LWN: Comments on "The ups and downs of strlcpy()"

strncpy() history

rlhamil — Thu, 25 May 2017 01:36:00 +0000

I thought that was the case too (14 character filenames in a directory entry only null-terminated if shorter than that), but looking through the v7 code, I didn't really see much in the way of examples of that, certainly not in the kernel. Here's everything I found (this is running on v7 on an emulated PDP-11 that is so much faster than the real thing that it reminds me how painfully slow they were by comparison):

unixv7# time find /usr/src /usr/sys -type f -name '*.[ch]' -exec grep strncpy /dev/null {} \;
/usr/src/libc/gen/strncpy.c:strncpy(s1, s2, n)
/usr/src/cmd/crypt.c: strncpy(buf, pw, 8);
/usr/src/cmd/ed.c: strncpy(buf, keyp, 8);
/usr/src/cmd/login.c:#define SCPYN(a, b) strncpy(a, b, sizeof(a))
/usr/src/cmd/mkdir.c: strncpy(pname, d, slash);
/usr/src/cmd/atrun.c: strncpy(file, dirent.d_name, DIRSIZ);
/usr/src/cmd/xsend/lib.c: strncpy(buf, s, 10);
/usr/src/cmd/ranlib.c: strncpy(firstname, arp.ar_name, 14);

real 7.0
user 2.0
sys 4.5

Without looking further, three appear to deal with pathname components, three deal with other things, one is a macro (turns out it was for a not-necessarily-terminated utmp.ut_name field), and one is the definition of strlcpy.

The ups and downs of strlcpy()

netghost — Thu, 01 May 2014 18:32:43 +0000

You wrote such a long article to say the "up" and "down"s to justify "strlcpy()" not going into glibc, and the ONLY "down" side you points out in the article is "if you do not check the return value of strlcpy(), then you can lose data"? Sorry, this seems an very far-fetched argument to me, since how many C library functions can do EXACTLY what you want it to do WITHOUT checking return value and handling the error accordingly?

The ups and downs of strlcpy()

mirabilos — Mon, 28 Apr 2014 16:56:47 +0000

If you want that, *and* if you want to safely use strcpy(), you’d better use memcpy() anyway.

The ups and downs of strlcpy()

mirabilos — Mon, 28 Apr 2014 16:56:17 +0000

“apparently for similar reasons to their rejection from glibc”

Well, the list of POSIX 2008 authors, half of them looks like a who’s who of glibc developers…

The ups and downs of strlcpy()

rmongiovi — Mon, 04 Nov 2013 20:07:40 +0000

Let me see if I've got this straight. Linux doesn't get strlcpy, which would be quite useful for code portability, because it could cause you to silently lose data?

Well, golly. I guess we'd better undefine snprintf then.

Or maybe we just ought to admit that even though they aren't perfect, having a consistent suite of routines that work with strings and have a maximum length parameter is a useful thing.

No one is forcing anyone to use these calls. And no one is preventing anyone from checking for truncation after the call. But I will go on record as saying that it's effing annoying to have someone decide for me what's good for me. I'm a grownup who can make that decision for myself.....

Another

hummassa — Mon, 26 Aug 2013 01:18:41 +0000

I had

const char *stringcopy(char *dst, size_t size, const char *orig)

and it was used like:

for( const char *neworig = stringcopy(dst, size, orig); neworig; neworig = stringcopy(dst, size, neworig) ) {
  // do something with dst buffer of size
}

The ups and downs of strlcpy()

tjc — Sat, 24 Aug 2013 23:57:48 +0000

> #define strlcpy(d,n,s) snprintf((d),(n),"%s",(s))

I just tried this (better late than never!), and parameters 2 and 3 are reversed. Fore the sake of posterity, it should be:

#define strlcpy(d,s,n) snprintf((d),(n),"%s",(s))

The ups and downs of strlcpy()

quotemstr — Wed, 01 Aug 2012 15:53:01 +0000

Emacs already uses vfork if it's available. (Read the source.) Perhaps something else was wrong with that system.

The ups and downs of strlcpy()

quotemstr — Wed, 01 Aug 2012 15:50:55 +0000

> Right. So I'm a mere user on a system with 250 users

That's a rare edge case these days, like it or not. If you do regularly use such a system, it's the administrator's job to make sure system resources are adequate. The kernel is there to accurately account for system resources, not work around your sysadmin's snobbery.

> This despite the fact that vfork() cannot do many of the things you do between a fork() and exec()

Such as?

> hence the appallingly insane complexity of the interface

I don't think the interface is particularly complex. It's less complex than pthreads, certainly.

> the latter in particular is absolutely crackpot.

Do you really need to make it personal?

> Thank goodness I can turn overcommit off on my own systems

I think you mean "on".

The ups and downs of strlcpy()

nix — Tue, 31 Jul 2012 23:38:26 +0000

True, so the underlying overcommit problem isn't actually fixed by it, except inasmuch as it sometimes falls back to vfork() for you. It just makes your software much much uglier, and makes it work better on major platforms such as MMU-less embedded systems, the Hurd, and Cygwin.

The ups and downs of strlcpy()

khc — Tue, 31 Jul 2012 01:30:36 +0000

nevermind that posix_spawn() uses fork/exec on linux anyway

The ups and downs of strlcpy()

nix — Mon, 30 Jul 2012 08:46:16 +0000

Yeah, true. But if ptrace() was something everyone had to use, a replacement would have been merged by now, because ptrace() is just so odious in so very many ways. (Though the improvements in recent kernels have been substantial, and in maybe as few as five to ten years I'll be able to rely on them enough to actually use them in real software, which these days means "meant to be portable between Linux distros, including the dinosaur-era RHELs too many people insist on running their bleeding-edge software on". sigh.)

The ups and downs of strlcpy()

foom — Mon, 30 Jul 2012 01:51:54 +0000

Apparently not *any* alternative, or a new userspace API would have been merged upstream by now. :)

The ups and downs of strlcpy()

pbonzini — Sun, 29 Jul 2012 15:34:40 +0000

Exception handler information pointed to by FS only tracks a (possibly very small) subset of frames, since most frames do not install an exception handler.

The ups and downs of strlcpy()

nix — Sun, 29 Jul 2012 13:26:55 +0000

It would be pretty cool if you could spawn an empty process in a stopped state, and then poke at it from the parent for a bit (open up new file descriptors/etc) before causing it to exec a real subprocess.

You can do that with PTRACE_O_TRACEFORK or PTRACE_O_TRACEEXEC, but as with anything involving ptrace() there are so many tentacles that virtually any alternative is preferable.

The ups and downs of strlcpy()

foom — Sun, 29 Jul 2012 02:18:13 +0000

fork() is pretty evil, especially now that we have multi-threaded programs.

It would be pretty cool if you could spawn an empty process in a stopped state, and then poke at it from the parent for a bit (open up new file descriptors/etc) before causing it to exec a real subprocess.

Doing things that way would avoid all the memory accounting issues, the performance issue of copying the page table for no good reason, and the significant complication of not actually being allowed to do anything that's not async-signal-handler-safe between fork() and exec(). (And nearly nothing actually falls into that category!)

The ups and downs of strlcpy()

nix — Sat, 28 Jul 2012 10:43:16 +0000

The reason why my rant above sounds terribly specific is that this scenario actually happened to me. And kept happening to me, every week or so, for *years*, costing me perhaps time begging people to close other jobs down each time.

Needless to say the thought of rewriting (then X)Emacs's ferociously complex subprocess-handling infrastructure to use posix_spawn() never crossed my mind. (I tried vfork(), but that was clearly out of the question.)

The ups and downs of strlcpy()

nix — Sat, 28 Jul 2012 10:41:02 +0000

If you find fork failing, you should either add more swap (which won't actually get used, as you note, except in the worst case) or change your program to use vfork or posix_spawn instead, both of which don't have the intrinsic commit-accounting problems of fork.

Right. So I'm a mere user on a system with 250 users. fork() is failing in my Emacs so I can't start a shell (Emacs is much bigger than a shell). And your proposal for fixing this awful user interface failure is either to beg the sysadmin to add swap (I did, he said no, of course turning overcommit off was out of the question as this machine was running a database, never mind that it was a test instance that nobody was using, also it was 'like Solaris does it' and he liked Solaris) or spend time hacking at Emacs and every other program that uses fork()/exec() -- i.e. nearly everything in Unix -- so it no longer does?! This despite the fact that vfork() cannot do many of the things you do between a fork() and exec(), and posix_spawn() cannot do any of them unless the developer of posix_spawn() thought of it, hence the appallingly insane complexity of the interface? And this on a machine with almost no memory left? And this when I'm supposed to be getting something else done?

Your former proposal betrays your single-user roots. Your latter proposal betrays your ignorance of what makes fork()/exec() better than the Windows model in the first place. Neither is at all times practical: the latter in particular is absolutely crackpot.

Thank goodness I can turn overcommit off on my own systems.

The ups and downs of strlcpy()

quotemstr — Fri, 27 Jul 2012 19:44:48 +0000

> OOM happens when you're out of RAM *and* swap

OOM happens when you're out of commit. If you're doing it right, you paid for the commit for the pages you'll need when you loaded the image, so making the backtrace tables resident should still be possible.

> a good thing too, it's fantastically annoying as even very-much-not-OOM systems can find fork() failing because there's not enough swap to back every page that might get dirtied in the new address space, even if it's only going to exec() and throw them all away

I disagree: strict commit accounting makes a system more predictable in practice. If you find fork failing, you should either add more swap (which won't actually get used, as you note, except in the worst case) or change your program to use vfork or posix_spawn instead, both of which don't have the intrinsic commit-accounting problems of fork.

The ups and downs of strlcpy()

nybble41 — Thu, 26 Jul 2012 16:27:24 +0000

The strxcpy function isn't just a wrapper; it does all of the real work. The strxcpy_abort, strxcpy_truncate functions only run when an overflow condition is detected. This allows you to substitute your own preferred method of error-handling.

This is actually rather similar to the way exceptions are handled in Common Lisp or Scheme programs, except that the Lisp version would use dynamic variables rather than explicit arguments for the handler code, which results in less cluttered code.

(define (default-error-handler error-value) (abort))
(define current-error-handler (make-parameter default-error-handler))

(define (do-something)
  (... (if (ok? var) var ((current-error-handler) var)) ... ))

; aborts on error
(do-something)

; evaluates to #t on success, or #f on error
(let/cc return
  (parameterize ([current-error-handler (lambda _ (return #f))])
    (do-something)
    #t)

; uses "value" in place of var on error
(parameterize ([current-error-handler (lambda _ value)])
  (do-something))

Scheme-style parameters are attached to the current continuation, meaning that they're not only thread-safe, but that the bindings only affect the inner dynamic scope of the (parameterize) form, even in exceptional cases such as non-local return (like the middle example above) and even re-entry into a dynamic scope which was previously exited.

The ups and downs of strlcpy()

hppnq — Thu, 26 Jul 2012 11:51:49 +0000

It would not make sense to handle the relatively complex case of failing modules correctly but not string manipulation routines.

The ups and downs of strlcpy()

renox — Thu, 26 Jul 2012 08:53:08 +0000

I'm not sure what is the point of the wrapper function, calling directly strxcpy_abort, strxcpy_truncate, etc would be simpler..

That said, one size doesn't fit all so having different function is reasonable, the biggest issue is that there is no sane default behaviour..

The ups and downs of strlcpy()

nybble41 — Thu, 26 Jul 2012 01:28:53 +0000

Here's a suggestion (only partly sarcastic):

typedef size_t (*strxcpy_handler_t)(char *dst, const char *src, size_t size, void *data);

size_t strxcpy(char *dst, const char *src, size_t size, strxcpy_handler_t overflow_fn, void *overflow_data)
{
  char *p;
  const char *q;

  for (p = dst, q = src; *q; ++p, ++q)
  {
    if ((p - dst) >= size)
    {
      return overflow_fn(dst, src, size, overflow_data);
    }

    *p = *q;
  }

  /* get here only if strlen(src) < size */
  *p++ = '\0';
  return (p - dst);
}

size_t strxcpy_truncate(char *dst, const char *src, size_t size, void *data)
{
  if (size <= 0) abort();
  dst[size - 1] = '\0';
  return size + strlen(src + size);
}

size_t strxcpy_abort(char *dst, const char *src, size_t size, void *data)
{
  abort();
  return size;
}

if (strxcpy(dst, src, dst_size, strxcpy_truncate, NULL) >= dst_size) ...;
(void)strxcpy(dst, src, dst_size, strxcpy_abort, NULL);
(void)strxcpy(dst, src, dst_size, strxcpy_subst, "(input too long)");
/* ... */

The ups and downs of strlcpy()

smurf — Wed, 25 Jul 2012 20:58:53 +0000

Aborting is not "safe". Aborting is one of at least five ways to handle this particular error. Whether that is 'safe' depends on the context, i.e. your definition of that word.

The fundamental point is that you cannot know beforehand whether looking at the string twice is a [performance] problem, whether truncating (with or without fixing incomplete UTF-8 codes) is better than not starting to fill the buffer in the first place, whether calling abort() is a good idea (I'd say that if you are a library, it almost never is), whether to return something negative or the new length or the source length or …, and a host of related questions, all of which do not lend themselves to consensus answers. As this discussion shows quite clearly, IMHO.

My point is that, with the sole exception of leaving the destination buffer undisturbed when the source won't fit, any of the aforementioned behaviors can be implemented with a reasonably-trivial O(1) wrapper around strlcpy(). Therefore, keeping strlcpy() out of libc is … kindof stupid. Again, IMHO.

Instead, people are told to use strncpy(). Which they'll do incorrectly. Let's face it, running off the end of a string into la-la land is always worse than truncating it.

The ups and downs of strlcpy()

bronson — Wed, 25 Jul 2012 19:13:12 +0000

Agreed. On top of that, I don't think anybody's saying that strlcpy must always abort, just that it should be safe by default. If you don't want safety, no problem, turn it off and and see what you get.

It would be nice if safety was always the default. Alas, libc (as standardized) only started thinking this way relatively recently.

The ups and downs of strlcpy()

nix — Wed, 25 Jul 2012 17:17:04 +0000

Yep. That's why frame pointers should be *enableable*. It doesn't mean they should be on by default.

The ups and downs of strlcpy()

mmorrow — Wed, 25 Jul 2012 15:55:58 +0000

9c9
<   const size_t n = 10
---
>   const size_t n = 10;

The ups and downs of strlcpy()

mmorrow — Wed, 25 Jul 2012 15:40:54 +0000

Backtracing on x86_64 is actually quite reasonable. Here are two methods:

#if defined(USE_BACKTRACE)
/*
  -DUSE_BACKTRACE -rdynamic
  (-rdynamic for backtrace_symbols)
*/
#include <execinfo.h>
void print_trace(void)
{
  const size_t n = 10
  void *array[n];
  size_t size = backtrace(array,n);
  char **strings = backtrace_symbols(array,size);
  for(size_t i = 0; i < size; i++)
    fprintf(stderr,"%s\n",strings[i]);
  free(strings);
}
#elif defined(USE_LIBUNWIND)
/*
  -DUSE_LIBUNWIND -lunwind-x86_64
*/
#include <libunwind.h>
void print_trace(void)
{
  unw_cursor_t cur;
  unw_context_t cxt;
  unw_getcontext(&cxt);
  unw_init_local(&cur,&cxt);
  while(unw_step(&cur) > 0)
  {
    unw_word_t off, pc;
    char fname[64] = {[0] = '\0'};
    unw_get_reg(&cur,UNW_REG_IP,&pc);
    unw_get_proc_name(&cur,fname,sizeof(fname),&off);
    printf("%p: (%s+0x%x) [%p]\n",pc,fname,off,pc);
  }
}
#endif

The ups and downs of strlcpy()

paulj — Wed, 25 Jul 2012 15:11:18 +0000

One reason is debugging stack corruption, where normal tools may not give meaningful backtraces. With frame-pointers, you can easily figure out where earlier, uncorrupted, frames really are, and figuring out why it crashed.

The ups and downs of strlcpy()

nix — Wed, 25 Jul 2012 13:45:45 +0000

If you have swap free somewhere, you are not in an OOM situation. OOM happens when you're out of RAM *and* swap (or when you hit your RLIMIT_AS boundaries, I suppose).

If you are in overcommit state 2 with an overcommit ratio of 0, you may be right, but neither of these are the default -- and even then, I don't believe Linux reserves swap pages for every in-memory page, like Solaris does (a good thing too, it's fantastically annoying as even very-much-not-OOM systems can find fork() failing because there's not enough swap to back every page that might get dirtied in the new address space, even if it's only going to exec() and throw them all away).

The ups and downs of strlcpy()

NAR — Wed, 25 Jul 2012 12:30:33 +0000

Why would aborting a single process shut down the whole gaming server? I mean if you have a sufficiently large and complicated software, there will be terminal bugs there that you have to handle (e.g. have a lightweight supervisor process that starts a separate server for each new client). But if you already handle these problems, then you might as well crash on purpose in the worker processes.

The ups and downs of strlcpy()

quotemstr — Wed, 25 Jul 2012 02:19:37 +0000

> In a severe OOM situation, it's very likely that you won't be able to map more pages in, and that not much of it is going to be mapped in at any given time.

If you run your system in a sane configuration, the kernel should have swap somewhere it can use to evict the pages it'll need to hardfault the pages from the unwind section. It might be slow, but it'll work. If you run overcommit, there's no such guarantee, but if you run overcommit, why the hell are you complaining about OOM behavior?

The ups and downs of strlcpy()

bronson — Wed, 25 Jul 2012 02:00:47 +0000

You don't need a managed language to make strlcpy abort instead of truncating.

I agree with the rest of your comment.

The ups and downs of strlcpy()

nix — Tue, 24 Jul 2012 23:19:09 +0000

Because the ABI allows it, because it still provides a performance improvement (somewhere between 1% and 5%, not insignificant, though well below the 8--12% I've seen reported for x86-32), and because it's useless -- everything from GDB through libgcj and now I find even glibc backtrace() uses the DWARF unwinder tables instead. Why maintain a 'feature' which costs a register and adds runtime overhead to every function call when nobody needs it?

The ups and downs of strlcpy()

nix — Tue, 24 Jul 2012 23:10:08 +0000

Sorry, I misspoke, was thinking of other DWARF sections. .eh_frame is loadable, but is, of course, like all ELF sections, *mapped* in. In a severe OOM situation, it's very likely that you won't be able to map more pages in, and that not much of it is going to be mapped in at any given time.

(I must have been unlucky with backtrace(). It's never been willing to do more than coredump for me without frame pointers. Mind you I haven't tried it for years because I was so sure it was broken :) time to try it out in my next debugging blitz.)

The ups and downs of strlcpy()

foom — Tue, 24 Jul 2012 20:37:11 +0000

The eh_frame section *is* loaded by default. And the backtrace function in glibc uses it, even.

So, no, it's not hard to produce a backtrace. You just call the function.

The ups and downs of strlcpy()

Cyberax — Tue, 24 Jul 2012 19:48:33 +0000

Windows does both. It uses the FS register pointing to thread information block to track frame pointers (since Windows uses SJLJ exceptions) and it also does crashdumps that contain offending threads' stacks. Minidumps are usually quite small.

The ups and downs of strlcpy()

renox — Tue, 24 Jul 2012 18:16:57 +0000

> x86-64 GCC disables them by default (as does x86-32 modern GCC)

I wonder why the GCC developpers chose this default behaviour, x86-64 isn't register starved like x86-32.

The ups and downs of strlcpy()

nix — Tue, 24 Jul 2012 17:16:55 +0000

Producing a backtrace of any sort without frame pointers is very hard, and x86-64 GCC disables them by default (as does x86-32 modern GCC). To produce backtraces on such systems, you need DWARF debugging info -- though perhaps the exception frame section would serve the same purpose, though of course it too is not loaded by default. I suppose you could write the entire stack to disk on OOM (as long as it's not too big -- coredumping may be much harder, as if you're out of memory the full coredump is likely to be huge).

The ups and downs of strlcpy()

kleptog — Tue, 24 Jul 2012 06:37:25 +0000

There are programs that attempt to recover from OOM, PostgreSQL for example. It has a pre-allocated area which it uses to create the error message to send to the client and the rip-cord allocator will quickly release any memory allocated to the current query context.

It's not perfect of course, if the client is using SSL then you have to rely on the SSL library to not do anything silly but it's worked every time for me. On the client you get a nice message along the lines of "server ran out of memory". Your transaction is aborted, but the rest of the server is still running.

This obviously only works if malloc() returns NULL, so memory overcommit needs to be off. OOM during stack growth is uncatchable, you can only try to mitigate the risk bit keeping your stack small.

I just wanted to point out that it is possible to create code that handles OOM, and it's not helpful if libraries assume they can just die in that case.