readdir_r() deprecated?

Application code isn't supposed to use NAME_MAX, but fpathconf(dirfd(dp), _PC_NAME_MAX). Indeed, both Solaris and Linux (or, at least, glibc GNU folks) push developers away from using NAME_MAX, PATH_MAX, etc, out of a dislike for fixed limits on filesystem names. Solaris has some convoluted syscall APIs for returning various data structures with dynamic (and effectively unbounded) sizes from kernel to user space.

I think that's misguided. For better or worse, a 255-character limit on traditional filesystems is here to stay. And while we can all easily _imagine_ scenarios where we could use (or abuse) longer file names, we all know that it's not very practical. (We could always up it to 1024 and call it a day.) In the end I think the complexities outweigh any possible benefit, especially given that the software interface with the traditional filesystem is effectively moving further and further down the stack. That doesn't make it any less important, but it means stability and consistency should be emphasized over flexibility. And in any event, other environments like Windows, OS X, and POSIX itself have much more onerous limitations to their interfaces, either in terms of length or content of filenames. So this isn't an area where sane people would attempt to push the limits when solving problems, at least not in a way which relies on filenames for storing arbitrary data.

Given the current security realities, I think it's better to focus on stabilizing and simplifying the syscall and libc landscape. While interfaces like readdir hide the complexity from application code, that's beside the point--there have been plenty of bugs and exploits in both glibc and the kernel related to their interface implementations. Arbitrary limits abound in software and that will never change. We shouldn't try to support dynamic limits at all costs, especially when the relative benefit is so miniscule and speculative.

That said, it's obviously far easier to use readdir than readdir_r. And, yeah, most implementations are thread safe. But at this point Linux is the only OS keeping up-to-date with POSIX, largely because Red Hat employees dominate the committee. The BSDs are falling further behind, and commercial vendors like Solaris and AIX effectively stopped caring altogether. Both AIX and Solaris seem to have focused on Linux/glibc compatibility (whether or not a POSIX addition) and other practical and usability concerns.

The thing is, if people never tried compiling their software on, e.g., Solaris, they would have never realized that NAME_MAX is not always defined. And while it's rare to see software that actually uses readdir_r, that only proves that few people paid attention to the issues to begin with. While we can sometimes improve the situation, as in this case by making readdir thread safe, we'll never be able to excise all the code that directly or indirectly relies on NAME_MAX, PATH_MAX, and similar limits. Heck, those constants influenced (and in some cases were influenced by) fixed limits in various RFCs. We'll never be able to leave them behind. I'd rather see us embrace them so we can work to simplify code bases.

I'm making far too much of this than it deserves. Mostly I'm responding to the opinion that seems to be widely held in some circles that we should support arbitrary limits in filesystem paths and similar kernel interfaces. While I don't agree with David Wheeler that we should restrict filenames to, e.g., UTF-8-encoding graphic characters (with some additional constraints on whitespace), I basically agree with the general sentiment. Secure interfaces are simple interfaces; simple interfaces often require drawing strict lines; and strict lines are sometimes arbitrary. So be it. More oever, in this area the lines have already been drawn by historical practice, and intentionally or unintentionally baked into the architecture of almost all software and software standards.