LWN: Comments on "Glibc feature test macros"

Glibc feature test macros

troglobit — Wed, 10 Sep 2014 05:56:47 +0000

At first when I read this, I thought http://xkcd.com/927/ ... now I hope
I've interpreted it correctly, the old -D_BSD_SOURCE will still work right?

_FILE_OFFSET_BITS default

wahern — Mon, 08 Sep 2014 22:22:28 +0000

The compiler will complain loudly about the narrowing.

And 64-bit programs have been around since the early 1990s. The first full-blown Linux distribution supporting 64-bit Alpha came out in 1995.

Compared to Windows, 64-bit issues on Linux have been few and far between. Part of that is because back then FOSS developers tended to be battle-hardened portability experts who took these issues seriously. And at least with 64-bit, that cautionary attitude still lingers.

In today's FOSS world where portability is looked down upon as a needless chore, where there's x86 and ARM and nothing else, and where IBM actually ships a Power chip in little-endian mode because it's apparently too difficult for their Linux customers to not conflate representation and value, I can't imagine we'd be able to make such a transition so smoothly again.

Best practice?

wahern — Thu, 03 Apr 2014 05:54:06 +0000

I compile and test all my projects on Linux, NetBSD, FreeBSD, OpenBSD, OS X, and Solaris. And I can say without a doubt that the last thing you should do, if you're writing portable programs, is to fiddle with POSIX or XOPEN feature macros!

Why?

Because most portable code actually uses one or more non-standard interfaces, and good luck making those routines visible again if you did something like -D_POSIX_C_SOURCE=200809L. FTM systems are wildly different on various platforms.

Also, many systems don't support the latest POSIX standard. For example, Solaris only supports POSIX up to 2001, even though it actually implements many of the new interfaces. So if you define _POSIX_C_SOURCE (to any value), you paradoxically _lose_ a ton of "POSIX" interfaces.

Or look at calls like mkstemp. It's in POSIX 2008, but previously it was only an X/Open extension. On Solaris there's no easy way to get it; if you define _POSIX_C_SOURCE, it disappears. If you define _XOPEN_SOURCE, you get it, but you lose the POSIX API. If you try to define both _POSIX_C_SOURCE and _XOPEN_SOURCE, the system headers barf.

Nope. In my experience, you define _GNU_SOURCE on Linux, and on every other system, leave well enough alone. Linux is the only widely used system that has a strict environment by default, AFAIK. Every other system exposes all their interfaces (excepting conflicting ones) by default, which means the last thing you want to do is define _POSIX_C_SOURCE or _XOPEN_SOURCE, unless you're a glutton for punishment, or are on some strict-POSIX crusade. Sometimes you want to define macros like _REENTRANT, _THREAD_SAFE, or the 64-bit FS macros, but those are orthogonal to the larger standards macros, and do not destroy visibility to other interfaces.

There's no substitute for testing your code on various systems. Virtual machines makes this easy-peasy. You're life is easier if you avoid FTMs. Then you can use convenient, de-facto standard APIs. It's 2014. Most of the interfaces you want are supported everywhere, but far fewer systems support the standard they were most recently specified in.

C standards are a slightly different issue, but fortunately you can have a compiler on a single platform warn you about certain constructs. Also, there's always the option of just using GCC everywhere. I try to support GCC, clang, and Solaris Studio (which require no special effort to get ahold of other than installing the OS), but you have to draw the line somewhere, because while being "portable" doesn't necessarily mean being less productive, it can make you less productive if you do it the wrong way.

Portability is a process, not a product. Most projects can be made way more portable than they are with little effort, especially if early in the development process you test-test-test, and don't code yourself into a corner.

Best practice?

bernat — Tue, 01 Apr 2014 16:05:38 +0000

What's the best practice when we want to write portable code?

Currently, each time I want to use, for example, asprintf, I have a define _GNU_SOURCE at the top of the file. Should I use -D_GNU_SOURCE=1 instead accross my whole project? Or should I stay away _GNU_SOURCE and try to use a more conservative _POSIX_C_SOURCE intead (and wrap asprintf in a dedicated file).

Glibc feature test macros

nix — Mon, 24 Mar 2014 18:04:10 +0000

They're both made available (via transparent unions).

The only place where you have to suck things up is the BSD-specific getpgrp(), but that hasn't actually worked as in BSD for ages, even if you built with -D_BSD_SOURCE. So we can safely say that this is basically unused.

_FILE_OFFSET_BITS default

roblucid — Mon, 17 Mar 2014 12:12:25 +0000

Indeed, I remember when I got my hands on the first 1GB IBM SCSI disk, which seemed unbelievably large, given ones just 3 years older at 300MB were in boxes the size of midi-tower PCs. Files, reaching sizes of 2GiB, seemed to be just in perpetuity broken.. we knew the UNIX FS imposed limits. In fact who wants to open a 2GiB text logfile with vi(1)? Even today with current RAM sizes, back then 32MiB was very large and text files were "small" and loaded into memory. BSD introduced nvi(1) which handled large files better.

Glibc feature test macros

kleptog — Sun, 16 Mar 2014 13:45:08 +0000

The only time I've really had to fiddle with the feature test macros was when dealing with the structure definitions in netinet/ip.h, netinet/tcp.h and netinet/udp.h. If BSD_SOURCE is enabled the fields are called th_sport, uh_sport, etc. Otherwise the names are different. This usually happens when you've downloaded some source from the internet.

Given they're talking about eliminating the BSD/SysV macros, does that mean GLibc will pick one of the two and if you're using the other you'll just have to suck it up?

_FILE_OFFSET_BITS default

giraffedata — Sat, 15 Mar 2014 23:00:29 +0000

I was not as imaginative as khim; I was just thinking of data structures that have 4 bytes for the file offset.

Like struct stat. If you used struct stat in a binary interface, and then recompiled with _FILE_OFFSET_BITS=64, your program would be using a struct stat64 while your communication partner would still be using stat and the file size field, not to mention the whole structure, would overrun.

If you just had your own data structure - maybe a database index - with a 4 byte offset field and assigned an off_t value to it, you'd end up with an arithmetic overrun, which is almost as bad as a buffer overrun. (And that's even before considering the signedness issue khim raised).

I'm sure there are people saying it was never right to assume the width of off_t or struct stat, and therefore essentially nobody did it, but that's hindsight. There was a time when the concept of a file larger than 2 GB was too ridiculous, and the nature of such a system too unpredictable, to justify mucking up your code with an overflow test. One could easily have reasoned that if files ever did get that big, the designers of the new interface could deal with the 32 bit assumption then (which they did).

_FILE_OFFSET_BITS default

kleptog — Sat, 15 Mar 2014 22:41:41 +0000

Nice example, thanks for that. I wouldn't write it that way, but I can imagine that kind of code being written.

In this example the reading in of 2G of data will probably fail but I can see now that it's not the quick win I think it was.

_FILE_OFFSET_BITS default

khim — Sat, 15 Mar 2014 21:58:34 +0000

Perhaps I'm being insufficiently imaginative, but I can't think of a way that compiling an old program with _FILE_OFFSET_BITS=64 could lead to a buffer overrun. Do you have an example?

Is this a joke? Something like this will overflow just fine with maliciously crafted file of size 2G+2*maxreqsize:
  int mail_size = lseek(fd, 0, SEEK_END);
  if (mail_size > MAX_MAIL_SIZE) {
    syslog(LOG_MAIL, "Mail is too big!\n");
    return;
  }
  lseek(fd, 0, SEEK_SET);
  read(fd, mail_bug, mail_size);
  …

Is it sloppy programming? Yeah, sure. But even if so it's safe with _FILE_OFFSET_BITS=32 and unusafe with _FILE_OFFSET_BITS=64.

In my experience programs rarely do anything with fseek/ftell.

Indeed. Most of the code is probably out there will probably work fine with _FILE_OFFSET_BITS=64. Unfortunately you only need one piece of code which does something sloppy with longs/ints/size_ts to have an exploit.

fseek/fteel use long with or without _FILE_OFFSET_BITS=64 but without _FILE_OFFSET_BITS=64 the actual size always fit in int while as with _FILE_OFFSET_BITS=64 it's easy to get values which are negative when stored in int but positive when stored in size_t. Hilarity ensues.

_FILE_OFFSET_BITS default

kleptog — Sat, 15 Mar 2014 18:46:47 +0000

Perhaps I'm being insufficiently imaginative, but I can't think of a way that compiling an old program with _FILE_OFFSET_BITS=64 could lead to a buffer overrun. Do you have an example?

In my experience programs rarely do anything with fseek/ftell. So not compiling with _FILE_OFFSET_BITS=64 just means these programs fail reading/writing large files without there actually being an issue. Especially if a file is opened append-only, it breaks when reaching 2G, even though the program clearly doesn't care.

Glibc feature test macros

jwakely — Fri, 14 Mar 2014 23:09:52 +0000

Many of them already do, so you can e.g. include <stdio.h> and only get the definition of FILE, then include it again and get everything else. Glibc is weird :)

_FILE_OFFSET_BITS default

giraffedata — Fri, 14 Mar 2014 22:32:05 +0000

_FILE_OFFSET_BITS=64, in particular, should always be set, or else it's easy to introduce latent bugs,

And you think that's a bigger risk than old programs, written in a day when file offsets greater than 32 bits were not even on the horizon, will have buffer overruns when you compile them for use on a new system?

Glibc feature test macros

mathstuf — Fri, 14 Mar 2014 19:09:48 +0000

Wouldn't headers which use features.h then need to support being included multiple times so that extra functions could be picked up with the new feature macros?

G++ defines _GNU_SOURCE

jwakely — Fri, 14 Mar 2014 15:21:47 +0000

> _GNU_SOURCE being missing, on the other hand, just leads to confused posts on programming forums questioning about missing functions.

Those people need to Read The Fantastic Article above :)

> But I have yet to find a situation where it being set wasn't what the programmer wanted.

http://gcc.gnu.org/PR60491
http://gcc.gnu.org/PR57582
http://gcc.gnu.org/PR59945
http://gcc.gnu.org/PR11196

These are all valid bug reports, the C++ standard library should not be polluting the global namespace and preventing users from declaring things called "clone" and "minor" which are not part of any C, C++ or POSIX standard. If you want those names there is a FTM you can use to get them, but 'g++ -std=c++11 -pedantic' should not force them on you.

G++ defines _GNU_SOURCE

cesarb — Fri, 14 Mar 2014 15:03:19 +0000

> Once that's done C++ developers on GNU/Linux will probably find they've been unintentionally relying on _GNU_SOURCE for years and will have to learn how to use FTMs properly to get access to various POSIX APIs.

Both _GNU_SOURCE and _FILE_OFFSET_BITS=64 should be the default, since they are what most programmers expect.

_FILE_OFFSET_BITS=64, in particular, should always be set, or else it's easy to introduce latent bugs, which will show up once you have a file with more than 2G. I have seen that happen: the previous sysadmin(s) had a hand-compiled squid proxy, which one day stopped working, because the log file had grown to 2G (yeah, the log rotation was missing too). The immediate solution was to hand-rotate the log files; the real solution was to use a squid compiled correctly (and with working log rotation).

_GNU_SOURCE being missing, on the other hand, just leads to confused posts on programming forums questioning about missing functions. But I have yet to find a situation where it being set wasn't what the programmer wanted.

G++ defines _GNU_SOURCE

jwakely — Thu, 13 Mar 2014 16:56:52 +0000

> I wonder if modules (C++1y) could help dealing with this problem?

Maybe. I'd like to think the answer is yes, but I bet I can solve the problem the old fashioned way before there's a modules proposal, let alone an implementation ;-)

> Is it true that with modules libstdc++ could be compiled with _GNU_SOURCE enabled, but dependencies of libstdc++ won't be exposed (don't pollute global namespace) to user, unless they affect signature of functions or layout of classes?

You're asking about the properties of a non-existent feature, so it's hard to give an accurate answer, but yes, I hope that will be true.

Glibc feature test macros

ejr — Thu, 13 Mar 2014 15:43:04 +0000

If anyone wants to see true feature test macro insanity, check out AIX... I truly appreciate the glibc FTM approach and this latest work, thank you!

G++ defines _GNU_SOURCE

sorokin — Thu, 13 Mar 2014 14:15:42 +0000

I wonder if modules (C++1y) could help dealing with this problem?

Is it true that with modules libstdc++ could be compiled with _GNU_SOURCE enabled, but dependencies of libstdc++ won't be exposed (don't pollute global namespace) to user, unless they affect signature of functions or layout of classes?

Glibc feature test macros

mchapman — Thu, 13 Mar 2014 11:17:38 +0000

> Notwithstanding occasional bugs in the documented FTM requirements, the problem is always the same: FTMs must be defined before including any header files. The reason for this requirement is that various header files may implicitly include other header files, and any of those files may in turn include <features.h>. If a macro is defined after <features.h> is included, then it has no effect, since that header file (like most library header files) includes a guard macro that ensures that the contents of the file are processed only on the first inclusion of the file.

I wonder if it would be possible for <features.h> to detect that it was being re-included with a different set of FTMs enabled -- implying that some macros were defined (or redefined) after the first inclusion of the header. It could emit an appropriate #warning in this case.

Unfortunately, looking at how <features.h> is implemented I don't think this would be quite straight-forward. You wouldn't just be able to capture and compare a snapshot of the macros' values, as some of them are actually used to set the values of others. But it would be a worthwhile change if it cuts down on needless bug reports.

G++ defines _GNU_SOURCE

jwakely — Thu, 13 Mar 2014 10:53:29 +0000

To confuse things a bit, G++ unconditionally defines _GNU_SOURCE on systems using glibc, because the C++ standard library wants to be able to make use of GNU extensions, so uses the "give me everything" approach.

This unfortunately means that standard C++ headers such as <algorithm> pollute the global namespace with dozens of non-standard names (and horrible macros like "minor"!)

I'm going to fix this in the next few months by replacing _GNU_SOURCE with more fine-grained use of FTMs, and (I hope) some help from the glibc maintainers to redeclare the handful of GNU extensions libstdc++ actually needs using alternative names.

Once that's done C++ developers on GNU/Linux will probably find they've been unintentionally relying on _GNU_SOURCE for years and will have to learn how to use FTMs properly to get access to various POSIX APIs.

Glibc feature test macros

ncm — Thu, 13 Mar 2014 10:28:41 +0000

That cleanup must have been mind-numbing work. Thanks, Joe.