G++ defines _GNU_SOURCE

Posted Mar 14, 2014 15:03 UTC (Fri) by cesarb (subscriber, #6266)
In reply to: G++ defines _GNU_SOURCE by jwakely
Parent article: Glibc feature test macros

> Once that's done C++ developers on GNU/Linux will probably find they've been unintentionally relying on _GNU_SOURCE for years and will have to learn how to use FTMs properly to get access to various POSIX APIs.

Both _GNU_SOURCE and _FILE_OFFSET_BITS=64 should be the default, since they are what most programmers expect.

_FILE_OFFSET_BITS=64, in particular, should always be set, or else it's easy to introduce latent bugs, which will show up once you have a file with more than 2G. I have seen that happen: the previous sysadmin(s) had a hand-compiled squid proxy, which one day stopped working, because the log file had grown to 2G (yeah, the log rotation was missing too). The immediate solution was to hand-rotate the log files; the real solution was to use a squid compiled correctly (and with working log rotation).

_GNU_SOURCE being missing, on the other hand, just leads to confused posts on programming forums questioning about missing functions. But I have yet to find a situation where it being set wasn't what the programmer wanted.

G++ defines _GNU_SOURCE

Posted Mar 14, 2014 15:21 UTC (Fri) by jwakely (subscriber, #60262) [Link]

> _GNU_SOURCE being missing, on the other hand, just leads to confused posts on programming forums questioning about missing functions.

Those people need to Read The Fantastic Article above :)

> But I have yet to find a situation where it being set wasn't what the programmer wanted.

http://gcc.gnu.org/PR60491
http://gcc.gnu.org/PR57582
http://gcc.gnu.org/PR59945
http://gcc.gnu.org/PR11196

These are all valid bug reports, the C++ standard library should not be polluting the global namespace and preventing users from declaring things called "clone" and "minor" which are not part of any C, C++ or POSIX standard. If you want those names there is a FTM you can use to get them, but 'g++ -std=c++11 -pedantic' should not force them on you.

_FILE_OFFSET_BITS default

Posted Mar 14, 2014 22:32 UTC (Fri) by giraffedata (guest, #1954) [Link] (6 responses)

_FILE_OFFSET_BITS=64, in particular, should always be set, or else it's easy to introduce latent bugs,

And you think that's a bigger risk than old programs, written in a day when file offsets greater than 32 bits were not even on the horizon, will have buffer overruns when you compile them for use on a new system?

_FILE_OFFSET_BITS default

Posted Mar 15, 2014 18:46 UTC (Sat) by kleptog (subscriber, #1183) [Link] (5 responses)

Perhaps I'm being insufficiently imaginative, but I can't think of a way that compiling an old program with _FILE_OFFSET_BITS=64 could lead to a buffer overrun. Do you have an example?

In my experience programs rarely do anything with fseek/ftell. So not compiling with _FILE_OFFSET_BITS=64 just means these programs fail reading/writing large files without there actually being an issue. Especially if a file is opened append-only, it breaks when reaching 2G, even though the program clearly doesn't care.

_FILE_OFFSET_BITS default

Posted Mar 15, 2014 21:58 UTC (Sat) by khim (subscriber, #9252) [Link] (4 responses)

Perhaps I'm being insufficiently imaginative, but I can't think of a way that compiling an old program with _FILE_OFFSET_BITS=64 could lead to a buffer overrun. Do you have an example?

Is this a joke? Something like this will overflow just fine with maliciously crafted file of size 2G+2*maxreqsize:
  int mail_size = lseek(fd, 0, SEEK_END);
  if (mail_size > MAX_MAIL_SIZE) {
    syslog(LOG_MAIL, "Mail is too big!\n");
    return;
  }
  lseek(fd, 0, SEEK_SET);
  read(fd, mail_bug, mail_size);
  …

Is it sloppy programming? Yeah, sure. But even if so it's safe with _FILE_OFFSET_BITS=32 and unusafe with _FILE_OFFSET_BITS=64.

In my experience programs rarely do anything with fseek/ftell.

Indeed. Most of the code is probably out there will probably work fine with _FILE_OFFSET_BITS=64. Unfortunately you only need one piece of code which does something sloppy with longs/ints/size_ts to have an exploit.

fseek/fteel use long with or without _FILE_OFFSET_BITS=64 but without _FILE_OFFSET_BITS=64 the actual size always fit in int while as with _FILE_OFFSET_BITS=64 it's easy to get values which are negative when stored in int but positive when stored in size_t. Hilarity ensues.

_FILE_OFFSET_BITS default

Posted Mar 15, 2014 22:41 UTC (Sat) by kleptog (subscriber, #1183) [Link]

Nice example, thanks for that. I wouldn't write it that way, but I can imagine that kind of code being written.

In this example the reading in of 2G of data will probably fail but I can see now that it's not the quick win I think it was.

_FILE_OFFSET_BITS default

Posted Mar 15, 2014 23:00 UTC (Sat) by giraffedata (guest, #1954) [Link] (1 responses)

I was not as imaginative as khim; I was just thinking of data structures that have 4 bytes for the file offset.

Like struct stat. If you used struct stat in a binary interface, and then recompiled with _FILE_OFFSET_BITS=64, your program would be using a struct stat64 while your communication partner would still be using stat and the file size field, not to mention the whole structure, would overrun.

If you just had your own data structure - maybe a database index - with a 4 byte offset field and assigned an off_t value to it, you'd end up with an arithmetic overrun, which is almost as bad as a buffer overrun. (And that's even before considering the signedness issue khim raised).

I'm sure there are people saying it was never right to assume the width of off_t or struct stat, and therefore essentially nobody did it, but that's hindsight. There was a time when the concept of a file larger than 2 GB was too ridiculous, and the nature of such a system too unpredictable, to justify mucking up your code with an overflow test. One could easily have reasoned that if files ever did get that big, the designers of the new interface could deal with the 32 bit assumption then (which they did).

_FILE_OFFSET_BITS default

Posted Mar 17, 2014 12:12 UTC (Mon) by roblucid (guest, #48964) [Link]

Indeed, I remember when I got my hands on the first 1GB IBM SCSI disk, which seemed unbelievably large, given ones just 3 years older at 300MB were in boxes the size of midi-tower PCs. Files, reaching sizes of 2GiB, seemed to be just in perpetuity broken.. we knew the UNIX FS imposed limits. In fact who wants to open a 2GiB text logfile with vi(1)? Even today with current RAM sizes, back then 32MiB was very large and text files were "small" and loaded into memory. BSD introduced nvi(1) which handled large files better.

_FILE_OFFSET_BITS default

Posted Sep 8, 2014 22:22 UTC (Mon) by wahern (subscriber, #37304) [Link]

The compiler will complain loudly about the narrowing.

And 64-bit programs have been around since the early 1990s. The first full-blown Linux distribution supporting 64-bit Alpha came out in 1995.

Compared to Windows, 64-bit issues on Linux have been few and far between. Part of that is because back then FOSS developers tended to be battle-hardened portability experts who took these issues seriously. And at least with 64-bit, that cautionary attitude still lingers.

In today's FOSS world where portability is looked down upon as a needless chore, where there's x86 and ARM and nothing else, and where IBM actually ships a Power chip in little-endian mode because it's apparently too difficult for their Linux customers to not conflate representation and value, I can't imagine we'd be able to make such a transition so smoothly again.