User: Password:
|
|
Subscribe / Log in / New account

Development

Glibc feature test macros

March 12, 2014

This article was contributed by Michael Kerrisk.

As evidenced by the frequency of misreported bugs to the Linux man-pages project, feature test macros (FTMs) would appear to be one of the topics that are most widely misunderstood by C programmers. The fundamental idea of FTMs is simple, but there are quite a few subtle details to master, and at least one pitfall that regularly traps programmers. This article will look at how FTMs are used in the GNU C library (glibc) and how to avoid some common problems.

FTMs are a technique used to control the definitions exposed by C header files. For example, in the clock_getres(3) manual page, we see the following text in the SYNOPSIS section:

    Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

       clock_getres(), clock_gettime(), clock_settime():
              _POSIX_C_SOURCE >= 199309L

This text tells us that the <time.h> header file will expose the declarations of these three system calls (or, more precisely, their C library wrapper functions) if _POSIX_C_SOURCE is defined with the value 199309L or greater. Beginning with man-pages version 2.64 in 2007, I started adding FTM annotations such as the above to various manual pages. By now, all of the pages that require such annotations (some 350) have them. (Any omissions or errors are bugs that should be reported.)

Feature test macros and formal standards

There are several different FTMs, and a full list can be found in the feature_test_macros(7) manual page. Many of the FTMs fall into two broad classes. The first of these are the FTMs that relate to various formal standards—for example, the C language standards or Unix standards such as POSIX and the Single Unix Specification. These macros have names such as _ISOC99_SOURCE, _POSIX_C_SOURCE, and _XOPEN_SOURCE. The purpose of these macros is to cause a header file to expose definitions in a manner that conforms to a particular standard.

To take an example, we can define _POSIX_C_SOURCE with different values in order to have header files expose definitions and declarations associated with different versions of POSIX. This FTM was specified in the first (1990) edition of POSIX; defining it with the value 1 causes various header files to expose definitions associated with that POSIX version. As that standard evolved, a more coherent numbering scheme was devised, based on the year and month that the standard was released. Thus, the 1993 POSIX.1b (realtime) standard specifies that the value 199309L should cause header files to expose definitions relating to that standard. Similarly, the value 200112L causes definitions associated with the POSIX.1-2001 standard to be exposed, and the value 200809L causes definitions associated with the POSIX.1-2008 standard to be exposed.

In the snippet from the clock_getres(2) manual page shown above, the value 199309L is required because those three APIs were first standardized in POSIX.1b (1993). Standards tend to be cumulative, so that most of the APIs specified in POSIX.1b were also specified in POSIX.1-2001 and POSIX.1-2008, which is why defining _POSIX_C_SOURCE with any value greater than 199309L will also cause the POSIX.1b declarations to be exposed.

There are a number of other FTMs associated with various standards. For example, defining _ISOC99_SOURCE (with any value) causes header files to expose definitions associated with the C99 standard. The _XOPEN_SOURCE macro causes header files to expose definitions associated with various versions of the Single Unix Specification (SUS): 1 (or greater) to get definitions for SUSv1, 500 (or greater) to get definitions associated with SUSv2, 600 (or greater) to get definitions associated with SUSv3, and so on.

The "00" numbering scheme used for the _XOPEN_SOURCE values relates to the XPG (X/Open Portability Guide) standards out of which SUS grew. Thus, SUSv2 is also (occasionally) known as XPG5, and so the number 500 was used for the corresponding FTM. The next SUS release, version 3, adopted the value 600, and SUSv4 adopted the value 700. Adding a couple of zeroes to the numbers presumably permits the standard to assign meaning to intermediate values between those numbers, for something like minor revisions of the standard, although in practice no intermediate values have so far been defined.

Feature test macros and implementation standards

A second class of glibc-specific FTMs relate to historical implementation standards, in particular System V and BSD. Defining one of these macros was sometimes helpful when porting older software from one of those implementations to Linux.

Defining the macro _BSD_SOURCE (with any value) causes header files to expose definitions derived from BSD. Thus, the setgroups(2) manual page says that we must define _BSD_SOURCE to obtain the declaration of that system call:

    Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

       setgroups(): _BSD_SOURCE

This corresponds to the fact that setgroups() is originally a BSD invention that is not specified in any formal standard.

Explicitly defining the _BSD_SOURCE macro has one other effect: in a very few cases where standards conflict, it causes BSD definitions to be favored. One example of such a conflict is the getpgrp() function, which exists in two forms in glibc: a POSIX version and a BSD version. By default, <unistd.h> exposes the POSIX version, unless _BSD_SOURCE is explicitly defined.

Analogously, the _SVID_SOURCE macro is used to obtain some definitions derived from the System V Interface Definition, a vendor-defined standard for System V.

The _GNU_SOURCE macro is an FTM invented by the glibc developers. Defining this macro (with any value) serves two main purposes. First, this macro is commonly used to guard Linux-specific and GNU-specific definitions that are part of standard header files. Thus, to get the declaration of the Linux-specific ppoll() system call from <poll.h>, which also provides the poll() call specified by various Unix standards, we must define _GNU_SOURCE.

The second purpose of _GNU_SOURCE is to say "give me everything". Defining this macro causes almost all of the other FTMs to be defined (with their maximum meaningful values), with the result that header files expose (almost) all of the definitions they contain.

It's complicated

Standards evolve and sometimes conflict; glibc has evolved to follow the standards, and occasionally changed its use of FTMs to fix bugs or be more consistent. Furthermore, sometimes standards are subtractive, meaning that a standard can remove an interface. A notable example occurred in the C11 standard, which removed the dangerous gets() function. Thus, recent versions of glibc don't expose this function if the _ISOC11_SOURCE FTM is defined.

The syntax used in the man pages reflects these complexities. Thus, for example, in the dprintf(3) page we see:

    Feature Test Macro Requirements for glibc (see feature_test_macros(7)):

       dprintf(), vdprintf():
           Since glibc 2.10:
               _XOPEN_SOURCE >= 700 || _POSIX_C_SOURCE >= 200809L
           Before glibc 2.10:
               _GNU_SOURCE

This tells us that before glibc version 2.10, it was necessary to define _GNU_SOURCE to obtain the declarations of these functions. However, these functions were eventually standardized in the combined POSIX.1-2008/SUSv4 standard, and glibc 2.10 and later adjusted to that fact, so that now it suffices either to define _XOPEN_SOURCE with a value greater than or equal to 700 or to define _POSIX_C_SOURCE with a value greater than or equal to 200809L.

The <features.h> header file

The logic dealing with FTMs is implemented within the <features.h> header file. That header file defines various internal macros which are used within each of the other glibc header files to control which definitions are exposed by the header file. (Programs never need to explicitly include <features.h>: it is implicitly included by each glibc header file that makes use of FTMs.)

The internal macros have names such as __USE_GNU (for _GNU_SOURCE), __USE_BSD (for _BSD_SOURCE), __USE_SVID (for _SVID_SOURCE), and __USE_MISC (defined if either _BSD_SOURCE or _SVID_SOURCE is defined). FTMs that can take different values are implemented as multiple internal macros. So, for example, there are __USE_POSIX199309, __USE_POSIX199506, __USE_XOPEN2K, and __USE_XOPEN2K8, each set when _POSIX_C_SOURCE is greater than or equal to, respectively, 199309L, 199506L, 200112L, and 200809L.

Certain FTMs are defined by default, principally _BSD_SOURCE, _SVID_SOURCE, and _POSIX_C_SOURCE with the value 200909L (which causes the definitions for the POSIX.1-2008 base standard to be exposed).

Individual macros can be explicitly defined, using either #define in the C source code or -D directives on the compiler command line. Multiple definitions are additive. However, there is a potential surprise: in most cases, if any FTM is explicitly defined, then the default FTMs are not defined by default, and they too must be explicitly defined. The details of the default definitions, as well as certain other implicit definitions that take place when certain FTMs are explicitly defined, have varied across glibc versions. Most of those details can be found in the feature_test_macros(7) manual page.

A regular pitfall

Over the years, probably the single most common recurrent bug report to the man-pages project is the claim that defining a particular FTM does not expose a definition as documented in a manual page. Often, the reporter has inspected the header files, and discovered that explicitly defining one of glibc's internal macros such as __USE_MISC (which should never be done outside of the glibc code), causes the desired definition to be exposed, and so the report includes a claim that the manual page should document the use of that macro.

Notwithstanding occasional bugs in the documented FTM requirements, the problem is always the same: FTMs must be defined before including any header files. The reason for this requirement is that various header files may implicitly include other header files, and any of those files may in turn include <features.h>. If a macro is defined after <features.h> is included, then it has no effect, since that header file (like most library header files) includes a guard macro that ensures that the contents of the file are processed only on the first inclusion of the file.

Changes in glibc 2.19 and 2.20

The original intent seems to have been that, within each of the glibc header files that employs FTMs, only one of the __USE_* internal macros should govern the exposure of any particular definition. Additionally, the macros should not be used in nested #ifdef directives. An inspection of the glibc header files quickly shows that the reality is far from the intent, a situation that led Roland McGrath to suggest that it was time for a major cleanup to bring things back to the intended situation. Roland thought that task could be simplified by eliminating the _BSD_SOURCE and _SVID_SOURCE FTMs, which, although they had a purpose historically, have ceased to be useful these days. Anymore, he said, the only macros that are needed for modern source code are those that relate to formal standards plus _GNU_SOURCE.

Joseph Myers duly obliged with a series of patches to implement the first steps in this work. The conservative approach encouraged by Roland meant that the deprecation of the _BSD_SOURCE and _SVID_SOURCE FTMs is taking place across two glibc versions. Version 2.19 of glibc added a new FTM, _DEFAULT_SOURCE. Defining this macro causes the default definitions to be exposed even when the explicit definition of other macros would cause that not to happen. The effect of defining this macro is equivalent to the effect of explicitly defining three macros in earlier glibc versions:

    cc -D_BSD_SOURCE -D_SVID_SOURCE -D_POSIX_C_SOURCE=200809C

In addition, glibc 2.19 removed the behavior that explicitly defining _BSD_SOURCE caused BSD definitions to be favored in the case of standards conflicts.

Some further patches from Joseph complete the deprecation of _BSD_SOURCE and _SVID_SOURCE for the upcoming glibc 2.20 release. The first of these patches converts _BSD_SOURCE and _SVID_SOURCE into synonyms for _DEFAULT_SOURCE. The deprecated macros remain effective, but cause compile-time warnings to be generated if they are used (unless _DEFAULT_SOURCE is also defined). The second patch (mechanically) converts all of the uses of the __USE_SVID and __USE_BSD internal macros in glibc header files into __USE_MISC.

The final patch does most of the real work: Joseph inspected all of the glibc header files to eliminate redundant uses of the __USE_MISC internal macro. These redundant uses range from doubled instances of __USE_MISC created by the preceding patch through to cases where __USE_MISC was combined (with a logical OR) with other macros that logically imply that __USE_MISC must also be defined. For example, in all cases where __USE_MISC is defined in glibc 2.20, __USE_XOPEN2K8 is also defined, meaning that conditionals such as the following in the glibc headers:

    #if defined __USE_MISC || defined __USE_XOPEN2K8

can be simplified to:

    #ifdef __USE_XOPEN2K8

From here on, there will probably be some further cleanups in the glibc headers. While it may not be practical in all cases to reach Roland's goal that only one of the internal __USE_* macros should govern the exposure of any particular definition, some simplifications can probably be achieved. There are likely also other cases similar to those involving __USE_MISC where there are conditionals involving two or more mutually dependent __USE_* macros that can be simplified by removing one of the macros.

Joseph has also hinted that there may be scope for obsoleting some interfaces or removing the __USE_MISC guard from conditionals such as __USE_MISC || __USE_UNIX98, so that some interfaces are not exposed by default. Certainly, there are currently a number of anomalous situations where some Linux-specific definitions are guarded by __USE_GNU and thus are exposed only if _GNU_SOURCE is defined (for example, the msgget() MSG_EXCEPT flag) while other (in some cases, closely related) Linux-specific definitions are guarded by __USE_MISC and are thus exposed by default (for example, the msgctl() MSG_STAT flag). Quite how the glibc developers will deal with issues like this, and what other FTM rationalizations and simplifications will be made for individual definitions, is matter that we are likely to see play out over the next couple of glibc releases.

Comments (22 posted)

Brief items

Quotes of the week

There should be a very high bias against changing documented behavior that is in fact working as documented. The document is a contract with users. Contracts shouldn't be violated without truly compelling cause.

"I am altering the deal. Pray I don't alter it any further." There's a reason Darth Vader wasn't proclaimed Python's BDFL ;-)

Tim Peters

The Web is to the Internet as Beer is to Alcohol.

For some people, sometimes, they are essentially synonymous, because they are often encountered together. But of course they are fundamentally different things

In this analogy, Email is like Wine: it’s the other universally popular use of the Internet/Alcohol.

But there are lots of other uses, too, somewhat more obscure. We could say the various chat protocols are the various Whiskeys. IRC is Scotch; XMPP is Bourbon.

gopher is obscure and obsolete, …. maybe melomel.

ssh is potato vodka.

Sandro Hawke

Comments (6 posted)

Calligra 2.8 released

The Calligra office suite has announced its 2.8 release. In addition to the Krita painting program's 2.8 release (which we reviewed in the March 6 edition of LWN), other components have new features and bug fixes as well. Words and Author gained support for document comments, Kexi (visual database tool) has many fixes and improvements based on user feedback, Sheets now has pivot tables, and so on. "There are also some general improvements in all apps. It is now possible to copy and paste any shape between any documents in Calligra applications. Moreover, copying and pasting of images and rich text is now more advanced."

Comments (none posted)

Xen Project Hypervisor version 4.4 available

The Xen Project has released version 4.4 of its hypervisor. This release enhances support for the ARM architecture, including support for AppliedMicro’s X-Gene, the Arndale Board, Calxeda ECX-2000, TI OMAP5 and the Allwinner SunXi boards, and 64-bit guest support on ARMv8 machines. Other changes in this release include a new event channel implementation contributed by Citrix and an experimental paravirtualization mode contributed by Oracle.

Comments (none posted)

Mozilla cutting support for Persona

Mozilla has announced that it is "transitioning Persona to community ownership" — or, in other words, dropping development support for this identity management project. "For a variety of reasons, Persona has received less adoption than we were hoping for by this point. However, we do still believe that Persona offers a unique and useful alternative to passwords, and we intend to support it as such. Reducing the scope of Persona and stabilizing its core APIs over the last quarter has shown us that adding more features was not the way forward." (LWN reported on Persona in March 2013).

Comments (18 posted)

Newsletters and articles

Development newsletters from the past week

Comments (none posted)

Packard: Glamor hacking

At his blog, Keith Packard has posted a write-up of recent work by Eric Anholt and himself to accelerate 2D functions in X with OpenGL, with promising results. "These two cases, text and points, are probably the hardest operations to accelerate with a GPU and yet a small amount of OpenGL code was able to meet or beat software easily. The advantage of this work over traditional GPU 2D acceleration implementations should be pretty clear — this same code should work well on any GPU which offers a reasonable OpenGL implementation. That means everyone shares the benefits of this code, and everyone can contribute to making 2D operations faster."

Comments (3 posted)

Page editor: Nathan Willis
Next page: Announcements>>


Copyright © 2014, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds