Toward safer GNU C Library tunable handling

By Jonathan Corbet
October 19, 2023

When considering the interface provided by the GNU C Library (glibc), thoughts naturally turn to the programming interface as specified by POSIX, along with numerous extensions added over the years. But glibc also provides a "tunables" interface to control how the library operates; rather than being managed by a C API, tunables are set with the GLIBC_TUNABLES environment variable. Glibc tunables have been a part of a few security problems involving setuid binaries, most recently the "Looney Tunables" bug disclosed at the beginning of October. The glibc developers are now considering significant changes to tunable handling in the hope of avoiding such problems in the future.

The tunables mechanism allows various aspects of the library's behavior to be adjusted at run time. Many of the provided knobs relate to memory-management algorithms hidden behind malloc(), but there are others as well. The list of available tunables on any given system can be seen by running:

    /lib64/ld-linux-x86-64.so.2 --list-tunables

The library provides no public functions for setting or querying tunable values; instead, users can set them, before running a program, by setting GLIBC_TUNABLES. This variable is processed early in the startup routine, well before a program's main() function is called. So a program has little control over which tunables are applied to it; in a sense, glibc tunables allow the library's behavior to be tweaked behind a program's back.

The ability to do such tweaking raises some obvious red flags when the program involved has the setuid or setgid permissions set. Those programs constitute a security boundary within the system, so allowing users on the unprivileged side of the boundary to adjust behavior on the privileged side is going to be fraught. Glibc has long recognized that some tunables should cease to be tunable when a program is setuid or setgid; to enforce that restriction, each tunable is marked, within glibc, with a "security level". If a tunable is marked SXID_ERASE, it will be ignored in privileged programs and erased from GLIBC_TUNABLES entirely, with the result that it will not be present in any child processes created by that program. SXID_IGNORE, instead, causes the tunable to be ignored but left in place. Without one of those settings, the tunable will be honored in privileged programs.

Even with this mechanism in place, the intersection of tunables and privileged programs has proved to be a reliable source of security bugs. So the glibc developers are looking at ways of eliminating that intersection entirely. Siddhesh Poyarekar posted a patch set setting all tunables to SXID_ERASE, essentially disabling the tunables mechanism for privileged programs entirely, shortly after the latest vulnerability disclosure. Some discussion followed and, one week later, Adhemerval Zanella proposed a deeper set of changes aimed at eliminating this class of problems.

Step one in Zanella's series is to eliminate support for the special /etc/suid-debug file. If that file exists (it can be empty), then the glibc.malloc.check tunable can turn on malloc() checking, even in privileged programs. Normally this feature is disabled in such programs because it can generate output to the standard error stream, which could be abused by an attacker. This debugging can be useful, so the check for this file was added as a way of giving the system administrator a way to make it available for privileged programs. It turns out, though, that the /etc/suid-debug check hasn't worked since the 2.34 release, when memory allocation was pushed out into a separate library; rather than fix this feature, this series just removes that support.

The next step is to ignore all tunables in privileged programs, essentially setting them all to SXID_IGNORE. A later patch then adds GLIBC_TUNABLES to the list of environment variables that are not passed through to child processes, effectively changing that setting to SXID_ERASE via a different means. There is another set of environment variables that can change many of the malloc() tunables as well; for example, MALLOC_MMAP_THRESHOLD can be used to change glibc.malloc.mmap_threshold. All of those variables are added to the "insecure variables" list as well.

That eliminates the handling of tunables from privileged programs, but the patch set makes another set of changes as well. Current glibc versions are relatively forgiving about invalid GLIBC_TUNABLES settings; a number of patches in this series put an end to that, aborting tunable parsing and emitting a warning when badly formatted tunable settings are found. Finally, there is a patch that causes GLIBC_TUNABLES to be parsed in place rather than copied to a separate string. That change is intended to avoid any problems that might come about as the result of memory-allocation failures; this processing happens before glibc's memory management is initialized, so allocation is trickier at this stage.

The end result should be a distinct reduction in tunable-related bugs and vulnerabilities in future glibc releases — if the series is accepted, of course. This change could come at a cost for users who have a reason to set these tunables for privileged programs, should any such users exist. But the glibc documentation is explicit in its statement that tunables are not considered part of the ABI and can change at any time. In this case, most users may well conclude that these changes are overdue and the cost is well worth paying if it results in fewer vulnerabilities in the future.

Toward safer GNU C Library tunable handling

Posted Oct 19, 2023 16:45 UTC (Thu) by sroracle (subscriber, #124960) [Link] (2 responses)

Note: --list-tunables was not introduced until glibc 2.33.

Toward safer GNU C Library tunable handling

Posted Oct 19, 2023 20:29 UTC (Thu) by itsmycpu (guest, #139639) [Link] (1 responses)

Just reading this without knowing much about it, I would want there to be a filter that can be set only as an admin, explicitly naming any tunables that may be applied to priviledged programs. Which means, by default, none.

Toward safer GNU C Library tunable handling

Posted Oct 19, 2023 21:06 UTC (Thu) by itsmycpu (guest, #139639) [Link]

It sounds like glibc.malloc.check might be useful even without the stderr output, so maybe for priviledged programs (or maybe in any case) it just wouldn't write anything to stderr.

Toward safer GNU C Library tunable handling

Posted Oct 19, 2023 22:36 UTC (Thu) by joib (subscriber, #8541) [Link]

I've used the glibc-specific secure_getenv() before, to avoid issues like this. https://man7.org/linux/man-pages/man3/getenv.3.html

Toward safer GNU C Library tunable handling

Posted Oct 20, 2023 0:43 UTC (Fri) by tilt12345678 (subscriber, #126336) [Link] (12 responses)

Shouldn't an effort for a permanent solution focus on a well-structured (versioned, extensible) documented configuration interface that offers deployment of textual, complete policies?
Shouldn't all this configuration go into a configuration file in /etc that has an expressive, descriptive language stating how a system behaves totally instead of offering the possibility to inject settings all over the place via environment?

Toward safer GNU C Library tunable handling

Posted Oct 20, 2023 7:45 UTC (Fri) by taladar (subscriber, #68407) [Link] (5 responses)

It does make sense to be able to set this per process, you wouldn't want the entire system to suddenly spew memory diagnostics.

However I agree, the environment variable system seems the wrong place for this, especially considering the low level nature of these settings and the issues around setting and getting environment variables from different threads of the same process without crashes.

Toward safer GNU C Library tunable handling

Posted Oct 20, 2023 17:14 UTC (Fri) by raven667 (subscriber, #5198) [Link] (2 responses)

In my opinion the environment is the simplest, most robust, most standard way to make ephemeral runtime changes to a piece of software, the environment is a core feature for passing settings to processes that I don't think should be discounted and setting environment variables has fewer unintended side effects than opening file descriptors unexpectedly in a library, which doesn't change the need to parse setting data.

The main takeaway for me is that a lot of tunable env vars like this may not be well documented on systems in a way that the millions and millions of people who admin and develop on Linux are generally aware they even exist, let alone could list the various settings and describe what they do. glibc is probably not the only library which has optional behavior which can be set by environment variables that could break a program, which the developers and users aren't taking sufficient precaution against because they aren't aware of the risk.

Toward safer GNU C Library tunable handling

Posted Oct 22, 2023 16:03 UTC (Sun) by dezgeg (subscriber, #92243) [Link] (1 responses)

Environment variables are annoying because they by default apply to any subprocesses, which can then cause them to break.

Command line would be much preferable, for example for Haskell code compiled with GHC you can pass 'tunables' to the GHC runtime by passing something like '+RTS -M128m -RTS' on the command line. Of course that GHC solution has the problem of in-band signalling.

At least for dynamically linked programs one neat way would be to explicitly invoke the dynamic linker and pass the tunables there, just like it already supports flags like --preload (ie. instead of 'LD_PRELOAD=foo.so /bin/ls' one can do '/lib64/ld-linux-x86-64.so.2 --preload foo.so /bin/ls').

Toward safer GNU C Library tunable handling

Posted Nov 2, 2023 14:20 UTC (Thu) by roblucid (guest, #48964) [Link]

This is NOT a valid criticism, the environment is for user preferences and things needed for their login session like display devices. The inheritance is required, it can allow programs written before a feature like ssh(1) to work correctly, while CLI arg would require enhancement to a mountain of software when adding functionality.

If your program system breaks it is written badly. Just sanitise the environment when you start your fragile application. Set things like PATH, adjust variables if required. Users are allowed to break their session.
When running with effective UID privilege the environment should not be trusted just as any user input needs careful verification.

Requiring every program anyone ever writes to pass through by CLI argument something like terminal type, printing width or display device is ridiculous.
Similarly you don't want low level routines continually opening config files to simulate the process environment.

Toward safer GNU C Library tunable handling

Posted Oct 22, 2023 17:17 UTC (Sun) by tilt12345678 (subscriber, #126336) [Link]

Sharing your views on the problematic behavior of the environment, also understanding that it is necessary to offer (at least) process-granular settings, maybe i should clarify my motivations when i asked those suggestive questions.

Problem 1: The method of applying tunables via environment is intrusive on a systems design/management level.
Being able to tune via environment is convenient, but - to state the obvious - it actually requires modifying the environment. Therefore, to be able to tune/trace every process, every potential process start has to be wrapped individually. A central policy mechanism could relieve system engineering from re-implementing this for every process that can be run in a system and that should be tuned/traced, making the tunables accessible where they right now are not (think containerized workloads etc.)

Problem 2: Being able to apply tunables via every process's environment makes a system's behavior less clear.
Given the current tuning configuration mechanism, it is a non-trivial task to state how a system currently actually behaves. Many corner cases have to be considered. Just some examples: Can a process apply tunables to itself by modifying its own environment? Is it safe to state that, if a process's environment contains a tunable setting X, then X is actually effective? Is it safe to state that X applies to all threads of X or only future threads? Also consider the special rules that apply to SUID/SGID binaries. Etc, etc. Much like discretionary access control, it's another semantical nightmare in audit situations.

Problem 3: The current design of the tunables configuration interface offers a questionable mixup of debugging and performance settings.
The current set of tunables mixes settings for tuning (e.g. hugeTLB) with settings for tracing (e.g. malloc check). A central policy syntax maybe could allow for a splitting of these substantially different activities into separate mechanisms, or maybe just a "sounder" design in general.

I am aware that my ramblings do not include even a hint of an idea of how to actually accomplish such improvements, because frankly i have none!

Toward safer GNU C Library tunable handling

Posted Nov 2, 2023 13:39 UTC (Thu) by roblucid (guest, #48964) [Link]

When running privileged it was longstanding standard practice to wipe out environment settings under UNIX.

It's hard to think of a valid use of environment passed in. For instance even dynamic linking with a local directory, you would optionally at build time insert a fixed directory to the default runtime .so search path. You could not rely on users not breaking environment.

This sounds like GNU lost the plot on this, when ld.so has an altered [ui]d it is simply asking for trouble to worry about a user's POSIXLY_CORRECT setting or other options that might introduce unanticipated behaviour.

Toward safer GNU C Library tunable handling

Posted Oct 20, 2023 16:48 UTC (Fri) by matthias (subscriber, #94967) [Link] (2 responses)

Some of these tunables are pretty low level. It is not even clear to me whether it would be possible to apply these late in the startup of a process. At least today most of this is processed even before there is an initialized malloc, which make sense as many of the tunables affect malloc. Parsing a well-structured (text)file without mallocing sounds like the worst nightmare.

And seeing that there are actually security holes in the parsing code for a pre-allocated very simple key-value store (environment variables), I can only imagine how hand crafted low level file parsing code without access to malloc would look like. Existing high level libraries will not be of much help in this restricted environment.

And while I do not really like environment variables, the problems are really not in the interface, but the language used to access the information. There will not be less buffer overflows in the checking code just because we switch from environment variables to files. The only real solution is to switch to a language that enforces bound checking.

And the problem that this kind of tunables crosses a security boundary for SUID files will also not go away if we switch the interface. We really should get rid of SUID. There is way too much state that crosses the boundary during fork+exec, not only the environment variables.

Toward safer GNU C Library tunable handling

Posted Nov 2, 2023 13:24 UTC (Thu) by sammythesnake (guest, #17693) [Link]

This is slightly off topic, but your mention of getting rid of SUID/SGID happened to trigger a thought - couldn't most SUID/SGID programs be reasonably easily replaced with a shim that could just send an IPC to some daemon already running with the right environment to do the equivalent work...?

Toward safer GNU C Library tunable handling

Posted Nov 2, 2023 14:43 UTC (Thu) by roblucid (guest, #48964) [Link]

GNU glibc introduced a complicated feature affecting startup of ld.so after exec.
It's self inflicted because it's providing features that are highly dubious in programs with setuid/setgid bit set.

Such programs are supposed to be small, carefully written and sanitise all input including the environment. Glibc tunables broke that contract.

Things worked before these GNU tunables, it's unclear why ld.so is EVEN looking at them when exec changes the effective [ug]id.

It's not environment that's broken it's a badly thought out feature created by glibc.

Toward safer GNU C Library tunable handling

Posted Oct 24, 2023 15:00 UTC (Tue) by rwmj (subscriber, #5474) [Link]

No please don't do that. We use tunables to enhance bug finding when doing 'make check' and we don't want to have to make sysadmin changes for that to work. What you're essentially proposing is having less testing, making programs more unsafe.

Toward safer GNU C Library tunable handling

Posted Oct 28, 2023 1:27 UTC (Sat) by siddhesh (guest, #64914) [Link]

There is a systemwide tunables proposal DJ Delorie is working on, which does all that.

Toward safer GNU C Library tunable handling

Posted Nov 2, 2023 13:52 UTC (Thu) by roblucid (guest, #48964) [Link]

Sysadmins has had ways to set up default environment for users helpfully.

The whole point of environment was to allow users to alter their preferences and also have things like their terminal type and X-server work.
Personal choices in ~/. profile are desirable, users are free to break their session settings.

BUT privileged programs with an effective [ug]id should not be affected by environmental whims of the misguided or malicious.