Leading items

Welcome to the LWN.net Weekly Edition for March 31, 2022

This edition contains the following feature content:

• Systemd discusses its kernel-version needs: what should be the earliest kernel version supported by systemd — and why?
• A way out for a.out: the kernel may finally drop support for an ancient executable format.
• 5.18 Merge window, part 1: the first set of changes merged for the 5.18 kernel.
• Pointer tagging for x86 systems: an attempt to support a new AMD CPU feature runs into trouble.
• Problems emerge for a unified /dev/*random: changes to the kernel's random-number generator can be trickier than expected.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Systemd discusses its kernel-version needs

A query regarding the possibility of dropping support for older kernels in systemd led to some discussion on the systemd-devel mailing list recently. As might be guessed, exactly which kernel would be the minimum supported, what kernel features systemd is using, and when those kernel features became available, were all part of that conversation. A component like systemd that is closely tied to the kernel, and the interfaces different versions provide, has a number of different factors to consider when making a decision of this sort.

Zbigniew Jędrzejewski-Szmek started things off by asking if changing the minimum required kernel version for systemd to 4.4 would cause problems for anyone. He said that if it did, "please substantiate why you are running new systemd with such old kernels". Currently, systemd minimally requires Linux 3.15 or later, as noted in its README file.

Which version?

But Greg Kroah-Hartman was quick to point out that the stable-kernel team recently stopped supporting 4.4, so the 4.9 kernel is the oldest version that is still getting stable updates. He suggested using that as the baseline instead. Jędrzejewski-Szmek noted that the Civil Infrastructure Platform (CIP) will continue to use 4.4, but that Debian 9.x ("stretch") uses 4.9 and has its end of life in June 2022. "Of course we'd like to move to 4.19, but we don't want to disrupt distros that use older kernels. Is ≤4.19 really unused?"

Michael Biebl filled in more details for the Debian picture. Newer systemd versions are being backported to Debian 11.x ("bullseye"), which uses the 5.10 kernel, but not to earlier Debian releases. He did note that downstream Debian derivatives (such as Raspbian) might have other requirements, as might those who build their own kernels. Neal Gompa said that there are active efforts to bring newer systemd releases to CentOS Stream 8, which is based on 4.18 plus a bunch of kernel-feature backports. He would like to see that continue to be supported.

Luca Boccassi said that he would like to see the minimum version only go as far as 4.4 since "what's on kernel.org is not really that important, as real usage is downstream from there anyway". The important thing is to identify what's needed for systemd core compatibility—all of that is available in 4.4, he said. He noted that support for version 2 of control groups (cgroupv2) came in 4.1; being able to count on cgroupv2 support would be helpful. Control groups are a resource-management mechanism that is used extensively by systemd.

Not surprisingly, Kroah-Hartman had a rather different take on that; "anyone still using 4.4.y today has an insecure and unsupported system". He does not want to encourage anyone to use that version. While CIP is supporting 4.4 until 2027, it is using that kernel "in a very limited set of use cases and configurations". He is somewhat skeptical that CIP will be able to keep that kernel "alive and secure"; for anyone else that is still using 4.4, "send them to me please".

Boccassi disagreed with that view; kernel-version choices are often out of the hands of the user because they have to use what their distribution or vendor provides. If the kernel version for systemd changes, he expects that the project will hear from "some poor soul stuck on 3.x because of $crappy_arm_vendor with no way to move on from there". Even so, moving to 4.4 has some advantages:

Jumping forward from 3.13 to 4.4 as the baseline, allowing to take cgroupsv2 for granted, seems like a good starting point to me. There's very obvious and public evidence of that being used in the wild. We can start to drop a bunch of backward-compat cruft, wait and see who complains, and if nobody does we can re-evaluate again in a couple of years.

But Kroah-Hartman thought it is unlikely that people stuck on 3.x kernels are updating to the latest systemd version; "if they are, they need to get $crappy_arm_vendor to do the work for them as they PAID for that support and work already".

Ulrich Windl said there can be value in sticking with the same kernel over time, though, because "new features intended to improve things sometimes actually make things worse". But in choosing which kernels to support for systemd, that is not really the issue, Kroah-Hartman said:

Do you want to run a kernel with known security problems, or one with "unknown potential problems." The latter is always the case, so please don't pick the known-insecure one, that's just foolish.

Boccassi was not impressed with that argument, noting that "'security problems' are a dime a dozen, as they say" and complaining that the stable kernels often have regressions of various sorts. He made a rather surprising claim about the nature of those regressions, however:

Upgrading major kernel version is like rolling a dice, you never know what kind of extremely expensive and time consuming rabbit hole you'll be dragged into because the kernel plays fast and loose with its userspace interfaces, and each and every time there's a chance one might end up having to do major reworks to deal with it.

Kroah-Hartman seemed surprised to hear that, since the kernel developers take user-space compatibility seriously. As readers of the LWN kernel page know, Linus Torvalds will quickly step in and remind kernel developers, sometimes rather harshly, that the kernel does not break user space. There are often complaints about regressions in the stable tree, but it does not typically involve breaking user-space interfaces. Kroah-Hartman said that he was interested in hearing any such reports and that the proper place to report them was the kernel's regressions mailing list.

That was something of a side conversation to the main point of the discussion, but the resistance to upgrading to new kernel versions is real; it is somewhat unclear why that resistance would not also extend to systemd versions, however. The kernel-upgrade resistance is something that the stable maintainers (and others) have been struggling with for a long time. Those who plan to stay on older releases, or not upgrade series that are still supported, are playing a dangerous game. Kroah-Hartman clearly sees the choice of the minimum supported kernel version as a means to help spread that message further, though it is unclear how successful it would be.

Control group evolution

Back to systemd itself, Lennart Poettering said that there had been an evolution in cgroupv2 support over time, so narrowing down exactly what systemd needs or wants in that support is needed:

Note that "cgroupv2 exists" and "cgroupv2 works well" are two distinct things. Initially too few controllers supported cgroupv2 for cgroupv2 to be actually useful.

What I am trying to say is that it would actually help us a lot if we'd not just be able to take [cgroupv2] for granted but to take a reasonably complete cgroupv2 for granted.

Boccassi agreed that information would be useful, especially which cgroupv2 controllers are needed and when they were added. As Jędrzejewski-Szmek pointed out, the README does list some features that systemd can use and what kernel is needed for them:

Linux kernel >= 4.2 for unified cgroup hierarchy support
Linux kernel >= 4.10 for cgroup-bpf egress and ingress hooks
Linux kernel >= 4.15 for cgroup-bpf device hook
Linux kernel >= 4.17 for cgroup-bpf socket address hooks

In this light, 4.19 is better than 4.4 or 4.9 ;)

Given that, Kroah-Hartman was in favor of choosing 4.19: "I strongly doubt that any distro that is using older kernels would ever be willing to update systemd." Windl added a data point, saying that the most recent service pack of SUSE Enterprise Linux 12 (SLES12 SP5) is using 4.12, but he agreed that it was not likely it would be moving to newer systemd versions.

The list of kernel versions in the README is mostly concerned with various BPF-related features, Boccassi said. But systemd should still run without support for BPF; "There's plenty of use cases that disable it entirely (in the sense, things shouldn't fall apart if we run on a non-bpf kernel and there are no bpf options configured in any unit). I have one of them." BPF-related features should remain optional in systemd forever, he said.

Poettering took a look through the systemd man pages to get a sense for what kernel features are being used and when they were added. He noted that cgroupv2 support was not really viable for systemd until 4.15, since before that it lacked the CPU controller, which is used to partition the CPU use between different groups. Beyond that:

some other interesting milestones:

• kcmp → 3.5
• renameat2 on all relevant file systems → 4.0
• pids controller in cgroupv1 → 4.3
• pids controller in cgroupv2 → 4.5
• cgroup namespaces → 4.6
• statx → 4.11
• pidfd → 5.3

This is just some quick search through man pages. There might be a lot of other stuff that would make sense for us to be able to rely on.

Windl wondered if some sort of conformance test suite for systemd made sense; "If the test suite succeeds, systemd might work; if it doesn't, manual steps are needed." Windl guessed that if manual steps were required, it might deter most people from going any further. But Poettering thought that was not the right path forward: "One goal here is to reduce our [maintenance] burden, not increase it. Another is to communicate clearly what we support and what we don't. Any such test suite collides with both these goals."

The conversation wound down, seemingly without any final conclusions. While support for 4.4 might send the wrong message, at least from the stable-kernel maintainers' point of view, it may make sense if the CIP project has plans to upgrade its systemd along the way. Jędrzejewski-Szmek opened an issue on the project's GitLab site to try to find out. It does seem reasonable to at least conjecture that distributions (and others) that are unwilling to upgrade their kernel might also be leery of upgrading a major component like systemd.

Since 4.4 lacks the cgroupv2 support that is desirable, 4.9 or even 4.19 seem like the next logical possibilities, though 4.9 reaches end of life in January 2023. In addition, 4.9 is lacking in the cgroupv2 department as well. The 4.19 kernel will be supported for nearly two years longer than 4.9, until December 2024, and has most of the desired features. It may well be where the project decides to land.

In the end, the systemd project will need to find a good balance between the features it needs from the kernel and the kernel versions that its users are running—if they still plan to upgrade to newer systemd versions anyway. Most distributions will likely shy away from kernels that have fallen outside of the stable-kernel support windows, though, so that helps bound the search space to a certain extent. Settling on a choice, and being able to remove a bunch of compatibility code for earlier kernels, will presumably result in a nice code cleanup as well.

Comments (55 posted)

A way out for a.out

The a.out executable format dates back to the earliest days of Linux — and before. It has not been used in any serious way for decades, but support still exists in the Linux kernel and has resisted all attempts at its removal. Back in January, Borislav Petkov tried yet again to delete support for this format, leading to another extended discussion. There is one difference this time around, though: the effort to get rid of a.out support might just succeed.

The a.out format dates back to the first edition of Unix. When MINIX came along, it naturally used that format for its executable files; that, in turn, led to a.out being used in Linux as well. It is a simple format, and its implementation on Linux was even simpler; among other things, every Linux shared library had to be centrally assigned its own portion of the address space, since libraries could not be relocated at run time. Still, Linux used a.out for some time, until support for the newfangled ELF format was first added to the 0.99.13 development kernel in 1993.

There was a time when the crazier people among us manually converted our Slackware systems from a.out to ELF in order to be able to try it out and gain the benefits before distributions were updated. They still bear the scars from that time. Not that your editor would ever admit to knowing anybody who would have engaged in any such activity.

ELF has been the standard executable format for Linux on most architectures since 1995. One might think that would have provided enough time for any users of a.out binaries to grudgingly move on to ELF; its adoption can probably be judged to not be a passing fad at this point. But, in the real world, surprises lurk.

The initial conversation on removing a.out faded out fairly quickly, but was restarted when Eric Biederman posted a patch disabling the building of a.out support on the two architectures (Alpha and m68k) that still enable it by default. This patch did not actually remove the support, it just turned it off to see if anybody would scream. If protests were raised, support could be turned back on quickly and easily; otherwise a full removal could be done.

Linus Torvalds quickly responded that he was "pretty sure we can't do this". He pointed out that the native executable format on Alpha systems running Digital's Unix was essentially a.out, even if it went by the newer name of ECOFF. The Linux a.out loader can run ECOFF programs by virtue of ignoring some of the newer ECOFF features; taking it out would break any systems still using that support. One might think that the number of users who are still using Alpha CPUs, running ECOFF binaries on those CPUs, and tracking current kernel releases might be a fairly small number, but one never knows.

Kees Cook did do a little research on, seemingly, the only distribution that still supports Alpha (Gentoo) and found that the only ECOFF files present contained firmware, which does not run on the CPU anyway. He concluded that there would be no harm from removing a.out support on this platform: "Let's do it".

It seems to be a universal rule that somebody always has to come along to ruin the party. In this case, just as it seemed like there were no further obstacles to the removal of a.out, James Jones showed up to let it be known that he was still using a.out:

The use case is running an old set of tools to build programs for the Atari Jaguar. Namely, Atari's assembler (mac) and linker (aln). The alternative is running windows versions in dosbox, or using some replacements that have been developed based on an even older, less-featureful version of the source code for mac and aln, but which still haven't managed to add back in all the features needed to build some programs or use the Atari debugging tools (Also available in a.out only).

He gave some more details on why he still needed to use these tools in this followup message

Moving a program to a new executable format is normally just a matter of rebuilding the program. But rebuilding is a rather more difficult task in the absence of the source code. Finding oneself stuck trying to run a decades-old binary on modern systems is just one of the many joys reserved for users of proprietary software — but many people have found themselves in that position at one time or another. They are legitimate Linux users, and there is no desire to break their systems. So Petkov duly gave up and asked only that Jones document his usage of a.out for the benefit of those who might try to remove it in the future.

Cook, however, was not so quick to throw in the towel; he looked at the programs in question and concluded that it might be possible to write an ELF wrapper program that could load and run old a.out binaries. One day later, he posted such a program, noting that it could run the aln program at least far enough for it to complain about its command-line arguments. Jones tried it out and was pleased with the results:

Yes, works perfectly, thank you. I like the idea of using this much better than getting emailed every time someone wants to delete the a.out code again. Consider my use case withdrawn. I've already pushed an update to my jaguar-sdk project to use this tool instead.

So removal of a.out support is back in the plans for the 5.18 kernel. Perhaps the effort will succeed this time, though there are still no guarantees; there may be a.out users who are unaware of the coming apocalypse and haven't yet made their objections known. If those users are unable or unwilling to use Cook's wrapper utility, the excision of a.out support may, once again, have to be deferred to some future kernel release.

Comments (42 posted)

5.18 Merge window, part 1

As of this writing, 4,127 non-merge changesets have found their way into the mainline repository for the 5.18 development cycle. That may seem like a relatively slow start to the merge window, but there are a lot of changes packed into those commits. Read on for a summary of the most significant changes to land in the first half of the 5.18 merge window.

Architecture-specific

• 32-Bit Arm systems have gained support for separate interrupt stacks and virtually-mapped kernel stacks.
• Support for older Arm systems (ARMv4 and ARMv5) without a memory-management unit has been removed. MMU-less support for ARMv7-M systems remains, though.
• The arm64 architecture supports the new "QARMA3" pointer-authentication algorithm. This variant of Arm's QARMA is evidently faster while still being sufficiently secure.
• Arm64 systems can be built with shadow-stack support using the GCC 12 compiler release.
• The PA-RISC architecture now has minimal vDSO support which, in turn, enables the system to run with a non-executable stack for the first time. The initial version of this patch was posted in 2006; some things take longer than others to get into the mainline, it seems.
• Support for the Intel "hardware feedback interface" has been added. This mechanism allows the hardware to inform the kernel about the current performance and energy-efficiency capability of each CPU in the system. These capabilities can change over time as the result of, for example, thermal constraints. This documentation patch has some more information.
• Support for the nds32 architecture has been removed. According to the merge changelog:

  The hardware is still used and the code is in reasonable shape, but the mainline port is not actively maintained any more, as all remaining users are thought to run vendor kernels that would never be updated to a future release.

Core kernel

• The io_uring subsystem has seen a number of improvements. The new IORING_SETUP_SUBMIT_ALL option will cause a full batch of requests to be submitted even if an error is encountered partway through. The file descriptors for the ring itself can be registered with the ring, providing a performance improvement for threaded applications; see this changelog for some details. The new IORING_OP_MSG_RING operation allows one ring to signal another. Finally, it is now possible to perform the NAPI busy poll on sockets directly from the ring.
• Support for the a.out executable format is no longer built by default for the alpha and m68k architectures — the last two that were still using it. The a.out code has not actually been removed yet but that is probably coming soon.
• Some tweaks to the restartable-sequences API have been merged in preparation for support in the GNU C Library.
• The DAMON operation schemes (DAMOS) mechanism gives user space more control over memory-management operations (and page reclaim in particular).
• The tracing system now supports "user events", which are essentially dynamic tracepoints in user-space applications. The feature is described in the merge changelog as:

  User space can register an event with the kernel describing the format of the event. Then it will receive a byte in a page mapping that it can check against. A privileged task can then enable that event like any other event, which will change the mapped byte to true, telling the user space application to start writing the event to the tracing buffer.

  See the commits adding documentation and a sample program for more information.

Filesystems and block I/O

• The inline-encryption capabilities of block request queues can now be viewed in sysfs; see this changelog for details.
• Direct I/O is not normally a possibility for encrypted files, since the data must be buffered through the kernel for encryption or decryption anyway. If the hardware does the crypto work, though, the situation is different. In 5.18, files encrypted with fscrypt can be accessed with direct I/O if inline encryption is in use. This documentation patch contains a little more information.
• The F2FS filesystem has gained support for ID-mapped mounts.
• Support for NFSv3 will always be built into the NFS server if NFS is enabled at all. This is done with the intent of making NFSv3 become the base, "always-supported" version of NFS in preparation for the eventual removal of NFSv2 support.
• There are two new ioctl() operations for Btrfs (BTRFS_IOC_ENCODED_READ and BTRFS_IOC_ENCODED_WRITE) that allow direct reading from and writing to a file's extents. The main use case for these commands is to support newer, more efficient send and receive operations.

Hardware support

• Hardware monitoring: ASUS ACPI embedded controllers, Vicor PLI1209BC digital power supervisors, Aquacomputer Farbwerk 360 RGB controllers, and Texas Instruments TMP464 and TMP468 temperature sensors.
• Media: Microchip CSI2 demux controllers, Hynix Hi-847 sensors, OmniVision OV08D10 and OG01A1B sensors, and Intersil ISL7998x video decoders.
• Miscellaneous: Qualcomm MSM power manager controllers, Xilinx ZynqMP SHA3 accelerators, TI TPS6286x power regulators, Richtek RT5190A power-management ICs, Sunplus SP7021 SPI controllers, LiteX MMC host controllers, and Tesla full-self-driving clock controllers.
• Sound: Texas Instruments TAS5805M speaker amplifiers, AMD PCI audio coprocessors, and Awinic AW8738 audio amplifiers.

Miscellaneous

• New documentation of interest includes some guidelines for researchers studying the kernel community, an overview of the readahead code, how to report regressions, and how developers should handle regressions.

Security-related

• There is a new kernel keyring called machine; it contains the machine-owner keys implemented by the shim bootloader interface. Keys in the machine keyring can be trusted within the kernel and thus used to sign artifacts (such as modules or integrity data) used after the initial boot process.
• Support for asymmetric TPM-backed private keys has been removed. This feature, initially added for the 3.7 release, depends on an obsolete TPM version and had some security issues of its own; it is hoped that nobody is using it.
• The random-number generator has seen a lot of work. The differences between /dev/random and /dev/urandom have been removed (though some of the urandom changes had to be reverted after a regression was reported). There is a new mechanism for the avoidance of random-stream duplication when a virtual machine forks. The BLAKE2s algorithm is now used internally. There is more; see the merge changelog and this page for lots more details.
• The kernel now provides saturating arithmetic helpers for size_t values; these can be used to harden code against integer-overflow bugs. See this commit for more information.

Internal kernel changes

• The first big chunk of work from the fast kernel-headers tree has found its way in with a significant reorganization of the scheduler header files.
• The block-layer congestion-tracking code, which was found to be unused last year, has been removed.
• The memory-management code has been enhanced with remote per-CPU page list draining.
• More of the folio patch series has been merged; this set converts internal memory-management functions (including the varieties of get_user_pages()) to folios and enables the creation of large folios in the readahead code. A second set converts a set of address_space_operations to folios.
• The set_fs() infrastructure has finally been fully removed.

A quick check shows that linux-next currently contains nearly 9,000 commits that have not yet been pulled into the mainline, so it would seem that the 5.18 kernel will have a lot more to offer still. The merge window can be expected to remain open until April 3; tune in shortly after that for a summary of the remaining work pulled for this release.

Comments (12 posted)

Pointer tagging for x86 systems

Pointers are a fact of life for developers working in numerous languages. It is often convenient to be able to associate a small amount — a few bits at most — of ancillary information with a pointer. This can often be done within the pointer value itself with some careful masking and shifting. CPU manufacturers have been adding ways to support the addition of this sort of "tag" to pointers; the most recent may be AMD's "upper address ignore" (UAI) feature, support for which was recently posted by Bharata B Rao. This feature has an uncertain future in Linux, though, as the result of a fundamental design decision.

On a 64-bit system, a pointer is, naturally, 64 bits wide. But the CPU does not actually need all of those bits to dereference an address stored in a pointer. There are no systems (yet) that require — or can provide — all of the memory that can be addressed by 64 bits, meaning that there are ranges of address space that do not map to physical memory. Normally, user-space addresses start at (or near) zero and increase from there; that means that the highest-order bits will be zero even with the largest possible addresses. As a result, it can be possible to use those high-order bits to store other types of information.

There are numerous use cases for stashing metadata into those unused bits. Memory allocators could use that space to track different memory pools, for example, or for garbage collection. Database management systems have their own uses for that space. Applications can implement this sort of tagging now, but it must be done with care; an address with extra bits set is no longer a valid pointer, so that metadata must be masked out before dereferencing that pointer or passing it into code that does not understand the tagging scheme. That is error-prone and may slow down the application.

To make life easier for the developers of this sort of application, CPU manufacturers have been adding the ability for the processor to simply ignore the non-address bits in an address value. Naturally, every manufacturer has invented its own way of supporting this feature. The AMD version, UAI, specifically allows the uppermost seven bits of an address to be used for ancillary data.

If accepted, AMD's implementation of this feature would not be the first; support for the Arm "top-byte ignore" feature was merged for the 5.4 kernel in 2019. At that time, a set of prctl() commands was added to control the use of this feature. Top-byte ignore can be enabled with:

    int prctl(PR_SET_TAGGED_ADDR_CTRL, PR_TAGGED_ADDR_ENABLE, 0, 0, 0);

This interface was designed around Arm's implementation, which makes eight bits available for tag data. The AMD implementation only allows for seven bits, meaning that applications wanting to use tagged addresses will need a way to discover how many bits are available. So Rao's patch set starts with a patch from Kirill Shutemov (intended to add support for a similar Intel feature, more about that below) adding two new parameters to the above prctl() call, both of which are integer pointers. The first of those is for the caller to specify how many bits they would like to use for pointer metadata; the kernel will update that value to reflect the number of bits that are actually available. The second pointer tells the kernel where to store the number of bits to right-shift a pointer value to obtain the tag data.

The subsequent patches then implement support for UAI in the Linux kernel.

The idea is simple enough, and this feature already exists for the Arm architecture, but the UAI patches have still run into pushback, for a number of reasons. Perhaps the most fundamental of those is that UAI allows the most-significant bit of the address to be used by user space. In current systems, only kernel-space addresses have that bit set. Turning on UAI would allow user space to create pointer values that look like kernel addresses, but which would actually be valid user-space pointers. Those pointers can, of course, be passed into the kernel via system calls where, in the absence of due care, they might be interpreted as kernel-space addresses. The consequences of such confusion would not be good, and the possibility of it happening is relatively high.

This mechanism could probably be made to work safely, but, as Andy Lutomirski said: "A lot of auditing of existing code would be needed to make it safe". Even more auditing would be required, of course, to keep it safe in a rapidly evolving kernel. It sounds like a recipe for ongoing security problems, which is why Thomas Gleixner said that "there is no justification for the bit 63 abuse". He suggested that AMD should rework the feature in its processors to disallow that bit in address tags; he did not say that this problem would block the merging of UAI, but the meaning was reasonably clear.

Another problem that Lutomirski pointed out is that UAI is not specific to any running context; once it is enabled, it is turned on for the entire CPU. That, too, could lead to unpleasant surprises, so he suggested that the kernel would need to make the UAI settings process-local, even if it slows down context switches considerably.

Finally, there is the issue of Intel's similar feature, called "Linear Address Masking" (LAM). It does not have the most-significant-bit issue that UAI has, and it is managed as part of the process context. It supports two modes, with either six or 15 bits being made available for ancillary data; the 15-bit mode only works if five-level page tables are not in use. LAM has been around for a while, and support patches were posted (by Shutemov) in early 2021. That work seems to have stalled after that posting, but can be expected to come back at some point.

Rao's UAI patch set deliberately keeps the AMD implementation entirely separate from the proposed LAM implementation, even though the two are doing essentially the same thing. That led recently appointed x86 co-maintainer Dave Hansen to object: "We'll have one x86 implementation of address bit masking. Both the Intel and AMD implementations will feed into a shared implementation". So this is something that would certainly need to be fixed before this work could be considered for mainline merging.

The other issues are tied to the design of the hardware, though, and will be rather harder to fix in kernel code. For these reasons, the sentiment among kernel developers seems to be that LAM is a better-designed implementation of pointer tagging and should perhaps be what all x86 systems use. In the above-linked message, Lutomirski concluded:

I believe it's possible for a high-quality kernel UAI implementation to exist, but, as above, I think it would be slow, and it might be quite complex and fragile. Are we sure that it's worth supporting it?

A better solution, he suggested, would be for AMD to go back to the drawing board and create its own implementation of LAM instead.

In the early days of Linux, kernel developers had to adapt to whatever the hardware manufacturers put out; the alternative was to not have hardware to run on at all. In 2022, though, those developers feel more confident in their ability to reject support for hardware features that, for whatever reason, they feel do not fit in well with the design of the system. If AMD is unable to get support for UAI into the kernel (it's worth noting that Rao hasn't given up yet), UAI is likely to go mostly unused and developers needing pointer tagging may gravitate toward competing CPUs. According to Gleixner (linked above), AMD was told about the problems with its implementation some time ago; the company may yet have reason to wish it had listened.

Comments (50 posted)

Problems emerge for a unified /dev/*random

In mid-February, we reported on the plan to unite the two kernel devices that provide random numbers; /dev/urandom was to effectively just be another way to access the random numbers provided by /dev/random. That change made it as far as the mainline during the Linux 5.18 merge window, but it was quickly reverted when problems were found. It may be possible to do that unification someday, but, for now, there are environments that need their random numbers early on—without entropy or the "Linus jitter dance" being available on the platform.

A bunch of changes for the kernel random-number generator (RNG) were merged by Linus Torvalds on March 21. Those changes included unifying the two RNG devices, because it was hoped that no mainstream platforms would lack a source of unpredictable data that would allow the RNG pool to initialize in short order at boot time. For several years now, the jitter dance has used CPU execution time jitter to initialize the pool in less than a second; it uses the differences in code-execution speed of repetitive operations due to unpredictability in modern CPUs, from caches, branch prediction, and the like. But some systems lack jitter and have no other source of unpredictable data. That leads to the boot process hanging waiting for the RNG pool to initialize.

De-unification

Guenter Roeck reported a problem the day after the code was merged. He saw "a large number of qemu boot test failures for various architectures (arm, m68k, microblaze, sparc32, xtensa are the ones I observed). Common denominator is that boot hangs at 'Saving random seed:'" He bisected the problem to the patch that unified the RNG devices, and noted that reverting it fixes the problems he found. As would be expected, a user-space regression of that sort led Torvalds to say that he would revert the patch. The idea was good, but it "causes problems for various platforms that can't do jitter entropy and have nothing else happening either". Jason A. Donenfeld, the author of the patch and one of the kernel RNG maintainers, agreed with that assessment. Later that day, Torvalds reverted the unification; "This isn't hugely unexpected - we tried it, it failed, so now we'll revert it."

But Donenfeld was interested in finding out more about the underlying problem and asked Roeck for information about the QEMU virtual machines (VMs) used. If the unified-RNG-devices idea is ever going to return, "understanding everything about why the previous time failed might be a good idea", Donenfeld said. He poked around in one of the VM images and discovered the boot-time shell script that was printing the message in question. It was using a stored random seed to initialize /dev/urandom by writing to it, then reading from the device to grab a new seed to store for the next boot.

There are some problems with that approach, however. The first is that writing to /dev/urandom will only mix the data into the pool; it does not credit any entropy, so the RNG subsystem still does not initialize. Because it does not initialize, and the changes merged (and since reverted) make /dev/urandom block until it is initialized, the boot process would hang when it tried to read the new seed. That was a clear user-space regression that is not going to be—was not—tolerated.

A more insidious problem, perhaps, is that a seed written to the RNG before it initializes will not actually be used until after the pool is initialized properly; "you might write in a perfectly good seed to /dev/urandom, but what you read out for the subsequent seed may be complete deterministic crap", Donenfeld said. The data that gets written to the device is not credited with any entropy unless the RNDADDTOENTCNT ioctl() command is used, but that also means the fast initialization pool is not updated with the seed data, so the random numbers returned from the non-blocking /dev/urandom before it is initialized are much worse. That behavior has been true for quite some time, he said, but it makes the "innocuous pattern" of writing a seed and reading a new one into a potentially serious flaw; he thought he had a "quick unobtrusive fix" for that.

Torvalds said that he hates the "no entropy means that we can't use it" idea that exists in the RNG; "It's a disease, I tell you." It is the direct cause of that second problem, he said, continuing:

By all means the code can say "I can't credit this as entropy", but the fact that it then doesn't even mix it into the fast pool is just wrong, wrong, wrong.

I think *that* is what we should fix. The fact is, urandom has long-standing semantics as "don't block", and that it shouldn't care about the (often completely insane) entropy crediting rules.

But that "don't care about entropy rules" should then also mean "oh, we'll mix things in even if we don't credit entropy".

Donenfeld agreed with that view: "In general, your intuition is correct, I think, that the entropy crediting scheme is sort of insane and leads to problems." He pointed to his recent report on kernel RNG changes, noting that it talks about other RNGs that could be used (e.g. Fortuna), which do not suffer from the same types of problems, so that "might be something to look at seriously in the future".

Another patch

Donenfeld posted a patch on March 22 to try to address the problem with reading poor-quality random numbers for seed files during early boot. He said that he had fixed a related problem in systemd, but that cleaner fix did not help the existing shell scripts.

So this patch fixes the issue by including /dev/urandom writes as part of the "fast init", but not crediting it as part of the fast init counter. This is more or less exactly what's already done for kernel-sourced entropy whose quality we don't know, when we use add_device_randomness(), which both contributes to the input pool and to the fast init key.

Torvalds wondered why reads from /dev/urandom did not simply use the initializing pool, since the data written to the device is already mixed into that. Donenfeld agreed that his approach was less-than-perfect, but that it is far better now that some changes he made for 5.18 are in place. He had a lengthy explanation on the goals of his changes and on the differences between the input pool and the fast init pool. The input pool is used to rekey the ChaCha cipher that actually provides the random bytes after initialization, while the fast init pool is used before the full 256 bits of entropy are gathered in the input pool. Until that entropy is gathered (from various kernel sources, a hardware RNG, entropy credited from user space, or from jitter), the input pool is not properly initialized, thus an alternative needs to be used:

The "pre init pool", the "fast init pool", the "super terrible but at least not zero pool", the "all bets are off pool", ... whatever you want to call it. Why a separate pool for pre init? Because the real input pool needs to accumulate 256 bits of entropy before it's safe to use.

Your suggestion is to instead not have a separate pool, but perhaps just do separate accounting. That might work to some degree, but the devil is in the details, and that sounds a lot harder and messier to code.

He reiterated there may be some other longer-term solutions to consider, but did not think Torvalds's suggestion would help much. Ted Ts'o, the other kernel RNG maintainer, cautioned that writing to /dev/urandom is not a privileged operation, so a malicious user-space program could write specific values to it; that is the reason why inputs to the device are not used "until there is a chance for it to be mixed in with other entropy which is hopefully not under the control of malicious userspace".

Now, I recognize that things are a bit special in early boot, and if we have a malicious script running in a systemd unit script, we might as well go home. But something to consider is whether we want to do [something] special if the process writing to /dev/[u]random has CAP_SYS_ADMIN, or some such.

Donenfeld said that the input provided by writing to the RNG devices is cryptographically hashed, so "we can haphazardly mix whatever any user wants, without too much concern" as long as it is not given entropy credit. While crediting entropy when the writes to /dev/urandom are done from a process with CAP_SYS_ADMIN (or some other privilege) might be reasonable, there may be user-space code that is expecting the current behavior. He noted that the problem he saw in the shell scripts being used by Roeck's tests is not really a kernel bug:

[...] this has _always_ been broken, and those shell scripts have _always_ been vulnerable. Maybe the kernel should fix that, but due to the ambiguity of the /dev/urandom write interface, maybe the best fix is actually in userspace itself, which means it'd work on old kernels too (which are rather common for the embedded devices that tend to have those types of shell scripts).

Donenfeld pointed to his systemd fix and one he submitted for Buildroot as a better approach. The idea is that user space hashes the old seed with the data it receives from reading the RNG device before storing it away for the next boot. That way, even if the RNG is not initialized, "the amount of entropy in the new seed will stay the same or get better, but not appreciably regress".

RNDADDTOENTCNT

In his proposed patch, Donenfeld noted that the RNDADDTOENTCNT ioctl() command is a poor interface for crediting entropy, in part because of the separation of the pools during initialization. The RNDADDTOENTCNT command simply tells the kernel to credit the number of entropy bits that is passed in, but the write to the RNG device has already happened (without credit). Repeatedly writing small amounts of data to the RNG device and then crediting it would give an attacker a window to brute-force the exact data that was written from the fast init pool. If the write-credit cycle is done enough times to fully initialize the input pool, the attacker could then brute-force the initial state of the pool.

The RNDADDENTROPY command combines the write and the credit in the same call, which allows the kernel to make a better decision on what to do with it, he said. With RNDADDTOENTCNT, the kernel does not know whether the data is simply meant to perturb the pool, or whether it should be counted as truly unpredictable data, until after the data has already been processed. He suggested that deprecating RNDADDTOENTCNT might be a good plan; Ts'o concurred on that.

Alex Xu did some research on the uses of RNDADDTOENTCNT in existing code, which led Donenfeld to question his patch. There are, it seems, programs that do exactly what Donenfeld was worried about. For example, maxwell is a daemon to feed jitter entropy into the Linux RNG; Xu said it operates as follows:

sandy-harris/maxwell is a "jitter entropy" daemon, similar to haveged. It writes 4 bytes of "generated entropy" to /dev/random, then calls RNDADDTOENTCNT, then repeats.

Donenfeld replied: "Okay bingo. The existence of this means that this patch will definitely introduce a new vulnerability." An attacker could brute-force the data 32 bits at a time, so his patch would need to change, he said. Part of the problem is that some early boot efforts to initialize the RNG in user space are not actually doing so unless there is entropy being credited during that process. David Laight said: "You can't really expect startup scripts to be issuing ioctl requests.". But that is exactly what is required if there are no other sources of entropy, as Donenfeld explained:

Crediting bits has required an ioctl since forever. Those shell scripts have been broken forever. The proposal here is to add new behavior to support those old broken shell scripts.

Fortunately, it seems sort of fixable. But only sort of. There are also a lot of complications, as detailed above. Part of it is that some people use /dev/urandom writes expecting it not to credit, while others use that and then later manually credit it with the ioctl. Both of those in general seem like not a very good interface for seeding the rng. The correct interface to use is RNDADDENTROPY, which takes both the data and whether it should be credited, since then the kernel knows what the intentions are and can do something smart with it. Barring that knowledge, we're in this vague middle ground where it's unclear what the user intends to do.

There was some discussion of possible heuristics for when to credit entropy for writes to the RNG devices. for example only during early boot or when the process has certain privileges, but all of those are fraught. There are other cases where crediting that entropy could be problematic or catastrophic. Eric Biggers provided an example of where things might go wrong. The Android system writes its kernel command line to /dev/urandom early in the boot process with the expectation that it will not count as entropy, "given that the command line might not contain much entropy, or any at all".

For those reasons, Donenfeld came to the conclusion that not changing the current behavior was the right thing to do.

Based on this, the fact that shell scripts cannot seed the RNG anyway, and due to the hazards in trying to retrofit some heuristics onto an interface that was never designed to work like this, I'm convinced at this point that the right course of action here is to leave this alone. There's no combination of /dev/urandom write hacks/heuristics that do the right thing without creating some big problem elsewhere. It just does not have the right semantics for it, and changing the existing semantics will break existing users.

His plan is to work with user-space utilities to make changes that reflect the current reality, so that file-based seeding works well everywhere. To that end he has created a simple SeedRNG program that is intended to be incorporated into programs that need this kind of functionality. If that work is successful, it "might lead to a better ecosystem and less boot time blocking and all that jazz".

It is clear that Donenfeld has injected some needed energy into the maintenance of the kernel RNG code. There is a lot of work going on in that area right now and, seemingly, more to come. For now, we are still facing that longtime kernel bugaboo, entropy gathering woes early in the boot process, but perhaps there is some light on the horizon in the form of other techniques that might improve the situation. In the meantime, reworking user space to properly use the facilities we do have looks like the right approach.

Comments (21 posted)