LWN.net Weekly Edition for July 14, 2022

Welcome to the LWN.net Weekly Edition for July 14, 2022

This edition contains the following feature content:

• "Critical" projects and volunteer maintainers: many maintainers aren't prepared for the responsibilities that come with a popular package.
• Distributors entering Flatpakland: community distributions are looking to change how they package at least some of their applications, but the user community has some doubts.
• The trouble with symbolic links: Jeremy Allison's SambaXP talk on why symbolic links were a mistake.
• Kernel support for hardware-based control-flow integrity: an overview of how well the kernel supports various control-flow integrity features offered by CPU vendors.
• Native Python support for units?: the Python community considers enabling literal values with attached units.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

"Critical" projects and volunteer maintainers

Over the last five decades or so, free and open-source software (FOSS) has gone from an almost unknown quantity available to only the most technically savvy to underpinning much of the infrastructure we rely on today. Much like software itself, FOSS is "eating the world". But that has changed—is changing—the role of the maintainers of all of that code; when "critical" infrastructure uses code from a FOSS project, suddenly, and perhaps without warning, that code itself becomes critical. But many maintainers of that software are volunteers who did not set out to become beholden to the needs of large companies and organizations when they released their code, they were just scratching their itch—now lots of others are clamoring for theirs to be scratched as well.

The supply-chain security problem is clearly a serious one that needs to be addressed. The Log4j incident provides a recent example of how a security vulnerability in a fairly small component can ripple out across the internet by way of dependency chains. Some projects depended directly on Log4j, but many others became vulnerable because they were using some other library or package that depended on Log4j—directly or indirectly.

Some of the places where dependency chains are often lengthy, and thus more vulnerable to the intentional injection of malware, are various language-specific repositories of packages. Sites like the Python Package Index (PyPI) provide a huge palette of components that can be used by applications or other libraries. The pip tool that comes with Python will happily install PyPI packages along with all of their dependencies, recursively. Many other languages have similar repositories and tooling.

Critical components

There are multiple efforts these days to identify the most critical dependencies and to provide assistance to those projects so that they do not end up in the position of a pre-Heartbleed OpenSSL—or represent that one project in the classic xkcd. For example, the Open Source Security Foundation (OpenSSF) has its Alpha-Omega project that is identifying projects needing assistance with their security. PyPI has also been identifying its packages that have been downloaded the most over the last six months based on its public data sets; those that are in the top 1% are deemed "critical". Roughly 3500 projects have been identified in this manner and the maintainers of those projects are being offered a free security key to help them set up two-factor authentication (2FA) for their PyPI accounts.

Authentication using 2FA is not currently required for any packages, but PyPI plans to require it for maintainers of critical projects "in the coming months". Once that goes into effect, maintainers who have not enabled 2FA (using a security key or time-based one-time password (TOTP) application) will presumably not be able to make changes, such as updating the package. That, of course, has its own risk, in that a critical package may not be able to get the update it needs for some serious vulnerability because its maintainers failed to sign up for 2FA.

On July 8, Skip Montanaro posted a message to the Python discussion forum noting that a defunct project of his, lockfile, had been identified as critical. The project had been marked as deprecated at the top of its README (with alternatives listed) and has not seen any releases since 2015. He wondered why it was considered critical and asked: "What should I do to get rid of this designation?"

Donald Stufft said that the package is being downloaded roughly 10-million times per month. Dustin Ingram pointed to the FAQ in the security-key giveaway announcement that says "once the project has been designated as critical it retains that designation indefinitely", so lockfile will be considered critical henceforth. The lockfile module is part of the OpenStack project; the README file for lockfile suggests contacting the openstack-dev mailing list for assistance in moving away from it.

It turns out that "no OpenStack projects declare direct dependencies on lockfile since May 2015", according to "fungi", who is a system administrator for OpenStack. But lockfile is still used by parts of the OpenStack project. In a perfect demonstration of the insidious nature of dependency chains, fungi tracked down its use by the project:

I've found that some OpenStack projects depend on ansible-runner, which in turn depends on python-daemon, which itself declares a dependency on lockfile. I'll need to confer with other contributors on a way forward, but probably it's to either help python-daemon maintainers replace their use of lockfile, or help ansible-runner maintainers replace their use of python-daemon.

So most or all of the downloads of this "critical" PyPI project are probably for continuous-integration testing of OpenStack and the components that use lockfile should likely have replaced it with something else nearly eight years ago. Hugo van Kemenade suggested encouraging people to stop using it; "if you're still in a position to release, emit a DeprecationWarning on import suggesting the replacements. Or something noisier like a UserWarning." Paul Moore noted that marking it as deprecated did not work, nor did ceasing releases in 2015; "I'm not at all sure 'tell people not to use it' is a viable strategy for getting marked as 'not critical'."

Opinions

On July 9, Armin Ronacher posted his thoughts about PyPI's 2FA requirement; that post was extensively discussed here at LWN, at Hacker News, and elsewhere. Ronacher makes it clear that he does not see 2FA as an unreasonable burden for maintainers of PyPI projects, but he does wonder where it all leads. For one thing, it is, apparently, only critical packages at PyPI that will be required to have 2FA set up, so "clearly the index [PyPI] considers it burdensome enough to not enforce it for everybody".

That creates something of a double standard. As Ronacher put it, he did not set out to create a critical package, that was something that happened organically. But the kinds of problems that can be prevented through 2FA, such as a malicious actor logging into PyPI with stolen credentials, can happen with any package, not just popular ones. "In theory that type of protection really should apply to every package."

But there is also a question of what else might be required down the road. When the projects at PyPI today were created, there was no mention of 2FA, so other things may be added down the road as well.

There is a hypothetical future where the rules tighten. One could imagine that an index would like to enforce cryptographic signing of newly released packages. Or the index wants to enable reclaiming of critical packages if the author does not respond or do bad things with the package. For instance a critical package being unpublished is a problem for the ecosystem. One could imagine a situation where in that case the Index maintainers take over the record of that package on the index to undo the damage. Likewise it's more than imaginable that an index of the future will require packages to enforce a minimum standard for critical packages such as a certain SLO [service level objective] for responding to critical incoming requests (security, trademark laws etc.).

Some of those requirements make perfect sense from a security standpoint; in fact, some should perhaps be in place already. But there is now an ongoing discussion about disallowing projects from being deleted from PyPI. Obviously deleting a project that other projects rely on is kind of an antisocial act, but it does seem like something the author (and probably copyright holder) should be allowed to do. It can lead to chaos like the famous left-pad fiasco, however.

The recent 2FA push from PyPI led a maintainer to accidentally remove all of the old releases of the atomicwrites package. As noted by Stufft in the PyPI deletion discussion linked above, he restored the atomicwrites releases at the request of the maintainer, but "it took about an hour to restore 35 files". Finding a way to head off those kinds of mistakes would be useful in addition to preventing downstream chaos when a maintainer deletes their project.

As Ronacher noted, he is using the resources of PyPI for the distribution of his projects, so he is willing to follow its rules, which are aimed at protecting the users of the index. But PyPI (and other similar repositories for other languages) have something close to a monopoly over project distribution, since pip is tied to it. He wondered if a solution along the lines of the cargo vet tool for Rust might mean that package indexes can get out of the job of enforcing security policies and to leave it to others to do so:

What I like about the cargo-vet approach is that it separates the concerns of running an index from vetting. It also means that in theory that multiple competing indexes could be provided and vetting can still be done. Most importantly it puts the friction of the vetting to the community that most cares about this: commercial users. Instead of Open Source maintainers having to jump through more hoops, the vetting can be outsourced to others. Trusted "Notaries" could appear that provide vetting for the most common library versions and won't approve of a new release until it undergoes some vetting.

Reaction

Django developer James Bennett had a sharply worded reply to Ronacher on July 11 (which was also discussed at Hacker News and no doubt elsewhere). In much of it, Bennett seems to be reacting to the arguments that others are making, rather than those that Ronacher made. But Bennett's main complaint with Ronacher is that he thinks the cargo vet approach is flawed and that those who release FOSS have a responsibility to users in an "ethical and social sense", even though any legal responsibility has been disclaimed in the license. "Yeah, if you publish open-source code you do have some responsibilities, whether you want them or not."

Bennett's list of responsibilities for a FOSS maintainer seem generally reasonable, "because what they really boil down to is the basic societal expectation of 'don't be an asshole'". But he is raising a strawman here, since Ronacher never argued that maintainers should be (allowed to be) assholes. Ronacher simply wondered what other requirements might be imposed on maintainers over time, some of those that he mentioned (e.g. a service level objective) would be quite onerous for a volunteer maintainer.

Bennett's weakest argument seems to be that Ronacher owes more to his users than he might voluntarily choose to give because his work on FOSS has opened various doors for him. It is a fairly strange argument, in truth. Overall, Bennett seems to be addressing lots of things that Ronacher did not say, or even imply. The heart of what Ronacher was trying to do was to try to figure out where the boundaries are, not to claim they had already been crossed.

It seems vanishingly unlikely that PyPI will be establishing two-day security-fix timelines, for example, on its critical projects, but there are surely lots of companies and other organizations out there that wish it would. There is a general tendency for all humans (and their constructs like companies) to shirk responsibilities if they can find another to pin them on. Companies and organizations that are shipping software that is dependent on the FOSS supply chain need to be deeply involved in ensuring that the code is secure.

Doing that work will cost a lot of money and take a lot of time. We are seeing efforts to do that work, and the PyPI 2FA requirement is one of those pieces. It is hardly a panacea, but it is a useful step.

As Luis Villa noted last year, FOSS maintainers are being asked to do more and more things; often they are being asked to do so without any compensation, though perhaps "doors opening" counts to a limited extent. As more critical projects are identified, it is likely we will see more conflicts of this nature. What happens when a maintainer does not want to follow the recommendations of OpenSSF (or some other similar effort) on changes? Forks are generally seen as a hostile move, but one suspects that may ultimately happen for projects that find themselves at odds with sponsoring organizations. That is a rather different world than the one FOSS grew up in.

Comments (67 posted)

Distributors entering Flatpakland

Linux distributions have changed quite a bit over the last 30 years, but the way that they package software has been relatively static. While the .deb and RPM formats (and others) have evolved with time, their current form would not be unrecognizable to their creators. Distributors are pushing for change, though. Both the Fedora and openSUSE projects are moving to reduce the role of the venerable RPM format and switch to Flatpak for much of their software distribution; some users are proving hard to convince that this is a good idea, though.

A traditional binary package from a distribution contains the program of interest, of course, along with any supplementary files that it needs. The package also carries metadata describing, among other things, which other packages must be installed for the program to work. The distribution's package manager uses that dependency information to ensure that every package is properly installed.

The Flatpak format has been described as "just another distribution format" and, to an extent, it is true. A Flatpak package (or, simply, "a flatpak") has everything that a .deb or RPM package would have, but there are some significant differences. Perhaps at the top of the list is the way that dependencies are handled. A traditional package will have a (possibly long) list indicating every other package that is needed; a Flatpak package, instead, will list a single "runtime" containing the base set of libraries against which the package is built. If there are libraries or other dependencies that do not appear in the runtime of choice, they are simply bundled with the application in its flatpak.

This arrangement has a certain appeal to packagers. The "runtime plus bundling" approach simplifies dependency management, and the ability to bundle patched versions of system or runtime libraries is called out as a Flatpak feature. A package built against a given runtime can be installed on any system that has that runtime installed, making it possible to build a single package that can be installed on multiple distributions. Distributors can thus use this format to make their lives easier; purveyors of proprietary packages also see some obvious charm in this idea.

In a sense, Flatpak has set out to solve many of the same problems that the ill-fated Linux Standard Base effort addressed many years ago.

Flatpak is designed to work with distributions that are built around an immutable core; it will not install into system directories like /etc or /usr/bin. That makes it attractive for distributors that are trying to create that sort of offering.

One other key selling point for Flatpak is the built-in sandboxing mechanism. Applications installed with Flatpak will, by default, be run inside a container that has almost no access to the underlying system or the Internet. Through declarations in the manifest file used to build the package, a specific application can request access to, for example, the user's local files, the Internet, or the window-system server. In theory, at least, this sandbox makes installing packages safer and limits the damage that can be caused by a compromised or malicious package.

Flatpak is not without its critics. While packages can share the same runtime, there is nothing to prevent a proliferation of runtimes and versions; that already seems to be happening to an extent. Both the runtimes and the packages themselves are large, significantly expanding the amount of storage space and memory consumed. It is common for applications to request access to local files and the Internet whether they need it or not, to the point that some see Flatpak's sandbox claims as being more hype than reality.

All of that adds up to a certain amount of opposition to the use of Flatpak. It also seems possible that much of the opposition to Flatpak may result from an instinctive resistance to changing the way things have always been done.

Fedora's (un)filtered Flathub

On the Fedora side, the discussion has been stirred up by this Fedora 37 change proposal removing the filters currently applied to the Flathub package repository in Fedora's GNOME Software application. Once the filter is gone, all of the packages available on Flathub, including those containing proprietary software or programs that, for reasons such as patent concerns, cannot be shipped with Fedora, will be available to be installed. Flathub as a whole will still be disabled, though, unless users enable third-party repositories.

The advantages of this change are said to be that it makes more software readily available to Fedora users and makes that software more secure by sandboxing it. The latter reason is part of why the GNOME Software tool will prefer flatpaks over traditional RPM packages when both are available. But there are some vocal members of the Fedora community who are strongly opposed to the idea. Vitaly Zaitsev, for example, complained about the preference for flatpaks, saying that it would cause flatpaks to replace RPM packages as a system is updated. A lot of the points mentioned above — install size and porous sandboxes, for example — have come up in this discussion, as has the perceived bias against the RPM Fusion repository, which was discussed in depth here in June.

It seems unlikely that these objections will stop the unfiltering of Flathub — or the transition toward Flatpak in general. The distribution seems committed to that direction, and the Silverblue variant, which is built around an immutable core installation, already requires Flatpak. In the discussion, Michael Catanzaro extolled the security benefits of the format and pointed out that, despite years of complaints, Flatpak opponents have not come up with a viable alternative that provides similar security benefits. Richard Hughes argued that "flatpaks are in almost all cases what users should be using". Unless something changes, it looks like full speed ahead for Fedora's Flatpak future.

OpenSUSE's ALPs are flat

Back in April, SUSE announced an effort known as the "Adaptable Linux Platform" (ALP) that is, in some way, meant to define the future of the SUSE Linux Enterprise (SLE) distribution. There was a lot of text about how the process would be open, but outside observers might be forgiven for thinking that ALP remains fairly murky so far. There is not much public information yet on just what ALP will be or how it will affect either SUSE's enterprise offerings or openSUSE. The hints that have come out, though, suggest a distribution built on a small, immutable core with applications managed separately, most likely using a format like Flatpak.

On July 5, Dan Čermák posted the minutes from a meeting of the ALP working group that was focused on "how the ALP Desktop is going to look like and how we can dispel fears around flatpaks and containers when it comes to the ALP desktop". The minutes note that Flatpak will indeed be used as a distribution format, but stressed that there would be no need to install flatpaks from external sources like Flathub; "the packaging model will not change, only the format".

When asked about the benefits driving a switch to flatpak, Čermák mentioned the sandboxing mechanism, but said that the main advantage was in the reduction of packaging work. Desktop applications like Firefox or LibreOffice could be built once on the rolling openSUSE Tumbleweed distribution, then shipped for Tumbleweed, Leap, ALP, and presumably anything else SUSE might create. It is not hard to see why distributors are attracted to an option that could significantly decrease the amount of work required to maintain current packages of complex applications.

There was some fundamental opposition to use of Flatpak in the openSUSE community as well. Robert Kaiser, for example, said "if there's no RPM-based distro left without that weird packaging like flatpak, it's probably time to go back to Windows". As a whole, though, the openSUSE community seems less stirred up by the prospect — so far at least. There may be a few reasons for that, including a relative lack of experience with Flatpak in the openSUSE community and the amorphous nature of ALP in general. But this reassurance from Čermák can only help as well:

Tumbleweed will be the place where development is going to take place, but there are no plans to fundamentally alter Tumbleweed. You'll still be able to use Tumbleweed like you used before. Additionally, you'll be able to use the "containerized stuff", but you won't have to, at least for the foreseeable future.

There may soon come a time where use of Flatpak on Fedora is unavoidable, at least when it comes to installing certain complex applications. Thus far, openSUSE does not appear to be driving in that direction, so Tumbleweed users (who make up a large part of the openSUSE community) need not fear being pushed unwillingly into the Flatpak world.

One should not underestimate the value of that last point. Much of the bitterness around the systemd transition was certainly caused by the feeling that it was being forced on users who were happy with their existing setup and felt no need to change. Even if Flatpak is as great as some claim, a poorly handled transition risks creating similar levels of unhappiness.

That transition seems likely to come; in a later posting Čermák questioned the value of packinging some complex applications and suggested that they may be made available only in flatpak form if they are shipped by the distribution at all. Robert Schweikert said more succinctly: "What we have today has worked reasonably well for probably about 30 years but the model is certainly showing it's age". Evolving to meet current needs is exactly what the distributors need to be doing, and that evolution will surely bring changes that are visible to end users. The key will be handling the changes with sympathy for those users and, hopefully, avoiding some of the turbulence we have seen with past changes.

Comments (136 posted)

The trouble with symbolic links

At the 2022 sambaXP conference, Jeremy Allison gave a talk titled "The UNIX Filesystem API is profoundly broken: What to do about it?". LWN regulars may recall hints of these talks in a recent comment thread. He started his talk with the problems that symbolic links ("symlinks") cause for application developers, then discussed how the solutions to the problems posed by symlinks led to substantial increases in the complexity of the APIs involved in working with pathnames.

Allison explained that hard links were the first "interesting addition" to the original Unix filesystem API; unlike symlinks, though, they are not dangerous, and are, in fact, easy to use. A hard link is simply the connection between a directory entry and the inode for the file (or directory) to which that entry refers. Unix systems allow multiple links to any file, but require that the inode and directory entries all reside on the same filesystem.

By contrast, symlinks contain another path as data, and the kernel transparently operates on the file at that path when system calls like open() or chown() are called on the symlink. This seemingly innocuous feature led to the addition of incredible amounts of complexity in the effort to fulfill the needs of programs that need to be aware of whether a pathname contains a symlink or not. Such programs include archival programs like tar, file synchronization and transfer programs such as rsync, network filesystem servers like Samba, and many more that suffer security problems as a result of not giving sufficient attention to symlinks in pathnames.

The variety of security problems resulting from symlinks can be seen in a search of CVE entries, which gave Allison 1,361 results when he ran it. These include vulnerabilities that facilitate information disclosure, privilege escalation, and arbitrary file manipulation including deletion, among other attacks. Without discussing any specific CVE in detail, he gave an example of the kind of security problem that can result from symlink-related vulnerabilities.

An application running as root may try to check that /data/mydir is a regular directory (not a symlink) before opening the file /data/mydir/passwd. In between the time the program does the directory check and the file open, an attacker could replace the mydir directory with a symlink to /etc, and now the file opened is, unexpectedly, /etc/passwd. This is a kind of race condition known as a time-of-check-to-time-of-use (TOCTOU) race.

Symlinks and complexity

Symlinks were created, Allison theorized, because hard links are restricted to linking within the same filesystem, so only symlinks (which lack that restriction) could be used if an administrator wanted to add new storage media without changing the paths to users' data. He quoted an advertisement for 4.2BSD, which touted, "This feature frees the user of the constraints of the strict hierarchy that a tree structure imposes. This flexibility is essential for good namespace management."

The addition of symlinks led to the lstat() system call, which provided the means to identify whether the last component in a pathname is a symlink. This was, unfortunately, insufficient for handling symlinks pointing to directories earlier in the path, he explained. An application could attempt to check each component of the path individually, but not atomically — another application could make a change to one of the components during this process, leading to security vulnerabilities.

An option to the open() system call, O_NOFOLLOW, exhibits the same problem as lstat(). O_NOFOLLOW instructs the system call to fail with ELOOP if the last component in the pathname is a symbolic link, but it only checks the last component. The realpath() C library function follows symlinks in a path and produces an absolute, canonical pathname that the application can then compare with the original. Allison described this as an appealing but incorrect solution to the problem. Another process could make a change in between the time realpath() is called and another function is used to manipulate the file in some fashion. In other words, the same TOCTOU race applies here.

Allison said that the openat() system call was designed as a solution to this problem; it introduces the idea of referring to files with respect to a directory that's indicated by an already-open file descriptor. The only reliable way to identify a file's path is to walk the hierarchy using multiple calls to openat(). Everything else would be vulnerable to race conditions.

But Allison also pointed out the flaw in this technique. "You cannot create a new directory with open(), you cannot remove a file, unlink a file, or delete a directory with an open() call." So, more functions following the pattern of openat() had to be created: mkdirat(), linkat(), unlinkat(), renameat(), and more. Some are still missing, like getxattrat() and setxattrat(). Some, like fchownat() and faccessat(), don't follow the pattern cleanly.

Allison didn't mince words: "So our original clean and beautiful POSIX filesystem API doesn’t look very clean and beautiful anymore...pathnames as a concept are now utterly broken in POSIX." One could reasonably attribute, in part at least, any perceived bitterness to Allison's struggles with the long road to a fix for CVE-2021-20316 in Samba.

Because of the talk's focus on the role of symlinks in complicating the Unix pathname API, Allison did not directly raise the point that race conditions involving pathnames can occur even without symlinks. It seems a major source of complexity is the lack of a mechanism for atomically batching together operations that involve walking directories and symlinks to eventually perform some operation on a file.

Workarounds

Allison then explained the use of the O_PATH flag to open(), which will return a file descriptor that is only useful for passing to the *at() system calls as the dirfd argument. Unfortunately for Samba, file descriptors opened with O_PATH cannot be used for reading or writing extended attributes. He found a workaround, demonstrated by a snippet of code that he described as "one of the most beautiful hacks I’ve ever seen, it’s so ugly it makes you want to vomit, but it’s amazing."

    int fd = openat(dirfd, "file", O_PATH|O_NOFOLLOW);
    sprintf(buf, "/proc/self/fd/%d", fd);
    getxattr(buf, ea_name, value, size);

The contents of /proc/self/fd are symlinks that represent every file descriptor the process has open. Allison explained the code: "If you print into a buffer '/proc/self/fd/' and then the number of the descriptor you just got back from O_PATH, you can then pass it to getxattr() or setxattr() as a path, and it can’t be symlink-raced." He wasn't sure whether to attribute this code to Android or Red Hat developers, but a similar use of /proc/self/fd/ can be found in the open() man page.

Allison reiterated the main point of his talk: "The concept of pathnames is unusable on POSIX, completely. For a non-trivial application, for a regular person writing code on POSIX, you will have symlink races in your code."

Examples of (since fixed) CVEs were then provided, including one in the Rust standard library, which was discussed extensively here. In the last few minutes of the talk, Allison noted several proposed solutions offered by LWN readers, including a special prctl() call and restrictions on when non-root symlinks are followed. He said that the MOUNT_NOSYMFOLLOW mount option, which simply forbids following symlinks within a filesystem, is his preferred solution: "It’s perfect. It does exactly what we need." Allison's talk concluded on that point.

While it certainly seems desirable to forbid symlinks in the name of cleaning up the POSIX API, they are a frequently used system-administration tool. Several popular "symlink managers" exist. Gnu Stow, for example, provides a way for administrators to install programs into a new directory hierarchy, such as /usr/local/stow/packagename-version/, and then create forests of symlinks from /usr/local/bin/example to /usr/local/stow/packagename-version/bin/example, using the minimum number of symlinks necessary. This makes it possible to "uninstall" a package simply by removing the symlinks with the help of stow -D.

The /etc/alternatives system created by Debian allows administrators to switch between substitutable packages in a similar manner without forcing the uninstallation or reinstallation of either package. In a similar vein, the Nix and Guix distributions make heavy use of symlinks — a Guix profile consists of a tree of symlinks to packages installed within /gnu/store/, making it easy to switch between grouped combinations of specific versions of packages.

Banning symlinks entirely would break these use cases, but restricting their creation to the root user would most likely suffice. Users may still have other legitimate needs for symlinks, though, and substantially restricting them would likely be an unpopular change.

SambaXP has made the talk's video and slides available.

Comments (82 posted)

Kernel support for hardware-based control-flow integrity

Once upon a time, a simple stack overflow was enough to mount a code-injection attack on a running system. In modern systems, though, stacks are not executable and, as a result, simple overflow-based attacks are no longer possible. In response, attackers have shifted to control-flow attacks that make use of the code already present in the target system. Hardware vendors have added a number of features intended to thwart control-flow attacks; some of these features have better support than others in the Linux kernel.

Control-flow integrity (CFI) is a set of techniques intended to prevent control-flow attacks, or at least to reduce the ability of an attacker to hijack the program's control flow. The general idea behind CFI is to label sources and destinations of indirect jumps (such as calls, branches, and return instructions) and verify at run time that the actual jump target matches the label. CFI can be implemented entirely in software, but there are also several hardware mechanisms from different vendors that assist in CFI implementation.

Coarse-grain forward-edge protection

One common way to corrupt a program's control flow is via indirect jumps, such as function calls through a pointer variable. If that pointer can be somehow overwritten, control can be rerouted to a location of an attacker's choosing, with unfortunate results. Just about any program of reasonable size will contain code segments that will do unwanted things if control lands in the middle of them. The term "forward edge" is used to describe attacks on outgoing control flow. Purely software-based protection against forward-edge attacks exists (forward-edge CFI in LLVM, for example), but hardware-based mechanisms exist as well.

The arm64 and x86 architectures enable coarse-grain protection of the forward edges using a dedicated BTI (arm64) and ENDBR{32,64} (x86) instructions, which create a "landing pad" for indirect branches. When an indirect branch is executed, the CPU will check for the landing pad at the target address; if a landing pad is not found, the CPU will trap. Linux supports these protections for the kernel itself on both arm64 and x86, however for now only arm64 supports enabling this feature for user space; the x86 version is still under development.

While protection of forward edges prevents jumps into the middle of code blocks, it is still possible to find useful code immediately following those landing pads and, of course, there are also the backward edges to exploit.

Return address integrity

Return-oriented programming (ROP) attacks use stack overflows to replace the return address of a function with the address of a machine-code sequence that performs some action the attacker needs and which ends with return instruction. Such a sequence is called "gadget". It is possible to chain multiple gadgets and, as the size of the program under attack grows, it becomes increasingly easy for an attacker to create a chain that performs any desired operation. The term "backward edge" is often used to describe ROP attacks.

Many architectures store return addresses in a special link register. The non-leaf functions, though, must save the contents of the link register on the stack before calling another function, then restore the link register after that function returns. To protect the on-stack copy of the return address, the arm64 and powerpc architectures allow storing a cryptographically calculated hash alongside the address. These architectures provide special instructions that can generate a random key for crypto operations, create of a hash based on that key and the address value, and validate that the hash matches the value loaded from the stack.

There are differences in where that hash is stored; on powerpc systems it's saved on the stack next to the pointer it protects, while on arm64 systems the hash occupies unused high bits in the pointer itself. The powerpc and arm64 mechanisms are both supported by GCC and LLVM, but only arm64 has proper support for this feature (called pointer authentication) in the Linux kernel.

Another way to ensure backward-edge integrity is to use a shadow stack to supplement the regular stack. The shadow stack stores the return address for each function that is called; whenever a function returns, the return address on the normal stack is compared to the address on the shadow stack. Execution is only allowed to continue if the two addresses match. Both Intel and AMD implement shadow-stack support in hardware, so that every CALL instruction saves the return address to both the normal and the shadow stack, and every RET instruction verifies that return addresses match. When the return addresses differ, the CPU generates a control-protection fault. The shadow-stack implementation also affects behavior of other instructions that change the control flow, including INT, IRET, SYSCALL, and SYSRET.

A shadow stack is more demanding to support than pointer authentication. The last version of the patches that enable support for shadow stacks for user space contained 35 patches and was covered in depth here. Along with enablement of the hardware's shadow-stack functionality and plumbing it into the core kernel, these patches define new kernel APIs that reserve memory for the shadow stack, enable and disable the shadow-stack functionality, and can also lock the state of the shadow-stack features. These APIs are intended to be used by the C library; the overall presence of a shadow stack should be transparent for most applications. These patches have not yet been merged into the mainline kernel, though.

Special cases

Some applications, like GDB and CRIU, need the ability to control the execution of other processes, meaning that they require ways to deal with the shadow stack in a nonstandard way. The GDB debugger, for example, often needs to jump between the stack frames of the program being debugged; it needs to keep the shadow stack in sync with the normal stack as it moves up and down the call chain.

The contents of the shadow stack can be updated using the PTRACE_POKEUSER command to ptrace(), but another ptrace() call is then required to update the shadow-stack pointer. There was a proposal to extend PTRACE_GETREGSET and PTRACE_SETREGSET to support access to the registers controlling the shadow-stack machinery; it was posted as a part of version 11 of the "Control-flow Enforcement: Indirect Branch Tracking, PTRACE" series, but has not been reposted since then. This interface is used by Intel's fork of GDB, but what the final form of the kernel API for manipulating shadow-stack control registers will be remains unclear.

Like GDB, CRIU has an intimate relationship with the processes it checkpoints and restores. The ptrace() interfaces intended for GDB are certainly useful for CRIU, but they are not enough. Beside the ability to adjust shadow-stack contents and the shadow-stack pointer, CRIU must be able to restore the shadow stack at the same virtual address as is was at the time of checkpoint. A possible way to achieve that is to extend the proposed map_shadow_stack() system call to accept an additional address parameter; when this parameter is not zero, map_shadow_stack() will behave similarly to mmap(MAP_FIXED) and will attempt to reserve the memory for the shadow stack at the requested location.

Another issue CRIU has to cope with is the need to restore the shadow stack's contents. Of course, this could be done using ptrace(), but that would be slow and would require major changes to the restore logic. A better way is to use a special WRSS instruction that lets an application write to its own shadow stack. But this instruction is only available when the shadow stack is enabled, which presents a problem of its own.

When the GNU C Library loads a program, it enables or disables shadow-stack features for that program based on its ELF header, then locks the feature state permanently. CRIU uses clone() to create the restored tasks, with the result that they inherit the shadow-stack state from the CRIU control process. So, if CRIU was built with the shadow stack enabled, it wouldn't be able to restore tasks without shadow stack and vice versa. A solution to this was an additional ptrace() call to override the shadow-stack feature lock.

CRIU also needs patches to support pointer authentication because, despite the feature being available for some time now, nobody got around to implementing it in CRIU.

What comes next

We have not yet seen what the final form of the kernel APIs exposed by the shadow-stack implementation will be after it actually lands in the upstream kernel. After that, GDB, CRIU, and maybe other applications that do tricky things with their control flow should be updated to cope with restrictions that shadow stacks pose. And, after the shadow-stack support for user space settles, there is another interesting question to answer: what would it take to support a shadow stack for the Linux kernel and are the complexity and the effort required worth it?

Comments (8 posted)

Native Python support for units?

Back in April, there was an interesting discussion on the python-ideas mailing list that started as a query about adding support for custom literals, a la C++, but branched off from there. Custom literals are frequently used for handling units and unit conversion in C++, so the Python discussion fairly quickly focused on that use case. While ideas about a possible feature were batted about, it does not seem like anything that is being pursued in earnest, at least at this point. But some of the facets of the problem are, perhaps surprisingly, more complex than might be guessed.

Custom literals

On April 1, Will Bradley posted a, presumably non-joking query about custom literal support for Python; "has this been considered and rejected, or is there a reason it's unpopular?" According to Stephen J. Turnbull, user-defined syntax for Python has a been a hard sell, in general, though literal syntax for units (e.g. "10m" for meters or "1.2kW" for kilowatts) has gotten somewhat further. In addition, the idea of adding a literal syntax for fixed-point constants that use the decimal package also crops up with some frequency, he said. His recollection is that "in general Python has rejected user-defined syntax on the grounds that it makes the language harder to parse both for the compiler and for human beings".

Brian McCall warned that he was getting up on his soap box, but said that he strongly supported Python (and, indeed, all programming languages) having native units. "It's not often that I would say that C++ is easier to read or more WYSIWYG than Python, but in this case, C++ is clearly well ahead of Python." He suggested that "lack of native language support for SI units" is particularly problematic because of the prevalence of scientific computing today. SI units are the international system of units, which are often referred to as the metric system.

Anyone who has ever dealt with units will immediately recognize a problem associated with enshrining only the SI units into Python (or anywhere else): other measurement systems exist, including imperial and US units and even other metric systems. As Ricky Teachey put it:

BUT- SI units isn't enough. Engineers in the US and Canada (I have many colleagues in Canada and when I ask they always say: we pretend to use SI but we don't) have all kinds of units.

Give us native, customizable units, or give us death! Who's with me??!!

Units

Beyond the conflict over measuring-system support, there are other fundamental questions that need to be resolved before new syntax could be added, Chris Angelico said. The number "4K" might mean four kelvins, 4000, or 4096, depending on context, for example. Angelico suggested that an existing bit of syntax could be repurposed for units:

But I would very much like to see a measure of language support for "number with alphabetic tag", without giving it any semantic meaning whatsoever. Python currently has precisely one such tag, and one conflicting piece of syntax: "10j" means "complex(imag=10)", and "10e1" means "100.0". (They can of course be combined, 10e1j does indeed mean 100*sqrt(-1).)

[...] In Python, I think it'd make sense to syntactically accept *any* suffix, and then have a run-time translation table that can have anything registered; if you use a suffix that isn't registered, it's a run-time error. Something like this:

    import sys
    # sys.register_numeric_suffix("j", lambda n: complex(imag=n))
    sys.register_numeric_suffix("m", lambda n: unit(n, "meter"))
    sys.register_numeric_suffix("mol", lambda n: unit(n, "mole"))

[...] Using it would look something like this:

    def spread():
	"""Calculate the thickness of avocado when spread on
	a single slice of bread"""
	qty = 1.5mol
	area = 200mm * 200mm
	return qty / area

Greg Ewing objected to the idea of a global registry since two libraries might want to use the same suffix for different units. But there are other reasons modules might need their own definitions, as Steven D'Aprano pointed out. There are multiple units called a "mile" (e.g. Roman, international, nautical, US survey, imperial, ...), for one thing, but even the definitions of the "same" unit may have changed over time: "a kilometre in 1920 is not the same as a kilometre in 2020, and applications that care about high precision may care about the difference". He thinks that units should be scoped like variables or, at least, have a separate per-module namespace where libraries can register their own units if they wish to.

Angelico did not agree, at least in part because he did not see the need to make things more complex for what he sees as a rare use case. He seemed to be only participant in the thread that thought that way, however, as D'Aprano, Ewing, and others were all fairly adamant that some kind of unit namespace would be needed. Though the reply is perhaps a bit on the rude side, D'Aprano put it this way:

Units are *values* that are used in calculations, not application wide settings. The idea that libraries shouldn't use their own units is as silly as the idea that libraries shouldn't use their own variables.

Units are not classes, but they are sort of like them. You wouldn't insist on a single, interpreter wide database of classes, or claim that "libraries shouldn't create their own classes".

Other possibilities

Ewing suggested a solution that could perhaps be done with no (or minimal) syntax changes:

Treating units as ordinary names looked up as usual would be the simplest thing to do.

If you really want units to be in a separate namespace, I think it would have to be per-module, with some variant of the import statement for getting things into it.

    from units.si import units *
    from units.imperial import units inch, ft, mile
    from units.nautical import units mile as nm

Teachey had mentioned the Pint library in his reply. It is perhaps the most popular Python Package Index (PyPI) library for working with units. Pint comes with a whole raft of units, which can be easily combined in various ways. For example:

    >>> import pint
    >>> ureg = pint.UnitRegistry()
    >>> speed = 17 * ureg.furlongs / ureg.fortnight
    >>> speed
    <Quantity(17.0, 'furlong / fortnight')>
    >>> speed.to('millimeter/second')
    <Quantity(2.82726756, 'millimeter / second')>
    >>> d = 1 * ureg.furlong
    >>> d.to('feet')
    <Quantity(660.00132, 'foot')>
    >>> d.to('mile')
    <Quantity(0.12500025, 'mile')>

At least on my system, Pint seems to have a slightly inaccurate value for a furlong, which is defined as 1/8 mile, or 660 feet; a fortnight is, of course, two weeks or 14 days. That oddity aside, Pint has much of the functionality users might want, but it (and other Python unit-handling libraries) have "so many shortfalls", Teachey said, mostly because they are not specified and used like real-world units are. Beyond Python libraries, the venerable Unix units utility has similar capabilities and can be used directly from the command line (its man page is where the classic "furlongs/fortnight" example comes from). As D'Aprano noted, units has over 3000 different units, which can be combined in a truly enormous number of ways.

Ethan Furman started a new thread from Teachey's message in order to focus specifically on native support for units. He floated his own suggestion for new syntax:

Well, if we're spit-balling ideas, what about:

    63_lbs

or

   77_km/hr

? Variables cannot start with a number, so there'd be no ambiguity there; we started allowing underbars for separating digits a few versions ago, so there is some precedent.

Teachey wondered about the behavior of the "simple tags" being suggested for units:

[...] What should the behavior of this be?

   height = 5ft + 4.5in

Surely we ought to be able to add these values. But what should the resulting tag be?

He also wondered if a more natural-language-like formulation (e.g. 5ft 4.5in) should be supported. Overall, he thinks that figuring out a solution for units in Python would be a "massive contribution" to the engineering world, "but boy howdy is it a tough [nut] of a problem to crack". Angelico said that the "5ft 4.5in" syntax was a step too far in his mind, but using addition should work. "It's not that hard to say '5ft + 4.5in', just like you'd say '3 + 4j' for a complex number."

Angelico went on to describe the benefits of the syntax change over simply defining constants for units, as Ewing suggested. Since, for example, "m" would only be valid as a unit when it was used as a suffix, it would not pollute the namespace for using "m" as a variable. It is also more readable:

If this were accepted, I would fully expect that libraries like pint would adopt it, so this example:

    >>> 3 * ureg.meter + 4 * ureg.cm
    <Quantity(3.04, 'meter')>

could look like this:

    >>> 3m + 4cm
    <Quantity(3.04, 'meter')>

with everything behaving the exact same after that point. Which would YOU prefer to write in your source code, assuming they have the same run-time behaviour?

But Ken Kundert said that units are primarily useful on input and output, not in the calculations within programs and libraries.

The idea that one carries units on variables interior to a program, and that those units are checked for all interior calculations, is naive. Doing such thing adds unnecessary and often undesired complexity.

His QuantiPhy library provides a means for "reading and writing physical quantities". It effectively adds units as an attribute to Python float values so that they can be used when converting the value to a string. But he said that it might make sense to incorporate scale factors and units into Python itself for readability purposes:

For example, consider the following three versions of the same line of code:

     virt /= 1048576
     virt /= 1.048576e6
     virt /= 1MiB

The last is the easiest to read and the least ambiguous. Using the units and scale factor on the scaling constant results in an easy to read line that makes it clear what is intended.

Notice that in this case the program does not use the specified units, rather the act of specifying the units clarifies the programmers intent and reduces the chance of misunderstandings or error when the code is modified by later programmers.

But this suggests that it is not necessary for Python to interpret the units. The most it needs do is to save the units as an attribute so that it is available if needed later.

Library deficiencies?

Beyond just Pint and QuantiPhy, the units module was also mentioned in the thread, so Paul Moore wondered why none of those solutions was acceptable. He pointed out that the @ matrix-multiplication operator was added to the language because of arguments from the NumPy community; "language changes are *more likely* based on a thriving community of library users, so starting with a library is a positive way of arguing for core changes". Turnbull echoed that and also wondered if the typing features could be harnessed to help solve the problem.

In a lengthy message, McCall tried to answer Moore's question, though it is not clear that he really changed any minds. He laid out a complicated calculation, with many different units, and showed how it looked using various existing libraries and the syntax proposed by Angelico; for each he listed a set of pros and cons. One could perhaps quibble with his analysis, but that is not really the point, Moore said; what it shows is that "the existing library solutions might not be ideal, but they do broadly address the requirement". Each has its own pain points, so:

Maybe that suggests that there's room for a unified library that takes the best ideas from all of the existing ones, and pulls them together into something that subject experts like yourself *would* be happy with (within the constraints of the existing language). And if new syntax is a clear win even with such a library, then designing a language feature that enables better syntax for that library would still be possible (and there would be a clear use case for it, making the arguments easier to make).

A somewhat late entrant into the syntax derby (though others had shown similar constructs along the way) came from Matt del Valle who suggested that the numeric types (e.g. int, float) could gain units by way of Python's subscript notation, which might look something like:

    from units.si import km, m, N, Pa

    3[km] + 4[m] == 3004[m]  # True
    5[N]/1[m**2] == 5[Pa]  # True

Moore thought that looked like a plausible syntax, but reiterated his belief that any change would necessarily need to come by way of a library that supporters of "units for Python" developed—and rallied around. That can all be done now, without any need for a PEP or core developer support. After that, a language change could be proposed if it made sense to do so:

Once that library has demonstrated its popularity, someone writes a PEP suggesting that the language adds support for the syntax `number[annotation]` that can be customised by user code. This would be very similar in principle to the PEP for the matrix multiplication @ operator - a popular 3rd party library demonstrates that a well-focused language change, designed to be generally useful, can significantly improve the UI of the library in a way which would be natural for that library's users (while still being general enough to allow others to experiment with the feature as well).

[...] But the library would be useful even if this doesn't happen (and conversely, if the library proves *not* to be useful, it demonstrates that the language change wouldn't actually be as valuable as people had hoped).

[...] So honestly, I'd encourage interested users to get on with implementing the library of their dreams. By all means look ahead to how language syntax improvements might help you, but don't let that stop you getting something useful working right now.

Those who want to try out different syntax changes without actually having to hack on the CPython interpreter directly may be interested in the ideas library. André Roberge, who developed the library, suggested using it as a way to prototype the changes. Ideas modifies the abstract syntax tree (AST) on the fly to enable changes to the input before handing it off to CPython. In another message, he noted that he had implemented the subscript notation for ideas so that it could be tested using Pint or astropy.units.

So far at least, it does not seem like there is a groundswell of activity toward yet another library for units, but one focused in the way that Moore suggested could lead to changes to the language. It may be that the disparate ideas of what unit support would actually mean—and how it would be used—make it hard to coalesce around a single solution. It may also be that the need for additional solutions for Python unit handling is not as pressing as some think. It seems likely that the idea will crop up again, however, so proponents may well want to consider Moore's advice and come up with a unified library before pursuing language changes.

Comments (83 posted)