Leading items

Welcome to the LWN.net Weekly Edition for June 10, 2021

This edition contains the following feature content:

• When and how to evaluate Python annotations: a planned change to annotation evaluation for Python 3.10 gets postponed.
• Fedora contemplates the driverless printing future: printers and their drivers are notoriously painful to set up, but that is changing.
• Rewriting the GNU Coreutils in Rust: providing memory-safe versions of the standard utilities, but under a permissive license.
• Auditing io_uring: adding support for LSMs and the audit subsystem to io_uring is proving somewhat difficult.
• The runtime verification subsystem: a way to verify the proper operation of the kernel through the use of models at run time.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

When and how to evaluate Python annotations

Annotations in Python came late to the party; they were introduced in Python 3 as a way to attach information to functions describing their arguments and return values. While that mechanism had obvious applications for adding type information to Python functions, standardized interpretations for the annotations came later with type hints. But evaluating the annotations at function-definition time caused some difficulties, especially with respect to forward references to type names, so a Python Enhancement Proposal (PEP) was created to postpone their evaluation until they were needed. The PEP-described behavior was set to become the default in the upcoming Python 3.10 release, but that is not to be; the postponement of evaluation by default has itself been postponed in the hopes of unwinding things.

History

It is, as might be guessed, a bit of a tangle, which will require some backstory in order to fully understand things. In 2006, PEP 3107 ("Function Annotations") was adopted for Python 3 to allow the value of arbitrary expressions to be associated with a function's arguments and its return value. The __annotations__ dictionary associated with the function would then contain the data. An example from the PEP is instructive, even if it is somewhat contrived:

For example, the following annotation:

def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
    ...

would result in an __annotations__ mapping of

{'a': 'x',
 'b': 11,
 'c': list,
 'return': 9}

The return key was chosen because it cannot conflict with the name of a parameter; any attempt to use return as a parameter name would result in a SyntaxError.

The interpretation of the values associated by annotations was specifically left out of the PEP: "[...] this PEP makes no attempt to introduce any kind of standard semantics, even for the built-in types. This work will be left to third-party libraries." That all changed with PEP 484 ("Type Hints"), which was adopted in 2015 for Python 3.5; it added a typing module to the standard library to provide "a standard vocabulary and baseline tools" for type annotations. It did not alter other uses of annotations, nor did it add any run-time type checking, it simply standardized the type information for use by static type-checkers. Variable annotations came the following year with PEP 526 ("Syntax for Variable Annotations"), which appeared in Python 3.6.

Some problems were encountered using these annotations, however. For one, forward references to types that have not yet been defined requires using string literals, instead of type names:

For example, the following code (the start of a simple binary tree implementation) does not work:

class Tree:
    def __init__(self, left: Tree, right: Tree):
        self.left = left
        self.right = right

To address this, we write:

class Tree:
    def __init__(self, left: 'Tree', right: 'Tree'):
        self.left = left
        self.right = right

The string literal should contain a valid Python expression (i.e., compile(lit, '', 'eval') should be a valid code object) and it should evaluate without errors once the module has been fully loaded.

Another problem is that evaluating these annotations requires computation, so all programs pay the price of the annotations, even if they are never needed. In addition, type checking was not meant to be done at run time, but potentially complex type annotations will be evaluated every time a module gets imported, which slows things down for no real gain.

The problems brought about PEP 563 ("Postponed Evaluation of Annotations"). Its goal, as the name would imply, was to defer the evaluation of the annotations until they were actually needed; instead, they would be stored in __annotations__ as strings. For static type-checkers, there should be no difference, since they are processing the source code anyway, but if the annotations are needed at run time, they will need to be evaluated—that's where the problems start cropping up.

For code that uses type hints, the typing.get_type_hints() function is meant to be used to return the evaluated annotations for any object; any code using the hints at run time should be making that call already, "since a type annotation can be expressed as a string literal". For other uses of annotations, users are expected to call eval() on the string stored in __annotations__. But there is a wrinkle; both of those functions optionally take global and local namespace parameters:

In both cases it's important to consider how globals and locals affect the postponed evaluation. An annotation is no longer evaluated at the time of definition and, more importantly, in the same scope where it was defined. Consequently, using local state in annotations is no longer possible in general. As for globals, the module where the annotation was defined is the correct context for postponed evaluation.

The switch to deferred evaluations was gated behind a __future__ import:

    from __future__ import annotations

The feature was available starting with Python 3.7 and the plan was to switch to deferred evaluations as the default behavior for Python 3.10, which is due in October. But in mid-January, Larry Hastings found some inconsistencies in the ways __annotations__ were handled for different types of objects (functions, classes, and modules) while he was working on a new PEP to address some of the problems encountered with PEP 563.

The PEP would eventually get turned into PEP 649 ("Deferred Evaluation Of Annotations Using Descriptors"). In it, Hastings listed a number of problems with the PEP 563 feature. By requiring Python implementations to turn annotations (which, syntactically, can be any valid expression) into strings, PEP 563 introduces difficulties that spread beyond CPython:

• It requires Python implementations to stringize their annotations. This is surprising behavior—unprecedented for a language-level feature. Also, adding this feature to CPython was complicated, and this complicated code would need to be reimplemented independently by every other Python implementation.
• [...]
• It adds an ongoing maintenance burden to Python implementations. Every time the language adds a new feature available in expressions, the implementation's stringizing code must be updated in tandem in order to support decompiling it.

But there are problems even strictly within the CPython ecosystem that PEP 649 also describes. PEP 563 necessitates a code change everywhere that annotations are going to be used. The use of eval() is problematic because it is slow, but it is also unavailable in some contexts because it has been removed for space reasons. All annotations are evaluated at module-level scope and cannot refer to local or class variables. In addition, evaluating the annotations on a class requires a reference to the class's global symbols, "which PEP 563 suggests should be done by looking up that class by name in sys.modules". That too is surprising for a language-level feature like annotations.

Overall, Hasting's analysis, coupled with other problems noted with annotations and PEP 563, gives the appearance of features that had rough edges, with fixes that filed them off, only to be left with more rough (or perhaps even sharp) edges. Annotations were bolted onto the language, then type hints bolted onto annotations, with deferred evaluation added, perhaps somewhat hastily, to fix the forward-reference problems that were introduced by type hints. But now, after several releases as an opt-in feature, deferred evaluation is slated to become the only behavior supported, with no way to opt-out for those who never chose to opt-in. It is all something of a tangle that seems to need some unsnarling.

Lazy evaluation

Hastings's solution in PEP 649 is to defer the evaluation of the annotations, but to effectively replace them with a function, rather than a string as PEP 563 does. That function would evaluate and return the annotations as a dictionary, while storing the result. It would do so in the same scope as the annotations were made, neatly sidestepping all of the weird scoping and namespace corner (and not-so-corner) cases that arise with PEP 563.

In this new approach, the code to generate the annotations dict is written to its own function which computes and returns the annotations dict. Then, __annotations__ is a "data descriptor" which calls this annotation function once and retains the result. This delays the evaluation of annotations expressions until the annotations are examined, at which point all circular references have likely been resolved. And if the annotations are never examined, the function is never called and the annotations are never computed.

PEP 649 would add a __co_annotations__ attribute to objects that would hold a callable object. The first time __annotations__ is accessed, __co_annotations__() is called, its return value is assigned to __annotations__, and the value of __co_annotations__ is set to None. All of that is described in the PEP, including pseudocode. PEP 649 would be gated behind its own __future__ import (co_annotations), but the idea is that it would replace the behavior of PEP 563, which would eventually be deprecated and removed.

In general, the response was favorable toward the PEP back in January. Guido van Rossum, who originated type hints back when he was the benevolent dictator of the language, seemed favorably disposed toward the idea and suggested some further refinements. Others, such as PEP 563 author Łukasz Langa, also expressed support: "I like the clever lazy-evaluation of the __annotations__ using a pre-set code object."

Typing-only annotations?

For several months, though, that's where things stood. Despite a prodding or two, Hastings did not post a second version of PEP 649 until April 11. That was pretty late in the Python 3.10 schedule, which had its first beta (thus feature freeze) coming on May 3. Van Rossum had given up on PEP 649 in the interim. In fact, he seemed to be ready to restrict annotations to types:

Nevertheless I think that it's time to accept that annotations are for types -- the intention of PEP 3107 was to experiment with different syntax and semantics for types, and that experiment has resulted in the successful adoption of a specific syntax for types that is wildly successful.

But annotations have been part of the language for a long time, with other use cases beyond just type hints for static type-checkers; Hastings wondered why it made sense to remove them now:

I'm glad that type hints have found success, but I don't see why that implies "and therefore we should restrict the use of annotations solely for type hints". Annotations are a useful, general-purpose feature of Python, with legitimate uses besides type hints. Why would it make Python better to restrict their use now?

For Van Rossum, though, "typing is, to many folks, a Really Important Concept", but using the same syntax for type information and, generally undefined, "other stuff" is confusing. Hastings is not convinced that type hints are so overwhelmingly important that they should trump other uses:

I'm not sure I understand your point. Are you saying that we need to take away the general-purpose functionality of annotations, that's been in the language since 3.0, and restrict annotations to just type hints... because otherwise an annotation might not be used for a type hint, and then the programmer would have to figure out what it means? We need to take away the functionality from all other use cases in order to lend clarity to one use case?

Van Rossum said: "Yes, that's how I see it." But he was unhappy with how Hastings's effort to change things has played out:

[...] the stringification of annotations has been in the making a long time, with the community's and the SC's [steering council's] support. You came up with a last-minute attempt to change it, using the PEP process to propose to *revert* the decision already codified in PEP 563 and implemented in the master branch. But you've waited until the last minute (feature freeze is in three weeks) and IMO you're making things awkward for the SC (who can and will speak for themselves).

There are Python libraries that use the type annotations at run time, but some have run aground on supporting the deferred evaluations as described in PEP 563. Since it looked like PEP 563 would become the default (with no way to preserve the existing behavior) for 3.10, the developers behind some of those libraries got a bit panicky. That resulted in a strident post (and GitHub issue) from Samuel Colvin, who maintains the pydantic data-validation library. Colvin lists a bunch of other bugs that have come up while trying to support PEP 563 in pydantic, noting:

The reasons are complicated but basically typing.get_type_hints() doesn't work all the time and neither do the numerous hacks we've introduced to try and get fix it. Even if typing.get_type_hints() was faultless, it would still be massively slower than the current semantics or PEP 649 [...]

In short - pydantic doesn't work very well with postponed annotations, perhaps it never will.

While the tone of Colvin's post was deemed over-dramatic and perhaps somewhat divisive, it turns out that others have encountered some of the same kinds of problems. Paul Ganssle pointed to the difficulties supporting the feature in the attrs package (which provided much of the inspiration for the dataclasses feature added in Python 3.7) as an example of how things may be going awry. He suggested a path forward as well:

[...] I wouldn't be surprised if PEP 563 is quietly throwing a spanner in the works in several other places as well), my vote is to leave PEP 563 opt-in until at least 3.11 rather than try to rush through a discussion on and implementation of PEP 649.

There was more discussion, along the way, including Langa's look at "PEP 563 in light of PEP 649" and Hastings's ideas on finding a compromise position that was meant to try to find a way to please both "camps". There were also side discussions on duck typing, the Python static-typing ecosystem, and more. But mostly, it seemed that folks were just marking time awaiting a pronouncement from the steering council.

Postponement

That came on April 20. Given the timing, the nature of the problems, and the importance of not locking the language into behavior that might not be beneficial long-term, it probably was not much of a surprise that the council "kicked the can down the road" a bit. It decided to postpone making the PEP 563 behavior the default until Python 3.11 at the earliest. It deferred PEP 649 or any other alternatives as well.

There was an assumption that type annotations would only be consumed by static type-checkers, Thomas Wouters said on behalf of the council, but: "There are clearly existing real-world, run-time uses of type annotations that would be adversely affected by this change." The existing users of pydantic (which includes the popular FastAPI web framework) would be impacted by the change, but there are also likely to be uses of annotations at run time that have not yet come to light. The least disruptive option is to roll back to the Python 3.9 behavior, but:

We need to continue discussing the issue and potential solutions, since this merely postpones the problem until 3.11. (For the record, postponing the change further is not off the table, either, for example if the final decision is to treat evaluated annotations as a deprecated feature, with warnings on use.)

For what it’s worth, the SC is also considering what we can do to reduce the odds of something like this happening again, but that’s a separate consideration, and a multi-faceted one at that.

For an optional feature, support for static typing and type hints has poked its nose into other parts of the language over the past five years or so. It was fairly easy to argue that the general-purpose annotations feature could be used to add some static-typing support, but we may be getting to the point where providing better support for typing means deprecating any other uses of annotations, which is not something that seems particularly Pythonic. If typing is to remain optional, and proponents are adamant that it will, other longstanding features should not have to be sacrificed in order to make the optional use case work better.

Collectively taking a deep breath and stepping back to consider possible alternatives is obviously the right approach here. Perhaps some compromise can be found so that all existing users of annotations—especially fringe uses whose developers may be completely unaware that there is even a change on the horizon—can be accommodated. That kind of outcome would be best for everyone concerned, of course. Taking the pressure off for a year or so might just provide enough space to make that happen.

[I would like to thank Salvo Tomaselli for giving us a "heads up" about this topic.]

Comments (8 posted)

Fedora contemplates the driverless printing future

Back in a distant time — longer ago than he cares to admit — your editor managed a system-administration group. At that time, most of the day-to-day pain reliably came from two types of devices: modems and printers. Modems are more plentiful than ever now, but they have disappeared into interface controllers and (usually) manage to behave themselves. Printers, instead, are still entirely capable of creating problems and forcing a reconsideration of one's life choices. Behind the scenes, though, the situation has been getting better but, as a recent conversation within the Fedora project made clear, taking advantage of those improvements will require some changes and a bit of a leap of faith.

Traditionally, getting a printer working on Linux has involved, among other things, locating and installing the appropriate printer drivers and PostScript printer definition (PPD) files to allow the system to communicate with the printer using whatever special dialect it favors. Often that involves installing a separate package like hplip, often supplied by the printer vendor. Some vendors have traditionally supported Linux better than others, but none of their products seem to work as smoothly as one would like. While printer setup on Linux has definitely improved over the years, it still easy to dread having to make a new printer work.

The Internet Printing Protocol

In this context, it is noteworthy that the maintainers of the CUPS system, which handles printing on Linux, have an outstanding issue (since moved) calling for the removal of its support for printer drivers. The Fedora development list also discussed this idea in late May. There are changes coming in the printing world, and users are naturally curious about when and how those changes will affect them.

Those who have not been following developments in printing may wonder why it would make sense to remove printer drivers, since those are what make the printers actually work. The answer is the Internet Printing Protocol (IPP) and, specifically, a derivative called IPP Everywhere. The idea is simple enough: what if all printers implemented the same protocol so that any software could print to them without the need to install special drivers or, indeed, perform any configuration work at all? Like many things, IPP Everywhere seems to be motivated by the desire to interoperate with smartphones, but it is useful in a wider context as well.

In the IPP Everywhere future, printers simply work. They make themselves known to the network via multicast DNS, allowing other machines on the network to discover them. The protocol allows the printer to communicate its capabilities and accept settings, so features like duplex printing and paper-tray selection work. IPP Everywhere describes a small set of known formats for material to be printed, so there is no need for special drivers for format conversion. Users simply need to indicate their desire to print something, select between the available printers and printing options, install the new toner cartridge that the printer inevitably demands, then file the resulting output in the recycling bin. It's a printing paradise.

Or, at least, it will be a printing paradise one of these days. Given that IPP Everywhere has been around since 2010, one might think that we should have arrived already, but there are some remaining issues.

Does it work?

CUPS has long had support for IPP Everywhere, and distributions have generally enabled that support. But it still doesn't seem to be the default way to use printers on Linux. For example, a search for how to set up your editor's Brother laser printer on Linux results in a visit to Brother's driver page; after a somewhat fraught process the printer can be made to work well — but it's not a transparent or driverless experience.

That printer is supposed to support IPP Everywhere, though, and it can be seen in the output of avahi-browse. Android devices can successfully print to it. That means, with any remotely modern version of Linux, the printer should show up automatically when trying to print a file, without the need to set up a queue at all. It does indeed appear, but an attempt to use it results in the print dialog hanging before the user can even hit the "print" button. That experience may be driverless, and it certainly results in a reduction of wasted paper, but it's also less than fully satisfying.

What about adding a queue explicitly? Firing up the "add a printer" dialog on a freshly installed Fedora 34 system produces a blank window — not the experience a printer user is looking for either. Typing in the IP address for the printer (something most users naturally have at the tips of their fingers) yields the display seen to the right: five options, with no real indication of which one — if any — is correct. In fact, only the final option in the list works at all, and it takes several minutes to print a page, for reasons that are unclear to anybody not initiated into the deep magic of CUPS.

If, instead, one simply tells CUPS to use the printer as an IPP Everywhere device with a command like:

    # lpadmin -p NewPrinter -E -v ipp://10.0.0.16/ipp -m everywhere

The printer works well, immediately. It began, of course, by communicating its appetite for a new toner cartridge to the Fedora system, which duly passed the request on. So IPP Everywhere does work, at some level, on both ends of the connection, but some of the glue is seemingly not yet fully in place.

Most distributors have been working on support for driverless printing. The curious can find instructions online for Arch Linux, Debian (and Ubuntu), Fedora, Gentoo, Red Hat, and others. SUSE and openSUSE seem to be the biggest exception, unless your editor's search skills have failed.

Old printers

In theory, almost all printers made in the last ten years should support IPP Everywhere. It seems that most do, even though the list of certified printers is conspicuously missing a number of vendors. But not everybody has a printer that recent; printers can stay around for a long time. The removal of printer drivers from CUPS (and the distributions using it) means the removal of support for all of those devices. It is fair to say that doesn't sit well with the owners of such hardware.

The intended solution to this problem is "printer applications": programs that can speak to a certain class of printers while implementing the IPP Everywhere protocol to connect those printers to CUPS. Only a handful of those exist now; they include LPrint, which can drive a set of label printers. Work is underway to support others, including the "foo2zjs" devices that show up as IPP Everywhere printers, but which still need special drivers to work. Implementing more printer applications features prominently in the OpenPrinting Google Summer of Code plans this year.

This does not satisfy all users out there, many of whom got their printers working after a fair amount of struggle and do not relish the idea of having to go through that process again with a new system. Some worry that there will be CUPS-supported printers that never get printer-application support; that may well happen, but it's not clear how many users will be affected by the loss of support for some ancient devices. Others complain that this transition is a lot of churn for no real benefit; Solomon Peachy answered that concern this way:

For many of us, that universal printing system can't come soon enough, if it works as well as it is intended to. Should that happen, there is likely to be little lamentation over the loss of printer drivers.

Fake printers

There is one other concern that came up in the Fedora conversation, though. In a world where printers just show up on the net, what is to prevent an attacker from putting up fake printers in the hope of capturing documents with interesting contents? There does not seem to have been much effort put into addressing this scenario; as Zdenek Dohnal put it: "CUPS discovery is designed to run on secure, private LAN, so it is expected that you have a protection against somebody connecting to your WIFI." Peachy added that, if an attacker has a sufficient presence on the local network to advertise a false printer, they can probably do worse things than that.

IPP does have provisions for authentication and such, but employing them would take away much of the convenience that the protocol is meant to provide in the first place. So the general attitude toward security is likely to remain as Dohnal described it: the local network is expected to be secure.

In a related issue, some participants in the discussion expressed concerns that some IPP Everywhere devices work by routing print jobs through cloud-based services. Indeed, the Android print dialog, when printing to the above-mentioned Brother printer, warns that the document may pass through third-party servers — something that does not appear to actually happen in this case. The idea of a local printer that fails when the Internet is down lacks appeal, but somehow we seem to be heading toward that world anyway.

When?

The CUPS project has been working toward a driverless future for many years; the use of PPD files was first deprecated in the 1.4 release in 2009, even though there was no replacement for them at that time. The project finally deprecated printer drivers with the 2.3 release in 2019. "Deprecated" is not the same as "removed", though; this deprecation was intended to draw attention to the forthcoming change. As can be seen with PPD files (which are still supported), there can be a long time between deprecation and removal.

That said, the plan is apparently to remove printer drivers in the next major CUPS release, for which there does not currently appear to be a target date. The project has not made a release since 2.3.3 came out in April 2020, so things do not seem to be moving all that quickly there at the moment. There is probably some time for distributors and users to prepare for this change.

That is a good thing. Even if most printers are supported reasonably well by IPP Everywhere, there will be a lot of users of printer drivers out there, and a lot of users with older printers as well. There will need to be a lot of work and testing done if the first distributions that release with CUPS 2.4 are to not break vast numbers of printing setups. If that can be done, though, perhaps printers can finally join modems as devices that simply work without the need for a lot of messing around.

Comments (97 posted)

Rewriting the GNU Coreutils in Rust

As movement toward memory-safe languages, and Rust in particular, continues to grow, it is worth looking at one of the larger scale efforts to port C code that has existed for decades to Rust. The uutils project aims to rewrite all of the individual utilities included in the GNU Coreutils project in Rust. Originally created by Jordi Boggiano in 2013, the project aims to provide drop-in replacements for the Coreutils programs, adding the data-race protection and memory safety that Rust provides.

Many readers will be familiar with the Coreutils project. It includes the basic file, process, and text manipulation programs that are expected to exist on every GNU-based operating system. The Coreutils project was created to consolidate three sets of tools that were previously offered separately, Fileutils, Textutils, and Shellutils, along with some other miscellaneous utilities. Many of the programs that are included in the project, such as rm, du, ls, and cat, have been around for multiple decades and, though other implementations exist, these utilities are not available for platforms like Windows in their original form.

Collectively, the Coreutils programs are seen as low-hanging fruit where a working Rust-based version can be produced in a reasonable amount of time. The requirements for each utility are clear and many of the them are conceptually straightforward, although that's not to suggest that the work is easy. While a lot of progress has been made to get uutils into a usable state, it will take some time for it to reach the stability and maturity of Coreutils.

The use of Rust for this project will help to speed this process along since a huge swathe of possible memory errors and other undefined behavior is eliminated entirely. It also opens the door to the use of efficient, race-free multithreading which has the potential to speed up some of the programs under certain conditions. The uutils rewrite also provides an opportunity to not just reimplement Coreutils but to also enhance the functionality of some of the utilities to yield a better user experience, while maintaining compatibility with the GNU versions. For example, feature requests that have long been rejected in the Coreutils project, like adding a progress bar option for utilities like mv and cp, are currently being entertained in this Rust rewrite.

What has been done so far

On the project's GitHub page, a table can be found with the utilities divided into three columns: "Done", "Semi-Done", and "To-Do". At the time of this writing, only 23 of the 106 utilities being worked on are not yet in the "Done" column, with 16 of them marked as "Semi-Done" and seven under the "To-Do" column. The utilities under "To-Do" have either not been worked on at all or are currently undergoing their initial implementation (like with pr and chcon). Those in the "Semi-Done" column are missing options that have not yet been implemented, or their behavior is slightly different from their GNU counterparts in certain situations. For example:

• tail does not support the -F or --retry flags.
• more fails when input is piped to it as in "cat foo.txt | more".
• install is missing both the -b and --backup flags.
• Several utilities do not support non-UTF-8 arguments, although this is largely being mitigated by migrating from getopts to clap for command-line argument parsing.

It is important to keep in mind that just because a program is marked as "Done" doesn't mean that all of the tests are passing or that the utility is as performant or memory-efficient as the GNU version. For example, there are open issues to improve the performance of factor (roughly 5x slower) and sort (between 1.5x to 6x slower). In some other cases, the uutils versions are faster than their GNU equivalents. An example is the cp utility in which a measurable performance improvement has been reported, primarily due to the use of the sendfile() and copy_file_range() system calls, which are not being used in the GNU version despite several proposals to do so.

At the moment, only 142 of 624 tests in the Coreutils test suite are passing compared to around 546 tests passing with the GNU version. However, it should be noted that many of the errors are due to differences in the output of the commands.

A separate table also exists to show all of the platforms and architectures that the uutils project currently supports. The major operating systems (Linux, macOS, and Windows) are well accounted for across various architectures although quite a few utilities are currently not building on Windows. FreeBSD, NetBSD, and Android also compile most binaries except for a handful of utilities including chroot, uptime, uname, stat, who, and others. The rows for Redox OS, Solaris, WebAssembly, and Fuchsia are currently all blank, which reflects the lower priority assigned to those platforms at the moment.

The uutils project, currently at version 0.0.6, has already been packaged in the repositories for various Linux distributions and packaging systems. Notably, Sylvestre Ledru, a director at Mozilla and prolific contributor to the Debian and Ubuntu projects, has led the way in getting the project packaged for Debian as an alternative to the GNU Coreutils. Its current state is deemed good enough to get a system with GNOME up and running, install a thousand of the most popular Debian packages, and to build Firefox, the Linux kernel, and LLVM/Clang. Additionally, uutils is present in the repositories for Arch Linux (Community), Homebrew for macOS, and the Exherbo Linux distribution.

Licensing

An important aspect of the uutils project to be aware of is its licensing. All of the utilities in the project are licensed under the permissive MIT License, instead of the GPLv3 license of GNU Coreutils. This potentially makes it more attractive for use in places where software licensed with GPLv3 is not adopted due to its restrictions on Tivoization among other things. The decision to use the MIT License is not without its critics; some who commented in a GitHub issue about the choice would rather see a copyleft license applied to a project of this sort.

The main criticism echoes arguments over FOSS licensing in the past: a non-copyleft license is harmful to the freedoms of end users since it allows a person or organization to incorporate any part of the project into a device or in the distribution of other software without providing the source code so it is impossible to study, change, or improve it. There is also a concern that the license choice is being made to maximize Rust usage without regard for other effects; replacing GPL-licensed tools with alternatives under a more-permissive license is seen by some as a step backward.

Contributing to uutils

The best way to follow the development of the uutils project is through its GitHub repository and official Discord server. Details on how to get started contributing to the project can be found in a document that is included in the repository.

A lot of work remains to be done to get uutils into a production-ready state. The project has been positioned as a good way get into Rust development and there is a list of issues for newcomers as a place to begin. The current focus of the project appears to be full compatibility with the GNU Coreutils and improving the test coverage before tackling other problems. Things like removing unnecessary dependencies, improving performance, and decreasing memory use are better suited to being addressed after the compatibility issues have been ironed out.

Comments (274 posted)

Auditing io_uring

The io_uring subsystem, first introduced in 2019, has quickly become the leading way to perform high-bandwidth, asynchronous I/O. It has drawn the attention of many developers, including, more recently, those who are focused more on security than performance. Now some members of the security community are lamenting a perceived lack of thought about security support in io_uring, and are trying to remedy that shortcoming by adding audit and Linux security module support there. That process is proving difficult, and has raised the prospect of an unpleasant fallback solution.

The Linux audit mechanism allows the monitoring and logging of all significant activity on the system. If somebody wants to know, for example, who looked at a specific file, an audit-enabled system can provide answers. This capability is required to obtain any of a number of security certifications which, in turn, are crucial if one wants to deploy Linux in certain types of security-conscious settings. It is probably fair to say that a relatively small percentage of Linux systems have auditing turned on, but distributors, almost without exception, enable auditing in their kernels.

The audit mechanism relies, in turn, on a large array of hooks sprinkled throughout the kernel source. Whenever an event that may be of interest occurs, it is reported via the appropriate hook to the audit code. There, a set of rules loaded from user space controls which events are reported to user space.

When io_uring was being developed (which is still happening now, of course), the developers involved were deeply concerned about performance and functionality. Supporting security features like auditing was not at the top of their list, so they duly neglected to add the needed hooks — or to think about how auditing could be supported in a way consistent with the performance goals. Now that io_uring is showing up in more distributor kernels (and, in particular, the sorts of kernels where auditing is relatively likely to be enabled), security-oriented developers are starting to worry about it. Having io_uring serve as a way to circumvent the otherwise all-seeing audit eye does not seem like a good way to maintain those security certifications.

Adding security support

In late May, Paul Moore (a maintainer of the audit subsystem) posted a set of patches adding Linux security module (LSM) and audit capabilities to io_uring. The LSM side is relatively straightforward; the operations performed by io_uring are already covered by LSM hooks, so all that was needed was a pair of new hooks to pass judgment on io_uring-specific actions. Specifically, these hooks control the sharing (between processes) of credentials that are stored with the ring buffer that is used to communicate operation requests to the kernel; see this patch for details. This part of the patch set does not seem to be controversial.

The audit code is another story. The core io_uring code has been carefully optimized to dispatch requests and their results as quickly as possible. Use cases for io_uring can involve performing millions of I/O operations per second, so any added overhead will prove most unwelcome. Adding the audit hooks to cover operations submitted through the ring slowed down one of the most performance-critical parts of io_uring, leading Pavel Begunkov to react negatively: "So, it adds two if's with memory loads (i.e. current->audit_context) per request in one of the hottest functions here... No way, nack".

Begunkov suggested that perhaps the audit hooks could be patched in at run time when they are actually enabled, the way tracepoints and kprobes work. Moore responded that the audit subsystem doesn't support that sort of patching, and that doing so could raise problems of its own: "I fear it would run afoul of the various security certifications". So that does not appear to be the route to a possible solution.

Meanwhile, Jens Axboe, the io_uring maintainer, ran some tests. A simple random-read test slowed down by nearly 5% with the audit hooks installed, even in the absence of any actual audit rules. Various other benchmarks, even when run with an updated version of the patch set (which was not posted publicly), gave the same results. Kernel developers can work for months for a 5% performance gain; losing that amount to audit hooks is a bitter pill for them to swallow.

Axboe pointed out that read and write operations are not audited when they are initiated through the older asynchronous I/O system calls. "In the past two decades, I take it that hasn't been a concern?" He agreed that some operations (such as opening or removing files) should be audited, but said that auditing read and write operations was "just utter noise and not useful at all". Since those operations are the ones where performance matters the most, taking the audit hooks out of the fast path for them might be a possible solution.

Moore suggested an approach based on that idea; only a specific, carefully chosen set of operations would have the audit hooks applied. There is a handy switch statement in the io_uring dispatcher that makes it easy to instrument just the desired operations. He asked for feedback, but has not gotten much so far. The important question, as Begunkov pointed out, is which operations in particular need audit support. Adding an audit call when opening a file is unlikely to bother anybody; a call added to, say, a poll operation would be another story. Moore has posted an initial set of operations that he thinks merit auditing.

Threats and grumbles

With luck, that solution will prove acceptable to everybody. The alternative to adding audit support to io_uring is, according to Moore, not particularly pleasant:

If we can't find a solution here we would need to make io_uring and audit mutually exclusive and I don't think that is in the best interests of the users, and would surely create a headache for the distros.

"Headache" is not really the word for it. If the two features are made exclusive, then it will not be possible to configure a kernel containing both of them. So distributors would have to either ship two different kernels (something they will go far out of their way to avoid) or pick one of the two features to support. Hopefully it will not come to that.

Meanwhile, there has been some disgruntlement expressed by developers on both sides, but the security developers have made it especially clear that they would have liked to see audit designed into io_uring from the beginning. As Casey Schaufler put it:

It would have been real handy had the audit and LSM requirements been designed into io_uring. The performance implications could have been addressed up front. Instead it all has to be retrofit.

Richard Guy Briggs also complained that "multiple aspects of security appear to have been a complete afterthought to this feature, necessitating it to be bolted on after the fact". The implication in both cases is that, with adequate forethought, the difficulties being encountered now could have been avoided.

That is arguably a fair criticism. Kernel developers working on new features often leave security as something to be thought about later; that is especially true for relatively niche features like auditing, which is unlikely to be enabled on development systems. The kernel community can be a bit unfriendly toward its security developers, characterizing them as prioritizing security above anything else (and above performance in particular). Such an environment seems like a recipe for leaving security concerns by the wayside, to be fixed up later.

It is also fair to point out, though, that io_uring has been developed in public since early 2019. It has been heavily discussed on the mailing lists (and in LWN), but the security community did not see that as their cue to make suggestions on how features like auditing could be supported. It is a rare kernel developer who can summon the focus to implement a million-operation-per-second I/O subsystem while simultaneously making provisions for security hooks that won't kill performance. Perhaps the io_uring developers should have been considering security from the beginning, but they should also have had help from the beginning.

The kernel community has surprisingly few rules regarding the addition of new features like io_uring. In theory, new system calls should come with manual pages, but it's somewhat surprising when that actually happens. In a project with a more bureaucratic process, it would make sense to insist that new features do not go in until they have proper support for mechanisms like LSM and auditing. That might force earlier interactions with security developers and avoid this kind of problem.

That is not the world that we live in, though; there is nobody with a checklist making sure that all of the relevant boxes have been marked before a new subsystem can be merged. So the kernel community will have to continue to muddle along, supporting the needed features as best it can. This is not the last time that a security mechanism will have to be retrofitted into an existing kernel feature. It's arguably not the best approach, but it generally gets the job done in the end.

Comments (23 posted)

The runtime verification subsystem

The realtime project has been the source of many of the innovations that have found their way into the core kernel in the last fifteen years or so. There is more to it than that, though; the wider realtime community is also doing interesting work in a number of areas that go beyond ensuring deterministic response. One example is Daniel Bristot de Oliveira's runtime verification patch set, which can monitor the kernel to ensure that it is behaving the way one thinks it should.

Realtime development in the kernel community is a pragmatic effort to add determinism to a production system, but there is also an active academic community focused on realtime work. Academic developers often struggle to collaborate effectively with projects like the kernel, where concerns about performance, regressions, and maintainability have been the downfall of many a bright idea. As a result, there is a lot of good academic work that takes a long time to make it into a production system, if it ever does.

Imagine, for a moment, a project to create a realtime system that absolutely cannot be allowed to fail; examples might include a controller for a nuclear reactor, a jetliner's flight-control system, or the image processor in a television set showing that important football game. In such a setting, it is nice to know that the system will always respond to events within the budgeted time. Simply observing that it seems to do so tends to be considered inadequate for these systems.

One way to get to a higher level of assurance is to create a formal model of the system, prove mathematically that the model produces the desired results, then run that model with every scenario that can be imagined. This approach can work, but it has its difficulties: ensuring that the model properly matches the real system is a challenge in its own right and, even if the model is perfect, it is almost certain to be far too slow for any sort of exhaustive testing. The complexity of real-world systems makes this approach impractical, at best.

Runtime verification

The runtime verification patches take a different approach. Developers can create a high-level model describing the states that the system can be in and the transitions that occur in response to events. The verification code will then watch the kernel to ensure that the system does, indeed, behave as expected. If a discrepancy is found in a running system, then there is either an error in the model or a bug in the system; either way, fixing it will improve confidence that the system is behaving correctly.

The use of this mechanism is described in this documentation patch helpfully included with the code; the following example is shamelessly stolen from there. The first step is to express one's model of an aspect of the system's behavior; the example given is whether a CPU can be preempted or not — a question that realtime researchers tend to be interested in. This model is described in the DOT language, making it easy to use Graphviz to view the result:

In the preemptive state, a CPU can be preempted by a higher-priority task. The preempt_disable event will put the CPU into the non_preemptive state where that can no longer happen; preempt_enable returns the CPU to the preemptive state. A sched_waking event while in the non_preemptive state will cause the CPU to remain in that state.

Pretty graph plots can dress up an academic paper (or an LWN article), but they are of limited utility when it comes to verifying whether the kernel actually behaves as described in that model. That might change, though, if this model could be used to generate code that can help with this verification. Part of the patch set is the dot2k tool, which will read the DOT description and output a set of template code that may be used for actual verification. There is, however, some work that must be done to connect that result to the kernel.

Connecting the model to the kernel

As a starting point, the template code generated by dot2k contains definitions of the states and events described in the model:

enum states { preemptive = 0, non_preemptive, state_max };

enum events { preempt_disable = 0, preempt_enable, sched_waking, event_max };

From these, a state machine is built to match the model. Also included are stub functions that will be called in the kernel when one of the defined events occurs; for example, the preempt_disable event is given this stub:

    void handle_preempt_disable(void *data, /* XXX: fill header */)
    {
	da_handle_event_wip(preempt_disable);
    }

The developer's job is to complete the prototype of the function to match how it will be called. That, in turn, depends on how it will be hooked into a running kernel. There is a fair amount of flexibility here; just about anything that will cause the kernel to call the function in response to the relevant event is fair game. The most common case, though, seems likely to be tracepoints, which already exist to mark the occurrence of events of interest. The kernel conveniently provides a tracepoint that fires when preemption is disabled; its name is, boringly, "preemption_disable", and it provides two parameters beyond the standard tracepoint data called ip and parent_ip. If our handle_preempt_disable() function is to be hooked to that tracepoint, its prototype must thus be:

    void handle_preempt_disable(void *data, unsigned long ip, unsigned long parent_ip);

Any other stubs must be fixed up in the same way. Then, the developer must arrange for the connection between the handler functions and their respective tracepoints; the template provided by dot2k makes that easy:

    #define NR_TP   3
    struct tracepoint_hook_helper tracepoints_to_hook[NR_TP] = {
        {
	    .probe = handle_preempt_disable,
	    .name = /* XXX: tracepoint name here */,
	    .registered = 0
    	},
    	/* ... */
    };

By filling in the names of the relevant tracepoints, the developer can make the connection to the handler functions. The template also provides functions that are used to start and stop the model, for cases where any sort of extra initialization or cleanup is required. Those functions need no modification for this simple model. Finally, dot2k generates a pair of tracepoints that can be used to watch for events and errors from the model.

Running the model

The resulting code is then built into the kernel, most likely as a loadable module. In a kernel with runtime verification enabled, there will be a new directory (called rv) in /sys/kernel/tracing with a set of control files; these can be used to turn specific models on or off. It is also possible to configure "reactors", which perform some action when the system's behavior diverges from what the model says should happen. The default is to do nothing, though the "error" tracepoint will fire in that case. Alternatives include logging a message with printk(), and panicking the system for those cases when somebody is especially unhappy that the system misbehaved.

Realtime researchers can use this mechanism to check that their models of the kernel's behavior match the reality. But it is not hard to imagine that runtime verification could have much broader applicability than that. It could be used to monitor the security of the system, ensuring that, for example, no process enters a privileged state in unexpected places. Regression testing is another obvious application; a suitably well-developed model of various kernel subsystems might be able to catch subtle bugs before they become visible at the user level. The use of DOT to define the models makes it easy to use them as documentation describing the expected behavior of the kernel as well.

The first step, though, would be to get this subsystem into the kernel. So far, there have not been many comments posted in response to this work, so it is unlikely to have seen a lot of review. As an add-on that should not bother anybody who is not using it, runtime verification should have a relatively low bar to get over, so it is not entirely fanciful to imagine that this work could be merged. Then there could be some interesting applications of it that come out of the woodwork.

Comments (8 posted)