LWN.net Weekly Edition for December 4, 2025

Welcome to the LWN.net Weekly Edition for December 4, 2025

This edition contains the following feature content:

• APT Rust requirement raises questions: how will Debian accommodate Rust in one of its core tools?
• Unpacking for Python comprehensions: a look at PEP 798 ("Unpacking in Comprehensions"), which was accepted in November.
• Zig's new plan for asynchronous programs: allowing Zig code to execute correctly using synchronous or asynchronous I/O.
• BPF and io_uring, two different ways: competing patch sets try to add BPF to io_uring.
• Checked-size array parameters in C: improving the safety of kernel code with an obscure C feature.
• Some 6.18 development statistics: a look at who contributed code and more to the 6.18 kernel.
• Just: a command runner: an alternative to make for running user-defined tasks.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

APT Rust requirement raises questions

It is rarely newsworthy when a project or package picks up a new dependency. However, changes in a core tool like Debian's Advanced Package Tool (APT) can have far-reaching effects. For example, Julian Andres Klode's declaration that APT would require Rust in May 2026 means that a few of Debian's unofficial ports must either acquire a working Rust toolchain or depend on an old version of APT. This has raised several questions within the project, particularly about the ability of a single maintainer to make changes that have widespread impact.

On October 31, Klode sent an announcement to the debian-devel mailing list that he intended to introduce Rust dependencies and code into APT as soon as May 2026:

This extends at first to the Rust compiler and standard library, and the Sequoia ecosystem.

In particular, our code to parse .deb, .ar, .tar, and the HTTP signature verification code would strongly benefit from memory safe languages and a stronger approach to unit testing.

If you maintain a port without a working Rust toolchain, please ensure it has one within the next 6 months, or sunset the port.

Klode added this was necessary so that the project as a whole could move forward, rely on modern technologies, "and not be held back by trying to shoehorn modern software on retro computing devices". Some Debian developers have welcomed the news. Paul Tagliamonte acknowledged that it would impact unofficial Debian ports but called the push toward Rust "welcome news".

However, John Paul Adrian Glaubitz complained that Klode's wording was unpleasant and that the approach was confrontational. In another message, he explained that he was not against adoption of Rust; he had worked on enabling Rust on many of the Debian architectures and helped to fix architecture-specific bugs in the Rust toolchain as well as LLVM upstream. However, the message strongly suggested there was no room for a change in plan: Klode had ended his message with "thank you for understanding", which invited no further discussion. Glaubitz was one of a few Debian developers who expressed discomfort with Klode's communication style in the message.

Klode noted, briefly, that Rust was already a hard requirement for all Debian release architectures and ports, except for Alpha (alpha), Motorola 680x0 (m68k), PA-RISC (hppa), and SuperH (sh4), because of APT's use of the Sequoia-PGP project's sqv tool to verify OpenPGP signatures. APT falls back to using the GNU Privacy Guard signature-verification tool, gpgv, on ports that do not have a Rust compiler. By depending directly on Rust, though, APT itself would not be available on ports without a Rust compiler. LWN recently covered the state of Linux architecture support, and the status of Rust support for each one.

None of the ports listed by Klode are among those officially supported by Debian today, or targeted for support in Debian 14 ("forky"). The sh4 port has never been officially supported, and none of the other ports have been supported since Debian 6.0. The actual impact on the ports lacking Rust is also less dramatic than it sounded at first. Glaubitz assured Antoni Boucher that "the ultimatum that Julian set doesn't really exist", but phrasing it that way "gets more attention in the news". Boucher is the maintainer of rust_codegen_gcc, a GCC ahead-of-time code generator for Rust. Nothing, Glaubitz said, stops ports from using a non-Rust version of APT until Boucher and others manage to bootstrap Rust for those ports.

Security theater?

David Kalnischkies, who is also a major contributor to APT, suggested that if the goal is to reduce bugs, it would be better to remove the code that is used to parse the .deb, .ar, and .tar formats that Klode mentioned from APT entirely. It is only needed for two tools, apt-ftparchive and apt-extracttemplates, he said, and the only "serious usage" of apt-ftparchive was by Klode's employer, Canonical, for its Launchpad software-collaboration platform. If those were taken out of the main APT code base, then it would not matter whether they were written in Rust, Python, or another language, since the tools are not directly necessary for any given port.

Kalnischkies also questioned the claim that Rust was necessary to achieve the stronger approach to unit testing that Klode mentioned:

You can certainly do unit tests in C++, we do. The main problem is that someone has to write those tests. Like docs.

Your new solver e.g. has none (apart from our preexisting integration tests). You don't seriously claim that is because of C++ ? If you don't like GoogleTest, which is what we currently have, I could suggest doctest (as I did in previous installments). Plenty other frameworks exist with similar or different styles.

Klode has not responded to those comments yet, which is a bit unfortunate given the fact that introducing hard dependencies on Rust has an impact beyond his own work on APT. It may well be that he has good answers to the questions, but it can also give the impression that Klode is simply embracing a trend toward Rust. He is involved in the Ubuntu work to migrate from GNU Coreutils to the Rust-based uutils. The reasons given for that work, again, are around modernization and better security—but security is not automatically guaranteed simply by switching to Rust, and there are a number of other considerations.

For example, Adrian Bunk pointed out that there are a number of Debian teams, as well as tooling, that will be impacted by writing some of APT in Rust. The release notes for Debian 13 ("trixie") mention that Debian's infrastructure "currently has problems with rebuilding packages of types that systematically use static linking", such as those with code written in Go and Rust. Thus, "these packages will be covered by limited security support until the infrastructure is improved to deal with them maintainably". Limited security support means that updates to Rust libraries are likely to only be released when Debian publishes a point release, which happens about every two months. The security team has specifically stated that sqv is fully supported, but there are still outstanding problems.

Due to the static-linking issue, any time one of sqv's dependencies, currently more than 40 Rust crates, have to be rebuilt due to a security issue, sqv (at least potentially) also needs to be rebuilt. There are also difficulties in tracking CVEs for all of its dependencies, and understanding when a security vulnerability in a Rust crate may require updating a Rust program that depends on it.

Fabian Grünbichler, a maintainer of Debian's Rust toolchain, listed several outstanding problems Debian has with dealing with Rust packages. One of the largest is the need for a consistent Debian policy for declaring statically linked libraries. In 2022, Guillem Jover added a control field for Debian packages called Static-Built-Using (SBU), which would list the source packages used to build a binary package. This would indicate when a binary package needs to be rebuilt due to an update in another source package. For example, sqv depends on more than 40 Rust crates that are packaged for Debian. Without declaring the SBUs, it may not be clear if sqv needs to be updated when one of its dependencies is updated. Debian has been working on a policy requirement for SBU since April 2024, but it is not yet finished or adopted.

The discussion sparked by Grünbichler makes clear that most of Debian's Rust-related problems are in the process of being solved. However, there's no evidence that Klode explored the problems before declaring that APT would depend on Rust, or even asked "is this a reasonable time frame to introduce this dependency?"

Where tradition meets tomorrow

Debian's tagline, or at least one of its taglines, is "the universal operating system", meaning that the project aims to run on a wide variety of hardware (old and new) and be usable on the desktop, server, IoT devices, and more. The "Why Debian" page lists a number of reasons users and developers should choose the distribution: multiple hardware architectures, long-term support, and its democratic governance structure are just a few of the arguments it puts forward in favor of Debian. It also notes that "Debian cannot be controlled by a single company". A single developer employed by a company to work on Debian tools pushing a change that seems beneficial to that company, without discussion or debate, that impacts multiple hardware architectures and that requires other volunteers to do unplanned work or meet an artificial deadline seems to go against many of the project's stated values.

Debian, of course, does have checks and balances that could be employed if other Debian developers feel it necessary. Someone could, for example, appeal to Debian's Technical Committee, or sponsor a general resolution to override a developer if they cannot be persuaded by discussion alone. That happened recently when the committee required systemd maintainers to provide the /var/lock directory "until a satisfactory migration of impacted software has occurred and Policy updated accordingly".

However, it also seems fair to point out that Debian can move slowly, even glacially, at times. APT added support for the DEB822 format for its source information lists in 2015. Despite APT supporting that format for years, Klode faced resistance in 2021, when he pushed for Debian to move to the new format ahead of the Debian 12 ("bookworm") release in 2021, but was unsuccessful. It is now the default for trixie with the move to APT 3.0, though APT will continue to support the old format for years to come.

The fact is, regardless of what Klode does with APT, more and more free software is being written (or rewritten) in Rust. Making it easier to support that software when it is packaged for Debian is to everyone's benefit. Perhaps the project needs some developers who will be aggressive about pushing the project to move more quickly in improving its support for Rust. However, what is really needed is more developers lending a hand to do the work that is needed to support Rust in Debian and elsewhere, such as gccrs. It does not seem in keeping with Debian's community focus for a single developer to simply declare dependencies that other volunteers will have to scramble to support.

Comments (343 posted)

Unpacking for Python comprehensions

Unpacking Python iterables of various sorts, such as dictionaries or lists, is useful in a number of contexts, including for function arguments, but there has long been a call for extending that capability to comprehensions. PEP 798 ("Unpacking in Comprehensions") was first proposed in June 2025 to fill that gap. In early November, the steering council accepted the PEP, which means that the feature will be coming to Python 3.15 in October 2026. It may be something of a niche feature, but it is an inconsistency that has been apparent for a while—to the point that some Python programmers assume that it is already present in the language.

Unpacking

One of the most common use cases for unpacking is to pass a list or dictionary to a function as a series of its elements rather than as a single object. The "*" unpacking operator (and its companion, "**", for dictionaries) can be used to expand an iterable in that fashion; a simple example might look like the following:

    def foo(a, b):
        a*b

    l = [ 2, 4 ]  # list
    t = ( 3, 5 )  # tuple
    d = { 'a' : 2, 'b' : 9 } # dict
    foo(*l)   # foo(2, 4) == 8
    foo(*t)   # foo(3, 5) == 15
    foo(**d)  # foo(a=2, b=9) == 18

In each case, the unpacking operator extracts the elements of the iterable to turn them into individual arguments to the function—keyword arguments for the dictionary.

Python comprehensions provide a mechanism to build a list or other iterable using a compact syntax, rather than a full loop. A classic example of that appears in the Python documentation linked just above:

    squares = []
    for x in range(10):
        squares.append(x**2)

    # can be replaced with

    squares = [ x**2 for x in range(10) ]

For both of those, the result is a list with the squares of the numbers zero through nine.

The unpacking operator can already be used to create iterables, such as:

    a = [ 1, 2 ]
    b = [ 3, 4 ]
    c = [ *a, *b ]  # [ 1, 2, 3, 4 ]

    # dictionaries can be merged in similar fashion

    newdict = { **d1, **d2, **d3 }

In the latter example, the order of the dictionaries matters, keys that are duplicated will take their value from the last dictionary where they were set (i.e. d3[key] takes precedence over the value for key in d1 or d2).

But what if there is a list of lists with a length that is not known except at run time? Currently, there is no easy way to use a list comprehension to build the flattened list of all of the entries of each list; it can be done using a comprehension with two loops, but that is error prone. There are other possibilities too, as described in the "Motivation" section of the PEP, but all of them suffer from semi-obscurity or complexity. Instead, Python will be allowing unpacking operators in comprehensions:

    # from the PEP
    
    [*it for it in its]  # list with the concatenation of iterables in 'its'
    {*it for it in its}  # set with the union of iterables in 'its'
    {**d for d in dicts} # dict with the combination of dicts in 'dicts'
    (*it for it in its)  # generator of the concatenation of iterables in 'its'

    # a usage example

    a = [ 1, 2 ]
    b = [ 3, 4, 5 ]
    c = [ 6 ]
    its = [ a, b, c ]
    [ *it for it in its ]   #  [ 1, 2, 3, 4, 5, 6 ]

    # current double-loop version

    [ x for it in its for x in it ]  #  [ 1, 2, 3, 4, 5, 6 ]

Discussion

The PEP authors, Adam Hartz and Erik Demaine, actually proposed the idea back in 2021 on the python-ideas mailing list. As noted in that message, though, the idea also came up in 2016 and perhaps even before that. In 2021, the proposal was generally well-received, and reached the pre-PEP stage, but was unable to attract a core developer as a sponsor. In late June, Hartz posted a lengthy pre-PEP message to the ideas category of the Python discussion forum, "hoping to find a sponsor for moving forward with the PEP process if there's still enthusiasm behind this idea".

Hartz noted that the idea had been raised in October 2023, as well, so it is a feature that is frequently brought up—generally to nodding approval. That 2023 message was posted by Alex Prengère, who was quick to reply to the pre-PEP saying that he had been working on unpacking in comprehensions as well. He, along with others, wondered about support for unpacking in asynchronous comprehensions (as described in PEP 530); it was not mentioned in the pre-PEP, but the implementation would allow them, he said. Hartz said that the intent was to support them and that he would update the text to reflect that.

There was also some discussion regarding the syntax for making function calls using a generator comprehension (e.g. f(x for x in it)); there is some ambiguity in the meaning, as Ben Hsing noted. PEP 448 ("Additional Unpacking Generalizations") added the ability to use the unpacking operators in more contexts, including function call arguments, but it explicitly did not extend that to generator comprehensions because it was not clear which meaning should be chosen. As Hsing put it:

That is, which one of these is intended?

    f(*x for x in it) == f((*x for x in it))

or:

    f(*x for x in it) == f(*(x for x in it))

In the thread, Hartz and others argued that, since the language already allows a generator comprehension (without any unpacking operators) as an argument without requiring an extra set of parentheses (e.g. f(x for x in it), the same should be true for those with the unpacking operator. In the reply linked above, Hartz noted that the error message for the syntax error points in that direction as well:

A little bit of support in this direction, perhaps, comes from the way that the syntax error for f(*x for x in it) is reported in 3.13, which suggests that this is interpreted as f(<a single malformed generator expression>) rather than as f(*<something>):

    >>> f(*x for x in its)
      File "<python-input-0>", line 1
        f(*x for x in its)
          ^^
    SyntaxError: iterable unpacking cannot be used in comprehension

The draft PEP continued to be discussed; it was updated a few days after it was first posted, on June 25, and then again on July 3. The latter posting caught the eye of core developer Jelle Zijlstra who said that it "is a nice little feature that I've missed several times in the past" and that he was willing to sponsor it. That started its path into the CPython mainline.

The now-numbered PEP 798 was posted for discussion in the PEPs category of the forum on July 19. Along the way, the PEP had picked up some extra pieces, including a section with examples of where standard library code could be simplified using the feature and an appendix on support in other languages. Most of the comments at that point were about other features that might also be considered, though Hartz and Zijlstra tried to keep things focused on the PEP itself.

One outstanding issue was the treatment of synchronous generator expressions versus the asynchronous variety. The PEP, which will be changing as we will see, currently makes a distinction between the two because "yield from" is not permitted in asynchronous generators. Another appendix goes into more detail; the difference comes down to whether the generator-protocol methods, such as send(), can be used. There are two ways that the semantics of an unpacking generator expression could be defined:

    g = (*x for x in it)

    # could be:

    def gen():
        for x in it:
            yield from x
    g = gen()

    # or:

    def gen():
        for x in it:
            for i in x:
                yield i
    g = gen()

Either of those works for synchronous generators, but:

    g = (*x async for x in ait())

    # must be:

    async def gen()
        async for x in ait():
            for i in x:
                yield i
    g = gen()

So the question is whether it makes sense to define the synchronous semantics differently so that those comprehensions could potentially use the generator-protocol methods. Hartz ran a poll in the thread, with several possibilities for the semantics, but no real consensus was reached—perhaps unsurprising given the esoteric nature of the question and that thread participants had likewise been unable to converge on the semantics.

In mid-September, after more than a month of quiet in the thread, Hartz submitted the PEP to the steering council for consideration. The council started looking at it a month later, with council member Pablo Galindo Salgado noting that the group was uncomfortable with positioning the new syntax "as offering 'much clearer' code compared to existing alternatives (such as itertools.chain.from_iterable, explicit loops, or nested comprehensions)" because readability is in the eye of the beholder. Instead, the council suggested that "the stronger, more objective argument is syntactic consistency as extending Python's existing unpacking patterns naturally into comprehensions". Hartz agreed and adjusted the PEP accordingly.

In the thread, "Nice Zombies" highlighted part of the "Rationale" section of the PEP, which nicely illustrates the argument for syntactic consistency:

This proposal was motivated in part by a written exam in a Python programming class, where several students used the proposed notation (specifically the set version) in their solutions, assuming that it already existed in Python. This suggests that the notation represents a logical, consistent extension to Python's existing syntax. By contrast, the existing double-loop version [x for it in its for x in it] is one that students often get wrong, the natural impulse for many students being to reverse the order of the for clauses.

One of the examples given in the PEP shows how an explicit loop to create a set could be changed in the shutil module of the standard library:

    # current:
    ignored_names = []
    for pattern in patterns:
        ignored_names.extend(fnmatch.filter(names, pattern))
    return set(ignored_names)

    # proposed:
    return {*fnmatch.filter(names, pattern) for pattern in patterns}

Instead of extending the list from the iterable returned by fnmatch.filter(), then converting it to a set, the new syntax allows creating the set directly. The existing code could have taken advantage of set.update() to avoid using the list, but the new syntax is in keeping with the ideas behind comprehensions—and was apparently intuitively obvious, but wrong, to Python students.

In its announcement of the PEP's acceptance, the SC also decided the question about generator comprehensions: "we require that both synchronous and asynchronous generator expressions use explicit loops rather than yield from for unpacking operations". That removes some advanced use cases "that are rarely relevant when writing comprehensions" but simplifies the mental model for the new feature. "We don't believe that developers writing comprehensions should have to think about the differences between sync and async generator semantics or about generator delegation protocols."

While it is certainly useful, the feature is not revolutionary in any sense, it simply fills a fairly longstanding hole that has been noticed and discussed several times over the years. Python is a mature language at this point, so revolutions are likely to be few and far between—if not absent entirely. The whole tale shows, however, that, with some persistence, a well-written PEP, and a well-shepherded discussion (by Hartz joined by Zijlstra, Demaine was absent this time around), changes can be made. Future Python students can rejoice starting next October.

Comments (15 posted)

Zig's new plan for asynchronous programs

The designers of the Zig programming language have been working to find a suitable design for asynchronous code for some time. Zig is a carefully minimalist language, and its initial design for asynchronous I/O did not fit well with its other features. Now, the project has announced (in a Zig SHOWTIME video) a new approach to asynchronous I/O that promises to solve the function coloring problem, and allows writing code that will execute correctly using either synchronous or asynchronous I/O.

In many languages (including Python, JavaScript, and Rust), asynchronous code uses special syntax. This can make it difficult to reuse code between synchronous and asynchronous parts of a program, introducing a number of headaches for library authors. Languages that don't make a syntactical distinction (such as Haskell) essentially solve the problem by making everything asynchronous, which typically requires the language's runtime to bake in ideas about how programs are allowed to execute.

Neither of those options was deemed suitable for Zig. Its designers wanted to find an approach that did not add too much complexity to the language, that still permitted fine control over asynchronous operations, and that still made it relatively painless to actually write high-performance event-driven I/O. The new approach solves this by hiding asynchronous operations behind a new generic interface, Io.

Any function that needs to perform an I/O operation will need to have access to an instance of the interface. Typically, that is provided by passing the instance to the function as a parameter, similar to Zig's Allocator interface for memory allocation. The standard library will include two built-in implementations of the interface: Io.Threaded and Io.Evented. The former uses synchronous operations except where explicitly asked to run things in parallel (with a special function; see below), in which case it uses threads. The latter (which is still a work-in-progress) uses an event loop and asynchronous I/O. Nothing in the design prevents a Zig programmer from implementing their own version, however, so Zig's users retain their fine control over how their programs execute.

Loris Cro, one of Zig's community organizers, wrote an explanation of the new behavior to justify the approach. Synchronous code is not much changed, other than using the standard library functions that have moved under Io, he explained. Functions like the example below, which don't involve explicit asynchronicity, will continue to work. This example creates a file, sets the file to close at the end of the function, and then writes a buffer of data to the file. It uses Zig's try keyword to handle errors, and defer to ensure the file is closed. The return type, !void, indicates that it could return an error, but doesn't return any data:

    const std = @import("std");
    const Io = std.Io;

    fn saveFile(io: Io, data: []const u8, name: []const u8) !void {
        const file = try Io.Dir.cwd().createFile(io, name, .{});
        defer file.close(io);
        try file.writeAll(io, data);
    }

If this function is given an instance of Io.Threaded, it will create the file, write data to it, and then close it using ordinary system calls. If it is given an instance of Io.Evented, it will instead use io_uring, kqueue, or some other asynchronous backend suitable to the target operating system. In doing so, it might pause the current execution and go work on a different asynchronous function. Either way, the operation is guaranteed to be complete by the time writeAll() returns. A library author writing a function that involves I/O doesn't need to care about which of these things the ultimate user of the library chooses to do.

On the other hand, suppose that a program wanted to save two files. These operations could profitably be done in parallel. If a library author wanted to enable that, they could use the Io interface's async() function to express that it does not matter which order the two files are saved in:

    fn saveData(io: Io, data: []const u8) !void {
        // Calls saveFile(io, data, "saveA.txt")
        var a_future = io.async(saveFile, .{io, data, "saveA.txt"});
        var b_future = io.async(saveFile, .{io, data, "saveB.txt"});

        const a_result = a_future.await(io);
        const b_result = b_future.await(io);

        try a_result;
        try b_result;

        const out: Io.File = .stdout();
        try out.writeAll(io, "save complete");
    }

When using an Io.Threaded instance, the async() function doesn't actually isn't actually required to do anything asynchronously [although the actual implementation may dispatch the function to a separate thread, depending on how it was configured] — it can just run the provided function right away. So, with that version of the interface, the function first saves file A and then file B. With an Io.Evented instance, the operations are actually asynchronous, and the program can save both files at once.

The real advantage of this approach is that it turns asynchronous code into a performance optimization. The first version of a program or library can write normal straight-line code. Later, if asynchronicity proves to be useful for performance, the author can come back and write it using asynchronous operations. If the ultimate user of the function has not enabled asynchronous execution, nothing changes. If they have, though, the function becomes faster transparently — nothing about the function signature or how it interacts with the rest of the code base changes.

One problem, however, is with programs where two parts are actually required to execute simultaneously for correctness. For example, suppose that a program wants to listen for connections on a port and simultaneously respond to user input. In that scenario, it wouldn't be correct to wait for a connection and only then ask for user input. For that use case, the Io interface provides a separate function, asyncConcurrent()concurrent() [this function was renamed during development; concurrent() is the most recent name] that explicitly asks for the provided function to be run in parallel. Io.Threaded uses a thread in a thread pool to accomplish this. Io.Evented treats it exactly the same as a normal call to async().

    const socket = try openServerSocket(io);
    var server = try io.concurrent(startAccepting, .{io, socket});
    defer server.cancel(io) catch {};

    try handleUserInput(io);

If the programmer uses async() where they should have used concurrent(), that is a bug. Zig's new model does not (and cannot) prevent programmers from writing incorrect code, so there are still some subtleties to keep in mind when adapting existing Zig code to use the new interface.

The style of code that results from this design is a bit more verbose than languages that give asynchronous functions special syntax, but Andrew Kelley, creator of the language, said that "it reads like standard, idiomatic Zig code." In particular, he noted that this approach lets the programmer use all of Zig's typical control-flow primitives, such as try and defer; it doesn't introduce any new language features specific to asynchronous code.

To demonstrate this, Kelley gave an example of using the new interface to implement asynchronous DNS resolution. The standard getaddrinfo() function for querying DNS information falls short because, although it makes requests to multiple servers (for IPv4 and IPv6) in parallel, it waits for all of the queries to complete before returning an answer. Kelley's example Zig code returns the first successful answer, canceling the other inflight requests.

Asynchronous I/O in Zig is far from done, however. Io.Evented is still experimental, and doesn't have implementations for all supported operating systems yet. A third kind of Io, one that is compatible with WebAssembly, is planned (although, as that issue details, implementing it depends on some other new language features). The original pull request for Io lists 24 planned follow-up items, most of which still need work.

Still, the overall design of asynchronous code in Zig appears to be set. Zig has not yet had its 1.0 release, because the community is still experimenting with the correct way to implement many features. Asynchronous I/O was one of the larger remaining priorities (along with native code generation, which was also enabled by default for debug builds on some architectures this year). Zig seems to be steadily working its way toward a finished design — which should decrease the number of times Zig programmers are asked to rewrite their I/O because the interface has changed again.

Comments (20 posted)

BPF and io_uring, two different ways

BPF allows programs uploaded from user space to be run, safely, within the kernel. The io_uring subsystem, too, can be thought of as a way of loading programs in the kernel, though the programs in question are mostly a sequence of I/O-related system calls. It has sometimes seemed inevitable that io_uring would, like many other parts of the kernel, gain BPF capabilities as a way of providing more flexibility to user space. That has not yet happened, but there are currently two patch sets under consideration that take different approaches to the problem.

An io_uring "program" is built by placing a series of entries in a submission queue managed in a ring buffer shared between the kernel and user space. Each submission-queue entry (SQE) describes a system call to be performed, and may make use of special buffers and file descriptors maintained within io_uring itself. Each SQE is normally executed asynchronously, but it is possible to link a series of SQEs so that each is only executed after the successful completion of the previous one. The result of each operation is stored in a completion-queue entry (CQE) in a second shared ring. Using io_uring, an application can keep many streams of I/O going concurrently with a minimum of system calls.

The io_uring linkage mechanism enables simple sequences of operations, such as creating a file, writing a buffer to that file, and closing the file. It does not offer much flexibility; one operation cannot pass information to the next or change how subsequent operations may execute. So it is not surprising that adding BPF support is seen as a way of filling that gap. So far, though, no attempts at adding that support have been seriously considered for merging into the mainline.

BPF operations

In early November, Ming Lei posted a patch set adding BPF support to io_uring in the form of a new operation, IORING_OP_BPF, that can be placed in the submission queue. The linkage mechanism can be used, for example, to cause a BPF program to be run between two other io_uring operations. The programs themselves can be set up as, essentially, new io_uring operations.

Specifically, the patch creates a new struct ops program type for defining BPF operations. A user-space program will fill in and register a uring_bpf_ops structure:

    struct uring_bpf_ops {
	unsigned short		id;
	uring_io_prep_t		prep_fn;
	uring_io_issue_t	issue_fn;
	uring_io_fail_t		fail_fn;
	uring_io_cleanup_t	cleanup_fn;
    };

The id is an operation ID; only the bottom eight bits are used, meaning that a program can establish up to 256 separate BPF-based operations. The rest of the fields are BPF programs that will implement the functions required by io_uring to set up, execute, and clean up after I/O operations. There are a couple of new kfuncs provided for those programs to obtain the request data from an SQE and to store a result of the operation in the proper CQE.

Once the operation has been set up, io_uring SQEs can make use of it with an IORING_OP_BPF operation specifying the appropriate ID. Two buffers can be passed to a BPF operation in each request; a new kfunc has been added to allow BPF programs to bulk-copy data between buffers. One of the use cases targeted by this work is to make it easy to copy data between user space and in-kernel buffers that are not readily accessible from user space; this feature would evidently be helpful for the increasingly capable ublk io_uring-based block driver subsystem.

The number of review comments on this work has been relatively small. Stefan Metzmacher said: "This sounds useful to me". But Pavel Begunkov was rather more negative, saying that attempts to add BPF operations to io_uring in the past did not work well. The performance of BPF programs in that context, he said, is poor due to the associated io_uring overhead. He has a different approach, he added, that seems more promising.

Hooking into the control loop

Shortly thereafter, Begunkov posted a new version of a series he has been working on sporadically to add BPF support to io_uring in a different way. Rather than add a new operation type, this series adds a new hook into the io_uring completion loop, allowing a BPF program to be run as operations finish. This implementation, he said, can improve performance by moving CQE processing from user space into the kernel. It also, he said, could eventually allow for the removal of the io_uring linkage mechanism, which he called "a large liability" due to the complexity it adds, entirely.

This series, which shows some signs of having been prepared in a hurry, also sets up a struct ops hook. It adds a single callback which, according to the changelog, should be called handle_events(), but is actually:

    int (*loop)(struct io_ring_ctx *ctx, struct iou_loop_state *ls);

The ctx field gives information about the submission and completion queues, while the iou_loop_state parameter can be used to control how often loop() is to be called, determined by the number of available CQEs and a timeout. When this program is called, it can look at the completed operations, if any, and possibly enqueue new operations in response.

There is a pair of new kfuncs to go along with this mechanism. Pointers to the various parts of the ring buffer can be had with bpf_io_uring_get_region() (though Begunkov says that this interface is likely to be replaced in a future version), and bpf_io_uring_submit_sqes() can be used to submit new operations. Using these kfuncs, a BPF program could replace links by waiting for operations of interest to complete, then submitting the next operation that should follow, perhaps using information from the operations that have already completed.

Lei, having looked at Begunkov's patches, said that they do not solve the problem as well as his operation-based approach. The key difference Lei pointed out is that, with IORING_OP_BPF, the bulk of the application logic, including the creation of SQEs, remains in user space. With Begunkov's series, instead, much of the application logic must be pushed into the kernel, necessitating a lot of communication between user space and the kernel that has the potential to hurt performance. Begunkov answered that the communication can be handled efficiently using a BPF arena, and that his approach provides a greater level of flexibility to handle more types of applications.

Neither developer appears to have convinced the other. Lei intends to continue work on IORING_OP_BPF, while Begunkov is likely to do the same with his patch. Both developers have said that there might be room in the kernel for both approaches, though one might reasonably expect resistance from the wider BPF community to adding what appears to be redundant functionality. A third possibility — that io_uring and BPF remain unintegrated as they have for years — remains a possibility as well.

Comments (none posted)

Checked-size array parameters in C

There are many possible programmer mistakes that are not caught by the minimal checks specified by the C language; among those is passing an array of the wrong size to a function. A recent attempt to add some safety around array parameters within the crypto layer involved the use of some clever tricks, but it turns out that clever tricks are unnecessary in this case. There is an obscure C feature that can cause this checking to happen, and it is already in use in a few places within the kernel.

The discussion started when Ard Biesheuvel sought to improve the safety of the poetically named function xchacha20poly1305_encrypt():

    void xchacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
			           const u8 *ad, const size_t ad_len,
			       	   const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE],
			       	   const u8 key[CHACHA20POLY1305_KEY_SIZE]);

A potential problem with this function is that it takes as parameters several pointers to arrays of type u8. As Biesheuvel pointed out, the size of the nonce and key arrays is not checked by the compiler, even though it is clearly specified in the function prototype. That makes it easy to, for example, give the parameters in the wrong order. The resulting vulnerabilities are generally not the outcome developers have in mind when they write cryptographic code.

Biesheuvel suggested that it was possible to write the prototype this way instead (differences shown in bold):

    void xchacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
			           const u8 *ad, const size_t ad_len,
			       	   const u8 (*nonce)[XCHACHA20POLY1305_NONCE_SIZE],
			       	   const u8 (*key)[CHACHA20POLY1305_KEY_SIZE]);

The types of the last two arguments have changed; there is a new level of pointer indirection, with the argument being a pointer to an array of a given size. Callers must change their calls by adding an additional & operator to obtain the desired pointer type, but the address that is passed is the same. In this case, though, the compiler will check the sizes of the array passed, and will now catch a reordering of the arguments to the function.

Jason Donenfeld was interested by the idea, but he knew of an arguably more straightforward way to address this problem. It seems that, buried deep within the C standard, is a strange usage of the static keyword, making it possible to write the prototype as:

    void xchacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
			           const u8 *ad, const size_t ad_len,
			       	   const u8 nonce[static XCHACHA20POLY1305_NONCE_SIZE],
			       	   const u8 key[static CHACHA20POLY1305_KEY_SIZE]);

This, too, will cause the compiler to check the sizes of the arrays, and it does not require changes on the caller side. Unlike the pointer trick, which requires an exact match on the array size, use of static will only generate a warning if the passed-in array is too small. So it will not catch all mistakes, though it is sufficient to prevent memory-safety and swapped-argument problems.

Eric Biggers pointed out that GCC can often generate "array too small" warnings even without static, but that the kernel currently disables those warnings; that would suppress them when static is used as well. (The warning was disabled in 6.8 due to false positives.) But he thought that adding static was worthwhile to get the warnings in Clang — if Linus Torvalds would be willing to accept use of "this relatively obscure feature of C".

Torvalds, as it turns out, has no objection to this usage; he likes the feature, if not the way it was designed:

The main issue with the whole 'static' thing is just that the syntax is such a horrible hack, where people obviously picked an existing keyword that made absolutely no sense, but was also guaranteed to have no backwards compatibility issues.

He pointed out that there are a number of places in the kernel that are already using it; for a simple example, see getconsxy() in the virtual-terminal driver. He suggested perhaps hiding it with a macro like min_array_size() just to make the usage more obvious, but didn't seem convinced that it was necessary. Donenfeld followed up with a patch to that effect a few days later, but then pivoted to an at_least marker instead.

A lot of work has gone into the kernel to make its use of C as safe as possible, but that does not mean that the low-hanging fruit has all been picked. The language has features, such as static when used to define formal array parameters, that can improve safety, but which are not generally known and not often used. In this particular case, though, it would not be surprising to see this "horrible hack" come into wider use in the near future.

Comments (25 posted)

Some 6.18 development statistics

Linus Torvalds released the 6.18 kernel as expected on November 30, closing the last full development cycle of 2025. It was another busy cycle, featuring a record number of developers. The time has come for a look at where the code came from for this kernel release, but also for the year-long long-term-support cycle which has also reached its conclusion with this release.

As a reminder: LWN subscribers can find much of the information below — and more — at any time in the LWN Kernel Source Database.

The 6.18 release

The 6.18 development cycle brought in 13,710 commits from 2,134 developers. The commit count is more-or-less average for recent releases, but the number of developers involved is the highest that has yet been seen in the kernel's development history. There were 333 first-time kernel contributors this time around, falling just barely short of the record 335 new developers who contributed to 6.12 one year ago.

The most active contributors this time around were:

Most active 6.18 developers

By changesets Takashi Iwai 205 1.5% Stefan Metzmacher 183 1.3% Krzysztof Kozlowski 150 1.1% Brian Masney 145 1.1% Laurent Pinchart 138 1.0% Sean Christopherson 126 0.9% Al Viro 121 0.9% Qianfeng Rong 109 0.8% Eric Biggers 107 0.8% Bartosz Golaszewski 100 0.7% Jakub Kicinski 97 0.7% Mauro Carvalho Chehab 97 0.7% Eric Dumazet 95 0.7% Ian Rogers 94 0.7% Jani Nikula 89 0.6% Russell King 88 0.6% Rob Herring 85 0.6% Thorsten Blum 85 0.6% Thomas Weißschuh 77 0.6% Xichao Zhao 73 0.5%

By changed lines Linus Torvalds 117581 14.9% Przemek Kitszel 16530 2.1% Ian Rogers 13869 1.8% Takashi Iwai 13147 1.7% Taniya Das 12000 1.5% Alice Ryhl 11836 1.5% Eric Biggers 10349 1.3% Abhijit Gangurde 9432 1.2% Fan Gong 7664 1.0% Jerome Brunet 7181 0.9% Jakub Kicinski 7103 0.9% Mauro Carvalho Chehab 7062 0.9% Tomeu Vizoso 6336 0.8% Laura Nao 6241 0.8% Dongsheng Yang 5472 0.7% Alex Deucher 5447 0.7% Dikshita Agarwal 5065 0.6% Beleswar Padhi 5001 0.6% Hector Martin 4870 0.6% Laurent Pinchart 4679 0.6%

Takashi Iwai, the maintainer of the sound subsystem, was the top contributor of changesets this time around; the bulk of them were bringing scope-based resource management to that subsystem. Stefan Metzmacher worked extensively in the SMB filesystem client and server implementations. Krzysztof Kozlowski, as is often the case, contributed a lot of devicetree (and related) changes. Brian Masney performed a large set of API conversions within the clock-driver subsystem, and Laurent Pinchart made a lot of updates within the media subsystem.

Linus Torvalds makes a rare appearance at the top of the "lines changed" column by virtue of having removed the bcachefs filesystem. Przemek Kitszel performed some significant refactoring of the Intel "ice" Ethernet driver. Ian Rogers worked extensively with the perf tool. Iwai's work has been noted above; Taniya Das added a number of clock drivers. Also worthy of mention is Alice Ryhl who, after years of work, contributed the Rust binder driver.

This time around, 9.0% of the commits to the mainline had Tested-by tags, while 53.6% had Reviewed-by tags — both typical numbers. The top testers and reviewers for 6.18 were:

Test and review credits in 6.18
Tested-by Dan Wheeler 134 8.9% Neil Armstrong 91 6.1% Arnaldo Carvalho de Melo 39 2.6% Bryan O'Donoghue 33 2.2% Vikash Garodia 26 1.7% Chris Packham 25 1.7% Rick Edgecombe 25 1.7% Venkat Rao Bagalkote 25 1.7% Mathias Krause 24 1.6% John Allen 24 1.6% Rinitha S 22 1.5% Hans de Goede 21 1.4% Thomas Falcon 20 1.3% Ramu R 18 1.2% Jayesh Choudhary 17 1.1%	Reviewed-by Simon Horman 227 2.3% Dmitry Baryshkov 199 2.0% Konrad Dybcio 179 1.8% Andy Shevchenko 139 1.4% Krzysztof Kozlowski 137 1.4% Geert Uytterhoeven 132 1.3% Lorenzo Stoakes 129 1.3% Frank Li 120 1.2% Linus Walleij 115 1.2% Rob Herring 111 1.1% Alice Ryhl 106 1.1% Andrew Lunn 104 1.1% David Sterba 99 1.0% Christian Brauner 97 1.0% Alex Deucher 94 1.0%

These listings change little from one release to the next; the community's most diligent testers and reviewers are seemingly in it for the long term.

The work in 6.18 was supported by 217 employers that we know of; those most active employers were:

Most active 6.18 employers

By changesets Intel 1420 10.4% (Unknown) 1307 9.5% Google 1080 7.9% Red Hat 866 6.3% AMD 682 5.0% (None) 633 4.6% Linaro 539 3.9% SUSE 534 3.9% Qualcomm 529 3.9% Meta 445 3.2% Renesas Electronics 358 2.6% Huawei Technologies 356 2.6% (Consultant) 309 2.3% NVIDIA 300 2.2% Oracle 257 1.9% vivo Mobile Communication Co 257 1.9% NXP Semiconductors 223 1.6% Arm 210 1.5% Collabora 189 1.4% SerNet 183 1.3%

By lines changed Linux Foundation 118142 15.0% Intel 74255 9.4% Google 67881 8.6% Qualcomm 56423 7.2% (Unknown) 55307 7.0% AMD 35901 4.6% (None) 33870 4.3% Red Hat 29083 3.7% SUSE 26027 3.3% Meta 24669 3.1% Huawei Technologies 23250 3.0% Linaro 19481 2.5% (Consultant) 15261 1.9% NVIDIA 14395 1.8% Collabora 13782 1.8% Texas Instruments 12434 1.6% Renesas Electronics 10756 1.4% NXP Semiconductors 9893 1.3% BayLibre 9025 1.1% Microsoft 8065 1.0%

These numbers, too, tend to be relatively consistent from one release to the next.

Finally, counting the number of Signed-off-by tags applied by developers to patches authored by others gives an insight into patch acceptance; those tags are usually applied when a patch is applied to a subsystem tree on its way toward the mainline. The top appliers of patches in 6.18 were:

Non-author Signed-off-by tags in 6.18

Individual Jakub Kicinski 1055 8.1% Alex Deucher 544 4.2% Andrew Morton 522 4.0% Hans Verkuil 432 3.3% Mark Brown 398 3.1% Greg Kroah-Hartman 363 2.8% Bjorn Andersson 333 2.6% Paolo Abeni 305 2.3% Steve French 265 2.0% Jens Axboe 197 1.5% Arnaldo Carvalho de Melo 192 1.5% Jonathan Cameron 191 1.5% Sakari Ailus 170 1.3% Borislav Petkov 164 1.3% Jonathan Corbet 159 1.2% Alexei Starovoitov 145 1.1% Shawn Guo 141 1.1% David Sterba 132 1.0% Martin K. Petersen 132 1.0% Lee Jones 116 0.9%

Employers Meta 1551 12.0% Google 1439 11.1% Intel 1231 9.5% Red Hat 1008 7.8% AMD 963 7.4% Linaro 768 5.9% Qualcomm 590 4.5% Arm 522 4.0% Linux Foundation 438 3.4% Microsoft 432 3.3% Cisco 432 3.3% SUSE 405 3.1% (Unknown) 329 2.5% NVIDIA 293 2.3% Huawei Technologies 286 2.2% Oracle 240 1.8% (None) 199 1.5% Renesas Electronics 175 1.3% LWN.net 159 1.2% Linutronix 113 0.9%

It still remains true that (just) over half of the patches landing in the mainline pass through the hands of maintainers employed by just five companies — two hyperscalers, two CPU manufacturers, and Red Hat. The Linux Foundation has moved down on this list, perhaps because the amount of activity in the staging tree has fallen considerably over the years, reducing the number of patches handled by Greg Kroah-Hartman.

The longer cycle

For some years now, the last kernel release of each calendar year is designated a long-term-support (LTS) release, which will receive support for a minimum of two years. Most Linux systems in the wild are not running arbitrary kernel releases; they are, instead, running one of these LTS releases. One can thus think of the kernel development cycle as having a one-year cadence; there are checkpoint releases every nine or ten weeks, but there is one major release at the end of the year that is widely deployed.

Last year's LTS release was 6.12; there have been six releases since then that incorporated 80,035 non-merge changesets from 5,275 developers. The most active developers over the course of the year were:

Most active 6.13-18 developers

By changesets Kent Overstreet 901 1.1% Krzysztof Kozlowski 739 0.9% Sean Christopherson 715 0.9% Jani Nikula 653 0.8% Thomas Weißschuh 650 0.8% Bartosz Golaszewski 650 0.8% Jakub Kicinski 579 0.7% Takashi Iwai 571 0.7% Ian Rogers 562 0.7% Eric Biggers 549 0.7% Dmitry Baryshkov 538 0.7% Christoph Hellwig 517 0.6% Thomas Zimmermann 487 0.6% Andy Shevchenko 483 0.6% Matthew Wilcox 478 0.6% Rob Herring 462 0.6% Alex Deucher 455 0.6% Ville Syrjälä 426 0.5% Filipe Manana 415 0.5% Al Viro 412 0.5%

By changed lines Linus Torvalds 123175 2.9% Wayne Lin 81574 1.9% Ian Rogers 80735 1.9% Philipp Hortmann 76407 1.8% Eric Biggers 49999 1.2% Dennis Dalessandro 48357 1.1% Johannes Berg 42369 1.0% Bitterblue Smith 39788 0.9% Jan Kara 36943 0.9% Takashi Iwai 35887 0.8% Dave Penkler 34047 0.8% Rob Herring 33894 0.8% Miri Korenblit 33729 0.8% Dmitry Baryshkov 31440 0.7% Kent Overstreet 29500 0.7% Taniya Das 29077 0.7% Jani Nikula 28008 0.7% Dr. David Alan Gilbert 26438 0.6% Andrew Donnellan 25845 0.6% Richard Fitzgerald 25402 0.6%

In 2024, Kent Overstreet topped the by-changesets list with nearly 4,000 commits; there were two other developers (Uwe Kleine-König and Krzysztof Kozlowski) with over 1,000 commits. This year, the curve is somewhat flatter, with nobody reaching 1,000 commits. The same is true of the lines-changed column, most resulting from a slowdown in the addition of massive, machine-generated amdgpu graphics-driver header files.

It is somewhat ironic, though, that the top of the by-changesets column reflects the addition of bcachefs, while the top of the by-lines column reflects its removal.

The top testers and reviewers over the longer cycle were:

Test and review credits in 6.13-18

Tested-by Dan Wheeler 760 8.9% Neil Armstrong 329 3.9% Arnaldo Carvalho de Melo 157 1.8% Thomas Falcon 113 1.3% Dmitry Osipenko 97 1.1% Rinitha S 92 1.1% Randy Dunlap 89 1.0% Venkat Rao Bagalkote 82 1.0% Rafal Romanowski 76 0.9% Tomi Valkeinen 69 0.8% Mark Pearson 68 0.8% Nicolin Chen 68 0.8% James Clark 64 0.8% Timur Tabi 60 0.7% Bryan O'Donoghue 58 0.7% Babu Moger 56 0.7% Alex Bennée 55 0.6% Pucha Himasekhar Reddy 54 0.6% Vikash Garodia 52 0.6% K Prateek Nayak 51 0.6%

Reviewed-by Simon Horman 1255 2.3% Dmitry Baryshkov 1033 1.9% Krzysztof Kozlowski 932 1.7% Ilpo Järvinen 762 1.4% Christoph Hellwig 736 1.3% Geert Uytterhoeven 735 1.3% Konrad Dybcio 684 1.2% Andy Shevchenko 633 1.1% Andrew Lunn 597 1.1% David Sterba 596 1.1% Rob Herring 556 1.0% Linus Walleij 467 0.8% Neil Armstrong 456 0.8% AngeloGioacchino Del Regno 447 0.8% Frank Li 431 0.8% Laurent Pinchart 427 0.8% Chao Yu 416 0.8% Darrick J. Wong 383 0.7% Christian König 373 0.7% Jeff Layton 373 0.7%

The most active employers (out of the 372 total) over the longer cycle were:

Most active 6.13-18 employers

By changesets Intel 8813 11.0% (Unknown) 6796 8.5% Google 6037 7.5% Red Hat 5181 6.5% (None) 4257 5.3% AMD 4098 5.1% Linaro 3701 4.6% SUSE 2750 3.4% Meta 2744 3.4% Qualcomm 2532 3.2% Huawei Technologies 1917 2.4% Oracle 1886 2.4% NVIDIA 1882 2.4% Renesas Electronics 1858 2.3% Arm 1547 1.9% IBM 1345 1.7% (Consultant) 1252 1.6% NXP Semiconductors 1218 1.5% Linutronix 1149 1.4% Microsoft 887 1.1%

By lines changed Intel 434943 10.2% (Unknown) 383181 9.0% Google 309767 7.2% AMD 277384 6.5% Qualcomm 235450 5.5% Red Hat 215725 5.0% (None) 176686 4.1% SUSE 133567 3.1% Linaro 129946 3.0% Linux Foundation 129058 3.0% Meta 124927 2.9% NVIDIA 107931 2.5% IBM 96906 2.3% Huawei Technologies 82132 1.9% Arm 77217 1.8% Emerson 76919 1.8% Collabora 64629 1.5% NXP Semiconductors 62549 1.5% Oracle 60553 1.4% Renesas Electronics 52732 1.2%

Now begins the 6.19 development cycle, starting this whole process over again. As of this writing, there are just over 12,500 changesets waiting in linux-next, suggesting that 6.19 will be another busy — but typical — development cycle. In other words, the kernel project will be beginning 2026 at full speed; we'll do everything we can to keep up.

Comments (2 posted)

Just: a command runner

Over time, many Linux users wind up with a collection of aliases, shell scripts, and makefiles to run simple commands (or a series of commands) that are often used, but challenging to remember and annoying to type out at length. The just command runner is a Rust-based utility that just does one thing and does it well: it reads recipes from a text file (aptly called a "justfile"), and runs the commands from an invoked recipe. Rather than accumulating a library of one-off shell scripts over time, just provides a cross-platform tool with a framework and well-documented syntax for collecting and documenting tasks that makes it useful for solo users and collaborative projects.

just what it is

Using just has a few advantages over a collection of scripts or bending make to uses it may not be intended for. It's certainly easier to get started with than make for newer Linux users who haven't had the need to learn make previously. Generally, just is more ergonomic for what it does; it isn't a complete replacement for shell scripts or scripting, but it provides a better framework, organization, and user experience.

A task is defined by a recipe. A recipe is, basically a series of commands that are to be run by just. This is a simple recipe that tells me a bit about the operating system just is running on:

    os:
        @echo "This is a {{os()}} system running on {{arch()}} with {{num_cpus()}} logical CPUs."

Running "just os" produces the following output on my system:

    This is a linux system running on x86_64 with 16 logical CPUs.

The recipes for tasks are collected in a justfile and stored in the user's home directory or in the root directory of a project. Technically, these can be broken out into multiple files and imported into the main justfile, but let's not go down that rabbit hole just now.

Say an open-source project includes a justfile in its Git repository for doing common tasks, such as updating the changelog, running tests, or publishing a release; all a new contributor needs to do is run "just -l" to discover the tasks that are available, along with their documentation. Assuming, of course, someone has written the documentation—just is useful, but it's not magic.

Another advantage is that just is expressly designed for running tasks across multiple platforms and for collaborative use. For instance, it has built-in functions for determining and acting on system information (such as os() for the operating system), environment variables, error handling, and much more. A task can be written to use sh on Linux, ksh on macOS, and PowerShell on Windows, if needed. Recipes can have shared variables and settings, so one need not redefine $TMPDIR in each shell script for a project. A recipe can also have dependencies, specified similarly to makefile dependencies, which can be reused for multiple recipes as needed.

Users can invoke multiple recipes at once, and a recipe's dependencies can be skipped by using the --no-deps option when a user just wants to execute the primary task. For example, perhaps a task has a dependency on downloading an ISO image or other large file; that can be skipped if it has been run recently and the user knows there is not a newer one.

Project history

Casey Rodarmor created just in 2016. The project is inspired by make; in fact it was originally just a shell script that called make. But just is not meant to be a replacement for make or a build system in general; it is just a command runner. It is licensed, somewhat unusually, under the Creative Commons Zero (CC0) 1.0 Universal deed: essentially, it is public-domain code. Rodarmor wrote that he chose CC0 because he had "no desire to impose any restrictions whatsoever on anyone who wants to use my code." Nearly 200 people have made small contributions to just, but the bulk of development is Rodarmor's work.

Like many maintainers of open-source projects these days, Rodarmor does not tend to do major releases with many new features. Instead, there is a frequent stream of minor releases with a few new features and miscellaneous improvements, along with point releases with bug fixes. The most recent minor release, 1.43.0, came out in September, with the 1.43.1 bug fix release on November 12. According to the Just Programmer's Manual, there will never be a 2.0 release. Any backward-incompatible changes, "will be opt-in on a per-justfile basis, so users may migrate at their leisure". Any recipes written for a version of just since 1.0 should continue to work without breakage indefinitely.

Most popular Linux distributions provide just packages, though the version in a distribution's repository may be somewhat out of date given how frequently the project is released. Users can also snag the latest version using "cargo install just", or use one of the project's pre-built binaries.

Justfiles

The just command looks for a file named justfile with recipes in the current directory or those above it. The idea is that a user might include a justfile with each project, as well as a justfile in their home directory for miscellaneous recipes. So, if the present working directory is ~/src/project, the "just foo" command will tell the utility to look in ~/src/project for a justfile. If it is not found there, then just will look in ~/src, and ~ for a justfile. Technically it will keep looking in /home and such, but one hopes users are not placing their recipe files there. One of the features added in 1.43.0 is a --ceiling option; this lets the user specify the "top" directory where just should stop looking for a justfile.

Note that the name is case-insensitive; just will accept Justfile, justfile, JUSTFILE, and .justfile if one prefers to hide configuration files. Even JuStFiLe will work, though it would be an affront to good taste. Naturally, there is a justfile in the just repository that has recipes for building project documentation, running demos, creating the just man page, updating Rust, and more.

Once just finds a justfile, it stops looking; this means that, for example, if one has a justfile in ~/ and another in ~/project/foo, running just in the ~/project/foo directory will usually not find any recipes above that directory. This can be changed by adding the set fallback directive in a recipe file. That way, if a user runs "just foo" and there is no recipe foo to be found in the first justfile, it will keep looking in the directories above it.

The syntax for just is inspired by make, but Rodarmor wanted to avoid what he calls the idiosyncrasies of make: such as the need to have .PHONY targets when there is a file with the same name as a make target (e.g., "clean") in the directory. Recipe lines must be indented at least one level from the recipe name, but users have a choice of spaces or tabs. Note that each recipe does have to be consistent: just does not allow indenting one line of a recipe with spaces, and another line with tabs.

Here is a simple recipe, called pub I use to update my Hugo-based static site:

    # update dissociatedpress.net
    [no-cd]
    pub:
        hugo
        rsync -avh --progress -e ssh /home/user/src/dissociatedpress/public/ \
        user@dissociatedpress.net:/path/to/www/html
        w3m https://dissociatedpress.net/

When I run "just pub", just steps through the recipe and runs each line in a separate shell. It runs hugo in the directory where the justfile is invoked; the default is for just to run a recipe in the directory where the justfile is stored, but [no-cd] instructs it to run the recipe in the current directory instead. It will rsync the contents to the server, and then open the site in w3m so I can verify that the site was updated.

Running "just -l" will list all of the available recipes in a justfile. If there is a comment above the recipe, it will be displayed:

    $ just -l
    pub #update dissociatedpress.net

The "just --choose" command will display an interactive menu that allows the user to choose the just recipe to run. This makes use of the fzf command-line fuzzy finder by default; it will work without it, but just prints a warning if it cannot find fzf. Most distributions should have fzf available.

Polyglot

Typically, just uses "sh -cu" to execute the commands in a recipe, unless configured otherwise. That can be changed with the "set shell" setting in a recipe, such as this:

    set shell := ["fish", "-c"]

One of just's more interesting features is its support for writing recipes in many programming languages, so users are not limited to shell interpreters; these are called shebang recipes. Basically, a user can write a recipe using JavaScript, Perl, Python, Ruby, and just about any other scripting language. A shebang recipe is saved to a temporary file by just and then run using the interpreter specified:

    python:
        #!/usr/bin/env python3
        print('GNU Terry Pratchett')

Running "just python" saves the Python script to /tmp to be executed by python3. The temporary directory can be specified in several ways if one wants to avoid placing scripts in a world-readable directory. Users can, of course, substitute their favorite languages; a justfile can contain shebang recipes in multiple languages with no problem.

Recipes can use conditional expressions, take variables from the command line, and much more. The programmer's manual does a fine job of laying out all of just's features and how to use them. There is also a cheatsheet for just syntax and grammar; it is a useful thing to have bookmarked if one starts to use just.

Simple and useful

The project has a Discord channel for discussion and support. There is a GitHub issue that Rodarmor uses to solicit user feedback when he is considering major new features, such as new recipe types. He suggests that users subscribe to that issue if they would like to provide input on features in the future. Sadly, GitHub does not offer RSS feeds for issues, so the only way to keep up with that is to have a GitHub account and use its subscribe feature.

I had heard of just but had not bothered to try it out until I started testing the Project Bluefin distribution in 2023. The project uses just, aliased to ujust, to use the system-wide justfiles for a number of administrative tasks or to toggle user features on and off. For example, Bluefin has "ujust benchmark" to run a one-minute benchmark test with the stress-ng utility, and "ujust setup-luks-tpm-unlock" to enable automatic disk unlocking. The justfiles for the distribution can be found on GitHub and the project also maintains a justfile with recipes for building the distribution images. Those files provide some interesting examples of just's capabilities.

Since then, I've been using just rather than shell scripts and such for automating little things (or medium-sized things...), and converting old scripts to recipes. It's one of the first tools I install on a new system, and it makes everything just a bit easier.

Comments (36 posted)