Leading items

Welcome to the LWN.net Weekly Edition for February 13, 2025

This edition contains the following feature content:

• Improved load-time checking for BPF kfuncs: the permissions mechanism for BPF programs is showing its age; the form if its replacement is still coming into focus.
• Smarter IRQ suspension in the networking stack: how a simple change can greatly improve the efficiency of the kernel's network stack.
• Maintainer opinions on Rust-for-Linux: a FOSDEM keynote on the state of Rust in the Linux kernel and how kernel maintainers feel about it.
• Rewriting essential Linux packages in Rust: a shiny new replacement for the GNU coreutils package.
• The selfish contributor revisited: what makes successful free-software projects work?
• Milliwatt machine learning with emlearn: machine learning does not have to be confined to data centers.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Improved load-time checking for BPF kfuncs

The BPF verifier is charged with the challenging task of ensuring that a BPF program is safe for the kernel to run before that program is loaded. Among many other concerns, the verifier must ensure that any kfuncs (kernel functions that have been exported to BPF programs) are called with the correct parameters and from the right context. The "context" part of that enforcement is showing its age in ways that are hurting performance; Juntong Deng has been working on infrastructure to provide finer-grained control over when a kfunc can be called.

Every BPF program is assigned a specific type; the full list of types can be found in Documentation/bpf/libbpf/program_types.rst. So, for example, BPF_PROG_TYPE_FLOW_DISSECTOR is for programs that implement network flow dissectors, while BPF_PROG_TYPE_LSM is for programs run from the BPF Linux Security Module. The kernel will not allow a program to be attached to a BPF hook if the program is not of the correct type for that hook. This restriction prevents BPF programs from being invoked from contexts that they are not designed for.

The kernel's kfunc mechanism is a relatively recent addition that allows any function within the kernel to be made available for direct calling from a BPF program. Here, too, it is important that kfuncs are only called from the correct context. So, whenever a set of kfuncs is registered with the BPF subsystem (using a call to register_btf_kfunc_id_set()), the program type must be supplied; the verifier will ensure that only programs of the given type can call a kfunc from that set.

This machinery works but has come under increasing strain over the years. The number of program types has grown considerably, and that has led to a desire to restrain that growth (without, of course, slowing the incursion of BPF into the few parts of the kernel where it is not yet found). That has resulted in BPF program types becoming more generic; in particular, the "struct ops" mechanism allows a BPF program to provide a structure full of functions that the kernel can call, under the BPF_PROG_TYPE_STRUCT_OPS program type. There are quite a few programs out there of this type that run in all kinds of contexts.

Any verification mechanism that relies on just the program type will be unable to tell one struct-ops program from another. Beyond that, though, there are reasons to treat the different functions called within a single struct-ops program as having different contexts. The BPF subsystem is currently unable to make that distinction, and that has complicated life.

In this patch set, Deng pointed to the sched_ext subsystem, which allows CPU schedulers to be written in BPF, as an example of this problem. A sched_ext program is of type BPF_PROG_TYPE_STRUCT_OPS; when it loads, it provides a structure of type sched_ext_ops with about three-dozen different function pointers, each for a callback that handles one aspect of the scheduling problem. The runnable() callback, for example, is invoked when a task becomes runnable and must be placed into a run queue, while cpu_offline() is called if a CPU is being removed from the system and tasks must be moved off of it. Clearly, the context in which these callbacks are called will vary considerably from one to the next.

The sched_ext subsystem also provides a number of kfuncs that allow BPF programs to perform scheduling tasks, such as putting a task onto a specific CPU. It is only appropriate to call some of those kfuncs from specific sched_ext_ops functions, though; they only make sense during the appropriate parts of the scheduling flow. To avoid problems, the sched_ext subsystem must track which BPF function is being called at the moment and, when a kfunc is invoked, ensure (with a call to scx_kf_allowed()) that the calling context is correct. This is an extra run-time check that would be more nicely done at load time; evidently this check is expensive enough to impact the performance of sched_ext schedulers.

Deng's first solution to this problem was to add a "capabilities" mechanism to the BPF subsystem. A capability mask (a 32-bit integer value) was added to each kfunc; bits set in that mask would indicate the capabilities needed to be able to call the kfunc. Each kfunc could then be registered along with the requisite capabilities. The patch set also provided a new callback (bpf_capabilities_adjust()) that would allow a subsystem (such as sched_ext) to specify which capabilities are held by a BPF program that it might run. This callback is invoked separately by the verifier during the checking of each BPF_PROG_TYPE_STRUCT_OPS function, allowing each to be provided with separate capabilities that may (or may not) make a given kfunc available. The end result is that the verifier gained the ability to prevent inappropriate kfunc calls at load time, and the run-time overhead was eliminated.

This implementation raised some concerns, though. Tejun Heo pointed out that a 32-bit mask for capabilities would surely be exhausted at some point. He also wondered if it was necessary to declare capabilities globally at all; given that the callback was needed in any case, it could just accept context information and make purely local decisions at that time. Alexei Starovoitov took issue with the term "capabilities", which already has a well-defined meaning in the kernel, but he also thought that the concept was unnecessary. He suggested just implementing this functionality as a filter callback instead.

Filters are a similar mechanism that were added during the 6.5 cycle by Aditi Ghag. They are yet another callback that is associated with each kfunc. The verifier will call this filter() function, if it exists, to determine whether a given call should be allowed or not. This functionality does seem quite similar to what capabilities brought, but Deng had some reservations about using it; in particular, filter functions do not work with kernel subsystems implemented as loadable modules. Starovoitov, though, was unworried about this restriction; the current user for this feature is sched_ext, which cannot be built as a module. The immediate problem (the performance impact on sched_ext) should be solved first, he said; other concerns can be addressed later if the need arises.

So Deng solved this problem anew, specifically for sched_ext, using the filter functionality. The new series adds some context information to the (already large) bpf_prog_aux structure that is available to filter functions. This context consists of a pointer to the operations structure itself and the byte offset within the structure of the specific function being called. Filter functions can use that information to determine which struct-ops function is being called and make a decision about whether that call should be allowed. As can be seen from this patch, for example, this mechanism is arguably not the most elegant ever, but it does get the job done.

In any case, it is sufficient to add load-time checks for sched_ext and eliminate the need for the run-time checks, once again addressing the performance problem. There are some residual glitches; if a kfunc appears in more than one exported set (which happens reasonably often when kfuncs must be exported to more than one program type), the filter no longer knows which offset to use and can make incorrect decisions. That causes at least one sched_ext program to fail to load with the current patch set. There are ways to work around this problem, including adding a simple wrapper for the kfunc to distinguish between the calling contexts, but this seems like the kind of trap that can easily snare unwary developers.

This new series is fresh as of this writing and has not yet generated any discussion. It does appear to have addressed the concerns raised the first time around, though, and to solve the immediate problem. So, while "capabilities" will not be coming to BPF programs anytime soon, better load-time decisions on the validity of kfunc calls would appear to be on the horizon.

Comments (none posted)

Smarter IRQ suspension in the networking stack

High-performance networking is a highly tuned activity; the amount of time available to deal with each packet may be measured in nanoseconds, so care must be taken to avoid anything that might slow the process down. Recently, there has been a fair amount of attention given to a patch set merged for 6.13 that, it is claimed, can improve processing efficiency (and, thus, power savings) in data centers by as much as 30%. The change itself, contributed by Joe Damato and Martin Karsten, is a relatively small tweak to existing optimization techniques; it shows just how much care is needed to optimize a high-bandwidth server.

In a simple configuration, a network interface will receive a packet, place it in RAM somewhere, then interrupt the CPU to let the kernel know that the packet is ready for processing. That worked well enough when networks were slow, but is decidedly less than optimal in current data centers. Older readers out there will remember a time when every email was special, so we configured our systems to beep (interrupt) at us whenever a message arrived. But that interrupt has a cost, and we do not do that anymore. Instead, we check email occasionally in the sure knowledge that there will be a bountiful supply of new phishing spam waiting for us when we get there.

The kernel's networking stack has long taken a similar approach. When there is a lot of traffic, the kernel will tell the interface to stop interrupting. Instead, the kernel will poll the interface occasionally to catch up with the new packets that are sure to be waiting. Masking interrupts in this way allows the kernel to process packets in batches, without being interrupted; that helps to improve throughput.

User-space polling

There is room for further improvements, though. By default, the kernel performs this polling in a software-interrupt routine that runs asynchronously from the application that is consuming the incoming data. The interrupt handler and the application will likely often run concurrently; since they are both working on the same network flows, the result can be locking contention and cache misses. If your time budget for processing a packet is measured in nanoseconds, even a single cache miss can cause that budget to be exceeded.

To address this problem (which only affects the most heavily loaded of servers, but there are a lot of those), the responsibility for polling can be pushed all the way out to user space. If the application selects a special "preferred busy polling" mode, it makes a solemn pledge to the kernel that it will frequently poll the incoming network stream and process the packets that have arrived. The kernel will respond by turning off its own software-interrupt-based packet processing. That processing can, instead, be done when the application polls, so that it will not contend with the application's user-space processing. This kind of polling can yield tiny packet-processing latencies, but it can also drive up CPU usage, especially during times when there are no packets waiting and the application burns CPU time polling without finding any work to do.

To minimize the CPU-usage problem (and to potentially allow the CPU to go into a lower-power state during slow times), the system can go back to an interrupt-driven mode. If it's not clear when the next packet will arrive, the kernel can simply stop polling, cause the application to block, and request an interrupt instead. There is a tradeoff here, though: in a moderately busy system, there is a good chance that a packet will arrive immediately after the switch to the interrupt-driven mode. Typically, it is better to wait for a little while before doing that.

To this end, the network stack has a couple of parameters that a high-performance application can tweak. napi_defer_hard_irqs is the number of times that an application should be allowed to poll without receiving any data before it is blocked and interrupts are enabled; that will keep the system in the polling mode over tiny gaps in the incoming packet stream. Even after that many attempts, though, interrupts are not enabled immediately; that would invite an interrupt on arrival of the first packet, when there is little work for the interrupt handler to do. It is better to wait a bit longer for traffic to accumulate. So the other parameter, gro_flush_timeout, tells the kernel how long it should wait (in nanoseconds) before re-enabling packet-receipt interrupts.

The gro_flush_timeout knob serves a second function as well: is a sort of safety factor, specifying a period of time during which the application should perform at least one poll. If that poll doesn't happen before the timeout period expires, the kernel assumes that the application has gotten distracted and forgotten about its promise to keep polling; it then restarts software-interrupt processing to take the polling responsibility back into its own hands.

A new knob

This dual role for gro_flush_timeout is at the root of the problem that was solved by the new patch set. Its value sets a lower bound for the response latency whenever polling stops; if it is set to an overly large value, response times will suffer during slower periods. Pausing for traffic to accumulate is good for throughput, but pausing for too long creates latency. If, instead, this value is set too small, the timeout will trigger while the application is processing packets; that will lead to software-interrupt processing happening concurrently, impacting performance. There is often no value that is perfect for both roles.

The answer is to split the roles by introducing yet another knob: irq_suspend_timeout, which is also specified in nanoseconds. When an application is running in the preferred busy polling mode and receiving data, the value of irq_suspend_timeout is used, rather than gro_flush_timeout, to determine how long the kernel should wait for the application's next poll before concluding that software-interrupt processing must resume. This timeout will be reset every time the application polls for more data and, importantly, successfully retrieves more data to process.

The regime changes the moment that a poll returns without finding any data; at that point, the kernel reverts to the older mode, allowing napi_defer_hard_irqs empty polls before starting the gro_flush_timeout delay, then re-enabling interrupts. In other words, the new timeout only applies while packets continue to arrive.

This mechanism allows irq_suspend_timeout to be set to a relatively long value, since it only applies during busy times when the application is actively processing data. Meanwhile, gro_flush_timeout, which only applies when a pause has been seen in incoming traffic, can be set to a relatively short value, with the result that processing will restart quickly once new data arrives. The promised result is both high throughput when traffic is high and low latency when things slow down, while also allowing the CPU to sleep (or perform other work) during those slower times.

The benchmark results included with the patch set would appear to back up this promise. When running in the new mode, a system is able to deliver consistent (and relatively low) latency as well as if it were running in a full busy-wait mode, but with CPU utilization that is much closer to the full interrupt-deferral case. This is where the claims of power savings come from; a server is able to deliver the required level of service, but without wasting lots of CPU time to contention or doing busy waiting. This one change can, evidently, remove most of the performance advantage that user-space networking solutions can have over the kernel.

Clearly, this new knob is not going to be something that most users, even those running servers, will want to play with. Enabling preferred busy polling is a balancing act, with a lot of attention required to find the right values for the relevant parameters, and constant monitoring is needed to ensure that the system is running optimally. Adding a new knob makes things a bit more complicated still. But for organizations running unimaginable numbers of servers and trying to get as much performance as possible out of each, this relatively simple tweak to the networking stack could make a world of difference.

Comments (1 posted)

Maintainer opinions on Rust-for-Linux

Miguel Ojeda gave a keynote at FOSDEM 2025 about the history of the Rust-for-Linux project, and the current attitude of people in the kernel community toward the experiment. Unlike his usual talks, this talk didn't focus so much on the current state of the project, but rather on discussing history and predictions for the future. He ended up presenting quotes from more than 30 people involved in kernel development about what they thought of the project and expected going forward.

Background information

Ojeda began by explaining Rust-for-Linux for those audience members who may not have been familiar with it, defining it as an attempt to add support for the Rust language to the Linux kernel. The project is not only about adding Rust to drivers, he said, it's about eventually having first-class support for the language in the core of the kernel itself. The long-term goal is for kernel maintainers to be able to choose freely between C and Rust.

Despite the enormity of such a task, many people are already building drivers on top of the in-progress Rust bindings. Ojeda took a moment to discuss why that is. The Linux kernel is already extensible, and introducing a new language has a huge cost — so why would people go to so much effort just to be able to write drivers in Rust, specifically?

In answering that question Ojeda said he wanted to give a more satisfying answer than the usual justification of "memory safety". He put up a piece of example code in C, asking whether the function did what the comment describes:

    /// Returns whether the integer pointed by `a`
    /// is equal to the integer pointed by `b`.
    bool f(int *a, int *b) {
        return *a == 42;
    }

The developers in the audience were all fairly certain that the presented code did not, in fact, follow the comment. Ojeda put up the obvious corrected version, and then showed the equivalent code in Rust:

    // Corrected C
    bool f(int *a, int *b) {
        return *a == *b;
    }

    // Equivalent Rust versions before and after being fixed
    fn f(a: &i32, b: &i32) -> bool {
        *a == 42
    }

    fn f(a: &i32, b: &i32) -> bool {
        *a == *b
    }

The Rust functions are nearly identical to the C functions. In fact, they compile to exactly the same machine code. Apart from how a function declaration is spelled, there's no obvious difference between C and Rust here. And neither language helps the programmer catch the mismatch between the comment and the behavior of the function. So why would anyone want to write the latter instead of the former?

Ojeda's answer is: confidence. As a maintainer, when somebody sends in a patch, he may or may not spot that it's broken. In an obvious case like this, hopefully he spots it. But more subtle logic bugs can and do slip by. Logic bugs are bad, but crashes (or compromises) of the kernel are worse. With the C version, some other code could be relying on this function to behave as documented in order to avoid overwriting some part of memory. That's equally true of the Rust version. The difference for a reviewer is that Rust splits things into "safe" and "unsafe" functions — and the reviewer can concentrate on the unsafe parts, focusing their limited time and attention on the part that could potentially have wide-ranging consequences. If a safe function uses the incorrect version of f, it can still be wrong, but it's not going to crash. This lets the reviewer be more confident in their review.

Ojeda "hopes this very simple example piques your curiosity, if you're a C developer", but it also answers why people want to write kernel components in Rust. Kernel programming is already complex; having a little more assurance that you aren't going to completely break the entire kernel is valuable.

The goals listed in the original Rust-for-Linux RFC do include other points, such as the hope that Rust code will reduce logic bugs and ease refactoring. But a core part of the vision of the project has always been to make it easier to contribute to the kernel, thereby helping to get people involved in kernel development who otherwise wouldn't feel comfortable.

History

With that context established, Ojeda then went into the history of the Rust-for-Linux project. The idea of using Rust with the Linux kernel is actually more than a decade old. The earliest example is rust.ko, a simple proof of concept written in 2013, before Rust had even reached its 1.0 version in 2015. The actual code that would become the base of the Rust-for-Linux project, linux-kernel-module-rust, was created in 2018 and maintained out of tree for several years.

Ojeda created the Rust-for-Linux GitHub organization in 2019, but it wasn't until 2020 that the project really got going. During the Linux Plumbers Conference that year, a number of people gave a collaborative talk about the project. It was "a pipe dream" at that point, but enough people were interested that the project started to pick up steam and a number of people joined.

Through the end of 2020 and the beginning of 2021, contributors put together the first patch set for in-tree Rust, set up the mailing list and Zulip chat instance, and got Rust infrastructure merged into linux-next. 2021 also saw the first Kangrejos, the Rust-for-Linux conference, run through LWN's BigBlueButton instance. In 2022, the set of Rust-for-Linux patches went from version 5 to version 10, before finally being merged in time for the Linux 6.1 long-term support release.

From that point on, the project (which had always been working with the upstream Rust project) began to collaborate with Rust language developers more closely. The project gained additional infrastructure, including a web site, and automatically rendered kernel documentation. Several contributors also began expanding the initial Rust bindings. Ojeda specifically called out work by Boqun Feng, Wedson Almeida Filho, Björn Roy Baron, Gary Guo, Benno Lossin, Andreas Hindborg, Alice Ryhl, Trevor Gross, and Danilo Krummrich, placing their contributions on a timeline to highlight the growth of the project over time.

The future

Ojeda ended his talk with a series of quotes that he had solicited from many different kernel developers and people associated with the Linux community about what they thought about the future of the project. People often form their impressions of what a community thinks of a topic based on what causes the most discussion — which has the effect of amplifying controversial opinions. To get a clear picture of what existing kernel maintainers think of the Rust-for-Linux experiment, Ojeda reached out to many people who had been involved in discussing the project so far — whether that was as a proponent or as a detractor. There were many more quotes than he could go through in the time remaining, but he did highlight a few as particularly important or insightful. Interested readers can find the full list of quotes in his slides.

Daniel Almeida, who has contributed several patches to the kernel, particularly around laying the groundwork for GPU drivers in Rust, said:

2025 will be the year of Rust GPU drivers. I am confident that a lot will be achieved once the main abstractions are in place, and so far, progress has been steady, with engineers from multiple companies joining together to tackle each piece of the puzzle. DRM maintainers have been very receptive too, as they see a clear path for Rust and C to coexist in a way that doesn't break their established maintainership processes. In fact, given all the buy-in from maintainers, companies and engineers, I'd say that Rust is definitely here to stay in this part of the kernel.

Ojeda agreed with Almeida's prediction, saying that he expected 2025 to be a big year for Rust in the graphics subsystem as well. Many Rust-for-Linux contributors were (unsurprisingly) also optimistic about the project's future. Even people who were less enthused about the language did generally agree that it was going to become a growing part of the kernel. Steven Rostedt, a kernel maintainer with contributions in several parts of the kernel, thought that the language would be hard for other kernel contributors to learn:

It requires thinking differently, and some of the syntax is a little counter intuitive. Especially the use of '!' for macros, but I did get use to it after a while.

Despite that, he thought Rust in the kernel was likely to continue growing. Even if Rust is eventually obsoleted by a safer subset of C for kernel programming, Rust would still have provided the push to develop things in that direction, Rostedt said. Wolfram Sang, who maintains many I2C drivers (an area the Rust-for-Linux project would like to expand into), also expressed concerns about the difficulty of learning Rust:

I am really open to including Rust and trying out what benefits it brings. Yet personally, I have zero bandwidth to learn it and no customer I have will pay me for learning it. I watched some high level talks about Rust in Linux and am positive about it. But I still have no experience with the language.

This left him with concerns around reviewing Rust code, saying that he "simply cannot review a driver written in Rust". He asked for help from someone who is able to do that, and hoped to learn Rust incrementally in the process.

Luis Chamberlain, who among other things maintains the kernel's loadable module support, thought that there were still some blocking requirements for the adoption of Rust, but was eager to see them tackled:

I can't yet write a single Rust program, yet I'll be considering it for anything new and serious for the kernel provided we get gcc support. I recently learned that the way in which we can leverage Coccinelle rules for APIs in the kernel for example are not needed for Rust -- the rules to follow APIs are provided by the compiler for us.

While both mild reservations and cautious optimism were common responses to Ojeda's request for comment, several respondents were less restrained. Josef Bacik, a maintainer for Btrfs and the block device I/O controller, said:

I've been working exclusively in Rust for the last 3 months, and I don't ever want to go back to C based development again.

Rust makes so many of the things I think about when writing C a non-issue. I spend way less time dealing with stupid bugs, I just have to get it to compile.

[...]

I wish Rust were more successful in the linux kernel, and it will be eventually. Unfortunately I do not have the patience to wait that long, I will be working on other projects where I can utilize Rust. I think Rust will make the whole system better and hopefully will attract more developers.

The quotes Ojeda gathered expressed a lot of thoughtful, nuanced opinions on the future of the Rust-for-Linux project. To crudely summarize: the majority of responses thought that the inclusion of Rust in the Linux kernel was a good thing; the vast majority thought that it was inevitable at this point, whether or not they approved. The main remaining obstacles that were cited were the difficulty of learning Rust, which may be difficult to change, and GCC support, which has been in progress for some time.

The responding kernel developers also thought that, even though Rust-for-Linux was clearly growing, there was a lot more work to be done. Overall, the expectation seems to be that it will take several more years of effort to have all of the current problems with Rust's integration addressed, but there is a willingness — or at least a tolerance — to see that work done.

[While LWN could not attend FOSDEM in person this year, and the video of Ojeda's talk is not yet available, I did watch the stream of the talk as it was happening in order to be able to report on it.]

Comments (172 posted)

Rewriting essential Linux packages in Rust

Most Linux systems depend on a suite of core utilities that the GNU Project started development on decades ago and are, of course, written in C. At FOSDEM 2025, Sylvestre Ledru made the case in his main stage talk that modern systems require safer, more maintainable tools. Over the past few years, Ledru has led the charge of rewriting the GNU Core Utilities (coreutils) in Rust, as the MIT-licensed uutils project. The goal is to offer what he said are more secure, and more performant drop-in replacements for the tools Linux users depend on. At FOSDEM, Ledru announced that the uutils project is setting its sights even higher.

Ledru has been a Debian developer for more than 20 years, and is a contributor to LLVM/Clang as well as other projects. He is a director at Mozilla, but said that the work he would be talking about is unrelated to his day job.

Bread, woodworking... or Rust?

Ledru said that learning Rust was a project he started during the COVID lockdown. Some people chose to learn to bake, others took up woodworking, and he took up rebuilding all the LEGO kits in the house with his son. But he wanted a project in the evening that would help him learn Rust. "I have been surrounded by upstream Rust developers inside the Paris office of Mozilla, so I wanted to learn it myself." He didn't want to take on a side project that would just sit on his hard drive, he wanted to do something with impact.

"So I was thinking, what about reimplementing the coreutils in Rust?" Well before COVID, Ledru had worked on rebuilding the Debian archive using Clang, a project that is documented at clang.debian.net. Ledru said that he had been inspired by Chris Lattner's work on Clang. One of the core fundamentals of Clang, he said, is the philosophy that "if you have different behaviors than GCC, it's a bug."

Next, he asked the audience who knew which programs were in GNU coreutils. Most people knew at least one, many knew at least five. But he was pretty sure no one in the audience knew everything in the list of coreutils, unless they happened to be an upstream developer on the project. Ledru said that pr, used to format text for printing "on actual paper", is one of his favorite coreutils programs. To start on the project he selected "all the fancy ones" like chmod, chown, ls, and mkdir—the commands that people on Linux and macOS use almost every day for their work.

Near full completion

Now, five years later, Ledru said that he has more gray hairs, and the project has Rust replacements for all of the more than 100 commands in coreutils. The project has more than 530 contributors, and more than 18,000 stars on GitHub, "[they're] meaningless, but it's one of the metrics we have".

A more meaningful measurement of the project's success is how well it fares against the GNU coreutils test suite. He said that the project only passed 139 tests when it started testing in 2021, and it was now close to 500 out of 617. "I should have worked last weekend to be at 500, but I didn't have the time," Ledru said. (According to the JSON file of results, it crossed 500 tests passed on February 4, with 42 skipped and 75 failed.) He displayed a slide with a graph of test suite runs from April 2021 to late January 2025, shown below.

Most of the tests that the project still fails have pull requests to fix the problems, or may be things that "nobody cares about" as well as some "weird ordering" problems. For example, if a user tries to use rm in a directory without the appropriate permissions, the output may be slightly different. "GNU is going to show something first, and we show it at the end, and it can make a small difference".

More and more of the programs are passing all of the GNU tests, but that may not mean the Rust version is fully compatible with the GNU implementation because the test suite itself has some limitations. Ledru said that the project has been contributing to the GNU coreutils project when it finds things that are not tested.

He said that the Rust coreutils are now "production ready", and that they support Linux, FreeBSD, NetBSD, OpenBSD, Illumos, Redox, Android, macOS, and Windows. There are also Wasm versions. According to Ledru, the Rust coreutils are used by the Debian-based Apertis distribution for electronic devices, the Spectacles smartglasses, and Microsoft is using the project for Visual Studio Code for the web. The Serpent OS distribution is using them by default, he said. He noted that, since the project is open source, it is probably in use elsewhere without his knowledge and asked any company using them to let him know. "I'm always excited to know when people are using our code".

Why Rust

Ledru's talk was immediately after Miguel Ojeda's keynote on Rust for Linux. Ledru said that Ojeda had already covered some of the reasons for using Rust, but that he would offer his point of view as well.

As part of the release-management team for Firefox, he was involved when the browser started shipping with Rust code. That made him "completely biased" about the suitability of Rust for the browser use case. He pointed out that Chrome, the Linux kernel, and Microsoft Windows are starting to include Rust. "I'm going to state the obvious, that Rust is very good for security, for parallelism, for performance".

The idea to replace GNU coreutils with Rust versions was not about security, though, because the GNU versions were already quite secure. "They did an amazing job. They almost don't have any security issues in their code base." And it's not about the licensing, he said. "I'm not interested in that debate."

One of the reasons that Ledru liked Rust for this project, he said, is that it's very portable. He is "almost certain" that code he writes in Rust is going to work well on everything from Android to Windows. That is very surprising, he said, given the complexity of "everything we do" but the cost of porting a change to an operating system is small "thanks to the ecosystem and quality of Rust".

Ledru cited laziness as another reason for using Rust. "So if there is a crate or library doing that work, I'm going to use it. I'm not going to implement it [myself]." There are between 200 and 300 dependencies in the uutils project. He said that he understood there is always a supply-chain-attack risk, "but that's a risk we are willing to take". There is more and more tooling around to help mitigate the risk, he said.

He is thinking about "what we are going to leave to the next generation". Developers starting out don't want to use COBOL, Fortran, or C, he said. They want to work with fancy stuff like Rust, Swift, Go, or Kotlin. It is a good investment to start planning for the future now and transition to new languages for Linux and the computer ecosystem in general.

Demo time

Even though the goal is for the Rust coreutils to be drop-in replacements, Ledru said, "we take the liberty at times to differentiate ourselves from the GNU implementation". Here he showed a demo of the cp command with a --progress option that is not available with the standard GNU version of cp. He said it was available with the mv command too, and invited the audience to ask if there were other places the project should add it. "In Rust, it's pretty easy to add that".

He also walked through a demo that compared the Rust implementation of sort to GNU's. He used the hyperfine command-line benchmarking tool to run a test ten times; sorting a text file containing all of Shakespeare's works to see which implementation was faster. The first time he performed the test, he used a debug build of the Rust version of sort. In that demo, Rust's version was 1.45x faster than the GNU version. Then he ran the test again using a non-debug version, which showed the Rust version performing the test six times faster than GNU's implementation.

Currently, the project has continuous integration (CI) and build systems for most of its supported platforms, with almost 88% of the code covered by a test suite. "If you are above 80, you usually are very happy. Here we are even happier with nearly 90%". Despite trying to demonstrate how the Rust implementation was better than GNU's, Ledru stressed that there is a friendly collaboration between projects and that they have been sending bug reports and patches upstream for GNU coreutils.

What's next

Rewriting more than 100 essential Unix and Linux utilities in a new language is an impressive achievement. Some, upon nearing completion of such a project, might stand back and admire the work and think about calling it a day. Ledru, apparently, is not one of those people.

He displayed a slide with the familiar adage "the best time to plant a tree is 20 years ago, the second-best time is now", and talked a bit about the age of the Unix core utilities. The original Unix utilities will be 55 years old in four or five months, he said. Despite their age, the utilities (albeit newer implementations of them) live on and continue to evolve. Ledru pointed out that the GNU project continues to add new options to the coreutils and introduce new algorithms "and we are going to do the same".

In parallel, Ledru said that the project had started working on rewrites of GNU Findutils and GNU Diffutils. Those have been less of a focus, he said, but people have been doing great work on implementing those and improving their performance on the bfs test suite used to test find. Now, Ledru said, "I'd like to do the same with most of the essential packages on Debian and Ubuntu". There isn't a better time than now to start that task, he said. What are the essential packages? He displayed the command he uses to find them:

    $ dpkg-query -Wf '${Package;-40}${Essential}\n' | grep yes$

The list includes procps (/proc file system utilities), util-linux (miscellaneous system utilities), acl (access-control-list utilities), bsdutils (standard BSD utilities) and several others.

Ledru then published a blog post from the stage formally announcing the plan to rewrite other parts of the modern Linux stack in Rust. He said that many people were already contributing to the project, and uutils was using the work that has already been done for the coreutils in Rust.

There are many, many functions we have in the coreutils that can be used for other programs as part of that world. So when you need to mount a file system or when you need to look at permissions, we already have all those functions.

He said that the project has a lot of low-hanging fruit and good first bugs for contributors to get started if they would like to learn Rust. "It's the way that I learned, and that's why we have so many contributors". Projects like rewriting and reinventing the wheel might sound crazy, he said, but he thought it would work because there is an appetite from the community.

As I was saying earlier, I'm getting older. We all do, but the new generation is not going to want to do C. And paving the way to do Rust is also a good opportunity for them to be involved in the open-source ecosystem.

The old tools are still using old build pipeline systems, some do not have CI. They are still using mailing lists to send patches. "I apologize for the Linux developers here, I still think that using mailing lists to do a patch review sucks". With GitLab and GitHub, he said, there was an opportunity to have a proper pipeline with CI, quality tools, and so on, which is one of the advantages that uutils has over the old projects. "To be clear, that works for them, so I'm happy, but we can do better as a community."

Finally, he wanted to mention again that uutils is not a Mozilla project and has no company behind it, no interest from big tech. "I'm doing it as a passion because I care about Debian and Linux in general, it's really a community effort, there is no structure behind it". He then opened the floor for questions with a little bit of time remaining.

Questions

The first question was about portability, and whether Ledru was tracking the other efforts toward additional Rust compilers to extend uutils coverage. Ledru said "we will when they are ready". Another audience member asked if there were any plans to develop a shell, like Zsh. Ledru said no, he wanted to replace existing utilities written in C, not to rewrite a shell, that there was "no need in that space".

The question that followed was about Ledru's stance on utilities that are written in Rust that are part of the "rewrite in Rust trend" like ripgrep, bat, and others. He said that they are amazing projects, and he uses ripgrep daily, but they are not drop-in replacements.

The final question was about packaging. A member of the audience observed that it is really hard to package Rust packages due to dependency management, and wanted to know what Ledru's experience with that was and how to work around it in future. Ledru agreed that it was a big deal and that Rust was "quite hard to package", but he thought it was going to stabilize.

[I was unable to attend FOSDEM in person, but watched the talk as it live-streamed on Saturday. Many thanks to the video team for their work in live-streaming all FOSDEM sessions.]

Comments (174 posted)

The selfish contributor revisited

Open source is often described as a "gift economy"—an ecosystem where contributors are motivated by a desire to make the world a better place. That is, sometimes, true. However, James Bottomley used his main track slot at FOSDEM 2025, on February 1, to make the case that it is better to bank on the selfish motivations of individuals to drive community success than to rely on their altruism.

His talk was titled "The Selfish Contributor Revisited". It was something of a follow-up to "The Selfish Contributor Explained", a presentation that Bottomley gave at FOSDEM in the "Community and Ethics" devroom in 2020. That talk focused on the selfish interests of corporations that contribute to open source. This time, he came to discuss the motivations of individual contributors instead.

Bottomley began with the disclaimer that, while Microsoft is his employer, the opinions expressed in his talk were his and his only. That said, his day job of managing engineers provides a certain insight into their motivations and how to persuade them to do what the corporation considers "productive work". Productive work, in this case, being defined by what VPs and CEOs want done.

Hunting truffles and herding cats

If asked, he said, engineers will claim that anything they're doing—including staring out the window while thinking—is productive work. Engineers work best on things they like doing. Trying to make them do other things is like herding cats. Companies, of course, want those cats herded. He opined that open source manages to move more rapidly than industry because its contributors are motivated, and that industry has been "jealous of what we were doing". Every engineer has a slightly different motivation. Some engineers, he joked, "even work on Rust".

He then turned to the topic of truffles. Bottomley enjoys cooking with truffles, and when he lived in the Pacific Northwest he observed a small industry based on finding and selling truffles to farmer's markets. People would seek them out with truffle-sniffing dogs, because they are good at finding them, and then sell the truffles to the markets and such.

In Europe, though, he said that truffle hunting was done with pigs. There are arguments about which are better, pigs or dogs, but Bottomley noted that pigs were more motivated than dogs because the pigs' motives were selfish. Dogs hunt truffles, he said, because they want to please their humans. Pigs hunt truffles because they want the truffles, and if they find a patch they have to be controlled to ensure they don't grub up the ground and eat the truffles. Harnessing selfishness, he said, could lead to better yield but has to be done correctly. And the same is true for open source.

People participating in open source typically do so, as the saying goes, to scratch their own itch. They form communities of volunteers that want to see something done for their own reasons. "Open source wins because contributors self-select for their interests." If that is so, then do open-source projects still need a cat herder? "Sometimes no, but mostly yes", he said.

When Linus Torvalds sent out the email saying he was working on Linux, he did not ask for contributions. Bottomley said that Torvalds thought he was going to write all the code, but was asking for suggestions and users. Within a year, contributions were flooding in and Linux was a self-hosting operating system. Not yet a good one, but on its way, and Torvalds was now an open-source cat herder. He still tries to write patches, Bottomley said, "because they certify you as dead" if a maintainer ceases to write code—but he is now primarily an open-source cat herder (or benevolent dictator, if one prefers that term).

Common goals, competing interests

Linux shows the power of natural community formation. It is a human thing, Bottomley said, not unique to open source—but it is a great strength of open source that it harnesses this tendency. Communities are formed when common goals emerge from competing interests. The competing interests stimulate ideas, but getting things done relies on negotiation and compromise between participants. "If everybody has a fixed view and is clashing, nothing gets done." There has to be a willingness to work together.

An equitable power balance is important for the ability to work together, Bottomley said. While human societies are naturally hierarchical, successful communities don't work that way. "It's a key requirement for people to be empowered, and if everybody is empowered, nobody is at the top."

Communities also need a diversity of viewpoints. Bottomley said that there is often a strong link between having people of different backgrounds and having diversity of thought in a community. And the more diverse the inputs, the more thoroughly a problem space is explored, which can lead to the best and most durable solutions.

However, having diverse inputs is not sufficient in and of itself. "Community must also reconcile them into code". That means having to reconcile viewpoints, or "you'll just disagree and have a schism". The process of bringing those viewpoints together is assisted by some kind of leader who can help people see how to combine their inputs into the best solution.

Doing all this correctly, Bottomley said, "results in enormous synergy" and can double or triple the efficiency of proprietary industry. Things move much faster in open source than traditional programming, which is "why industry is so keen to embrace the model". The key to that is to understand the motivations of contributors, he said.

People are a problem

Bringing a community together does not ensure success. Bottomley said that there are many things that can drag down a community and make it less successful. For example, there is the problem of "fixed idea dominance" where the community leader "has a fixed idea of what you do to solve whatever you're trying to do". That shackles the community to the view of a single person. Often the participants in a project don't notice this until a project becomes big enough for it to be a problem—or it fails to attract enough contributors and dies naturally.

Another problem is the inability to reconcile inputs, he said. Negotiation between contributors doesn't produce results, it simply goes round and round without success. Bottomley said that this was often caused by ineffective leadership.

Having a diverse community with diverse views is important, but won't lead to success if those participants have entrenched views they are unwilling to budge from. Bottomley said that a community must be able to come to agreement, and that means participants need to be willing to embrace other ideas. That is difficult if participants have a win/lose mentality. Some people define problems in terms of opposites. "If my opponent wins anything, I have lost." People have to be able to make accommodations for the views of others.

The problem, really, is people. Working with others, that is to say negotiating, is a learned behavior. But many people don't learn to negotiate, they learn to manipulate. This is not all bad, he said, politics is "the art of manipulation by incentive". Community politicians see achievable compromises. A leader, Bottomley said, can see and drive compromises that others can't see. "Manipulation by incentive is a quality a leader has".

Passive aggression

But, some people try to manipulate by other means—through express or implied threats. "Look at a bad community, you'll often find passive aggression as the basis". Codes of conduct, he said, don't help much with passive aggression. They are effective against direct threats. "If you push this patch, I'll break your kneecaps", is something a code of conduct can deal with. But they don't help much with passive aggressive behavior.

He illustrated this with an example that may be familiar to many who have participated in open-source communities. A contributor says, "a number of us have reviewed this patch, and as a group urge you to withdraw it or there would be fracture and dissent" because it's not very good or some other reason. Typically the technical reasons are not well defined. This contains subtle threats and tactics to negatively manipulate another contributor.

First, Bottomley noted, it implies that there is a group behind the speaker. "[They] always imply they're on behalf of a group". Everybody is ganging up on the person trying to push the patch, or so the contributor is supposed to think. The implication that the patch does not meet the expectations of a group also is an attempt to lower the contributor's self-esteem—to make them more likely to be manipulated. Finally, if the contributor persists, there is the implied threat that "the rest of the community is going to take umbrage". It doesn't say what the community might do, but the threat is out there.

Put together, this works as a passive-aggressive tactic to try to stop a patch, and Bottomley noted "nothing that a code of conduct says will help me combat this".

Insiders versus outsiders

Another problem communities may suffer from is the divide between insiders and outsiders, which implicitly discriminates against drive-by contributors. Bottomley said that he is a "fairly prolific" drive-by contributor because he uses open source for almost everything he does, and submits patches to other projects often. The drive-by contributors are often users with "a valid point" and patches that are not that bad. And, he noted, outsiders may become insiders if contributing is easy. "Originally they only wanted to get one thing done, but it was so easy they stick around". But, even if they don't, it's not a bad thing.

He cautioned against the "origin trap"—that is, the trap of inquiring into a patch's origins as a basis for acceptance rather than the quality of the patch itself, because it is a slippery slope. Using an extreme example, he said that it might seem fine to not accept patches from people convicted of crimes. But then, what about those who are merely indicted, or just suspected? Then, he said, the community will soon be refusing patches from people "who disagree with a key goal". Or have the wrong political affiliation. Don't do this, he said, but if a community must discriminate in some way "use an objective, externally-vetted standard". Otherwise, "when you get to the bottom [of this slippery slope] you'll see your community destroyed".

Persuading the pig

It is possible to become a good leader. That is a process, Bottomley said, that involves learning to see all sides of an argument. The problem is that most people think they already can do this, but actually very few do. There are, however, exercises that he recommended to develop this skill.

One thing to do is to learn to debate the contrary position. He recommended, "in an argument with a good friend", to stop and try to argue for the other side. Reverse the viewpoints to become an advocate for the opposite position. This only works, he cautioned, with good friends, though. It is important to learn how to argue well for a cause you do not support, and develop the ability to see all angles of a problem. This is most difficult when arguing against yourself, Bottomley said. "Try to think of reasons not to accept your own patch, helping you to see other sides of the argument."

Good leaders can learn this ability to see the motivations of others, and to promote negotiation and compromise in the community. They make suggestions to give others effective negotiating positions, rather than making decisions. "A community that comes to its own conclusions is healthier". It is even, he said, possible to learn incentive-based manipulation. "It's now what I get paid for".

Leaders also have to learn to deal with passive-aggressive people in email. He reminded the audience that leaders in open-source communities have no real power, "you have to persuade the pig". Bottomley suggested that people should respond only to the technical content in that kind of email and ignore the passive aggression. People employing passive aggression seek responses to their aggression and he said the victim should "never, ever" respond to the passive-aggressive part of the message. Nothing is going to nudge them toward more effective engagement. It is, of course, easier to say that people should not respond to passive aggression than it is to avoid doing so. "There will be times when you don't achieve this advice, but that shouldn't stop you from trying the next time".

Bottomley coached the audience to pay attention to the economics of engaging with the passive-aggressive types. If the cost balance is positive, he said, "accept it". But if the cost of engaging is more than the patch is worth, think about ways to ease the person out of the community. He also suggested that leaders in a community could help to deflect passive-aggressive tactics by intervening. "I pipe up and say 'actually, I think it's pretty good'".

Finally, he said, "be willing to be wrong and admit when you are."

With the session nearly at an end, and no time for questions, he noted that he had created the slides with Impress.js, which in turn "makes me a web developer" too. The slides will eventually be published on his site, hansenpartnership.com, but he said the site was currently down due to the wildfires in Los Angeles. The video of the talk is not yet available on the FOSDEM site, but should appear within a few days or weeks.

[I was unable to attend FOSDEM in person, but watched the talk as it live-streamed on Saturday. Many thanks to the video team for their work in live-streaming all FOSDEM sessions.]

Comments (37 posted)

Milliwatt machine learning with emlearn

While large language models and the expensive hardware they require are all the rage now, other areas of artificial intelligence work within much more constrained hardware environments. At FOSDEM 2025, Jon Nordby presented his open-source machine-learning inference engine for microcontrollers, named emlearn. The project also boasts bindings for MicroPython, thus making machine-learning applications even more accessible.

Nordby's talk, "Milliwatt sized Machine Learning on microcontrollers with emlearn", was part of the Low-level AI Engineering and Hacking developer room (devroom) at FOSDEM. Nordby is the CTO and co-founder of Soundsensing, a Norwegian company that monitors ventilation systems and other technical infrastructure for its clients. The company uses machine learning to detect anomalies in data from sound and vibration sensors, thereby providing early warning of potential failures.

Tiny chips, massive scale

The niche field of running machine learning on microcontrollers is also known as TinyML or, more recently, Edge AI. It's primarily used for analyzing sensor data. The idea involves a microcontroller processing data from connected sensors, running a machine-learning model to analyze the data, and then extracting useful information from it. The results of this analysis can then be sent over the network or used to control a locally connected motor if the microcontroller is part of a robotic system.

Although machine-learning models could be run on much more powerful hardware in the cloud, Nordby explained that the TinyML approach has some key benefits. It allows for the creation of standalone systems, without the need of a network connection. Another benefit is low latency: sending data over a network to a server and then retrieving the results introduces latency that should be avoided in many industrial applications. Furthermore, transmitting raw sensor data to a server instead of just the useful information requires more power. With TinyML, "sensors can run for multiple years on a single battery". Also, by not transferring all of the potentially sensitive data, a TinyML application can be "privacy-compatible". And finally, AI on microcontrollers is cost-effective, which allows using it on a massive scale.

TinyML is already used in many areas, Nordby noted, citing examples such as keyword spotting in smart speakers to recognize phrases like "Hey Google" or "Hey Siri", and sleep trackers that monitor sleep quality. The technology is also used for tracking animal health and, in the case of Soundsensing, the health of machines.

To give an idea about the constraints we're talking about, a microcontroller often has just a thousandth or even less of the CPU power and memory of a smartphone. RAM could be up to 1MB and program space up to 10MB. Over 20 billion microcontrollers are shipped each year, and they're becoming increasingly available for hobbyists. Nordby highlighted the Arduino Uno, ESP32, RP2040, and RP2350 as available hardware platforms, as well as the Arduino IDE and MicroPython as software platforms for hobbyists.

Real-world use cases

Nordby started the emlearn project in 2018, as part of his Master's degree in Data Science at the Norwegian University of Life Sciences, and published it under an MIT license. Emlearn fits in the regular workflow of machine-learning researchers. They can just create a model and train it on a data set using standard Python libraries such as scikit-learn or Keras on their PC. Emlearn comes into play only after they have trained a model.

Emlearn is used to convert the trained model from Python objects to a C data structure and save it to a C header file that can then be included in the microcontroller's code. This header file has portable C99 code, without dynamic allocations and using only integer or fixed-point arithmetic, making it a fit for many embedded devices and software frameworks. This includes Arduino, Zephyr, and FreeRTOS. The C code generated by emlearn can also be used from other programming languages, as long as they support C bindings.

Not all models from scikit-learn or Keras are supported, but emlearn is able to convert models for the most common tasks relevant for analyzing sensor data: classification, regression, and anomaly detection. So emlearn can't be used to run large language models on microcontrollers, but Nordby summarized that "you have access to decision trees, random forests, k-nearest neighbors, and some smaller neural networks". Of course, the trained model must fit into the microcontroller's RAM and storage constraints.

Nordby emphasized that emlearn is not just a toy project: it has many real-world uses around the globe and it's referenced in at least 40 scientific publications. For example, Abhishek Damle used the project for health monitoring of cattle, as described in his Master's thesis. He trained a decision tree classifier on accelerometer data from a sensor on the cow's head. The device uses this model for sending information to the farmer about the animal's current activity: lying, walking, standing, grazing, or ruminating. This allows the farmer to detect abnormal behavior that could indicate health issues or other problems, for example when a cow is not eating for a long time. Running the classifier model on-device with emlearn consumed less than 1mW, and the TinyML approach used 50 times less power than sending the raw data to the cloud for analysis, significantly improving battery life.

Samsung Research even trained a model with emlearn for monitoring people with respiratory health problems using the motion and acoustic sensors in earbuds. Because the TinyML approach has a low power consumption, this "can be used as a practical solution worn every day, not just in a clinical context".

Last year, Nordby undertook the challenge of building a sensor board capable of running a useful machine-learning model at a total component cost of less than one US dollar. His "1 dollar TinyML" project is based on a Puya PY32F003 microcontroller featuring a 32-bit Arm Cortex-M0 core running at 32MHz, with 8KB RAM and 64KB flash storage. While emlearn is often applied on slightly more powerful microcontrollers, Nordby succeeded in creating something useful even within those constraints, analyzing audio from a microphone or movements from an accelerometer.

Python all the way down

To make emlearn more accessible, Nordby ported it to MicroPython. Since many who are involved in machine learning primarily know Python, they can now develop TinyML solutions without needing to learn C. MicroPython implements a subset of Python for microcontrollers with 16KB RAM and more. We looked at in 2023 and Nordby gave another talk at FOSDEM 2025 on it, as well.

One of MicroPython's appealing features is that it can be extended with modules written in C, which offer greater performance than plain MicroPython. The emlearn-micropython project offers various models from emlearn as a prebuilt .mpy module with native machine code for multiple microcontrollers. Users don't need to set up a C toolchain or SDK, but can easily install the module onto their microcontroller's flash storage using MicroPython's "mpremote mip install" command and copy the model weights from their trained model to the device as a CSV file.

The MicroPython code on the device then creates the emlearn model from the .mpy file and loads the model weights from the CSV file. With the model ready, it can then read sensor data, preprocess the data, and run the model to return a prediction or classification.

As an example, Nordby developed an automatic toothbrush timer. He mounted M5Stack's M5StickC PLUS 2, which has an ESP32 microcontroller and an accelerometer, onto a 3D-printed case together with a toothbrush. By collecting and labeling one hour of accelerometer data while brushing his teeth and engaging in other activities, he trained a model to recognize these activities. In 500 lines of MicroPython code, his solution counts the number of seconds during which the device detects brushing activity. Once the user completes two minutes of brushing, the device's buzzer plays a song.

Conclusion

Emlearn shows how, even under the severe hardware limitations of embedded devices, one can develop practical AI applications. A couple of kilobytes of RAM, a couple of milliwatts of power, and a couple of dollars are enough for applications such as human or animal activity recognition. Even without knowing C, the necessary performance is possible thanks to emlearn-micropython's native module. It will be interesting to see what applications people will build with emlearn for sensor devices.

[While I was unable to attend FOSDEM in person, I watched Nordby's talks as they live-streamed on Saturday and Sunday.]

Comments (1 posted)