A "fireside" chat

In something of an Open Source Summit tradition, Linus Torvalds and Dirk Hohndel sit down for a discussion on various topics related to open source and, of course, the Linux kernel. Open Source Summit North America (OSSNA) 2022 in Austin, Texas was no exception, as they reprised their keynote on the first day of the conference. The headline-grabbing part of the chat was Torvalds's declaration that Rust for Linux might get merged as soon as the next merge window, which opens in just a few weeks, but there was plenty more of interest there.

Hohndel introduced himself as the chief open source officer at the Cardano Foundation; he is working to help foster an open-source ecosystem around the foundation's blockchain technology. Torvalds said that these "fireside chats" are held because of his wishes; "I do software", not public speaking, he said, so the format makes it easier for him. He effectively has outsourced figuring out what people are interested in hearing about to Hohndel; with a grin, Torvalds said, "if he asks bad questions, it's not my fault".

Hohndel said that it was "super exciting" to see so many people in the room for the keynote; there were around 1200 in-person attendees for OSSNA. Things have been quite different over the last few years and he asked Torvalds how all of that had affected him and the kernel. "This whole COVID mess affected the kernel almost not at all", Torvalds said; certainly individuals were affected in various negative ways, but from a development standpoint, the kernel project continued apace—or even sped up during the lockdowns. In part, that is because kernel development has always been a distributed project where many participants were already working from home.

Torvalds noted that he has been working from home for 20 years now. Hohndel said that it was interesting in the early days of the pandemic when companies were trying to figure out how to work remotely and rely on email more, which is something that the kernel community figured out long ago. "To us, it didn't feel as new."

Hohndel asked whether Torvalds had noticed any changes in the kernel process from the events of the last few years. Torvalds spent a little bit of time describing the normal nine- to ten-week kernel development cycle that will be familiar to LWN readers; it has been the standard for kernel releases for 15 years at this point. It is a calm, staid process, "which is, I think, exactly what you want".

But that "boring and staid development process" does not lead to the same characteristics in the development itself. There are lots of changes going on in the core kernel, "and I'm really happy to see that". One might think that a 30-year-old project would have gotten boring, but he is actively trying to encourage people to "do the exciting stuff". There are new architectures being added, people are trying out new languages for kernel development, and core parts of the kernel are being improved in fundamental ways. "It is not just at the edge that the kernel is growing," which is one of the things that he personally enjoys the most. "We're not a dead project."

Don't break user space

Hohndel then referred to a recent discussion about the limits of the kernel's "don't break user space" promise, especially with regard to BPF programs. More and more things are being implemented in the kernel using BPF rather than, say, system calls, he said; what is the point where "your API ends and crazy space begins?" Torvalds said that "'crazy space' is hopefully all internal" to the kernel for one thing.

But he does not like talking about this in terms of APIs, because people will look at the documentation for an API and argue that if someone is not following what was written, it is their problem, not the kernel's. "I feel that's a complete cop-out and it's just bad policy. Documentation is worthless compared to reality." He did note that his opinion is biased because he "never writes documentation".

Torvalds's rule has never been that the API cannot be broken, but that the kernel "can't break people's loads, you can't break what people do". If someone takes advantage of a bug in the kernel, "that bug is not a bug, it's a feature". The kernel will maintain that feature forever, unless there are "really pressing concerns", almost all of which are security-related, that require it to change. "We will go to insane lengths to actually keep bug-for-bug compatibility."

Torvalds is not just a developer, he is also a user, and he thinks that the most annoying problem that users experience is "doing a software upgrade and things stop working". He cannot change all of the other software projects out there, which generally have different policies, but he has been "very hard-nosed" with kernel policy to try to ensure that programs continue to run after a upgrade. If that does not happen, "you're supposed to scream at us". The kernel has had great success with that policy, he said, and he wished that everyone in the room would push for this in their projects as well.

Hohndel probed a bit deeper, though. For example, BPF programs are loaded from user space, as are kernel modules, including those that are out of tree; would breaking those cross the line, he asked. Torvalds said that he does not consider breaking kernel modules to be breaking user space. Those who are writing their own modules need to update them if the kernel changes; out-of-tree modules are "heavily discouraged" though they sometimes make sense as a step in the development path. He is not a lawyer, so he does not want to even look at the question of when third-party modules may run afoul of the kernel's GPL license; "I don't know when that gray area turns into black."

But for user-space programs "that are part of user workflow", whether it uses BPF or any other kernel feature, it is generally on the kernel developers to ensure that they do not break, Torvalds said. Some people use BPF for low-level tracing, statistics gathering, and debugging, though; when it is used that way, kernel changes may break those programs "and we can't do anything about it". He is talking about users, he said, "not people who are digging around in the kernel".

Rust

Hohndel asked about the status of Rust for the kernel; "I haven't seen any actual patches being merged yet." Torvalds said that the patches are available, but they have not been merged. Part of the issue is that the kernel project has gotten more careful over the years; "we were more freewheeling 30 years ago." But some believe that the kernel has become too risk-averse at this point.

Rust for the kernel has been discussed for years at this point, so "real soon now we will actually have it merged in the kernel—maybe next release". That was met with a round of applause, but Torvalds was quick to caution that "to me it's a trial run". The memory safety offered by Rust means it has real technical benefits. But the kernel tried C++ more than 25 years ago; "we tried it for two weeks and then ... we stopped trying it."

To Torvalds, "Rust is a way to try something new" for the kernel. People have been working hard on it and "I really hope it works out because otherwise they will be bummed". But it will start "very small" and in "very specific parts of the kernel"; "we're not rewriting it all in Rust." Some kernel developers want to do something new and fun; "I think Rust makes a lot of technical sense."

But Rust is a much different language than C, so Hohndel wondered what the challenges would be for kernel maintainers who might start getting code in a language they do not know, or know well. Torvalds does not "see that as a big issue"; there are multiple languages in the kernel today. For example, the kernel makefiles are "makefiles in name only"; they are an "unholy mess of various macros and other helper functions that are really hard to understand". Beyond that, he gets patches with Perl code in them, but does not "even pretend to understand Perl"; in fact, he thinks it is a write-only language. But he generally trusts the maintainers to do the right thing.

It is his long-standing policy to trust maintainers—at least "until they screw up". When that happens, he sometimes responds in an "overly impolite" fashion and he was quite apologetic about that. "It is a personal failing of mine and I mean that very seriously; sometimes I go overboard". Hohndel suggested that perhaps it was "in a loving way", but Torvalds disagreed: "I wish I could say that." He said that he should "preemptively apologize to the Rust people".

He has seen the worries from some developers in the kernel community with regard to not understanding Rust. He thinks it is fine not to understand it; people do not understand the kernel's virtual memory (VM) subsystem, even though it is written in C. "The language is generally not the biggest hindrance to understanding." There will be Rust maintainers just as there are VM maintainers today; Rust is a "small technical change, not a fundamental one".

Hardware security woes

Fixes for various hardware security bugs, especially those related to speculative execution, have been with us for a number of years now, Hohndel said. Some of the "very well-maintained architectures", like x86 and Arm64, have a steady stream of these kinds of problems that need to be addressed in the kernel. There are lots of other architectures in the kernel, however, and he wondered why they did not seem to be affected by that. "Are the others safe or are they just not as well-maintained?" "I think it is a bit of both", Torvalds said.

These kinds of problems go back to Meltdown and Spectre, he said. Those problems made people realize that even if the software is written in a secure manner—which is rare and hard to do—"the hardware sometimes ends up really screwing you over anyway". The kernel is not the only software project affected, browser and virtualization developers have also been affected. "It's very frustrating" when the hardware cannot be trusted and "you have to do a lot of extra work to fix hardware bugs". The good news is that the kinds of vulnerabilities being found are getting more esoteric and are affecting fewer people.

The architectures affected by these kinds of bugs are mostly just x86, Arm, PowerPC, and s390, he said, in part because there are more Linux systems running on those processors. But the kernel supports 15 other architectures "that never see these kinds of security problems". Sometimes that is because they are embedded, in-order processors that do little or no speculation. Other times, there is less scrutiny of the architectures and there are fewer people "digging into what could go wrong" on those systems.

He has gotten used to these hardware security problems; "I used to be very very frustrated about this." After five years, "you kind of grow inured to the pain". Hohndel asked if the hardware vendors had gotten better at working with the kernel community on handling and fixing these kinds of bugs. Torvalds said that they had; much of the early pain was due to something of a culture clash between the kernel community and the hardware vendors.

Normally, kernel security fixes are disclosed within seven days, he said; the patches are posted publicly, though the implications may not be disclosed at that point. The hardware companies want to have security embargoes of, say, 12 months, which makes it difficult. He does not want to have to try to remember which bugs have not been disclosed, so that they should not be discussed, and so on.

The whole process goes against the way things normally work in open source. Another problem is that the people who can fix the bug in question may be people who "you can't talk to". So a small team has to be gathered to fix the bug, without talking to all of the relevant people, without using all of the public testing infrastructure, and so on. The next two weeks after a release of a bug fix of this sort are always spent fixing up the details that slipped through the cracks because the normal process could not be followed.

Moving onto the wider question of security for our systems, Torvalds emphasized that bugs need to be accepted. "Bugs will happen, if they don't happen in hardware, they will happen in software, and if they don't happen in your software, they will happen in somebody else's software." Some of those bugs will be security bugs; the "only way to try to do security right" is to have multiple layers of defense.

The kernel provides a layer of its own, he said, but it has multiple security layers within it as well. There are various hardening efforts going on to try to reduce the scope of bugs when they occur. The compilers are being used to insert various kinds of checks into the kernel, some of which are expensive, and some of which are assisted by various hardware features for things like pointer authentication. But defensive programming at all layers of today's software stacks is going to be needed. "Anybody who thinks you can get 100% security is living in some dream world that is not this reality," Torvalds said.

Fun stuff

"I don't want to make this a security talk, I want to talk about stuff that's fun", Hohndel said, to some laughter in the audience. He asked Torvalds about what he did outside of software development, though Torvalds disclaimed any knowledge of that at first. "Is there life outside of the screen in front of you?"

He did admit to working on a hardware project during the pandemic. The problem, he found, was one of motivation. He really likes "doing social software", though not in the social networking sense of that term. He likes talking with people during the development—over email, not face-to-face, he emphasized. With a laugh, Hohndel noted that there were a lot of caveats in all of that.

Torvalds said that he designed a circuit board with a microcontroller on it, which worked correctly when it came back from being manufactured in China. But at that point, he thought "now what?" It was an interesting learning experience, but it turns out he really did not want to create hardware.

Beyond that, Torvalds likes to scuba dive. He started another open-source project, Subsurface, to log his dives, because the dive computer makers did not consider Linux a big target for their software. Subsurface is now maintained by Hohndel, who thanked him for the plug.

"Linux is my baby", Torvalds said, but he found it amusing that when his oldest daughter went to college (for computer science—"I didn't push her"), he was much better known for Git in the CS department. "I only did Git for six months"; he will take the credit, but in reality Junio Hamano has done an excellent job of maintaining the project. If Git-using attendees ever see Hamano "buy him a beer or something". Torvalds said: "My name comes up much too often when it comes to Git." With that, the keynote session ran out of time; they plan to be back on the stage in Dublin for OSS Europe (OSSEU) in September.

[I would like to thank LWN subscribers for supporting my travel to Austin for OSSNA.]