Resources for learning Rust for kernel development

Dirk Behme led a second session, back-to-back with his session on error handling at Kangrejos 2024, discussing providing better guidance for users of the kernel's Rust abstractions. Just after that, Carlos Bilbao and Miguel Ojeda had their own time slot dedicated to collecting resources that could be of use to someone trying to come up to speed on kernel development in Rust. The attendees provided a lot of guidance in both sessions, and discussed what they could do to make things easier for people coming from non-Rust backgrounds.

He opened the session by noting that "most of you are special" — that the attendees were, by and large, already knowledgeable about Rust. They have written drivers, seen that abstractions were missing, and written the abstractions as well. So nearly everyone in the room was an expert, who knew all of the details of how Rust works in the Linux kernel. Behme isn't a computer-science person, though. His background is in electrical engineering.

He put up a picture of Linux Device Drivers, 3rd edition, asking: does there also need to be a book about Rust kernel abstractions? Rust is said to have a steep learning curve — and Rust-for-Linux goes even further, since it involves writing low-level code in a particular style and the kernel is always under heavy development.

To illustrate his point, Behme put up some examples of beginners asking about writing kernel Rust. One person was having trouble writing a module. Alice Ryhl had replied to them that the abstraction they were using had changed its API, and explained how to adapt their module. This isn't an uncommon problem — others have also reported needing time to adapt, he said. Behme himself took some time to figure out the devicetree abstraction — about a week. He said that this wasn't a complaint, just an example of how learning the necessary prerequisites can be hard, and how the project could have better learning materials.

Andreas Hindborg said that when an abstraction goes into a kernel tree, the requirement is for there to be a user of that abstraction — so there should be an example right there in the tree. In practice, he said, the abstractions that do go in also tend to have good examples in the documentation. So the project certainly intends for the type of learning material Behme was asking for to exist.

Miguel Ojeda pointed out that there may be books about Linux device drivers, but that it's still early days for Rust in the kernel. It took time for those books to be written, he said. "We were thinking about writing a book," he continued, but it was just too much work right now.

Behme replied that he did think that the project was doing a good job with documentation, but that it was not enough. At his work, he had asked about whether they could start using Rust-for-Linux soon; his manager said no, not for technical reasons, but for social reasons — the learning curve from C to Rust was too steep for most of the engineers at his company.

One audience member asked whether that difficult curve was due to the language, or the Rust-for-Linux project. Behme said that the main concern was Rust, but that the project adds complexity on top of that. He gave the example of looking at some Rust code and seeing that it called spin_lock(), but "why the hell was there no unlock"? (Answer: the spin_lock() Rust abstraction returns a guard object that automatically releases the lock when it is dropped — either explicitly by the programmer, or implicitly at the end of the function.) There are examples for these things, but the underlying reasoning is different from C, and that takes time to learn.

Paul McKenney noted that modern kernel C code actually has similar lock guards now, and that maybe this would make Rust's use of lock-guard objects less counterintuitive. Ryhl wondered whether having translations for common conventions between languages would be helpful.

Hindborg agreed that it takes time to learn a language, and that Rust is fairly difficult to learn as imperative languages go. But "you need to invest time in learning that; it doesn't come for free". Once you have put the time in, there are substantial benefits. He suggested that Behme tell his manager that, noting that Google claimed a 3x productivity increase with Rust. He also said that while Rust-for-Linux does add some additional details on top of plain Rust, that's nothing new for the kernel — C in the kernel is pretty different from C in user space.

Ryhl asked what it takes to teach other developers to write Rust, in the other attendees' experience. She noted that there would be an upcoming talk at RustConf on that topic, actually. Another attendee pointed out that Rust-for-Linux patches go via the mailing lists, just like any kernel patch — so the documentation that justifies or explains a change is often there. Behme asked if it would be possible to get something like a set of release notes for each kernel, talking about what changed.

Hindborg replied that all of the changes are there in Git. Ojeda suggested that when Behme saw a change to an API, he should go look at the corresponding commit.

Greg Kroah-Hartman pointed out that the kernel's C developers don't provide internal kernel-change information — so why should the Rust developers do that? On a related note, he advised against using Linux Device Drivers, because it is now seriously out of date. Ojeda agreed, noting that the changes Behme had highlighted were all to internal Rust APIs — and just like other kernel interfaces, there is no guarantee that they can't change at any time.

Richard Weinberger thought Behme had a good point, however — he noted that most people writing device drivers are electrical engineers, not computer scientists. From that point of view, Rust looks "outlandish and hostile". If you know OCaml and Haskell, Rust looks awesome, he said. The Rust-for-Linux developers should be careful not to assume that kernel hackers who only know C have the same positive impression.

Hindborg replied that he understood Weinberger's point, but that he was himself an electrical engineer who learned C and then Rust. It's not impossible, he said, and you can expect people to learn new tools.

Yes, but you need to give them motivation to do so, Weinberger responded. Benno Lossin said that as a writer of Rust documentation, it's often hard to know what a beginner won't understand. If you're coming at it from the Rust side, the reason that there's no corresponding unlock() in the code is pretty clear. We need to listen to the people coming from the kernel side who have problems in order to improve our documentation, he said. He asked Behme to write down some of the problems he had encountered, so they could turn it into some documentation. Behme agreed.

Lossin also agreed that the changes to the Rust APIs were frustrating, but that they could become better over time — there's a preexisting plan to split up some of the functionality of the kernel crate into smaller crates that should therefore see less frequent changes. He said that there are so many people working on the kernel crate right now that it's hard for anyone to track all of the changes.

Gary Guo thought that Rust would actually shine for developers writing device drivers. It's unrealistic to expect every engineer to understand C's many sharp edges, he said. In Rust, the APIs are not all there yet, but it's possible that they could only ever need to write safe Rust code. So there's actually value in getting less experienced engineers to write Rust — the compiler will help them write fewer bugs.

Simona Vetter said that, in her experience, the average kernel C developer doesn't understand C either. There are, in theory, five people who could write a bug-free driver, and in practice zero. It's practically impossible to write bug-free kernel code, she stated.

Behme replied that, in industry, you have to write drivers. So getting acceptable drivers out of real engineers is a requirement. Vetter thought that Rust could actually be helpful with that — her hope is that new engineers could just type out "random code" in Rust and get a correct driver out, which would never happen in C.

Hindborg thought that was an interesting observation. He predicted that lots of people would be angry if their employer told them to write Rust, because nobody likes being told off by the compiler. But despite that, when the necessary libraries are in place, perhaps we can just never compile a buggy driver.

Ryhl noted that she has seen other people contribute to her driver without adding any abstractions. So in at least one domain, things have gotten to the point that things are mostly stable.

Collecting resources

At that point, it was officially time to go to the next session. But, luckily, the next session was scheduled to be a roundup of different educational materials, with the aim of producing a recommended list for learning Rust in the kernel.

Ojeda asked that people list out the resources they found most helpful while learning rust. For his own part, he found an online book from Brown University's Cognitive Engineering Lab with an interactive borrow-checker helpful.

Guo joked that the best way to learn Rust was to learn C++, hate it, and then learn Rust. Lots of concepts map, he said, but Rust is much better. Adrian Taylor suggested the New Rustacean podcast. He thought that audio was a weird way to learn a programming language, but he liked it in this case. He also suggested a series of articles, "Learn Rust the Dangerous Way", which shows the incremental conversion of a C program to Rust.

Kroah-Hartman said that the Linux Foundation has a free online course for learning Rust. Hindborg said that Google had a free five-day course as well, "Comprehensive Rust".

Lossin gave a more general recommendation — read blogs. There are lots of good posts on advanced topics, he said. He particularly liked Amos Wenger's explanation of Pin. Ojeda suggested the Master's thesis "You Can't Spell Trust Without Rust" by Aria Desires as a good resource for advanced topics as well. She also wrote "Learn Rust With Entirely Too Many Linked Lists", which nobody recommended at the time, but that is also intended as an introduction to Rust for programmers with existing C experience.

With the resources collected, the discussion turned to what to do with them. Bilbao said that the project should make a distinction between people just starting out, and people who have been writing Rust in Linux for some time — they have different needs. He suggested using the Rust-for-Linux web site as a central location for hosting good blog posts, but also thought that it was important that the project be "serious" about ensuring things are well documented.

Lossin noted that there is already a linter rule requiring that all public items (functions and types) be documented. There was a brief discussion of current kernel conventions for documenting C code.

Vetter ended up pointing out one problem with kernel-doc, the tool that checks whether C code is documented. It doesn't complain when there are no comments, but it does complain when there is one comment and some still missing. This makes people not want to add documentation where it doesn't already exist. Rust is ahead, she said, because just requiring that documentation exists, even if people don't put effort into it, makes it easier to improve later. She pointed out that the overlap between people who are good at writing complicated code, and people who are good at writing documentation is often not big — so it's okay to encourage people to collaborate.

In all, there was a clear consensus that the Rust-for-Linux project could make it easier for people to get up to speed with the knowledge necessary to write Rust in the kernel. So the project will continue to encourage good documentation standards, centralize learning resources, and work with other kernel developers who bring up pain points to figure out what else needs to be covered.

[ Thanks to the Linux Foundation, LWN's travel sponsor, for supporting our coverage of Kangrejos. ]

    drop(guard);

    struct Node {
        some_unprotected_data: usize,
        protected_data: Mutex<NodeInner>,
    }
    struct NodeInner {
        more_protected_data: usize,
        children: Vec<Node>,
    }