LWN: Comments on "Using Rust for kernel development"

Using Rust for kernel development

NAR — Sun, 10 Oct 2021 14:42:50 +0000

In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.

Interesting. I thought it's quite common - have the frontend in Java or Javascript, the backend in C++ or Elixir or whatever, throw in some SQL for the stored procedures, some Python for test glue, maybe a Perl binding... Even if there are separate teams working on the frontend and backend, they might still look into each others code from time to time... I didn't need to be a Python expert to add a testcase or a Javascript expert to figure out the parameters passed to the backend. Granted, there are few projects needing the level of C expertise as the Linux kernel, so maybe being an expert in two languages might be harder - but I don't think (maybe C++ is the big exception, that's a horrible complicated language).

Using Rust for kernel development

piexil — Thu, 30 Sep 2021 21:53:50 +0000

For real, it was completely unnecessary, and it's so smug. People also already do things similar to fixing transpiled code in their freetime. (reverse engineering decompiled assembly to c like the super mario 64 decomp comes to mind)

Using Rust for kernel development

mpr22 — Thu, 30 Sep 2021 00:06:42 +0000

(And, of course, it turns the rest into process-local data structure corruption. Which could still be disastrous when the program has control of a DMA-capable peripheral device and your IOMMU capabilities are insufficient...)

Using Rust for kernel development

mpr22 — Thu, 30 Sep 2021 00:03:52 +0000

Writing in Rust turns many kinds of memory safety violation into compile-time errors, meaning that you have to fix them before you can start testing the code.

Running in user-space turns some subset of memory safety violations, which partially overlap with those that Rust turns into compile-time errors, into SIGSEGVs instead of kernel data structure corruption.

Using Rust for kernel development

MrWim — Wed, 29 Sep 2021 23:47:39 +0000

In case you're not already aware there's the c2rust project: https://c2rust.com/ . The idea was that it does an automatic conversion for you to unsafe rust and then provides tools to help with conversion to idiomatic rust, and to help confirm that the conversion was faithful.

I can thoroughly recommend the blog series on the same: https://immunant.com/blog/ and the wiki contains a discussion of some of the limitations of the approach: https://github.com/immunant/c2rust/wiki/Known-Limitations... . It's sufficiently advanced to allow transpiling Quake 3 with only minor changes required to the C source.

From looking at the github history it seems like the project has slowed over the last year or two. I don't know of its current state.

(This isn't an endorsement of the approach, just a pointer to it)

Using Rust for kernel development

lwnli — Wed, 29 Sep 2021 21:42:24 +0000

read somewhere else. wonder if writing it in user space gives the same benefit as writing it in Rust.

Using Rust for kernel development

excors — Wed, 29 Sep 2021 10:41:14 +0000

> I don't know if there is any transpilation in the "original" Mozilla, but the HTML (5, at the time) parser was converted from Java to C++ - https://johnresig.com/blog/html-5-parsing/

The HTML5 parser is a special case since it's a large amount of mostly very repetitive code, defining a complex state machine to turn characters into tokens and another to turn tokens into a DOM tree. (There's enough special cases and weird rules, for backward compatibility reasons, that it's not very practical to implement the state machines declaratively - it's easier to write as imperative code. Plus the code closely mirrors the structure of the HTML5 specification, which makes it easier to verify that the spec and implementation are consistent.)

Since it's so repetitive, it's relatively easy to mechanically translate between languages. And I think in this case the transpiler was custom-written for the HTML5 parser, and the Java version of the code was modified to be easy to transpile. E.g. compare "startTag" in Java (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...) and C++ (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...): the Java has some explicit "// [NOCPP[" markers for sections to omit from the C++, function calls like err() are deleted from the C++ (because the Java version was meant for use in a validator, which needs to report parse errors with as much detail as possible, whereas a browser doesn't), enum values are renamed to Mozilla's naming convention, string literals are replaced with global constants, etc, so there's a lot of translation rules specific to this codebase.

This is very different from general-purpose transpiling of arbitrary code that wasn't designed for it.

Using Rust for kernel development

Fowl — Wed, 29 Sep 2021 09:33:04 +0000

I don't know if there is any transpilation in the "original" Mozilla, but the HTML (5, at the time) parser was converted from Java to C++ - https://johnresig.com/blog/html-5-parsing/

Using Rust for kernel development

farnz — Wed, 29 Sep 2021 09:13:45 +0000

Have you got a reference for this? I can trace the core of Firefox's codebase back to Netscape 1.0, which predates Java, so if a transpiler was involved, it was during the rewrite.

Using Rust for kernel development

maxfragg — Wed, 29 Sep 2021 06:29:43 +0000

you might be surprised to learn, that mozillas codebase started with transpiler-output, as what later became the mozilla suit and in turn firefox had a transpiled java core.

Using Rust for kernel development

rahulsundaram — Tue, 28 Sep 2021 21:57:41 +0000

> If drivers are encouraged to be implemented in user space, what benefit does writing drivers in Rust bring about?

The article doesn't talk about user space drivers. Where are you getting that from?

Using Rust for kernel development

lwnli — Tue, 28 Sep 2021 21:03:14 +0000

If drivers are encouraged to be implemented in user space, what benefit does writing drivers in Rust bring about?

Using Rust for kernel development

PaulMcKenney — Tue, 28 Sep 2021 17:24:17 +0000

And that reuse of the backend code is the challenge!

At the informal meeting called out in the article, I committed to write a blog series on the specific topics that I am concerned about. As Dan Frye was wont to say, "It should be good clean fun." ;-)

Using Rust for kernel development

pbonzini — Tue, 28 Sep 2021 16:40:23 +0000

> It might be unfair to say that the Kernel memory model is x86, but there's a heavy overlap on the assumptions Linus Torvalds made 30 years ago.

I think that's quite incorrect since:

* any x86-based memory model would have serious problem on weakly-ordered architectures, and also would have serious problems with optimizations that do not respect the TSO (total store ordering) model that x86 uses.

* I might be wrong on this, but the first architecture for which Linux supported SMP was either SPARC or Alpha, almost certainly not x86

Right now, the Rust memory model _is_ the C/C++ model since it just reuses the same backend code.

Using Rust for kernel development

linusw — Tue, 28 Sep 2021 16:04:44 +0000

As "fake Linus" I can just acknowledge the protocol here: I am happy about Rust when done right (read: with true grit) and don't mind to go first.

However during review I did point out that the security hurdles that Rust can help to solve are not in drivers, and I pointed to filesystems and suggested that the developers look at reimplementing the (then out-of-tree) ksmbd in Rust. ksmbd was then merged unbeknownst to me and my worries confirmed to be true as some out-of-bounds bugs (which are impossible in Rust) surfaced immediately.

Using Rust for kernel development

k3ninho — Tue, 28 Sep 2021 14:46:53 +0000

>Thomas Gleixner ... is worried, though, about "the blank page where the memory model should be". Ojeda ... encouraged Paul McKenney that this is the best time to go to the Rust community and tell them how things could be.

I'm a fan of cross-pollination and have huge respect for Paul McKenney's ability to build consensus on a way forward while also teaching the lessons of the past that went wrong. It might be unfair to say that the Kernel memory model is x86, but there's a heavy overlap on the assumptions Linus Torvalds made 30 years ago. I might put a slots on the bingo card that: other memory models do it better; Rust wants to learn those lessons; mismatch with the LKMM when it comes to the essential bikeshedding.

K3n.

Using Rust for kernel development

jgg — Tue, 28 Sep 2021 13:41:43 +0000

Just making C compile in a C++ compiler is not at all what I mean by upgrade. Proper C++11 code looks and operates very differently from C code. Human led, machine assisted incremental improvement is required to get from C to C++11 stuff. This is things like converting to use unique_ptr/smart_ptr, changing open coded containers into STL stuff, relying on the language to do cleanup, using move semantics, references, etc/etc. This is where the code becomes safer and more understandable.

My point is there is a proven path from C code to high quality C++11 code that is incremental and auditable (and a wack of human work). It starts with a lightweight 'transpile' to get C code into the C++ compiler in the first place, and no, it doesn't completely destroy the readability of the code base.

To my mind this should be the gold standard. This approach to add Rust parallel to C without any sane migration path is not. The fact that C to Rust, to even get started on the human-led incremental improvement, is basically impossible is not inherent. It comes from the deliberate design of Rust.

Despite Al's virtrol, I think there is merit in exploring what an ideal language upgrade for the kernel would be like - at least we can understand how far away this Rust proposal is.

Using Rust for kernel development

alonz — Tue, 28 Sep 2021 12:41:08 +0000

Upgrading C to C++ (and C++03 to C++11) is somewhat of a joke; the language is generally 99.9% identical, and the only "conversion" required is often just to replace identifiers which conflict with keywords in the other language with something else. (And even then, in some such "uplifted" codebases I have seen, the resulting identifiers were jarring and clearly not what a human programmer would choose to use.)

Conversion between languages with larger semantic differences - such as the case with Rust - would indeed be a feat worthy of a Nobel-prize-equivalent. And that's even before you try to care about readability of the resulting code. (I had myself written two such "transpilers" in my career; both were very special-purpose, targeting very limited code-bases, and both produced code that was nearly impossible to maintain.)

Using Rust for kernel development

dvrabel — Tue, 28 Sep 2021 09:39:34 +0000

And kernel maintainers wonder why they have trouble attracting new developers and maintainers. Only the last sentence has actual, valid content and the rest is entirely unnessesary vitriol.

FWIW, this sort of toxic response and culture is why I'm highly unlikely to ever be a kernel subsystem maintainer again.

Using Rust for kernel development

tux3 — Tue, 28 Sep 2021 07:21:36 +0000

Mozilla seems to think having two languages in the same codebase is reasonable. Given Rust was created at Mozilla and used to ship major Firefox components, I imagine they have thought about this quite a bit.

Unsurprisingly, Mozilla is not trying to convert their existing C++ codebase into transpiler output.

Using Rust for kernel development

jgg — Tue, 28 Sep 2021 02:13:36 +0000

Who said core parts? I said obsolete drivers. I said incremental change.

Advocating for a permentant state of two completely different languages bridged together with a vast hodpodge of bindgen and hand written code only a rare expert can understand is barely a better outcome.

And yes, I know C to Rust-as-it-is-today is halfway insane, but come on, lets at least be open to imagining there is some outcome out there where we could have a language improvement and not end up with a permanent dual-language mess.

There is historical context for upgrading codebases in-place, many C to C++ and C++03 to C+11 uplifts have gone down an incremental upgrade path over the last 20 years without hitting your doomsday prophecy. Mechanically fix enough to be compilable in the new standard, then incremental passes to update high value areas. Programs like clang-tidy and coccinelle are even a proven path for machine guided incremental improvement.

In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.

Using Rust for kernel development

viro — Tue, 28 Sep 2021 00:39:37 +0000

Lovely. When you are done with your "convert not quite safe C into Rust, without turning it into completely undebuggable mess" tool (and after having received the Nevanlinna Prize, richly deserved for such achievement), make sure to contact us mere mortals. Until then, comp.lang.advocacy and alt.sex.masturbation are over -> that way.

And no, it's not too harsh - anyone who suggest that automated transliteration from one language to another can improve anything is delusional and ignorant in the best case. It's not even "one can write Fortran in any language" (and I've seen the results of that), it's "replace the core parts of the kernel with a mess of horribly unidiomatic Rust, produced by automated conversion with unknown amount of bugs introduced by that and zero odds of discovering them in the resulting unreadable obsce^Wcodebase".

Using Rust for kernel development

jgg — Mon, 27 Sep 2021 23:36:02 +0000

I hope the outcome of all of this is a widespread recongition that C is no longer a great language to write the kind of software the Kernel has turned out to be.

I also hope sanity prevals and top level kernel contributors are not required to be experts in two languages. It is hard to reconcile Linus suggesting we need a 10:1 submitter/maintainer ratio while making it ridiculously hard to actually train a kernel maintainer. I'd put up with this for ~5 years, but not forever.

Which suggests to me any path to fixing the problem requires a long-term outcome where all of the kernel is in a single language.

For instance I'd be much happier if someone was working on something like the python 2to3 tool with the idea of mechnically converting entire swaths of the kernel into rust. Eg lets make the *entire* GPIO subsystem rust/etc. Yes, the conversion may be riddled with unsafe, but at least it is one language and can get fixed in an incremental way. IMHO this is a fine way to let old HW drivers ride out into the sunset.

Using Rust for kernel development

pbonzini — Mon, 27 Sep 2021 17:33:56 +0000

> He is worried, though, about "the blank page where the memory model should be".

I should really write the "C11 memory model for kernel programmers" article that I've been mulling over for a while. The same issues pretty much occur with Rust, as Miguel pointed out.