Using Rust for kernel development

Posted Sep 28, 2021 0:39 UTC (Tue) by viro (subscriber, #7872)
In reply to: Using Rust for kernel development by jgg
Parent article: Using Rust for kernel development

Lovely. When you are done with your "convert not quite safe C into Rust, without turning it into completely undebuggable mess" tool (and after having received the Nevanlinna Prize, richly deserved for such achievement), make sure to contact us mere mortals. Until then, comp.lang.advocacy and alt.sex.masturbation are over -> that way.

And no, it's not too harsh - anyone who suggest that automated transliteration from one language to another can improve anything is delusional and ignorant in the best case. It's not even "one can write Fortran in any language" (and I've seen the results of that), it's "replace the core parts of the kernel with a mess of horribly unidiomatic Rust, produced by automated conversion with unknown amount of bugs introduced by that and zero odds of discovering them in the resulting unreadable obsce^Wcodebase".

Using Rust for kernel development

Posted Sep 28, 2021 2:13 UTC (Tue) by jgg (subscriber, #55211) [Link] (9 responses)

Who said core parts? I said obsolete drivers. I said incremental change.

Advocating for a permentant state of two completely different languages bridged together with a vast hodpodge of bindgen and hand written code only a rare expert can understand is barely a better outcome.

And yes, I know C to Rust-as-it-is-today is halfway insane, but come on, lets at least be open to imagining there is some outcome out there where we could have a language improvement and not end up with a permanent dual-language mess.

There is historical context for upgrading codebases in-place, many C to C++ and C++03 to C+11 uplifts have gone down an incremental upgrade path over the last 20 years without hitting your doomsday prophecy. Mechanically fix enough to be compilable in the new standard, then incremental passes to update high value areas. Programs like clang-tidy and coccinelle are even a proven path for machine guided incremental improvement.

In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.

Using Rust for kernel development

Posted Sep 28, 2021 7:21 UTC (Tue) by tux3 (subscriber, #101245) [Link] (4 responses)

Mozilla seems to think having two languages in the same codebase is reasonable. Given Rust was created at Mozilla and used to ship major Firefox components, I imagine they have thought about this quite a bit.

Unsurprisingly, Mozilla is not trying to convert their existing C++ codebase into transpiler output.

Using Rust for kernel development

Posted Sep 29, 2021 6:29 UTC (Wed) by maxfragg (guest, #122266) [Link] (3 responses)

you might be surprised to learn, that mozillas codebase started with transpiler-output, as what later became the mozilla suit and in turn firefox had a transpiled java core.

Using Rust for kernel development

Posted Sep 29, 2021 9:13 UTC (Wed) by farnz (subscriber, #17727) [Link] (2 responses)

Have you got a reference for this? I can trace the core of Firefox's codebase back to Netscape 1.0, which predates Java, so if a transpiler was involved, it was during the rewrite.

Using Rust for kernel development

Posted Sep 29, 2021 9:33 UTC (Wed) by Fowl (subscriber, #65667) [Link] (1 responses)

I don't know if there is any transpilation in the "original" Mozilla, but the HTML (5, at the time) parser was converted from Java to C++ - https://johnresig.com/blog/html-5-parsing/

Using Rust for kernel development

Posted Sep 29, 2021 10:41 UTC (Wed) by excors (subscriber, #95769) [Link]

> I don't know if there is any transpilation in the "original" Mozilla, but the HTML (5, at the time) parser was converted from Java to C++ - https://johnresig.com/blog/html-5-parsing/

The HTML5 parser is a special case since it's a large amount of mostly very repetitive code, defining a complex state machine to turn characters into tokens and another to turn tokens into a DOM tree. (There's enough special cases and weird rules, for backward compatibility reasons, that it's not very practical to implement the state machines declaratively - it's easier to write as imperative code. Plus the code closely mirrors the structure of the HTML5 specification, which makes it easier to verify that the spec and implementation are consistent.)

Since it's so repetitive, it's relatively easy to mechanically translate between languages. And I think in this case the transpiler was custom-written for the HTML5 parser, and the Java version of the code was modified to be easy to transpile. E.g. compare "startTag" in Java (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...) and C++ (https://hg.mozilla.org/mozilla-central/diff/8b5103cb12a6b...): the Java has some explicit "// [NOCPP[" markers for sections to omit from the C++, function calls like err() are deleted from the C++ (because the Java version was meant for use in a validator, which needs to report parse errors with as much detail as possible, whereas a browser doesn't), enum values are renamed to Mozilla's naming convention, string literals are replaced with global constants, etc, so there's a lot of translation rules specific to this codebase.

This is very different from general-purpose transpiling of arbitrary code that wasn't designed for it.

Using Rust for kernel development

Posted Sep 28, 2021 12:41 UTC (Tue) by alonz (subscriber, #815) [Link] (1 responses)

Upgrading C to C++ (and C++03 to C++11) is somewhat of a joke; the language is generally 99.9% identical, and the only "conversion" required is often just to replace identifiers which conflict with keywords in the other language with something else. (And even then, in some such "uplifted" codebases I have seen, the resulting identifiers were jarring and clearly not what a human programmer would choose to use.)

Conversion between languages with larger semantic differences - such as the case with Rust - would indeed be a feat worthy of a Nobel-prize-equivalent. And that's even before you try to care about readability of the resulting code. (I had myself written two such "transpilers" in my career; both were very special-purpose, targeting very limited code-bases, and both produced code that was nearly impossible to maintain.)

Using Rust for kernel development

Posted Sep 28, 2021 13:41 UTC (Tue) by jgg (subscriber, #55211) [Link]

Just making C compile in a C++ compiler is not at all what I mean by upgrade. Proper C++11 code looks and operates very differently from C code. Human led, machine assisted incremental improvement is required to get from C to C++11 stuff. This is things like converting to use unique_ptr/smart_ptr, changing open coded containers into STL stuff, relying on the language to do cleanup, using move semantics, references, etc/etc. This is where the code becomes safer and more understandable.

My point is there is a proven path from C code to high quality C++11 code that is incremental and auditable (and a wack of human work). It starts with a lightweight 'transpile' to get C code into the C++ compiler in the first place, and no, it doesn't completely destroy the readability of the code base.

To my mind this should be the gold standard. This approach to add Rust parallel to C without any sane migration path is not. The fact that C to Rust, to even get started on the human-led incremental improvement, is basically impossible is not inherent. It comes from the deliberate design of Rust.

Despite Al's virtrol, I think there is merit in exploring what an ideal language upgrade for the kernel would be like - at least we can understand how far away this Rust proposal is.

Using Rust for kernel development

Posted Sep 29, 2021 23:47 UTC (Wed) by MrWim (subscriber, #47432) [Link]

In case you're not already aware there's the c2rust project: https://c2rust.com/ . The idea was that it does an automatic conversion for you to unsafe rust and then provides tools to help with conversion to idiomatic rust, and to help confirm that the conversion was faithful.

I can thoroughly recommend the blog series on the same: https://immunant.com/blog/ and the wiki contains a discussion of some of the limitations of the approach: https://github.com/immunant/c2rust/wiki/Known-Limitations... . It's sufficiently advanced to allow transpiling Quake 3 with only minor changes required to the C source.

From looking at the github history it seems like the project has slowed over the last year or two. I don't know of its current state.

(This isn't an endorsement of the approach, just a pointer to it)

Using Rust for kernel development

Posted Oct 10, 2021 14:42 UTC (Sun) by NAR (subscriber, #1313) [Link]

In my experiance dual language is really rare. The only direct experiances I have doing this has ended up being pretty horrible in the interfacing layer.

Interesting. I thought it's quite common - have the frontend in Java or Javascript, the backend in C++ or Elixir or whatever, throw in some SQL for the stored procedures, some Python for test glue, maybe a Perl binding... Even if there are separate teams working on the frontend and backend, they might still look into each others code from time to time... I didn't need to be a Python expert to add a testcase or a Javascript expert to figure out the parameters passed to the backend. Granted, there are few projects needing the level of C expertise as the Linux kernel, so maybe being an expert in two languages might be harder - but I don't think (maybe C++ is the big exception, that's a horrible complicated language).

Using Rust for kernel development

Posted Sep 28, 2021 9:39 UTC (Tue) by dvrabel (subscriber, #9500) [Link] (1 responses)

And kernel maintainers wonder why they have trouble attracting new developers and maintainers. Only the last sentence has actual, valid content and the rest is entirely unnessesary vitriol.

FWIW, this sort of toxic response and culture is why I'm highly unlikely to ever be a kernel subsystem maintainer again.

Using Rust for kernel development

Posted Sep 30, 2021 21:53 UTC (Thu) by piexil (guest, #145099) [Link]

For real, it was completely unnecessary, and it's so smug. People also already do things similar to fixing transpiled code in their freetime. (reverse engineering decompiled assembly to c like the super mario 64 decomp comes to mind)