Linux-for-Rust or Rust-for-Linux [LWN.net]

Linux-for-Rust or Rust-for-Linux

Posted Aug 30, 2024 15:47 UTC (Fri) by jgg (subscriber, #55211)
In reply to: Linux-for-Rust or Rust-for-Linux by willy
Parent article: Rust-for-Linux developer Wedson Almeida Filho drops out

I was in the room and sort of thought Ted was "capturing the mood" - lets be blunt - there are some loud voices in FS/Block that are very anti-rust.

As above, I think the Linux project needs to make a clear unambigous decision, or at least set a timeline to make it.

Either we are doing Rust and what Ted said is out of line, or we are not, and Ted's position is reasonable - do not burden busy people with a Rust experiment that is nobody is going to use.

I thought the point of this inbetween state was supposed to be doing experiments to decide if Rust is feasible. Is there a result yet? Your concern about inlines is news to me, that sounds kind of fatal honestly. What was the result of the nvme experiment? I saw some graphs that it was worse performance, that's not encouraging?? How much of the toolchain situation is sorted out? Last LPC they were saying constant upgrades required? GCC was working on a front end?

to post comments

Linux-for-Rust or Rust-for-Linux

Posted Aug 30, 2024 17:55 UTC (Fri) by Wol (subscriber, #4433) [Link] (5 responses)

> Your concern about inlines is news to me, that sounds kind of fatal honestly.

I was under the impression that - compiling huge monolithic blobs - Rust was quite capable of spotting and inlining functions like that all by itself. What's the point of an "inline" keyword if you can rely on the compiler to spot it?

(Dunno whether I like the downsides of huge blobs - it makes libraries more complicated - but it's horses for courses, or pick you poison, whichever suits ...)

Cheers,
Wol

Linux-for-Rust or Rust-for-Linux

Posted Aug 30, 2024 20:12 UTC (Fri) by pbonzini (subscriber, #60935) [Link] (4 responses)

These are cross-language calls, so the call from Rust to C (say, to spin_lock()) will not be inlined.

Linux-for-Rust or Rust-for-Linux

Posted Aug 30, 2024 22:55 UTC (Fri) by roc (subscriber, #30627) [Link] (2 responses)

Mozilla has been inlining across the Rust-C++ boundary, using LTO, for five years: https://www.reddit.com/r/cpp/comments/ch7g6n/mozilla_just...

Linux-for-Rust or Rust-for-Linux

Posted Aug 31, 2024 5:38 UTC (Sat) by pbonzini (subscriber, #60935) [Link] (1 responses)

Linux however it's typically compiled with GCC. I remember people working on LTO for Linux a few years ago but I think that's not particularly common.

Linux-for-Rust or Rust-for-Linux

Posted Aug 31, 2024 6:16 UTC (Sat) by roc (subscriber, #30627) [Link]

OK, but the important thing is that there's an obvious fix for cross-language inlining if that becomes important.

Linux-for-Rust or Rust-for-Linux

Posted Aug 31, 2024 18:02 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

Even without LTO, a simple practical solution might be to duplicate some of the hottest code (e.g. spin_lock) in Rust. It's not great, but most of such code has been stable for quite a while.

Linux-for-Rust or Rust-for-Linux

Posted Aug 30, 2024 22:46 UTC (Fri) by kees (subscriber, #27264) [Link] (15 responses)

We are doing Rust and what Ted said is out of line. :)

Yes, there continues to be technical issues to be worked out. This is no different from C. It's the nature of technology. We're always moving to new compiler versions, refactoring to get rid of bad APIs, etc.

And Rust has shown its strengths very well, IMO. The M1 GPU driver is excellent and in constant use by a large set of distro users. The Binder driver is a drop-in replacement for the C version and will be shipped in Android soon.

I think the characterization of these Rust drivers being "toy projects" is pretty wildly inaccurate, if not outright insulting.

It's just a big change that is taking time to get everything resolved. I don't understand the resistance to learning new languages, especially given the large developer community associated with Rust. Us kernel devs are always hoping to get more people involved in Linux... Why push people away like this, especially given how many problems Rust permanently solves?

But to answer your questions:

Yes, Rust is feasible.

The NVMe driver works fine and the interfaces needed for it are going through review, though it's an uphill battle politically: as you said there are a few very loud voices in FS/Block that seem to be overwhelmed by their work loads.

The NVMe performance graphs looked very encouraging to me! A proof-of-concept driver with no explicit optimizations is identical to the C performance, except in 2 cases: one where it's within roughly 2%, and in another was _faster_.

What do you consider "the toolchain situation"? It works fine for me. :)

Yes, the upgrade cycle is faster than for GCC and Clang currently, but this is already slowing as the language features needed for Linux are stabilizing, so now there is a minimum version needed, not an exact version.

Yes, GCC has, I think, 2 front-end projects for Rust. I haven't paid too much attention to this myself, though.

GCC and Rust

Posted Aug 30, 2024 22:53 UTC (Fri) by corbet (editor, #1) [Link] (12 responses)

I'm hoping to learn more about the state of gccrs at Cauldron soon. That is one of my biggest concerns with this whole thing... GCC support is needed to reach all of the targets supported by the kernel, but the gccrs project seems to be languishing with little effort going into it. Somehow, I think, we have to find a way to bring some resources to bear on that problem.

GCC and Rust

Posted Aug 31, 2024 18:43 UTC (Sat) by josh (subscriber, #17465) [Link]

> GCC support is needed to reach all of the targets supported by the kernel

rustc_codegen_gcc is making steady progress.

(That's leaving aside the question of how much value those targets provide.)

GCC and Rust

Posted Sep 1, 2024 11:38 UTC (Sun) by ralfj (subscriber, #172874) [Link] (10 responses)

I think the focus should be on rustc_codegen_gcc: reusing the entire Rust frontend and middle-end, and just replacing the codegen backend. That has a much higher chance of delivering a compiler that is able to keep up with Rust's development and access GCC's backends.

gccrs is attempting an entirely independent second Rust implementation -- that's orders of magnitude more work than rustc_codegen_gcc, and I don't think there are significant benefits that would justify the cost. (This is not to say the gccrs devs should stop, if they're having fun doing what they do then by all means continue, but in terms of where to allocate resources and where to watch for medium-term results, I think rustc_codegen_gcc is clearly the better choice.)

rustc_codegen_gcc is unfortunately held back by GCC's reluctance to provide a nice library API for accessing its backends, but it seems using the libgccjit library works reasonably well.

Standardization - two independent implementations are good.

Posted Sep 2, 2024 15:12 UTC (Mon) by jjs (guest, #10315) [Link] (9 responses)

gccrs being a second, independent implementation is good. It ensures that the specification is, in fact, clear. I've seen many projects (HW & SW) where the specifications seemed clear to the writers who did an implementation, but someone else followed the specifications, made something that matched the specifications, yet it was not interoperable with the original version. This is the nature of language - there's lots of places where the meaning of a word is not a singular, universally agreed meaning (check any dictionary).

Law dictionaries exist to help ensure legal language is precise & unambiguous. There's a reason IETF requires two, independent implementations before declaring something a Internet Standard - https://www.ietf.org/participate/runningcode/implementati.... If two implementations don't produce the same product, it's time to go back and fine tune the specification to clarify the ambiguities that arise. And the only way to check for ambiguities is via an independent implementation.

Standardization - two independent implementations are good.

Posted Sep 2, 2024 18:43 UTC (Mon) by ralfj (subscriber, #172874) [Link] (8 responses)

> gccrs being a second, independent implementation is good. It ensures that the specification is, in fact, clear.

It may do that. Or it may cause endless issues due to differences in behavior between implementations, as is the case in C. One reason why the standard leaves so many things as "Undefined Behavior" is that implementations happened to implement different behavior, and none of them wanted to change. It's easy for them to agree to make things UB, the consequences are beard by programmers... just look at the entire debacle with realloc-of-size-0 now being UB: https://queue.acm.org/detail.cfm?id=3588242

I don't deny that multiple independent implementations have advantages. But they also have serious disadvantages. And given the resources required to build and maintain them, I am not convinced that it's worth it overall. The fact that language implementations are typically open-source these days has removed one of the biggest arguments in favor of multiple implementations.

Standardization - two independent implementations are good.

Posted Sep 2, 2024 22:09 UTC (Mon) by jjs (guest, #10315) [Link] (2 responses)

Yes, they can define the behavior as UB -which means they've changed the spec. If you have a spec with defined behavior, and two implementations have different behavior, but meet the spec, you really have two choices, IMO -
1. Follow what appears to be the C way - declare it UB in the spec. Also, what I understand from this article & other things I've read about Rust that the Rust community is trying to avoid.
2. Clarify the spec. Choose which behavior is correct (or a third way), and rewrite the spec to clarify it.

In either case, the spec is changed. I suppose a 3rd way is to ignore the problem, but, IMO that's worse.

"The fact that language implementations are typically open-source these days has removed one of the biggest arguments in favor of multiple implementations."

I'll argue the opposite - it's the language implementations being open source is one of the biggest arguments in favor of multiple implementations. Look at what went on with Linux and GCC/LLVM as LLVM began to work to compile the kernel. More defined behavior, from what I can tell. And a huge advantage of open source is everyone can contribute.

Standardization - two independent implementations are good.

Posted Sep 5, 2024 11:40 UTC (Thu) by taladar (subscriber, #68407) [Link]

The C way was to declare it undefined behavior in the spec because the committee of representatives from multiple implementations that already implemented things differently failed to find a consensus whose code should change, not because there is ever any advantage at all in having undefined parts in a spec.

Standardization - two independent implementations are good.

Posted Sep 5, 2024 14:28 UTC (Thu) by ralfj (subscriber, #172874) [Link]

> If you have a spec with defined behavior, and two implementations have different behavior, but meet the spec, you really have two choices, IMO

That's not what happened here. In this case, the C standard was unambiguous since at least C89: "If size is zero and ptr is not a null pointer, the object it points to is freed". Some implementations violated the standard, and somehow it was deemed better to introduce UB into tons of existing code than to fix the buggy implementations.

Such a hypothetical case could of course happen, though. IMO in that case you have a buggy (unintentionally underdefined) standard -- which happens and which needs to be dealt with reasonably well. If you have multiple different implementations of the standard, they are very hard to fix (other than by making the standard so weak that it encompasses all implementations), and that explains some (but not all) of the oddities in C. If you only have a single implementation, it is a lot easier to fix such bugs in the standard/specification by adjusting either the spec (to still have a *defined* behavior! just maybe not the one that we'd ideally have liked to see) or the implementation. These kinds of things happen in Rust fairly regularly. A big part of what makes this possible is that we have the ability to add "future compatibility" lints to Rust so that there's many months or even years of advance notice to all code that might be affected by a compiler change. I worry that with multiple implementations, this kind of language evolution will become even harder than it already is due to the added friction of having to coordinate this across implementations.

Standardization - two independent implementations are good.

Posted Sep 2, 2024 22:40 UTC (Mon) by viro (subscriber, #7872) [Link] (4 responses)

When specification is "whatever the interpreter actually does", you get wonders like sh(1). Which is _not_ a good language to write in...

Standardization - two independent implementations are good.

Posted Sep 5, 2024 11:41 UTC (Thu) by taladar (subscriber, #68407) [Link] (3 responses)

Which is why there is an effort to develop a Rust spec but that still doesn't require a second implementation, just a test suite that checks if the one implementation conforms to the spec.

2nd Implementation tests the meaning of the specification

Posted Sep 7, 2024 17:10 UTC (Sat) by jjs (guest, #10315) [Link] (2 responses)

That test suite can determine if one implementation meets what the spec writers interpret the spec to mean. It can't detect if the spec always means what the spec writers think it means (the wonders of human language). The purpose of a second implementation is to check that the wording of the spec actually only means what the spec writers think it means. I.e. catch unseen errors in the spec. Again, there's a reason IETF requires two independent implementations of an RFC before they declare it a standard.

2nd Implementation tests the meaning of the specification

Posted Sep 7, 2024 17:40 UTC (Sat) by intelfx (subscriber, #130118) [Link]

Or you can simply have one team write the compiler (and maybe the spec) and some other team to write the tests using the spec.

2nd Implementation tests the meaning of the specification

Posted Sep 8, 2024 12:02 UTC (Sun) by farnz (subscriber, #17727) [Link]

That's where two implementations of the test suite comes in handy, since you now have two separate groups of people who've read the specification and agree on what it means; where one test suite fails and the other passes, you need to resolve that by either fixing the specification, or getting the passing test suite to agree that they had a gap in test coverage.

Linux-for-Rust or Rust-for-Linux

Posted Aug 31, 2024 2:03 UTC (Sat) by sam_c (subscriber, #139836) [Link] (1 responses)

> Yes, the upgrade cycle is faster than for GCC and Clang currently, but this is already slowing as the language features needed for Linux are stabilizing, so now there is a minimum version needed, not an exact version.

I think faster is being generous. Nightly crates are still being used and the minimum Rust version is still extremely recent and being cranked up regularly. I think it's inevitable that Rust will be used in the kernel, but I find it hard to accept it's mature enough to merit being there when so many unstable features are needed.

Rust also doesn't, AFAIK, have any LTS versions for its compiler.

Linux-for-Rust or Rust-for-Linux

Posted Sep 2, 2024 11:55 UTC (Mon) by taladar (subscriber, #68407) [Link]

What benefits do you expect from an LTS version for a compiler where the most recent version is supposed to compile all code previous versions compiled? Sure, there might be the occasional bug compromising that goal but that can (and does) happen with backports to LTS versions too.