Thoughts and clarifications

Posted Sep 15, 2024 12:19 UTC (Sun) by asahilina (subscriber, #166071)
In reply to: Thoughts and clarifications by pizza
Parent article: Whither the Apple AGX graphics driver?

> the assertion of "just change the definition and your job is done when the compiler stops complaining" is laughably naive.

My experience having gone through several major refactorings of the drm/asahi driver is that it is correct. Some of the bigger ones are:

- Going from a "demo" fully blocking implementation (the first one we shipped) to implementing queues and scheduling and asynchronous work
- Dropping in GPUVM and fully implementing VM_BIND, which along the way ended up changing how non-GPUVM kernel objects are managed including making changes to the heap allocators [1]

It is hard to explain just how liberating it is to be able to refactor and rearrange non-trivial things in the code, fix all the compiler errors, and then end up with working code more often than not. Sure, if you're unlucky you might run into a logic error or a deadlock or something... but with C it's almost impossible to escape adding a new handful of memory safety errors and new flakiness and bugs, every time you make any non-trivial change to the structure of the code.

This is true even when you're interfacing with C code, as long as its API is documented or can be easily understood. The GPUVM change involved first writing abstractions for the underlying C code. That API is reasonably nice and well documented, so it wasn't hard to get the Rust abstraction right (with some care) [2], and then when it came to dropping it into the Rust driver, everything just worked.

Most people don't believe this until they actually start working with Rust on larger projects. All the Rust evangelism isn't just zealotry. There really is something magical about it, even if it might be overstated sometimes.

[1] https://github.com/AsahiLinux/linux/commit/93b390cce8a303...
[2] https://github.com/AsahiLinux/linux/commit/e3012f87bf98c0...

Thoughts and clarifications

Posted Sep 16, 2024 9:05 UTC (Mon) by Wol (subscriber, #4433) [Link] (4 responses)

> > the assertion of "just change the definition and your job is done when the compiler stops complaining" is laughably naive.

> My experience having gone through several major refactorings of the drm/asahi driver is that it is correct. Some of the bigger ones are:

Out of curiosity, would you describe *your* codebase as complex? Or would you say "my code is simple because Rust handles the complexity for me"?

Or even "the driver problem itself is fairly simple, and Rust just makes it easy to express it"? (Put another way, "C makes the problem a lot more complicated than it should be"!)

Cheers,
Wol

Thoughts and clarifications

Posted Sep 16, 2024 9:48 UTC (Mon) by asahilina (subscriber, #166071) [Link] (3 responses)

Hmm... there are a few dimensions here.

I would say the driver has medium complexity for a GPU driver (in terms of line count it's almost 4x drm/panfrost and around the same as the GPU part of drm/msm). Rust doesn't directly reduce complexity (the driver has to do what it has to do), but it does handle a lot of error-prone boilerplate for you (for example enforced RAII) and it strongly encourages design that makes it easier to reason about the complexity (separation of concerns/encapsulation). So Rust makes it easier to maintain the complexity, understand it, and avoid bugs caused by it. I'm a lot more comfortable dealing with complex code in Rust than in C.

Then, there are some aspects where Rust is specifically a very good fit for this particular GPU driver. One of them is using Rust proc macro magic to describe multiple firmware version and GPU generation interfaces (the firmware interface is not stable) in a single implementation, as cleanly as possible. To do the same thing in C you either end up duplicating all the code, or using ugly preprocessor or build system tricks (drm/apple in our tree is a C driver that has to do this, and it's not pretty. Rust would be a good fit for a rewrite of that driver too for multiple reasons, but we need DRM KMS bindings first). The other one is (ab)using Rust lifetimes to represent GPU firmware interface lifetimes, which makes handling the firmware interface much less error-prone (and this is critical, because an error crashes the GPU firmware irrecoverably). So Rust helps with those more specific kinds of complexity.

At the end of the day it all really boils down to Rust benefiting from decades of programming experience and history in its design. C was designed at a time when programs were a tiny fraction of the size they are today. The entire 1983 UNIX kernel had around the same line count in C as my drm/asahi driver does in Rust. Linux is more than a thousand times more code today, and it shouldn't be a surprise that a programming language designed for codebases 1000x smaller might not be the best option these days. We have learned a lot since then about how to manage complexity, and Rust takes a lot of that and applies it to the kind of systems language that is suitable for writing kernels.

Thoughts and clarifications

Posted Sep 16, 2024 11:16 UTC (Mon) by Wol (subscriber, #4433) [Link] (2 responses)

> Rust doesn't directly reduce complexity (the driver has to do what it has to do), but it does handle a lot of error-prone boilerplate for you (for example enforced RAII) and it strongly encourages design that makes it easier to reason about the complexity (separation of concerns/encapsulation). So Rust makes it easier to maintain the complexity, understand it, and avoid bugs caused by it. I'm a lot more comfortable dealing with complex code in Rust than in C.

So in other words "It's a complex problem, but Rust makes it simple to express that complexity"?

I'm just trying to get a handle on where Rust lies on the problem-complexity / language-complexity graph. I'll upset Jon with this, but I hate Relational/SQL because imnsho Relational lies too far on the simplicity side of the graph, so SQL has to lie way over on the complexity side. So in terms of Einstein's "make things as simple as possible, but no simpler", Relational/SQL lies way above the local minimum. Rustaceans probably feel the same is true of C and modern hardware.

Do you feel Rust lies close to the sweet spot of minimal possible complexity? It certainly comes over you find it easy to express the complexity of the hardware.

Cheers,
Wol

Thoughts and clarifications

Posted Sep 16, 2024 11:57 UTC (Mon) by jake (editor, #205) [Link] (1 responses)

> I'll upset Jon with this

Wol, it's more than just Jon who is tired of you bringing up this stuff in every thread, often multiple times, when it is not particularly relevant. Your comments are voluminous, people have complained to you about that and the content of your posts in comments here, and you are one of the most filtered commenters we have. I think you should perhaps reduce your volume of comments and try to ensure that the drum you are banging does not come up everywhere.

just fyi,

jake

Thoughts and clarifications

Posted Sep 16, 2024 12:21 UTC (Mon) by paulj (subscriber, #341) [Link]

Perhaps putting some stats on "ranking by volume of comments (over the site in last X period, for X in {a, b, c} time|this story)" for a user to that user somewhere would help softly nudge people, where needed, on a self-educating basis?

Thoughts and clarifications

Posted Sep 22, 2024 18:05 UTC (Sun) by Rudd-O (guest, #61155) [Link] (4 responses)

I have reason to believe that you are talking to people who have never seen a match statement in their lives. And so they're used to knowing that when they make a change somewhere in the code, somewhere else very, very far away, and if or case select statement no longer matches that condition that you just added to the code, and therefore things break spectacularly at runtime.

That lack of experience is why they continue issuing the truism that refactoring is "very difficult" and you don't really know when you're changing code if something else is going to break. They haven't gotten the compiler to yell at them, "you're missing this case", because they have never experienced it. Reflectoring is super easy when the computer is doing the thinking of myriad otherwise irrelevant trivialities for you!

There really is something magical about it. And to try and explain to people that haven't seen that, quote, magic, is almost impossible. It's like trying to explain electricity to someone from the 1600s. And it is equally frustrating. In fact, it is doubly frustrating because unlike electricity in the 1600s, this is something that is very easy to witness, you just have to read a little code and push a button in a webpage and you can see it. And they just refuse. It is so oddly disconcerting.

Thoughts and clarifications

Posted Sep 22, 2024 18:29 UTC (Sun) by pizza (subscriber, #46) [Link] (3 responses)

> you just have to read a little code and push a button in a webpage and you can see it. And they just refuse. It is so oddly disconcerting.

$ sloccount `find projdir -name *.[ch]`
[...]
Total Physical Source Lines of Code (SLOC) = 2,278,858

Call me naive, but "read a little code and push a button on a web page" isn't going to cut it.

Thoughts and clarifications

Posted Sep 25, 2024 9:44 UTC (Wed) by Rudd-O (guest, #61155) [Link] (2 responses)

That's a lot of code.

To get back to the topic (common pitfalls of refactoring and how Rust helps avoid errors):

Can you articulate what a match statement does, and how it behaves when you add a new case somewhere very far away from the match statement? How is it different from, say, a chain of if/else or a select case? If your codebase was (hypothetically) Rust, what would the compiler say to such a change, versus what the C compiler says today?

My intention is to figure out if you have had a chance to compare both C and Rust in order to form an honest, informed opinion.

Thanks in advance.

Perhaps that is far enough

Posted Sep 25, 2024 9:48 UTC (Wed) by corbet (editor, #1) [Link] (1 responses)

It is my suspicion that this conversation will go nowhere useful after this point. Perhaps it's best to stop it here?

Perhaps that is far enough

Posted Sep 25, 2024 10:01 UTC (Wed) by Rudd-O (guest, #61155) [Link]

Sure. Have a nice day.