LWN: Comments on "Progress toward a GCC-based Rust compiler"

Progress toward a GCC-based Rust compiler

dvdeug — Thu, 21 Dec 2023 18:55:28 +0000

There was a GCC version that had a particular proprietary C compiler noted in its documentation because that compiler had a bug in its compiling of GCC that made certain optimizations NOPs. Thus a three stage bootstrap of GCC would produce a slightly broken first stage, and an unoptimized but correct second-stage, which would break bootstrap because it wouldn't match the optimized third stage.

So it's conceivable that a bad bootstrap compiler could break the borrow checker in rustc. It'd be unlikely to carry over to next generations, though.

Progress toward a GCC-based Rust compiler

farnz — Wed, 20 Dec 2023 13:13:04 +0000

The part of safety that the borrow checker is responsible for is purely a property of the source code; it confirms that (for example) that the shared XOR mutable invariant is maintained by safe Rust.

Anything which can't be machine-checked at the source level as maintaining the safety invariants is supposed to marked with unsafe, which indicates that the human programmer is responsible for checking that things are safe. This means that anything outside Rust source (such as machine-specific primitives) should be marked as unsafe, and it's on the humans writing a safe wrapper to ensure that the wrapper maintains invariants at all times.

Rust does insist that all instantiations are either safe, or marked appropriately with unsafe; additionally, the caller of an instantiation of a generic that's marked with unsafe must mark their calling block with unsafe to indicate that they're OK with this. You cannot have an unsafe instantiation of a safe generic in safe code - you need the markers to tell Rust that you've thought about this and are going to uphold the language invariants, even if the compiler can't check your working (which is true of platform interfaces, for example, where the compiler can't check the platform behaviour).

Progress toward a GCC-based Rust compiler

rrolls — Wed, 20 Dec 2023 08:20:39 +0000

Aha, so basically, the solution "borrow-check it with rustc, then compile it with whatever" effectively _is_ the solution "write the borrow-checker in a lower-level language". (Since IIRC, each rustc is built with an older rustc, but at some point you'd get back to a version of rustc that wasn't written in Rust.)

One more complication, then. Is "safety" _purely_ a property of the source code, or does it depend on the environment it's being compiled in as well? For example, you use rustc to borrow-check your source code on a typical consumer-grade x86-64 linux platform, and now you compile the same code on some esoteric architecture and operating system. Presumably, the environment can affect, for example, how generics get instantiated, which might turn something that was deemed safe on the first platform into something that's not fine at all on the second? Or does Rust insist that code is only deemed safe if every possible instantiation is safe? I suppose you could request cross-compilation in your borrow-check step, which might address this, but it does add some complication.

Progress toward a GCC-based Rust compiler

steveklabnik — Tue, 19 Dec 2023 16:37:52 +0000

Borrow checking does not affect codegen in any way. As long as the code runs successfully under a borrow checker, a compiler without one can compile it with zero issues. As rustc has been checked by rustc's borrow checker, mrustc will (modulo other bugs, of course) not miscompile it.

At one point mrustc produced a byte-identical compiler to rustc, I am not sure if that's a one-time thing or if they always do that, though.

Progress toward a GCC-based Rust compiler

mathstuf — Tue, 19 Dec 2023 16:36:33 +0000

Note that the borrow checker itself is not part of the language, but of the implementation. Rust's rule is "shared XOR mutable"; the borrow checker is an inexact implementation of an enforcement for this. It has been improved over time from the original lexical borrow checker to non-lexical lifetimes to polonius in the future. I presume that a runtime could also enforce it (at the cost of runtime overhead). In fact, I wonder if valgrind could be enhanced to verify this for any given program at runtime…that would give any implementation more trust in its enforcement of the rule.

Progress toward a GCC-based Rust compiler

farnz — Tue, 19 Dec 2023 15:17:19 +0000

The bootstrap compiler can only be safely used on sources that have been built successfully by a compiler on another platform that never went through a stage without a borrow checker; that way, if there is a bug caused by an invalid borrow (noting that borrow checking is purely a safety net - it does not affect codegen at all), the pre-existing compiler on another platform will have found it.

Bootstrapping and borrow-checking

timon — Tue, 19 Dec 2023 14:49:56 +0000

I don’t think you’re wrong, but I want to add another idea:

Being borrow-checked is more a property of the input (source code) rather than the output (binary), so I’d say you can reasonably treat it as orthogonal to your bootstrap chain.

The problem of having a not-borrow-checked borrow-checker is similar to the “trusting trust” problem. For countering the trusting trust problem, you can do “diverse double-compiling” [1].

I woud propose “borrowing borrow-checkers” as an even better countermeasure here. Just throw your bootstrap compiler Rust source at as many borrow-checkers as you can find, and optimally they should all agree whether your borrows are fine.

[1]: https://dwheeler.com/trusting-trust/

Progress toward a GCC-based Rust compiler

rrolls — Tue, 19 Dec 2023 14:05:06 +0000

I was considering the question "is it safe for a bootstrap Rust compiler to not include a borrow checker" before I even read the comments, so it's pleasing to see this discussed here.

Surely the approach of "the final recursive build will discover [any invalid borrows] and fail" isn't 100% safe, because if an invalid borrow exists it's _possible_ (unlikely but possible) that it creates an edge case that allows exactly that invalid borrow through the borrow checker that was just compiled, right?

I'd describe it as 99% safe, since in practice it'd be much more likely that an invalid borrow in the bootstrap compiler would cause some noticeable effect somewhere else and _not_ prevent its own discovery... but I think for 100% safety, you'd have no option but to write the borrow-checker in a lower-level, already-proven language.

Do correct me if I'm wrong, though!

Progress toward a GCC-based Rust compiler

ssokolow — Tue, 19 Dec 2023 12:58:20 +0000

Regardless, my understanding is that gccrs intends to borrow rustc's implementation of the borrow checker, written in Rust, because, as mrustc demonstrates, you don't need a borrow checker to bootstrap the compiler from a non-Rust language.

Progress toward a GCC-based Rust compiler

SAI_Peregrinus — Mon, 18 Dec 2023 20:12:25 +0000

Polonius is both an alggorithm/model for borrow checking and a Datalog implementation of that algorithm. The implementation in rustc today is written in Rust, not Datalog. There are no longer plans to use the Datalog implementation in rustc, since there's a port to Rust.

Progress toward a GCC-based Rust compiler

tialaramex — Mon, 18 Dec 2023 02:13:36 +0000

"I'm sure it's fine" is of course completely reasonable - for a bootstrap compiler. If it's not fine then our final recursive build will discover that and fail. This is no sort of way to do routine development, but that's not what a bootstrap compiler is for.

Progress toward a GCC-based Rust compiler

Cyberax — Mon, 18 Dec 2023 02:13:30 +0000

mrustc can bootstrap 1.54

Progress toward a GCC-based Rust compiler

mathstuf — Mon, 18 Dec 2023 01:20:40 +0000

`mrustc` is stuck on bootstrapping 1.39 last I heard (the primary author went off to complete their degree or something). Note that its "borrow checker" implementation is "I'm sure it's fine".

Progress toward a GCC-based Rust compiler

mcon147 — Sun, 17 Dec 2023 19:34:58 +0000

It helps shake out the specification for the language itself

Progress toward a GCC-based Rust compiler

atnot — Sun, 17 Dec 2023 02:28:04 +0000

It doesn't really change the bootstrapping story much, given that mrustc exists, which is a C++14 compiler that compiles valid rust to bad C11. So depending on whether you're okay pre-generating those sources, it is either around equally as easy or significantly easier to bootstrap than gccrs would be.

Progress toward a GCC-based Rust compiler

roc — Sun, 17 Dec 2023 01:10:53 +0000

Reimplementing the front-end is likely to expose bugs where the rustc frontend does something unintended.

I think it will also be useful for bootstrapping.

Progress toward a GCC-based Rust compiler

Phantom_Hoover — Sun, 17 Dec 2023 00:25:00 +0000

I really do struggle to understand what the practical reason is for gccrs to exist. Integrating Rust with the GCC toolchain is a worthy goal with practical benefits mentioned in this article… and it’s already been achieved with the GCC codegen. I see no benefits to rewriting the frontend, and it will take a ton of constant work given how quickly Rust evolves. The only thing I can think of is easier bootstrapping, which is a pretty marginal concern.

Actually I am a bit confused by this part of the article:

> There is a wide range of existing GCC plugins that can aid in debugging, static analysis, or hardening; these work on the GCC intermediate representation.

Wouldn’t the GCC codegen for rustc be able to take advantage of this, given it’s emitting GCC IR?

Progress toward a GCC-based Rust compiler

josh — Sat, 16 Dec 2023 20:14:33 +0000

As noted elsewhere in this thread, the question is whether you mean Polonius the standalone datalog-based implementation, or the algorithm that that implementation prototyped.

Progress toward a GCC-based Rust compiler

atnot — Sat, 16 Dec 2023 14:21:30 +0000

You're linking to the same article quoted above, which (in the part quoted) explains the difference between polonius-the-implementation and polonius-the-mathematical-model, which was created while developing polonius-the-implementation. It announces that they no longer intend to use polonius-the-implementation in rustc.

This is a recent development, because for the projects multi-year existence, the general consensus was that polonius-the-implementation was going to be used in rustc.

What has been implemented in rustc is, thus, polonius-the-mathematical-model.

Progress toward a GCC-based Rust compiler

khim — Sat, 16 Dec 2023 13:40:28 +0000

What talks? Where? By whom? Who abandoned Polonius? Nightly Rust still accepts -Zpolonius flag today…

Progress toward a GCC-based Rust compiler

khim — Sat, 16 Dec 2023 13:37:53 +0000

What “recent development” are you talking about? Polonius is already integrated into rustc and was on the way to becoming default in Rust 2024 only two months ago.

What happened in these two months???

Progress toward a GCC-based Rust compiler

khim — Sat, 16 Dec 2023 13:35:26 +0000

I think you are mixing story of Polonius and Chalk.

Attempts to bring Chalk into rustc were, indeed, abandoned. And new replacement is supposed to be built, more tightly integrated with rustc.

Polonius, on the other hand, was integrated long ago. It's not enabled by default, but that is supposed to replace current borrow checker in Rust 2024.

Progress toward a GCC-based Rust compiler

josh — Fri, 15 Dec 2023 22:31:02 +0000

That's true, but there's still discussion about librarifying chunks of rustc, even if it isn't the datalog-based borrow checker prototype.

Progress toward a GCC-based Rust compiler

atnot — Fri, 15 Dec 2023 16:23:44 +0000

It should be noted that is a relatively recent development. Originally polonius was very much written with the idea that it would be used in rustc.

Progress toward a GCC-based Rust compiler

Bigos — Fri, 15 Dec 2023 15:41:34 +0000

I don't think rustc will use Polonius directly:

> Polonius refers to a few things. It is a new formulation of the borrow checker. It is also a specific project that implemented that analysis, based on datalog. Our current plan does not make use of that datalog-based implementation, but uses what we learned implementing it to focus on reimplementing Polonius within rustc.

https://blog.rust-lang.org/inside-rust/2023/10/06/poloniu...

I understand that as they working on reimplementing Polonius within rustc (not a separate crate). It will use the original Polonius ideas but work in a different way.