Debian, Rust, and librsvg [LWN.net]

Debian, Rust, and librsvg

Posted Nov 15, 2018 22:15 UTC (Thu) by ceplm (subscriber, #41334) [Link] (9 responses)

Hmm, but it seems that for OCaml https://is.gd/hU4vAG we are able to build some minimal version of the compiler which then generates the full compiler. I wonder whether something like that would be possible with Rust.

Debian, Rust, and librsvg

Posted Nov 15, 2018 23:01 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (7 responses)

> Hmm, but it seems that for OCaml https://is.gd/hU4vAG we are able to build some minimal version of the compiler which then generates the full compiler.

To build the minimal version still requires GCC, and to bootstrap GCC you need some form of C compiler (though not necessarily GCC itself).

I don't know of any distros with a requirement to be able to build every package "from scratch" with no outside dependencies. Frankly I'm not even sure how that would work; does every rebuild need to start with procuring a blank-slate system with a direct hardware interface for loading hand-generated machine code? In practice you always start out using host tools of one form or another. If you look at the bootstrap process for something like Linux From Scratch[1] it involves building the toolchain three times: First you build a cross-compiler using the host tools, then you use that cross-compiler on the host to build a native toolchain and other packages required for standalone operation (what LFS calls the "temporary system"), and finally use this temporary system to build the host-independent tools.

This is for a complete system, of course, but bootstrapping a compiler is similar. Use the host's Rust compiler to build a cross-compiler which doesn't rely on the host environment, then use that to build the native Rust compiler. The result of the second build should be deterministic even if the first stage isn't due to variations in host compilers.

[1] http://www.linuxfromscratch.org/

Debian, Rust, and librsvg

Posted Nov 16, 2018 1:11 UTC (Fri) by pabs (subscriber, #43278) [Link]

These folks are attempting to solve the bootstrap problem:

https://bootstrappable.org/

Debian, Rust, and librsvg

Posted Nov 16, 2018 15:39 UTC (Fri) by moltonel (subscriber, #45207) [Link] (5 responses)

We've "always" depended on existing compilers to compile a new compiler, but somehow at some stage the industry decided that C had special status : it's mandatory for "true bootstraping", you have to go all the way down to C but no further. Why is nobody complaining that you can't bootstrap GCC using a Fortran or Lisp compiler, but many people are complaining that you can't bootstrap this or that language using a C compiler ?

I understand that many platforms get a C compiler before any other, but that doesn't change anything to the fact that a pre-existing binary was used to compile that "initial" compiler. The language of the pre-existing binary shouldn't matter, as long as it can generate native code for your target platform. It's a mindset issue, not a technical issue.

For fun, have a look at https://github.com/nebulet/nebulet, a microkernel written in Rust that only executes wasm, and does so in ring 0. No C in sight ;)

Debian, Rust, and librsvg

Posted Nov 16, 2018 16:47 UTC (Fri) by zlynx (guest, #2285) [Link] (1 responses)

I don't have a link right now, but I remember reading about a project that did a full bootstrap on x86. Starting with a Forth interpreter written in machine code that could be hand written to a boot device or DOS DEBUG.EXE. Then super-simple assembler and tiny K&R C compiler written in Forth. Then bootstrap up to GCC 2.95 and from there you can build anything.

Debian, Rust, and librsvg

Posted Nov 23, 2018 10:03 UTC (Fri) by ewen (subscriber, #4772) [Link]

Maybe https://www.owlfolio.org/research/bootstrapping-trust-in-... which speculates using FORTH as part of this bootstrap? Or https://github.com/oriansj/stage0 which mentions FORTH but ultimately takes a different route via assembly and C.

If you’re thinking of another one and do find the link I’d be curious to read about their approach.

Ewen

Debian, Rust, and librsvg

Posted Nov 16, 2018 17:11 UTC (Fri) by madscientist (subscriber, #16861) [Link] (2 responses)

I don't know anything about Rust bootstrapping, but if what others have implied here is true, that Rust only guarantees that rustc can be built with the immediately previous version of the compiler AND a new version is released every 6 weeks or so, it's a major difference from GCC and a serious headache for distributions.

You can find a C compiler from decades ago that is sufficient to build GCC (because GCC starts out by building a "stage 1" version of the GCC compiler which relies only on very portable C, then builds the rest of GCC, including all the different front-ends like C++, gfortran, etc., using that stage 1 compiler). You can cache some obsolete version of GCC, or get TinyCC or something like that, and use that for bootstrapping. It's super-easy and virtually never has to change.

Debian, Rust, and librsvg

Posted Nov 16, 2018 18:30 UTC (Fri) by josh (subscriber, #17465) [Link]

> Rust only guarantees that rustc can be built with the immediately previous version of the compiler AND a new version is released every 6 weeks or so, it's a major difference from GCC and a serious headache for distributions.

Rust doesn't systematically demand the latest prior release to build itself; it simply does not refrain from using features as they become available. As mentioned elsewhere in this thread, folks are working on creating a minimized series of rustc versions to successively bootstrap from the last OCaml version.

Debian, Rust, and librsvg

Posted Nov 20, 2018 5:57 UTC (Tue) by mirabilos (subscriber, #84359) [Link]

And, more importantly than “from decades ago”, is “simple enough to port quickly to new architectures”.

Debian, Rust, and librsvg

Posted Nov 16, 2018 14:09 UTC (Fri) by glondu (subscriber, #70274) [Link]

> it seems that for OCaml we are able to build some minimal version of the compiler which then generates the full compiler

Not exactly.

In OCaml world, there is a portable bytecode for binaries (like Java) and the OCaml sources contain a binary of the full compiler (to bytecode) that is used to compile everything (including the compiler itself). The interpreter for that bytecode is written in C. Hence, "bootstrapping" a new architecture is just porting the interpreter to the new architecture, which is easy.

Additionally, there is a compiler to native code, but it is not strictly needed to enjoy OCaml on a new architecture. And it can be compiled using the bytecode compiler, so no bootstrapping issues.

Debian, Rust, and librsvg

Posted Nov 15, 2018 22:25 UTC (Thu) by nix (subscriber, #2304) [Link] (3 responses)

Though of course in most cases you can use a range of earlier compiler versions to build the current compiler. The only compiler I can recall with a strict 'previous version only guaranteed to work, try with one a few versions old and it might well break' policy was GNAT, and even they relented a few versions back.

For Rust, this makes sense, because the language is evolving fast and this does mean that at least the compiler can dogfood experimental features added in compiler version N in version N+1, then mark them non-experimental once they know they work for at least that use case. :)

Debian, Rust, and librsvg

Posted Nov 16, 2018 0:01 UTC (Fri) by josh (subscriber, #17465) [Link]

You could theoretically go back to the original versions of the Rust compiler written in OCaml and bootstrap forward from there. (And likewise you'd need to bootstrap OCaml.) The folks behind mrustc are also thinking about this kind of problem, and trying to figure out how far forward you can get in each build step.

Debian, Rust, and librsvg

Posted Nov 18, 2018 1:45 UTC (Sun) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

Maybe it's just me, but it's hard to see a system like this as ready for building OS critical components. When the compiler demands bleeding edge features such that it can't be built with a version of itself that's even 3 months old, it is not a stable platform on which to build a system.

Debian, Rust, and librsvg

Posted Nov 18, 2018 23:51 UTC (Sun) by nix (subscriber, #2304) [Link]

The OS critical components are not likely to depend on any particularly recent Rust compiler version. The latest released Firefox is happy with Rust 1.28, which is two releases old now. So you only need to upgrade Rust occasionally, in bursts, when its deps require it. (Note that it is only recently that FF came to depend on anything so recent: only one release ago it depended on 1.24.)

Debian, Rust, and librsvg

Posted Nov 16, 2018 14:11 UTC (Fri) by plugwash (subscriber, #29694) [Link] (5 responses)

That is true.

As the maintainer of Raspbian there have been several things that have made rust particuarly painful for us. I suspect some of these would also be applicable to re-bootstrapping efforts.

1. The upgrade treadmill is unusually rapid. You need the previous release of rust and rust makes a new release about every 6 weeks.
2. There are interdependencies between rust versions and cargo versions that aren't always obvious until you try particular combinations.
3. There does not seem to be any way to disable multithreading in the build. Threading can easilly lead to failures that only show their face on particular hardware/kernel configurations.
4. After a failed build attempting to re-run the failed command manually using the command line printed by the build process (for example to run it in a debugger or under strace or something) produces completely unrelated errors. I guess there are some special environment variables or something but I can't seem to find any documentation of them if there are.
5. It's virtually impossible to google anything about rust. If I google say "rust fabricate" most of the results have nothing to do with the language.

Debian, Rust, and librsvg

Posted Nov 16, 2018 15:14 UTC (Fri) by Ieee754 (guest, #128655) [Link] (1 responses)

> It's virtually impossible to google anything about rust. If I google say "rust fabricate" most of the results have nothing to do with the language.

I have no idea what you expected from googling "rust fabricate", but if I just go to the documentation site of the toolchain: https://doc.rust-lang.org/rustc/index.html , there is a frickin book about how the Rust toolchain is built, how to run tests, debug issues, configure everything, the overall design of the compiler, the different modules, etc.

Debian, Rust, and librsvg

Posted Nov 17, 2018 2:02 UTC (Sat) by plugwash (subscriber, #29694) [Link]

> I have no idea what you expected from googling "rust fabricate"

I was hoping to find some documentation on what "fabricate" (which was hanging during the rustc build on one of our buildboxes, I suspect a bug in the arm32 compatibility layer of the arm64 kernel on said box) did and how to troubleshoot problems with it.

But instead I just found a load of results about metal fabrication :(

Someone did later tell me what fabricate was (apparently it's the rust-installer crate), though I haven't got around to trying to take a closer look. For now I have been working around the issue by doing our rust builds on a different system.

> there is a frickin book about how the Rust toolchain is built

Using the search tool in said book to search for fabricate doesn't seem to turn up any results.

Still thanks for the pointer.

Debian, Rust, and librsvg

Posted Nov 16, 2018 21:09 UTC (Fri) by josh (subscriber, #17465) [Link] (2 responses)

> 2. There are interdependencies between rust versions and cargo versions that aren't always obvious until you try particular combinations.

My understanding is that both Rust and Cargo only require the previous stable version at most, and should never have dependency loops between their latest versions.

> 4. After a failed build attempting to re-run the failed command manually using the command line printed by the build process (for example to run it in a debugger or under strace or something) produces completely unrelated errors. I guess there are some special environment variables or something but I can't seem to find any documentation of them if there are.

This is being fixed right now, and yes, there are environment variables. See https://github.com/rust-lang/cargo/pull/5683 for details.

Debian, Rust, and librsvg

Posted Nov 19, 2018 14:15 UTC (Mon) by dgm (subscriber, #49227) [Link] (1 responses)

> My understanding is that both Rust and Cargo only require the previous stable version at most, and should never have dependency loops between their latest versions.

Doesn't that imply that you (potentially) have to build each and every version in succession to get the latest one?

Debian, Rust, and librsvg

Posted Nov 20, 2018 11:57 UTC (Tue) by nix (subscriber, #2304) [Link]

Yes, but you don't have to do it when they come out: you can do it when you run into something that needs the latest version, and build the intermediate versions only for the sake of building the next in the chain (and then throw them away again). No need to ship, validate etc every version in turn.