Debian, Rust, and librsvg

Posted Nov 14, 2018 16:05 UTC (Wed) by mathstuf (subscriber, #69389)
In reply to: Debian, Rust, and librsvg by moltonel
Parent article: Debian, Rust, and librsvg

> The fact that Rust is releasing often and trying out fairly cutting-edge concepts make me fear that those projects will never catch up

This is no different than CPython being the only implementation of 3.x while PyPy and other projects only had 3.x-2 support or so. Now with CPython explicitly slowing down after the BDFL stuff (and calling out letting other implementations catch up). I imagine that Rust will, at some point, slow down for other implementations, but it took CPython 20+ years to do that. And 10 years (Jython started in 2001) for alternate implementations to even get an initial release.

Debian, Rust, and librsvg

Posted Nov 15, 2018 6:25 UTC (Thu) by edomaur (subscriber, #14520) [Link] (34 responses)

I think that the Edition approach will help with that : in a few weeks, the Rust Language 2018 Edition will be out

https://blog.rust-lang.org/2018/07/27/what-is-rust-2018.html

Debian, Rust, and librsvg

Posted Nov 15, 2018 9:54 UTC (Thu) by ceplm (subscriber, #41334) [Link] (32 responses)

Well, I hope we get reproducible build of Rust soon. On platforms which care strongly about reproducibility (e.g., openSUSE) it is a real problem, it is not possible to build Rust without Rust. https://build.opensuse.org/package/show/devel:languages:r... seems still like a bit of disaster (not sure, how the current build finishes, perhaps this is the first build of Rust which will actually work). Anybody willing to help? I would be glad to be able to finally build neovim-gtk.

Debian, Rust, and librsvg

Posted Nov 15, 2018 18:23 UTC (Thu) by peter-b (guest, #66996) [Link] (1 responses)

> On platforms which care strongly about reproducibility (e.g., openSUSE) it is a real problem, it is not possible to build Rust without Rust.

Sorry, I don't really understand this argument. Does openSUSE build GCC without GCC?

Debian, Rust, and librsvg

Posted Nov 18, 2018 10:27 UTC (Sun) by roblucid (guest, #48964) [Link]

I have built GCC in past without GCC.
A phase of the compiler build, created a GCC subset, that could then bootstrap the full compiler.
If you didn't have a C working on target, then cross compiling was another possibility. GCC was used on Sun h/w to build code for telephone exchanges for example, unsuitable to run a compiler natively.

Debian, Rust, and librsvg

Posted Nov 15, 2018 20:14 UTC (Thu) by josh (subscriber, #17465) [Link] (29 responses)

Reproducible builds can work just fine for self-hosting compilers. If there's an issue building Rust reproducibly, please report it, and let's fix it.

Debian, Rust, and librsvg

Posted Nov 15, 2018 20:57 UTC (Thu) by ceplm (subscriber, #41334) [Link] (28 responses)

Just take a look at the way we (think are forced to) build Rust. https://is.gd/emZco5 It seems to me in order to build Rust, I need binaries of the previous Rust. It seems it is not possible to bootstrap Rust properly without Rust. Is it true, or we are just missing something?

Debian, Rust, and librsvg

Posted Nov 15, 2018 21:14 UTC (Thu) by josh (subscriber, #17465) [Link] (27 responses)

rustc and cargo are written in Rust, so you do need rustc and cargo to build them. However, you should always be able to build each new version of rustc and cargo with your packaged versions of the previous stable version of rustc and cargo. You should never need to download a binary, except for your first initial bootstrap to get packages into the archive.

Debian, Rust, and librsvg

Posted Nov 15, 2018 21:23 UTC (Thu) by ceplm (subscriber, #41334) [Link] (24 responses)

That's the problem: with the reproducible builds, we need to be able to rebuild everything in our repositories from scratch (I don't know the details, but we do it with other compilers, not sure about details of GCC itself).

Debian, Rust, and librsvg

Posted Nov 15, 2018 21:45 UTC (Thu) by ssokolow (guest, #94568) [Link]

If it helps, I don't have a citation URL, but I remember reading in an /r/rust/ comment that someone is keeping track of the chain of rustc versions necessary to bootstrap a current release compiler from C++ using mrustc.

(mrustc is a Rust compiler, written in C++, which makes some assumptions about the correctness of the code, but is complete enough to compile certain older versions of rustc so that it can be proved free of a Trusting Trust attack.)

Debian, Rust, and librsvg

Posted Nov 15, 2018 21:46 UTC (Thu) by josh (subscriber, #17465) [Link] (20 responses)

Rust is hardly the only compiler written in its own language. You're going to have a stack of such loops, including ocaml, C, C++, Haskell, various core utilities often used in build systems, and so on.

You might check with the Debian folks on reproducible build strategies for self-hosting compilers.

Debian, Rust, and librsvg

Posted Nov 15, 2018 22:15 UTC (Thu) by ceplm (subscriber, #41334) [Link] (9 responses)

Hmm, but it seems that for OCaml https://is.gd/hU4vAG we are able to build some minimal version of the compiler which then generates the full compiler. I wonder whether something like that would be possible with Rust.

Debian, Rust, and librsvg

Posted Nov 15, 2018 23:01 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (7 responses)

> Hmm, but it seems that for OCaml https://is.gd/hU4vAG we are able to build some minimal version of the compiler which then generates the full compiler.

To build the minimal version still requires GCC, and to bootstrap GCC you need some form of C compiler (though not necessarily GCC itself).

I don't know of any distros with a requirement to be able to build every package "from scratch" with no outside dependencies. Frankly I'm not even sure how that would work; does every rebuild need to start with procuring a blank-slate system with a direct hardware interface for loading hand-generated machine code? In practice you always start out using host tools of one form or another. If you look at the bootstrap process for something like Linux From Scratch[1] it involves building the toolchain three times: First you build a cross-compiler using the host tools, then you use that cross-compiler on the host to build a native toolchain and other packages required for standalone operation (what LFS calls the "temporary system"), and finally use this temporary system to build the host-independent tools.

This is for a complete system, of course, but bootstrapping a compiler is similar. Use the host's Rust compiler to build a cross-compiler which doesn't rely on the host environment, then use that to build the native Rust compiler. The result of the second build should be deterministic even if the first stage isn't due to variations in host compilers.

[1] http://www.linuxfromscratch.org/

Debian, Rust, and librsvg

Posted Nov 16, 2018 1:11 UTC (Fri) by pabs (subscriber, #43278) [Link]

These folks are attempting to solve the bootstrap problem:

https://bootstrappable.org/

Debian, Rust, and librsvg

Posted Nov 16, 2018 15:39 UTC (Fri) by moltonel (guest, #45207) [Link] (5 responses)

We've "always" depended on existing compilers to compile a new compiler, but somehow at some stage the industry decided that C had special status : it's mandatory for "true bootstraping", you have to go all the way down to C but no further. Why is nobody complaining that you can't bootstrap GCC using a Fortran or Lisp compiler, but many people are complaining that you can't bootstrap this or that language using a C compiler ?

I understand that many platforms get a C compiler before any other, but that doesn't change anything to the fact that a pre-existing binary was used to compile that "initial" compiler. The language of the pre-existing binary shouldn't matter, as long as it can generate native code for your target platform. It's a mindset issue, not a technical issue.

For fun, have a look at https://github.com/nebulet/nebulet, a microkernel written in Rust that only executes wasm, and does so in ring 0. No C in sight ;)

Debian, Rust, and librsvg

Posted Nov 16, 2018 16:47 UTC (Fri) by zlynx (guest, #2285) [Link] (1 responses)

I don't have a link right now, but I remember reading about a project that did a full bootstrap on x86. Starting with a Forth interpreter written in machine code that could be hand written to a boot device or DOS DEBUG.EXE. Then super-simple assembler and tiny K&R C compiler written in Forth. Then bootstrap up to GCC 2.95 and from there you can build anything.

Debian, Rust, and librsvg

Posted Nov 23, 2018 10:03 UTC (Fri) by ewen (subscriber, #4772) [Link]

Maybe https://www.owlfolio.org/research/bootstrapping-trust-in-... which speculates using FORTH as part of this bootstrap? Or https://github.com/oriansj/stage0 which mentions FORTH but ultimately takes a different route via assembly and C.

If you’re thinking of another one and do find the link I’d be curious to read about their approach.

Ewen

Debian, Rust, and librsvg

Posted Nov 16, 2018 17:11 UTC (Fri) by madscientist (subscriber, #16861) [Link] (2 responses)

I don't know anything about Rust bootstrapping, but if what others have implied here is true, that Rust only guarantees that rustc can be built with the immediately previous version of the compiler AND a new version is released every 6 weeks or so, it's a major difference from GCC and a serious headache for distributions.

You can find a C compiler from decades ago that is sufficient to build GCC (because GCC starts out by building a "stage 1" version of the GCC compiler which relies only on very portable C, then builds the rest of GCC, including all the different front-ends like C++, gfortran, etc., using that stage 1 compiler). You can cache some obsolete version of GCC, or get TinyCC or something like that, and use that for bootstrapping. It's super-easy and virtually never has to change.

Debian, Rust, and librsvg

Posted Nov 16, 2018 18:30 UTC (Fri) by josh (subscriber, #17465) [Link]

> Rust only guarantees that rustc can be built with the immediately previous version of the compiler AND a new version is released every 6 weeks or so, it's a major difference from GCC and a serious headache for distributions.

Rust doesn't systematically demand the latest prior release to build itself; it simply does not refrain from using features as they become available. As mentioned elsewhere in this thread, folks are working on creating a minimized series of rustc versions to successively bootstrap from the last OCaml version.

Debian, Rust, and librsvg

Posted Nov 20, 2018 5:57 UTC (Tue) by mirabilos (subscriber, #84359) [Link]

And, more importantly than “from decades ago”, is “simple enough to port quickly to new architectures”.

Debian, Rust, and librsvg

Posted Nov 16, 2018 14:09 UTC (Fri) by glondu (subscriber, #70274) [Link]

> it seems that for OCaml we are able to build some minimal version of the compiler which then generates the full compiler

Not exactly.

In OCaml world, there is a portable bytecode for binaries (like Java) and the OCaml sources contain a binary of the full compiler (to bytecode) that is used to compile everything (including the compiler itself). The interpreter for that bytecode is written in C. Hence, "bootstrapping" a new architecture is just porting the interpreter to the new architecture, which is easy.

Additionally, there is a compiler to native code, but it is not strictly needed to enjoy OCaml on a new architecture. And it can be compiled using the bytecode compiler, so no bootstrapping issues.

Debian, Rust, and librsvg

Posted Nov 15, 2018 22:25 UTC (Thu) by nix (subscriber, #2304) [Link] (3 responses)

Though of course in most cases you can use a range of earlier compiler versions to build the current compiler. The only compiler I can recall with a strict 'previous version only guaranteed to work, try with one a few versions old and it might well break' policy was GNAT, and even they relented a few versions back.

For Rust, this makes sense, because the language is evolving fast and this does mean that at least the compiler can dogfood experimental features added in compiler version N in version N+1, then mark them non-experimental once they know they work for at least that use case. :)

Debian, Rust, and librsvg

Posted Nov 16, 2018 0:01 UTC (Fri) by josh (subscriber, #17465) [Link]

You could theoretically go back to the original versions of the Rust compiler written in OCaml and bootstrap forward from there. (And likewise you'd need to bootstrap OCaml.) The folks behind mrustc are also thinking about this kind of problem, and trying to figure out how far forward you can get in each build step.

Debian, Rust, and librsvg

Posted Nov 18, 2018 1:45 UTC (Sun) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

Maybe it's just me, but it's hard to see a system like this as ready for building OS critical components. When the compiler demands bleeding edge features such that it can't be built with a version of itself that's even 3 months old, it is not a stable platform on which to build a system.

Debian, Rust, and librsvg

Posted Nov 18, 2018 23:51 UTC (Sun) by nix (subscriber, #2304) [Link]

The OS critical components are not likely to depend on any particularly recent Rust compiler version. The latest released Firefox is happy with Rust 1.28, which is two releases old now. So you only need to upgrade Rust occasionally, in bursts, when its deps require it. (Note that it is only recently that FF came to depend on anything so recent: only one release ago it depended on 1.24.)

Debian, Rust, and librsvg

Posted Nov 16, 2018 14:11 UTC (Fri) by plugwash (subscriber, #29694) [Link] (5 responses)

That is true.

As the maintainer of Raspbian there have been several things that have made rust particuarly painful for us. I suspect some of these would also be applicable to re-bootstrapping efforts.

1. The upgrade treadmill is unusually rapid. You need the previous release of rust and rust makes a new release about every 6 weeks.
2. There are interdependencies between rust versions and cargo versions that aren't always obvious until you try particular combinations.
3. There does not seem to be any way to disable multithreading in the build. Threading can easilly lead to failures that only show their face on particular hardware/kernel configurations.
4. After a failed build attempting to re-run the failed command manually using the command line printed by the build process (for example to run it in a debugger or under strace or something) produces completely unrelated errors. I guess there are some special environment variables or something but I can't seem to find any documentation of them if there are.
5. It's virtually impossible to google anything about rust. If I google say "rust fabricate" most of the results have nothing to do with the language.

Debian, Rust, and librsvg

Posted Nov 16, 2018 15:14 UTC (Fri) by Ieee754 (guest, #128655) [Link] (1 responses)

> It's virtually impossible to google anything about rust. If I google say "rust fabricate" most of the results have nothing to do with the language.

I have no idea what you expected from googling "rust fabricate", but if I just go to the documentation site of the toolchain: https://doc.rust-lang.org/rustc/index.html , there is a frickin book about how the Rust toolchain is built, how to run tests, debug issues, configure everything, the overall design of the compiler, the different modules, etc.

Debian, Rust, and librsvg

Posted Nov 17, 2018 2:02 UTC (Sat) by plugwash (subscriber, #29694) [Link]

> I have no idea what you expected from googling "rust fabricate"

I was hoping to find some documentation on what "fabricate" (which was hanging during the rustc build on one of our buildboxes, I suspect a bug in the arm32 compatibility layer of the arm64 kernel on said box) did and how to troubleshoot problems with it.

But instead I just found a load of results about metal fabrication :(

Someone did later tell me what fabricate was (apparently it's the rust-installer crate), though I haven't got around to trying to take a closer look. For now I have been working around the issue by doing our rust builds on a different system.

> there is a frickin book about how the Rust toolchain is built

Using the search tool in said book to search for fabricate doesn't seem to turn up any results.

Still thanks for the pointer.

Debian, Rust, and librsvg

Posted Nov 16, 2018 21:09 UTC (Fri) by josh (subscriber, #17465) [Link] (2 responses)

> 2. There are interdependencies between rust versions and cargo versions that aren't always obvious until you try particular combinations.

My understanding is that both Rust and Cargo only require the previous stable version at most, and should never have dependency loops between their latest versions.

> 4. After a failed build attempting to re-run the failed command manually using the command line printed by the build process (for example to run it in a debugger or under strace or something) produces completely unrelated errors. I guess there are some special environment variables or something but I can't seem to find any documentation of them if there are.

This is being fixed right now, and yes, there are environment variables. See https://github.com/rust-lang/cargo/pull/5683 for details.

Debian, Rust, and librsvg

Posted Nov 19, 2018 14:15 UTC (Mon) by dgm (subscriber, #49227) [Link] (1 responses)

> My understanding is that both Rust and Cargo only require the previous stable version at most, and should never have dependency loops between their latest versions.

Doesn't that imply that you (potentially) have to build each and every version in succession to get the latest one?

Debian, Rust, and librsvg

Posted Nov 20, 2018 11:57 UTC (Tue) by nix (subscriber, #2304) [Link]

Yes, but you don't have to do it when they come out: you can do it when you run into something that needs the latest version, and build the intermediate versions only for the sake of building the next in the chain (and then throw them away again). No need to ship, validate etc every version in turn.

Debian, Rust, and librsvg

Posted Nov 16, 2018 15:04 UTC (Fri) by Ieee754 (guest, #128655) [Link] (1 responses)

How do you build GCC from scratch without GCC ?

(what you are saying does not make sense for a toolchain)

Debian, Rust, and librsvg

Posted Nov 18, 2018 0:01 UTC (Sun) by JoeBuck (subscriber, #2330) [Link]

You build it with some other C compiler (for recent releases you need a C++ compiler, but if you only have a C compiler you can first build an older GCC release and then use that to build the latest version). If you don't have a C compiler for the platform, GCC works as a cross-compiler.

Debian, Rust, and librsvg

Posted Nov 20, 2018 5:56 UTC (Tue) by mirabilos (subscriber, #84359) [Link] (1 responses)

So one’s expected to port Every. Single. Ever. Released. version of rust to a new CPU architecture, when one comes along, in order to get it bootstrapped?

That sounds ridiculous.

Debian, Rust, and librsvg

Posted Nov 20, 2018 9:01 UTC (Tue) by Jonno (subscriber, #49613) [Link]

> So one’s expected to port Every. Single. Ever. Released. version of rust to a new CPU architecture, when one comes along, in order to get it bootstrapped?

Of course not, rust support cross-compilation perfectly well!

Sure, that means that if you want to re-verify the chain of trust from scratch you are going to need two computers (one of each arch), but that is a comparatively minor inconvenience...

Debian, Rust, and librsvg

Posted Nov 15, 2018 20:13 UTC (Thu) by josh (subscriber, #17465) [Link]

The edition will not affect that. "Rust 2018" isn't a long-term language target where you can implement *just* what the initial Rust 2018 release had and nothing more. There will be more language features after Rust 2018, in the months afterward, not just years.