Shrinking the kernel with a hammer

Posted Mar 7, 2018 2:13 UTC (Wed) by abufrejoval (guest, #100159)
Parent article: Shrinking the kernel with a hammer

<old-memories>
I remember running Microport Unix on my 80286 (fully loaded with 640K of base RAM) with an Intel Above Board that added 1.5MB of RAM, I believe (could have been 2MB). It also gave me a free 8087 math co-processor as a gift and a set of disks labelled "Microsoft Windows 1.01".

Since I dual booted it with DOS and the mapping between expanded (paged in a 64K "BIOS area" window in real mode) and extended (above 1MB range, available only in protected mode) mapping of RAM was set via DIP switches, I allocated around 50% to each. It meant a little more than 1MB of RAM overall for Microport.

Full UNIX (TM) file system, full multi-user (via serial ports), very much like a PDP-11 in fact, where Unix was born. The 286 had a MMU but not at a page, but segment level, again pretty much like a PDP-11. Of course UNIX System V, Release 2 didn't have 400 system calls and the kernel was statically built and linked. I did some fiddling with the serial drivers to have them support the internal queues of the UARTs, that avoided having to interrupt after every character set or received. That's what made 115kbit possible. Also fiddled with Adaptec bus master SCSI drivers.

Ah and it ran a DOS box, long before OS/2 ever did, one single DOS task which run in the lower 640k by resetting the 286 CPU via the keyboard controller on timer interrupts. The BIOS would then know via a CMOS byte, that the computer had in fact not just been turned on, but come back from protected mode: A glorious hack made possible by IBM for the PC-AT, so it could perform a RAM test after boot on systems which had more than 640K of RAM installed.

For kicks I ran a CP/M emulator in the DOS box, while running a compile job on the Unix...
</old-memories>

<other-old-memories>
A couple of years later I had to port X11R4 to a Motorolla 68020 system that ran AX, a µ-kernel OS somewhere between QNX and Mach. Basically a fixed demo system, that ran a couple of X applications on a true color HDTV display using a TI TMS34020 TIGA board. Had to make X11R4 true-color capable, too: It was only 1 and 8 bit color depth at that point.

MMU wasn't enabled on 68020, there was no file system and the µ-kernel just gave me task scheduling. So I had to write a Unix emulator, basically a library that emulated all system calls required by X11. The "file system" was a memory mapped archive (uncompressed) that just got included into the BLOB along with everything else.

Used a GCC 1.31 cross compiler on a Sun SPARC Station. After several months of working through the X11 source code to make it true color and run some accellerated routines on the TIGA GPU (TMS 34020 was a full CPU that used bit instead of byte addressing for up to 16MB of RAM with 32-bit addresses!) it just worked perfectly at first launch! Without any debugging facilities I'd have been screwed if it didn't...

Dunno what the 68020 had for RAM, but I doubt it was more than 1 or 2MB.
</other-old-memories>

So where it all comes together is that the process you describe is somewhat similar to turning Linux plus a payload into a Unikernel or Library OS, where everything not needed by the payload app is removed from the image.

I sure wouldn't mind if Linux could support that out of the box, including randomization of the LTO phase for RoP protection. And yes, I believe the GPL is not a good match for that.

The Linux build process must be one of the most wasteful things you can do on a computer, starting with the endless parsing of source files which have motivated Google to Go.

I keep dreaming about an AI that can take the Linux source code and convert it into something that is a Unikernel/LibraryOS image, which is only incrementally compiled/merged where needed when you change some kernel or application code at run-time.

I believe I'd call it Multics.

to post comments

Shrinking the kernel with a hammer

Posted Mar 7, 2018 18:34 UTC (Wed) by nix (subscriber, #2304) [Link] (10 responses)

Fascinating reminiscences, but..

The Linux build process must be one of the most wasteful things you can do on a computer

Oh God no. Compiler build processes with multiple-stage bootstrapping is the first thing that springs to mind (GCC building is *far* harder on a machine than Linux kernel building and most of it is thrown away); but then you look at new stuff like Rust, with, uh, no support for separate compilation or non-source libraries to speak of so everything you depend on is recompiled and relinked in statically for every single thing you build... the Linux build process is nice and trim. The oddest thing it does is edit module object files in a few simple ways after building.

Shrinking the kernel with a hammer

Posted Mar 9, 2018 16:43 UTC (Fri) by fratti (subscriber, #105722) [Link] (9 responses)

not to forget things like compiling a modern web browser. Chromium needs more than 20 GiB of disk space just to build, and you'll be at it for several hours on a modern system. Sure, ccache can save you some time, but yikes, talk about bad first contributor experiences.

Firefox has also been getting worse now that they're using some Rust. I genuinely hope Rust gets its ABI stuff sorted so that we do not end up living in a world where everything is a >2 MiB static binary in need of recompilation with every dependency update.

Shrinking the kernel with a hammer

Posted Mar 9, 2018 17:56 UTC (Fri) by excors (subscriber, #95769) [Link]

And don't forget Android, which often builds the Linux kernel, and most of a web browser or two, and some of Clang, and a thousand other things. I have several Android trees at about 200GB each. But a reasonable PC can still build the entire thing in under an hour, so it's not too bad really. The kernel itself is trivial.

Shrinking the kernel with a hammer

Posted Mar 10, 2018 0:48 UTC (Sat) by pabs (subscriber, #43278) [Link] (7 responses)

> static binary in need of recompilation with every dependency update.

Is that considered a feature in the Rust community like it is with Go?

Shrinking the kernel with a hammer

Posted Mar 10, 2018 6:42 UTC (Sat) by fratti (subscriber, #105722) [Link] (5 responses)

Yes, I've been told their religious beliefs state that the dynamic linker is Unsafe™.

Shrinking the kernel with a hammer

Posted Mar 10, 2018 8:13 UTC (Sat) by jdub (guest, #27) [Link] (3 responses)

Hrm, no, practically all Rust Linux binaries dynamically link to glibc by default (and by design), and you can easily dynamically link to C ABI shared libraries. If you want to build a static executable, you have to go out of your way to use the musl target.

There's nothing "unsafe" about dynamic linking, just the challenge of safety across C ABI boundaries (which exists for statically linked code as well) and the lack of a stable Rust ABI (which is pretty reasonable).

Shrinking the kernel with a hammer

Posted Mar 10, 2018 9:40 UTC (Sat) by fratti (subscriber, #105722) [Link] (2 responses)

I can understand that there is no stable Rust ABI, after all there's no stable C++ ABI either, but the issue is exaggerated by the Rust ecosystem's obsession with microdependencies (there are modules which are really just one function, à la npm), and the fast speed at which the Rust compiler moves.

>practically all Rust Linux binaries dynamically link to glibc by default (and by design)

Indeed, though as far as I know they statically link the Rust standard library. Despite the glibc being dynamically linked, e.g. oxipng still clocks in at 2.8M. Compare that to 86K for optipng.

Shrinking the kernel with a hammer

Posted Mar 11, 2018 17:10 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

> no stable C++ ABI

There isn't in the ISO standard sense, but there are de facto ABIs. GCC and MSVC have declared their ABIs long ago and stick to them. The Rust compiler does not commit to any given ABI between two releases. I suspect there may be one eventually, but it's not in the same area as C++.

Shrinking the kernel with a hammer

Posted Mar 12, 2018 10:48 UTC (Mon) by iq-0 (subscriber, #36655) [Link]

> [...] but the issue is exaggerated by the Rust ecosystem's obsession with microdependencies (there are modules which are really just one function, à la npm),

The reason you have to compile a lot of crates (rust libraries) while the thing you're building only uses a few parts of a few crates directly, has to do with how coherency-rules effectively cause many crates to depend on other crates in order to offer possibly relevant implementation of traits for there types or implementations of their traits on it's types.

To minimize the pain of these type/trait dependencies, and also to ease semver stability guarantees, a number of projects have extracted their basic types and/or traits in single purpose (and thus relatively small) crates. This helps these common crates to have few changes and reduce their compile times.

The fact that the crate dependency explosion often seems worse is due to different crates being able to have different (incompatible) dependencies on different versions of the same crate. Rust often handles these issues gracefully, which in many programming languages would have been painfull version conflicts, at the cost of sitting through additional crate compilations.

But to counter that, they only get build once for a project, unless you switch compiler versions, and thus often have the effect of reducing rebuild times. First time builds can be pretty long, but you only incur that cost occasionally. You do want to keep this in mind when configuring possible CI so that you cache these compiled dependencies.

> and the fast speed at which the Rust compiler moves.

Unless you really depend on the unstable (nightly) rust version the compiler normally is only updated every six weeks.

If you're using the unstable channel, you get to pick when you want to go through the bother of updating and thus recompiling everything. But I agree that that's hardly a consolation.

> Indeed, though as far as I know they statically link the Rust standard library. Despite the glibc being dynamically linked, e.g. oxipng still clocks in at 2.8M. Compare that to 86K for optipng.

All rust dependencies are, by default, statically linked, though LTO will prevent 90% of the standard library and other dependencies from being included in the final binary. A very large part of the resultant binary is debugging information (Rust's multi-versioning, types and module support has a big impact on the symbol length) and unwind information (in order to perform gracefull panics as opposed to plain aborts).

Both can be disabled and, with some effort, Rust binaries can be reasonably small. But things like monomorphization, while generating more optimized code, will almost always result in more code being generated. For most applications this usually isn't a big problem as the larger binaries don't really have a performance impact and greatly aid in error message information and debugging possibilities.

Luckily the people working on Rust support in Debian are working at making Rust programs integrate better with their distribution philosophy (dynamic linking, separating debug info and each dependency in a dedicated package), and I really hope that a number of their requirements and solutions will find their way back to the upstream Rust project.

Shrinking the kernel with a hammer

Posted Mar 10, 2018 8:19 UTC (Sat) by bof (subscriber, #110741) [Link]

"Yes, I've been told their religious beliefs state that the dynamic linker is Unsafe™."

Recently having had openSUSE tumbleweed running crond coredump on me until restarted due to weird DL loading of PAM stuff which was apparently updated, again makes me strongly sympatise with that sentiment...

Shrinking the kernel with a hammer

Posted Mar 11, 2018 17:15 UTC (Sun) by mathstuf (subscriber, #69389) [Link]

It is a feature for deployment, not so much for things one would expect from a distribution (i.e., development tools). One could do the same with C or C++ deployments, but it's a PITA to wrangle build systems in that stack without embedding dependencies, so "no one" does it. I suspect Rust (and not Go[1]) will get dynamic linking before C or C++ have viable "everything static" deployment solutions.

[1]AFAICT, Go has much more of a "non-Go code doesn't exist" mentality than Rust folks do for non-Rust code.

Shrinking the kernel with a hammer

Posted Mar 7, 2018 22:42 UTC (Wed) by anselm (subscriber, #2796) [Link] (2 responses)

The Linux build process must be one of the most wasteful things you can do on a computer

Crypto-“currency” mining?

Shrinking the kernel with a hammer

Posted Mar 8, 2018 8:52 UTC (Thu) by epa (subscriber, #39769) [Link] (1 responses)

In Linuscoin the computational challenge is to start with the SHA256 of a Linux kernel image and work out the combination of build options needed to produce it.

Shrinking the kernel with a hammer

Posted Mar 9, 2018 16:46 UTC (Fri) by fratti (subscriber, #105722) [Link]

Something semi-related:

https://marcan.st/2017/12/debugging-an-evil-go-runtime-bug/ (heading "Hash-based differential compilation")