LWN: Comments on "McIntyre: Bootstrapping arm64 in Debian"

McIntyre: Bootstrapping arm64 in Debian

wookey — Fri, 16 Jan 2015 15:03:27 +0000

Yes we have spent quite a lot of time getting cross-building in Debian working. In fact the initial arm64 port was the first Debian port bootstrapped from/on Debian (except that it was actually Ubuntu built entirely from Ubuntu, due to the Wheezy freeze and multiarch status, which for these purposes is very nearly the same thing).

That was cross-built with 'profiled' (reduced-dependencies) builds to get a build-essential system, then more cross (and a bit of native for recalcitrant) building to get more packages. See https://wiki.debian.org/Arm64Port#Status_details_and_history and http://lwn.net/Articles/540244/

Pabs has posted a few relevant links.

The problem with cross-building is as much to do with packaging as upstreams, but mostly because it's not tested by default, and tests usually cannot be run, cross-building is more fragile than native building. You _could_ make all of debian cross-buildable, but its a _huge_ undertaking, with some genuinely thorny bits. The current target is to have debian cross-bootstrappable, which means being able to cross-build the bottom ~150 packages of a build-essential image, and at least some of the 400 packages of the SCC (Strongly connected component) which is full of circular dependencies.

See http://bootstrap.debian.net/ for a pile of analysis on this.

There are issues with build-systems that know nothing of crossing, build-time tests which rely (often incorrectly) on the build hardware, cross-dependency satisfaction, tests, packaging cross-toolchains, etc.

We've done a lot of work on the underlying systems (multiarch, build profiles, cross toolchains, fixing up packages, which is actually in a pretty useable state in jessie, which has only taken about 4 years so far :-), but there are still some gaps. On my last test about 50% of the base set 'just crosses' OK in Jessie: http://people.linaro.org/~wookey/buildd/testing/sbuild/la...
(and we'd get a pile more if docbook-xml was marked 'multiarch:foreign' https://bugs.debian.org/732096). There are a few issues like that which hide just how slick cross- build-dependency installation now is.

More people interested in working on this are of course always welcome (#debian-bootstrap IRC, debian-cross mailing list). There are a lot of patches in the BTS which just need uploads and would improve the situation. I'd hope that Stretch (next release) will be genuinely bootstrappable-from-scratch, the whole core cross-buildable, with in-archive cross-toolchains and a generally rather slick setup.

So far the bootstrap-CI gets about 50 packages in at best: https://jenkins.debian.net/view/rebootstrap/

Steve, interesting server setup...

pixelpapst — Wed, 14 Jan 2015 17:21:39 +0000

When trying to visit your blog with HTTPS Everywhere enabled:

* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE AUTH=PLAIN] Dovecot ready.
GET BAD Error in IMAP command received by server.
Host: BAD Error in IMAP command received by server.
User-Agent: BAD Error in IMAP command received by server.
Accept: BAD Error in IMAP command received by server.
Accept-Language: BAD Error in IMAP command received by server.
Accept-Encoding: BAD Error in IMAP command received by server.
DNT: BAD Error in IMAP command received by server.
Connection: BAD Error in IMAP command received by server.
* BAD Error in IMAP command received by server.
* BYE Disconnected for inactivity.

Such fun ! :)

McIntyre: Bootstrapping arm64 in Debian

fb — Sun, 11 Jan 2015 13:20:46 +0000

I meant to say:

IIRC Nokia was the first to release a *Debian* Linux based device using dash as /bin/sh. (Again IIRC) Ubuntu was the first Linux desktop/"regular" distribution to do it. Debian followed soon after.

McIntyre: Bootstrapping arm64 in Debian

fb — Sun, 11 Jan 2015 13:18:16 +0000

I remember that the time required to boot my (old) desktop went down by a huge factor when I swapped /bin/bash by /bin/dash to be my "/bin/sh" (I'm not quoting numbers because I don't remember them exactly but it was a real significant difference.

To achieve that, all I had to do (I used to do this long before Ubuntu or Debian did it 'officially') was to issue a single dpkg-reconfigure command. I understand billion dollar companies can assign people to work for years on something to improve things further but it does not remove the merit of Debian/Ubuntu/Nokia early efforts to make Linux work better by default for end users (who then did not need to learn about bash/dash differences to get a machine that booted faster.

IIRC Nokia was the first to release a Linux based device using dash as /bin/sh. (Again IIRC) Ubuntu was the first Linux "regular" distribution to do it. Debian followed soon after.

McIntyre: Bootstrapping arm64 in Debian

ballombe — Thu, 08 Jan 2015 19:11:16 +0000

Did you try using distcc on a native host (or emulator) with distcc pointing to a cross-compiler on a fast machine, like I did with m68k ? This cut down cross building failure to almost none.

McIntyre: Bootstrapping arm64 in Debian

BlueLightning — Thu, 08 Jan 2015 09:49:33 +0000

Whilst it did seem like a painful and somewhat unnecessary change at the time it was introduced, there was one side benefit at least for another part of the Linux ecosystem - in embedded, where it's common to want to remove bash in favour of busybox (or at least know for sure when you do need to pull in bash). There's a lot more awareness of bash-specific shell syntax now and I believe that has translated to a cleanup of a lot of the common shell scripts on our systems such that they either use standard POSIX shell syntax or explicitly mark themselves as requiring bash with #!/bin/bash.

McIntyre: Bootstrapping arm64 in Debian

dgp — Thu, 08 Jan 2015 04:35:42 +0000

I think you're being a bit over dramatic. People are cross compiling code with GCC and friends on a daily basis.. there are plenty of platforms that GCC supports that it can't run on natively. There are also plenty of machines out there that can run Linux + busybox just fine but have nowhere near the resources needed to host a toolchain and compile a kernel.. tools like buildroot are designed with cross compiling in mind for that exact reason.

Most of the issues you'll see from cross compiling Linux applications are going to be issues with build tools trying to work stuff out and getting results for the host instead of the target. If you can find a situation where GCC generates different code when cross compiling you should report it.

McIntyre: Bootstrapping arm64 in Debian

pabs — Thu, 08 Jan 2015 03:19:07 +0000

New that we have in-archive cross-compilers we can do some automated cross-build QA and make some progress in fixing that stuff, yay!

McIntyre: Bootstrapping arm64 in Debian

thedevil — Thu, 08 Jan 2015 03:13:53 +0000

Subtle trooling is also trolling.

McIntyre: Bootstrapping arm64 in Debian

HelloWorld — Thu, 08 Jan 2015 00:46:29 +0000

> Swapping out bash for dash as system shell was a very good move
Actually, it was a move that nobody really cared about and that caused infinite amounts of pointless breakage. Now, if they had tried to remove shell scripts from the booting process altogether there would have been a point, but they didn't. Oh well, someone else did by now.

McIntyre: Bootstrapping arm64 in Debian

rahvin — Wed, 07 Jan 2015 18:30:05 +0000

Regardless of whether you can cross-compile, when you first start the architecture you need to know if you can actually compile on the architecture at some point. It was my understanding from having experimented with cross-compilation that bugs can appear that aren't there under native compile and bugs that will be there under native compile won't be there.

In other words even if you get a clean cross-compile you may not have package that compiles on the actual architecture and there is a risk it will have problems that wouldn't exist under native compilation. At the time I was fooling around with it Gentoo warned heavily against attempting to use it for anything but an emergency because of the risks. You can also pretty well write off any help from upstream if the compile fails as soon as they find out you were cross-compiling. At least that was my experience at the time though I haven't used Gentoo is more than a decade at this point.

McIntyre: Bootstrapping arm64 in Debian

stevem — Wed, 07 Jan 2015 15:16:31 +0000

*grin* I wonder exactly who that one is, then. I have my suspicions...

McIntyre: Bootstrapping arm64 in Debian

stevem — Wed, 07 Jan 2015 15:15:16 +0000

Exactly, yes. We end up having to cross-build quite a few of the packages to start with when doing the very early bits of bootstrapping a port, but we don't build the final packages that way. There are *so* many packages that don't cross build correctly that we can't rely on it.

McIntyre: Bootstrapping arm64 in Debian

pboddie — Wed, 07 Jan 2015 15:09:37 +0000

As far as I recall, the Debian goal is to be able to compile the distribution on each of the supported platforms. Supporting cross-compilation doesn't remove the need to bring everything up on real hardware. It obviously does mean more work, too, but the Emdebian people have managed to make some steps towards it possibly happening for the entire archive one day. Personally, I find build systems that sniff around and run things on the build host in order to configure the built software, assuming that the architectures or environments involved are the same ("Hey! I just built a program... running it now!"), to be annoying.

For certain architectures, cross-compiling the distribution (or at least cross-building packages yourself) would be highly beneficial: there are plenty of fast x86-64 machines out there with lots of RAM that would make light work of it, whereas suitably capable hardware is scarce or almost non-existent for some platforms. Suggestions like "Raspberry Pi build farm!!1!" sound like quick and easy remedies until you hit limits on RAM or just get frustrated at the general lack of performance.

McIntyre: Bootstrapping arm64 in Debian

pabs — Wed, 07 Jan 2015 14:58:46 +0000

Debian has had Win32 and other cross-compilers for a long time but we haven't had cross-compilers for Debian architectures until 2014 and unfortunately the work happened too late to be included in Debian jessie. Hopefully future Debian ports will be able to be fully automatically bootstrapped from cross-compilers, some links:

https://wiki.debian.org/CrossToolchains
https://wiki.debian.org/DebianBootstrap

McIntyre: Bootstrapping arm64 in Debian

MKesper — Wed, 07 Jan 2015 14:38:25 +0000

Swapping out bash for dash as system shell was a very good move, reducing needed system resources and attack vectors (think shellshock).
NB: The standard interactive shell still is bash.

McIntyre: Bootstrapping arm64 in Debian

epa — Wed, 07 Jan 2015 13:21:45 +0000

Yes, understandable. I expected that of all the quixotic quests Debian has gone on over the years (allowing you to swap out the kernel for a FreeBSD kernel, changing the standard shell to dash when every other Linux uses bash, etc) cross-compiling the whole distribution would be among them. But it depends on what individual developers are motivated to work on. Chasing broken upstreams to fix cross-compilation could be a full-time job.

McIntyre: Bootstrapping arm64 in Debian

gb — Wed, 07 Jan 2015 12:40:38 +0000

There is 1 arm64 system at popcon =)

McIntyre: Bootstrapping arm64 in Debian

kevinm — Wed, 07 Jan 2015 11:36:29 +0000

I believe that Debian package are built natively because a great deal of those 21,000 upstream packages simply can't be relied on to build correctly when cross-compiled.

It's understandable why that is - upstream developer compile natively, and a cross-compile-clean build system is not really something that users tend to be clamouring for.

McIntyre: Bootstrapping arm64 in Debian

epa — Wed, 07 Jan 2015 10:40:19 +0000

Do I understand rightly that they needed to use arm64 hardware (physical or simulated) in order to build arm64 packages? I expected that cross-compilation would be one of the things Debian is good at.

(Obviously it is important that building on the target platform work, but you don't need to grind through the whole Debian archive that way... build most of it with a cross-compiler and then sanity check that native building gives the same results for a small sample.)