LWN.net Weekly Edition for January 7, 2021

Welcome to the LWN.net Weekly Edition for January 7, 2021

This edition contains the following feature content:

• Some unlikely 2021 predictions: an ill-advised tradition continues for another year.
• Bootstrappable builds: reproducible builds are nice, but what if you didn't have to rely on an existing compiler to create them?
• LibreSSL languishes on Linux: an effort to create a cleaned-up TLS library fails to gain traction on Linux systems.
• 5.11 Merge window, part 2: the rest of what was merged for the 5.11 kernel release.
• Portable and reproducible kernel builds with TuxMake: a new tool to help with kernel building.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Some unlikely 2021 predictions

Just because something is traditional does not imply that it is necessarily a good idea. As a case in point, consider LWN's tradition of starting the year with some predictions for what is to come; some may be obvious while others are implausible, but none of them are reliable. Nonetheless, we've been doing this since 2002 so we can't stop now. Read on for our wild guesses as to what might transpire in 2021.

With luck, the world will emerge from the depths of the pandemic this year. In many ways, the effects of the pandemic on the free-software community were relatively minor, so it is to be expected that the recovery will not change much either. We will continue to create great software as always. The recovery may mean, though, that we can start meeting in-person later in the year, which is important for the long-term health of our community. That said, many people will likely remain reluctant to travel, and companies may discover a reluctance to pay for a return to travel. So many of our meetings will have, at least, an online component for the entire year.

The other thing that may happen, though, as the world opens up, is a conclusion by some in our community that life is too short and precarious to spend it tied to a keyboard. It would not be surprising to see retirements increase over the course of the next year or two.

Support for CentOS 8 will end at the end of the year; users will have to transition to CentOS Stream or find another solution altogether. For all the screaming, CentOS Stream may well turn out to be good enough for many of the deployments that are currently using a stable CentOS build. Others are likely to find that, in this era of cloud computing, a long-term-stable distribution isn't as important as it used to be. If the "machines" running the distribution will not last for years, why does the distribution they run need such a long life? The end of CentOS could have the unintended effect of undermining the demand for ultra-stable "enterprise" distributions in general.

There will be attempts to recreate CentOS as it was, of course; most or all of them are likely to fail. Maintaining a stable distribution for years takes a lot of work — and tedious, unrewarding work at that. CentOS struggled before Red Hat picked it up; there is no real reason to believe that its successors will have an easier time of it. The fact that the alternative with the most mindshare currently, Rocky Linux, has no publicly archived discussions and only seems to communicate on the proprietary Slack platform is also worrisome.

For better or for worse, the Fedora project has a well-established relationship with Red Hat. The status of openSUSE is nowhere near as clear, which is one of the causes of the ongoing strife on its mailing lists over the last year. OpenSUSE will need to better define its relationship with SUSE in 2021, even if additional stresses, such as the creation of the independent openSUSE Foundation or the rumored public offering by SUSE, don't happen. Like Fedora, openSUSE is the descendant of one of our earliest and most influential distributions; it will be with us for a long time yet, but exactly how that will happen needs to be worked out.

It will become possible to submit kernel patches without touching an email client — but few people will do that in 2021. The kernel community will, eventually, be dragged into more contemporary ways of doing development. The kernel project's current processes are there for a reason, though; few other projects have anything near the kernel's developer or patch counts. Some significant thought will have to go into making "modern" development processes work at kernel scale. As has happened in the past, the result may be innovations that are felt far beyond the kernel community.

The commercial side of BPF will become more prominent in 2021. BPF, which allows code to be loaded into and executed within a running kernel, has been growing rapidly in power and adoption over the last several years. This year, we'll see how companies are using it to build their products and services. BPF makes it much easier to add interesting functionality to the kernel, but it also serves to keep code implementing that functionality separate from the kernel source. Our systems in the future may be more flexible and capable, but they may also thus be a bit more proprietary, even if all the code is ostensibly free.

GNOME 40 will bring a new shell interface (described here). It's a GNOME interface change, so a fair amount of loud complaining is inevitable; people had finally started to make peace with the current GNOME shell, after all. And yes, it will be GNOME 40 rather than GNOME 3.40.

Python developers will have to think hard about the future of the language. The Python 3.0 release happened twelve years ago now, and most of the angst over moving from Python 2 is behind us. There is a reasonable case to be made that, to a great extent, the Python 3 language is "done" and need not continue to undergo significant change. On the other hand, proposed features like structural pattern matching show that some developers still have an appetite for big changes. One can safely predict that there will be no disruptive Python 4 release anytime soon; what is harder to predict is when developers wanting stability will start to put the brakes on attempts to continue to evolve Python 3.

Software supply-chain attacks will be a serious threat for the community this year. The SolarWinds attack, which was used to compromise a number of US government agencies, was carried out by slipping the malware payload into a routine software update. We can read this SolarWinds blog entry from 2019 with amusement; it claimed that open-source software makes one's chance of downloading malicious software "much higher". That post has not aged well, but this attack could also happen with free software, which is distributed in binary form through a large number of trusted channels. Malicious code inserted into one of those supply chains could be used with devastating effect; we can only hope that the suppliers we trust are truly trustworthy.

Antitrust enforcement against companies like Facebook and Google in the US and Europe will pick up in 2021. The long-term result of these efforts could be huge, and not necessarily good or bad for our community, but the glacial pace of the courts will keep anything serious from happening this year. Being under the antitrust microscope may make those companies, which are significant contributors to the free-software community, more circumspect in their actions, though. If a software contribution could look like an attempt to further entrench a monopoly, it may just not happen.

OpenStreetMap will continue to grow in importance as companies realize that it provides the best way to compete with Google Maps. As a result, welcome resources will pour into the project; indeed, it would appear that corporate-sponsored contributions are already the majority of the edits going into the OpenStreetMap database. Inevitably, there will be clashes with the hobbyists who built OpenStreetMap up from the beginning, but the end result should be good for everybody involved. As with software, free data is better when everybody works to improve it.

Through all of this, Linux and free software will just be stronger at the end of 2021. That trend has held for decades, through economic crises, terrorist attacks, a pandemic, the dotcom crash, and more; predicting its continuation should be a safe thing to do.

Finally, these predictions will be reviewed and duly mocked in the December 23, 2021 LWN Weekly Edition — the other half of this tradition of ours. That is a small part of our larger mission of reporting from within the free-software community — a tradition that is about to begin its 24th year. Certainly none of us predicted that back at the start, but it is safe to say we'll still be at it when the year ends. As always, many thanks to all of you who have supported our work for all of these years; you are the reason that we are still at it. Best wishes to all LWN readers for a safe and rewarding 2021.

Comments (67 posted)

Bootstrappable builds

The idea of Reproducible Builds—being able to recreate bit-for-bit identical binaries using the same source code—has gained momentum over the last few years. Reproducible builds provide some safeguards against bad actors in the software supply chain. But building software depends on the tools used to construct the binary, including compilers and build-automation tools, many of which depend on pre-existing binaries. Minimizing the reliance on opaque binaries for building our software ecosystem is the goal of the Bootstrappable Builds project.

For example, GCC is written in C and C++, which means that it requires compilers for those two languages in order to be built from source. In practice, that generally means a distribution would use its existing binary executables of those tools to build a new GCC version, which would then be released to users. One of the concerns with that approach is described in Unix inventor Ken Thompson's Turing Award lecture "Reflections on Trusting Trust" [PDF]. In a nutshell, Thompson said that trusting the output of a binary compiler is an act of faith that someone has not tampered with the creation of that binary—even if the source code is available.

The Bootstrappable Builds project was started as an offshoot of the Reproducible Builds project during the latter's 2016 summit in Berlin. A bootstrappable build takes the idea of reproducibility one step further, in some sense. The build of a target binary can be reproduced alongside the build of the tools required to do so. It is, conceptually, almost like building a house from a large collection of atoms of different elements.

While it is obviously an interesting intellectual puzzle, bootstrapping a Linux distribution from the ground up is a lot of work—and the benefits may not be immediately apparent. The project has a web page outlining the benefits, which are largely about security and portability of the source code. For users, bootstrapping and reproducibility help provide protection against malicious backdoors, while distributions and tool developers will have an easier path in porting code to new architectures.

Since C is at the heart of much of the open-source ecosystem, having a way to bootstrap a C compiler, such as GCC, is among the projects that Bootstrappable Builds is pursuing. One such effort is maintaining a subset of GCC version 4.7, which is the last version that can be built with only a C compiler. GCC 4.7 will be easier to bootstrap from simpler C compilers, such as the Tiny C Compiler (TinyCC or tcc), without requiring a C++ compiler too.

A related effort revolves around GNU Mes, which is the combination of a Scheme interpreter written in C and a C compiler written in Scheme. The two parts are mutually self-hosting, so one can be built from the other (or from a separate binary C compiler or Scheme interpreter). This has been used to halve the size of bootstrap binaries (or "seeds") required to create a version of the GNU Guix distribution:

While that has greatly reduced the amount of binary code that is needed to create a distribution from scratch, there are plans to go even further. Stage0 is a project aimed at bootstrapping from a truly minimal base: a less-than-500-byte hex monitor ("How you create it is up to you; I like toggling it in manually myself"). That monitor implements a simple hex-code-to-binary translator that can be used to build ever-more complex binaries, some of which are available from the project repository.

In a recent posting to the bootstrappable mailing list, Jan Nieuwenhuizen reports on some progress. Mes was eliminated as a bootstrap seed for Guix by building it starting from the hex-code tool (hex0). There are also efforts outside of Guix to bootstrap a system just using two small seeds (including the hex0 tool); it is currently able to build Mes and TinyCC is in progress.

In addition, at the 2019 Reproducible Builds summit, three distributions created bit-for-bit identical binaries of Mes using three different versions of GCC. Guix, NixOS, and Debian first built Mes with GCC, then built it again using that Mes, which resulted in identical binaries. As noted by David A. Wheeler, that exercise was a real-world application of his diverse double-compiling (DDC) approach to countering Thompson's "trusting trust" attack.

By the sound of things, bootstrappability is super fiddly, low-level work. It is not for everyone, but it is important. If we can ensure that the foundations of our software ecosystem are sound, and build up from there, we can be reasonably certain that there is no backdoor hiding in our build tools and subverting everything else. That is a great outcome, but it only pushes the problem down a level, in truth. Some kind of hardware or firmware backdoor could still be lurking. Solutions to that problem will be rather more difficult.

[Thanks to Paul Wise for suggesting the topic.]

Comments (26 posted)

LibreSSL languishes on Linux

The LibreSSL project has been developing a fork of the OpenSSL package since 2014; it is supported as part of OpenBSD. Adoption of LibreSSL on the Linux side has been slow from the start, though, and it would appear that the situation is about to get worse. LibreSSL is starting to look like an idea whose time may never come in the Linux world.

OpenSSL provides the low-level plumbing for a number of important cryptographic functions; it provides TLS and SSL implementations and a number of utilities for functions like key generation and signing. Most programs that need to communicate securely over the network end up linking to OpenSSL for that functionality. OpenSSL has always had a bit of a strange position in the Linux world due to its special license, which contains an advertising requirement that is deemed to be incompatible with the GNU General Public License. To get around this problem, many GPL-licensed programs include a special exception allowing linking to OpenSSL.

Our systems are built upon a frightening number of crucial components that everybody uses, but which nobody maintains. In 2014, the disclosure of the Heartbleed vulnerability made it clear that OpenSSL was one of those components; it had languished with minimal development effort for years. As a result, the OpenSSL code base had filled up with unmaintained cruft — and catastrophic vulnerabilities. Heartbleed was seen as being bad enough that Something Had To Be Done to improve that situation.

One of the things that was done was the establishment of the Core Infrastructure Initiative (CII) by the Linux Foundation; its goal was to direct resources to crucial components that were insufficiently supported. The CII has faded somewhat over the years, but it did succeed in bringing developers to projects like OpenSSL and patching over many of our worst problems with unsupported infrastructure.

The OpenBSD project, as is its wont, took its own approach; the result was the LibreSSL fork. Within a couple weeks of the Heartbleed announcement, the LibreSSL developers claimed to have removed about half of the code forked from OpenSSL. The OpenBSD distribution switched over to the new library, and the project seemed to be off to a running start. Over nearly six years, LibreSSL has remained a viable project, producing regular releases as it goes. Since the 2.9.0 release at the end of 2018, the project has merged 1,554 patches from 36 developers, showing that it is definitely alive and getting work done.

The OpenSSL project, though, has merged over 5,000 patches during approximately the same time period; that work came from 276 developers. Just as importantly, much of that work is supported by organizations that depend on OpenSSL; large contributors include Oracle, Siemens, Akamai, Red Hat, IBM, VMware, Intel, and Arm — along with the OpenSSL Software Foundation itself. This level of support has enabled the OpenSSL project to address many of its longstanding problems; by 2016, the project was on a much more stable footing. Security problems still exist, of course — this is software we are talking about, after all — but they are dealt with in a coordinated way and people don't worry about OpenSSL as they once did.

One result of all this work is that Linux distributions have, in general, not shifted away from OpenSSL. Two distributions that did attempt to provide LibreSSL support were Alpine Linux and Gentoo. Alpine Linux supported LibreSSL as its primary TLS library for a while, but switched back to OpenSSL with the 3.9.0 release in January 2019. Gentoo never tried to switch over completely, but it supports LibreSSL as an alternative.

That support will end in February, though. Gentoo developer Michał Górny first suggested this change at the end of December, saying that LibreSSL offers no benefit over OpenSSL at this point while imposing a lot of costs. In particular, he complained about the large number of packages that require patches to work with LibreSSL and the constant stream of regressions that the project must deal with. Various other developers were quick to support this change. Hanno Böck, who claimed to be the first to have built Gentoo with LibreSSL, said:

Anthony Basile, who is the current lead for LibreSSL on Gentoo, agreed, saying that LibreSSL is "more of a hassle than it's worth".

The primary opposition came from Peter Stuge, who asserted that "Gentoo is about choice and this particular component is one of the most important in a system". He also made the point that monocultures are not healthy in any part of the system, so it is good to support alternatives like LibreSSL. Even Stuge seemed to agree, though, that supporting LibreSSL at its current level was not sustainable and didn't seem to make sense. But he thought LibreSSL should remain part of the distribution for those who want to go to the trouble to try to use it.

Even that may be a bit challenging, since it is not possible to install both OpenSSL and LibreSSL on the same system. But that is the approach that is being taken; LibreSSL will remain in the Gentoo repository — for now — but it will be "masked" to prevent its installation by users who have not taken extra action to enable it. As noted in the announcement, the Gentoo project will also stop carrying patches to make other packages work with LibreSSL:

It has now become difficult indeed to find a Linux distribution that is still trying to support LibreSSL over OpenSSL.

One might try to argue that the LibreSSL fork has failed, but that is clearly not the case; it is under active development and is used by at least a couple BSD variants. One could argue, though, that this fork was done too soon. This project was created only a couple of weeks after the disclosure of Heartbleed, which seems rather too early to conclude that the problems with OpenSSL itself could not be addressed without a fork. OpenSSL turned out not to be as unfixable as it seemed, and OpenBSD is saddled with the cost of carrying its own fork rather than benefiting from the rejuvenated OpenSSL effort.

Perhaps, someday, OpenSSL will run aground again and the world will be glad that there is a LibreSSL project out there to fall back to. Alternatives are good, and the ability to create those alternatives is one of the great strengths of free software. But alternatives can also be expensive to support, and it would appear that the Linux community has decided that this particular alternative is not worth the price.

Comments (72 posted)

5.11 Merge window, part 2

Linus Torvalds released the 5.11-rc1 prepatch and closed the 5.11 merge window on December 27. By that time, 12,498 non-merge changesets had been pulled into the mainline; nearly 2,500 of those wandered in after the first merge-window summary was written. Activity slowed down in the second week, as expected, but there were still a number of interesting features that found their way into the mainline.

Architecture-specific

• The RISC-V architecture has gained support for the contiguous memory allocator, strict protection for /dev/mem, and interrupt-time accounting.
• KASAN is now supported on 32-bit Arm systems.
• KASAN can now use the 64-bit Arm memory tagging extension rather than software tagging and shadow memory.

Core kernel

• The epoll_pwait2() system call, which enables higher-resolution timeouts, has been added.

Filesystems and block I/O

• The F2FS filesystem has gained new ioctl() calls giving user space control over which files are stored in compressed form. There is a new compress_mode= mount option controlling whether decisions about compression are made in kernel or user space.
• Overlayfs filesystems can now be mounted within a user namespace by an otherwise unprivileged process. According to this commit message, a full audit of overlayfs functions has been performed to try to ensure that overlayfs cannot be used to gain new privileges.
• The Ceph filesystem has gained support for the "msgr2.1" protocol, which enables encryption of transmitted data using AES-GCM.
• The XFS filesystem has a new "needs repair" bit; if this bit is set by the kernel, further mounts of that filesystem will be refused until it has been given a clean bill of health by the xfs_repair utility.
• The dm-multipath device-mapper target has a new path selector that takes CPU affinity into account when choosing how to route I/O requests. The intent is to preserve CPU locality for systems that have been configured with that in mind, with possibly significant improvements in throughput.

Hardware support

• Clock: Qualcomm SC7180 camera clock controllers, Qualcomm SDX55 global clock controllers, Freescale FlexSPI clocks, and SiFive FU740 PRCI clock controllers.
• Miscellaneous: ARM MHUv2 mailbox controllers, Qualcomm Application Data Mover (ADM) DMA controllers, Qualcomm GPI DMA controllers, MStar MSC313 GPIO controllers, HiSilicon GPIO controllers, Intel LGM and Keem Bay pulse-width modulators, Synopsys DesignWare pulse-width modulators, and MIPI Alliance I3C host controller interfaces.

Security-related

• A new configuration option, concisely called "DM_VERITY_VERIFY_ROOTHASH_SIG_SECONDARY_KEYRING", allows the dm-verity subsystem to verify hash signatures against certificates in the secondary trusted keyring. In practice, that allows verification against certificates loaded at run time rather than those built into the kernel itself; that, in turn, allows updating certificates without having to update the whole kernel.

Virtualization and containers

• User-mode Linux now supports suspend-to-idle; a suspended instance can be resumed with a SIGUSR1 signal.
• The virtio-mem mechanism, which enables adding memory to (or removing memory from) virtual machines, has gained support for "big block mode". In short, this allows memory to be added or removed in units larger than the host kernel's memory-block size, facilitating the use of huge pages (in the 1GB range) for guest memory.

Internal kernel changes

• The kernel build system will now fail with an error when building loadable modules lacking a MODULE_LICENSE() declaration.
• The use of EXPORT_SYMBOL() on static functions never made much sense; now it will generate an error at build time.

Now the time has come to, as Torvalds said in the -rc1 announcement, "go kick the tires" and find the bugs that were inevitably added with all of that code. That process should lead to a reasonably stable 5.11 release that will, with luck, be released as a special Valentine's Day present in February.

Comments (6 posted)

Portable and reproducible kernel builds with TuxMake

TuxMake is an open-source project from Linaro that began in May 2020 and is designed to make building Linux kernels easier. It provides a command-line interface and a Python library, along with a full set of curated portable build environments distributed as container images. With TuxMake, a developer can build any supported combination of target architecture, toolchain, kernel configuration, and make targets.

Building a Linux kernel is not difficult. Follow the documentation, install the dependencies, and run a couple of make commands. However, if a developer wants to build for multiple architectures, with multiple toolchains, things get complicated quickly. Most developers and maintainers have a set of custom scripts that they have written and maintained to perform their required set of builds. TuxMake provides a common layer of abstraction to reduce the need for every developer to write their own build scripts.

TuxMake publishes containers for various toolchain/architecture combinations. These containers eliminate the need for individual developers to source and install multiple toolchains and toolchain versions on their systems. It also makes builds reproducible and portable because now the environment in which a kernel is built is versioned and shareable across the internet and on mailing lists.

TuxMake has two goals. First, remove the friction that may cause developers, especially new developers, to skip build testing for uncommon toolchain/architecture combinations. Second, to make it easier for builds and build problems to be described and reproduced.

Features

Architectures supported are arc, arm, arm64, i386, mips, parisc, powerpc, riscv, s390, sh, sparc, and x86_64. Toolchains supported are GCC versions 8, 9, and 10, and Clang versions 10, 11, and nightly-latest. Targets supported for all relevant architectures are the kernel config (Kconfig), kernel image, modules, device tree binaries (DTBs), and a debug kernel image. The TuxMake team plans to add additional targets including kselftest, cpupower, perf, and even documentation.

Containers and container runtimes that are compliant with Open Container Initiative (OCI) specifications are supported to enable build portability and reproducibility. Both Docker and Podman runtimes are supported and interchangeable, depending on each user's preference. Podman is a popular alternative to Docker because it is daemonless and does not require root privileges. Additional container runtimes can be added as needed.

How does it work?

In a terminal on a Linux system with TuxMake, you can navigate to a Linux kernel source directory where you would usually run make and instead run tuxmake. Without any arguments, tuxmake will perform a build using default values for all options. It will look something like the following:

    $ tuxmake
    # to reproduce this build locally: tuxmake --target-arch=x86_64 \
        --kconfig=defconfig --toolchain=gcc --wrapper=none --runtime=null \
        config kernel xipkernel debugkernel modules dtbs

First, the tuxmake reproducer command is printed with all of the arguments provided. This information is helpful for reproducing the build and sharing with colleagues.

    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp defconfig

A .config is built using the defconfig target. Note that, by default, it is saved to a unique directory that is automatically created under ~/.cache/tuxmake. All intermediary files and build artifacts will be stored there.

    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp
    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp bzImage

The default kernel as well as a bzImage will be built; it might seem redundant to explicitly build bzImage for x86_64 since the previous make invocation should already do that. As it happens, not all architectures handle this consistently. Rather than special case those architectures and bake in knowledge about quirks in the kernel tree, tuxmake explicitly builds the final image. In most cases that is a no-op.

    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp vmlinux
    xz -T0 --keep /home/drue/.cache/tuxmake/builds/676/tmp/vmlinux

A debug kernel image will be built, saved to the output directory, and compressed with xz. As before, make vmlinux might look redundant given vmlinux is already built by make. However, the debug kernel image can be built independently. In a full build, make vmlinux will be a no-op, but if only the debug kernel is being built, it will be the main build step.

    grep -q CONFIG_MODULES=y /home/drue/.cache/tuxmake/builds/676/tmp/.config
    make --silent --keep-going --jobs=16 \
        O=/home/drue/.cache/tuxmake/builds/676/tmp modules_install INSTALL_MOD_STRIP=1 \
        INSTALL_MOD_PATH=/home/drue/.cache/tuxmake/builds/676/tmp/modinstall
    tar -caf /home/drue/.cache/tuxmake/builds/676/tmp/modules.tar.xz \
        -C /home/drue/.cache/tuxmake/builds/676/tmp/modinstall lib

If kernel modules are enabled in the build config, modules will be built, gathered using modules_install, and stored in a tar.xz file in the output directory.

    I: config: PASS in 0:00:01.305649
    I: kernel: PASS in 0:01:31.887716
    I: debugkernel: PASS in 0:00:08.207033
    I: modules: PASS in 0:00:00.869124
    I: build output in /home/drue/.cache/tuxmake/builds/676

Finally, each target's build status (PASS/FAIL/SKIP) and build time are displayed, along with the path to the build's output directory.

Containers are not required, and are not used by default. When running without a container runtime, TuxMake will use locally available toolchains. When a container runtime is specified, TuxMake will download a container image at build time (if necessary) and perform the build inside the container. It does this by mounting the Linux source directory and the build output directory into the container and performing each build step in an ephemeral container — the container only runs for the duration of the build step and then exits.

Putting it all together, here is a more elaborate example of using Podman to build an arm64 kernel using Clang with KASAN enabled:

    $ tuxmake -r podman -a arm64 -t clang-11 -k defconfig -K CONFIG_KASAN=y -w ccache
    # to reproduce this build locally: tuxmake --target-arch=arm64 --kconfig=defconfig \
        --kconfig-add=CONFIG_KASAN=y --toolchain=clang-11 --wrapper=ccache \
        --runtime=podman --image=tuxmake/arm64_clang-11 \
	config kernel xipkernel debugkernel modules dtbs

When podman is specified as the runtime, TuxMake will use Podman to perform the build. The kernel will be built for aarch64 (-a arm64) with Clang version 11 (-t clang-11). The kernel configuration will be generated using the defconfig target, then KASAN will be explicitly enabled. Finally, ccache will be enabled (-w ccache) to reduce the build time.

    Trying to pull docker.io/tuxmake/arm64_clang-11...

A container for arm64_clang-11 is fetched from TuxMake's public container registry at hub.docker.com/u/tuxmake, if needed.

    # CONFIG_KASAN=y -> /home/drue/.cache/tuxmake/builds/685/tmp/0.config
    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
        ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
        'HOSTCC=ccache clang' 'CC=ccache clang' defconfig
    scripts/kconfig/merge_config.sh -m -O /home/drue/.cache/tuxmake/builds/685/tmp \
        /home/drue/.cache/tuxmake/builds/685/tmp/.config \
	/home/drue/.cache/tuxmake/builds/685/tmp/0.config
    Using /home/drue/.cache/tuxmake/builds/685/tmp/.config as base
    Merging /home/drue/.cache/tuxmake/builds/685/tmp/0.config
    #
    # merged configuration written to /home/drue/.cache/tuxmake/builds/685/tmp/.config (needs make)
    #
    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
        ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
	'HOSTCC=ccache clang' 'CC=ccache clang' olddefconfig

The .config is generated by building defconfig and then using merge_config.sh to merge in any config fragments that were specified. The rest of the build proceeds as expected. The only difference from the first example is the addition of building DTBs, since it is an arm64 kernel.

    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
        ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
	'HOSTCC=ccache clang' 'CC=ccache clang' dtbs
    mkdir -p /home/drue/.cache/tuxmake/builds/685/tmp/dtbsinstall/dtbs
    make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
        ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
	'HOSTCC=ccache clang' 'CC=ccache clang' dtbs_install \
	INSTALL_DTBS_PATH=/home/drue/.cache/tuxmake/builds/685/tmp/dtbsinstall/dtbs
    tar caf /home/drue/.cache/tuxmake/builds/685/tmp/dtbs.tar.xz \
        -C /home/drue/.cache/tuxmake/builds/685/tmp/dtbsinstall dtbs

The result of the DTBs build is a dtbs.tar.xz artifact in the output directory, containing all of the DTBs.

Specifying and managing Linux kernel build configuration is a difficult problem. TuxMake provides a --kconfig argument, which defaults to defconfig. When specified, the given config target (such as tinyconfig, allmodconfig, or allnoconfig) will be built. If a path or URL of a pre-built config file is specified, it will be used directly.

Additional Kconfig options can be specified by one or more --kconfig-add arguments. --kconfig-add takes a path or URL to a config fragment or a literal Kconfig string such as CONFIG_KASAN=y. If any URLs are provided, they will be downloaded. All of the configs specified are merged using scripts/kconfig/merge_config.sh and make olddefconfig.

After a build, the build directory will contain a build log, a compressed kernel image, the kernel config, a metadata file in JSON that describes every aspect of the build and the build environment, a system map, a compressed debug kernel, a compressed modules tar file if applicable, and a compressed DTB tar file if applicable. Additional artifacts will be added as additional targets are implemented.

Portability and reproducibility

Because TuxMake can use shareable build environments, and things like Kconfig can be specified using URLs, TuxMake build commands can be shared with others. In the case of reporting a build problem to a mailing list, a one-line TuxMake command communicates precise instructions for reproducing the build problem. Any user who runs the same TuxMake command against the same Linux source tree will see the same build.

Full bit-for-bit reproducible builds are also supported, but require additional build arguments. First, it is critical that the exact same container is used. This can be accomplished with the --image argument which can take a full explicit container path, including sha256 digest. Second, environment variables KBUILD_BUILD_TIMESTAMP, KBUILD_BUILD_USER, and KBUILD_BUILD_HOST must be set using -e because they influence the resulting kernel image. In the normal case, the resulting build binaries will match bit for bit as long as all of the above inputs are the same (aside from the additional reproducible build caveats detailed at kernel.org).

For example, the following command run on an x86_64 host against Linux sources at tag v5.10 will result in a bzImage sha256 that begins with 8d066f679eac. It works with both -r podman and -r docker:

    $ tuxmake --image \
        docker.io/tuxmake/x86_64_gcc@sha256:f8218cbfad8ecf6628fc44db864a402070feb87ff43a880e1409649172d4bc8c \
        -r podman -k tinyconfig \
	-e "KBUILD_BUILD_TIMESTAMP='Tue May 26 16:16:14 2020 -0500'" \
	-e "KBUILD_BUILD_USER=tuxmake" \
	-e "KBUILD_BUILD_HOST=tuxmake"

Note that this was run using TuxMake 0.11.0, and should work for the foreseeable future. However, it is possible that future versions of TuxMake will introduce additional default variables into the build environment that break the backward compatibility of this example.

Quality

It is critical that TuxMake’s behavior is obvious, transparent, and reliable. Without basic trust and quality, such a tool is more trouble than it's worth. TuxMake enforces 100% unit test coverage, meaning that every line of code is covered by at least one basic test. Additionally, it incorporates comprehensive integration testing using an embedded (to its Git repository) "fakelinux" tree that can be used to simulate a large number of fake kernel builds and corner cases with each supported runtime.

Mutation testing is frequently used on the TuxMake code base to discover edge cases by applying generated mutations to the running code and ensuring that every mutation has a corresponding failing test case.

The project also employs automated testing for the contents of the provided container images, to avoid regressions. These include both checking that the required tools and compilers are available in the default $PATH, as well as integration testing where the container images are used in actual builds.

All of the tests are run on every merge request and merge made to TuxMake using an automated GitLab pipeline.

Getting Started

TuxMake can be installed from source or with pip. To use it with a container runtime, install either Docker or Podman and ensure that your user has the ability to run containers. Alternative installation options and complete documentation are available at docs.tuxmake.org. Issues such as feature requests and bugs are tracked as GitLab issues.

In addition to the command-line interface, TuxMake also provides a Python interface that can be used to perform Linux kernel builds from Python. Most of the arguments that are available at the command line are also available to the build constructor Build(), but a minimal example can be seen below:

    import tuxmake.build

    build = tuxmake.build.Build("/home/drue/src/linux-mainline")
    build.run()

TuxMake is funded by Linaro and the TuxBuild commercial build service. While TuxMake runs locally to perform individual kernel builds, TuxBuild is an API that integrates into continuous integration (CI) systems to perform massive numbers of Linux kernel builds in parallel and on demand.

TuxMake was written to generally solve the problem of automating Linux kernel builds across targets, architectures, toolchains, kernel configurations, and build-host environments. Git solved the Linux source code problem by making it easy to reproduce and communicate specific versions of the tree across time and space. Hopefully, TuxMake can help solve the Linux build problem by providing a common interface to performing, reproducing, and communicating any possible Linux kernel build.

[I would like to thank Antonio Terceiro, TuxMake's author and maintainer, for his help with this article.]

Comments (21 posted)