Portable and reproducible kernel builds with TuxMake
TuxMake is an open-source project from Linaro that began in May 2020 and is designed to make building Linux kernels easier. It provides a command-line interface and a Python library, along with a full set of curated portable build environments distributed as container images. With TuxMake, a developer can build any supported combination of target architecture, toolchain, kernel configuration, and make targets.
Building a Linux kernel is not difficult. Follow the documentation, install the dependencies, and run a couple of make commands. However, if a developer wants to build for multiple architectures, with multiple toolchains, things get complicated quickly. Most developers and maintainers have a set of custom scripts that they have written and maintained to perform their required set of builds. TuxMake provides a common layer of abstraction to reduce the need for every developer to write their own build scripts.
TuxMake publishes containers for various toolchain/architecture combinations. These containers eliminate the need for individual developers to source and install multiple toolchains and toolchain versions on their systems. It also makes builds reproducible and portable because now the environment in which a kernel is built is versioned and shareable across the internet and on mailing lists.
TuxMake has two goals. First, remove the friction that may cause developers, especially new developers, to skip build testing for uncommon toolchain/architecture combinations. Second, to make it easier for builds and build problems to be described and reproduced.
Features
Architectures supported are arc, arm, arm64, i386, mips, parisc, powerpc, riscv, s390, sh, sparc, and x86_64. Toolchains supported are GCC versions 8, 9, and 10, and Clang versions 10, 11, and nightly-latest. Targets supported for all relevant architectures are the kernel config (Kconfig), kernel image, modules, device tree binaries (DTBs), and a debug kernel image. The TuxMake team plans to add additional targets including kselftest, cpupower, perf, and even documentation.
Containers and container runtimes that are compliant with Open Container Initiative (OCI) specifications are supported to enable build portability and reproducibility. Both Docker and Podman runtimes are supported and interchangeable, depending on each user's preference. Podman is a popular alternative to Docker because it is daemonless and does not require root privileges. Additional container runtimes can be added as needed.
How does it work?
In a terminal on a Linux system with TuxMake, you can navigate to a Linux kernel source directory where you would usually run make and instead run tuxmake. Without any arguments, tuxmake will perform a build using default values for all options. It will look something like the following:
$ tuxmake
# to reproduce this build locally: tuxmake --target-arch=x86_64 \
--kconfig=defconfig --toolchain=gcc --wrapper=none --runtime=null \
config kernel xipkernel debugkernel modules dtbs
First, the tuxmake reproducer command is printed with all of the arguments provided. This information is helpful for reproducing the build and sharing with colleagues.
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp defconfig
A .config is built using the defconfig target. Note that, by default, it is saved to a unique directory that is automatically created under ~/.cache/tuxmake. All intermediary files and build artifacts will be stored there.
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp bzImage
The default kernel as well as a bzImage will be built; it might seem redundant to explicitly build bzImage for x86_64 since the previous make invocation should already do that. As it happens, not all architectures handle this consistently. Rather than special case those architectures and bake in knowledge about quirks in the kernel tree, tuxmake explicitly builds the final image. In most cases that is a no-op.
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/676/tmp vmlinux
xz -T0 --keep /home/drue/.cache/tuxmake/builds/676/tmp/vmlinux
A debug kernel image will be built, saved to the output directory, and compressed with xz. As before, make vmlinux might look redundant given vmlinux is already built by make. However, the debug kernel image can be built independently. In a full build, make vmlinux will be a no-op, but if only the debug kernel is being built, it will be the main build step.
grep -q CONFIG_MODULES=y /home/drue/.cache/tuxmake/builds/676/tmp/.config
make --silent --keep-going --jobs=16 \
O=/home/drue/.cache/tuxmake/builds/676/tmp modules_install INSTALL_MOD_STRIP=1 \
INSTALL_MOD_PATH=/home/drue/.cache/tuxmake/builds/676/tmp/modinstall
tar -caf /home/drue/.cache/tuxmake/builds/676/tmp/modules.tar.xz \
-C /home/drue/.cache/tuxmake/builds/676/tmp/modinstall lib
If kernel modules are enabled in the build config, modules will be built, gathered using modules_install, and stored in a tar.xz file in the output directory.
I: config: PASS in 0:00:01.305649
I: kernel: PASS in 0:01:31.887716
I: debugkernel: PASS in 0:00:08.207033
I: modules: PASS in 0:00:00.869124
I: build output in /home/drue/.cache/tuxmake/builds/676
Finally, each target's build status (PASS/FAIL/SKIP) and build time are displayed, along with the path to the build's output directory.
Containers are not required, and are not used by default. When running without a container runtime, TuxMake will use locally available toolchains. When a container runtime is specified, TuxMake will download a container image at build time (if necessary) and perform the build inside the container. It does this by mounting the Linux source directory and the build output directory into the container and performing each build step in an ephemeral container — the container only runs for the duration of the build step and then exits.
Putting it all together, here is a more elaborate example of using Podman to build an arm64 kernel using Clang with KASAN enabled:
$ tuxmake -r podman -a arm64 -t clang-11 -k defconfig -K CONFIG_KASAN=y -w ccache
# to reproduce this build locally: tuxmake --target-arch=arm64 --kconfig=defconfig \
--kconfig-add=CONFIG_KASAN=y --toolchain=clang-11 --wrapper=ccache \
--runtime=podman --image=tuxmake/arm64_clang-11 \
config kernel xipkernel debugkernel modules dtbs
When podman is specified as the runtime, TuxMake will use Podman to perform the build. The kernel will be built for aarch64 (-a arm64) with Clang version 11 (-t clang-11). The kernel configuration will be generated using the defconfig target, then KASAN will be explicitly enabled. Finally, ccache will be enabled (-w ccache) to reduce the build time.
Trying to pull docker.io/tuxmake/arm64_clang-11...
A container for arm64_clang-11 is fetched from TuxMake's public container registry at hub.docker.com/u/tuxmake, if needed.
# CONFIG_KASAN=y -> /home/drue/.cache/tuxmake/builds/685/tmp/0.config
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
'HOSTCC=ccache clang' 'CC=ccache clang' defconfig
scripts/kconfig/merge_config.sh -m -O /home/drue/.cache/tuxmake/builds/685/tmp \
/home/drue/.cache/tuxmake/builds/685/tmp/.config \
/home/drue/.cache/tuxmake/builds/685/tmp/0.config
Using /home/drue/.cache/tuxmake/builds/685/tmp/.config as base
Merging /home/drue/.cache/tuxmake/builds/685/tmp/0.config
#
# merged configuration written to /home/drue/.cache/tuxmake/builds/685/tmp/.config (needs make)
#
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
'HOSTCC=ccache clang' 'CC=ccache clang' olddefconfig
The .config is generated by building defconfig and then using merge_config.sh to merge in any config fragments that were specified. The rest of the build proceeds as expected. The only difference from the first example is the addition of building DTBs, since it is an arm64 kernel.
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
'HOSTCC=ccache clang' 'CC=ccache clang' dtbs
mkdir -p /home/drue/.cache/tuxmake/builds/685/tmp/dtbsinstall/dtbs
make --silent --keep-going --jobs=16 O=/home/drue/.cache/tuxmake/builds/685/tmp \
ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
'HOSTCC=ccache clang' 'CC=ccache clang' dtbs_install \
INSTALL_DTBS_PATH=/home/drue/.cache/tuxmake/builds/685/tmp/dtbsinstall/dtbs
tar caf /home/drue/.cache/tuxmake/builds/685/tmp/dtbs.tar.xz \
-C /home/drue/.cache/tuxmake/builds/685/tmp/dtbsinstall dtbs
The result of the DTBs build is a dtbs.tar.xz artifact in the output directory, containing all of the DTBs.
Specifying and managing Linux kernel build configuration is a difficult problem. TuxMake provides a --kconfig argument, which defaults to defconfig. When specified, the given config target (such as tinyconfig, allmodconfig, or allnoconfig) will be built. If a path or URL of a pre-built config file is specified, it will be used directly.
Additional Kconfig options can be specified by one or more --kconfig-add arguments. --kconfig-add takes a path or URL to a config fragment or a literal Kconfig string such as CONFIG_KASAN=y. If any URLs are provided, they will be downloaded. All of the configs specified are merged using scripts/kconfig/merge_config.sh and make olddefconfig.
After a build, the build directory will contain a build log, a compressed kernel image, the kernel config, a metadata file in JSON that describes every aspect of the build and the build environment, a system map, a compressed debug kernel, a compressed modules tar file if applicable, and a compressed DTB tar file if applicable. Additional artifacts will be added as additional targets are implemented.
Portability and reproducibility
Because TuxMake can use shareable build environments, and things like Kconfig can be specified using URLs, TuxMake build commands can be shared with others. In the case of reporting a build problem to a mailing list, a one-line TuxMake command communicates precise instructions for reproducing the build problem. Any user who runs the same TuxMake command against the same Linux source tree will see the same build.
Full bit-for-bit reproducible builds are also supported, but require additional build arguments. First, it is critical that the exact same container is used. This can be accomplished with the --image argument which can take a full explicit container path, including sha256 digest. Second, environment variables KBUILD_BUILD_TIMESTAMP, KBUILD_BUILD_USER, and KBUILD_BUILD_HOST must be set using -e because they influence the resulting kernel image. In the normal case, the resulting build binaries will match bit for bit as long as all of the above inputs are the same (aside from the additional reproducible build caveats detailed at kernel.org).
For example, the following command run on an x86_64 host against Linux sources at tag v5.10 will result in a bzImage sha256 that begins with 8d066f679eac. It works with both -r podman and -r docker:
$ tuxmake --image \
docker.io/tuxmake/x86_64_gcc@sha256:f8218cbfad8ecf6628fc44db864a402070feb87ff43a880e1409649172d4bc8c \
-r podman -k tinyconfig \
-e "KBUILD_BUILD_TIMESTAMP='Tue May 26 16:16:14 2020 -0500'" \
-e "KBUILD_BUILD_USER=tuxmake" \
-e "KBUILD_BUILD_HOST=tuxmake"
Note that this was run using TuxMake 0.11.0, and should work for the foreseeable future. However, it is possible that future versions of TuxMake will introduce additional default variables into the build environment that break the backward compatibility of this example.
Quality
It is critical that TuxMake’s behavior is obvious, transparent, and reliable. Without basic trust and quality, such a tool is more trouble than it's worth. TuxMake enforces 100% unit test coverage, meaning that every line of code is covered by at least one basic test. Additionally, it incorporates comprehensive integration testing using an embedded (to its Git repository) "fakelinux" tree that can be used to simulate a large number of fake kernel builds and corner cases with each supported runtime.
Mutation testing is frequently used on the TuxMake code base to discover edge cases by applying generated mutations to the running code and ensuring that every mutation has a corresponding failing test case.
The project also employs automated testing for the contents of the provided container images, to avoid regressions. These include both checking that the required tools and compilers are available in the default $PATH, as well as integration testing where the container images are used in actual builds.
All of the tests are run on every merge request and merge made to TuxMake using an automated GitLab pipeline.
Getting Started
TuxMake can be installed from source or with pip. To use it with a container runtime, install either Docker or Podman and ensure that your user has the ability to run containers. Alternative installation options and complete documentation are available at docs.tuxmake.org. Issues such as feature requests and bugs are tracked as GitLab issues.
In addition to the command-line interface, TuxMake also provides a Python interface that can be used to perform Linux kernel builds from Python. Most of the arguments that are available at the command line are also available to the build constructor Build(), but a minimal example can be seen below:
import tuxmake.build
build = tuxmake.build.Build("/home/drue/src/linux-mainline")
build.run()
TuxMake is funded by Linaro and the TuxBuild commercial build service. While TuxMake runs locally to perform individual kernel builds, TuxBuild is an API that integrates into continuous integration (CI) systems to perform massive numbers of Linux kernel builds in parallel and on demand.
TuxMake was written to generally solve the problem of automating Linux kernel builds across targets, architectures, toolchains, kernel configurations, and build-host environments. Git solved the Linux source code problem by making it easy to reproduce and communicate specific versions of the tree across time and space. Hopefully, TuxMake can help solve the Linux build problem by providing a common interface to performing, reproducing, and communicating any possible Linux kernel build.
[I would like to thank Antonio Terceiro, TuxMake's author and
maintainer, for his help with this article.]
| Index entries for this article | |
|---|---|
| GuestArticles | Rue, Dan |
Posted Jan 5, 2021 22:05 UTC (Tue)
by roc (subscriber, #30627)
[Link] (1 responses)
Posted Jan 6, 2021 10:08 UTC (Wed)
by dottedmag (subscriber, #18590)
[Link]
Posted Jan 5, 2021 23:19 UTC (Tue)
by unixbhaskar (guest, #44758)
[Link] (9 responses)
Plus bending it makes thing even more complex and no point . Flooded with assumption.
But having said that, it must be good excercise by the authors to enahance their understanding of kernel.
Why make simple thing more comlicated???? When things can be done in 3 steps...why brings all those fuzzz???
YMMV
Posted Jan 6, 2021 0:00 UTC (Wed)
by ttuttle (subscriber, #51118)
[Link] (4 responses)
If you're only building for yourself, TuxMake may in fact be overkill. But if you're working as part of a larger system and may need to reproduce other builds or have others reproduce yours, it's a way to simplify identifying and sharing those environmental dependencies.
Posted Jan 6, 2021 3:19 UTC (Wed)
by unixbhaskar (guest, #44758)
[Link] (3 responses)
Probably ,somebody more aware can confirm that. Anyway, It is always a temptation ,when you build something and and think it is "absolutely required" to do better job.
"Large environment" is garbage.
YMMV and I am sure I am telling from my small/little and importantly very limited experiance(NO PUN INTENDED) , you might have more exposure to conclude on that.
Okay, This tool might a boon for some...not a generalised one.As you have rightly pointed out. So, it might missed few "mundane" and "ordinary" user who are fond of building stuff.
Sincerely, I have no objection , just express my views, am I not suppose to do so???
Posted Jan 6, 2021 9:13 UTC (Wed)
by Sesse (subscriber, #53779)
[Link]
Only if those views are useful in some way, really.
Posted Jan 6, 2021 11:13 UTC (Wed)
by k3ninho (subscriber, #50375)
[Link]
The thing that made Linux the success it is is entirely its community. People got together to add features and share-back the code they changed. Patches need the transitivity of 'your code + my changes' in either your build system or my build system to arrive at similar-enough program binaries that my claims to make something better in my built edition of the program are actually making the program better in your built edition of the program.
Seems easy when it works. It's a massive pain when it goes wrong and you've moved so fast breaking things that you can't quite understand which small change caused a chain of breakage.
Reproducible builds and a reference-type build environment might be monocultural and eventually a weak spot. For now there's gains in assembling and maintaining this suite -- and look at Linaro as an organisation empowering small-board and industrial-computing builders to target esoteric and exotic devices: it's hard to standardise the wide swathe of 'get it working, ship the product and move on' hacking for these devices, but this helps.
Dont forget it's a team game,
Posted Jan 8, 2021 1:22 UTC (Fri)
by mcon147 (subscriber, #56569)
[Link]
Posted Jan 6, 2021 1:10 UTC (Wed)
by sashal (✭ supporter ✭, #81842)
[Link] (3 responses)
At least Linus got a good exercise to enhance his understanding of SCMs...
But seriously, doing it "by hand" works fine if it's just your own kernel, but its far more complex to build kernels for multiple architectures and config files, or being able to quickly build a kernel that a user point out. I don't want to maintain toolchains, I just want to get my kernel work done.
Posted Jan 6, 2021 3:08 UTC (Wed)
by unixbhaskar (guest, #44758)
[Link] (2 responses)
A single person build the kernel, whether it's in team environment or for indidual need.
Flame, if I read between the lines. Oh, btw , I would like maintain a toolchain of my own which match the expectation of kernel....well, now brain ring a alerm bell and you frown, probably justified ...that's the way I do .
That is not efficent , you might opined ...but it's okay ...whatever I have read in the article looks not so straight forward and laden with assumptions.
Well, I probably grossly wrong....
Posted Jan 6, 2021 3:25 UTC (Wed)
by unixbhaskar (guest, #44758)
[Link]
:) Teach me/correct me/show me ...I believe I have "growth mindset" and not stubbornly cling on something for the sake statisfy my stupid ego(yes I have, probably other don't have that....he he he he h...probably others control better) ...
Posted Jan 7, 2021 18:01 UTC (Thu)
by jezuch (subscriber, #52988)
[Link]
Posted Jan 6, 2021 0:01 UTC (Wed)
by dxin (guest, #136611)
[Link]
On the other hand, I'm surprised that the embedded world, notably AOSP, lived with the broken and bloated old way for so long.
Posted Jan 6, 2021 2:36 UTC (Wed)
by darwi (subscriber, #131202)
[Link] (6 responses)
In the examples shown, the printed tuxmake command "to reproduce this build locally" does not show the GCC version. How do you plan to handle future GCC versions?
It would also be nice if there is a way to extract all the pure build parameters and defconfig in a simple file, so that bug reporters can send *that data* instead of a "tuxmake command". Kernel developers / maintainers on the receiving end should not be forced to use "tuxmake" to reproduce bug reports.
(I'm honestly not a fan. At least on Debian, all the needed toolchains are one "apt-get gcc-${ARCH}-linux-gnu" away. But I understand the possible need for CI).
Posted Jan 6, 2021 9:15 UTC (Wed)
by amacater (subscriber, #790)
[Link] (2 responses)
Posted Jan 6, 2021 9:42 UTC (Wed)
by smurf (subscriber, #17840)
[Link] (1 responses)
Reproducible kernels are a very good idea, but they need to be based on reproducibly-built tools. Otherwise you have containers with SHA256s which you base your build on all you want, but what assurance do you have that the container was built with non-compromised tools in the first place? Does TuxMake address this?
Posted Jan 6, 2021 17:15 UTC (Wed)
by terceiro (subscriber, #83820)
[Link]
The TuxMake container images are built upon the Debian images provided by Docker Inc. They use only official Debian packages, with the exception of daily toolchain builds for which we get packages from the upstream project. They are built on Gitlab CI, with arm64 builds done by a Linaro server, and x86_64 done by Gitlab.com workers. Therefore at the moment the integrity of the TuxMake images relies on the integrity of Docker Hub, Debian, LLVM nightly builds, Gitlab.com, and a Linaro server.
Given the current state of reproducible builds in the free software community, would say the TuxMake containers are just good enough to get started. Of course, we can and should improve upon that (both TuxMake and the rest of the community). On the other hand, except for that Linaro server, a compromise in any of those means we all have bigger problems than the non-reproducibility of the TuxMake container images.
Posted Jan 6, 2021 15:07 UTC (Wed)
by mndrue (guest, #115498)
[Link] (1 responses)
We agree. Currently, most build reports from users on lkml look something like "My build broke with the following error". If the recipient is lucky, the author mentions a config, toolchain, or architecture. Usually, in the case of a tricky build problem, it's a few round trips to figure out how to reproduce the build error reported.
The tuxmake reproducer already gives more information to the recipient, whether or not they choose to use tuxmake, than most build reports. It's designed to be self describing. From the article:
> tuxmake -r podman -a arm64 -t clang-11 -k defconfig -K CONFIG_KASAN=y -w ccache
I'm hopeful that even that short version of the command communicates enough information to reproduce a build problem, with or without tuxmake.
We've tried to make tuxmake as stupid as possible, so that it's not doing any clever things that a kernel developer wouldn't expect.
Posted Jan 6, 2021 18:40 UTC (Wed)
by darwi (subscriber, #131202)
[Link]
Yes. What grabbed my attention though was the GCC case, where (unlike clang) the version was not explicitly stated in the "tuxmake reproducer" line :)
> We've tried to make tuxmake as stupid as possible, so that it's not doing any clever things that a kernel developer wouldn't expect.
I see. Thanks for the clarification (and for popularizing podman!).
Posted Jan 6, 2021 15:40 UTC (Wed)
by terceiro (subscriber, #83820)
[Link]
When using containers, you can request a (major) compiler version explicitly, and that's what you will get. Otherwise, all that TuxMake can do is use whatever gcc you have on $PATH. In any case the exact compiler version that was used is recorded in a metadata file which is collected from the build environment.
> It would also be nice if there is a way to extract all the pure build parameters and defconfig in a simple file, so that bug reporters can send *that data* instead of a "tuxmake command". Kernel developers / maintainers on the receiving end should not be forced to use "tuxmake" to reproduce bug reports.
Beyond tool versions, the metadata file also contains info about the input tree and build output, plus all the inputs that have been provided by the user such as toolchain, target architecture, config arguments etc. The final kernel config is also included in the build artifacts.
Posted Jan 6, 2021 8:37 UTC (Wed)
by linusw (subscriber, #40300)
[Link]
I would say it brings the kernel development into what all the cool kids call "DevOps". This is a somewhat elusive term, but involves versioning build environment and build artifacts which is something TuxMake does.
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
K3n.
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake
Portable and reproducible kernel builds with TuxMake