LWN: Comments on "Portable and reproducible kernel builds with TuxMake"

Portable and reproducible kernel builds with TuxMake

mcon147 — Fri, 08 Jan 2021 01:22:15 +0000

Using repeated "??" and ".." and chains of capital letter words should be treated with some caution

Portable and reproducible kernel builds with TuxMake

jezuch — Thu, 07 Jan 2021 18:01:18 +0000

So you wrote your own scripts to build the kernel. So what's wrong with someone else writing their own scripts to build the kernel? :)

Portable and reproducible kernel builds with TuxMake

darwi — Wed, 06 Jan 2021 18:40:27 +0000

> It's designed to be self describing. From the article: "tuxmake -r podman -a arm64 -t clang-11 -k defconfig -K CONFIG_KASAN=y -w ccache".

Yes. What grabbed my attention though was the GCC case, where (unlike clang) the version was not explicitly stated in the "tuxmake reproducer" line :)

> We've tried to make tuxmake as stupid as possible, so that it's not doing any clever things that a kernel developer wouldn't expect.

I see. Thanks for the clarification (and for popularizing podman!).

Portable and reproducible kernel builds with TuxMake

terceiro — Wed, 06 Jan 2021 17:15:10 +0000

> Reproducible kernels are a very good idea, but they need to be based on reproducibly-built tools. Otherwise you have containers with SHA256s which you base your build on all you want, but what assurance do you have that the container was built with non-compromised tools in the first place? Does TuxMake address this?

The TuxMake container images are built upon the Debian images provided by Docker Inc. They use only official Debian packages, with the exception of daily toolchain builds for which we get packages from the upstream project. They are built on Gitlab CI, with arm64 builds done by a Linaro server, and x86_64 done by Gitlab.com workers. Therefore at the moment the integrity of the TuxMake images relies on the integrity of Docker Hub, Debian, LLVM nightly builds, Gitlab.com, and a Linaro server.

Given the current state of reproducible builds in the free software community, would say the TuxMake containers are just good enough to get started. Of course, we can and should improve upon that (both TuxMake and the rest of the community). On the other hand, except for that Linaro server, a compromise in any of those means we all have bigger problems than the non-reproducibility of the TuxMake container images.

Portable and reproducible kernel builds with TuxMake

terceiro — Wed, 06 Jan 2021 15:40:55 +0000

> In the examples shown, the printed tuxmake command "to reproduce this build locally" does not show the GCC version. How do you plan to handle future GCC versions?

When using containers, you can request a (major) compiler version explicitly, and that's what you will get. Otherwise, all that TuxMake can do is use whatever gcc you have on $PATH. In any case the exact compiler version that was used is recorded in a metadata file which is collected from the build environment.

> It would also be nice if there is a way to extract all the pure build parameters and defconfig in a simple file, so that bug reporters can send *that data* instead of a "tuxmake command". Kernel developers / maintainers on the receiving end should not be forced to use "tuxmake" to reproduce bug reports.

Beyond tool versions, the metadata file also contains info about the input tree and build output, plus all the inputs that have been provided by the user such as toolchain, target architecture, config arguments etc. The final kernel config is also included in the build artifacts.

Portable and reproducible kernel builds with TuxMake

mndrue — Wed, 06 Jan 2021 15:07:18 +0000

> Kernel developers / maintainers on the receiving end should not be forced to use "tuxmake" to reproduce bug reports.

We agree. Currently, most build reports from users on lkml look something like "My build broke with the following error". If the recipient is lucky, the author mentions a config, toolchain, or architecture. Usually, in the case of a tricky build problem, it's a few round trips to figure out how to reproduce the build error reported.

The tuxmake reproducer already gives more information to the recipient, whether or not they choose to use tuxmake, than most build reports. It's designed to be self describing. From the article:

> tuxmake -r podman -a arm64 -t clang-11 -k defconfig -K CONFIG_KASAN=y -w ccache

I'm hopeful that even that short version of the command communicates enough information to reproduce a build problem, with or without tuxmake.

We've tried to make tuxmake as stupid as possible, so that it's not doing any clever things that a kernel developer wouldn't expect.

Portable and reproducible kernel builds with TuxMake

k3ninho — Wed, 06 Jan 2021 11:13:12 +0000

You're very welcome to your opinion. You're wrong, though, in light of the community you're (inadvertently) part of. Please, express your opinion and your assumptions and the context in which you've developed those views. But along with that, expect that people will (courteously and respectfully) correct the misapprehensions you've absorbed.

The thing that made Linux the success it is is entirely its community. People got together to add features and share-back the code they changed. Patches need the transitivity of 'your code + my changes' in either your build system or my build system to arrive at similar-enough program binaries that my claims to make something better in my built edition of the program are actually making the program better in your built edition of the program.

Seems easy when it works. It's a massive pain when it goes wrong and you've moved so fast breaking things that you can't quite understand which small change caused a chain of breakage.

Reproducible builds and a reference-type build environment might be monocultural and eventually a weak spot. For now there's gains in assembling and maintaining this suite -- and look at Linaro as an organisation empowering small-board and industrial-computing builders to target esoteric and exotic devices: it's hard to standardise the wide swathe of 'get it working, ship the product and move on' hacking for these devices, but this helps.

Dont forget it's a team game,
K3n.

Portable and reproducible kernel builds with TuxMake

dottedmag — Wed, 06 Jan 2021 10:08:41 +0000

Amusing, but not surprising: testing a kernel is harder than testing a build system.

Portable and reproducible kernel builds with TuxMake

smurf — Wed, 06 Jan 2021 09:42:27 +0000

Debian's builds, strictly-reproducible or not, already create a .buildinfo artefact that lists the versions of all tools used to effect the build. Thus there's no need for yet another, container-based (ugh), tool like TuxMake that only works for a single package (the kernel).

Reproducible kernels are a very good idea, but they need to be based on reproducibly-built tools. Otherwise you have containers with SHA256s which you base your build on all you want, but what assurance do you have that the container was built with non-compromised tools in the first place? Does TuxMake address this?

Portable and reproducible kernel builds with TuxMake

amacater — Wed, 06 Jan 2021 09:15:42 +0000

Antonio is a Debian developer: this has direct relevance to making Debian reproducible builds, for example.. It's a worthy step forward for its use cases.

Portable and reproducible kernel builds with TuxMake

Sesse — Wed, 06 Jan 2021 09:13:29 +0000

> Sincerely, I have no objection , just express my views, am I not suppose to do so???

Only if those views are useful in some way, really.

Portable and reproducible kernel builds with TuxMake

linusw — Wed, 06 Jan 2021 08:37:14 +0000

I've used TuxMake a little and very happy with it for what I do.

I would say it brings the kernel development into what all the cool kids call "DevOps". This is a somewhat elusive term, but involves versioning build environment and build artifacts which is something TuxMake does.

Portable and reproducible kernel builds with TuxMake

unixbhaskar — Wed, 06 Jan 2021 03:25:35 +0000

Okay, if you prefer, curse me for the "misnomer" ...I am bugged by that ,being in the specific industry and seeing it's been practiced everywhere and importantly everyone(smart people too!) , so laser mortals like me , also get injected with the bad idea.

:) Teach me/correct me/show me ...I believe I have "growth mindset" and not stubbornly cling on something for the sake statisfy my stupid ego(yes I have, probably other don't have that....he he he he h...probably others control better) ...

Portable and reproducible kernel builds with TuxMake

unixbhaskar — Wed, 06 Jan 2021 03:19:47 +0000

Okay, IIRC , if my faint memory serves well, don't we have a "reproducible thing" for the kernel???? Wondering....

Probably ,somebody more aware can confirm that. Anyway, It is always a temptation ,when you build something and and think it is "absolutely required" to do better job.

"Large environment" is garbage.

YMMV and I am sure I am telling from my small/little and importantly very limited experiance(NO PUN INTENDED) , you might have more exposure to conclude on that.

Okay, This tool might a boon for some...not a generalised one.As you have rightly pointed out. So, it might missed few "mundane" and "ordinary" user who are fond of building stuff.

Sincerely, I have no objection , just express my views, am I not suppose to do so???

Portable and reproducible kernel builds with TuxMake

unixbhaskar — Wed, 06 Jan 2021 03:08:21 +0000

Sasha, "by hand" means I have my frivilious automated tool, which does the job for me. And doing thing for a team, I believe not every on involed int he kernel building stage?? At least I am not aware of it.

A single person build the kernel, whether it's in team environment or for indidual need.

Flame, if I read between the lines. Oh, btw , I would like maintain a toolchain of my own which match the expectation of kernel....well, now brain ring a alerm bell and you frown, probably justified ...that's the way I do .

That is not efficent , you might opined ...but it's okay ...whatever I have read in the article looks not so straight forward and laden with assumptions.

Well, I probably grossly wrong....

Portable and reproducible kernel builds with TuxMake

darwi — Wed, 06 Jan 2021 02:36:10 +0000

> In the case of reporting a build problem to a mailing list, a one-line TuxMake command communicates precise instructions for reproducing the build problem. Any user who runs the same TuxMake command against the same Linux source tree will see the same build.

In the examples shown, the printed tuxmake command "to reproduce this build locally" does not show the GCC version. How do you plan to handle future GCC versions?

It would also be nice if there is a way to extract all the pure build parameters and defconfig in a simple file, so that bug reporters can send *that data* instead of a "tuxmake command". Kernel developers / maintainers on the receiving end should not be forced to use "tuxmake" to reproduce bug reports.

(I'm honestly not a fan. At least on Debian, all the needed toolchains are one "apt-get gcc-${ARCH}-linux-gnu" away. But I understand the possible need for CI).

Portable and reproducible kernel builds with TuxMake

sashal — Wed, 06 Jan 2021 01:10:51 +0000

I too prefer indexing my own versions of files rather than use some third party program like Git!

At least Linus got a good exercise to enhance his understanding of SCMs...

But seriously, doing it "by hand" works fine if it's just your own kernel, but its far more complex to build kernels for multiple architectures and config files, or being able to quickly build a kernel that a user point out. I don't want to maintain toolchains, I just want to get my kernel work done.

Portable and reproducible kernel builds with TuxMake

dxin — Wed, 06 Jan 2021 00:01:09 +0000

Containerized toolchains makes a lot of sense. Very few machines needs to run purely natively compiled code. It's always such that one machines compiles the code while another runs it.

On the other hand, I'm surprised that the embedded world, notably AOSP, lived with the broken and bloated old way for so long.

Portable and reproducible kernel builds with TuxMake

ttuttle — Wed, 06 Jan 2021 00:00:19 +0000

The simplicity you cite is deceptive: it's simple to build a kernel in your own workspace, but it's not simple to build one in an identical workspace to a colleague's, or to the one in a continuous instrumentation system, or to the one used in a larger build system.

If you're only building for yourself, TuxMake may in fact be overkill. But if you're working as part of a larger system and may need to reproduce other builds or have others reproduce yours, it's a way to simplify identifying and sharing those environmental dependencies.

Portable and reproducible kernel builds with TuxMake

unixbhaskar — Tue, 05 Jan 2021 23:19:38 +0000

Nope, I would prefer to do all the things by hand , then doing the important part decided by some thrid party program .

Plus bending it makes thing even more complex and no point . Flooded with assumption.

But having said that, it must be good excercise by the authors to enahance their understanding of kernel.

Why make simple thing more comlicated???? When things can be done in 3 steps...why brings all those fuzzz???

YMMV

Portable and reproducible kernel builds with TuxMake

roc — Tue, 05 Jan 2021 22:05:21 +0000

It's amusing that TuxMake's testing protocols are much more stringent than those of the kernel itself.