|
|
Subscribe / Log in / New account

Hash-based module integrity checking

By Daroc Alden
March 7, 2025

On January 20, Thomas Weißschuh shared a new patch set implementing an alternate method for checking the integrity of loadable kernel modules. This mechanism, which checks module integrity based on hashes computed at build time instead of using cryptographic signatures, could enable reproducible kernel builds in more contexts. Several distributions have already expressed interest in the patch set if Weißschuh can get it into the kernel.

Linux has supported signing loadable kernel modules since 2012, when David Howells introduced the first code for it. Since then, the implementation has not changed too much. Users can enable the CONFIG_MODULE_SIG option to turn on module signing; by default, this will simply taint the kernel if an unsigned module is loaded. Enabling CONFIG_MODULE_SIG_FORCE or the Lockdown Linux Security Module (LSM) prevents the kernel from loading unsigned modules. The public keys needed to verify signatures are baked into the kernel at build time. The build process can be configured to use an existing keypair, or to automatically generate the necessary asymmetric keys.

That automatic generation is Weißschuh's gripe with the current code. Reproducible builds are important for security, since they allow independent verification that an open-source project has been compiled without inserting extra, malicious changes. For something as foundational as the Linux kernel, it would be nice to be able to verify that a build of the kernel is unmodified. But when signing keys are needed for the build, it cannot be made reproducible without distributing the key. Currently, this puts users in a bind. They cannot have loadable modules, reproducible builds, and signature verification all turned on at the same time.

It's tempting to search for a clever cryptographic solution, but nobody has yet proposed one. If the signing keys are publicly available for use in recreating the build, malicious actors could also sign modified loadable modules with them. If they aren't publicly available, the build can't be reproduced. So Weißschuh's patch set takes a much simpler approach: instead of trying to create a signature for loadable modules, the patch set calculates cryptographic hashes for all modules built with the kernel, and embeds those hashes as a static list to verify prospective modules before they are loaded.

This has the benefit of simplicity — if a module has the same hash that it had at build time, it certainly hasn't been tampered with — but it is a bit less flexible than the signature-based approach. Specifically, Weißschuh's patch set only works for modules that are built at the same time as the kernel. The default build configuration, which generates new keys for each build, has the same limitation, but users can use their own long-term signing keys if they prefer. Out-of-tree modules, DKMS modules, and so on can't have their hashes included in the kernel, and therefore can't be verified by hash.

Nothing prevents users from enabling both Weißschuh's new code (with CONFIG_MODULE_HASHES) and the existing signature verification support, although in that case the resulting kernel isn't reproducible. This will build all the in-tree modules with hashes (but not signatures), and still allow the loading of signed (but not hashed) out-of-tree modules. Arch Linux contributor "kpcyrd" summarized the possible options.

Combining the settings for module hashing and module signing yield four possibilities. When neither CONFIG_MODULE_HASHES nor CONFIG_MODULE_SIG are enabled, the kernel does not perform any checks (and therefore Lockdown will reject loading any modules). When only CONFIG_MODULE_SIG is enabled, the behavior is the same as today — signed modules can be loaded. When CONFIG_MODULE_HASHES is enabled, modules with a known hash can be loaded. When both are enabled, a module that has either a known hash or a valid signature can be loaded. If CONFIG_MODULE_SIG_FORCE is enabled, CONFIG_MODULE_SIG must also be enabled. If neither CONFIG_MODULE_SIG_FORCE nor the Lockdown LSM are enabled, the kernel won't disallow modules without valid signatures or hashes, and therefore there is little point in configuring either signed or hashed modules.

For users of Arch Linux, Proxmox, and SUSE — projects that Weißschuh indicated were interested in this work based on the previous discussion — the last option is the most likely configuration for a normal kernel build. NixOS does not enable module signing by default, and is more likely to only enable CONFIG_MODULE_HASHES, but was still interested. The patch set is not yet quite ready to go into the mainline kernel, however. Petr Pavlu suggested a few tweaks for Weißschuh to make. A handful of other people participated in the discussion of the patch set, although there was not too much objection to the patch set in face of the evident interest from the different distributions. Once he revises the patch set, it seems likely to remove one of the few remaining obstacles to deploying reproducible kernel builds.



to post comments

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 16:06 UTC (Fri) by bluca (subscriber, #118303) [Link] (19 responses)

> Currently, this puts users in a bind. They cannot have loadable modules, reproducible builds, and signature verification all turned on at the same time.

Why not just improve the tools that do the diffing to strip the signatures from the modules before comparing them? If they are bit-by-bit identical, the same signature could even be just reattached and the result should be identical.
It seems the binutils strip is able to remove the signature from a signed kernel module, from a quick local test.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 17:25 UTC (Fri) by josh (subscriber, #17465) [Link] (16 responses)

That's definitely possible, but it means you can't (for instance) have whole *software packages* (e.g. debs) that are reproducible.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 18:41 UTC (Fri) by bluca (subscriber, #118303) [Link] (1 responses)

You still can't have the kernel package reproducible without the signing key, as the kernel PE image itself needs to be signed.

Considering inline signatures as part of the reproducible envelope doesn't provide any useful information about the state of the builds. It's not interesting data - it's just RSA. Given the same input, and the same key, you get the same output, so it's reproducible by definition.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 13, 2025 14:24 UTC (Thu) by pjones (subscriber, #31722) [Link]

I think this is a problem of how we're discussing it more than anything else - the signed kernel is reproducible in the meaningful sense, but the data to validate it is effectively encapsulating it.

Which means you're right that this is primarily a tooling problem in our comparisons.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 18:58 UTC (Fri) by bluca (subscriber, #118303) [Link] (13 responses)

Also that statement ignores that half of the world signs packages too - RPMs are themselves signed.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 20:01 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

Well, don't use RPMs. Problem solved.

For PE binaries, they can be generated on package installation, with a signature shipped in a _different_ package.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 22:59 UTC (Fri) by bluca (subscriber, #118303) [Link] (6 responses)

Right, so that instead of having some packages that are not independently reproducible and need to have exceptions applied, you have some packages that are not independently reproducible and need to have exceptions applied. Great success!

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 8, 2025 0:33 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> Right, so that instead of having some packages that are not independently reproducible

A small subset of packages with signature files won't be reproducible (by design). These packages can be audited to not have anything but signature data, while large packages like the kernel can be completely bit-for-bit reproducible.

And yes, it's strictly better than the status quo.

It can even be done in a flexible way, something like `/usr/lib/share/signtab` directory with files containing hash-to-signature mappings.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 8, 2025 0:44 UTC (Sat) by bluca (subscriber, #118303) [Link] (4 responses)

> And yes, it's strictly better than the status quo.

No, it really is not - it's made-up nonsense that fails to solve a problem that doesn't exist while at the same time making everything worse on all aspects. But nice try.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 8, 2025 0:48 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

I understand why systemd is such a mess now...

First, not _everyone_ needs or wants the kernel signatures. I don't need them, I have enrolled my own keys into the Secure Boot. But I for sure want to have a guarantee that my kernel was indeed built from the supplied sources. Ideally integrated into the package management system.

In my scenario, I just won't bother installing the non-reproducible signature packages. They can even be put into a separate package repository, actually (like it's done with non-free right now).

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 9, 2025 20:03 UTC (Sun) by k3ninho (subscriber, #50375) [Link] (2 responses)

I'm happier with a VM on a hyperscaler hosting platform that's able to chain together integrity measurements and for systemd to enable the chain.

K3n.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 10, 2025 8:28 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Which hyperscaler? AWS doesn't support TPM measurements.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 12, 2025 13:45 UTC (Wed) by surajm (subscriber, #135863) [Link]

AWS supports it on bare metal VMs. Google Cloud and Azure support it on normal VMs.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 13, 2025 14:27 UTC (Thu) by pjones (subscriber, #31722) [Link] (1 responses)

This sounds like an okay idea, but it is going to be /fraught/ with bugs because of how PE signatures are embedded. You really want the signature to be bundled to avoid discrepancies computing the hash due to the alignment and padding at the end, which is optional if the binary doesn't have a signature embedded.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 13, 2025 18:38 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

But that's the same issue, just in reverse. If you want to strip signatures, you have to parse the PE format anyway. With signatures in a separate package, you can have easy checks by just doing `diff`, and the non-deterministic PE artifacts are produced locally.

This is fine for two reasons:
1. The PE binaries need to be placed on a separate EFI partition anyway.
2. You still need to handle custom signing keys for people using their own secure boot keys.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 10, 2025 13:28 UTC (Mon) by Foxboron (subscriber, #108330) [Link] (2 responses)

We (as in the repro community) have sorta come to the conclusion that embedding signatures into artifacts is a mistake for this very reason. apt, apk, pacman and... more(?) provide these as separate artifacts and it solves quite a few problems you would not normally have.

If RPM insists on embedded signatures then I suspect RPM needs to change.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 12, 2025 3:37 UTC (Wed) by dbnichol (subscriber, #39622) [Link] (1 responses)

The primary reason to embed the signature is so it's always in sync with the data. That's why apt prefers InRelease vs Release + Release.gpg. Having the signature embedded eliminates an entire class of errors. I understand the downside for reproducibility, though.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 12, 2025 10:44 UTC (Wed) by bluca (subscriber, #118303) [Link]

Not just that, but the consumers cannot take detached signatures, neither for the EFI binariers case nor the kmods case. Which means you need a post-install step to reattach them, which means there's extra work to do on every node, which means it's extra things that can (and will) go wrong, and another step back from being able to get rid of maintainer scripts in packages, which _reduce_ reproducibility of the system as a whole as they need to run on the targets rather than on the build systems, so there will invariably be differences in behaviours and results.

Tools like diffoscope already deep dive into binaries being inspected, it shouldn't be hard to make them ignore difference in signatures. There is no point whatsoever in comparing signature data when comparing binaries anyway, it's completely pointless and doesn't provide any useful information or data. In fact, being forced to do so downgrades security of the target systems, as you are no longer able to, for example, use ephemeral keys for kernel modules, which is much better and safer (generate key at beginning of the build, throw it away at the end, no way for attackers to use it).

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 17:44 UTC (Fri) by t-8ch (subscriber, #90907) [Link] (1 responses)

> Why not just improve the tools that do the diffing to strip the signatures from the modules before comparing them? If they are bit-by-bit identical, the same signature could even be just reattached and the result should be identical.

One-off exceptions would have to be maintained in all comparison tools.
And it gets much more complicated with derived build artifacts.
For example a package index of an archive containing a package repository with a kernel package.
Suddenly the tool would need to know which checksum in that package index are valid to be non-reproducible.
Or a built-from-source root filesystem.

Also in addition to the appended signatures the keyring embedded into vmlinux (but only the one used for modules!) would need to be ignored, which is not as straight-forward.

(disclaimer: I'm the author of the patchset under discussion)

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 18:39 UTC (Fri) by bluca (subscriber, #118303) [Link]

You need to be able to handle exceptions anyway, as there are many signed artifacts. For example, EFI binaries.

Can’t public-key verification be used for third party modules?

Posted Mar 7, 2025 22:58 UTC (Fri) by epa (subscriber, #39769) [Link] (2 responses)

Out-of-tree modules, DKMS modules, and so on can't have their hashes included in the kernel, and therefore can't be verified by hash.
No — but they could still be signed with half of a keypair, and the other half included with the kernel so it can validate the signature. You don’t need both keys to build a kernel image, so you can have a reproducible build that accepts signed modules, while not handing out the key required to sign them.

I understand the problem the article explains for modules produced as part of the kernel build: those can’t be signed without also distributing the signing key. But for modules distributed separately that problem doesn’t apply. So the kernel should support two ways to check a module: either a fixed set of hashes (for modules part of the original build), or public-key cryptography (for third party modules you want to sign).

Can’t public-key verification be used for third party modules?

Posted Mar 10, 2025 12:18 UTC (Mon) by daroc (editor, #160859) [Link]

Oh, that's a good point. I had assumed that since DKMS modules are built after the kernel on which they depend that this couldn't work, but you're right that you could generate a key, embed the public key in the kernel, and then sign the DKMS build with the private key.

Can’t public-key verification be used for third party modules?

Posted Mar 10, 2025 15:43 UTC (Mon) by t-8ch (subscriber, #90907) [Link]

> So the kernel should support two ways to check a module: either a fixed set of hashes (for modules part of the original build), or public-key cryptography (for third party modules you want to sign).

This is indeed possible since v2 of the patchset.

Reproducible but individually signed?

Posted Mar 7, 2025 23:13 UTC (Fri) by SLi (subscriber, #53131) [Link] (3 responses)

This sounds to me like one of those problems that, if defined the way I'd guess from the article, is not just technically impossible but definitionally impossible.

A signature is a piece of data that satisfies these properties:

1. Given your private key, you can produce it efficiently.
2. It can be verified efficiently without access to your private key.
3. It cannot be produced efficiently without your private key.

In other words, non-reproducibility without private data is a very essential part of what a signature is.

We might be able to get reproducibility in the sense that the outputs are identical given the same secret key, but that doesn't really sound very useful.

I like both reproducibility and signatures a lot. I think both are necessary. But they seem clearly mutually exclusive, unless you manage to move the signature out of the part for which you require reproducibility, or are happy with "reproducibility if you can access the key".

This does feel like a problem where I'd say the only way to have this in general would be for the reproducibility projects to define sufficiently general means to ship signatures outside the measured blob.

Reproducible but individually signed?

Posted Mar 8, 2025 1:29 UTC (Sat) by JoeBuck (subscriber, #2330) [Link]

Seems that if there were an efficient way to segregate everything that is expected to be reproducible and bit-for-bit identical, from the small amount of data (signatures, perhaps build info) that is expected not to be, in such a way to guarantee that all of the code falls into the first bucket, we could satisfy both requirements. So, instead of just running the equivalent of 'cmp' on everything, there'd be a smarter compare tool.

I recall that the GCC bootstrap procedure, where it checks that the stage 2 and stage 3 compilers are bit-for-bit identical, had a way to exclude time stamps on OSes that add time stamps to object files and only compare the rest. So, something like that.

Reproducible but individually signed?

Posted Mar 10, 2025 8:33 UTC (Mon) by Niflmir (subscriber, #175249) [Link] (1 responses)

I don't think you need to be able to reproduce a signature to verify that a build is reproducible. The verification of the signature itself proves that if I had the private key I could reproduce the signature which proves that the entire build with the signature is reproducible if the signed content has been proven reproducible.

I don't ever actually need to reproduce the entire build to gain the benefits of reproducible builds, I just need to prove that I could. That is the asymmetry of PKI: zero knowledge proofs as opposed to constructive proofs.

Reproducible but individually signed?

Posted Mar 10, 2025 10:11 UTC (Mon) by SLi (subscriber, #53131) [Link]

I'm not entirely sure if I understood correctly what you are saying, but I'll respond to what I think you are. Is this a fair rephrasing of your comment?

> If you have independently verified that a build is reproducible, then having an external signature does not in practice harm the reproducibility; and if you trust the signer to have signed the authentic output, verifying that signature against your build guarantees that you have the same result.

If so:

I think that depends on whether we want to have signatures _within_ the artifact being reproduced (such as the kernel) or external to it. If the signatures need to be inside what we measure *and* what we want to be reproducible, I don't think that's feasible. I think your comment aligns with what I suggested about moving the signature outside the reproducibility scope.

That is, if we have a package that doesn't itself contain PKI that you are supposed to be in control of inside the scope of reproducibility, then, yes, we could verify an external signature against the build artifact. But that's not the use case here. Here we want to have a kernel package that inherently contains a public key and verifies signatures on modules that are loaded. It needs this to implement the functionality of verifying that the modules it loads are legitimate (blessed by you).

Sure, we could make a "reproducible" kernel build where we just embed a key that someone else generated and have all kernel modules contain a signature by this third party, as long as we never modify them. This would allow us to load exactly these blessed versions. It would also enable us to rebuild that precise kernel and load exactly those modules, but we couldn't modify a module's source code and load it because the signature check would fail. On the other hand, this third party would be able to sign a malicious module and load it the computer. I find this less than ideal.

And my point is that unless we have a method to have the signatures and the public key outside the scope of reproducibility, that is fundamentally, *definitionally* incompatible with the user having control over the signatures (it's by definition not a useful cryptographic signature if anyone can reproduce it).

OTOH what I think could be done, in principle, is to move the signing part out of the kernel building part, but that would require some infrastructure. We could have a "kernel with holes for signatures" that is reproducible and that could be turned into a non-reproducible "kernel with signatures". If something like this were to happen, I'd hope the mechanism would be generic enough that it's not a one-off for the kernel use case, since the problem is clearly more generic than that.

But I also don't think it's possible to just wave hands here and say that the kernel people or cryptographers need to find a solution that satisfies the constraints the way we now measure reproducibility, because that is actually very deeply, definitionally impossible, like a square with three sides.

Maybe use a signature oracle

Posted Mar 8, 2025 18:11 UTC (Sat) by iabervon (subscriber, #722) [Link] (1 responses)

Rather that having the private key and generating new signatures (that won't be bit-for-bit identical to any other signatures), it should be possible to have a list of signatures previously made with the correct private key and just attach the appropriate ones, provided your new build only produces output which is bit-for-bit identical to output from a previous build. Presumably, it would be okay for the attempt to reproduce an earlier build to fail during the build, rather than succeeding but producing different fully-worked output, and someone would want to investigate the first different intermediate product, rather than taking two final packages and trying to figure out why they aren't the same.

Maybe use a signature oracle

Posted Mar 8, 2025 20:15 UTC (Sat) by mathstuf (subscriber, #69389) [Link]

Doesn't F-Droid do this so that upstreams can sign with their key but F-Droid compiles-from-source verify on its own?


Copyright © 2025, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds