Hash-based module integrity checking
On January 20, Thomas Weißschuh shared a new patch set implementing an alternate method for checking the integrity of loadable kernel modules. This mechanism, which checks module integrity based on hashes computed at build time instead of using cryptographic signatures, could enable reproducible kernel builds in more contexts. Several distributions have already expressed interest in the patch set if Weißschuh can get it into the kernel.
Linux has supported signing loadable kernel modules since 2012, when David Howells introduced the first code for it. Since then, the implementation has not changed too much. Users can enable the CONFIG_MODULE_SIG option to turn on module signing; by default, this will simply taint the kernel if an unsigned module is loaded. Enabling CONFIG_MODULE_SIG_FORCE or the Lockdown Linux Security Module (LSM) prevents the kernel from loading unsigned modules. The public keys needed to verify signatures are baked into the kernel at build time. The build process can be configured to use an existing keypair, or to automatically generate the necessary asymmetric keys.
That automatic generation is Weißschuh's gripe with the current code. Reproducible builds are important for security, since they allow independent verification that an open-source project has been compiled without inserting extra, malicious changes. For something as foundational as the Linux kernel, it would be nice to be able to verify that a build of the kernel is unmodified. But when signing keys are needed for the build, it cannot be made reproducible without distributing the key. Currently, this puts users in a bind. They cannot have loadable modules, reproducible builds, and signature verification all turned on at the same time.
It's tempting to search for a clever cryptographic solution, but nobody has yet proposed one. If the signing keys are publicly available for use in recreating the build, malicious actors could also sign modified loadable modules with them. If they aren't publicly available, the build can't be reproduced. So Weißschuh's patch set takes a much simpler approach: instead of trying to create a signature for loadable modules, the patch set calculates cryptographic hashes for all modules built with the kernel, and embeds those hashes as a static list to verify prospective modules before they are loaded.
This has the benefit of simplicity — if a module has the same hash that it had at build time, it certainly hasn't been tampered with — but it is a bit less flexible than the signature-based approach. Specifically, Weißschuh's patch set only works for modules that are built at the same time as the kernel. The default build configuration, which generates new keys for each build, has the same limitation, but users can use their own long-term signing keys if they prefer. Out-of-tree modules, DKMS modules, and so on can't have their hashes included in the kernel, and therefore can't be verified by hash.
Nothing prevents users from enabling both Weißschuh's new code (with CONFIG_MODULE_HASHES) and the existing signature verification support, although in that case the resulting kernel isn't reproducible. This will build all the in-tree modules with hashes (but not signatures), and still allow the loading of signed (but not hashed) out-of-tree modules. Arch Linux contributor "kpcyrd" summarized the possible options.
Combining the settings for module hashing and module signing yield four possibilities. When neither CONFIG_MODULE_HASHES nor CONFIG_MODULE_SIG are enabled, the kernel does not perform any checks (and therefore Lockdown will reject loading any modules). When only CONFIG_MODULE_SIG is enabled, the behavior is the same as today — signed modules can be loaded. When CONFIG_MODULE_HASHES is enabled, modules with a known hash can be loaded. When both are enabled, a module that has either a known hash or a valid signature can be loaded. If CONFIG_MODULE_SIG_FORCE is enabled, CONFIG_MODULE_SIG must also be enabled. If neither CONFIG_MODULE_SIG_FORCE nor the Lockdown LSM are enabled, the kernel won't disallow modules without valid signatures or hashes, and therefore there is little point in configuring either signed or hashed modules.
For users of Arch Linux, Proxmox, and SUSE — projects that Weißschuh indicated were interested in this work based on the previous discussion — the last option is the most likely configuration for a normal kernel build. NixOS does not enable module signing by default, and is more likely to only enable CONFIG_MODULE_HASHES, but was still interested. The patch set is not yet quite ready to go into the mainline kernel, however. Petr Pavlu suggested a few tweaks for Weißschuh to make. A handful of other people participated in the discussion of the patch set, although there was not too much objection to the patch set in face of the evident interest from the different distributions. Once he revises the patch set, it seems likely to remove one of the few remaining obstacles to deploying reproducible kernel builds.
Posted Mar 7, 2025 16:06 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (19 responses)
Why not just improve the tools that do the diffing to strip the signatures from the modules before comparing them? If they are bit-by-bit identical, the same signature could even be just reattached and the result should be identical.
Posted Mar 7, 2025 17:25 UTC (Fri)
by josh (subscriber, #17465)
[Link] (16 responses)
Posted Mar 7, 2025 18:41 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (1 responses)
Considering inline signatures as part of the reproducible envelope doesn't provide any useful information about the state of the builds. It's not interesting data - it's just RSA. Given the same input, and the same key, you get the same output, so it's reproducible by definition.
Posted Mar 13, 2025 14:24 UTC (Thu)
by pjones (subscriber, #31722)
[Link]
Which means you're right that this is primarily a tooling problem in our comparisons.
Posted Mar 7, 2025 18:58 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (13 responses)
Posted Mar 7, 2025 20:01 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (9 responses)
For PE binaries, they can be generated on package installation, with a signature shipped in a _different_ package.
Posted Mar 7, 2025 22:59 UTC (Fri)
by bluca (subscriber, #118303)
[Link] (6 responses)
Posted Mar 8, 2025 0:33 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (5 responses)
A small subset of packages with signature files won't be reproducible (by design). These packages can be audited to not have anything but signature data, while large packages like the kernel can be completely bit-for-bit reproducible.
And yes, it's strictly better than the status quo.
It can even be done in a flexible way, something like `/usr/lib/share/signtab` directory with files containing hash-to-signature mappings.
Posted Mar 8, 2025 0:44 UTC (Sat)
by bluca (subscriber, #118303)
[Link] (4 responses)
No, it really is not - it's made-up nonsense that fails to solve a problem that doesn't exist while at the same time making everything worse on all aspects. But nice try.
Posted Mar 8, 2025 0:48 UTC (Sat)
by Cyberax (✭ supporter ✭, #52523)
[Link] (3 responses)
First, not _everyone_ needs or wants the kernel signatures. I don't need them, I have enrolled my own keys into the Secure Boot. But I for sure want to have a guarantee that my kernel was indeed built from the supplied sources. Ideally integrated into the package management system.
In my scenario, I just won't bother installing the non-reproducible signature packages. They can even be put into a separate package repository, actually (like it's done with non-free right now).
Posted Mar 9, 2025 20:03 UTC (Sun)
by k3ninho (subscriber, #50375)
[Link] (2 responses)
K3n.
Posted Mar 10, 2025 8:28 UTC (Mon)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Mar 12, 2025 13:45 UTC (Wed)
by surajm (subscriber, #135863)
[Link]
Posted Mar 13, 2025 14:27 UTC (Thu)
by pjones (subscriber, #31722)
[Link] (1 responses)
Posted Mar 13, 2025 18:38 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link]
This is fine for two reasons:
Posted Mar 10, 2025 13:28 UTC (Mon)
by Foxboron (subscriber, #108330)
[Link] (2 responses)
If RPM insists on embedded signatures then I suspect RPM needs to change.
Posted Mar 12, 2025 3:37 UTC (Wed)
by dbnichol (subscriber, #39622)
[Link] (1 responses)
Posted Mar 12, 2025 10:44 UTC (Wed)
by bluca (subscriber, #118303)
[Link]
Tools like diffoscope already deep dive into binaries being inspected, it shouldn't be hard to make them ignore difference in signatures. There is no point whatsoever in comparing signature data when comparing binaries anyway, it's completely pointless and doesn't provide any useful information or data. In fact, being forced to do so downgrades security of the target systems, as you are no longer able to, for example, use ephemeral keys for kernel modules, which is much better and safer (generate key at beginning of the build, throw it away at the end, no way for attackers to use it).
Posted Mar 7, 2025 17:44 UTC (Fri)
by t-8ch (subscriber, #90907)
[Link] (1 responses)
One-off exceptions would have to be maintained in all comparison tools.
Also in addition to the appended signatures the keyring embedded into vmlinux (but only the one used for modules!) would need to be ignored, which is not as straight-forward.
(disclaimer: I'm the author of the patchset under discussion)
Posted Mar 7, 2025 18:39 UTC (Fri)
by bluca (subscriber, #118303)
[Link]
Posted Mar 7, 2025 22:58 UTC (Fri)
by epa (subscriber, #39769)
[Link] (2 responses)
I understand the problem the article explains for modules produced as part of the kernel build: those can’t be signed without also distributing the signing key. But for modules distributed separately that problem doesn’t apply. So the kernel should support two ways to check a module: either a fixed set of hashes (for modules part of the original build), or public-key cryptography (for third party modules you want to sign).
Posted Mar 10, 2025 12:18 UTC (Mon)
by daroc (editor, #160859)
[Link]
Posted Mar 10, 2025 15:43 UTC (Mon)
by t-8ch (subscriber, #90907)
[Link]
This is indeed possible since v2 of the patchset.
Posted Mar 7, 2025 23:13 UTC (Fri)
by SLi (subscriber, #53131)
[Link] (3 responses)
A signature is a piece of data that satisfies these properties:
1. Given your private key, you can produce it efficiently.
In other words, non-reproducibility without private data is a very essential part of what a signature is.
We might be able to get reproducibility in the sense that the outputs are identical given the same secret key, but that doesn't really sound very useful.
I like both reproducibility and signatures a lot. I think both are necessary. But they seem clearly mutually exclusive, unless you manage to move the signature out of the part for which you require reproducibility, or are happy with "reproducibility if you can access the key".
This does feel like a problem where I'd say the only way to have this in general would be for the reproducibility projects to define sufficiently general means to ship signatures outside the measured blob.
Posted Mar 8, 2025 1:29 UTC (Sat)
by JoeBuck (subscriber, #2330)
[Link]
Seems that if there were an efficient way to segregate everything that is expected to be reproducible and bit-for-bit identical, from the small amount of data (signatures, perhaps build info) that is expected not to be, in such a way to guarantee that all of the code falls into the first bucket, we could satisfy both requirements. So, instead of just running the equivalent of 'cmp' on everything, there'd be a smarter compare tool.
I recall that the GCC bootstrap procedure, where it checks that the stage 2 and stage 3 compilers are bit-for-bit identical, had a way to exclude time stamps on OSes that add time stamps to object files and only compare the rest. So, something like that.
Posted Mar 10, 2025 8:33 UTC (Mon)
by Niflmir (subscriber, #175249)
[Link] (1 responses)
I don't ever actually need to reproduce the entire build to gain the benefits of reproducible builds, I just need to prove that I could. That is the asymmetry of PKI: zero knowledge proofs as opposed to constructive proofs.
Posted Mar 10, 2025 10:11 UTC (Mon)
by SLi (subscriber, #53131)
[Link]
> If you have independently verified that a build is reproducible, then having an external signature does not in practice harm the reproducibility; and if you trust the signer to have signed the authentic output, verifying that signature against your build guarantees that you have the same result.
If so:
I think that depends on whether we want to have signatures _within_ the artifact being reproduced (such as the kernel) or external to it. If the signatures need to be inside what we measure *and* what we want to be reproducible, I don't think that's feasible. I think your comment aligns with what I suggested about moving the signature outside the reproducibility scope.
That is, if we have a package that doesn't itself contain PKI that you are supposed to be in control of inside the scope of reproducibility, then, yes, we could verify an external signature against the build artifact. But that's not the use case here. Here we want to have a kernel package that inherently contains a public key and verifies signatures on modules that are loaded. It needs this to implement the functionality of verifying that the modules it loads are legitimate (blessed by you).
Sure, we could make a "reproducible" kernel build where we just embed a key that someone else generated and have all kernel modules contain a signature by this third party, as long as we never modify them. This would allow us to load exactly these blessed versions. It would also enable us to rebuild that precise kernel and load exactly those modules, but we couldn't modify a module's source code and load it because the signature check would fail. On the other hand, this third party would be able to sign a malicious module and load it the computer. I find this less than ideal.
And my point is that unless we have a method to have the signatures and the public key outside the scope of reproducibility, that is fundamentally, *definitionally* incompatible with the user having control over the signatures (it's by definition not a useful cryptographic signature if anyone can reproduce it).
OTOH what I think could be done, in principle, is to move the signing part out of the kernel building part, but that would require some infrastructure. We could have a "kernel with holes for signatures" that is reproducible and that could be turned into a non-reproducible "kernel with signatures". If something like this were to happen, I'd hope the mechanism would be generic enough that it's not a one-off for the kernel use case, since the problem is clearly more generic than that.
But I also don't think it's possible to just wave hands here and say that the kernel people or cryptographers need to find a solution that satisfies the constraints the way we now measure reproducibility, because that is actually very deeply, definitionally impossible, like a square with three sides.
Posted Mar 8, 2025 18:11 UTC (Sat)
by iabervon (subscriber, #722)
[Link] (1 responses)
Posted Mar 8, 2025 20:15 UTC (Sat)
by mathstuf (subscriber, #69389)
[Link]
Just strip the signature before comparing the rebuilt artifacts?
It seems the binutils strip is able to remove the signature from a signed kernel module, from a quick local test.
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
1. The PE binaries need to be placed on a separate EFI partition anyway.
2. You still need to handle custom signing keys for people using their own secure boot keys.
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
Just strip the signature before comparing the rebuilt artifacts?
And it gets much more complicated with derived build artifacts.
For example a package index of an archive containing a package repository with a kernel package.
Suddenly the tool would need to know which checksum in that package index are valid to be non-reproducible.
Or a built-from-source root filesystem.
Just strip the signature before comparing the rebuilt artifacts?
Can’t public-key verification be used for third party modules?
Out-of-tree modules, DKMS modules, and so on can't have their hashes included in the kernel, and therefore can't be verified by hash.
No — but they could still be signed with half of a keypair, and the other half included with the kernel so it can validate the signature. You don’t need both keys to build a kernel image, so you can have a reproducible build that accepts signed modules, while not handing out the key required to sign them.
Can’t public-key verification be used for third party modules?
Can’t public-key verification be used for third party modules?
Reproducible but individually signed?
2. It can be verified efficiently without access to your private key.
3. It cannot be produced efficiently without your private key.
Reproducible but individually signed?
Reproducible but individually signed?
Reproducible but individually signed?
Maybe use a signature oracle
Maybe use a signature oracle
