Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 16:06 UTC (Fri) by bluca (subscriber, #118303)
Parent article: Hash-based module integrity checking

> Currently, this puts users in a bind. They cannot have loadable modules, reproducible builds, and signature verification all turned on at the same time.

Why not just improve the tools that do the diffing to strip the signatures from the modules before comparing them? If they are bit-by-bit identical, the same signature could even be just reattached and the result should be identical.
It seems the binutils strip is able to remove the signature from a signed kernel module, from a quick local test.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 17:25 UTC (Fri) by josh (subscriber, #17465) [Link] (16 responses)

That's definitely possible, but it means you can't (for instance) have whole *software packages* (e.g. debs) that are reproducible.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 18:41 UTC (Fri) by bluca (subscriber, #118303) [Link] (1 responses)

You still can't have the kernel package reproducible without the signing key, as the kernel PE image itself needs to be signed.

Considering inline signatures as part of the reproducible envelope doesn't provide any useful information about the state of the builds. It's not interesting data - it's just RSA. Given the same input, and the same key, you get the same output, so it's reproducible by definition.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 13, 2025 14:24 UTC (Thu) by pjones (subscriber, #31722) [Link]

I think this is a problem of how we're discussing it more than anything else - the signed kernel is reproducible in the meaningful sense, but the data to validate it is effectively encapsulating it.

Which means you're right that this is primarily a tooling problem in our comparisons.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 18:58 UTC (Fri) by bluca (subscriber, #118303) [Link] (13 responses)

Also that statement ignores that half of the world signs packages too - RPMs are themselves signed.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 20:01 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

Well, don't use RPMs. Problem solved.

For PE binaries, they can be generated on package installation, with a signature shipped in a _different_ package.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 22:59 UTC (Fri) by bluca (subscriber, #118303) [Link] (6 responses)

Right, so that instead of having some packages that are not independently reproducible and need to have exceptions applied, you have some packages that are not independently reproducible and need to have exceptions applied. Great success!

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 8, 2025 0:33 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> Right, so that instead of having some packages that are not independently reproducible

A small subset of packages with signature files won't be reproducible (by design). These packages can be audited to not have anything but signature data, while large packages like the kernel can be completely bit-for-bit reproducible.

And yes, it's strictly better than the status quo.

It can even be done in a flexible way, something like `/usr/lib/share/signtab` directory with files containing hash-to-signature mappings.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 8, 2025 0:44 UTC (Sat) by bluca (subscriber, #118303) [Link] (4 responses)

> And yes, it's strictly better than the status quo.

No, it really is not - it's made-up nonsense that fails to solve a problem that doesn't exist while at the same time making everything worse on all aspects. But nice try.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 8, 2025 0:48 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

I understand why systemd is such a mess now...

First, not _everyone_ needs or wants the kernel signatures. I don't need them, I have enrolled my own keys into the Secure Boot. But I for sure want to have a guarantee that my kernel was indeed built from the supplied sources. Ideally integrated into the package management system.

In my scenario, I just won't bother installing the non-reproducible signature packages. They can even be put into a separate package repository, actually (like it's done with non-free right now).

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 9, 2025 20:03 UTC (Sun) by k3ninho (subscriber, #50375) [Link] (2 responses)

I'm happier with a VM on a hyperscaler hosting platform that's able to chain together integrity measurements and for systemd to enable the chain.

K3n.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 10, 2025 8:28 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

Which hyperscaler? AWS doesn't support TPM measurements.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 12, 2025 13:45 UTC (Wed) by surajm (subscriber, #135863) [Link]

AWS supports it on bare metal VMs. Google Cloud and Azure support it on normal VMs.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 13, 2025 14:27 UTC (Thu) by pjones (subscriber, #31722) [Link] (1 responses)

This sounds like an okay idea, but it is going to be /fraught/ with bugs because of how PE signatures are embedded. You really want the signature to be bundled to avoid discrepancies computing the hash due to the alignment and padding at the end, which is optional if the binary doesn't have a signature embedded.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 13, 2025 18:38 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

But that's the same issue, just in reverse. If you want to strip signatures, you have to parse the PE format anyway. With signatures in a separate package, you can have easy checks by just doing `diff`, and the non-deterministic PE artifacts are produced locally.

This is fine for two reasons:
1. The PE binaries need to be placed on a separate EFI partition anyway.
2. You still need to handle custom signing keys for people using their own secure boot keys.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 10, 2025 13:28 UTC (Mon) by Foxboron (subscriber, #108330) [Link] (2 responses)

We (as in the repro community) have sorta come to the conclusion that embedding signatures into artifacts is a mistake for this very reason. apt, apk, pacman and... more(?) provide these as separate artifacts and it solves quite a few problems you would not normally have.

If RPM insists on embedded signatures then I suspect RPM needs to change.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 12, 2025 3:37 UTC (Wed) by dbnichol (subscriber, #39622) [Link] (1 responses)

The primary reason to embed the signature is so it's always in sync with the data. That's why apt prefers InRelease vs Release + Release.gpg. Having the signature embedded eliminates an entire class of errors. I understand the downside for reproducibility, though.

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 12, 2025 10:44 UTC (Wed) by bluca (subscriber, #118303) [Link]

Not just that, but the consumers cannot take detached signatures, neither for the EFI binariers case nor the kmods case. Which means you need a post-install step to reattach them, which means there's extra work to do on every node, which means it's extra things that can (and will) go wrong, and another step back from being able to get rid of maintainer scripts in packages, which _reduce_ reproducibility of the system as a whole as they need to run on the targets rather than on the build systems, so there will invariably be differences in behaviours and results.

Tools like diffoscope already deep dive into binaries being inspected, it shouldn't be hard to make them ignore difference in signatures. There is no point whatsoever in comparing signature data when comparing binaries anyway, it's completely pointless and doesn't provide any useful information or data. In fact, being forced to do so downgrades security of the target systems, as you are no longer able to, for example, use ephemeral keys for kernel modules, which is much better and safer (generate key at beginning of the build, throw it away at the end, no way for attackers to use it).

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 17:44 UTC (Fri) by t-8ch (subscriber, #90907) [Link] (1 responses)

> Why not just improve the tools that do the diffing to strip the signatures from the modules before comparing them? If they are bit-by-bit identical, the same signature could even be just reattached and the result should be identical.

One-off exceptions would have to be maintained in all comparison tools.
And it gets much more complicated with derived build artifacts.
For example a package index of an archive containing a package repository with a kernel package.
Suddenly the tool would need to know which checksum in that package index are valid to be non-reproducible.
Or a built-from-source root filesystem.

Also in addition to the appended signatures the keyring embedded into vmlinux (but only the one used for modules!) would need to be ignored, which is not as straight-forward.

(disclaimer: I'm the author of the patchset under discussion)

Just strip the signature before comparing the rebuilt artifacts?

Posted Mar 7, 2025 18:39 UTC (Fri) by bluca (subscriber, #118303) [Link]

You need to be able to handle exceptions anyway, as there are many signed artifacts. For example, EFI binaries.