Testing

Posted Apr 2, 2025 5:52 UTC (Wed) by marcH (subscriber, #57642)
Parent article: Fedora change aims for 99% package reproducibility

> Rebuilding a package the second time in the same build environment means "we are not testing much".

As often, the discussion seems to fall a bit short on the testing side. This is not specific to reproducibility, it affects all software in general. Disclaimer: I've only read the LWN article, nothing it points to.

If some feature / parameter is not tested then it does not work. The test suite and efforts are the real specification. But testing (and claiming) build reproducibility is surprisingly hard, probably as hard as testing race conditions (which funny enough can expose build reproducibility issues). You can build in 4 different environments and claim victory until someone tries a slightly different version of some very minor build time dependency that everyone forgot about. While this forgotten dependency may have no security consequence, it's always enough to break the checksums.

In my case the "breakthrough" happened when the same toolchain was available on both Linux and Windows (+ some MinGW magic). That exposed the very "last" round of reproducibility issues. All other issues after that were recent regressions. Two random systems are incredibly unlikely to have some unexpected environment / dependency differences that a Linux and a Windows build systems don't also have. Now you're entering the realm of "theoretical" bugs that are possible in theory but never happen in practice.

It seems possible to build (RPM) packages on "foreign" Linux distributions, maybe that would provide good test coverage?

Testing

Posted Apr 2, 2025 8:21 UTC (Wed) by bmwiedemann (subscriber, #71319) [Link]

You need to build with the same gcc, glibc, headers etc... so normally you don't get identical results on different distributions.

The exception is when you build in a container/chroot. Then only embedding uname -r output would break reproducibility (several packages do that).

And the usual rules apply:
https://reproducible-builds.org/docs/commandments/

Having worked on a OS that has all its 3000 packages bit-reproducible, race-conditions were indeed among the hardest. You are not guaranteed to trigger them anywhere and you cannot be certain they are gone after a fix.
Some race only occurred on a build-machine with HDD. Another race only showed up on the fastest machine with NVMe (8GB/s)

OTOH, when a race is sufficiently rare, it is still possible to verify official binaries with few tries.