LWN: Comments on "A backdoor in a popular Ruby gem"

A backdoor in a popular Ruby gem

njs — Sat, 20 Apr 2019 05:56:46 +0000

Reproducible builds are a great thing, but they're not the first step I'd take.

If RubyGems switched to a model where you gave it the VCS tag, and then it used that to build the distributed artifacts itself, then that would already get the main guarantee, without any reproducible builds infrastructure. This is strictly simpler than your proposal, since you also require that RubyGems build the artifacts itself.

Once you have that, adding reproducible builds on top would add some value: it would let third-parties validate that the RubyGems build infrastructure hasn't been compromised. This would certainly be a nice capability to have, but it's not really the first priority. (Everyone who downloads from RubyGems already trusts the RubyGems infrastructure in lots of ways!)

A backdoor in a popular Ruby gem

flussence — Mon, 15 Apr 2019 15:57:01 +0000

It's a sad state of affairs. One would think some lessons would be learned after the first 50 or so high profile incidents like this.

So… which language ecosystems (if any) *have* bothered to learn? I've seen a few attempts at papering over the multiple-upstreams-single-package problem which could only generously be described as “advisory security”.

A backdoor in a popular Ruby gem

k8to — Mon, 15 Apr 2019 15:54:09 +0000

I think the question I have about articles like this is what does a package repo that is not the wild west look like?

Go notary proposal

ms — Mon, 15 Apr 2019 11:19:39 +0000

Yeah, this is currently true because "replace" is only honoured in the main module. And you can't even have a go.mod file include another go.mod-style file so you can't do convenient hacks using eg symlinks. Which means your "build" system is going to have to include random set of "go mod edit" commands, or you start messing with stuff like gohack.

They do seem to have admitted the design has a number of holes in it. I'm not aware of the rate of progress addressing said holes....

A backdoor in a popular Ruby gem

cyphar — Mon, 15 Apr 2019 09:31:19 +0000

I'm not a fan of Go (despite having been developing with it for the past ~6 years), but I would like to point out that this isn't a new problem with Go -- nor is it really related to reproducible builds (other than the fact that Go's builds are reproducible).

The proposal you've linked is talking about effectively adding more checks to go.sum to make it so that go.sum hashes need to be signed by a notary. There are definitely some concerns I have about how this probably would break the Debian "Deserted Island" test for free software, and how it entrenches Google as being central to you being able to build Go code (though they say there will be other notaries managed by other people) -- rather than providing some way for the module author to control the signing of their module hashes.

But this doesn't make patching any more difficult than it has been in the past. Outside of vendor/, Go has always made patching exceptionally painful (even pre-modules). And while you could patch vendor/, that's been considered a Very Bad Idea™ for quite a while. Obviously if you need to do it for a private build you can (and in the case of Go modules you could do it with "go mod vendor"), but publishing code that patches vendor/ in that way has been considered a bad practice for quite a long time.

It's also not related to reproducible builds because the go.sum hashes are the hashes of the source code, not binaries (so the reproducibility of Go is not relevant to the system being proposed).

A backdoor in a popular Ruby gem

robert_s — Sat, 13 Apr 2019 19:12:43 +0000

Newsflash: wild-west package repositories are not to be trusted.

Go notary proposal

Hattifnattar — Sat, 13 Apr 2019 14:21:51 +0000

I never worked with Go, but my impression was that Go designers generally like to put the coder in a straitjacket.
(For a really funny review of Go, see https://www.evanmiller.org/four-days-of-go.html).

So this begs the question: are these problems and inconveniences just some "Go things", or this is a general requirement for anybody trying to implement stable builds?

Go notary proposal

nim-nim — Sat, 13 Apr 2019 13:43:11 +0000

Besides, with replaces you can not even share your fixed fork within your builds, you need to add a replace directive to every single app, that uses the component requiring patching

Go notary proposal

nim-nim — Sat, 13 Apr 2019 13:41:07 +0000

And that amounts to `forcing you to fork and rename everything`, as I wrote in my original message.

Go notary proposal

Cyberax — Sat, 13 Apr 2019 09:46:54 +0000

Go modules support replacing, allowing you to use local forks. go get is not at all required.

Go notary proposal

nim-nim — Sat, 13 Apr 2019 09:40:55 +0000

To be clearer: in module mode (which will become the default in Go in August) the compiler only knows about the current module it is working on. Every other module the current module needs is taken from a cache. The cache is populated by "go get". Doing a local fix of a module, to use in other modules, therefore, becomes a "publish it for go get" operation, notary checks included.

Who is "them"?

nim-nim — Sat, 13 Apr 2019 09:23:27 +0000

The notaries automate the independant versification. And I need them, because the people writing the language spec felt they needed them.

Go notary proposal

nim-nim — Sat, 13 Apr 2019 09:20:37 +0000

`go get` and `go build` are deeply imbricated, `go build` builds things from the local project and a cache populated by `go get` (and can trigger `go get` calls during the build process). If `go get` does not accept something, basically, you can not build against this something in go.

Go notary proposal

skybrian — Sat, 13 Apr 2019 03:26:50 +0000

This proposal looks like it's all about authenticating downloads using "go get". It doesn't say that the Go compiler will be checking anything or that you can't modify the code after it was downloaded? I didn't see any mention of the Go compiler.

A backdoor in a popular Ruby gem

david.a.wheeler — Fri, 12 Apr 2019 18:43:56 +0000

Fair enough. If you're doing anything other than directly fetching from the source distribution (e.g., GitHub), then it makes sense to ensure it's a reproducible build.

A backdoor in a popular Ruby gem

david.a.wheeler — Fri, 12 Apr 2019 18:41:41 +0000

Debian certainly doesn't own the concept, that's true. But Debian also doesn't require that you create a new project for every patch, even with reproducible builds. It's simply not true that reproducible builds require the creation of a new project for every patch. Reproducible builds simply give you a way to verify that the binary and source correspond given a build toolsuite. That's a big improvement over "trust us" - you can now "trust but verify".

Sandboxing

david.a.wheeler — Fri, 12 Apr 2019 18:38:57 +0000

Travis doesn't make the promise. Travis simply runs the build and can return its results. You can then compare the new build with the old build. The build might include malicious steps... but now you can review the source (including any build instructions) instead of blindly trusting that a precompiled package is the same thing.

Who is "them"?

Cyberax — Fri, 12 Apr 2019 18:16:22 +0000

Why do you need notaries at all?

A backdoor in a popular Ruby gem

hmh — Fri, 12 Apr 2019 15:14:21 +0000

Ah, golang.

Thanks for the link, I have to digest that proposal document before commenting any further.

A backdoor in a popular Ruby gem

nim-nim — Fri, 12 Apr 2019 13:14:27 +0000

Close but no hit

https://go.googlesource.com/proposal/+/master/design/2553...

The consequence of this design is that any change to sources (ie patching to fix bugs or security issues) requires setting up a new project, since obviously you can not push your patches to the original project (if you could you would not need patching in the first place), and any patch will invalidate the notary attestation checked by the compiler.

Who is "them"?

nim-nim — Fri, 12 Apr 2019 13:04:16 +0000

And an independently-verifiable path can and has been implemented via notary robots, that perform the independant verification.

A backdoor in a popular Ruby gem

nim-nim — Fri, 12 Apr 2019 12:59:32 +0000

Debian is not the only entity working on reproducible builds and it does not own the concept.

A backdoor in a popular Ruby gem

NAR — Fri, 12 Apr 2019 10:05:47 +0000

However, very few people rebuild *EVERY* package. Most people use a lot of precompiled packages.

I think it depends on the language and ecosystem. For Erlang the most common way to get dependencies to fetch the sources from GitHub. For Elixir it looks like source files are downloaded from hex.pm (and compiled once). I don't know about ruby though.

Sandboxing

NYKevin — Fri, 12 Apr 2019 04:20:13 +0000

> If the returned hash is the same as the hash of the file provided,

I was not aware that TravisCI and friends made that guarantee. It sounds like it would be a difficult promise to keep.

Sandboxing

david.a.wheeler — Thu, 11 Apr 2019 23:36:23 +0000

> > If the sandboxed build environment can inject vulnerabilities (malicious code) into the build result, then you need to release a new version.

> No go. RubyGems has to protect itself against hostile projects. You cannot push this back onto those projects, because they are untrusted.

RubyGems currently completely and totally trusts all projects that send data to it. If a project says, "here is my package xyz version 1.2.3" then if the login is correct RubyGems says "sure" and redistributes it to everyone. That's obviously easy to implement, but it means there's no verification that the package actually *is* a compiled version as claimed.

The alternative mooted here is that RubyGems tells some service "please rebuild package xyz version 1.2.3 and send me its cryptographic hash". That external service would be something like CircleCI or TravisCI (something already designed to run externally-provided programs and produce results). If the returned hash is the same as the hash of the file provided, all is well, otherwise there's a problem that needs to be fixed. RubyGems could even provide the rebuilt file to help debug it.

It could be third parties instead of RubyGems itself, but that might create a big window of time where the correspondence was not checked.

A backdoor in a popular Ruby gem

david.a.wheeler — Thu, 11 Apr 2019 23:27:14 +0000

> Reproducible builds enable checking that the released package (gem, etc.) was indeed generated from the tagged version in GitHub/GitLab/etc. As far as I understand, in this backdooring case only the released package was modified, not the source, so an automated check on the site hosting the packages could have caught this issue before anyone downloaded the backdoored package.

Exactly right. In this case, requiring a reproducible build would have completely prevented the problem, since the attacker appears to have not had rights to the corresponding GitHub account.

But even if the attacker *had* control over the GitHub account, this check would have prevented *hidden* changes not visible in the official source repo. Instead, the exact changes would have to be directly and publicly visible (if the repo is public). While that doesn't *prevent* an attack, it does make the attack immediately visible.

> If I use dependency X in my project (be it libfoo, or gembar, or whatever), any deployment needs to get X from somewhere: either grab the source and build it yourself, or get a pre-built package (say from rubygems.org).

> If X has bugs that I want fixed, but X’s upstream is slow to incorporate these (or totally MIA), I can fork off X′. But I can neither force it into the source repository of X nor replace X on rubygems.org with my X′. I can of course upload X′ to rubygems as a different package.

That's true, but not the point.

The problem is that it's easy to put a package on distribution site (like RubyGems) that is *supposed* to correspond to some source code, but doesn't actually correspond. Most people would assume that Rubygems gem xyzzy version 1.2.3.4 was generated from xyzzy's official source repo (wherever that is), but no mechanism today actually ensures that. It's all honor system. Which works until there's an attacker.

Reproducible builds simply let you confirm that a given compiled package of some package P corresponds to its putative source code. They don't guarantee there is no malicious code... but they do guarantee that if there's malicious code, it comes from either the source or the underlying system used to build it. (If you're worried about the latter point, my Diverse Double-Compiling approach will counter that, but let's get reproducible builds first.)

You can still use X' if you want. If you have your own source, you can just build it yourself. However, very few people rebuild *EVERY* package. Most people use a lot of precompiled packages. Reproducible builds simply let you confirm that the precompiled packages you use correspond to their source code (if the build tools are not malicious). And that's a significant improvement.

Who is "them"?

david.a.wheeler — Thu, 11 Apr 2019 23:13:15 +0000

I have no idea what you're talking about.

I didn't find anything like that on Google.

The front page of https://reproducible-builds.org/ instead says: "Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code." No notaries, no requirements that require you to change the universe to release a new patch.

Again: citation needed. I think you have a misunderstanding of what the reproducible builds folks are trying to do. That's no crime, but I suggest checking it out further.

A backdoor in a popular Ruby gem

hmh — Thu, 11 Apr 2019 21:11:32 +0000

Where did you find this? Who are "them"?

Because that sort of definition really reeks of enterprise-grade supply chain hardening, gone subtly wrong. Or not so subtly.

A backdoor in a popular Ruby gem

mathstuf — Thu, 11 Apr 2019 20:28:55 +0000

> The "integration" for a distribution service is really just downloading from the repo (e.g., "git clone") when it receives a new module to distribute

An improvement would be to fetch it from one of the wonderful software archives we've had articles about here (Software Heritage among others). If it isn't in there, submit it and then build from that.

A backdoor in a popular Ruby gem

NAR — Thu, 11 Apr 2019 20:11:55 +0000

Reproducible builds enable checking that the released package (gem, etc.) was indeed generated from the tagged version in GitHub/GitLab/etc. As far as I understand, in this backdooring case only the released package was modified, not the source, so an automated check on the site hosting the packages could have caught this issue before anyone downloaded the backdoored package.

A backdoor in a popular Ruby gem

Cyberax — Thu, 11 Apr 2019 17:15:32 +0000

> You've visibly not encountered the latest advancements in reproducibility (lucky you).
I have.

> They define reproducible as "source code hash, patches included, match the hash computed by a notary robot that attempted to download the code at some unknown point of time".
Uh, this is incorrect. Debian uses regular packages for reproducible builds. A system I worked on used internal repositories.

You simply need a way to recreate the build environment exactly. No notary robots need or anything.

A backdoor in a popular Ruby gem

NYKevin — Thu, 11 Apr 2019 17:02:33 +0000

> If the sandboxed build environment can inject vulnerabilities (malicious code) into the build result, then you need to release a new version.

No go. RubyGems has to protect itself against hostile projects. You cannot push this back onto those projects, because they are untrusted.

A backdoor in a popular Ruby gem

david.a.wheeler — Thu, 11 Apr 2019 14:03:54 +0000

So basically the only way to patch anything is to create a new project, with a new name, at a new URL, ...

[citation needed]. I think you're misunderstanding things.

A backdoor in a popular Ruby gem

david.a.wheeler — Thu, 11 Apr 2019 13:59:21 +0000

> However, applying reproducible builds to a free-for-all package repository like RubyGems, in practice, seems like it would be challenging. At a minimum, you would have to impose these restrictions on projects using reproducible builds ("reproducible projects"):

> 1. RubyGems has to integrate with GitHub, and at least a half dozen smaller services-that-are-not-GitHub, or else force all reproducible projects onto GitHub. This may sound trivial, but it's a deal-breaker for the likes of the FSF.

It'd need to support multiple services, but that's easy. The "integration" for a distribution service is really just downloading from the repo (e.g., "git clone") when it receives a new module to distribute, rerunning the build instructions, and confirming that the results are identical. You don't need to support hooks or anything.

> A standardized build process that every reproducible project uses, with no exceptions allowed, and probably with some restrictions on extensions and modifications to the build process.

No, you just need a standardized way to invoke whatever build process is provided by the project. If the build process isn't part of the project, then you need to fix that anyway.

> A standardized set of dependencies that reproducible projects' build processes are allowed to use, with no exceptions allowed.

Not at all. The project just needs to specify its dependencies, and then the build process uses those. The goal is to be able to reproduce a project's build result, not force all projects to use the same thing.

> A standardized sandboxed environment in which the builds may be run without endangering the service as a whole.

You really only need to allow projects to *specify* the sandboxed environment (and check in that specification into the project). Different projects don't need to use the same one. There's no reason different projects need to use the exact same environment, and different versions of a project's results could use different environments too. It's just that when you're reproducing a *particular* version you need to know which environment.

> That sandbox is kept up-to-date (at least in terms of security vulnerabilities), but we must not alter the output of any builds. So the build's interaction with the sandbox, and with other software running in the sandbox if any, must be very carefully circumscribed to prevent it from behaving differently in response to updates.

Um, no. The sandbox is what it is. It might be okay to make an exception for the OS kernel (Linux), but otherwise, any "keep up to date" may change the result, so you update the sandbox as part of updating the build process. If the sandboxed build environment can inject vulnerabilities (malicious code) into the build result, then you need to release a new version.

Make it easy, not hard, and make it easy to do in stages. Otherwise no one will do it.

A backdoor in a popular Ruby gem

robbe — Thu, 11 Apr 2019 13:31:03 +0000

I fail to grasp what reproducible builds will prevent you from doing.

If I use dependency X in my project (be it libfoo, or gembar, or whatever), any deployment needs to get X from somewhere: either grab the source and build it yourself, or get a pre-built package (say from rubygems.org).

If X has bugs that I want fixed, but X’s upstream is slow to incorporate these (or totally MIA), I can fork off X′. But I can neither force it into the source repository of X nor replace X on rubygems.org with my X′. I can of course upload X′ to rubygems as a different package.

Reproducibe builds changes nothing on that front.

A backdoor in a popular Ruby gem

nim-nim — Thu, 11 Apr 2019 09:03:56 +0000

You've visibly not encountered the latest advancements in reproducibility (lucky you).

The latest iterations of the concept refuse, at the compiler level, to build anything that uses non reproducible third-party sources. Third-party is any code you reuse in your project, it can even be your own bleeding edge code, if it's published as another component, is not totally solid yet, and needs some small short-term adjustments to be reused.

They define reproducible as "source code hash, patches included, match the hash computed by a notary robot that attempted to download the code at some unknown point of time".

So basically the only way to patch anything is to create a new project, with a new name, at a new URL, upload all your changes here, so the notary robot can vouch you are indeed using the patched code published there. And the notary will happily vouch any code hash is good, as long as it matches its download (so if it downloads at a time the code repository was holed, it will vouch all the holed versions are good).

Of course once you've gone to the effort to produce a full-fledged fork, you have zero interest left in merging back anything.

A backdoor in a popular Ruby gem

Cyberax — Thu, 11 Apr 2019 08:26:50 +0000

> You end up with ”reproducible” systems that forbid you to apply any quick fix to third party code while upstream is busy somewhere else (because that would break the notary hash), forcing you to fork and rename everything.
Uhh... What? You apply fixes just like everywhere else, by making a patch to a fixed version of the package.

A backdoor in a popular Ruby gem

nim-nim — Thu, 11 Apr 2019 08:24:29 +0000

All the hype about reproducible builds is perfectly pointless without an upgrade and merge-back strategy.

You end up with ”reproducible” systems that forbid you to apply any quick fix to third party code while upstream is busy somewhere else (because that would break the notary hash), forcing you to fork and rename everything.

Fast forward a few years no one is using the same fork, not two forks have the same fixes, the only thing the reproducible setup can vouch for is that each build is a one of a kind, that does not share any component with anything else. But, each component has not been modified since its download from fork source (hurrah! the internet notary vouches your backdoored download matches the original badware commit). CVEs to original third party projects may apply (or not), who knows, it's a perfect mess.

Code is not something that can be frozen, it needs to evolve (because no code is bug-free and the code environment and security state of the art *will* evolve), none of the reproducibility efforts I know of take this necessary evolution into account.

A backdoor in a popular Ruby gem

roc — Thu, 11 Apr 2019 07:18:10 +0000

Developer workstations are incredibly inefficient. For a big project like Firefox you need lots of cores to make the build latency tolerable, but then they're mostly idle.

A backdoor in a popular Ruby gem

nilsmeyer — Thu, 11 Apr 2019 06:36:41 +0000

Though it can be said the compute capacity already exists and is paid for on the developers workstation while the funds to pay for it again on the other end have to be organized.