Native dependency cache

Posted Feb 6, 2025 12:54 UTC (Thu) by bluca (subscriber, #118303)
In reply to: Native dependency cache by Cyberax
Parent article: Vendoring Go packages by default in Fedora

> > Yes, that's it, you have now ingested, vendored and locked down some malware, as it came from a cesspool with no gatekeeping whatsoever. Success!

> Nope. It will get the locked versions of packages that are specified in the dependency list (go.mod and go.sum). They are cryptographically verified to be the same (go.sum contains the hashes of packages). If you think that the package author is malicious, then don't package the software in the first place.

Aged like a fine wine:

https://arstechnica.com/security/2025/02/backdoored-packa...

> A mirror proxy Google runs on behalf of developers of the Go programming language pushed a backdoored package for more than three years

lol, lmao

Native dependency cache

Posted Feb 6, 2025 13:35 UTC (Thu) by excors (subscriber, #95769) [Link] (1 responses)

That's basically just typosquatting, with the malicious module named "github.com/boltdb-go/bolt" impersonating the popular "github.com/boltdb/bolt".

The novel part is that the attacker removed the malicious code from the corresponding GitHub repository, after publishing the module. Since Go modules are immutable (unlike Git repositories), the mirror kept serving the original code that was first published, and everything was successfully cryptographically verified against that.

The lesson is that anyone wanting to audit a module's code needs to download the crytographically-verified version and read that, instead of downloading the same tag name from the same repository URL and assuming it's still the same code. And of course you also need to be careful of typosquatting, which is a non-trivial problem, but that's nothing new or Go-specific.

Native dependency cache

Posted Feb 6, 2025 15:16 UTC (Thu) by bluca (subscriber, #118303) [Link]

So, "just don't use the malware" then? Very useful advice.

How about "do not allow anyone to upload whatever they want with no gatekeeping whatsoever and requiring just an email address", instead? Like every Linux distro has been doing for 30 years? Just a thought!

Native dependency cache

Posted Feb 6, 2025 21:30 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

This happened with classic distros: https://bugs.gentoo.org/323691#c19 or https://blog.linuxmint.com/?p=2994

Native dependency cache

Posted Feb 7, 2025 17:50 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

I looked at it again, and I don't see how it can work. Sumdb protects against this very attack, if a module referenced by a tag changes, it complains loudly and refuses to build the code.

The attack won't work unless you use the proxy service _and_ ignore the sumdb validation.

Native dependency cache

Posted Feb 7, 2025 18:38 UTC (Fri) by excors (subscriber, #95769) [Link] (6 responses)

Where would that change be detected? As far as I can see, publishing the module (via the process in https://go.dev/doc/modules/publishing) will cause it to be downloaded from GitHub and cached forever by proxy.golang.org and added to the global checksum database (sum.golang.org). Anyone subsequently using the module with the default GOPROXY will receive the cached copy, and will successfully verify it against the global checksum database. Nobody will download it from GitHub again (unless they disable the default mirror), so nobody will notice the repository has been changed to hide the backdoor.

Native dependency cache

Posted Feb 7, 2025 20:32 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

When you download a tag, Go complains if sumdb contains a different hash for the tag. It does not invalidate the previously stored artifact, though. I might be misremembering something, though.

Native dependency cache

Posted Feb 8, 2025 0:54 UTC (Sat) by excors (subscriber, #95769) [Link] (4 responses)

What is downloading the tag, and from where? It sounds like the mirror downloads it from GitHub once (before the backdoor was removed from the repository), caches it immutably forever, and most `go` users will subsequently download it from the mirror. Nobody will look at GitHub again, other than rare users who disable the default mirror (the attacker can just hope to avoid their notice), or naive security researchers who look for backdoors exclusively on GitHub and not in the mirror's verified copy of the code (which is what the article is about).

Native dependency cache

Posted Feb 10, 2025 3:55 UTC (Mon) by raven667 (subscriber, #5198) [Link]

In the case of a distro, or an in-house sysadmin packaging deps needed for in-house software, if the packager falls for the con and downloads the typosquatted release, "verifies" it and distributed in their repo then it'll be there as long as they are working from that srpm or source archive, the only thing preventing this is dilligence from the person who caches the artifacts to package, right. And if the distro, or inhouse packager, uses a local goproxy then it's incumbent upon them to vet everything that is requested to be pulled into it with the same dilligence. A typosquatted project on GitHub can look pretty convincing, esp if you are just grabbing a lib on the way to get something else working, and getting fooled for a moment can lead to long persistence until you need to revisit that library again and maybe notice the discrepancy.

I think a lot of people hold illusions that they wouldn't fall for this, that of course they'd know right away and have a good laugh, but i don't think this is true, a GitHub project with all the assets cloned from the original with just the URL changed is very difficult to spot if you have no prior familiarity and a shallow interaction with the project. The hall of mirrors can be pretty comprehensive and convincing, how could a packager know for sure in a way that would never fail or be missed, or skipped, that wouldn't be subject to human fallibility?

Native dependency cache

Posted Feb 10, 2025 19:22 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Ah. I see.

I was thinking of a case where you would push malicious code to a project's repo, tag it, and let users download it. Then you can force-push a "clean" version, erasing the trace. Go will detect this, once the next person tries to download the module, the Go module proxy will complain about different hashes for the same tag.

It won't help with pure typosquatting.

Native dependency cache

Posted Feb 11, 2025 15:47 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (1 responses)

When does the Go module proxy download the upstream and re-check that the source no longer matches? It sounds like the cache is heavily preferred to the point of just never checking again (otherwise the rewritten code would be detected as soon as the typosquat found a victim).

Native dependency cache

Posted Feb 11, 2025 15:53 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

I have not checked recently, but when I accidentally tripped it in my project, it was detected within a day. I have an open source module that I'm using in my own code, and I put a tag incorrectly. Nobody else is using this module, so I just changed the tag. My builds started failing when I pushed a change on the same day.