LWN: Comments on "Vendoring Go packages by default in Fedora"

Distros package managers don't meet upstream needs

Cyberax — Sat, 16 Aug 2025 02:41:00 +0000

> So are these "open source developers" going to pay the distributions for the stable, tested base platform their entire ecosystem depends upon?

That's an interesting question. My attitude for production deployment shifted entirely from "use as many distro packages as possible" to "get the distro away from anything important". These days, I'm using pretty much nothing out of distros outside of the bare minimum: libc, shells, standard utilities, ca-certificates, and a handful of standard libraries like zlib.

Distros package managers don't meet upstream needs

DemiMarie — Sat, 16 Aug 2025 01:55:08 +0000

Do those developers care if a distro uses bundled or unbundled dependencies, though?

Distros package managers don't meet upstream needs

pizza — Fri, 15 Aug 2025 21:53:36 +0000

> In the case of commercial distributions like SLES and RHEL, it really should be the distros paying upstreams

...You left out Ubuntu.

RHEL's users collectively make far more money using (and even developing for) RHEL than Red Hat themselves make.

Traditional distributions are "upstreams" of all of those application (and even language-specific ecosystem) developers, because that software is worthless without a stable foundation to run on.

Distros package managers don't meet upstream needs

DemiMarie — Fri, 15 Aug 2025 21:26:52 +0000

In the case of commercial distributions like SLES and RHEL, it really should be the distros paying upstreams. Upstream developers usually do not use these distros, but these distros get a lot of value from upstreams.

I expect a lot of upstream developers do not see downstream unbundling as providing much if any value, and see language-specific package management as providing a very large amount of value. Furthermore, inclusion in distributions is no longer the only way for software to be widely adopted on Linux. This means that upstream developers have much less incentive to make their software packaging-friendly. Asking upstream developers to put in substantial additional work for a use-case they are not particularly interested in is not likely to be successful unless there is a financial incentive.

Additionally, cross-platform projects must bundle dependencies for Windows, macOS, and mobile *anyway*. This means that bundled dependencies is the case that upstreams need to make sure works, and supporting unbundling is optional.

Upstream developers really have all the power here. I expect that in the vast majority of cases being discussed here, not packaging something hurts distros far more than it hurts upstream developers. Therefore, it is really the distros that need to be making things work for upstreams, not the other way around.

Distros package managers don't meet upstream needs

pizza — Fri, 15 Aug 2025 21:03:27 +0000

> Open source developers are generally not paid to meet the needs of distributions. If distributions want them to start caring more, they should start paying for that or provide tooling that is a better fit for the developers’ needs - not just the distributions’ - than what the developers currently use.

So are these "open source developers" going to pay the distributions for the stable, tested base platform their entire ecosystem depends upon?

If not, why do you think money deserves to flow in the other direction instead?

Distros package managers don't meet upstream needs

DemiMarie — Fri, 15 Aug 2025 20:31:07 +0000

Language package managers work on all major platforms and are fun and easy to use. They also avoid having to wait for distros to ship a new version of a library you need, and ensure those building your software don’t need to chase down libraries their distro doesn’t provide. Finally, they make it much easier to ensure that what the user uses is the same as what the developer tested, avoiding entire classes of otherwise-unreproducible bugs and making it much easier for developers to provide support. When language-specific package managers are not an option, containers provide the same benefits.

Open source developers are generally not paid to meet the needs of distributions. If distributions want them to start caring more, they should start paying for that or provide tooling that is a better fit for the developers’ needs - not just the distributions’ - than what the developers currently use.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

ssokolow — Tue, 11 Mar 2025 14:34:19 +0000

I know it's a month and a half late, but I think it's important to point to this post:

The impact of C++ templates on library ABI by Michał Górny

TL;DR: C++ suffers from exactly the same problems that Rust does... it's just more willing to emit an empty .so file when a library uses templates everywhere and doesn't use them as pervasively as Rust does.

Dynamic linking in the face of templates/generics without making the compromises Swift does is an unsolved problem.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

ssokolow — Tue, 11 Mar 2025 14:15:04 +0000

*nod* For example, I install Inkscape off Flathub instead of through the distro repos because it allows me to get the latest version when it works, on top of Kubuntu LTS, but trusting that I can reliably downgrade my way around the occasional crash bugs they ship with a single command.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

nye — Thu, 06 Mar 2025 17:21:44 +0000

I know it's been over a month but I just can't keep silent about this any longer.

Corbet, can we please just get a permaban for this malignant troll? LtWorf poisons almost every thread with his mindless toxic bile, and it's ruining LWN.

Native dependency cache

Cyberax — Tue, 11 Feb 2025 15:53:46 +0000

I have not checked recently, but when I accidentally tripped it in my project, it was detected within a day. I have an open source module that I'm using in my own code, and I put a tag incorrectly. Nobody else is using this module, so I just changed the tag. My builds started failing when I pushed a change on the same day.

Native dependency cache

mathstuf — Tue, 11 Feb 2025 15:47:44 +0000

When does the Go module proxy download the upstream and re-check that the source no longer matches? It sounds like the cache is heavily preferred to the point of just never checking again (otherwise the rewritten code would be detected as soon as the typosquat found a victim).

Native dependency cache

Cyberax — Mon, 10 Feb 2025 19:22:04 +0000

Ah. I see.

I was thinking of a case where you would push malicious code to a project's repo, tag it, and let users download it. Then you can force-push a "clean" version, erasing the trace. Go will detect this, once the next person tries to download the module, the Go module proxy will complain about different hashes for the same tag.

It won't help with pure typosquatting.

Native dependency cache

raven667 — Mon, 10 Feb 2025 03:55:44 +0000

In the case of a distro, or an in-house sysadmin packaging deps needed for in-house software, if the packager falls for the con and downloads the typosquatted release, "verifies" it and distributed in their repo then it'll be there as long as they are working from that srpm or source archive, the only thing preventing this is dilligence from the person who caches the artifacts to package, right. And if the distro, or inhouse packager, uses a local goproxy then it's incumbent upon them to vet everything that is requested to be pulled into it with the same dilligence. A typosquatted project on GitHub can look pretty convincing, esp if you are just grabbing a lib on the way to get something else working, and getting fooled for a moment can lead to long persistence until you need to revisit that library again and maybe notice the discrepancy.

I think a lot of people hold illusions that they wouldn't fall for this, that of course they'd know right away and have a good laugh, but i don't think this is true, a GitHub project with all the assets cloned from the original with just the URL changed is very difficult to spot if you have no prior familiarity and a shallow interaction with the project. The hall of mirrors can be pretty comprehensive and convincing, how could a packager know for sure in a way that would never fail or be missed, or skipped, that wouldn't be subject to human fallibility?

Native dependency cache

dskoll — Sat, 08 Feb 2025 22:21:50 +0000

What program written in go does a end user who isn't a developer need? Can you cite one such program?

Oh, sure. Here are two, in fact:

Mattermost, a chat system.
Galéne, a videoconference server.

I know you wrote "need" and could argue that you don't need the above programs, but they are very nice and very useful.

Native dependency cache

intelfx — Sat, 08 Feb 2025 21:05:26 +0000

Syncthing is a “random experiment on github”?

You are obviously arguing in bad faith.

Native dependency cache

LtWorf — Sat, 08 Feb 2025 20:32:46 +0000

I meant some project with users and so on, not some random experiments on github :)

Native dependency cache

excors — Sat, 08 Feb 2025 00:54:46 +0000

What is downloading the tag, and from where? It sounds like the mirror downloads it from GitHub once (before the backdoor was removed from the repository), caches it immutably forever, and most `go` users will subsequently download it from the mirror. Nobody will look at GitHub again, other than rare users who disable the default mirror (the attacker can just hope to avoid their notice), or naive security researchers who look for backdoors exclusively on GitHub and not in the mirror's verified copy of the code (which is what the article is about).

Native dependency cache

Cyberax — Fri, 07 Feb 2025 20:32:51 +0000

When you download a tag, Go complains if sumdb contains a different hash for the tag. It does not invalidate the previously stored artifact, though. I might be misremembering something, though.

Native dependency cache

TheBicPen — Fri, 07 Feb 2025 19:04:50 +0000

> What program written in go does a end user who isn't a developer need? Can you cite one such program?

A cursory look at GitHub shows plenty of examples. Syncthing, Alist, AdGuard Home, Photoprism, LocalAI, Croc, lux, filebrowser. I don't know why you think that go is only for writing developer tools.

Native dependency cache

excors — Fri, 07 Feb 2025 18:38:54 +0000

Where would that change be detected? As far as I can see, publishing the module (via the process in https://go.dev/doc/modules/publishing) will cause it to be downloaded from GitHub and cached forever by proxy.golang.org and added to the global checksum database (sum.golang.org). Anyone subsequently using the module with the default GOPROXY will receive the cached copy, and will successfully verify it against the global checksum database. Nobody will download it from GitHub again (unless they disable the default mirror), so nobody will notice the repository has been changed to hide the backdoor.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

TheBicPen — Fri, 07 Feb 2025 18:35:36 +0000

> flatpak and snap are there mostly for proprietary applications.

That's not true at all. I can't speak for snap, but on flathub, the vast majority of applications are FOSS.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

bluca — Fri, 07 Feb 2025 18:34:38 +0000

https://maskray.me/blog/2022-08-21-glibc-and-dt-gnu-hash

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

TheBicPen — Fri, 07 Feb 2025 18:17:54 +0000

> glibc last broke compat what, 20 years ago

August 2022. https://lwn.net/Articles/904892/

Native dependency cache

Cyberax — Fri, 07 Feb 2025 17:50:37 +0000

I looked at it again, and I don't see how it can work. Sumdb protects against this very attack, if a module referenced by a tag changes, it complains loudly and refuses to build the code.

The attack won't work unless you use the proxy service _and_ ignore the sumdb validation.

Dynamic linking isn't compatible with modern languages

daroc — Fri, 07 Feb 2025 14:18:56 +0000

I think there were a few comments that mentioned that, actually. Monomorphization specifically. But you're right that a lot of Rust's language features don't work well with dynamic linking.

Dynamic linking isn't compatible with modern languages

dragonn — Fri, 07 Feb 2025 14:16:56 +0000

Probably around 100 comments and no a single one comment about that how dynamic linking simply doesn't work with a lot of concepts used in modern languages (I am talking about the compiled ones, not interpreted). Stuff like generics, code monomorphization, procedural macros etc.
Rust didn't choose static linking "out of some hate for dynamic linking" but simple because that is the only option that works with features it has.
You could spend all the time in the world and money to try to package libs like serde for dynamic linking and it would simple not lead anywhere.

Native dependency cache

Cyberax — Thu, 06 Feb 2025 21:30:57 +0000

This happened with classic distros: https://bugs.gentoo.org/323691#c19 or https://blog.linuxmint.com/?p=2994

Native dependency cache

bluca — Thu, 06 Feb 2025 15:16:01 +0000

So, "just don't use the malware" then? Very useful advice.

How about "do not allow anyone to upload whatever they want with no gatekeeping whatsoever and requiring just an email address", instead? Like every Linux distro has been doing for 30 years? Just a thought!

Native dependency cache

excors — Thu, 06 Feb 2025 13:35:29 +0000

That's basically just typosquatting, with the malicious module named "github.com/boltdb-go/bolt" impersonating the popular "github.com/boltdb/bolt".

The novel part is that the attacker removed the malicious code from the corresponding GitHub repository, after publishing the module. Since Go modules are immutable (unlike Git repositories), the mirror kept serving the original code that was first published, and everything was successfully cryptographically verified against that.

The lesson is that anyone wanting to audit a module's code needs to download the crytographically-verified version and read that, instead of downloading the same tag name from the same repository URL and assuming it's still the same code. And of course you also need to be careful of typosquatting, which is a non-trivial problem, but that's nothing new or Go-specific.

Native dependency cache

bluca — Thu, 06 Feb 2025 12:54:17 +0000

> > Yes, that's it, you have now ingested, vendored and locked down some malware, as it came from a cesspool with no gatekeeping whatsoever. Success!

> Nope. It will get the locked versions of packages that are specified in the dependency list (go.mod and go.sum). They are cryptographically verified to be the same (go.sum contains the hashes of packages). If you think that the package author is malicious, then don't package the software in the first place.

Aged like a fine wine:

https://arstechnica.com/security/2025/02/backdoored-packa...

> A mirror proxy Google runs on behalf of developers of the Go programming language pushed a backdoored package for more than three years

lol, lmao

Questions

callegar — Thu, 06 Feb 2025 10:25:15 +0000

While a lot of the discussion appears to be focused on the advantages/disadvantages of vendoring or of the "go" model vs the traditional distribution model, and what is better, maybe there should be other factors to consider once a distribution decides to adopt a model different from its traditional one, related to the expectations of the distro users.
- should packages that use vendoring be immediately recognizable as such?
- should packages that bundle dependencies in, and thus somehow refuse the traditional distribution dependency management be allowed to have dependencies? Should only a very restricted set of dependencies be allowed? Should other packages be allowed to depend on them?
- should distros have the burden to update the bundled dependencies when there is an issue with them even if that breaks the alignment with the upstream developer?
Based on the answers to these questions, there could be a simplified/different packaging for the code that uses bundling and vendoring going side to side to the traditional distro one. Which is maybe, what is already done with flatpaks: users immediately recognize that it is a different type of packaging with a different type of dependency management. So maybe the point is just to extend flatpaks to make them more similar to regular packages wrt isolation or lack thereof and to fully support CLI.

Hmmm...

Baylink — Thu, 06 Feb 2025 04:11:58 +0000

Based on nothing other than reading this article (ok, and the first few comments, which seem compatible with my view) I gotta ask:

What has Go done wrong in its architecture design? 1/2 :-)

It's not uncommon for new entrants in a certain space not to realize what all the constraints are for being in that space, and that sounds like what's going (no pun intended) on here...

but the real pinch is if people are building distributions with *system programs* built in those languages which are making the distros hard to maintain.

This is vaguely similar to when SystemD (for no fathomable reason) killed off sysVinit, since packagers stopped making the necessary rc files to support it.

Native dependency cache

raven667 — Tue, 04 Feb 2025 04:39:33 +0000

As a tangent, I think these two technological approaches to code reuse can be combined as well. Going back to SuSE JeOS and OBS, the traditional packaging systems can help build containers and manage larger conglomeration of software bigger than one library or app, but they do need to grow to better bridge and encompass the hard work and knowledge built into the language-specific package managers like pip, npm, cpan, etc so that you can have one tool that is capable of showing the full dependency tree for a larger multi-language, multi-project system. cpanspec is pretty good, you can build anything from cpan with high fidelity of the dependency requirements known to cpan with only occasional fixes needed, this is not my experience with pip and setuptools as the bridge between ecosystems is rudimentary and the chance of irreconcilable dependency conflict is too high, which is why virtualenv contains dependency management for the single runtime much like how OCI containers can do for other apps.

For systems like how go and Rust manage code reuse the bridge tooling could be even more useful, providing a standard way to document build dependencies even when the result is a static binary without dynamic linking to inspect. Ask the package manager what software in the repo was built with git@git.example.org/libfoo.git and what commits were used, because the package format can be made to save that info reliably, so you can patch yourself or help the distro automatically do what needs to be done. OpenBuildService already does d
full CI and dep tracking across multiple languages for an entire distro of software using RPM spec as the CI language. You can do very slick things for one project on one language with the standard tools for that language, but the distro world has built great tooling for managing a whole runtime that could be made even better, even if the end result is building an OSTree or Flatpak image.

shared memory and loading savings

Cyberax — Mon, 03 Feb 2025 22:17:40 +0000

What are these wonderful unicorn-grade container managers?

shared memory and loading savings

bluca — Mon, 03 Feb 2025 22:14:47 +0000

It does, if you run good container managers. If you run crappy ones like docker, well, you get what you pay for.

shared memory and loading savings

shiz — Mon, 03 Feb 2025 20:35:19 +0000

> This is not really the case. Alpine Linux that prides itself on mostly doing static linking is snappier than regular Linuxes.

As a former Alpine maintainer: Alpine does not pride itself on doing static linking as it is dynamically linked like most other other distros.
In fact, we carry patches to enforce dynamic linking for some packages[1] that assume being musl-based means being statically linked.

> With the current state of hardware, the savings from shared text pages are generally insignificant,

Those savings can end up mattering more than you'd think, by their effect on the cache usage rather than simply the RAM footprint.
This was (back when I was active) one of the reasons we linked with -Os in Alpine, aside from fitting the goal for a small base system, for a surprising number of cases it actually lead to speedups over -O2.

>but the time to do linking adds up. This is not helped by the general design of the GNU linker, it's obsolete by 20 years. Newer linkers like `mold` help, but they still are not free.

Linking (as in the part that would be done by a program like mold) is done only once in the build process, so it shouldn't matter with regards to snappiness. The dynamic loader (/lib/ld-*.so), like the parent comment mentions, can actually benefit from things like lazy loading to improve runtime performance, however for security reasons that is disabled in most distributions (-z relro -z now) and I don't believe the musl dynamic loader supports it to begin with.

[1]: https://gitlab.alpinelinux.org/alpine/aports/-/blob/3.21-...

Native dependency cache

Cyberax — Mon, 03 Feb 2025 18:20:17 +0000

> What is a good way to package software these days? And when you wrote this what were your definitions of :
> obsolete

Not having the two properties below.

> atomicity
> reproducibility

The system updates are atomic, they either happen or not, and they don't leave the system in a broken state midway. Reproducibility means that I can recreate the state of the system now, at any point in the future or on another hardware.

The computing world is coalescing around small "immutable" distributions that run Docker/Podman/etc. It is even happening for desktop-based OSes.

The next level is reproducibility for containers. There isn't a good universal solution that works in a general case, mostly because of the "obsolete" status of classic distros. People work around that by using immutable tags in Docker for base images. Nixos is also getting popular.

Native dependency cache

Wol — Mon, 03 Feb 2025 13:14:40 +0000

> I am not going to argue against your view, I am just trying to get a grasp of what you meant.

_Thank_you_ for trying to "win" the discussion the right way.

In other words arrive at the CORRECT answer, not at the answer you want to hear.

Cheers,
Wol

Let it explode

amacater — Mon, 03 Feb 2025 12:38:25 +0000

Yes, that's (potentially) easier and for some languages that may be unavoidable. Disk space and computer power is cheap - for the cost of extra compilation and building, meh, who cares? It isn't necessarily a cure-all, however, and not everybody has large scale build farms or infinite disk space.

This also becomes much more difficult when there is a security problem / a severe bug and you need to know *exactly
how many library copies you have / where they all are. That's also the reason for some folk to dislike snaps, Flatpak and similar - it's software bloat to some extent and each flatpak is essentially an unknown black box.

For other languages or on the scale of a large project like Debian, the amount of extra archive space would become problematic. There's also no possibility of deduplication. That's one of the reasons why distributions try to de-vendor ... and that takes us round to a whole other part of this series of threads.

Native dependency cache

smoogen — Mon, 03 Feb 2025 11:36:20 +0000

@Cyberax

Not sure you are still following this, but I am interested in your line:

> RPMs and DEBs are an obsolete way of packaging software, that doesn't provide atomicity or reproducibility.

What is a good way to package software these days? And when you wrote this what were your definitions of :
obsolete
atomicity
reproducibility

so I can better understand where you are coming from? [I am not going to argue against your view, I am just trying to get a grasp of what you meant.]

Thank you