Packaging Kubernetes for Debian

Posted Oct 31, 2020 10:23 UTC (Sat) by cyphar (subscriber, #110703)
In reply to: Packaging Kubernetes for Debian by geuder
Parent article: Packaging Kubernetes for Debian

In openSUSE with OBS we don't package Go programs this way, we just use what upstream vendored or if they use go.mod we have an OBS service which can fetch the modules before build (these downloaded modules are not packaged). In either case, while OBS in theory could make managing this really nice, we don't use it that way. There are a couple of reasons why, which all boil down to how libraries are practically used in Go:

Go libraries are commonly (though this is no longer as common as it used to be) imported such that they are pinned by commit ID. Therefore, different projects are unlikely to use the same version, meaning that you would need to package the library multiple times or otherwise maintain several versions. This is a problem you would immediately hit because of the fact that all of the golang.org/x/... libraries do not have version tags at all. (OBS can and does keep around old versions of packages for stable distributions but not for devel projects or Tumbleweed -- also you would need to be able to handle packaging a new project which uses a yet-unpackaged-but-older version of a commonly-used library.)
Because of widespread vendoring, Go libraries aren't quite as strict about API compatibility as C libraries. This means that you really can't swap out dependencies from under the project -- so the obvious solution to the previous point won't work. (Yes, Go.mod and semver do in theory eliminate this problem but in my experience this is still a very common issue.)
Libraries may have buildtags (used to determine whether to build a file or not and can be supplied at build time) which a downstream project sets based on some other configuration or feature detection. This means that for a single library you may have a power set of all possible subsets of buildtag combinations which could be set -- meaning you either have to build all of them or otherwise handle this situation.
Go (as a language) doesn't really support compiling libraries separately from the downstream project which depends on them. You can do it, but the Go folks say it's not a supported setup -- especially in the context of a distribution.

Note this is not an issue that only affects Go, most modern languages (as well as oldies like Perl and Python) have their own package managers and package ecosystems. When trying to package such programs into a distribution, distributions are kind of stuck. What we have historically done is (in an automated fashion) wrap every language package needed by a project into a corresponding distribution package and hope that there aren't too many to deal with. Go managed to break this model by not having packages or package registries at all and instead relying entirely on import paths. So we ended up just using vendoring (which is what upstreams also used). Debian bucked the trend and tried to "un-vendor" all Go projects, but the net result is a massive amount of extra work done by one distribution without any other distributions helping out.

Personally I think this situation would be much nicer if distributions and languages would have some way to merge their respective package ecosystems into a single package namespace, which would permit distributions to directly reference the native language packages -- making it easier for developers to have their software packaged and easier for distributions to keep track of security issues and maintenance. But naturally -- because we're talking about free software ecosystems -- this isn't going to happen because it appears that we all deep down thrive on having fragmented communities.

Packaging Kubernetes for Debian

Posted Oct 31, 2020 12:19 UTC (Sat) by geuder (subscriber, #62854) [Link] (3 responses)

> Therefore, different projects are unlikely to use the same version, meaning that you would need to package the library multiple times

I know, I had this in mind, but then my comment already got too long. So I skipped it. It would be another argument against ever growing bloat and workload. You need to keep several versions around and patch them in case of vulnerabilities.

It seems to be a development we should not participate in.

Packaging Kubernetes for Debian

Posted Oct 31, 2020 14:25 UTC (Sat) by cyphar (subscriber, #110703) [Link] (2 responses)

Choosing to not use such languages is one reaction, but as I said this is not a problem that only exists in Go. Rust and most other modern languages have many of the same caveats (and you could argue that even languages like Python and Perl also are similar in certain respects). And while you can go and write all of your own projects in C/C++, the wider software community is simply not going to do that -- very few developers deeply care about difficulty packaging their code in Linux distributions. Not to mention that if a useful project is written using one of these languages, distributions will need to package it somehow.

Packaging Kubernetes for Debian

Posted Oct 31, 2020 15:21 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (1 responses)

However, Rust crates that I've interacted with tend to be much more cognizant of API breakage than (it would seem) Go modules are. However, it sounds like go.mod might help this? Time will tell. I would like to see cargo support using a distro-local registry that can be used, but I don't know how much work that would entail.

Maybe someday we'll have the ability to use Rust APIs across a shared library boundary (sure, you can technically do it today, but limiting yourself to the C ABI for inter-library calls and compiler checks is quite restrictive in the Rust world).

Packaging Kubernetes for Debian

Posted Oct 31, 2020 15:45 UTC (Sat) by cyphar (subscriber, #110703) [Link]

> However, Rust crates that I've interacted with tend to be much more cognizant of API breakage than (it would seem) Go modules are. However, it sounds like go.mod might help this?

Yeah that is definitely true for Rust -- and I would argue this is because Rust grew the idea of strong versioning very early on (before Go modules you could argue that this wasn't a first-class feature in Go for most of it's lifetime). In theory Go modules might improve this situation, but to be honest I'm fairly skeptical that this will change enough Go projects that it will allow us to change how we would package the majority of them.

> Maybe someday we'll have the ability to use Rust APIs across a shared library boundary (sure, you can technically do it today, but limiting yourself to the C ABI for inter-library calls and compiler checks is quite restrictive in the Rust world).

This would make packaging more efficient in practice (only compile libraries once and reuse the compiled files as much as possible) -- and would be great to see -- but in principle the organisation of packages doesn't need to mean that library packages are actually distributed as compiled code. If you have a package for each (library) crate dependency which is used by a binary crate you can still do the actual compilation in the final binary. In this instance the issue is more about being able to represent crate dependencies using the distribution's method of representing such dependencies (BuildRequires in RPM).

Packaging Kubernetes for Debian

Posted Oct 31, 2020 14:42 UTC (Sat) by Conan_Kudo (subscriber, #103240) [Link] (1 responses)

Well, the reason openSUSE doesn't have Go dependencies de-vendored like both Debian and Fedora do is because openSUSE can't handle it. There are only 10 people who can review packages to merge into openSUSE Factory, and 7 of those 10 people have done zero reviews in the past three years. The effort to de-vendor Rust in openSUSE stalled on that fact alone, too. The openSUSE review process does not scale like the Fedora package review process does.

Fedora has been de-vendoring Go dependencies for years now, and we have a high level of reuse across applications using Go dependencies. We did not do it for Kubernetes (or Docker/Moby for that matter) because historically both projects forked and modified their dependencies when they vendored them. In this case, we declare them accordingly as bundled libraries, audit the licensing accordingly, and move on.

The transition to Go modules is actually expected to improve things even more for de-vendoring, since Go modules are required to establish an API contract with their versioning. This allows us to more freely upgrade dependencies with confidence. We have tooling to trigger rebuild chains as needed, even though I would like OBS-like auto-rebuild magic to make it even simpler.

Packaging Kubernetes for Debian

Posted Oct 31, 2020 15:48 UTC (Sat) by cyphar (subscriber, #110703) [Link]

Yeah I don't disagree about the review resources for openSUSE, it is a bit of a shame. Though I'd still argue that having distribution maintainers have to do busywork to maintain packages which are just wrappers around language packages is inefficient (even though Fedora and Debian have shown they can handle the review load). Ideally distribution maintainers would only ever have to step in for critical security or other such fixes IMHO.

> The transition to Go modules is actually expected to improve things even more for de-vendoring, since Go modules are required to establish an API contract with their versioning.

That is the idea (and a laudable one), though to be honest I'm a little pessimistic that much will change for the vast majority of projects.

Packaging Kubernetes for Debian

Posted Nov 2, 2020 15:05 UTC (Mon) by LtWorf (subscriber, #124958) [Link] (4 responses)

Go was never designed with distributing in mind.

It is a language developed at google to run on their own internal systems. When they made it they just wanted a binary blob so that whatever distributions they might use internally that month, it would just work.

It was never the idea to make go programs for others.

Packaging Kubernetes for Debian

Posted Nov 2, 2020 16:58 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

This is pure nonsense. Go was designed to be a general-purpose language.

The static compilation has always been a welcome feature. I remember that I was introduced to Go by HPC (High Performance Computation) folks who really LOVED that they could compile a binary locally and run it on a cluster, without caring for mismatching glibc versions.

Packaging Kubernetes for Debian

Posted Nov 2, 2020 20:14 UTC (Mon) by smurf (subscriber, #17840) [Link] (1 responses)

> This is pure nonsense. Go was designed to be a general-purpose language.

"It's a general purpose language" and "it's not designed for distribution" is not a contradiction.

Go binaries can be distributed like hell. Go sources? not so much, esp. if by "distributed" one means "included in a distribution". You basically need network access, github (and whatever else the code in question and/or any of its its dependencies want) needs to be up and reachable, and so on.

If you plan to work on a Go program offline, you basically need to start the compiler once, before disconnecting from the net, so it can cache all that stuff.

And that's the polar opposite of what a distribution is trying to accomplish.

Packaging Kubernetes for Debian

Posted Nov 3, 2020 0:43 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

Ironically, the older Go code was even more autonomous when the current one (vendored packages used to be committed into the repo along with the code).

But even module-based Go doesn't need Github access, it needs a way to resolve packages and you can use one of the many repo managers for that (like Artifactory).

Packaging Kubernetes for Debian

Posted Nov 13, 2020 3:19 UTC (Fri) by nivedita76 (subscriber, #121790) [Link]

"Go is a programming language designed by Google to help solve Google's problems, and Google has big problems."

Packaging Kubernetes for Debian

Posted Nov 6, 2020 5:46 UTC (Fri) by jccleaver (guest, #127418) [Link] (2 responses)

> Note this is not an issue that only affects Go, most modern languages (as well as oldies like Perl and Python) have their own package managers and package ecosystems. When trying to package such programs into a distribution, distributions are kind of stuck. What we have historically done is (in an automated fashion) wrap every language package needed by a project into a corresponding distribution package and hope that there aren't too many to deal with.

I mean, isn't that the crux of the matter? Perl has had CPAN since the days of RH 5, and once cpan2rpm was reliable to use in a mostly-automated fashion it made keeping that in sync for trivial things... trivial. It's only the more involved things that require much manual tweaking, but that's pretty much how it should be since that's why you have humans in the loop to begin with.

Go decided it didn't care, and more to the point the people pushing it decided they didn't care, and now we're stuck.

I can't speak for the Go or rust ecosystem, but I still have difficulty understanding why something like k8s or terraform couldn't be re-implemented in a more traditional language which doesn't force an up-ending of ecosystems that are tried-and-tested.

Packaging Kubernetes for Debian

Posted Nov 6, 2020 5:48 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> I can't speak for the Go or rust ecosystem, but I still have difficulty understanding why something like k8s or terraform couldn't be re-implemented in a more traditional language which doesn't force an up-ending of ecosystems that are tried-and-tested.
Why should somebody bend over backwards to make stuff easier for Linux distros?

Packaging Kubernetes for Debian

Posted Nov 6, 2020 10:44 UTC (Fri) by amacater (subscriber, #790) [Link]

They don't have to, at all - but the big industry players all rely on Linux, even if it's only internally - Google provide a Linux desktop based on Debian testing, for example, if that's what you want for their own staff. There's benefits to having a consistent infrastructure: various languages have done their own thing, an ecosystem or two are now impossible to maintain (gnuradio seems to be an environment which relies on magic, voodoo and blessed builds to keep going :) ). Working with, say, Debian, Ubuntu and CentOS would provide a core that's known.

The comment about security auditing somewhere above in this thread: it's not great, but it's easier in a distribution than a randomly structured ecosystem like NPM / PIP.
Looking from a distance at OpenStack / k8s - there's still a lot of magic, blessed github repositories or whatever around this if you don't go with a single vendor stack (Red Hat/Canonical are effectively vendoring their commercial offerings and depend on you having paid for support).

All of that may be completely immaterial if you're Alphabet/Facebook/AWS and can afford to throw human resources at the relevant language or system for internal use: external use is merely good publicity but you don't have to guarantee users there anything. And yes, I'm coming at this from 25 years of experience with Linux so I may be completely out of touch/too old to appreciate how the real world works.