Packaging Kubernetes for Debian

Posted Nov 5, 2020 16:20 UTC (Thu) by Cyberax (✭ supporter ✭, #52523)
In reply to: Packaging Kubernetes for Debian by smurf
Parent article: Packaging Kubernetes for Debian

> K8s's Go package dependency hell is just the most extreme example of a program written in a language whose library infrastructure decidedly is not up to the task of providing that kind of stability.
I violently disagree with that. I have NEVER had an issue with Go's libraries breaking the ABI, while I've had more than one issue fighting with subtly broken C-based ABI.

Go provides fully deterministic and replicable builds. If I check out a Go project then I'm guaranteed to get byte-for-byte identical build environment to that on the developer's machine. There are zero surprises.

It is IMPOSSIBLE to do that with classical Linux packaging methods. My environment will be subtly different to that of the package developer, unless we install everything at the same time.

So the proper question should be: "How Debian can achieve Go's stability?"

Packaging Kubernetes for Debian

Posted Nov 5, 2020 16:46 UTC (Thu) by pizza (subscriber, #46) [Link] (5 responses)

> I violently disagree with that. I have NEVER had an issue with Go's libraries breaking the ABI, while I've had more than one issue fighting with subtly broken C-based ABI.

So... if you take two different versions of a random Go library, you are saying that can never be API (and thus, ABI) differences between them?

> Go provides fully deterministic and replicable builds. If I check out a Go project then I'm guaranteed to get byte-for-byte identical build environment to that on the developer's machine. There are zero surprises.

And if you revision control a C-based project and all of its dependent libraries (as well as identical toolchains) you won't have any build-time surprises either.

(I mean, duh, if you change nothing, nothing will be changed)

Packaging Kubernetes for Debian

Posted Nov 5, 2020 16:57 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

> So... if you take two different versions of a random Go library, you are saying that can never be API (and thus, ABI) differences between them?
No. I'm saying that a project won't accidentally be broken when one library gets updated. Breakages happen only when a project explicitly updates its dependencies.

> And if you revision control a C-based project and all of its dependent libraries (as well as identical toolchains) you won't have any build-time surprises either.
Sure. Except Go doesn't require me to commit all libraries into the same project, it uses a coherent module system for that. The project code simply stores the SHA hashes of dependent libraries.

> (I mean, duh, if you change nothing, nothing will be changed)
Sure. But there's no way to do that with classic Linux distros. You will basically always get a subtly different build environment, as packages get updated without changing their name.

Packaging Kubernetes for Debian

Posted Nov 5, 2020 18:04 UTC (Thu) by pizza (subscriber, #46) [Link] (3 responses)

> No. I'm saying that a project won't accidentally be broken when one library gets updated. Breakages happen only when a project explicitly updates its dependencies.

Yep. But this has led to the current situation where the libraries themselves are anything but stable, so in practice updates rarely happen, bugs be damned.

> Sure. Except Go doesn't require me to commit all libraries into the same project, it uses a coherent module system for that. The project code simply stores the SHA hashes of dependent libraries.

... _and_ statically links everything into a standalone binary, so any library/dependency change necessitates a rebuild.

(And you know what? Apply the "rebuild all dependents" rule to "classic distributions" too, and your "subtle ABI differences" argument vanishes entirely)

> Sure. But there's no way to do that with classic Linux distros. You will basically always get a subtly different build environment, as packages get updated without changing their name.

If I install two systems off the same install media, and configure them identically, the results are going to be the same.

("but her updates!" you say? to which I point out that you're explicitly not updating anything in Go-land either; if it's good for Go, surely it's good for everything else too?)

Packaging Kubernetes for Debian

Posted Nov 5, 2020 20:27 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Yep. But this has led to the current situation where the libraries themselves are anything but stable, so in practice updates rarely happen, bugs be damned.
I'm developing mainly in Go for the last 3 years and honestly this hasn't happened once. In my experience bugs are fixed rapidly and have never resulted in API breaks.

If anything, bugs are fixed much more readily because authors are not afraid that they might accidentally break some old software that depended on some arcane accidental behavior.

> ... _and_ statically links everything into a standalone binary, so any library/dependency change necessitates a rebuild.
Not quite. You can have dynamic libraries, it's just that you're allowed to install multiple versions of them.

> (And you know what? Apply the "rebuild all dependents" rule to "classic distributions" too, and your "subtle ABI differences" argument vanishes entirely)
But classic distros don't allow to do that easily. I know, I tried. You have to do stuff like spinning up your own mirror with snapshots of package lists.

Non-classic distros like Nix (and I think guix) do allow it.

> If I install two systems off the same install media, and configure them identically, the results are going to be the same.
Now do that with Debian or Fedora net install. You will get a different image if the code is updated between installations. And there are no real ways to tell: "Debian Unstable as it looked at 2020-03-31 21:12:41 UTC".

Packaging Kubernetes for Debian

Posted Nov 5, 2020 22:52 UTC (Thu) by ballombe (subscriber, #9523) [Link] (1 responses)

> And there are no real ways to tell: "Debian Unstable as it looked at 2020-03-31 21:12:41 UTC".
Sure there is, see https://snapshot.debian.org

Packaging Kubernetes for Debian

Posted Nov 5, 2020 23:23 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Oh wow, this is actually super-helpful! I'm going to start using it, like right now.

Packaging Kubernetes for Debian

Posted Nov 5, 2020 18:44 UTC (Thu) by smurf (subscriber, #17840) [Link] (1 responses)

> I have NEVER had an issue with Go's libraries breaking the ABI

That is by definition impossible, and IMHO it's anything but an advantage.
One of the major points of assembling a distribution is to have exactly one version of any library on the system, so that if there is an issue with any one of these libraries you update this exact library and majickally fix the issue for every program using them. Instead of, say, rebuilding and re-deploying ten Go programs.

And no, stability is not impossible. The package developer doesn't build the binary. The distro's build system does that, in a controlled environment.

> There are zero surprises.

There also are zero security bugfixes to any of the libraries you depend on. Thanks but no thanks.

Packaging Kubernetes for Debian

Posted Nov 5, 2020 20:34 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> And no, stability is not impossible. The package developer doesn't build the binary. The distro's build system does that, in a controlled environment.
Distros do almost exactly zero QA, though. So if anything is broken, it'll be discovered by users. This results in very slow moving "stable" distros as a result.

> There also are zero security bugfixes to any of the libraries you depend on. Thanks but no thanks.
Well, Go is fairly safe in itself so security fixes are fairly rare. When they do happen, you'll have to update dependent applications, it's true.

And having better tooling to do that would be great. Github has a proprietary thingie (Dependabot) for that, and OpenSource solution to do the same would be awesome.