LWN: Comments on "Packaging Kubernetes for Debian"

Packaging Kubernetes for Debian

debacle — Sun, 10 Jan 2021 01:32:45 +0000

Is this https://bugs.debian.org/120645 opened 2001-11-22 and closed 2001-11-25?
Maybe you should reopen it, if setting of /etc/papersize to "a4" does not work on your system.
I just tried to print an A4 PDF into a PS file using xpdf 3.04-14 on Debian testing and the result is A4.
Works for me, but that's 19 years later ;-)

Packaging Kubernetes for Debian

jnxx — Sun, 06 Dec 2020 13:00:31 +0000

I think parts of the problems described here are even more acute with TensorFlow.

Strictly from the license, it is open source, as it uses the Apache License 2.0. But to build it from source, one has not only to use the bazel build system, but the bazel build scripts load down dependencies and code from over 200 internet locations which when building runs and downloads even more code from, even more locations.... it seems (to me) humanly impossible for a package maintainer to tell what all this code does, let alone to take any responsibility for this. So, it is "Open Source" but not actually in a way that provides real control to the user and owner of the computer, which is what free software is all about.

And this is merely an extreme of the problem that more and more tools and libraries suggest to install using something curl | bash. Even Rust does this (which is inconceivable to me because it tries to appeal to infrastructure people which usually do not fancy running untrusted code).

I think that Guix might be a partial solution to these problems. It seems to work very well with Rust. Perhaps it could help not to see it as simply some competition to Debian, but as a kind of complement, because I believe that many of the principal goals of both projects are very similar if not identical. Guix can build reproducible, fast-moving software from source with ease, and it can do that on top of a really solid and stable system like Debian.

Packaging Kubernetes for Debian

emorrp1 — Mon, 23 Nov 2020 17:48:57 +0000

> I wonder why there is ever any need to package binaries of any Go program.

So that the user doesn't have to know that application X is written in language Y before even being able to install it and know it'll work. Distros like Debian do have a lot of complexity to support language specific ecosystems, but that's all done on behalf of the end users to (hopefully) fit their expectations of how an OS behaves.

> to install any Go program the installer clones or pulls and compiles and links the program on the spot

Which is fine for developers, or for source based distros, but why should an end-user buy dev-spec hardware so they can install some software - one of the advantages of a binary package is the compilation can be offloaded to beefy servers.

Packaging Kubernetes for Debian

jak90 — Mon, 23 Nov 2020 13:50:25 +0000

The most comparable language in that regard is Ada (or Haskell in relation to GHC), since the most prominent and featured compiler GNAT has a pretty hard and long-lasting dependency on itself. There is no proprietary software at the end of the path, however.
Bootstrapping C is one of the core issues of the Bootstrappable project, but so far they have set up a pretty straightforward path of hex dumper -> machine monitor -> assembler -> Scheme interpreter -> naive C compiler -> simple C compiler (like TinyCC) -> GCC 4.7 -> modern compilers written in C++.
With Java, they started out with a Java 5 compiler written in C++ and gradually bootstrapped a JVM and classpath on top that understood enough Java 6 features to build OpenJDK 6 from it.
https://bootstrappable.org/projects/java.html
As far as I have read, there's still a multi-stage approach to bootstrap the Mono compiler from C, and Microsoft compiled their "native" C# compilers using their own C++ compiler until switching to Roslyn.
https://news.ycombinator.com/item?id=7528264

Packaging Kubernetes for Debian

nivedita76 — Fri, 13 Nov 2020 03:19:16 +0000

"Go is a programming language designed by Google to help solve Google's problems, and Google has big problems."

Packaging Kubernetes for Debian

anton — Thu, 12 Nov 2020 09:55:49 +0000

xpdf does not allow to copy text from PDFs that have been marked accordingly, and that was documented (I don't find that documentation at the moment, though). So it's a misfeature, not a bug; however, the documentation explained that it is easy to change xpdf to ignore the flag, so if that's what Debian's maintainers want to do, I don't think they need to ignore all sources of the desired paper size to do it.

Packaging Kubernetes for Debian

Wol — Thu, 12 Nov 2020 09:14:20 +0000

I believe the pdf spec allows you to put flags in saying "you can do this, you can't do that".

I guess Debian honour those flags to avoid potential legal liability. Sounds pretty typical for them.

Cheers,
Wol

Packaging Kubernetes for Debian

foom — Thu, 12 Nov 2020 00:18:51 +0000

That sounds like it may be a bug indeed; have you reported it?

On the other hand, the upstream version of xpdf inexplicably refuses to let you copy text from certain PDFs, and upstream refuses to fix this. The debian version has fixed that bug.

Packaging Kubernetes for Debian

anton — Tue, 10 Nov 2020 11:04:19 +0000

For some packages, the contribution by Debian is negative. E.g., for many years now, I have needed to build xpdf from source on every Debian/Ubuntu system because the Debian-crippled version only prints on letter paper (which we don't have around here); all configuration options for A4 paper (both the upstream xpdf ones as well as the Debian ones) are ignored.

Maybe the overall contribution by Debian is positive, but it seems to me that some Debian people are too convinced of their importance, and would do better by just packaging the upstream with as few changes as possible.

Packaging Kubernetes for Debian

amacater — Fri, 06 Nov 2020 10:44:48 +0000

They don't have to, at all - but the big industry players all rely on Linux, even if it's only internally - Google provide a Linux desktop based on Debian testing, for example, if that's what you want for their own staff. There's benefits to having a consistent infrastructure: various languages have done their own thing, an ecosystem or two are now impossible to maintain (gnuradio seems to be an environment which relies on magic, voodoo and blessed builds to keep going :) ). Working with, say, Debian, Ubuntu and CentOS would provide a core that's known.

The comment about security auditing somewhere above in this thread: it's not great, but it's easier in a distribution than a randomly structured ecosystem like NPM / PIP.
Looking from a distance at OpenStack / k8s - there's still a lot of magic, blessed github repositories or whatever around this if you don't go with a single vendor stack (Red Hat/Canonical are effectively vendoring their commercial offerings and depend on you having paid for support).

All of that may be completely immaterial if you're Alphabet/Facebook/AWS and can afford to throw human resources at the relevant language or system for internal use: external use is merely good publicity but you don't have to guarantee users there anything. And yes, I'm coming at this from 25 years of experience with Linux so I may be completely out of touch/too old to appreciate how the real world works.

Packaging Kubernetes for Debian

Cyberax — Fri, 06 Nov 2020 05:48:32 +0000

> I can't speak for the Go or rust ecosystem, but I still have difficulty understanding why something like k8s or terraform couldn't be re-implemented in a more traditional language which doesn't force an up-ending of ecosystems that are tried-and-tested.
Why should somebody bend over backwards to make stuff easier for Linux distros?

Packaging Kubernetes for Debian

jccleaver — Fri, 06 Nov 2020 05:46:47 +0000

> Note this is not an issue that only affects Go, most modern languages (as well as oldies like Perl and Python) have their own package managers and package ecosystems. When trying to package such programs into a distribution, distributions are kind of stuck. What we have historically done is (in an automated fashion) wrap every language package needed by a project into a corresponding distribution package and hope that there aren't too many to deal with.

I mean, isn't that the crux of the matter? Perl has had CPAN since the days of RH 5, and once cpan2rpm was reliable to use in a mostly-automated fashion it made keeping that in sync for trivial things... trivial. It's only the more involved things that require much manual tweaking, but that's pretty much how it should be since that's why you have humans in the loop to begin with.

Go decided it didn't care, and more to the point the people pushing it decided they didn't care, and now we're stuck.

I can't speak for the Go or rust ecosystem, but I still have difficulty understanding why something like k8s or terraform couldn't be re-implemented in a more traditional language which doesn't force an up-ending of ecosystems that are tried-and-tested.

Packaging Kubernetes for Debian

ringerc — Fri, 06 Nov 2020 02:30:22 +0000

Yes and no?

Increasingly a browser is a (terrible and inefficient) application runtime that maintains persistent state including connections to remote hosts.

You can't easily serialize the Javascript runtime state and the DOM to disk if you want to maintain remote connections. You'd need at least a way for part of the application to remain resident to respond to server side chatter / keepalives etc, or changes to redefine client/server expectations for idle and background tabs.

What I'd really like is a way to set resource limits on webapps. Impose some pressure on webapp developers, so they stop offloading gigabytes of epic JavaScript horror to onto all their users because there's no incentive for them not to.

Packaging Kubernetes for Debian

ringerc — Fri, 06 Nov 2020 01:10:53 +0000

Agreed. I work mostly with PostgreSQL - our palloc() is a heirachical memory allocator, not dissimilar to Samba's talloc. So yes I agree - you can just use your own allocator.

Packaging Kubernetes for Debian

Cyberax — Thu, 05 Nov 2020 23:23:41 +0000

Oh wow, this is actually super-helpful! I'm going to start using it, like right now.

Packaging Kubernetes for Debian

ballombe — Thu, 05 Nov 2020 22:52:10 +0000

> And there are no real ways to tell: "Debian Unstable as it looked at 2020-03-31 21:12:41 UTC".
Sure there is, see https://snapshot.debian.org

Packaging Kubernetes for Debian

Cyberax — Thu, 05 Nov 2020 20:34:49 +0000

> And no, stability is not impossible. The package developer doesn't build the binary. The distro's build system does that, in a controlled environment.
Distros do almost exactly zero QA, though. So if anything is broken, it'll be discovered by users. This results in very slow moving "stable" distros as a result.

> There also are zero security bugfixes to any of the libraries you depend on. Thanks but no thanks.
Well, Go is fairly safe in itself so security fixes are fairly rare. When they do happen, you'll have to update dependent applications, it's true.

And having better tooling to do that would be great. Github has a proprietary thingie (Dependabot) for that, and OpenSource solution to do the same would be awesome.

Packaging Kubernetes for Debian

Cyberax — Thu, 05 Nov 2020 20:27:13 +0000

> Yep. But this has led to the current situation where the libraries themselves are anything but stable, so in practice updates rarely happen, bugs be damned.
I'm developing mainly in Go for the last 3 years and honestly this hasn't happened once. In my experience bugs are fixed rapidly and have never resulted in API breaks.

If anything, bugs are fixed much more readily because authors are not afraid that they might accidentally break some old software that depended on some arcane accidental behavior.

> ... _and_ statically links everything into a standalone binary, so any library/dependency change necessitates a rebuild.
Not quite. You can have dynamic libraries, it's just that you're allowed to install multiple versions of them.

> (And you know what? Apply the "rebuild all dependents" rule to "classic distributions" too, and your "subtle ABI differences" argument vanishes entirely)
But classic distros don't allow to do that easily. I know, I tried. You have to do stuff like spinning up your own mirror with snapshots of package lists.

Non-classic distros like Nix (and I think guix) do allow it.

> If I install two systems off the same install media, and configure them identically, the results are going to be the same.
Now do that with Debian or Fedora net install. You will get a different image if the code is updated between installations. And there are no real ways to tell: "Debian Unstable as it looked at 2020-03-31 21:12:41 UTC".

Packaging Kubernetes for Debian

smurf — Thu, 05 Nov 2020 18:44:01 +0000

> I have NEVER had an issue with Go's libraries breaking the ABI

That is by definition impossible, and IMHO it's anything but an advantage.
One of the major points of assembling a distribution is to have exactly one version of any library on the system, so that if there is an issue with any one of these libraries you update this exact library and majickally fix the issue for every program using them. Instead of, say, rebuilding and re-deploying ten Go programs.

And no, stability is not impossible. The package developer doesn't build the binary. The distro's build system does that, in a controlled environment.

> There are zero surprises.

There also are zero security bugfixes to any of the libraries you depend on. Thanks but no thanks.

Packaging Kubernetes for Debian

pizza — Thu, 05 Nov 2020 18:04:46 +0000

> No. I'm saying that a project won't accidentally be broken when one library gets updated. Breakages happen only when a project explicitly updates its dependencies.

Yep. But this has led to the current situation where the libraries themselves are anything but stable, so in practice updates rarely happen, bugs be damned.

> Sure. Except Go doesn't require me to commit all libraries into the same project, it uses a coherent module system for that. The project code simply stores the SHA hashes of dependent libraries.

... _and_ statically links everything into a standalone binary, so any library/dependency change necessitates a rebuild.

(And you know what? Apply the "rebuild all dependents" rule to "classic distributions" too, and your "subtle ABI differences" argument vanishes entirely)

> Sure. But there's no way to do that with classic Linux distros. You will basically always get a subtly different build environment, as packages get updated without changing their name.

If I install two systems off the same install media, and configure them identically, the results are going to be the same.

("but her updates!" you say? to which I point out that you're explicitly not updating anything in Go-land either; if it's good for Go, surely it's good for everything else too?)

Packaging Kubernetes for Debian

mpr22 — Thu, 05 Nov 2020 18:03:25 +0000

> There is really no excuse for any tab you are not looking at to occupy any more memory than the pixels in the tab at the time you switched from it.

"Dump DOM state, necessitating a page reload and possible loss of work when switching back to the tab, if the user switches to a different tab for more than X minutes" is a perfectly reasonable behaviour to want, but it is not remotely a sane default behaviour.

Packaging Kubernetes for Debian

Cyberax — Thu, 05 Nov 2020 16:57:51 +0000

> So... if you take two different versions of a random Go library, you are saying that can never be API (and thus, ABI) differences between them?
No. I'm saying that a project won't accidentally be broken when one library gets updated. Breakages happen only when a project explicitly updates its dependencies.

> And if you revision control a C-based project and all of its dependent libraries (as well as identical toolchains) you won't have any build-time surprises either.
Sure. Except Go doesn't require me to commit all libraries into the same project, it uses a coherent module system for that. The project code simply stores the SHA hashes of dependent libraries.

> (I mean, duh, if you change nothing, nothing will be changed)
Sure. But there's no way to do that with classic Linux distros. You will basically always get a subtly different build environment, as packages get updated without changing their name.

Packaging Kubernetes for Debian

pizza — Thu, 05 Nov 2020 16:46:06 +0000

> I violently disagree with that. I have NEVER had an issue with Go's libraries breaking the ABI, while I've had more than one issue fighting with subtly broken C-based ABI.

So... if you take two different versions of a random Go library, you are saying that can never be API (and thus, ABI) differences between them?

> Go provides fully deterministic and replicable builds. If I check out a Go project then I'm guaranteed to get byte-for-byte identical build environment to that on the developer's machine. There are zero surprises.

And if you revision control a C-based project and all of its dependent libraries (as well as identical toolchains) you won't have any build-time surprises either.

(I mean, duh, if you change nothing, nothing will be changed)

Packaging Kubernetes for Debian

Cyberax — Thu, 05 Nov 2020 16:20:17 +0000

> K8s's Go package dependency hell is just the most extreme example of a program written in a language whose library infrastructure decidedly is not up to the task of providing that kind of stability.
I violently disagree with that. I have NEVER had an issue with Go's libraries breaking the ABI, while I've had more than one issue fighting with subtly broken C-based ABI.

Go provides fully deterministic and replicable builds. If I check out a Go project then I'm guaranteed to get byte-for-byte identical build environment to that on the developer's machine. There are zero surprises.

It is IMPOSSIBLE to do that with classical Linux packaging methods. My environment will be subtly different to that of the package developer, unless we install everything at the same time.

So the proper question should be: "How Debian can achieve Go's stability?"

Packaging Kubernetes for Debian

smurf — Thu, 05 Nov 2020 15:14:00 +0000

The problem isn't gnu/gcc, the problems are "can we all agree on a single version of a dependency that works for every program/deoending library in the distro that's written in X" and "is there a version of Y that's going to be supportable until the distro's next release".

These issues aren't going away just because you use Python or Perl or Go or JS instead of C. There have been a few big-ticket C libraries with dependency issues that the distros ran afoul of that took years to untangle; ffmpeg comes to mind.

K8s's Go package dependency hell is just the most extreme example of a program written in a language whose library infrastructure decidedly is not up to the task of providing that kind of stability. At the moment it's not packageable, period, no matter what the system which would like to package it is is or is not geared for. You can find examples of that in other languages, true, but Go and JS celebrate this kind of chaos and seem to regard it as a feature.

Maybe that'll change. Maybe not. In the meantime, Debian's best solution is to package an installer and then keep out of its way.

Packaging Kubernetes for Debian

flussence — Thu, 05 Nov 2020 10:33:54 +0000

The problem here, I feel, is that most distros were never intended to be general-purpose packagers, but GNU/GCC ecosystem packagers. The versioning scheme typically revolves around C ABI binaries and there's often metadata specifically for them, but the system shows its sharp edges immediately even for ancient things like Perl or Python. FWIW Gentoo recently also threw its hands up and allowed Go programs that vendor dozens of dependencies, and is in fact going through its own Kubernetes shake up.

In the long term I think the answer to this is going to require better acceptance that those other worlds exist, and ways to sandbox their packaging tools and extract build artifacts so they can be used “as intended” while preserving the OS's integrity.

Packaging Kubernetes for Debian

farnz — Thu, 05 Nov 2020 10:01:39 +0000

Yeah - we do rolling deploys of our server updates, so at any time, some servers are on today's code, and others are on yesterday's code. If today's code is bad (manually noticed, or detected by automation), it gets rolled back and ignored, ready for the new version that drops later.

This does mean that we have to be careful to make upgrades easy and roll-back safe within limits, but the frequency of updates means this isn't an issue because anyone who forgets this gets burnt regularly until they learn.

Packaging Kubernetes for Debian

ncm — Thu, 05 Nov 2020 09:00:00 +0000

I wonder why there is ever any need to package binaries of any Go program.

Suppose we just said that for any Go dependency, the package installer just does a "git clone" or "git pull" of upstream source. Then, to install any Go program the installer clones or pulls and compiles and links the program on the spot, taking the right revision from each dependency's local repository.

This is different from what we are used to for compiled programs, but a lot like what we already do for scripting languages. Go compilation is fast enough that there is not so much difference from a scripting language.

There are still a lot of dependency packages, but the amount of work to maintain them drops to near zero, as you only change them when their own dependency list changes.

This method would not work so well for Rust because the Rust compiler is so damn slow, but Rust people are not fighting with package management. Even for Rust, though, linking as part of installation could reduce problems.

Packaging Kubernetes for Debian

ncm — Thu, 05 Nov 2020 08:41:36 +0000

There is really no excuse for any tab you are not looking at to occupy any more memory than the pixels in the tab at the time you switched from it.

If the browser wants to cache more stuff from recently used tabs, that should be something tunable. But gigabytes of stuff I can't even see squatting on memory I need for things I am actually doing is an abomination.

Packaging Kubernetes for Debian

jwarnica — Thu, 05 Nov 2020 05:49:38 +0000

Or.... you do both. And you do live A/B testing.

Yeah, there is some absolutely pure sense of Big O good and bad.

But most software is not built to run on a system we are building down to a cost of melted sand, or to where it is unchanging in its exactness hoping for a human to override.

Packaging Kubernetes for Debian

NYKevin — Thu, 05 Nov 2020 01:33:57 +0000

There's another bonus, too: If the new release turns out to be bad (for any definition of "bad"), you can just abort it, roll back to last week's release, and re-release next week once you have a handle on what went wrong. That way, you don't have to do (much) troubleshooting in production in the first place (besides figuring out that the problem started around the same time as the release hit the affected systems, so it could plausibly have been the cause). While this is possible with big and heavy releases, it's significantly more painful.

This is less effective when you are rolling out to devices you don't control, because you'll have to tell the user "Hey, um, actually, we think you should go back to the old version for now" and users don't like that kind of churn. But it's great for servers, which is where k8s is supposed to run.

Packaging Kubernetes for Debian

Cyberax — Wed, 04 Nov 2020 18:34:39 +0000

I'm not sure Debian will actually scale for that. My fairly complex Go application has 40 dependencies, which I consider reasonable enough. I can see packaging it "classically" with fine-grained dependencies.

On the other hand, its ReactJS web-frontend that basically shows Google Maps and a couple of forms has 2120 dependencies. Simply building all of them as DEB files would take a LONG time, never mind doing the dependency resolution.

Packaging Kubernetes for Debian

jelmer — Wed, 04 Nov 2020 18:28:06 +0000

As a user, I like the ability to install and update my go binaries in the same way that I update the other binaries on my system.

The current overhead for creating a new package is significant if you have to manually package a lot of dependencies (such as for Kubernetes). Debian has certain assumptions about how much human-customized packaging overhead each package needs, and in the past that overhead was acceptable - shared libraries in e.g. the C world tend to be larger and less predictable.

For many of the newer ecosystems (go, rust, haskell, node) packages could be generated with a lot less overhead, because . The approach taken by the rust team with debcargo is promising - the overhead for adding an extra package is minimal, and makes it easy to add many more packages. See for example https://salsa.debian.org/rust-team/debcargo-conf/-/tree/m...

Packaging Kubernetes for Debian

khim — Wed, 04 Nov 2020 14:29:48 +0000

And how is scala different from Ada, C, C# or Java in this regard?

Packaging Kubernetes for Debian

khim — Wed, 04 Nov 2020 14:26:52 +0000

> Nobody's going to argue that malloc() can afford to throw around chunks of memory for developer convenience because RAM is cheap now.

Why would they argue if they can just use tcmalloc which does just that? It's used in web-browsers, too, BTW.

And the fact that web-browsers are using muti-process model for security which increases memory consumption about 2x-3x is just the fact of life now, too…

Packaging Kubernetes for Debian

gdt — Wed, 04 Nov 2020 10:53:09 +0000

I suspect the bare minimum is listing the names and versions of the vendored libraries in the .deb metadata. That gives the minimum function: a systems administrator can automate a check that the package isn't vulnerable to Known Bug X. For these recent languages than information is known when building the software, so it's not a huge hack to export that information to the package.

Packaging Kubernetes for Debian

pabs — Wed, 04 Nov 2020 03:56:40 +0000

That proposal is now implemented as fasttrack:

https://fasttrack.debian.net/

Packaging Kubernetes for Debian

smcv — Tue, 03 Nov 2020 21:03:09 +0000

volatile no longer exists, and backports explicitly doesn't allow packages which aren't in testing, which means it can't have packages that are unsuitable for the next stable release.

Various people have proposed adding a new suite/repository/archive area/thing that would accept packages that are intended to be used *with* the stable release but are not suitable to be shipped *in* the stable release (a bit like the apt repositories published by upstream projects like Docker and Salt, but general rather than package-specific, and within Debian); but that isn't something that a single package's maintainer can do unilaterally.

Packaging Kubernetes for Debian

farnz — Tue, 03 Nov 2020 10:41:58 +0000

Perhaps paradoxically, but we tell users asking for support to update much more rarely now that we're deploying daily than we did when we deployed monthly.

We've always used "are you up to date" as a way of dealing with issues that we are confident were fixed in a recent release; with daily deployment, our window for "recent release" has shrunk from 6 months to a week (something about the human brain seems to have us remembering what happened in the last 5 to 7 releases, and no more), which means that problems are much more likely to get investigated than they used to be, because there's no incorrectly remembered "recent" release with a fix for a similar problem.

Plus, because it's a rapid turnaround, we're happier to do a partial fix that solves the problem for you, and then use our time to fix up the resulting technical debt. In the past, because we had a long turnaround time, we'd prefer to avoid the technical debt at all - if it's going to take a month for the fix to get to you, we might as well aim for perfect first attempt.

Packaging Kubernetes for Debian

Cyberax — Tue, 03 Nov 2020 00:43:34 +0000

Ironically, the older Go code was even more autonomous when the current one (vendored packages used to be committed into the repo along with the code).

But even module-based Go doesn't need Github access, it needs a way to resolve packages and you can use one of the many repo managers for that (like Artifactory).