Packaging Kubernetes for Debian

Posted Oct 30, 2020 23:20 UTC (Fri) by Paf (subscriber, #91811)
In reply to: Packaging Kubernetes for Debian by ballombe
Parent article: Packaging Kubernetes for Debian

Or, perhaps, rather than being bloated and insecure, they have found a way to persuade their users to accept rapid upgrade cycles, sparing the developer the significant ongoing costs of supporting old versions! This is a different optimization point but it seems like a potent one.

It might just be possible there is a reason that doesn’t involve the new software being crap. It might just be optimized differently,

Packaging Kubernetes for Debian

Posted Oct 31, 2020 10:51 UTC (Sat) by misc (subscriber, #73730) [Link] (7 responses)

Another interpretation is that by having the software move fast and be supported for a short period of time, this create a gap that vendors can fill with LTS supported versions, which in turn make the ecosystem sustainable. I am not saying this is made on purpose, nor even that's the only reason (likely far from it), but it seems to be a self reinforcing system.

Packaging Kubernetes for Debian

Posted Oct 31, 2020 20:31 UTC (Sat) by jwarnica (subscriber, #27492) [Link] (6 responses)

I'm not sure exactly what camp it comes from, but one of the theories of modern software development is "do the hard things always". Which is why there is CI/CD, for example.

If upgrades are small, safe, and frequent there is no reason not to do upgrades at that rapid pace. System administration is no longer
split between firefighting and planning and managing huge lift-and-shift operations, it day to day minor changes. Or should be.

"Vendor support" thus becomes "making *our* point releases seamlessly upgrade to *our* point releases in a specific path" not "supporting systems for ever". And k8s, even outside a commercial vendor, is very much focused on that theory of operation, allowing for zero downtime rolling updates across a cluster.

Actually, I'm not sure how k8s could manage a rolling upgrade if it was triggered from an OS level package manager.

Packaging Kubernetes for Debian

Posted Nov 1, 2020 15:43 UTC (Sun) by farnz (subscriber, #17727) [Link] (5 responses)

From direct personal experience (moving the system my team at work is responsible for from monthly to daily deployment), the advantage of rapid deployment is that there's simply less for the triage team to think about when a user complains about a change, because instead of having to be aware of ~1.5 months worth of changes, you have to be aware of around a week's worth of changes.

The rate of breakage is about the same as it was with monthly deployment, it's just that there's a small amount broken and in need of fixing each day, instead of a huge amount broken each month.

Packaging Kubernetes for Debian

Posted Nov 2, 2020 20:49 UTC (Mon) by tpo (subscriber, #25713) [Link] (1 responses)

> From direct personal experience (moving the system my team at work is responsible for from monthly to daily deployment), the advantage of rapid deployment is that there's simply less for the triage team to think about when a user complains about a change, because instead of having to be aware of ~1.5 months worth of changes, you have to be aware of around a week's worth of changes.

There is another advantage: when a user comes up with a bug or problem, the first reply (can be trivially automated == no human ressources necessary, yay!) can always be: do you have the latest version from very few days ago?

Now the user has to put in effort to update. Once he has managed to go through the effort, which usually carries a non-ignorable cost, the latest release has moved on. Repeat.

At some point the user gives up and accepts the bug or plasters some workaround over it ("reboot machine", "reinstall", "reinstantiate container" ...) and you can close the bug with "no feedback".

Of course I am being sarcastic here, but exactly this is happening in reality in many a project I've encountered.

A related trendy flavor of this "I the developper am the center of the universe" approach is "there was no activity on this ticket in x days, closing".

That perspective stinks. I'm no exception to it.

Packaging Kubernetes for Debian

Posted Nov 3, 2020 10:41 UTC (Tue) by farnz (subscriber, #17727) [Link]

Perhaps paradoxically, but we tell users asking for support to update much more rarely now that we're deploying daily than we did when we deployed monthly.

We've always used "are you up to date" as a way of dealing with issues that we are confident were fixed in a recent release; with daily deployment, our window for "recent release" has shrunk from 6 months to a week (something about the human brain seems to have us remembering what happened in the last 5 to 7 releases, and no more), which means that problems are much more likely to get investigated than they used to be, because there's no incorrectly remembered "recent" release with a fix for a similar problem.

Plus, because it's a rapid turnaround, we're happier to do a partial fix that solves the problem for you, and then use our time to fix up the resulting technical debt. In the past, because we had a long turnaround time, we'd prefer to avoid the technical debt at all - if it's going to take a month for the fix to get to you, we might as well aim for perfect first attempt.

Packaging Kubernetes for Debian

Posted Nov 5, 2020 1:33 UTC (Thu) by NYKevin (subscriber, #129325) [Link] (2 responses)

There's another bonus, too: If the new release turns out to be bad (for any definition of "bad"), you can just abort it, roll back to last week's release, and re-release next week once you have a handle on what went wrong. That way, you don't have to do (much) troubleshooting in production in the first place (besides figuring out that the problem started around the same time as the release hit the affected systems, so it could plausibly have been the cause). While this is possible with big and heavy releases, it's significantly more painful.

This is less effective when you are rolling out to devices you don't control, because you'll have to tell the user "Hey, um, actually, we think you should go back to the old version for now" and users don't like that kind of churn. But it's great for servers, which is where k8s is supposed to run.

Packaging Kubernetes for Debian

Posted Nov 5, 2020 5:49 UTC (Thu) by jwarnica (subscriber, #27492) [Link]

Or.... you do both. And you do live A/B testing.

Yeah, there is some absolutely pure sense of Big O good and bad.

But most software is not built to run on a system we are building down to a cost of melted sand, or to where it is unchanging in its exactness hoping for a human to override.

Packaging Kubernetes for Debian

Posted Nov 5, 2020 10:01 UTC (Thu) by farnz (subscriber, #17727) [Link]

Yeah - we do rolling deploys of our server updates, so at any time, some servers are on today's code, and others are on yesterday's code. If today's code is bad (manually noticed, or detected by automation), it gets rolled back and ignored, ready for the new version that drops later.

This does mean that we have to be careful to make upgrades easy and roll-back safe within limits, but the frequency of updates means this isn't an issue because anyone who forgets this gets burnt regularly until they learn.