ABI stability funding [LWN.net]

ABI stability funding

Posted Nov 25, 2025 20:37 UTC (Tue) by jhoblitt (subscriber, #77733)
In reply to: ABI stability funding by bluca
Parent article: APT Rust requirement raises questions

If this was true, kubernetes would have been dead upon arrival. Even an extreme case of needing a terabyte of OCI layers is an incremental cost per server on the order of $100, which is probably less than 1% of the acquisition cost of a new 1U server.

At my $day_job, I have increased the k8s per node pod limit on most clusters up to 250 from the default of 110 as nodes were routinely hitting the pod limit but could easily handle more load.

to post comments

ABI stability funding

Posted Nov 25, 2025 20:49 UTC (Tue) by bluca (subscriber, #118303) [Link] (10 responses)

So with a million servers one has to spend an extra 100 millions for no particular reason other than because kubernetes is crap? Sounds about right

ABI stability funding

Posted Nov 25, 2025 21:22 UTC (Tue) by khim (subscriber, #9252) [Link] (9 responses)

If you have million servers then you would spend way more than $100 million on stuff not even remotely related to what you would pay for these servers. Wouldn't be surprised to find out that just building permits would cost more.

Heck, with million servers one, single, outage caused by problems with shared library compatibility may cost you more than $100 million!

An attempt to inflate price of something by shouting “but what if there are thousand, ten thousand, millions servers” would never work because not just expenses grow linearly but also cost of potential problems also grow linearly. Time when dynamic linking was feasible is long in the past for this very reason: in a world where human labor is cheap and hardware is expensive saving of one byte made sense. In today's world… not so much.

You may win some, in rare cases, if you have some component that's shared between hundreds and thousand of different programs (maybe some core OS library) but everything above that is cheaper not to share.

ABI stability funding

Posted Nov 25, 2025 21:48 UTC (Tue) by bluca (subscriber, #118303) [Link] (8 responses)

...and yet, tons of money is being spent developing various solutions to this very problem. How curious!

ABI stability funding

Posted Nov 25, 2025 21:52 UTC (Tue) by khim (subscriber, #9252) [Link] (7 responses)

Nothing curious, really. It's just matter of priorities: upgrade of one shared library on a server farm with million servers that would bring down your whole datacenter may cost you a lot more than $100 million thus you install Kubernetes and don't do that, but, of course, $100 million are still $100 — if you may, somehow, save them without exposing yourself to instability caused by distros quicksand then you will do that.

It's matter of priorities.

ABI stability funding

Posted Nov 26, 2025 1:06 UTC (Wed) by bluca (subscriber, #118303) [Link] (6 responses)

Yeah because famously kubernetes runs on thin air, it most definitely doesn't run on "distros quicksand". Also it never needs to be updated, and never, ever breaks

ABI stability funding

Posted Nov 26, 2025 2:53 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> Yeah because famously kubernetes runs on thin air

That's actually close to truth. There was even a project to run K8s as PID1. Most installations don't go _that_ far, and just limit themselves with something minimalistic like Alpine.

ABI stability funding

Posted Nov 26, 2025 3:18 UTC (Wed) by jhoblitt (subscriber, #77733) [Link] (1 responses)

There are compelling reasons to let kubelet use systemd to manage slices, so Alpine is probably not a popular host OS. However, it is incredibly popular as an OCI base layer.

ABI stability funding

Posted Nov 26, 2025 5:29 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

I was thinking more about the host for the control plane (kube-apiserver and such). Worker nodes are more diverse (there are even Windows nodes).

My desktop Docker host uses a cut-down Debian version without systemd running (there are just 4 processes: /initd services, /usr/bin/containerd-shim-runc-v2, /usr/bin/containerd, /usr/bin/rosetta-mount).

ABI stability funding

Posted Nov 26, 2025 11:15 UTC (Wed) by bluca (subscriber, #118303) [Link] (2 responses)

> That's actually close to truth.

In your fantasy land perhaps - back down here in the real world, everyone and their dog deploys on Ubuntu as the host, with RHEL and its derivatives distant contenders

> something minimalistic like Alpine

...which is also famously not a "distribution" but consists entirely of aether, right

ABI stability funding

Posted Nov 27, 2025 17:12 UTC (Thu) by ssmith32 (subscriber, #72404) [Link] (1 responses)

Er, actually, if we're going to go down the "well, back in the real world" route:

1) In the real world, most folks are going to use whatever OS their cloud provider uses for their k8s solution. So for EKS, Amazon Linux or Bottlerocket or something. And if it's GCP, they're probably rebuilding the whole world for funsies even if they use Ubuntu, because Monorepos Are (not) Awesome.

2) In the real world, people don't choose one or the other. Both have trade-offs, and most end up using both, depending on the situation. A base image of Ubuntu, with the occasional application installed via flatpak, etc.. is often the best solution for home use. k8s when deploying a large number of services at scale for commercial use.

ABI stability funding

Posted Nov 27, 2025 18:50 UTC (Thu) by bluca (subscriber, #118303) [Link]

For managed solutions sure, but the topic at hand here was custom hosts that one chooses and sets up to run their containers or VMs or whatevers.