Shared libraries [LWN.net]

Shared libraries

Posted Nov 26, 2025 18:37 UTC (Wed) by Cyberax (✭ supporter ✭, #52523)
In reply to: Shared libraries by bluca
Parent article: APT Rust requirement raises questions

OK. I'll bite.

Can you explain exactly why even simple brain-dead binary diffs are crap? I did an experiment, changed an "if" condition in a large binary, and did a diff. This is probably what most security patches are.

I used simple `rsync --only-write-batch=diff file1 file2`, and for a 50Mb binary the diff was 1Mb. I have not tried bsdiff or xdelta.

to post comments

Shared libraries

Posted Nov 26, 2025 18:55 UTC (Wed) by bluca (subscriber, #118303) [Link] (13 responses)

Because operational reality down in the real world is very, very far away from experiments with spherical cows in a vacuum. It's the same with kernel live patching: great idea in theory, but in practice it's so damn expensive to make it work for real, on real systems, ran by real people, with real random variations, that depend on them for real production use cases, that in reality you need an entire paid team to carefully shepherd them in a production scenario. And it happens for live patching because it's worth real money, as the only alternative to apply kernel secury updates is rebooting and thus very long and measurable downtimes, and it's somewhat simpler because you have _one_ kernel on any given system to delta patch. So if you pay Canonical/RH/Oracle/SUSE you can get access to them, and then you can sort of manage it with enough engineering resources.

It's orders of magnitude worse for deltarpm and similar because instead of one kernel to build and manage patches for you have N packages, and every node will have a different and unique combination. Combinatorial explosion. Complex, costly to manage, and benefits on a well designed systems that ships critical components such as libc or libssl as shared libraries that can get easily updated are too small to notice.

Shared libraries

Posted Nov 26, 2025 19:20 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

I don't get it. We have a globe-spanning binary-diff-driven system right now. You're using it every day, in fact. It's called "git".

It works just fine, somehow. All you need is protocol-level diffing support that deltarpms just half-assed. Even with the most naïve implementation, the end result should be bit-for-bit identical to just transferring files.

You want more examples? OSM (OpenStreet Maps) distributes up-to-minute incremental changes as diff files, and their full database is around 100GB. Yet they manage.

You want even more examples? Dockerhub and Docker images are represented as a set of deltas on top of a base image.

The real problem is, in fact, the RPMs and DEBs themselves. It routinely takes _longer_ to unpack and install Fedora or Debian updates than it takes to download them. Heck, the `apt-get update` step in my Dockerfiles is usually one of the slowest!

Shared libraries

Posted Nov 26, 2025 20:49 UTC (Wed) by bluca (subscriber, #118303) [Link] (9 responses)

> I don't get it. We have a globe-spanning binary-diff-driven system right now. You're using it every day, in fact. It's called "git".

Here's git in the real world: https://xkcd.com/1597/

> You want more examples? OSM

I'm not familiar with it, I suspect the bits are not user-facing pick-and-choose, but entirely behind the scenes, with no user control of installations and updates. And it's pure data, not running code. That makes it way, way easier and more manageable.

> You want even more examples? Dockerhub and Docker images are represented as a set of deltas on top of a base image.

Docker is a dumpster fire

Shared libraries

Posted Nov 27, 2025 0:15 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (8 responses)

> Docker is a dumpster fire

Yeah. Because it highlights just how terrible APT or RPM are. I can update 20 of my self-hosted apps _faster_ than it takes to update the host OS (Fedora). And to remind you, Docker is very much a user-facing, "pick-and-choose" system that is based on delta updates. A total impossibility, in other words.

And it works because it was architected correctly. Not _perfectly_, but correctly.

Why have the RPM diffs and APT deltas failed? It's because each package is installed separately. Apt has to pretend that each package is stand-alone, so it can't install them in parallel. And when you have thousands of packages, adding just a tenth of a second to the sequential critical path becomes painful, for not much gain for most people.

A more modern update system would dispense with all the individual package nonsense. It would fetch the updates (in parallel if possible) and in parallel pre-stage the changed files, resolving the deltas as soon as it gets the data. Then it would do atomic replacement of changed files (or as close to atomic as filesystems make it possible) and run whatever scripts needed to complete the update.

Shared libraries

Posted Nov 27, 2025 0:37 UTC (Thu) by bluca (subscriber, #118303) [Link] (5 responses)

> Yeah. Because it highlights just how terrible APT or RPM are.

Nah, it's shite with pacman and everything else too

Shared libraries

Posted Nov 27, 2025 0:47 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

So, correct me if I'm wrong, everything that solves the users' problems with updates is bad? As long as it makes the updates fast, seamless, reliable, and atomic?

BTW, even the _actual_ user-facing versions of Fedora are switching away from RPMs. See: Bazzite.

Shared libraries

Posted Nov 27, 2025 1:06 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> So, correct me if I'm wrong, everything that solves the users' problems with updates is bad?

Ladders are really, really useful for climbing to the top of a one or two-floor building.

But for 3-4 floors, portable ladders are no longer an option. Beyond 5.. ladders of any sort are worthless and a completely different solution must be employed.

In case you can't follow this analogy, something that is optimal for one application, or even a couple of applications, rapidly runs into fundamental scaling problems when you try to scale it to dozens or hundreds of applications.

But application writers don't care about bigger picture problems; they only care about *their* application.

Shared libraries

Posted Nov 27, 2025 6:58 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> But application writers don't care about bigger picture problems; they only care about *their* application.

I agree with your analogy. Except that classical distros are these rickety wooden home-made ladders. And modern Docker or immutable distros are tower cranes.

Shared libraries

Posted Dec 2, 2025 21:06 UTC (Tue) by raven667 (subscriber, #5198) [Link] (1 responses)

I think it is a good fit for the OS distributor using tools like RPM to manage the internal complexity of their software, then roll that up into an image with deltas for distribution in the way that Bazzite and other rpm-ostree distros are built, same for building Flatpak and Docker runtimes and apps. You need a tool like RPM/deb to track how a complicated runtime gets built.

I've been working with package-based OS since Redhat 4.2 (Colgate IIRC) and can see the limitations in scaling up a full desktop workstation using rpm/dpkg and yum/apt, where update performance is slow (my Fedora MacPro I used to use would take *hours* to update between releases on spinning rust) and the cost of de-duplicating all application libraries is the local admin has to use the packaging tool to manage long dependency chains, and things get complicated if you want to install an app which the OS distributor hasn't packaged themselves.

For someone who just wants to _use_ the computer to do computer things, rather than making OS design and development their hobby, the image-based systems with container-based apps work a lot better and more reliably than the package-based model, at least in my experience. Heck, having a good way to rollback because you aren't trying to mutate the one-and-only live system comes in really handy and makes me sleep better at night that the computer isn't going to immolate itself if something goes wrong.

Shared libraries

Posted Dec 3, 2025 2:50 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Yup. As many people noted, Linux works well for local development. And image-based systems allow that local environment to be used more easily for deployment. Package-based distros are also great for low-level tinkering and experimentation.

Although classic distros might also eventually change a bit. I think something like nix or guix might end up a better solution.

Shared libraries

Posted Nov 27, 2025 23:08 UTC (Thu) by intgr (subscriber, #39733) [Link] (1 responses)

> Docker is a [...] system that is based on delta updates.
> it was architected correctly.

I agree that compared to classical package managers it's impressively fast.

But no, Docker's deltas aren't even optimal for updating your application from one version to the next. They are a delta between one layer and the next one, so only efficient when the last few layers have changed.

Fast updates are often a symptom of applications not updating their base image with security patches as often as they should.

Docker could be a lot faster even, if it computed deltas between versions, or had a protocol like rsync or casync that allows figuring out the delta on the fly.

Shared libraries

Posted Dec 1, 2025 17:21 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

There was an AllSystemsGo! talk years ago about an OCIv2 which did chunking for image delivery and storage, but I believe it is DOA.

Simple in theory, but .deb format makes it a little complicated.

Posted Dec 5, 2025 11:02 UTC (Fri) by gmatht (subscriber, #58961) [Link]

I did a bit of work on this back in the day.

In principle this is really simple. Grab the .gz files out of the .deb using ar, zsync [1] them with the .gz files stored on some server (using either a cached .deb file or just $(cat `dpkg -L myoldpackage`). Then use ar to pack the .gz files back into a deb. Optionally also store some binary diffs from some particular versions the user might be likely to have lying round, as this is more efficient than zsyncing between arbitrary versions.

However, there are some issues with the .deb format. The compressed files are signed, so once you have updated your files you have to recompress them before you can feed them back into dpkg. Apart from slowing things down a bit this also means that you have to use tricks to get the resulting .gz to be bit for bit identical to the official .gz files. OK, it is possible create reproducable gzip files, but Debian seemed to be moving away from .gz compression.

If Debian decided to switch to a hsynz/zsync friendly format it should be easy. For example, if Debian switched to using uncompressed tars in their .deb files and instead distributed .deb.zstd files it should be way easier to support incremental updates than say, rewriting apt in rust.

[1] These days we would probably instead use https://github.com/sisong/hsynz

Delta updates work great with image-based OSs

Posted Nov 27, 2025 18:17 UTC (Thu) by DemiMarie (subscriber, #164188) [Link]

On image-based OSs, delta updates work great because there is only one old version.