Shared libraries [LWN.net]

Shared libraries

Posted Nov 25, 2025 15:10 UTC (Tue) by paulj (subscriber, #341)
In reply to: Shared libraries by farnz
Parent article: APT Rust requirement raises questions

This one can easily be flipped the other way. Upstream statically links in libtiff with legacy, bug ridden stuff enabled. When said bugs become known, the distro updates the system shared library. All the dynamically linked apps are now secured. Except of course your statically linked upstream.

Which scenario is the more common? Which has the better track record at quickly updating to fix bugs? The random statically linked upstream-packaged apps or the Linux distros? I'd say the distros.

But let's say Linux distros are just average. Say we have 100 upstream-packaged statically-linked apps, and 100 apps using the distro shared library... ~50 of the upstream apps will update before the distro, and ~50 after - with a long tail. So - even if distros are not very good at shipping security updates, the statically linked approach will still leave you with a number of vulnerable apps for a long time to come.

to post comments

Shared libraries

Posted Nov 25, 2025 15:18 UTC (Tue) by farnz (subscriber, #17727) [Link] (14 responses)

Oh yes - both ways round are possible.

Note that the distro is quite capable of using the dependency information it already has (BuildRequires and the like) to rebuild statically linked binaries - dynamically linked versus statically linked is more about how much automated work has to be done to get you a fixed version in place, rather than about which is "more secure".

And I don't believe anyone has done the study to determine which is actually more secure in practice - static linked executables, with unused parts of libraries turned off, or dynamically linked executables sharing a library with more used components. Once you allow for things like time to determine that an update is needed, it's quite a complex space to think about, and (like so much in computing), we're more going on "what feels right" than on hard data.

Shared libraries

Posted Nov 25, 2025 17:42 UTC (Tue) by paulj (subscriber, #341) [Link] (1 responses)

My point was that, even IF distros were not great at updating the shared library, i.e. just average, it would still result in the system applications generally ALL being 'fixed' in an average and bounded amount of time - if they use shared libraries. Whereas, with apps statically linking, assuming a decent number of app, you will have a much longer and unbounded period of time where you have apps installed with a security problem.

I don't see any reason why app would or would not choose more locked-down build options than the distro. Assume that varies randomly across apps, and assume it varies randomly with libraries. The point still stands: The 'average' Linux distro gives you a bounded, shorter window of time where you have vulnerable apps from a bad library.

Shared libraries

Posted Nov 26, 2025 8:51 UTC (Wed) by taladar (subscriber, #68407) [Link]

This is not quite true. If the shared library solution works there is a strong chance that rebuilding the unmodified binary will work too so the distro could also do that, the builds might take a bit more CPU-time but overall this is definitely also bounded.

If, on the other hand, the binary needs to be modified to support the fixed library version there is a strong chance there is no ABI-compatibility for the shared library to take advantage of anyway.

Shared libraries

Posted Nov 25, 2025 17:56 UTC (Tue) by nim-nim (subscriber, #34454) [Link] (11 responses)

> Note that the distro is quite capable of using the dependency information it already has (BuildRequires and the like) to rebuild statically linked binaries

That’s a workaround not a feature, it transforms updates in mass rebuilds, that require a lot more build power, and add an economic barrier to entry to the distribution game (not that the big distributors will complain much about this part).

Nevertheless the real major drawback of static building is that it removes any developer incentive to converge on specific component versions. With dynamic building you have to choose one of the handful of versions packaged by your distributors. With static building there is no reason to make the effort (which is why developers are in deep love with static building).

The consequence of this lack of version convergence, is that static building is not only massively more wasteful on building power, it is massively more demanding in maintainer power. You need to tailor security patches (and security impact analysis) to each and every component version individual developers chose to vendor in their static build. In effect you promote technical debt (increase short term benefits at the expense of long term maintenance).

The problem does not exist FAANG side, because FAANGs force their dev teams to use the same golden versions. Guess how promoters of static building would welcome a distribution, that told them “you can use static building, but only with the following vendored versions, because we do not have the resources to patch others”.

Shared libraries

Posted Nov 25, 2025 18:31 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> That’s a workaround not a feature, it transforms updates in mass rebuilds, that require a lot more build power, and add an economic barrier to entry to the distribution game (not that the big distributors will complain much about this part).

Eh, no. Unless you're targeting something like m68k, rebuilding is cheap. We can use Gentoo as an example, rebuilding the world takes about 12 hours on a reasonably fast computer. And this is for a catastrophic bug, where everything needs to be updated.

In most cases, you're looking at maybe a handful of packages.

Shared libraries

Posted Nov 26, 2025 8:54 UTC (Wed) by taladar (subscriber, #68407) [Link] (4 responses)

On the other hand why is converging on a version that isn't the latest version a good thing? Shared library distro builds are essentially always stuck on the oldest version that some reverse dependency requires while static linking can just use the latest version, becoming part of the much, much larger group of people who aren't years behind.

This gets rid of all the effort required for distro specific bugs when developers don't want bug reports for ancient versions, all the backporting,...

Shared libraries

Posted Nov 28, 2025 9:37 UTC (Fri) by nim-nim (subscriber, #34454) [Link] (1 responses)

The latest version is not necessarily reliable, give the developers of other components a break, they also need to explore new approaches. Converging on a common version means a lot of people found this particular version reliable;

Static building is the opposite of using the latest version, people use the latest version *at the time they bother to check* and then lock it down and accumulate technical debt. People do not like making the effort to update, period, they can’t complain using dynamic libraries forces them to update on a regular cadence (that would expose that they do not want to make this effort) they complain that the update forced on them by distributions is not new and shiny enough.

Shared libraries

Posted Nov 29, 2025 16:20 UTC (Sat) by khim (subscriber, #9252) [Link]

> Converging on a common version means a lot of people found this particular version reliable;

That's only true for forges like PyPi or crates.io. Where people are free to pick any version they like.

With distros is the exact opposite: few packagers, often just one, single, person decides what version of library to package.

That's the opposite of the process that may lead to the outcome you describe.

> Static building is the opposite of using the latest version, people use the latest version *at the time they bother to check*

Still better than some random version that was picked by god-know-who for god-know-what reason and which wasn't even tested in conjunction with app.

> they complain that the update forced on them by distributions is not new and shiny enough.

People complain when something doesn't work, period. It may be too new or too old or anything in between.

But with distros they are often in position that they have no one to even complain to because person that assembled the crazy combination of libraries that breaks things is not even result of conscious decision, but more of result of random dice throwing.

According to the distro makers every package should be ready to work with every version of library… but that's rarely a thing that any sane developer would accept: to know whether something works or not you need to test… and distro makers make that exceedingly difficult.

Shared libraries

Posted Nov 28, 2025 11:25 UTC (Fri) by intelfx (subscriber, #130118) [Link] (1 responses)

> Shared library distro builds are essentially always stuck on the oldest version that some reverse dependency requires while static linking can just use the latest version, becoming part of the much, much larger group of people who aren't years behind.

Yeah, that's just the exact opposite of how it happens in reality.

In reality, applications in ecosystems with vendor-controlled static linking/bundling/vendoring get locked to the first version of each dependency that happens to work and never upgraded again (until a vulnerability is exploited in the wild, or until maintenance of a suitably obsolete build environment becomes untenable through actions of others).

Maintainers of distributions, on the other hand, care about health of the overall "ecosystem" Of course, the quality of said care may differ, but the overall concept is always there. You, as a user, may be briefly stuck on non-latest versions (or even on legacy branches) of some dependencies, but the entire distribution moves forward as fast as manpower allows.

Shared libraries

Posted Dec 1, 2025 9:59 UTC (Mon) by taladar (subscriber, #68407) [Link]

That is the way it works in C/C++ when they vendor dependencies because the tooling to upgrade sucks but e.g. on Rust, when upgrading dependencies is as simple as running cargo upgrade and cargo update and then fixing the minor compile issues 99% of the time (i.e. every time some dependency didn't completely revamp their API which is very rare) keeping up with dependency versions is incredibly easy.

Shared libraries

Posted Nov 26, 2025 9:22 UTC (Wed) by taladar (subscriber, #68407) [Link]

If rebuilding thousands of packages was such a big deal, why can the Rust project easily do https://rustc-dev-guide.rust-lang.org/tests/crater.html runs regularly to test if the compiler breaks any crate on crates.io?

ABI stability can block security updates

Posted Nov 27, 2025 18:14 UTC (Thu) by DemiMarie (subscriber, #164188) [Link] (3 responses)

Static linking allows security updates to libraries to be applied to applications that are ready for them, even when other applications are stuck on vulnerable versions. See gRPC in Fedora.

ABI stability can block security updates

Posted Dec 1, 2025 9:17 UTC (Mon) by nim-nim (subscriber, #34454) [Link] (2 responses)

Unfortunately baddies do not care if you have double plated the main door with the latest uber-expensive alloy, if the service door right next still uses rotting medium (a very common situation in proprietary setups).

What matters for security is that *every* deployed software bit uses fully patched components, even when those components are slightly old because no security event required their update and a full OS update is expensive effort-wise.

Static linking as you wrote promotes fixing highly visible main doors while keeping service doors wide open.

ABI stability can block security updates

Posted Dec 1, 2025 10:50 UTC (Mon) by taladar (subscriber, #68407) [Link]

You are completely ignoring that dynamic linking keeps the entire distro on an old version (or rather often a new, completely untested version somehow considered stable because it is called a "backported fix") because the many less popular programs that most systems don't even install have not been updated yet.

ABI stability can block security updates

Posted Dec 1, 2025 10:59 UTC (Mon) by muase (subscriber, #178466) [Link]

But that’s not true, as different components have different attack surfaces. gRPC is a good example, because it’s a huge difference if you have a public service that works with untrusted gRPC messages from the outside, or if your coordinator just uses a fixed set of gRPC messages to distribute your physics simulation across multiple worker processes via stdin/stdout on your own desktop.

In the first case, that’s a real-world RCE, in the second case you probably cannot even exploit the bug because you can’t control the input in the first place (messages are generated by the coordinator, not you).

Different use cases have different attack scenarios and different attack surfaces. There are tons of applications where a gRPC vulnerability will be a serious incident; but there are also a lot of cases where it – realistically speaking – it’s simply not exploitable.

Ideally, you should fix all bugs asap – but given the limited resources IRL, it sometimes is simply necessary to triage. That includes that if you cannot fix all packages at once, you can at least try to fix those where the bug has the most impact; i.e. users*impact (“reduce the bleeding”). It simply does not make sense to hold a fix back for the majority of users because you’re blocked by some obscure package nobody is using, or to wait for a package where the bug has no relevant real-world impact.

Shared libraries

Posted Nov 25, 2025 17:58 UTC (Tue) by JanC_ (guest, #34940) [Link] (19 responses)

There is also the issue of having to upgrade 100 packages instead of 1, which can make a big difference in time, network usage, required disk space, …

Shared libraries

Posted Nov 25, 2025 18:33 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (18 responses)

But why? I'd argue that this is more of a problem of existing packaging systems, they are inefficient by design.

There are no technical problems in storing updates as binary deltas. It's just that nobody cared to spin up infrastructure for this.

Shared libraries

Posted Nov 25, 2025 19:21 UTC (Tue) by bluca (subscriber, #118303) [Link] (17 responses)

That's because they are crap, and work only in theory. There's a reason stuff like deltarpm was abandoned. It just doesn't work in the real world, and it's effectively a workaround for a problem that doesn't need to exist in the first place.

Shared libraries

Posted Nov 26, 2025 2:34 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (16 responses)

deltarpm was just bad, just as debdiffs. Not in the least because DEB/RPM themselves are inefficient.

Format-aware binary diffs can be incredibly compact. RedBend Software used to provide target-aware OTA diffs, and they could compress a simple binary patch to just about the source code difference in bytes. Its differ used additional instructions to represent address shifts, and it could extract locations from the ELF relocation section.

And these days, not many people are going to care about update sizes anyway.

Shared libraries

Posted Nov 26, 2025 11:18 UTC (Wed) by bluca (subscriber, #118303) [Link] (15 responses)

So everything that actually exists was crap but it was just a coincidence, and the implementations that actually don't exist are the bee's knees - got it

Shared libraries

Posted Nov 26, 2025 18:37 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (14 responses)

OK. I'll bite.

Can you explain exactly why even simple brain-dead binary diffs are crap? I did an experiment, changed an "if" condition in a large binary, and did a diff. This is probably what most security patches are.

I used simple `rsync --only-write-batch=diff file1 file2`, and for a 50Mb binary the diff was 1Mb. I have not tried bsdiff or xdelta.

Shared libraries

Posted Nov 26, 2025 18:55 UTC (Wed) by bluca (subscriber, #118303) [Link] (13 responses)

Because operational reality down in the real world is very, very far away from experiments with spherical cows in a vacuum. It's the same with kernel live patching: great idea in theory, but in practice it's so damn expensive to make it work for real, on real systems, ran by real people, with real random variations, that depend on them for real production use cases, that in reality you need an entire paid team to carefully shepherd them in a production scenario. And it happens for live patching because it's worth real money, as the only alternative to apply kernel secury updates is rebooting and thus very long and measurable downtimes, and it's somewhat simpler because you have _one_ kernel on any given system to delta patch. So if you pay Canonical/RH/Oracle/SUSE you can get access to them, and then you can sort of manage it with enough engineering resources.

It's orders of magnitude worse for deltarpm and similar because instead of one kernel to build and manage patches for you have N packages, and every node will have a different and unique combination. Combinatorial explosion. Complex, costly to manage, and benefits on a well designed systems that ships critical components such as libc or libssl as shared libraries that can get easily updated are too small to notice.

Shared libraries

Posted Nov 26, 2025 19:20 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

I don't get it. We have a globe-spanning binary-diff-driven system right now. You're using it every day, in fact. It's called "git".

It works just fine, somehow. All you need is protocol-level diffing support that deltarpms just half-assed. Even with the most naïve implementation, the end result should be bit-for-bit identical to just transferring files.

You want more examples? OSM (OpenStreet Maps) distributes up-to-minute incremental changes as diff files, and their full database is around 100GB. Yet they manage.

You want even more examples? Dockerhub and Docker images are represented as a set of deltas on top of a base image.

The real problem is, in fact, the RPMs and DEBs themselves. It routinely takes _longer_ to unpack and install Fedora or Debian updates than it takes to download them. Heck, the `apt-get update` step in my Dockerfiles is usually one of the slowest!

Shared libraries

Posted Nov 26, 2025 20:49 UTC (Wed) by bluca (subscriber, #118303) [Link] (9 responses)

> I don't get it. We have a globe-spanning binary-diff-driven system right now. You're using it every day, in fact. It's called "git".

Here's git in the real world: https://xkcd.com/1597/

> You want more examples? OSM

I'm not familiar with it, I suspect the bits are not user-facing pick-and-choose, but entirely behind the scenes, with no user control of installations and updates. And it's pure data, not running code. That makes it way, way easier and more manageable.

> You want even more examples? Dockerhub and Docker images are represented as a set of deltas on top of a base image.

Docker is a dumpster fire

Shared libraries

Posted Nov 27, 2025 0:15 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (8 responses)

> Docker is a dumpster fire

Yeah. Because it highlights just how terrible APT or RPM are. I can update 20 of my self-hosted apps _faster_ than it takes to update the host OS (Fedora). And to remind you, Docker is very much a user-facing, "pick-and-choose" system that is based on delta updates. A total impossibility, in other words.

And it works because it was architected correctly. Not _perfectly_, but correctly.

Why have the RPM diffs and APT deltas failed? It's because each package is installed separately. Apt has to pretend that each package is stand-alone, so it can't install them in parallel. And when you have thousands of packages, adding just a tenth of a second to the sequential critical path becomes painful, for not much gain for most people.

A more modern update system would dispense with all the individual package nonsense. It would fetch the updates (in parallel if possible) and in parallel pre-stage the changed files, resolving the deltas as soon as it gets the data. Then it would do atomic replacement of changed files (or as close to atomic as filesystems make it possible) and run whatever scripts needed to complete the update.

Shared libraries

Posted Nov 27, 2025 0:37 UTC (Thu) by bluca (subscriber, #118303) [Link] (5 responses)

> Yeah. Because it highlights just how terrible APT or RPM are.

Nah, it's shite with pacman and everything else too

Shared libraries

Posted Nov 27, 2025 0:47 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

So, correct me if I'm wrong, everything that solves the users' problems with updates is bad? As long as it makes the updates fast, seamless, reliable, and atomic?

BTW, even the _actual_ user-facing versions of Fedora are switching away from RPMs. See: Bazzite.

Shared libraries

Posted Nov 27, 2025 1:06 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> So, correct me if I'm wrong, everything that solves the users' problems with updates is bad?

Ladders are really, really useful for climbing to the top of a one or two-floor building.

But for 3-4 floors, portable ladders are no longer an option. Beyond 5.. ladders of any sort are worthless and a completely different solution must be employed.

In case you can't follow this analogy, something that is optimal for one application, or even a couple of applications, rapidly runs into fundamental scaling problems when you try to scale it to dozens or hundreds of applications.

But application writers don't care about bigger picture problems; they only care about *their* application.

Shared libraries

Posted Nov 27, 2025 6:58 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> But application writers don't care about bigger picture problems; they only care about *their* application.

I agree with your analogy. Except that classical distros are these rickety wooden home-made ladders. And modern Docker or immutable distros are tower cranes.

Shared libraries

Posted Dec 2, 2025 21:06 UTC (Tue) by raven667 (guest, #5198) [Link] (1 responses)

I think it is a good fit for the OS distributor using tools like RPM to manage the internal complexity of their software, then roll that up into an image with deltas for distribution in the way that Bazzite and other rpm-ostree distros are built, same for building Flatpak and Docker runtimes and apps. You need a tool like RPM/deb to track how a complicated runtime gets built.

I've been working with package-based OS since Redhat 4.2 (Colgate IIRC) and can see the limitations in scaling up a full desktop workstation using rpm/dpkg and yum/apt, where update performance is slow (my Fedora MacPro I used to use would take *hours* to update between releases on spinning rust) and the cost of de-duplicating all application libraries is the local admin has to use the packaging tool to manage long dependency chains, and things get complicated if you want to install an app which the OS distributor hasn't packaged themselves.

For someone who just wants to _use_ the computer to do computer things, rather than making OS design and development their hobby, the image-based systems with container-based apps work a lot better and more reliably than the package-based model, at least in my experience. Heck, having a good way to rollback because you aren't trying to mutate the one-and-only live system comes in really handy and makes me sleep better at night that the computer isn't going to immolate itself if something goes wrong.

Shared libraries

Posted Dec 3, 2025 2:50 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Yup. As many people noted, Linux works well for local development. And image-based systems allow that local environment to be used more easily for deployment. Package-based distros are also great for low-level tinkering and experimentation.

Although classic distros might also eventually change a bit. I think something like nix or guix might end up a better solution.

Shared libraries

Posted Nov 27, 2025 23:08 UTC (Thu) by intgr (subscriber, #39733) [Link] (1 responses)

> Docker is a [...] system that is based on delta updates.
> it was architected correctly.

I agree that compared to classical package managers it's impressively fast.

But no, Docker's deltas aren't even optimal for updating your application from one version to the next. They are a delta between one layer and the next one, so only efficient when the last few layers have changed.

Fast updates are often a symptom of applications not updating their base image with security patches as often as they should.

Docker could be a lot faster even, if it computed deltas between versions, or had a protocol like rsync or casync that allows figuring out the delta on the fly.

Shared libraries

Posted Dec 1, 2025 17:21 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

There was an AllSystemsGo! talk years ago about an OCIv2 which did chunking for image delivery and storage, but I believe it is DOA.

Simple in theory, but .deb format makes it a little complicated.

Posted Dec 5, 2025 11:02 UTC (Fri) by gmatht (subscriber, #58961) [Link]

I did a bit of work on this back in the day.

In principle this is really simple. Grab the .gz files out of the .deb using ar, zsync [1] them with the .gz files stored on some server (using either a cached .deb file or just $(cat `dpkg -L myoldpackage`). Then use ar to pack the .gz files back into a deb. Optionally also store some binary diffs from some particular versions the user might be likely to have lying round, as this is more efficient than zsyncing between arbitrary versions.

However, there are some issues with the .deb format. The compressed files are signed, so once you have updated your files you have to recompress them before you can feed them back into dpkg. Apart from slowing things down a bit this also means that you have to use tricks to get the resulting .gz to be bit for bit identical to the official .gz files. OK, it is possible create reproducable gzip files, but Debian seemed to be moving away from .gz compression.

If Debian decided to switch to a hsynz/zsync friendly format it should be easy. For example, if Debian switched to using uncompressed tars in their .deb files and instead distributed .deb.zstd files it should be way easier to support incremental updates than say, rewriting apt in rust.

[1] These days we would probably instead use https://github.com/sisong/hsynz

Delta updates work great with image-based OSs

Posted Nov 27, 2025 18:17 UTC (Thu) by DemiMarie (subscriber, #164188) [Link]

On image-based OSs, delta updates work great because there is only one old version.