Shared libraries [LWN.net]

Shared libraries

Posted Nov 24, 2025 20:12 UTC (Mon) by ballombe (subscriber, #9523)
In reply to: Shared libraries by Wol
Parent article: APT Rust requirement raises questions

> Okay, you lose the ability to just drop in a new fixed library, that fixes all your apps in one hit, but how well does that really work in practice?

It work pretty well. For example each time a new CVE is fixed in libtiff, the libtiff library is upgraded and there is no need to rebuild every software that directly or indirectly process TIFF files.

Making very costly to apply a security fix does not increase security.

to post comments

Shared libraries

Posted Nov 25, 2025 8:54 UTC (Tue) by taladar (subscriber, #68407) [Link] (18 responses)

Oddly enough none of the distros that advocate for this approach officially cover the part of that process that needrestart does, i.e. actually tell the user what needs to be restarted to get the benefit of that update.

Shared libraries

Posted Nov 25, 2025 9:45 UTC (Tue) by leromarinvit (subscriber, #56850) [Link] (14 responses)

Is that so? My Debian and Ubuntu systems will often restart services after installing updates. Does this not cover shared library updates - i.e. is a service only restarted after it itself was updated? Genuinely curious, I've always just assumed dependencies would also trigger restarts (at least if they're shared libs).

If they don't, just setting APT to auto-install security updates, without somehow restarting individual services or the whole system afterwards, is clearly not enough to at least keep a system free of known (and fixed) vulnerabilities.

Shared libraries

Posted Nov 25, 2025 10:03 UTC (Tue) by taladar (subscriber, #68407) [Link]

There is that "Daemons using outdated libraries" thing that sometimes pops up but it only covers a small subset of the running binaries or even systemd services actually using updated files.

Shared libraries

Posted Nov 25, 2025 10:05 UTC (Tue) by epa (subscriber, #39769) [Link] (12 responses)

Yes, that's the shameful secret of security updates in shared libraries. We've long mocked the Microsoft approach of rebooting the device for every change, but it's hard to beat if you want to guarantee that the vulnerable code is no longer running anywhere on the system.

In principle a program could be re-linked against the new shared library code while it stays running, but that requires an even stronger ABI stability guarantee than most libraries provide.

Shared libraries

Posted Nov 25, 2025 11:26 UTC (Tue) by SLi (subscriber, #53131) [Link] (10 responses)

Hmm. Why is this a shared vs static library problem? Doesn't the application need to be restarted in either case to get the benefits?

Shared libraries

Posted Nov 25, 2025 14:02 UTC (Tue) by farnz (subscriber, #17727) [Link] (9 responses)

The claim being made for shared libraries is that I can just update the library, and all the applications are immediately patched, which reduces admin effort as compared to static linking, where I have to update the binaries and then restart the applications.

The point is that it's not that big a reduction in effort - and it's a reduction in effort in the automated part, to boot.

Shared libraries

Posted Nov 25, 2025 14:59 UTC (Tue) by intelfx (subscriber, #130118) [Link] (8 responses)

> The claim being made for shared libraries is that I can just update the library, and all the applications are immediately patched, which reduces admin effort as compared to static linking, where I have to update the binaries and then restart the applications.

Nobody is making the claim for shared libraries to somehow obviate the need to *restart the applications*. You invented this claim out of thin air.

Shared libraries obviate the need to *update the binaries*, no more, no less.

Shared libraries

Posted Nov 25, 2025 15:12 UTC (Tue) by farnz (subscriber, #17727) [Link] (7 responses)

They don't even do that - you have to update the binaries that are supplied by the shared library, and the in-memory copies of the binaries, too.

The only thing they do is mean that you don't have to replace the executables - but replacing binaries (libraries or executables) is the bit of the update process that's simple to automate.

Shared libraries

Posted Nov 25, 2025 18:23 UTC (Tue) by intelfx (subscriber, #130118) [Link] (6 responses)

Clearly, "updating the binaries" in this argument means updating the _dependent_ binaries (the executables). I intended to include this clarification but omitted it due to sheer obviousness.

> but replacing binaries (libraries or executables) is the bit of the update process that's simple to automate.

It's not so simple (or cheap) to automate _creating_ those binaries in the first place. This is the part shared libraries help with: you do not need to rebuild and redistribute O(n) dependents when all you need is to make a compatible change in O(1) libraries.

Shared libraries

Posted Nov 25, 2025 19:14 UTC (Tue) by NYKevin (subscriber, #129325) [Link] (5 responses)

> It's not so simple (or cheap) to automate _creating_ those binaries in the first place.

The argument made by the newer languages is that it should be. And they have largely succeeded. When is the last time you saw a Rust or Go project that *couldn't* be built with cargo build, go build, etc.?

Yes, yes, the subsequent packaging step is frequently more complicated. But that step is entirely the distro's invention. They don't get to make it everyone else's problem.

Shared libraries

Posted Nov 28, 2025 3:16 UTC (Fri) by wtarreau (subscriber, #51152) [Link] (4 responses)

This full rebuild of applications however is a real problem for end users because they cannot choose anymore to only update the part that causes trouble. When application X has dependency Y and a full rebuild of X+Y has to be done to fix Y, most of the time it also comes with fixes (or even minor updates) of X, that users have to accept just to fix Y. This is a real problem because regressions do happen (frequently) and the more dependencies are involved in a rebuild, the more likely unintended changes are visible to the end user, who don't always have the choice of running the software that works for them and only update problematic dependencies.

Shared libraries

Posted Nov 28, 2025 5:35 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

I keep reminding people here that Gentoo exists. And it's not a new fad, it's been around for more than two decades. And Nix is a newer example.

Shared libraries

Posted Nov 28, 2025 8:33 UTC (Fri) by taladar (subscriber, #68407) [Link] (2 responses)

You mean they don't get a choice in the way they don't get a choice to keep the old version of a shared library (or indeed choose an arbitrary version of a shared library) for each of the other applications using that same library on the system that might not be affected by a fix?

Besides, updating all the dependencies to the same combination of dependency versions used by everyone else using the application actually reduces the number of truly different combinations in use, making it more likely that someone else is running the same version and has run into a bug before you do.

Shared libraries

Posted Jan 14, 2026 4:48 UTC (Wed) by wtarreau (subscriber, #51152) [Link] (1 responses)

> You mean they don't get a choice in the way they don't get a choice to keep the old version of a shared library (or indeed choose an arbitrary version of a shared library) for each of the other applications using that same library on the system that might not be affected by a fix?

No I mean they don't have the choice to only updating the library without updating the application. The lifecycle of applications is not always the same as that of libraries. Take firefox as an example, you might experience issues caused by one of the thousand libs it depends on, but otherwise match your needs. By only updating that broken lib you get a perfectly functional browser. When using static builds, it's less likely that they'll rebuild your version based on an updated dependency so instead you're more likely to be forced to upgrade to the next version that probably doesn't please you at all anymore, or introduces new issues. The alternative is to not update the static application and that's an issue with security stuff (e.g. crypto libraries), which will overall degrade the security of what's effectively deployed in the field.

Shared libraries

Posted Jan 14, 2026 5:20 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

The corollary is that if you have a static version, it's usually easy to build a custom version, because the project will have tooling to easily manage the bundled libraries. Otherwise they'd go mad.

This absolutely applies to Rust and Go. I can make my own mini-fork within minutes for most of the projects, except for really hairy ones like Docker or K8s.

Shared libraries

Posted Dec 6, 2025 10:38 UTC (Sat) by jengelh (subscriber, #33263) [Link]

>We've long mocked the Microsoft approach of rebooting the device for every change, but it's hard to beat if you want to guarantee that the vulnerable code is no longer running anywhere on the system

The more evolved and hostile attached networks became between 1990 and now, the *less* Windows versions needed to reboot. So it does not seem like the mocking was ever done because of (in)security concerns.

Shared libraries

Posted Nov 25, 2025 16:02 UTC (Tue) by draco (subscriber, #1792) [Link] (2 responses)

Officially, Fedora only supports offline updates (reboot, apply updates in constrained environment, then boot with updated software). You can do it online, but then it's up to you to pick up the pieces.

You keep saying distros don't handle this case, but they do.

Extra services during offline upgrade

Posted Nov 27, 2025 18:13 UTC (Thu) by DemiMarie (subscriber, #164188) [Link] (1 responses)

Is there a way to run extra services during an offline upgrade? Qubes OS requires that a VM be running certain services at all times.

Extra services during offline upgrade

Posted Nov 28, 2025 12:10 UTC (Fri) by neverpanic (subscriber, #99747) [Link]

AFAIR, the "offline" upgrade really is just a special systemd target that gets activated when a specific symlink exists in the rootfs, and a successful update deletes that symlink.

So, yes, you can run extra services by making sure your systemd service file is configured to start in that target.

Shared libraries

Posted Nov 25, 2025 12:17 UTC (Tue) by NAR (subscriber, #1313) [Link] (54 responses)

Don't forget that it goes the other way too: introduce one bug into libtiff and now all libtiff-using programs on your system have the bug. It's not unheard that new versions of libraries have (new) bugs... I'd even be inclined to think it happens more often that a shared library introduces a bug than it fixes an actual, exploitable security vulnerability.

Shared libraries

Posted Nov 25, 2025 14:06 UTC (Tue) by farnz (subscriber, #17727) [Link] (53 responses)

And there's a particularly nasty subset of that, induced by the increased scope of feature unification.

Imagine a new version of libtiff which introduces a security-relevant bug into the decompressor for TIFF compression scheme 32809 (ThunderScan 4-bit RLE). Upstream's statically linked builds of the program are not vulnerable, because they don't enable the bits of libtiff needed to handle files from ancient Macs, but because your distro includes a utility that's supposed to analyse an ancient Mac disk image and convert all the data to modern formats that you can work with, your distro build of libtiff has this support enabled.

Hey presto, an application that was not vulnerable in the upstream configuration (and may not be vulnerable on other distros that don't support reading TIFF files from ancient Macs) is now vulnerable, because you're running a configuration of the code that's necessary for a different application.

Worst case, you've opened up a network-accessible vulnerability in an application that was unaware that you could build libtiff this way, in order to give more functionality to an application that's carefully sandboxed in case the files are corrupt and trigger a bug.

Shared libraries

Posted Nov 25, 2025 15:10 UTC (Tue) by paulj (subscriber, #341) [Link] (35 responses)

This one can easily be flipped the other way. Upstream statically links in libtiff with legacy, bug ridden stuff enabled. When said bugs become known, the distro updates the system shared library. All the dynamically linked apps are now secured. Except of course your statically linked upstream.

Which scenario is the more common? Which has the better track record at quickly updating to fix bugs? The random statically linked upstream-packaged apps or the Linux distros? I'd say the distros.

But let's say Linux distros are just average. Say we have 100 upstream-packaged statically-linked apps, and 100 apps using the distro shared library... ~50 of the upstream apps will update before the distro, and ~50 after - with a long tail. So - even if distros are not very good at shipping security updates, the statically linked approach will still leave you with a number of vulnerable apps for a long time to come.

Shared libraries

Posted Nov 25, 2025 15:18 UTC (Tue) by farnz (subscriber, #17727) [Link] (14 responses)

Oh yes - both ways round are possible.

Note that the distro is quite capable of using the dependency information it already has (BuildRequires and the like) to rebuild statically linked binaries - dynamically linked versus statically linked is more about how much automated work has to be done to get you a fixed version in place, rather than about which is "more secure".

And I don't believe anyone has done the study to determine which is actually more secure in practice - static linked executables, with unused parts of libraries turned off, or dynamically linked executables sharing a library with more used components. Once you allow for things like time to determine that an update is needed, it's quite a complex space to think about, and (like so much in computing), we're more going on "what feels right" than on hard data.

Shared libraries

Posted Nov 25, 2025 17:42 UTC (Tue) by paulj (subscriber, #341) [Link] (1 responses)

My point was that, even IF distros were not great at updating the shared library, i.e. just average, it would still result in the system applications generally ALL being 'fixed' in an average and bounded amount of time - if they use shared libraries. Whereas, with apps statically linking, assuming a decent number of app, you will have a much longer and unbounded period of time where you have apps installed with a security problem.

I don't see any reason why app would or would not choose more locked-down build options than the distro. Assume that varies randomly across apps, and assume it varies randomly with libraries. The point still stands: The 'average' Linux distro gives you a bounded, shorter window of time where you have vulnerable apps from a bad library.

Shared libraries

Posted Nov 26, 2025 8:51 UTC (Wed) by taladar (subscriber, #68407) [Link]

This is not quite true. If the shared library solution works there is a strong chance that rebuilding the unmodified binary will work too so the distro could also do that, the builds might take a bit more CPU-time but overall this is definitely also bounded.

If, on the other hand, the binary needs to be modified to support the fixed library version there is a strong chance there is no ABI-compatibility for the shared library to take advantage of anyway.

Shared libraries

Posted Nov 25, 2025 17:56 UTC (Tue) by nim-nim (subscriber, #34454) [Link] (11 responses)

> Note that the distro is quite capable of using the dependency information it already has (BuildRequires and the like) to rebuild statically linked binaries

That’s a workaround not a feature, it transforms updates in mass rebuilds, that require a lot more build power, and add an economic barrier to entry to the distribution game (not that the big distributors will complain much about this part).

Nevertheless the real major drawback of static building is that it removes any developer incentive to converge on specific component versions. With dynamic building you have to choose one of the handful of versions packaged by your distributors. With static building there is no reason to make the effort (which is why developers are in deep love with static building).

The consequence of this lack of version convergence, is that static building is not only massively more wasteful on building power, it is massively more demanding in maintainer power. You need to tailor security patches (and security impact analysis) to each and every component version individual developers chose to vendor in their static build. In effect you promote technical debt (increase short term benefits at the expense of long term maintenance).

The problem does not exist FAANG side, because FAANGs force their dev teams to use the same golden versions. Guess how promoters of static building would welcome a distribution, that told them “you can use static building, but only with the following vendored versions, because we do not have the resources to patch others”.

Shared libraries

Posted Nov 25, 2025 18:31 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link]

> That’s a workaround not a feature, it transforms updates in mass rebuilds, that require a lot more build power, and add an economic barrier to entry to the distribution game (not that the big distributors will complain much about this part).

Eh, no. Unless you're targeting something like m68k, rebuilding is cheap. We can use Gentoo as an example, rebuilding the world takes about 12 hours on a reasonably fast computer. And this is for a catastrophic bug, where everything needs to be updated.

In most cases, you're looking at maybe a handful of packages.

Shared libraries

Posted Nov 26, 2025 8:54 UTC (Wed) by taladar (subscriber, #68407) [Link] (4 responses)

On the other hand why is converging on a version that isn't the latest version a good thing? Shared library distro builds are essentially always stuck on the oldest version that some reverse dependency requires while static linking can just use the latest version, becoming part of the much, much larger group of people who aren't years behind.

This gets rid of all the effort required for distro specific bugs when developers don't want bug reports for ancient versions, all the backporting,...

Shared libraries

Posted Nov 28, 2025 9:37 UTC (Fri) by nim-nim (subscriber, #34454) [Link] (1 responses)

The latest version is not necessarily reliable, give the developers of other components a break, they also need to explore new approaches. Converging on a common version means a lot of people found this particular version reliable;

Static building is the opposite of using the latest version, people use the latest version *at the time they bother to check* and then lock it down and accumulate technical debt. People do not like making the effort to update, period, they can’t complain using dynamic libraries forces them to update on a regular cadence (that would expose that they do not want to make this effort) they complain that the update forced on them by distributions is not new and shiny enough.

Shared libraries

Posted Nov 29, 2025 16:20 UTC (Sat) by khim (subscriber, #9252) [Link]

> Converging on a common version means a lot of people found this particular version reliable;

That's only true for forges like PyPi or crates.io. Where people are free to pick any version they like.

With distros is the exact opposite: few packagers, often just one, single, person decides what version of library to package.

That's the opposite of the process that may lead to the outcome you describe.

> Static building is the opposite of using the latest version, people use the latest version *at the time they bother to check*

Still better than some random version that was picked by god-know-who for god-know-what reason and which wasn't even tested in conjunction with app.

> they complain that the update forced on them by distributions is not new and shiny enough.

People complain when something doesn't work, period. It may be too new or too old or anything in between.

But with distros they are often in position that they have no one to even complain to because person that assembled the crazy combination of libraries that breaks things is not even result of conscious decision, but more of result of random dice throwing.

According to the distro makers every package should be ready to work with every version of library… but that's rarely a thing that any sane developer would accept: to know whether something works or not you need to test… and distro makers make that exceedingly difficult.

Shared libraries

Posted Nov 28, 2025 11:25 UTC (Fri) by intelfx (subscriber, #130118) [Link] (1 responses)

> Shared library distro builds are essentially always stuck on the oldest version that some reverse dependency requires while static linking can just use the latest version, becoming part of the much, much larger group of people who aren't years behind.

Yeah, that's just the exact opposite of how it happens in reality.

In reality, applications in ecosystems with vendor-controlled static linking/bundling/vendoring get locked to the first version of each dependency that happens to work and never upgraded again (until a vulnerability is exploited in the wild, or until maintenance of a suitably obsolete build environment becomes untenable through actions of others).

Maintainers of distributions, on the other hand, care about health of the overall "ecosystem" Of course, the quality of said care may differ, but the overall concept is always there. You, as a user, may be briefly stuck on non-latest versions (or even on legacy branches) of some dependencies, but the entire distribution moves forward as fast as manpower allows.

Shared libraries

Posted Dec 1, 2025 9:59 UTC (Mon) by taladar (subscriber, #68407) [Link]

That is the way it works in C/C++ when they vendor dependencies because the tooling to upgrade sucks but e.g. on Rust, when upgrading dependencies is as simple as running cargo upgrade and cargo update and then fixing the minor compile issues 99% of the time (i.e. every time some dependency didn't completely revamp their API which is very rare) keeping up with dependency versions is incredibly easy.

Shared libraries

Posted Nov 26, 2025 9:22 UTC (Wed) by taladar (subscriber, #68407) [Link]

If rebuilding thousands of packages was such a big deal, why can the Rust project easily do https://rustc-dev-guide.rust-lang.org/tests/crater.html runs regularly to test if the compiler breaks any crate on crates.io?

ABI stability can block security updates

Posted Nov 27, 2025 18:14 UTC (Thu) by DemiMarie (subscriber, #164188) [Link] (3 responses)

Static linking allows security updates to libraries to be applied to applications that are ready for them, even when other applications are stuck on vulnerable versions. See gRPC in Fedora.

ABI stability can block security updates

Posted Dec 1, 2025 9:17 UTC (Mon) by nim-nim (subscriber, #34454) [Link] (2 responses)

Unfortunately baddies do not care if you have double plated the main door with the latest uber-expensive alloy, if the service door right next still uses rotting medium (a very common situation in proprietary setups).

What matters for security is that *every* deployed software bit uses fully patched components, even when those components are slightly old because no security event required their update and a full OS update is expensive effort-wise.

Static linking as you wrote promotes fixing highly visible main doors while keeping service doors wide open.

ABI stability can block security updates

Posted Dec 1, 2025 10:50 UTC (Mon) by taladar (subscriber, #68407) [Link]

You are completely ignoring that dynamic linking keeps the entire distro on an old version (or rather often a new, completely untested version somehow considered stable because it is called a "backported fix") because the many less popular programs that most systems don't even install have not been updated yet.

ABI stability can block security updates

Posted Dec 1, 2025 10:59 UTC (Mon) by muase (subscriber, #178466) [Link]

But that’s not true, as different components have different attack surfaces. gRPC is a good example, because it’s a huge difference if you have a public service that works with untrusted gRPC messages from the outside, or if your coordinator just uses a fixed set of gRPC messages to distribute your physics simulation across multiple worker processes via stdin/stdout on your own desktop.

In the first case, that’s a real-world RCE, in the second case you probably cannot even exploit the bug because you can’t control the input in the first place (messages are generated by the coordinator, not you).

Different use cases have different attack scenarios and different attack surfaces. There are tons of applications where a gRPC vulnerability will be a serious incident; but there are also a lot of cases where it – realistically speaking – it’s simply not exploitable.

Ideally, you should fix all bugs asap – but given the limited resources IRL, it sometimes is simply necessary to triage. That includes that if you cannot fix all packages at once, you can at least try to fix those where the bug has the most impact; i.e. users*impact (“reduce the bleeding”). It simply does not make sense to hold a fix back for the majority of users because you’re blocked by some obscure package nobody is using, or to wait for a package where the bug has no relevant real-world impact.

Shared libraries

Posted Nov 25, 2025 17:58 UTC (Tue) by JanC_ (guest, #34940) [Link] (19 responses)

There is also the issue of having to upgrade 100 packages instead of 1, which can make a big difference in time, network usage, required disk space, …

Shared libraries

Posted Nov 25, 2025 18:33 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (18 responses)

But why? I'd argue that this is more of a problem of existing packaging systems, they are inefficient by design.

There are no technical problems in storing updates as binary deltas. It's just that nobody cared to spin up infrastructure for this.

Shared libraries

Posted Nov 25, 2025 19:21 UTC (Tue) by bluca (subscriber, #118303) [Link] (17 responses)

That's because they are crap, and work only in theory. There's a reason stuff like deltarpm was abandoned. It just doesn't work in the real world, and it's effectively a workaround for a problem that doesn't need to exist in the first place.

Shared libraries

Posted Nov 26, 2025 2:34 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (16 responses)

deltarpm was just bad, just as debdiffs. Not in the least because DEB/RPM themselves are inefficient.

Format-aware binary diffs can be incredibly compact. RedBend Software used to provide target-aware OTA diffs, and they could compress a simple binary patch to just about the source code difference in bytes. Its differ used additional instructions to represent address shifts, and it could extract locations from the ELF relocation section.

And these days, not many people are going to care about update sizes anyway.

Shared libraries

Posted Nov 26, 2025 11:18 UTC (Wed) by bluca (subscriber, #118303) [Link] (15 responses)

So everything that actually exists was crap but it was just a coincidence, and the implementations that actually don't exist are the bee's knees - got it

Shared libraries

Posted Nov 26, 2025 18:37 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (14 responses)

OK. I'll bite.

Can you explain exactly why even simple brain-dead binary diffs are crap? I did an experiment, changed an "if" condition in a large binary, and did a diff. This is probably what most security patches are.

I used simple `rsync --only-write-batch=diff file1 file2`, and for a 50Mb binary the diff was 1Mb. I have not tried bsdiff or xdelta.

Shared libraries

Posted Nov 26, 2025 18:55 UTC (Wed) by bluca (subscriber, #118303) [Link] (13 responses)

Because operational reality down in the real world is very, very far away from experiments with spherical cows in a vacuum. It's the same with kernel live patching: great idea in theory, but in practice it's so damn expensive to make it work for real, on real systems, ran by real people, with real random variations, that depend on them for real production use cases, that in reality you need an entire paid team to carefully shepherd them in a production scenario. And it happens for live patching because it's worth real money, as the only alternative to apply kernel secury updates is rebooting and thus very long and measurable downtimes, and it's somewhat simpler because you have _one_ kernel on any given system to delta patch. So if you pay Canonical/RH/Oracle/SUSE you can get access to them, and then you can sort of manage it with enough engineering resources.

It's orders of magnitude worse for deltarpm and similar because instead of one kernel to build and manage patches for you have N packages, and every node will have a different and unique combination. Combinatorial explosion. Complex, costly to manage, and benefits on a well designed systems that ships critical components such as libc or libssl as shared libraries that can get easily updated are too small to notice.

Shared libraries

Posted Nov 26, 2025 19:20 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

I don't get it. We have a globe-spanning binary-diff-driven system right now. You're using it every day, in fact. It's called "git".

It works just fine, somehow. All you need is protocol-level diffing support that deltarpms just half-assed. Even with the most naïve implementation, the end result should be bit-for-bit identical to just transferring files.

You want more examples? OSM (OpenStreet Maps) distributes up-to-minute incremental changes as diff files, and their full database is around 100GB. Yet they manage.

You want even more examples? Dockerhub and Docker images are represented as a set of deltas on top of a base image.

The real problem is, in fact, the RPMs and DEBs themselves. It routinely takes _longer_ to unpack and install Fedora or Debian updates than it takes to download them. Heck, the `apt-get update` step in my Dockerfiles is usually one of the slowest!

Shared libraries

Posted Nov 26, 2025 20:49 UTC (Wed) by bluca (subscriber, #118303) [Link] (9 responses)

> I don't get it. We have a globe-spanning binary-diff-driven system right now. You're using it every day, in fact. It's called "git".

Here's git in the real world: https://xkcd.com/1597/

> You want more examples? OSM

I'm not familiar with it, I suspect the bits are not user-facing pick-and-choose, but entirely behind the scenes, with no user control of installations and updates. And it's pure data, not running code. That makes it way, way easier and more manageable.

> You want even more examples? Dockerhub and Docker images are represented as a set of deltas on top of a base image.

Docker is a dumpster fire

Shared libraries

Posted Nov 27, 2025 0:15 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (8 responses)

> Docker is a dumpster fire

Yeah. Because it highlights just how terrible APT or RPM are. I can update 20 of my self-hosted apps _faster_ than it takes to update the host OS (Fedora). And to remind you, Docker is very much a user-facing, "pick-and-choose" system that is based on delta updates. A total impossibility, in other words.

And it works because it was architected correctly. Not _perfectly_, but correctly.

Why have the RPM diffs and APT deltas failed? It's because each package is installed separately. Apt has to pretend that each package is stand-alone, so it can't install them in parallel. And when you have thousands of packages, adding just a tenth of a second to the sequential critical path becomes painful, for not much gain for most people.

A more modern update system would dispense with all the individual package nonsense. It would fetch the updates (in parallel if possible) and in parallel pre-stage the changed files, resolving the deltas as soon as it gets the data. Then it would do atomic replacement of changed files (or as close to atomic as filesystems make it possible) and run whatever scripts needed to complete the update.

Shared libraries

Posted Nov 27, 2025 0:37 UTC (Thu) by bluca (subscriber, #118303) [Link] (5 responses)

> Yeah. Because it highlights just how terrible APT or RPM are.

Nah, it's shite with pacman and everything else too

Shared libraries

Posted Nov 27, 2025 0:47 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (4 responses)

So, correct me if I'm wrong, everything that solves the users' problems with updates is bad? As long as it makes the updates fast, seamless, reliable, and atomic?

BTW, even the _actual_ user-facing versions of Fedora are switching away from RPMs. See: Bazzite.

Shared libraries

Posted Nov 27, 2025 1:06 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> So, correct me if I'm wrong, everything that solves the users' problems with updates is bad?

Ladders are really, really useful for climbing to the top of a one or two-floor building.

But for 3-4 floors, portable ladders are no longer an option. Beyond 5.. ladders of any sort are worthless and a completely different solution must be employed.

In case you can't follow this analogy, something that is optimal for one application, or even a couple of applications, rapidly runs into fundamental scaling problems when you try to scale it to dozens or hundreds of applications.

But application writers don't care about bigger picture problems; they only care about *their* application.

Shared libraries

Posted Nov 27, 2025 6:58 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> But application writers don't care about bigger picture problems; they only care about *their* application.

I agree with your analogy. Except that classical distros are these rickety wooden home-made ladders. And modern Docker or immutable distros are tower cranes.

Shared libraries

Posted Dec 2, 2025 21:06 UTC (Tue) by raven667 (guest, #5198) [Link] (1 responses)

I think it is a good fit for the OS distributor using tools like RPM to manage the internal complexity of their software, then roll that up into an image with deltas for distribution in the way that Bazzite and other rpm-ostree distros are built, same for building Flatpak and Docker runtimes and apps. You need a tool like RPM/deb to track how a complicated runtime gets built.

I've been working with package-based OS since Redhat 4.2 (Colgate IIRC) and can see the limitations in scaling up a full desktop workstation using rpm/dpkg and yum/apt, where update performance is slow (my Fedora MacPro I used to use would take *hours* to update between releases on spinning rust) and the cost of de-duplicating all application libraries is the local admin has to use the packaging tool to manage long dependency chains, and things get complicated if you want to install an app which the OS distributor hasn't packaged themselves.

For someone who just wants to _use_ the computer to do computer things, rather than making OS design and development their hobby, the image-based systems with container-based apps work a lot better and more reliably than the package-based model, at least in my experience. Heck, having a good way to rollback because you aren't trying to mutate the one-and-only live system comes in really handy and makes me sleep better at night that the computer isn't going to immolate itself if something goes wrong.

Shared libraries

Posted Dec 3, 2025 2:50 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link]

Yup. As many people noted, Linux works well for local development. And image-based systems allow that local environment to be used more easily for deployment. Package-based distros are also great for low-level tinkering and experimentation.

Although classic distros might also eventually change a bit. I think something like nix or guix might end up a better solution.

Shared libraries

Posted Nov 27, 2025 23:08 UTC (Thu) by intgr (subscriber, #39733) [Link] (1 responses)

> Docker is a [...] system that is based on delta updates.
> it was architected correctly.

I agree that compared to classical package managers it's impressively fast.

But no, Docker's deltas aren't even optimal for updating your application from one version to the next. They are a delta between one layer and the next one, so only efficient when the last few layers have changed.

Fast updates are often a symptom of applications not updating their base image with security patches as often as they should.

Docker could be a lot faster even, if it computed deltas between versions, or had a protocol like rsync or casync that allows figuring out the delta on the fly.

Shared libraries

Posted Dec 1, 2025 17:21 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

There was an AllSystemsGo! talk years ago about an OCIv2 which did chunking for image delivery and storage, but I believe it is DOA.

Simple in theory, but .deb format makes it a little complicated.

Posted Dec 5, 2025 11:02 UTC (Fri) by gmatht (subscriber, #58961) [Link]

I did a bit of work on this back in the day.

In principle this is really simple. Grab the .gz files out of the .deb using ar, zsync [1] them with the .gz files stored on some server (using either a cached .deb file or just $(cat `dpkg -L myoldpackage`). Then use ar to pack the .gz files back into a deb. Optionally also store some binary diffs from some particular versions the user might be likely to have lying round, as this is more efficient than zsyncing between arbitrary versions.

However, there are some issues with the .deb format. The compressed files are signed, so once you have updated your files you have to recompress them before you can feed them back into dpkg. Apart from slowing things down a bit this also means that you have to use tricks to get the resulting .gz to be bit for bit identical to the official .gz files. OK, it is possible create reproducable gzip files, but Debian seemed to be moving away from .gz compression.

If Debian decided to switch to a hsynz/zsync friendly format it should be easy. For example, if Debian switched to using uncompressed tars in their .deb files and instead distributed .deb.zstd files it should be way easier to support incremental updates than say, rewriting apt in rust.

[1] These days we would probably instead use https://github.com/sisong/hsynz

Delta updates work great with image-based OSs

Posted Nov 27, 2025 18:17 UTC (Thu) by DemiMarie (subscriber, #164188) [Link]

On image-based OSs, delta updates work great because there is only one old version.

Shared libraries

Posted Dec 5, 2025 2:20 UTC (Fri) by dvdeug (subscriber, #10998) [Link] (16 responses)

> Imagine a new version of libtiff which introduces a security-relevant bug into the decompressor for TIFF compression scheme 32809 (ThunderScan 4-bit RLE). Upstream's statically linked builds of the program are not vulnerable, because they don't enable the bits of libtiff needed to handle files from ancient Macs, but because your distro includes a utility that's supposed to analyse an ancient Mac disk image and convert all the data to modern formats that you can work with, your distro build of libtiff has this support enabled.

Imagine using a bunch of old TIFF files and finding that one program doesn't read them, despite the fact they all use libtiff. You're the type of person who reads LWN, so after an hour of work, you might figure out that the program uses some brain-damaged version of libtiff. It would be a lot more painful for less technically skilled users.

libtiff is not the most secure library in the world, but randomly turning off features is painful to users. TIFF is annoying as a complex standard that no one completely supports; breaking support for random subformats (which are not visible to the average user, and even if they were, that this program doesn't support ThunderScan 4-bit RLE is probably documented nowhere) just makes things more miserable.

To boot, nobody is messing with the code for that compression scheme. There's a good chance that the bugs that affect it have always affected it. Changes to libtiff are more likely to break compression schemes that are worth optimizing, or some combination of important features that the programmer wasn't thinking about in combination and nobody wants to disable.

Shared libraries

Posted Dec 5, 2025 2:38 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

So you convert from the ancient compressed TIFF to something more modern, like PNG. Problem solved in practice.

Shared libraries

Posted Dec 5, 2025 9:33 UTC (Fri) by dvdeug (subscriber, #10998) [Link]

Except that TIFF has rich metadata support that PNG doesn't, including a whole set of private tags, and it's likely anyone still using TIFF is depending on one or the other. Changing to PNG is likely going to involve exporting that data to some format, and fixing up your system to import that data alongside the PNG.

And what's stopping someone from making a local copy of libpng and removing, say, Adam7 interlacing from it? "Nobody uses it", and it's more code surface. TIFF is a little weird, but if you support a file format, you should support it as far as reasonably possible, not randomly turning off features you're not interested in.

Shared libraries

Posted Dec 5, 2025 11:11 UTC (Fri) by farnz (subscriber, #17727) [Link] (13 responses)

In the scenario I outlined, you'd read the TIFF with the software that knows about legacy Mac formats, and convert it a modern TIFF in the process - you're already having to use that software to get the TIFF format out, and it can decompress ThunderScan 4-bit RLE and either leave the data uncompressed, or recompress with a more modern lossless compression scheme.

And I chose that format very, very deliberately; it was never commonly used with TIFFs, and the only time you're going to encounter it is if you're extracting files from a legacy Mac where you'd bought the ThunderScan attachment for your ImageWriter. Most people won't ever encounter a file with ThunderScan RLE encoding - very few programs outside Macs with the ThunderScan software installed could even read them, and if you opened it in (say) Photoshop, then saved as a TIFF, it'd compress it differently anyway.

While nobody may be messing with the code for it, there's also a good chance that until the compiler changes, the bug was latent - if it contains UB, then the compiler is perfectly capable of compiling it as the programmer intended, up until it changes and starts compiling it in a way that's exploitable (but that correctly decodes all valid ThunderScan compressed TIFFs). At this point, you've created a vulnerability in many programs, to support the one or two that still need to decode ThunderScan RLE compression.

Shared libraries

Posted Dec 5, 2025 21:49 UTC (Fri) by dvdeug (subscriber, #10998) [Link] (12 responses)

> In the scenario I outlined, you'd read the TIFF with the software that knows about legacy Mac formats, and convert it a modern TIFF in the process

There's a lot of times the solution is set VXTMLR=1 and it just works. A fact you learn six weeks after you finish redesigning the whole system with new software and custom code, or replaced a piece of hardware. You have a TIFF file that is read by TIFF reading programs; the fact that it's not being read here will frustrate all non-technical users and many technical ones. Then you get the tech in, and they realize it's running libtiff like the rest of the system, so why isn't this working?

> And I chose that format very, very deliberately

And I stand by my case; not supporting certain files in a format because you don't think they're being used is a recipe for pain and annoyance. I'll go further and point out that the gain for disabling this is tiny; if you don't trust libtiff, then you should protect yourself in some way, not just disable one or two compression schemes that you don't feel are being used that have a tiny chance of being the angle someone uses to attack libtiff. TIFF, like various media schemes, is a format you'll never support all files in the wild, but you should either support some narrowly and clearly defined subset, or support everything that e.g. libtiff/ffmpeg does. Or don't support it at all, which means people know to run a different tool or convert it to a format you support.

I'm also not convinced that everyone will be so careful as you. It's easy to get carried away; most attempts at making an X clone with only the features that people really use fail because while people only use 20% of the functionality, they all use a different 20%.

Shared libraries

Posted Dec 6, 2025 21:24 UTC (Sat) by farnz (subscriber, #17727) [Link] (11 responses)

Most of the time, TIFF reading programs don't support ThunderScan RLE encoding. That's a big part of why I chose it - it's rarely supported unless you've deliberately enabled it, and you only need it if you're dealing with deeply legacy media.

I could, of course, have chosen Internet-facing streaming media software that uses its own build of ffmpeg in the upstream configuration, where only the formats that are expected to work are enabled, but where the distro has turned on support for all the weird and wonderful formats that ffmpeg supports.

In both cases, though, your argument is that it's important to introduce security issues into software whose upstreams don't have that issue because they run with cut-down dependencies, because there might be a rare user who actually wants to deal with ThunderScan RLE TIFF files, or upload LucasArts SANM videos for transcoding to AV1 for streaming, or otherwise do something deeply niche and weird. I would suggest that actually, what you want is more than one build of libtiff, or ffmpeg, or other functionality, where the local-only media player can decode LucasArts SANM, or the legacy 68k Mac (ThunderScan was not supported on PPC Macs) file reader can convert your scanned documents to a modern format, but the Internet-exposed one doesn't have the extra codec support by default, precisely because of the security risk.

Shared libraries

Posted Dec 6, 2025 21:48 UTC (Sat) by johill (subscriber, #25196) [Link] (6 responses)

On the flip side, when a distro chooses to have a single libtiff/ffmpeg/whatever build installed at a time, it could actually give users a better choice. Default to libtiff-simple, but let a user install libtiff-everything, and both provide a virtual libtiff package. Now all software can either do "ThunderScan RLE encoding" or not, depending on which libtiff you installed, and you don't even need to rebuild the software if you do have such a special case. (Though obviously the problem with this is making it discoverable.)

Which is basically the "multiple builds", but as long as you don't need to do "internet facing server" and "legacy media" on the same machine, it'd be much simpler.

(You could argue that maybe a web browser should have a restricted version and the local viewer not, but realistically you might even need the browser to have the full version if you're working with such files since you might have a web-based organisation tool etc., so I don't think that really works)

Shared libraries

Posted Dec 8, 2025 9:28 UTC (Mon) by farnz (subscriber, #17727) [Link] (5 responses)

The trouble with that is that what you want is not "simple" and "everything", but a variety of different choices: there's faxes, medical imaging, scanned paperwork, prepress artwork and more, to name some common use cases for TIFFs.

In the extreme, you end up with a different libtiff for each use case, and you want them to be parallel installed and each only used by one program on a system - e.g. the fax handler has the fax subset (and doesn't have any others), while the paper document archive program supports the subsets used by faxes and scanned paperwork, but not medical imaging or prepress artwork.

And once you get there, where's the advantage of dynamic linking?

Shared libraries

Posted Dec 24, 2025 14:49 UTC (Wed) by sammythesnake (guest, #17693) [Link] (4 responses)

If there's a need for that kind of thing, then a plugin mechanism is the real answer - nothing to do with static linking or otherwise...

Shared libraries

Posted Dec 24, 2025 17:31 UTC (Wed) by farnz (subscriber, #17727) [Link] (3 responses)

A plugin mechanism doesn't help - it just rearranges the deckchairs.

Let's switch to video formats, and use GStreamer, which has a plugin mechanism. How does my "upload a video from your phone via the Internet" program know that a given plugin is not for it to use? In the upstream version, it's fine - they just don't install vulnerable plugins, so by definition, all plugins are fine for it to use.

But the distribution has a problem: a phone will never record in (say) LucasArts SANM format, but other software installed by the user might well have a legitimate reason to want the GStreamer plugin for LucasArts SANM to be available and working. Equally, though, the program might well benefit from you adding an AV2 plugin (so that phones that record AV2 video instead of HEVC are supported), even though it doesn't know about AV2.

This, however, is the same problem as before, just rearranged - you need parallel installable "profiles" of the library (in this case GStreamer), such that each program gets the profile it wants. And in the extreme, each program has its own profile.

Shared libraries

Posted Dec 24, 2025 18:22 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

> This, however, is the same problem as before, just rearranged - you need parallel installable "profiles" of the library (in this case GStreamer), such that each program gets the profile it wants. And in the extreme, each program has its own profile.

So `/etc/pam.d`-like files but for plugins then? Sounds like fun ;) .

Shared libraries

Posted Dec 24, 2025 18:56 UTC (Wed) by excors (subscriber, #95769) [Link] (1 responses)

That sounds like a pretty easy problem to solve: the application can have an allowlist of plugins. When a significant new codec like AV2 is released, maybe a couple of times per decade, the developers can add it to the list. Maybe have a config file so users of old releases can add it without upgrading. This should only be needed for applications that are processing untrusted input on systems where they can't trust the administrator not to install insecure plugins, so it's going to be end-user things like web browsers that have to be constantly upgraded for security reasons anyway and it's minimal extra effort to maintain the plugin list.

(GStreamer already splits plugins into "good" (safe to use), "ugly" (good quality code but copyright/patent licensing issues), and "bad" (unsupported; maybe low quality, unmaintained, rarely used, etc), with separate Linux packages for each set, plus a number of single-plugin packages (for licensing or dependency reasons). On systems with a responsible administrator, that means you can already choose what profile of plugins are available, and there's no technical reason it couldn't be more fine-grained.)

Shared libraries

Posted Dec 25, 2025 19:14 UTC (Thu) by farnz (subscriber, #17727) [Link]

But this is the same mess as multiple shared libraries, just moved around - it's not actually a solution if you want to have all programs use a unified feature set, which is an explicit goal of how the distro mechanism works (see the earlier comment from dvdeug).

And that's the root of the problem I'm raising. There exist cases where unifying features introduces a vulnerability - indeed, the allowlist mechanism you're described also introduces one if an insecure plugin is added that upstream explicitly don't support because they know it opens a security hole in their software. As a consequence, turning a simple "statically link ffmpeg, configured just for the plugins upstream has reviewed and know are good" into "port to a plugin architecture, maintain allowlists, somehow ensure that downstream never adds something insecure to the allowlist" problem, which is a lot of extra complexity, and all so that downstream can "unify" features for things that don't benefit from the unification (since the allowlist promptly bans use of all the extra features so enabled).

This is the flip-side of distros spotting problems as they package - because they change things (they're effectively a friendly fork of upstream), they run the risk of introducing new bugs. It is not nearly as simple as "static linking bad, dynamic linking good"; it can also be "by unifying features, I've introduced a bug that does not exist upstream, because I de-vendored a library, linked to the shared version, and now we have problems".

Shared libraries

Posted Dec 7, 2025 8:56 UTC (Sun) by dvdeug (subscriber, #10998) [Link] (3 responses)

> your argument is that it's important to introduce security issues into software whose upstreams don't have that issue because they run with cut-down dependencies, because there might be a rare user who actually wants to deal with ThunderScan RLE TIFF files, or upload LucasArts SANM videos for transcoding to AV1 for streaming, or otherwise do something deeply niche and weird.

My argument is that first, you aren't gaining much by stripping features because they're rare. Either you trust libtiff, or you don't. FFMPEG has more dependencies and sometimes more sketchy dependencies, so that's slightly different. But in either of those cases, you're taking a small chunk of code in a much, much bigger library, and treating it as a security win to disable it, not because it's known to have anything wrong with it, but just because you can. Don't; if you don't trust libtiff, sandbox it. If you find libtiff trustworthy enough, you're not gaining anything measurable by disabling random tiny parts of it.

Two, to you, a ThunderScan RLE TIFF file is an obscure Macintosh format. To a user, it's a TIFF file, a relatively common format. If I try and use an .snm file in a tool that plays movies, and it doesn't work, I look for a tool that supports .snm files. If I try and use a valid .png file in a tool that supports PNG files, and it doesn't work, I'm going to treat that tool as broken. I've read the TIFF standard, and understand how complete support is impossible, but the user is going to treat that as the PNG case, not the SNM case.

Third, I think this is a slippery slope that would make software less pleasant to use. ThunderScan RLE is a rare format, yes. But as I said, it's a small chunk of code, so what else are you going to take out? I have 16000 TIFF files on my hard drive, and most of them are in CCITT Group 4 fax encoding, which is not in the TIFF baseline. They may be few, but I have 2, 3 and 4 respectively in PackBits (baseline, but considered rare on Wikipedia), Deflate (PKZIP) (obsolete, uncommon), and LZW (not in baseline, and who uses LZW any more?). Having to spend any time to figure out why the program that views TIFF files doesn't view these is a waste of my time, and for many users, that could be a waste of hours and possibly expensive bills to call in a technician.

> I would suggest that actually, what you want is more than one build of libtiff, or ffmpeg, or other functionality,

No; I want something that processes TIFF files to actually process all TIFF files. That's not realistic, but approaching it is still important for the quality of user experience. If you don't trust libtiff, I expect you to not just disable a couple of minor compression schemes and call it done; you should protect the program from libtiff bugs. ffmpeg is a gnarlier mess, but I suspect there's even less real data about what video codecs people are using, and probably even more reason to sandbox it instead of trying to figure out which codecs are unused and blindly trusting the rest.

Shared libraries

Posted Dec 8, 2025 9:24 UTC (Mon) by farnz (subscriber, #17727) [Link] (2 responses)

Just to be clear, then: you want a fax handling program to have extra vulnerabilities over and above upstream, because it's possible that someone might want to reuse a fax handling program for legacy Macintosh scans.

There is no program out there that supports all TIFF files - libtiff certainly doesn't.

And coming back round to my initial point: by adding in the extra file format support to someone's fax handler, the distro has introduced a vulnerability that is not present upstream, and where upstream is quite likely to say "well, why did you add ThunderScan RLE decoding to my fax program? That makes no sense at all, since faxes are 1 bit (by definition), and ThunderScan RLE is 4 bit (by definition)". This is not a win for users, or for the upstream, and it's a security hole opened by the distro insisting that there is a shared libtiff.

Shared libraries

Posted Dec 8, 2025 11:16 UTC (Mon) by dvdeug (subscriber, #10998) [Link] (1 responses)

>Just to be clear, then: you want a fax handling program

Just to be clear, the idea that this was a fax handling program is present nowhere else in this discussion. At no point did you offer a reason ThunderScan RLE wouldn't be supported besides "it's rare".

> There is no program out there that supports all TIFF files - libtiff certainly doesn't.

There's no program out there that supports all C++ files; g++ certainly doesn't. Is that argument going to pacify you when it fails on your C++ file? I'm asking programs that claim to support TIFF files to try their best at supporting TIFF files, and not put extra work into disabling support for certain TIFF files.

>And coming back round to my initial point: by adding in the extra file format support to someone's fax handler, the distro has introduced a vulnerability that is not present upstream, and where upstream is quite likely to say "well, why did you add ThunderScan RLE decoding to my fax program? That makes no sense at all, since faxes are 1 bit (by definition), and ThunderScan RLE is 4 bit (by definition)". This is not a win for users, or for the upstream, and it's a security hole opened by the distro insisting that there is a shared libtiff.

No, the distro has not introduced a vulnerability that is not present upstream. It may have introduced a vulnerability, and the probability it did so is key to judging whether or the action should be taken. Again, it is a win for users to have as much support for their files as possible, and this idea that the program wouldn't accept ThunderScan RLE anyway is something new. The very fact that it only supports 1 bit files means that it should bail out of reading the file long before any ThunderScan RLE specific code is run.

It's possible that just adding the support code will close a vulnerability; mishandling of an error state was what destroyed the first Ariane 5, one of the most expensive software errors ever, after all. The log4j problems, similarly, came from issuing errors to a log file and could have been blocked in some cases by just changing which error message was logged.

More importantly, by linking the program to the shared libtiff, it avoids any vulnerabilities that could come from the window between the shared libtiff being fixed and the program being fixed. That's why distros demand that programs be linked to shared libraries instead of vendored ones, so security holes in the library get quickly patched on being fixed, instead of fixed in every copy on the system individually.

Even in the fax case, it's not as simple as the distro introducing a vulnerability; the distro could be patching a vulnerability just by making the change, and they could be minimizing the time that an openly known vulnerability exists in the program. Even if the user support is irrelevant, how likely a vulnerability is to be created by a change and how likely it is to be fixed by a change is important.

Shared libraries

Posted Dec 8, 2025 11:25 UTC (Mon) by farnz (subscriber, #17727) [Link]

You made assumptions about what the program might be, and asserted that a bug introduced by ThunderScan RLE support would be worth having for the extra supported formats; I've introduced a bit more information, explaining why ThunderScan RLE support is not worth having.

And just to be completely and utterly clear: you are saying that a vulnerability that is not present in the upstream version of the project, or in their binary builds, but is only present in the distro build (where they've unbundled libtiff and linked a version that has a security issue that can be triggered just by loading a TIFF file) is not introduced by the distro?

And my argument is that the distro is not always fixing things by unbundling dependencies and unifying on a single version - it can be both fixing some problems and introducing new ones. Which tradeoff is better is not backed (in either direction) by data.