Shared libraries [LWN.net]

Shared libraries

Posted Nov 25, 2025 14:23 UTC (Tue) by gspr (subscriber, #91542)
In reply to: Shared libraries by keithp
Parent article: APT Rust requirement raises questions

Not knowing much about this, I've always wondered why a relatively closed set of packages – like those that constitute a distro – couldn't be transitively monomorphized *internally* in the set.

For example, take the directed graph of dependencies between Rust packages in Debian. Pick any package that is not a library (i.e. not a librust-foo-dev package). This package surely uses, in its dependencies, either monomorphized versions of functions and types, or dynamic dispatch. Note down all the monomorphized versions, and add them to a list for each dependency. Traverse the graph in topological order, and build these monomorphization lists for all dependencies. Then build all library packages as shared objects with all of those monomorphic instances explicitly stamped out (I understand there's no compiler support for this at the moment, but it shouldn't be too hard to fake it by generating stubs?). Will this not allow dynamic linking and bug-fixing in shared objects *within* Debian at least? For a given compiler version, of course. Non-Debian software that uses the libraries are no better off than before (unless they happen to need the same monomorphizations), but they're also no worse off.

I'm sure I'm overlooking something here, but I'd love to learn :)

to post comments

Shared libraries

Posted Nov 25, 2025 14:39 UTC (Tue) by farnz (subscriber, #17727) [Link] (8 responses)

The problem comes with updates. If you update (say) ripgrep to fix a bug, and it uses a new monomorphization, that new monomorphization can rely on a new monomorphization inside a library package, and so on.

You end up with the same problem as the rebuild problem, since you cannot determine ahead of time that no bug fixes will involve a new monomorphization. You will probably reduce the number of total rebuilds you need, but if you're unlucky, you won't.

Shared libraries

Posted Nov 25, 2025 14:43 UTC (Tue) by gspr (subscriber, #91542) [Link] (6 responses)

> The problem comes with updates. If you update (say) ripgrep to fix a bug, and it uses a new monomorphization, that new monomorphization can rely on a new monomorphization inside a library package, and so on.

Is that likely? Or, is it any more more likely than, say, a bugfix in a classical C library needing to break the ABI?

Shared libraries

Posted Nov 25, 2025 14:46 UTC (Tue) by farnz (subscriber, #17727) [Link] (4 responses)

If you're doing the change downstream, then yes it is quite likely - something as "trivial" to upstream as "add a new variant to an error enum" is a new monomorphization, with the resulting need to recompile everything that knows the layout of that enum.

Shared libraries

Posted Nov 25, 2025 14:51 UTC (Tue) by gspr (subscriber, #91542) [Link] (3 responses)

> If you're doing the change downstream, then yes it is quite likely - something as "trivial" to upstream as "add a new variant to an error enum" is a new monomorphization, with the resulting need to recompile everything that knows the layout of that enum.

Definitely. But a similar change in a classical C library would be to return a new error value. That wouldn't technically break the ABI, but it would sure require depending packages to acquire knowledge of the new error value. That would take *more* than just recompiling.

I guess what I'm saying is that this approach doesn't always work, but it's not much worse than the situation for classical C libraries.

Shared libraries

Posted Nov 25, 2025 14:59 UTC (Tue) by farnz (subscriber, #17727) [Link] (2 responses)

Returning a new error value that was previously impossible is an ABI break, in both C and Rust, unless it's clearly documented beforehand that other errors are possible.

For example, if I truncate the error value to 8 bits to make it fit an existing struct, because all known error values are under 255, and you introduce error value 256, I've got a problem in C. This gets worse in Rust, because enums aren't just a value, they can carry data, too, so the enum may get larger as a result of the change, and upstream won't care that the old enum compiled by Debian was 72 bytes, and the new one is 80 bytes - especially if compiled with a newer compiler, they're both 64 bytes.

Shared libraries

Posted Nov 25, 2025 15:34 UTC (Tue) by gspr (subscriber, #91542) [Link] (1 responses)

Ok, sure, we have an ABI break for both C and Rust. I'm just saying: the situation doesn't get worse, does it?

Shared libraries

Posted Nov 25, 2025 15:42 UTC (Tue) by farnz (subscriber, #17727) [Link]

It's slightly worse, because years of habit mean that C programmers are used to thinking about whether a change will break distro ABIs; Rust programmers aren't, and because Rust makes it easier to express things that are hard to express in C (sum types, for example), they're more likely to make changes to fix a bug that make the situation worse.

Remember that the state we're in with C is in part because the language requires programmers to get it right, or risk UB, and as a result, C programmers doing security fixes tend to be thinking about all the ways they can accidentally break someone; Rust programmers tend not to be doing that, because the result of breaking someone is a compiler error, not UB.

That cultural difference matters, and is part of why the aim on the Rust side is to have a state where swapping in an incompatible dynamic library is a dynamic linker failure, not UB as it is in C.

Shared libraries

Posted Nov 25, 2025 17:16 UTC (Tue) by ssokolow (guest, #94568) [Link]

Rust has automatic struct packing and niche optimization and, were it not for the need for reproducible builds, the support for randomizing struct layouts to help people catch Hyrum's law mistakes might be on by default.

Shared libraries

Posted Nov 25, 2025 17:05 UTC (Tue) by Wol (subscriber, #4433) [Link]

Or you go back to the old FORTRAN libraries that I worked with. The linker pulled in all the modules it knew it needed (and if it had recursive dependencies you had to link the same library several times to get them all). That also had the side effect that all the functions that your program didn't need didn't end up in the executable.

So we then have some fancy update tool (sorry) that takes two allegedly identical libraries, and goes through merging all the different modules into a new updated library. If it finds the same module in both precursor libraries, it would need to check and enforce the rule that the extern definition was identical, before choosing the version with the newest reference number (maybe defined as the most recent modified date - I think that might be a tricky problem?). Or maybe just updates the original library with new modules that didn't originally exist.

Cheers,
Wol

Whole-image optimization

Posted Nov 27, 2025 22:48 UTC (Thu) by DemiMarie (subscriber, #164188) [Link]

I think this could work if one is shipping an image that is updated by replacing it.