Shared libraries [LWN.net]

Shared libraries

Posted Nov 24, 2025 18:42 UTC (Mon) by keithp (subscriber, #5140)
In reply to: Shared libraries by bluca
Parent article: APT Rust requirement raises questions

I've come to accept the reality that compiling parametric polymorphic languages (like Rust and C++) inherently eliminates any notion of strong ABI. To avoid just stuffing everything in boxes and using virtual dispatch for all operations, you have to codegen functions using the concrete type.

So, you either get responsible language design with actual type checking across interfaces, or you get shared libraries. I haven't seen any plan for getting both. It kinda sucks, but given that I have to make a choice, I know which I'm willing to accept.

At this point, I'd assume any time a package using Rust anywhere should trigger a rebuild of any reverse dependencies, at least until policy tells us how to avoid that.

to post comments

Shared libraries

Posted Nov 24, 2025 18:57 UTC (Mon) by ballombe (subscriber, #9523) [Link] (37 responses)

Then there should be a subset of rust without parametric polymorphism.
This is not required to replace C code.

Shared libraries

Posted Nov 24, 2025 21:05 UTC (Mon) by DemiMarie (subscriber, #164188) [Link]

And nobody is going to use it. Too much code would have to be duplicated.

Shared libraries

Posted Nov 24, 2025 21:27 UTC (Mon) by mb (subscriber, #50428) [Link]

It's called "extern C".
You can basically do almost all the things you can do in C. Including dynamic linking.

Shared libraries

Posted Nov 25, 2025 12:02 UTC (Tue) by farnz (subscriber, #17727) [Link] (33 responses)

The problem is more than just parametric polymorphism; it's things like defined constants, semantic meaning of functions and more.

Polymorphism is absolutely fine as long as you are aware that this means that the polymorphic parts of your library live in the caller's binary, not in your binary. Same with defined constants in a header, struct layout etc.

The thing that you need is something that tells you when you've modified something that will be in the caller's binary, not your binary, so that you can undo that breakage. Ideally, you'd also have a way to "shim" your new library, so that old binaries can still link against the new library, and go via the shim that fixes things up so that they continue to work without a rebuild.

But this is a really hard tool to develop; there's a lot hiding in those two sentences. Even just doing the "modified something that will cause breakage" for static linking is hard; and dynamic linking ups the difficulty a notch.

Shared libraries

Posted Nov 25, 2025 13:38 UTC (Tue) by khim (subscriber, #9252) [Link] (32 responses)

> Polymorphism is absolutely fine as long as you are aware that this means that the polymorphic parts of your library live in the caller's binary, not in your binary.

This would only work if your library provides ABI without things like Option or Result… and ABI that doesn't use these is as almost far from idiomatic Rust as "C"

Shared libraries

Posted Nov 25, 2025 13:56 UTC (Tue) by farnz (subscriber, #17727) [Link] (31 responses)

Why? Option and Result can be fully monomorphized in your API, in which case there's no polymorphic parts (even though pub struct Foo<T>(Option<T>) is polymorphic, pub struct Foo(Result<u32, MyError>) is not).

Second, I didn't say that you can't have polymorphism; I said that you have to be aware that your polymorphic components live outside your binary. You can have, for example, pub fn foo<P: AsRef<Path>>(path: P) -> u32 { foo_impl(path.as_ref() }, as long as you are happy that foo is inlined into the caller's binary, while fn foo_impl(path: &Path) -> u32 is in your binary.

The important part is that you're aware of what's in your dynamic library, and what's outside it, and that you have a way to cope with the subset of your code that's in the caller not changing when your dynamic library changes. That might be shims and symbol versions like glibc, or not changing things once they've been exposed in a way that breaks the ABI.

Shared libraries

Posted Nov 25, 2025 14:32 UTC (Tue) by khim (subscriber, #9252) [Link] (30 responses)

> Option and Result can be fully monomorphized in your API

Yes. But not with Rust as it exists today.

> in which case there's no polymorphic parts (even though pub struct Foo<T>(Option<T>) is polymorphic, pub struct Foo(Result<u32, MyError&ht;) is not).

Even pub struct Foo(Result<u32, MyError>) is polymorphic because it depends on a compiler version. Compile is free to change the representation of pub struct Foo(Result<u32, MyError>) at any time, in fact nightly have a flag to do that and stable does it from time, to time, too.

> That might be shims and symbol versions like glibc, or not changing things once they've been exposed in a way that breaks the ABI

Well… compiler upgrade [potentially] break ABI which means you would have to specify precisely which version of the compiler defines it… and never upgrade.

RenderScript tried that and died as a result, Apple ended up in the exact same potion, etc.

You couldn't build a stable platform on a quicksand.

Shared libraries

Posted Nov 25, 2025 15:09 UTC (Tue) by farnz (subscriber, #17727) [Link] (29 responses)

Sure, you'd need the compiler to not break things that are marked as ABI - and you'd have to accept that the stable ABI is not necessarily as efficient as the unstable ABI.

Indeed, you might well end up with a v1, v2, v3 etc stable ABI, where v1 is what we thought was good enough next year, v2 is a decade later with all the small improvements that we've accumulated since v1 was marked stable, with downstream users deciding when it's worth moving to a new version of the ABI and breaking older binaries - or even provide a stable ABI v1 shim that uses the stable ABI v5 code to implement things, and does whatever is needed to get compatibility (copies of data structures etc).

But that's something the compiler team has to commit to. None of this works if the compiler team won't stabilize the ABI (replacing the compiler version dependency with a stable ABI version dependency).

Shared libraries

Posted Nov 25, 2025 15:18 UTC (Tue) by khim (subscriber, #9252) [Link] (28 responses)

> But that's something the compiler team has to commit to. None of this works if the compiler team won't stabilize the ABI (replacing the compiler version dependency with a stable ABI version dependency).

Then what's the point of limiting the whole thing to statically known types? Polymorphic ABIs work with the compiler buy-in just fine: there are Swift, C#, Java, Ada… it's not a rocket science, it's well-tested tech. Know for decades, not years.

Shared libraries

Posted Nov 25, 2025 15:33 UTC (Tue) by farnz (subscriber, #17727) [Link] (20 responses)

I don't know why you'd limit it to statically known types - that's not my proposal at all.

Mine is that the provider of a library with a stable ABI needs to be aware of what they're actually offering - what parts have to be kept stable (including because they're embedded in user code - e.g. user code knows this is a thin pointer, ergo you can't change it to a fat pointer, or user code knows this data structure is 108 bytes in size, so that can't change), and what parts are safe to change (e.g. user code calls into your library at this point, so you can change the implementation).

Note that you might want to allow some of your library code to be inlined for performance - e.g. Vec::len is something you'd want inlined, you might want to inline most of Vec::push, only calling out-of-line code if the Vec needs to grow, for two examples. And when you do that, you need to know that part of your stable ABI is ensuring that the inlined parts still work as designed, even if the parts that you've kept in your shared binary are changing.

Shared libraries

Posted Nov 25, 2025 17:06 UTC (Tue) by ssokolow (guest, #94568) [Link] (4 responses)

What you're describing sounds like what they aim to produce with CrABI.

An opt-in stable ABI for Rust that serves as a higher-level alternative to what you currently get from extern "C" without having to use the abi_stable crate to marshal your types through the C ABI.

Shared libraries

Posted Nov 25, 2025 17:18 UTC (Tue) by farnz (subscriber, #17727) [Link] (3 responses)

It's more than CrABI, but CrABI is a necessary component of it.

I want both CrABI, so that I have a compiler that can give me a stable ABI, and a way to cleanly identify which things are ABI, so that a future version of cargo-semver-checks can tell me not only that I've broken my API stability promises (leaving me to decide if I bump the major version, or if I fix my API), but also that I've broken my ABI stability promises.

I also want to be able to say that my stable ABI depends on you using a compatible stable API crate as part of the build process - just as I can't use a random header file version in C, and rely on it working with a random shared library version - that allows for things like carefully chosen inlining of part of my code into the caller, while still having chunks of my internals in my shared library (e.g. so that fast path code can be inlined, calling a slow path in my shared library).

On top of that, while I'm demanding perfect tooling, moon-onna-stick, and free unicorns for everyone, I want tooling that allows me to knowingly break ABI as long as I provide the necessary shims to let people who linked against the older ABI to continue to work.

Shared libraries

Posted Nov 25, 2025 19:16 UTC (Tue) by ssokolow (guest, #94568) [Link] (2 responses)

just as I can't use a random header file version in C, and rely on it working with a random shared library version

For varying definitions of "rely on"

https://abi-laboratory.pro/index.php?view=tracker

On top of that, while I'm demanding perfect tooling, moon-onna-stick, and free unicorns for everyone, I want tooling that allows me to knowingly break ABI as long as I provide the necessary shims to let people who linked against the older ABI to continue to work.

Knowing the Rust community, give it a decade and you'll probably have all that.

Shared libraries

Posted Nov 26, 2025 10:00 UTC (Wed) by farnz (subscriber, #17727) [Link] (1 responses)

My definition of "rely on" is "the tooling gives me an error if I mismatch the shared library version and the header file". Very few C and C++ libraries make that guarantee - the only library I'm aware of that does make that guarantee is glibc, and they do a lot of work with symbol versioning to make that happen.

Shared libraries

Posted Nov 26, 2025 10:15 UTC (Wed) by ssokolow (guest, #94568) [Link]

I missed that you said "can't" instead of "can". Sorry.

I suspect anything which satisfies both your goals and the Rust devs' goals will be paired with some kind of link-time hackery to ensure that what would otherwise have been statically linked will refuse to start if the subset of the provided library that's being used is not 100% ABI compatible.

"'Just don't make mistakes' doesn't scale" is, after all, a core element of Rust's design philosophy.

Shared libraries

Posted Nov 25, 2025 17:17 UTC (Tue) by khim (subscriber, #9252) [Link] (14 responses)

I understand what you are offering, but I don't understand why would you offer that.

C++ went this way because it could without asking compiler developers to cooperate. People just started providing binary libraries without asking anyone for permission — and that worked because C++ haven't been breaking their ABIs quickly enough.

Rust couldn't do that without asking compiler developers to cooperate… the breakage possibility is real enough… and if compiler developers have to involved anyway… why not give developers ability to make nice, easy to use, generic ABIs and restrict them to this strange subset?

Note that it only works in C++ because of incredible efforts that compiler developers, now, have to provide… no one would have accepted such burden if they wouldn't have been put in the bind “provide backward compatibility or forget about user adopting new versions of your compiler”.

Rust developers may simply refuse to participate in that crazyness (because, as you note, without them nothing would work) or, if they would agree — they can design something more useful and sane.

Shared libraries

Posted Nov 26, 2025 10:49 UTC (Wed) by farnz (subscriber, #17727) [Link] (13 responses)

Because it's not just about generic ABIs, and I'm not talking about restricting developers to a "strange subset". You're projecting your own prejudices on top of what I'm saying, and letting it blind you to what I'm talking about.

I'm saying that we need the tools to allow developers to decide what falls on the "shared binary" side of the library, and what falls on the "inlined into the user" side of the library; in C++, this got forced on compilers by the header file/object file split, where things in the header files are "inlined into the user", and things in the object files are not.

But there's no particular reason that this couldn't be well-designed, such that everything (even constants and data layouts) is fixed up by the dynamic linker at run time by default - but there are performance wins to be had from inlining other things. For example, doing a full-blown function call for Vec::len is an expensive way to read a single usize from a known location in the structure; if you export the layout of Vec and the definition of Vec::len such that they can both be inlined into the caller, then you need to ensure that the layout of Vec and the definition of Vec::len remain compatible into the future. Or you might decide that optimizations enabled by knowing that Vec is always going to be 24 bytes, 8 byte aligned, on this platform, are worth enabling, and thus export that layout.

And that's why I want to offer that - I want library authors (who presumably understand the library code) to be able to decide what is inlined into the calling binary, and what is put in the library's binary, with tooling support for stopping them from accidentally changing something from inlined into the caller's binary to part of the library's binary.

Shared libraries

Posted Nov 26, 2025 13:42 UTC (Wed) by khim (subscriber, #9252) [Link] (12 responses)

> I'm saying that we need the tools to allow developers to decide what falls on the "shared binary" side of the library

Nope. We need tools that simply work. Developer shouldn't consult some large document to find out what would be embedded where. Except in rare cases marked unsafe. C/C++ tried that “you are holding it wrong” approach — and it doesn't scale.

Developers have some ideas about what it takes to have stable API (and there are checkers for that), that should be enough. How exactly compiler would ensure that stable API would become stable ABI, also — is on the shoulders of the compiler.

> in C++, this got forced on compilers by the header file/object file split, where things in the header files are "inlined into the user", and things in the object files are not.

Yes. In C++ they had this split from the day one — that's why it worked. Rust had no such split, thus it wouldn't work.

> And that's why I want to offer that - I want library authors (who presumably understand the library code) to be able to decide what is inlined into the calling binary, and what is put in the library's binary, with tooling support for stopping them from accidentally changing something from inlined into the caller's binary to part of the library's binary.

This could be an external addon, sure. But by default things should just work. Otherwise no one would adopt that solution.

Developers are not malicious, but lazy. Take that into account — and that would be enough to develop sane approach.

Provide tools that allow some kind of perfection after you'll provide something that “just works”.

Shared libraries

Posted Nov 26, 2025 14:01 UTC (Wed) by farnz (subscriber, #17727) [Link] (11 responses)

You continue to misrepresent my position; I'm saying that we need the tooling to enable library developers to safely inline parts of their library into the caller, otherwise we're going to see C/C++ "you're holding it wrong" hacks because they're needed for performance.

The compiler cannot do this unaided - how does the compiler know whether a given small function is "hot path" in your users (and therefore needs to be inlined, with tooling assistance to prevent you from breaking the inlined paths), or "cold path" (and therefore can be out-of-line as a symbol in the shared binary)?

If you don't offer this as part of shared library support, then developers are going to hack around it in ways that tooling cannot check, because they're lazy - fixing the compiler to support this is a lot harder than hacking around it in interesting and unsupportable ways.

And the bit you keep ignoring is that I'm saying that we need tooling that's aware of this, and that can error out if you accidentally break your users. That is table stakes for this - otherwise you end up in the C++ situation.

Shared libraries

Posted Nov 26, 2025 14:15 UTC (Wed) by khim (subscriber, #9252) [Link] (10 responses)

> You continue to misrepresent my position; I'm saying that we need the tooling to enable library developers to safely inline parts of their library into the caller,

Why?

> otherwise we're going to see C/C++ "you're holding it wrong" hacks because they're needed for performance.

Why that doesn't happen with bazillion other languages that offer similar capabilities?

> If you don't offer this as part of shared library support, then developers are going to hack around it in ways that tooling cannot check, because they're lazy - fixing the compiler to support this is a lot harder than hacking around it in interesting and unsupportable ways.

Not exposing “lightweight” objects in way that requires such hacks is even easier.

We are talking shared libraries that are used to offer some kind of stable ABI here. Supporting stable ABI is hard. People wouldn't add crazy hack where they couldn't guarantee if they would work or.

Look on Vulkan as an example of stable ABI that have to be performant, too: no crazy hacks, no inline function, just a bunch of non-opaque structures. With accessors to these structures in static libraries (that you may or may not want to use). Zero support from the compiler, zero magic, guaranteed to work, can be easily replicated in Rust, if needed.

> That is table stakes for this - otherwise you end up in the C++ situation.

Nope, you wouldn't. C++ situation arose because C++ already had that difference between .h and .c from the day one, because it already offered a way to have functions partially inlined in .so and partially in .h.

Don't give developers that capability — and they wouldn't use it. They would work with dynamic dispatch and limitations of dynamic dispatch. It's as simple as that.

Sure, optional availability of such capability wouldn't hurt… but lack of these are not show-stoppers at all. Developers can tolerate slow, they wouldn't tolerate “you need an entirely different API for shared libraries”, they would just link everything statically.

Shared libraries

Posted Nov 26, 2025 15:28 UTC (Wed) by farnz (subscriber, #17727) [Link] (8 responses)

Because these hacks do happen with other languages - the exceptions are those with a JIT, where it doesn't matter. And compiler developers just have to get used to this - like they've been forced to with C and C++. Either you provide the tooling to do a good job, or people come up with hacks.

And if you don't allow exposure of lightweight objects from a library, then you're ruling out such things as Option<&str> in the library's ABI surface. That's a lightweight object being exposed - and thus needs a stable ABI.

Vulkan's a really good example, because it does expose all sorts of things that get inlined into the caller - there's inline functions, for example, and those struct definitions are inlined into the caller, such that Vulkan cannot redefine the structures to a different size and still work with older libraries.

In an ideal world, there would be tooling that told people working on Vulkan "hey, you've changed this structure in such a way that it now has different size/alignment. You can't do that!", so that they couldn't make that mistake - but as it is, the normal C rules apply of "you're holding it wrong".

Shared libraries

Posted Nov 26, 2025 15:40 UTC (Wed) by khim (subscriber, #9252) [Link] (7 responses)

> In an ideal world, there would be tooling that told people working on Vulkan "hey, you've changed this structure in such a way that it now has different size/alignment. You can't do that!"

That tool is called Libabigail. And yes, it's used when Vulkan is upgraded.

> That's a lightweight object being exposed - and thus needs a stable ABI.

It just needs stable layout. But yes, it's an interesting dilemma there: if you just freeze the layout someone still needs to ensure that said Option<&str> would be valid when it crosses library boundary.

Whether this needs to be exposed to developers as some kind of universally accessible tooling or kept to the standard library is an interesting question. Today there are a lot of things that standard library does yet regular programs can not do. Providing inlining for methods of Option<str> without giving such ability to third-party library could be a good first step.

After all it's not possible to take self: Foo<Bar<Self>> today as target for the method, but only self: Rc<Self> or self: Arc<Self>… and sky haven't fallen on earth.

Shared libraries

Posted Nov 26, 2025 16:01 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

> It just needs stable layout. But yes, it's an interesting dilemma there: if you just freeze the layout someone still needs to ensure that said Option<&str> would be valid when it crosses library boundary.

So you allow Rust to use the "extern" keyword. Which freezes the layout according to certain rules.

One likely mechanism is to require (a) that extern tells the compiler to use a simplistic layout as declared (no optimisation) so it's guaranteed to be consistent across programs. And then (b) (copying Rust "Editions") you allow "extern "version"" so you CAN update the layout, and more importantly, you can tell the Rust compiler about both layouts so it can auto-generate a shim, or you can choose not to, so it generates a linking error.

But either way, you do have to say "extern" is similar to "unsafe", in that you are relying on the programmer to enforce invariants, but that the compiler will do its job properly if the programmer does theirs (primarily ensures version is updated correctly).

Cheers,
Wol

Shared libraries

Posted Nov 26, 2025 16:11 UTC (Wed) by khim (subscriber, #9252) [Link] (1 responses)

> So you allow Rust to use the "extern" keyword. Which freezes the layout according to certain rules.

It's not enough to just freeze a layout. Option<&str> contains either None or reference to a valid UTF-8 string. Nothing else is allowed. If handling of it is inlined into both binary and library then they should agree about upholding such invariants.

> But either way, you do have to say "extern" is similar to "unsafe", in that you are relying on the programmer to enforce invariants

This would never work. The big advantage of Rust is the fact that compiler (and, most notably, not a developer) upholds many such invariants.

We need this guarantee kept across shared libraries boundary or it'll never work.

Shared libraries

Posted Nov 26, 2025 16:55 UTC (Wed) by Wol (subscriber, #4433) [Link]

> We need this guarantee kept across shared libraries boundary or it'll never work.

Which is farnz' point. You need tooling.

Which is the point of extern. It says this is where it's likely to go wrong. It tells the compiler "here is a boundary where abstractions mustn't leak". It tells the programmer "here is a boundary where you need to be careful what goes across".

It's a major improvement on C / C++ where the header file is part of the library, but contains loads of stuff that ends up in the application binary - a major abstraction leak that C / C++ sprinkles heavily with "here be dragons" pixie dust. A Rust compiler could cleanly enforce most of that, for example by ensuring structs passed through an extern have to be declared as extern, and have the same version as the function they are passing through.

Yes it's not perfect. But it's a damn sight better than what we have in the C ecosystem. Given Rust's concern about pointer provenance and stuff, you could use the same approach to argument provenance across an extern. Doesn't stop a programmer cheating and feeding two different versions of the same header file into the two different halves - application and library - in order to fool the compiler - but that's a clear breach of his obligation to enforce integrity across that boundary.

You might even be able to get the compiler (when compiling a library) to output an Intermediate Language Description of the call interface, which the compiler (when compiling an application) imports to guarantee compatibility. Again, you're then heavily dependent on versioning, and the programmer doing it properly, but it'll only take a few crashes on linking as attempts to do so cause problems, and the programmers will learn to "get it right".

Cheers,
Wol

Shared libraries

Posted Nov 26, 2025 16:50 UTC (Wed) by farnz (subscriber, #17727) [Link] (3 responses)

Libabigail is part of the tooling I'd like to see in proper shared library tooling - it covers size and alignment, but not field ordering within the structure (which also ought to be checked by tools). And, of course, if you want something that "just works", you need the size and alignment of structures to not be hard-coded in the user, but rather filled in by the dynamic linker via a COPY or SIZE relocation (causing you to have to do arithmetic all over the place when a structure from the dynamic library is embedded in a user-supplied structure).

Basically, what I want is tooling that tells me if I've made any change that breaks either my API (which currently exists) or my ABI, and that I can have in my build process to tell me when I've made a mistake. And, as Vulkan shows, this is not just about having a way to express "this item has a fixed layout regardless of compiler", but also "you said this was supposed to be unchanging, but you've changed it in a way that's significant. Did you mean to do that?", complete with the ability to reserve space for future expansion (and not in the C-like hacky way of unused fields, but a very deliberate and well-thought through way to reserve sufficient space with good enough alignment for what you might want to fill in that space with, allowing for some fields being fixed in location because they're exposed ABI, and others being mobile because they're opaque to callers).

And yes, this tooling needs co-operation with the compiler - that's table stakes for doing a stable ABI properly. But it also needs a bunch of other things, so that I have to tell my tools that yes, I mean to break the stable ABI here, just as I have to tell my tools (via SemVer changes) that yes, I mean to break the stable API.

Shared libraries

Posted Nov 26, 2025 17:04 UTC (Wed) by khim (subscriber, #9252) [Link] (2 responses)

> And, of course, if you want something that "just works", you need the size and alignment of structures to not be hard-coded in the user, but rather filled in by the dynamic linker via a COPY or SIZE relocation (causing you to have to do arithmetic all over the place when a structure from the dynamic library is embedded in a user-supplied structure).

It's not “of course”, but “do we really need it?” If structure has a potential of changing size then it's highly unlikely then it's “trivial” structure that would be used somewhere in the critical path. More likely it's some descriptor that needs some non-trivial work to handle it. Swift handles such structures with runtime descriptors and it may be enough to do that.

I would say that if you would try to do that you'll end up with “success” of OSI protocols: lots of hype, lots of talks, zero working implementations.

I would rather see that being marked as “maybe in 10 years if we would have the resources we might attempt that” rather then “that's something that we would use as justification not to do anything”.

> But it also needs a bunch of other things

It needs some things, yes. You are trying to stuff it full of things that are not, strictly speaking, needed, but only desired. That's willful ignorance of RFC1925: It is always possible to aglutenate multiple separate problems into a single complex interdependent solution. In most cases this is a bad idea.

I would say that starting with dynamic dispatch everywhere but some “blessed” types from the standard library (which is essentially what Swift did) would get us 90% there (if not 99% there) and the remaining 1% may be discussed later (or handled with hacky solutions, if needed).

But yes, basic library types like Option<&str> needs to be covered, somehow.

Shared libraries

Posted Nov 26, 2025 17:14 UTC (Wed) by farnz (subscriber, #17727) [Link] (1 responses)

Runtime descriptors get filled in by the dynamic linker - and all of the things I'm describing are things that are done semi-manually to maintain a stable ABI by Vulkan and glibc developers today.

I'm not arguing that these are required for a minimal solution - but rather that all of these problems have to be solved if you're going to have a stable ABI, and given that Rust needs tooling updates anyway to have a stable ABI, it would be better to have tooling that prevents accidental ABI breakage from day 1 of the Rust stable ABI for dynamic linking, rather than (as you seem to be suggesting) relying on "programmers don't make that mistake".

After all, the lesson of C++ UB is that "programmers don't make that mistake" is virtually always false.

Shared libraries

Posted Nov 26, 2025 17:23 UTC (Wed) by khim (subscriber, #9252) [Link]

> Runtime descriptors get filled in by the dynamic linker

Nope. Not even remotely close. Rather it's extension of current dyn Trait scheme: in the vtable there are size of the object (so it can be allocated on stack, alloca-style), reference to the constructor (so you can create a vector) and so on. Then generic may have one, common code for that thing and work — even without any dynamic linkage involved.

Very natural extension of the Rust language, not special code for the dynamic libraries. It would be well-received even without dynamic libraries, the ability to have true polymorphic callback is something people desire often.

> After all, the lesson of C++ UB is that "programmers don't make that mistake" is virtually always false.

Yes, but that doesn't mean we have to invent something grandiose, that would require next 10 or 20 years to develop. Simple solution may be better, because it's actually implementable… and we know, from Swift example, what can be implemented in the limited time.

Hint: anything that requires changes to the dynamic linker automatically makes the whole thing DOA. Simply because on Windows dynamic linker is part of the OS and if this scheme wouldn't support Windows then nobody serious would use it.

Shared libraries

Posted Nov 26, 2025 16:09 UTC (Wed) by excors (subscriber, #95769) [Link]

> Look on Vulkan as an example of stable ABI that have to be performant, too

Vulkan's ABI is stable and extensible and C-based (allowing FFI from many languages), with the tradeoff that it's painful and bug-prone to use directly.

That's only acceptable because most application developers won't use it directly. They'll use something like https://github.com/KhronosGroup/Vulkan-Hpp which provides an easier-to-use and safer C++ API, but gives no promises about API stability. That's a header-only library, so effectively it's statically linked into every application. (Header-only libraries are popular because C++'s ABI support is so poor that even static linking of separately-compiled libraries is unreliable). Or they'll use some other wrapper or engine, or write their own, with the same issues.

So now the problem is: how could you implement something like Vulkan-Hpp as a shared library with a stable ABI? And the current answer is, you can't. Vulkan isn't solving the problem of safe ABIs; it's just shifting it to another layer which doesn't solve it either.

If you want shared libraries and safety (which I think was the premise of this thread), it seems worth trying to come up with a better solution than that.

(Vulkan is also a peculiar example because it has a complex architecture where applications link (dynamically or sometimes statically) to a Vulkan Loader which dynamically links to multiple independent driver implementations, and drivers export the whole Vulkan ABI through a GetProcAddr interface, and applications call a trampoline function in the Loader that dispatches to the appropriate driver. (If you have multiple GPUs, the application might use multiple drivers at once). Applications can avoid the trampoline cost by using a GetProcAddr to get a function pointer directly into the driver, though it may actually be a function pointer into a separate Layer library that can intercept and modify each call before the driver sees it (for debugging etc). I wouldn't call that "no crazy hacks". (https://github.com/KhronosGroup/Vulkan-Loader/blob/main/d...))

Shared libraries

Posted Nov 25, 2025 17:59 UTC (Tue) by paulj (subscriber, #341) [Link] (6 responses)

Java elides the parameterised type information from the binary ABI I thought. AFAIK linking to a jar won't catch an incompatible change - I'd have to experiment a bit to be sure. So... ICBW, or otherwise perhaps you're talking about a slightly different point to me.

Shared libraries

Posted Nov 25, 2025 18:55 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> Java elides the parameterised type information from the binary ABI I thought.

It doesn't. And the type information is available throught reflection. It simply ignores it during the bytecode interpretation (or JIT-compilation).

Shared libraries

Posted Nov 26, 2025 10:44 UTC (Wed) by paulj (subscriber, #341) [Link] (4 responses)

Interesting... Hadn't considered reflection. But that's not part of the binary ABI as such, but the bolt-on reflection API. It's the equivalent of additional debug symbols - useful for introspection, but not used for link-time validation. What you say confirms my understanding, the runtime byte-code linking process does not take parameterised types into account.

Shared libraries

Posted Nov 26, 2025 13:33 UTC (Wed) by khim (subscriber, #9252) [Link]

It's implementation detail. Developers don't care about whether something is “proper” part of the ABI or not “proper” part of the ABI. They can and use foreign types with generics and this just works.

Rust may do the same if it would impose that foreign types can only be used with the dynamic dispatch.

Shared libraries

Posted Nov 26, 2025 18:19 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> Interesting... Hadn't considered reflection. But that's not part of the binary ABI as such, but the bolt-on reflection API. It's the equivalent of additional debug symbols - useful for introspection, but not used for link-time validation.

Technically, there is no "linking" in Java. All the code is dynamically loaded at runtime (through ClassLoaders). You can do additional validation during loading and/or rewrite ("instrument") the bytecode entirely.

Type information available through reflection was used to implement reified generics by some languages targeting the JVM. Kotlin is one of them, for example. So in practice, the type erasure on the bytecode level is mostly an implementation detail.

Shared libraries

Posted Nov 27, 2025 16:54 UTC (Thu) by ssmith32 (subscriber, #72404) [Link] (1 responses)

Oh no. I've gotten bit by type erasure (of generics) more times then I can remember.

You can definitely have code that does:

for( var a: someCollection<SomeTypeA> )

And it ends up with someCollection<SomeTypeB> at runtime, and you're stuck debugging the damnedest runtime error, once you do something with it in the for loop..

I had this happen on a nested, templated collection.

Granted, the thing that usually triggers this is the odd corner case of deserializing data back into Java.

Unfortunately, this odd corner is something that happens all over the place when you're working with a system that contains HTTP services, which, if you work with Java, is what you're usually paid to do, nowadays.

And, since you always need some sort of deserializer for JSON (or, if you're particularly unlucky, XML) in your HTTP service, oh, what the heck, we'll use it for config files as well, for any data we read in, for..

And Boom.

When working with Java on a daily basis, type erasure is anything but an implementation detail.

It's is a very real pain point (granted it was done by smart people with sound reasoning and good intentions [1], but you know what they say about good intentions...)

There are other cases I've ran into at odd times, but I have trouble recalling the specifics, because the serde issue happens often enough that it looms large in the memory.

[1] https://openjdk.org/projects/valhalla/design-notes/in-def...

Shared libraries

Posted Nov 28, 2025 5:31 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

Oh yes. Back when I was writing tons of Java, I had my own deserializer for my homegrown RPC protocol that made sure that types of collections match.

I also distinctly remember collection wrappers that enforced the correct types using reflection. Maybe in apache-commons?

Shared libraries

Posted Dec 1, 2025 23:30 UTC (Mon) by NYKevin (subscriber, #129325) [Link]

There is not. You would need to remove or severely restrict all of the following:

* References (lifetimes are considered type parameters), but maybe we can make an exception since lifetime parameters do not result in true monomorphization.
* Slices and arrays (the element is considered a type parameter, and gives rise to monomorphization in the usual fashion).
* Async (Future<T>)
* for loops and iteration (Iterator<T>)
* Smart pointers (Deref<T>)
* Result<V, E> and Option<T>, and all APIs that return things in one of those two forms (i.e. every fallible function in std and core). That includes basically all I/O.

And probably a dozen other things I didn't think of. Needless to say, the resulting language would be so constrained as to make WUFFS look like a "normal" language in comparison.

Shared libraries

Posted Nov 24, 2025 21:16 UTC (Mon) by zyga (subscriber, #81533) [Link] (15 responses)

This is not quite true. It is "just" insanely expensive to do well.

Apple paid for that support in Swift so that apps for their platforms can benefit from base OS library updates without having to be rebuilt.

Rust and Go didn't have the money or desire to implement that, respectively.

I recommend reading what Swift can do today, on Linux. You can load a library with a type. Load another with a container and efficiently instantiate container specialized with that type, all with dynamic libararies and stable ABIs. It is compiler voodoo but it is not impossible.

I kind of think we are all doomed in the long run (e.g. imagine all of GTK and Qt are written in rust and require a complete world rebuild for every tiny update). IMO that is not scalable and the trend to move to Rust or another langue like that, will bounce at some point.

Either someone steps in and does the heavy lifting to solve this problem, or distributions will just grind down to a halt.

Shared libraries

Posted Nov 24, 2025 21:50 UTC (Mon) by zyga (subscriber, #81533) [Link] (3 responses)

Replying to myself. Apologies for not posting this above but it is well worth reading. "How Swift Achieved Dynamic Linking Where Rust Couldn't": https://faultlore.com/blah/swift-abi/

Shared libraries

Posted Nov 25, 2025 12:17 UTC (Tue) by paulj (subscriber, #341) [Link] (1 responses)

I think that's pretty much exactly what keithp had in mind with "To avoid just stuffing everything in boxes and using virtual dispatch for all operations". I read his comment as fairly explicitly referring to the "You can have (strong ABI AND expensive runtime) OR (limited dynlib ABI AND compile time concrete implementation of parametric types to obtain fast runtime)" dichotomy.

Shared libraries

Posted Nov 25, 2025 13:40 UTC (Tue) by khim (subscriber, #9252) [Link]

You can have both (just not at the same time). That's what Swift does.

Shared libraries

Posted Nov 27, 2025 10:15 UTC (Thu) by Karellen (subscriber, #67644) [Link]

Great link, thanks for posting!

Swift vs Rust ABI

Posted Nov 24, 2025 23:15 UTC (Mon) by DemiMarie (subscriber, #164188) [Link] (5 responses)

Part of how Swift achieved ABI stability is to implicitly heap-allocate data in certain situations. I don’t believe this is permitted in Rust. In particular, Rust doesn’t require that a heap even exists.

Swift vs Rust ABI

Posted Nov 25, 2025 2:11 UTC (Tue) by khim (subscriber, #9252) [Link] (4 responses)

That can be solved by declaring that thing an “std-only” feature.

There's nothing impossible there, but it's a lot of work—means it's unlikely to happen without serious funding… who can provide it?

Swift vs Rust ABI

Posted Nov 25, 2025 6:52 UTC (Tue) by josh (subscriber, #17465) [Link] (3 responses)

It's not just funding, because we've got a strong interest in trying it. It's design, because the Swift design is a good source of inspiration but not *exactly* what would work for Rust, in various different ways.

We're working on it, though.

Good to know that Rust is working on ABI!

Posted Nov 27, 2025 18:23 UTC (Thu) by DemiMarie (subscriber, #164188) [Link]

I’m glad to read this!

Swift vs Rust ABI

Posted Nov 28, 2025 9:51 UTC (Fri) by farnz (subscriber, #17727) [Link]

It's a really hard problem to get right, and I am extremely grateful that the Rust team is thinking it through, and not getting stuck with the first design that solves 99% of the problem space.

Swift vs Rust ABI

Posted Nov 28, 2025 10:26 UTC (Fri) by bluca (subscriber, #118303) [Link]

Excellent news, looking forward to learn more about these developments

Shared libraries

Posted Nov 25, 2025 9:01 UTC (Tue) by taladar (subscriber, #68407) [Link] (3 responses)

I mean lets be honest, if Qt were a Rust library it would also be much, much smaller because it wouldn't re-implement an entire ecosystem of dependencies inside the library itself the way it currently does in C++.

Shared libraries

Posted Nov 25, 2025 10:25 UTC (Tue) by intelfx (subscriber, #130118) [Link] (2 responses)

> I mean lets be honest, if Qt were a Rust library it would also be much, much smaller because it wouldn't re-implement an entire ecosystem of dependencies inside the library itself the way it currently does in C++.

It would have been smaller in source code, but not in binary, for obvious reasons: it might not need to reimplement an ecosystem of dependencies, but the object code generated from those dependencies would still have to exist somewhere.

Unless, of course, it was a hypothetical *shared* Rust library, linking to *shared* Rust libraries of those dependencies. Right.

Shared libraries

Posted Nov 26, 2025 16:25 UTC (Wed) by tialaramex (subscriber, #21167) [Link] (1 responses)

Qt is a "kitchen sink" library which tries to offer everything you might need, so it's hard to compare, maybe a Rust Qt-like would insist on offering all these container types Rust already provides for example, just as Qt feels the need to supply QList (the growable array type) QStack (the same type, with a more stack flavoured API) and QQueue (the same type again AFAIK but now used to back a ring buffer abstraction presented as a FIFO Queue)

C++ doesn't provide very good container types, but it does provide a reasonably good growable array type, with the expected amortized constant time growth, the basic reservation feature - and QList isn't actually better. So the fact QList was provided anyway suggests the same mindset would apply in Rust for a library in that mould.

But on the other hand, a library which didn't insist on offering its own version of things you already have might actually reduce the redundancy and the extra marshalling overhead from the duplication, so that might actually be a net win in size terms.

Shared libraries

Posted Nov 26, 2025 17:19 UTC (Wed) by excors (subscriber, #95769) [Link]

> C++ doesn't provide very good container types, but it does provide a reasonably good growable array type, with the expected amortized constant time growth, the basic reservation feature - and QList isn't actually better. So the fact QList was provided anyway suggests the same mindset would apply in Rust for a library in that mould.

Back in 1998 they explained: (https://github.com/KDE/qt1/blob/25d30943816da9c28cded9ac7...)

> Qt's containers are much more portable than the STL is. When we started writing Qt, STL was far away in the future, and when we released Qt 1.0, no widely-used compilers could compile the STL. For a library such as Qt, it is of course very important to compile on the widest possible variety of compilers.

They also justified it by having different space/time tradeoffs to STL containers (and still make that argument in today's documentation); but if the STL had been widely available with decent-quality implementations when they started, I imagine they would have simply used that and put up with its other limitations. And anyone starting today in Rust would simply use Rust's std containers.

Shared libraries

Posted Nov 26, 2025 1:58 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

> I kind of think we are all doomed in the long run (e.g. imagine all of GTK and Qt are written in rust and require a complete world rebuild for every tiny update). IMO that is not scalable and the trend to move to Rust or another langue like that, will bounce at some point.

GTK has language-independent object descriptions (GObject). I don't see why that couldn't be provided from a Rust core as well. Qt relies more on C++ magic and would be harder translation, but a QML interface could be provided from a Rust core.

Also, I feel that Rust implementations would have more, smaller crates to provide "all" of these UI libraries/frameworks than we have today, so you might actually "win" if something can depend on a smaller footprint in the first place.

Shared libraries

Posted Nov 25, 2025 14:23 UTC (Tue) by gspr (subscriber, #91542) [Link] (10 responses)

Not knowing much about this, I've always wondered why a relatively closed set of packages – like those that constitute a distro – couldn't be transitively monomorphized *internally* in the set.

For example, take the directed graph of dependencies between Rust packages in Debian. Pick any package that is not a library (i.e. not a librust-foo-dev package). This package surely uses, in its dependencies, either monomorphized versions of functions and types, or dynamic dispatch. Note down all the monomorphized versions, and add them to a list for each dependency. Traverse the graph in topological order, and build these monomorphization lists for all dependencies. Then build all library packages as shared objects with all of those monomorphic instances explicitly stamped out (I understand there's no compiler support for this at the moment, but it shouldn't be too hard to fake it by generating stubs?). Will this not allow dynamic linking and bug-fixing in shared objects *within* Debian at least? For a given compiler version, of course. Non-Debian software that uses the libraries are no better off than before (unless they happen to need the same monomorphizations), but they're also no worse off.

I'm sure I'm overlooking something here, but I'd love to learn :)

Shared libraries

Posted Nov 25, 2025 14:39 UTC (Tue) by farnz (subscriber, #17727) [Link] (8 responses)

The problem comes with updates. If you update (say) ripgrep to fix a bug, and it uses a new monomorphization, that new monomorphization can rely on a new monomorphization inside a library package, and so on.

You end up with the same problem as the rebuild problem, since you cannot determine ahead of time that no bug fixes will involve a new monomorphization. You will probably reduce the number of total rebuilds you need, but if you're unlucky, you won't.

Shared libraries

Posted Nov 25, 2025 14:43 UTC (Tue) by gspr (subscriber, #91542) [Link] (6 responses)

> The problem comes with updates. If you update (say) ripgrep to fix a bug, and it uses a new monomorphization, that new monomorphization can rely on a new monomorphization inside a library package, and so on.

Is that likely? Or, is it any more more likely than, say, a bugfix in a classical C library needing to break the ABI?

Shared libraries

Posted Nov 25, 2025 14:46 UTC (Tue) by farnz (subscriber, #17727) [Link] (4 responses)

If you're doing the change downstream, then yes it is quite likely - something as "trivial" to upstream as "add a new variant to an error enum" is a new monomorphization, with the resulting need to recompile everything that knows the layout of that enum.

Shared libraries

Posted Nov 25, 2025 14:51 UTC (Tue) by gspr (subscriber, #91542) [Link] (3 responses)

> If you're doing the change downstream, then yes it is quite likely - something as "trivial" to upstream as "add a new variant to an error enum" is a new monomorphization, with the resulting need to recompile everything that knows the layout of that enum.

Definitely. But a similar change in a classical C library would be to return a new error value. That wouldn't technically break the ABI, but it would sure require depending packages to acquire knowledge of the new error value. That would take *more* than just recompiling.

I guess what I'm saying is that this approach doesn't always work, but it's not much worse than the situation for classical C libraries.

Shared libraries

Posted Nov 25, 2025 14:59 UTC (Tue) by farnz (subscriber, #17727) [Link] (2 responses)

Returning a new error value that was previously impossible is an ABI break, in both C and Rust, unless it's clearly documented beforehand that other errors are possible.

For example, if I truncate the error value to 8 bits to make it fit an existing struct, because all known error values are under 255, and you introduce error value 256, I've got a problem in C. This gets worse in Rust, because enums aren't just a value, they can carry data, too, so the enum may get larger as a result of the change, and upstream won't care that the old enum compiled by Debian was 72 bytes, and the new one is 80 bytes - especially if compiled with a newer compiler, they're both 64 bytes.

Shared libraries

Posted Nov 25, 2025 15:34 UTC (Tue) by gspr (subscriber, #91542) [Link] (1 responses)

Ok, sure, we have an ABI break for both C and Rust. I'm just saying: the situation doesn't get worse, does it?

Shared libraries

Posted Nov 25, 2025 15:42 UTC (Tue) by farnz (subscriber, #17727) [Link]

It's slightly worse, because years of habit mean that C programmers are used to thinking about whether a change will break distro ABIs; Rust programmers aren't, and because Rust makes it easier to express things that are hard to express in C (sum types, for example), they're more likely to make changes to fix a bug that make the situation worse.

Remember that the state we're in with C is in part because the language requires programmers to get it right, or risk UB, and as a result, C programmers doing security fixes tend to be thinking about all the ways they can accidentally break someone; Rust programmers tend not to be doing that, because the result of breaking someone is a compiler error, not UB.

That cultural difference matters, and is part of why the aim on the Rust side is to have a state where swapping in an incompatible dynamic library is a dynamic linker failure, not UB as it is in C.

Shared libraries

Posted Nov 25, 2025 17:16 UTC (Tue) by ssokolow (guest, #94568) [Link]

Rust has automatic struct packing and niche optimization and, were it not for the need for reproducible builds, the support for randomizing struct layouts to help people catch Hyrum's law mistakes might be on by default.

Shared libraries

Posted Nov 25, 2025 17:05 UTC (Tue) by Wol (subscriber, #4433) [Link]

Or you go back to the old FORTRAN libraries that I worked with. The linker pulled in all the modules it knew it needed (and if it had recursive dependencies you had to link the same library several times to get them all). That also had the side effect that all the functions that your program didn't need didn't end up in the executable.

So we then have some fancy update tool (sorry) that takes two allegedly identical libraries, and goes through merging all the different modules into a new updated library. If it finds the same module in both precursor libraries, it would need to check and enforce the rule that the extern definition was identical, before choosing the version with the newest reference number (maybe defined as the most recent modified date - I think that might be a tricky problem?). Or maybe just updates the original library with new modules that didn't originally exist.

Cheers,
Wol

Whole-image optimization

Posted Nov 27, 2025 22:48 UTC (Thu) by DemiMarie (subscriber, #164188) [Link]

I think this could work if one is shipping an image that is updated by replacing it.

Shared libraries

Posted Nov 25, 2025 20:46 UTC (Tue) by jhoblitt (subscriber, #77733) [Link] (4 responses)

The reality is that the burden of rolling out security fixes is shifting from distros to upstream projects. So instead of a shared lib getting updated at the distro level, the upstream(s) tag a new release with the version dep(s) bumped. This isn't a big deal for most upstreams as odds are, a bot will automatically open a pull/merge request when a dep patches a security issue.

What is probably needed to make this new model palletted for distros is standardized change log data that the distros can poll looking for security updates.

Shared libraries

Posted Nov 25, 2025 20:51 UTC (Tue) by mb (subscriber, #50428) [Link] (3 responses)

That's not what actually happens, though.

Distros don't care about locked upstream dependencies.
And I think that's fine, if they always pick the latest compatible semver.
That tends to work very well in the crates.io ecosystem with only extremely few exceptions.

But distros often try to use older dependencies than locked upstream. Which is not Ok, IMO.

Shared libraries

Posted Nov 25, 2025 22:35 UTC (Tue) by jhoblitt (subscriber, #77733) [Link] (2 responses)

I don't dispute that distros override Cargo.lock files but I don't agree that behavior is a feature. Changing a single dep version can also result in changes to transitive deps and my preference is that I am running the exact version of a software package which passed the upstream project's CI pipeline.

Shared libraries

Posted Nov 26, 2025 8:22 UTC (Wed) by joib (subscriber, #8541) [Link] (1 responses)

Case in point being Debian breaking bcachefs-tools by changing one of the deps to an older one.

Maybe there's a position in between "only one version of each library" and "whatever is currently in Cargo.lock across a hundred different projects"? E.g. some algorithm that could calculate the minimum set of versions to satisfy all the version requirements in all the Cargo.toml files that are used in the distro?

Shared libraries

Posted Nov 26, 2025 8:59 UTC (Wed) by zdzichu (subscriber, #17118) [Link]

Debian breakage should be caught by unit tests during the package build. But last time I've checked, there were no such tests in bcachefs-tools. Not exactly evoking confidence in its code.