Defining the Rust 2024 edition

Posted Jan 29, 2024 21:37 UTC (Mon) by quotemstr (subscriber, #45331)
In reply to: Defining the Rust 2024 edition by tialaramex
Parent article: Defining the Rust 2024 edition

> But what we tend to see from say C++ people is the fantasy that you can have C++-style comprehensive "everything just works" ABI freeze without the inevitable consequences of poorer performance and inability to fix obvious problems. If you actively _want_ worse performance and problems you can't fix, C++ already has those. They don't seem very happy about it though, "Break the ABI" is almost as popular as "Unspecialize vector<bool>" from the r/cpp crowds for example.

100% correct. Inter-DSO APIs should be specified in explicitly ABI-stable language-neutral API (e.g. gobject), not the mangled artifacts of one's language of choice.

If I could wave a magic wand, I'd outright ban any mangled symbol appearing in a dynamic symbol table.

Defining the Rust 2024 edition

Posted Jan 30, 2024 17:21 UTC (Tue) by josh (subscriber, #17465) [Link] (9 responses)

Much like the C ABI itself, that would force everything to the least common denominator that any language supports.

Defining the Rust 2024 edition

Posted Jan 30, 2024 17:37 UTC (Tue) by quotemstr (subscriber, #45331) [Link]

Only across DSO boundaries. Systems should have far fewer DSOs than they typically today.

Have an ABI stable API? Spell it using gobject or whatever. Mangled names aren't ABI stable anyway (or, for the sake of language flexibility, should not be). Don't need an ABI-stable API? Statically link and gain the benefits of LTO.

Putting ABI-stable high level APIs across DSO boundaries is the worst of both worlds.

Defining the Rust 2024 edition

Posted Jan 31, 2024 13:33 UTC (Wed) by farnz (subscriber, #17727) [Link] (7 responses)

That's the point, though - for a stable ABI, you should be forcing yourself to consider every exported item with care, and deciding whether or not it should be part of the stable ABI, or whether this should be part of the per-language stable API (that might call a stable ABI function, but is statically linked to your program).

After all, this is how dynamically linked libraries already work in C today on ELF systems (just like they did on SunOS 4.0 in 1988); you have a .so that's the DLL itself, and is dynamically linked. You have a .lib that's statically linked, and that mostly tells ld to put references to the DLL in place. And you have .h (and in some edge cases .c) files that the consumer adds to their set of sources and actually builds against. The idea is that the person who creates the DLL has considered what parts of it are sensible to dynamically link, and what parts must be statically linked, and has carefully planned out the stable ABI for the dynamically linked component; distros have kinda broken this model by saying "well, actually, all symbols can be dynamically linked, even if there's no stable ABI, and we will make someone's life miserable if you don't ensure that all symbols from your library have a stable ABI, even if they were ones you never intended to export at all".

Defining the Rust 2024 edition

Posted Jan 31, 2024 14:52 UTC (Wed) by paulj (subscriber, #341) [Link] (5 responses)

Didn't Sun also come up with symbol versioning, and visibility specifications - via linker maps - pretty soon after? It looks like it was introduced in Solaris 2.5 in 1995. The Solaris 2.5 "Linked and Libraries" doc has a chapter on symbol versioning, and how to specify these in the mapfile, which is not in the Solaris 2.4 version.

https://docs.oracle.com/cd/E19641-01/

Glibc has been using symbol versioning for a long time to support multiple incompatible ABIs within the same library - allowing old and new applications to be supported without breakage, without DLL versioning hell.

If you write a library, and you write a map to describe your ABI, and maintain it, C and ELF gives you powerful tools to maintain compatibility, seamlessly. Since the 90s.

Defining the Rust 2024 edition

Posted Jan 31, 2024 15:22 UTC (Wed) by farnz (subscriber, #17727) [Link] (4 responses)

Yes. But the vast majority of libraries don't do that - they are designed to be statically linked and never swapped out, and distributions "cheat" by using one version of the header file and a different version of code, and getting very upset (see also "unvendoring" of sources) when upstreams don't comply with their demand that everything be designed for dynamic linking with a stable ABI.

Looking at the stuff I have installed, glibc is the only library that goes the full mile on providing a stable ABI; several others (glib, for example) clearly limit the exported symbols to a set that they're likely to support, but don't have symbol versioning, but quite a few commonly used libraries (libz for example) look like they just export everything as-if I was supplying them to my code as source.

And this is a large component of the problem; for C, the "correct" way to do it comes from Solaris 2.5 in 1995, where you supply a dynamic library where all symbols are versioned and thus supply a stable ABI, along with a combination of C source and static libraries that supply a stable API implemented atop the stable ABI of the dynamic library. But the "default" is that unless you do the work to do things properly, every non-static symbol in your C library becomes part of your stable ABI whether you like it or not; if the default was flipped round to "nothing is exported unless you supply a version script", then we'd be in a different place for these discussions, since it would be very clear that much of the C ecosystem doesn't actually have a stable ABI either (since you need a stable API to build a stable ABI on top of).

Defining the Rust 2024 edition

Posted Jan 31, 2024 15:36 UTC (Wed) by joib (subscriber, #8541) [Link] (2 responses)

Even if the default of the symbol exporting toggle would be the right way, it's still easy to accidentally introduce ABI breakage, and there's really nothing that fundamentally prevents it (yay, another footgun in the C world, who would have guessed?). Nowadays there are tools like libabigail that help with it, but that's yet another thing the developer needs to be familiar with and systematically use.

Defining the Rust 2024 edition

Posted Jan 31, 2024 16:02 UTC (Wed) by paulj (subscriber, #341) [Link]

A mapfile is not that hard. It only needs to be a few lines initially to just catch all symbols and assign a version.

Also, it was not hard to check and control visibility of symbols even prior to symbol versioning. Checking is just nm. A little sanity-check script to dump your libraries symbols and diff them to what's intended to be there is... trivial. The Solaris 2.4 linker guide has examples on editing linker symbols.

It's just a culture of refusing to use long-standing tooling.

It was not hard in Sun to just learn what you had to do - with the exact same tools as Linux has had since almost same time. What Sun had was technical leadership willing to make legions of engineers bashing out code take a stap back and spend 5 minutes a day checking what symbols they'd added / modified. The FOSS world lacks that leadership.

Defining the Rust 2024 edition

Posted Jan 31, 2024 16:17 UTC (Wed) by farnz (subscriber, #17727) [Link]

Sure, and C relies on documentation and skilled developers to avoid making mistakes, but that's the C culture. At least with the defaults set correctly, it becomes easier to avoid making the mistake - you mark the exported symbols in the source code, and review can catch changes to exported symbols (but not implementation details).

Defining the Rust 2024 edition

Posted Jan 31, 2024 15:56 UTC (Wed) by paulj (subscriber, #341) [Link]

Many FOSS libraries don't bother, indeed. I've actually already replied on that point in the other sub-thread: https://lwn.net/Articles/960252/

;)

I think it's ridiculous. It's not that hard to maintain C interface mapfiles. It's a socio-cultural problem in FOSS - just ignorance for most, but willful, reckless, disdain even just for /managing/ stability in some cases (I've fallen out with a former co-maintainer in small part over this). [note, managing stability could mean no more than just documenting "This bit is stable, all the rest is free-for-all"].

Defining the Rust 2024 edition

Posted Feb 2, 2024 6:51 UTC (Fri) by josh (subscriber, #17465) [Link]

> That's the point, though - for a stable ABI, you should be forcing yourself to consider every exported item with care

That's not the same issue I was raising, though. I *absolutely* think that you should consider every part of your public ABI with care. However, that does *not* mean I want every public ABI to have to use raw pointers, eschew things like Option and Result and String and Vec and HashMap, be unable to provide objects that have methods, be unable to accept parameters that can be anything-that-implements-a-trait, and many many other features.

crabi will solve some of this for cross-language ABIs, and there's another effort to provide a much broader stable ABI surface that's just for Rust-to-Rust calls, but in both cases it'll take a while before we get there. And in the meantime, I don't think it's reasonable to require that people provide a C-only ABI just so it can be a shared library; that'd be a substantial step backwards.