Defining the Rust 2024 edition
Defining the Rust 2024 edition
Posted Jan 30, 2024 14:24 UTC (Tue) by farnz (subscriber, #17727)In reply to: Defining the Rust 2024 edition by gspr
Parent article: Defining the Rust 2024 edition
It's also worth noting, in this context, that dynamic linking with a stable ABI for C code the way we do it on Linux is a historical accident, and not something that was carefully planned with a view to programming languages other than FORTRAN and C
Sun, in the late 1980s, chose to develop their dynamic linking technology (which was later adopted into ELF) as a way of simply delaying the final link stage. Thus, the decision was made that only things that the SunOS implementation of ld handled - basically just symbol lookup and nothing else - would be part of the dynamic linker's remit. Everything else would be exactly the same as if you were building a static binary using the traditional SunOS toolchain, and thus you did not need to consider the ABI of your shared library; if you had a stable C (or FORTRAN) API, the SunOS 4.0 dynamic linker would automatically give you a shared library with a stable ABI, and any instability in the ABI would be traceable to a change in the API. However, you must then ensure that the API is completely stable from the point of view of ld - any changes, no matter how minor, immediately break the stability of the DLL.
This contrasts to the approach other systems (notably Windows) took; in the alternative approach, you have to explicitly label which parts of your API are part of your stable ABI for your dynamic library, and instability in your ABI is a consequence of you deciding to change the stable ABI. In this approach, you can supply statically-linked code around your ABI that supports API changes, but the stable ABI must stay stable.
We later adopted symbol versioning, and symbol visibility, so that library authors could provide a stable ABI under the ELF rules, but we kept the SunOS thing of "your DLL ABI is deduced from the objects you present to the final link stage" rather than switching to the Windows thing of "your DLL ABI is defined based on the things that you expect to treat as stable". If we'd switched to the latter at any point (thus had a requirement for a curated "module definition" to define a shared object, instead of deducing what should be exposed based on the C source code), then we would be headed to a very different place, since we'd be expecting languages taking part in dynamic linking (including C) to define what parts of their API are stable ABIs, and we would not have the situation we have today where it's possible to dynamically link against parts of a C API that aren't stable, while not possible to dynamically link against parts of a stable C++ API that happen to be inexpressible in C after the preprocessor has run.
Posted Jan 30, 2024 15:22 UTC (Tue)
by smcv (subscriber, #53363)
[Link] (6 responses)
It's possible to do this in a way that works with GNU or clang ELF toolchains: for example, GLib and GTK do this. Briefly, you compile and link with -fvisibility=hidden on ELF platforms, and then decorate each public/stable symbol declaration with a macro that expands to __attribute__((visibility("default"))) on ELF platforms. In portable code, it's convenient for the same macro to expand to __declspec(dllexport) or __declspec(dllimport) (as appropriate) on Windows.
I agree that this is much less obvious than it would ideally be, with a lot of things that the library maintainer is expected to "just know".
Posted Jan 30, 2024 16:12 UTC (Tue)
by farnz (subscriber, #17727)
[Link] (5 responses)
It's possible, but it's not the default, and it's a lot of work for the library developers. Most of the stuff we dynamically link is stuff like zlib, where we rely on the fact that ELF dynamic linking and ELF static linking work in almost the same way, just at different times.
Your brief guide misses the other thing, which is that you need symbol versioning as well - yet another bit of build complexity to add versions to symbols, and to deliberately handle cases where you need to change your stable ABI without breaking old users. If we'd gone for deliberate module definitions (as Windows used to require, but I believe has moved away from to closer match the way Unix systems handle dynamic linking), then this would be required if you wanted to be dynamically linked at all.
Posted Jan 30, 2024 17:26 UTC (Tue)
by josh (subscriber, #17465)
[Link] (3 responses)
And that turns out to be a monumental pain for anyone who wants to use a different zlib. Symbol conflicts between, for instance, zlib and zlib-ng-compat are painful.
The entire concept of symbol interposition and a global symbol namespace are *terrible*, and it's unfortunate that we all pay the cost for them just because people like the ability to override symbols via LD_PRELOAD.
Posted Jan 30, 2024 18:54 UTC (Tue)
by farnz (subscriber, #17727)
[Link] (2 responses)
Indeed.
If you look at this historically, the dynamic linking model we mostly use is from SunOS 4.0 in 1998, where all symbols not explicitly hidden from the dynamic linker are linked at run time, not at compile time, and the purpose of the dynamic linker is to defer symbol resolution from compile time to run time. It was not meant as a stable ABI mechanism; rather, the assumption was that you'd only supply dynamic libraries where you already had a stable ABI for your static libraries. Later, in the mid 1990s, Solaris 2.5 added symbol versioning, since it was clear that even if you had a stable ABI for your libraries, you'd run into cases where people depended on the implementation, not the defined behaviour, and you'd need to give people who thought they'd linked against the old version the old behaviour, not the new behaviour.
However, Linux distros have turned everything into dynamic libraries, without considering whether or not the library ABI is meant to be stable; by and large, this has worked out because people have done huge amounts of work to make it work. And a lot of upstreams (take Chromium for an example) have handled this by copying libraries into their source trees and patching them (since they don't have a stable ABI anyway), which now needs more work in distributions to handle de-vendoring these libraries and ensuring that the patch is either upstream, or not needed.
Posted Jan 31, 2024 10:06 UTC (Wed)
by farnz (subscriber, #17727)
[Link] (1 responses)
And I notice my typo - SunOS 4.0 is not 1998, it's 1988 - ten years earlier than I said.
Posted Jan 31, 2024 16:14 UTC (Wed)
by tialaramex (subscriber, #21167)
[Link]
Posted Jan 30, 2024 22:47 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Windows now supports the "throw everything over the wall" approach, but everything native to Windows uses either explicit exports. Or COM objects that are basically completely handled in runtime.
Posted Jan 31, 2024 15:49 UTC (Wed)
by paulj (subscriber, #341)
[Link] (10 responses)
<<we kept the SunOS thing of "your DLL ABI is deduced from the objects you present to the final link stage">>
Is that really the case? We have mapfiles in which you specify your interface(s). Along with versions, with composability of these versioned interfaces though inheritance. Sun and Glibc have been doing so since the 90s. And have been doing a good job. We havn't had to suffer libC "upgrade everything on your computer" flag days since Glibc... 2.0 (?) I think - which IVR was the first glibc to really get shipped by distros (there might have been a bleeding edge release of RHL with a pre-release Glibc (1.99?) which then had to suffer a flag day upgrade to 2+ - but it was clearly marked as bleeding edge; I avoided that release anyway and went from Linux libc5 to glibc 2).
[Sun did an even better job with Solaris. Particularly on the process side, ensuring that libraries shipped with Solaris were required to specify their interface stability - both API and ABI. At a minimum, the library would ship with a man page detailing the interfaces had no stability. For anything a customer was to be allowed to have some reliance on, there were other levels of stability - with increased guarantees of stability, came increased requirements to document the interfaces in manual pages and in map files, along with increased levels of reviews to changes internally within Sun.]
Anyway, linker map files with versioned interface descriptions work _really_ well for managing C library compatibility.
For some strange reason many libraries in the open-source world continued to rely on (largely unworkable) SONAME versioning. Indeed, many don't even bother with that.
That's a _social_ problem though, not really a technical problem in the Unix C world. Cause the technical tools were there since the 90s. Sun and Glibc also solved the process and social sides, on top of the tooling.
Posted Jan 31, 2024 15:52 UTC (Wed)
by paulj (subscriber, #341)
[Link] (6 responses)
And end-users, the wider world, we all suffer for it.
Posted Jan 31, 2024 16:20 UTC (Wed)
by farnz (subscriber, #17727)
[Link] (5 responses)
The social problem is worse than that - you have upstreams working in the "wild west" style, and distros quietly doing the work to keep stable interfaces downstream. That work done by distros is then assumed to be an inherent part of the C way of working, and not serious effort by skilled people, and thus languages where no-one is doing that work are assumed to be doing it wrong, whereas in fact all they're doing is exposing the fact that somebody needs to do this work if we're to have stable interfaces.
Posted Jan 31, 2024 16:40 UTC (Wed)
by paulj (subscriber, #341)
[Link] (4 responses)
I think there was actually a moment in tech, in the mid-90s, where a bunch of engineers /did/ think about stability. And they /did/ bring in tools and processes to try tame some of the dependency and stability issues that had been seen. We got tools to at least define C interfaces and manage their stability - even if the default wasn't ideal, as you say (also, unfortunately GNU libtool went with soname versions as the recommended approach rather than symbol versioning - and soname versions just.. doesn't work - I fear this damaged the FOSS side). We got social processes to manage stability, both in some tech companies and in a (v small) number of FOSS projects (Glibc, on top of symbol versioning; Linux, of userspace API and ABI). You could argue Linux distros and packaging tools also played a significant part in trying to manage stability and tame the hell of dependencies without such management.
Sadly, we seem to have lost the drive - collectively - to manage stability.
We see many projects just ignoring tooling and packaging approaches to managing stability, and just... vendoring the fsck out of everything and giving users "wget | bash" scripts. And this has become very normalised. And the more projects do this, the more unstable everything becomes and hence the more other projects do this, because they can't rely on stability of stuff either and have to vendor.
And it's not just FOSS, this happens internally in proprietary corps too. With internal projects sucking in external and internal repos into big blobs. Even checking in .sos into build directories for various platforms and arches cause they can't figure out anything more reasonable. Cause nobody has the time to care about managing stability and dependencies, and the resulting industry wide mess means it's too complicated to even start to figure it out.
But wait, I know the answer, let's ship a script that builds container images that contain our build, so it can build...
It's really awful, and it seems to be getting worse.
Posted Jan 31, 2024 17:27 UTC (Wed)
by farnz (subscriber, #17727)
[Link] (3 responses)
The only part of it that I see as a "C problem" is the idea that C inherently has a stable ABI for dynamic linking, while other languages don't. That's a false statement - it's a historical coincidence that as engineers at Sun build what became ELF dynamic linking, they built something that assumed that if you hadn't put the effort in to first limit symbol exposure (SunOS 4.0) and then version your stable ABI (Solaris 2.5), the "right" thing to do was to expose everything as-if it's a stable ABI, rather than hide everything by default and require you to explicitly expose a stable ABI to dynamic linking.
With engineering discipline, you can work well in the Solaris 2.5 model of a version script indicating exactly what symbols are part of your stable ABI, and what symbols are internal, combined with a static library and source code that provides a "useful" stable API on top of the stable ABI (but where the ABI of the static library and matching source is itself unstable). But we don't do that for any language; Rust, Go, C++ et al merely expose that our "stable ABI" trickery is not engineered in, but something that only exists for C because there's a lot of invisible work done to hide the fact that the libraries don't have a stable ABI by default.
Posted Jan 31, 2024 17:44 UTC (Wed)
by paulj (subscriber, #341)
[Link] (2 responses)
It was a little bit of sound engineering work, and quite manageable.
Really, where we are today, it's largely a cultural thing - we just don't want stability. Or more precisely, we want stability from everyone else's software, but we don't want to held to that ourselves. We - as an industry - are largely a bunch of degens who just want to slap code around, and not be held responsible much later. Either because we work in some big-walled-garden tech corp, and all that matters is throwing enough code around to get past the next performance review; or because we work somewhere where managers are putting pressure on you to get that next hacky-bug-fix or half-arsed-feature-update shipped to the customers, so that you can move onto the next hacky-bug-fix or half-arsed-feature-update.
Sorry... I've become very cynical. But then, I've been watching some of the videos of testimony from the UK Post Office scandal, particularly the software engineer describing the code issues and practices and.. I really am not being cynical, we really are just a completely degenerate industry.
Posted Feb 1, 2024 10:59 UTC (Thu)
by farnz (subscriber, #17727)
[Link]
See also a subset of the fuss around the EU's Cyber Resilience Act; not the people talking to MEPs and national Commissioners about getting the details right (e.g. ensuring that I can't be held liable if you use my software for a purpose I did not advertise it as suitable for), but the people arguing that any form of liability for defects in software will destroy the software industry completely. In effect, they're saying that the software industry (including embedded) cannot exist if software vendors can be held liable for faults in the software they sell.
Posted Feb 1, 2024 16:09 UTC (Thu)
by Hattifnattar (subscriber, #93737)
[Link]
Early on, programming was an elite undertaking, with people practicing it having a mindset closer to scientists. It was elite job, commanded elite compensation (not necessary monetary, esteem too). The "industry" also had been largely driven by people with more-or-less engineer or scientist mindset.
Now software is a real and huge industry, programmer work is commoditized, the whole thing is more and more run by people with "financist" and/or "bureaucrat" mindset, "buccaneer" in the best case.
There are still "islands of quality" in this mess, mostly dealing with foundations. But they cannot completely isolate themselves from the overall dynamics. However, understanding it can can help to draw some boundaries and do better in specific areas/projects etc.
Posted Jan 31, 2024 16:22 UTC (Wed)
by farnz (subscriber, #17727)
[Link]
Yes, it is - like Solaris 2.5, we kept the thing where if you don't supply a version script specifying your interface, we assume that you intended for every single linker-visible symbol in your library to be part of your static ABI.
If the default were the other way round, where libraries that don't carefully specify their exported ABI have no exported ABI at all, it'd be abundantly obvious to everyone that most C libraries don't actually have a stable ABI for dynamic linking, either (glibc an important and notable exception here, and there are others like glib that would specify their exported ABI in this world).
And then the distro thing of "un-vendoring" sources and exporting a suitable "stable ABI" from libraries would be highly visible work, not (as it is today) invisible work where distro packagers put in a lot of effort to make useful stable ABIs for C libraries whose upstreams don't even care about having a stable API, let alone a stable ABI.
As it is, though, we have a set of upstreams who don't care, but where distros can do the work so that as long as you don't look too closely (and especially if you don't look at the work done by distro packagers), it looks a bit like all C libraries have a stable ABI with no effort. Following from this, any language where this work can't be done quietly by the distros, but will need invasive changes to upstream "looks" bad compared to C, because it's assumed that the distros are "just" building binaries, when there's a lot more social work involved around keeping stable interfaces in place.
Posted Jan 31, 2024 16:48 UTC (Wed)
by Wol (subscriber, #4433)
[Link] (1 responses)
Linux didn't UPGRADE to glibc2, it CHANGED to glibc2.
Before that we had libc5, which was a completely different libc. I bang on about WordPerfect, but the native linux WP was a libc5 beast. So I gather you can still force it to run, but it's a lot of work.
Cheers,
Posted Jan 31, 2024 17:10 UTC (Wed)
by paulj (subscriber, #341)
[Link]
Libc.so.5 was a "Upgrade everything on your computer" type libc though. Any major updates to it would have required updating most of your Linux box. That this didn't happen was only because Linux libc lasted only a relative short period of time, as a stop-gap fork of an earlier Glibc 1.
That I and others "changed" from Linux libc to Glibc 2.0, to gain symbol versioned libc - which then meant "upgrade your entire distro" was largely avoided (there was some foible in compatibility with an ancillary library around Glibc 2.2 IVR) - is kind of irrelevant. Cause it would equally have required a complete distro upgrade to have "upgraded" Linux libc to a symbol versioned one. And anyway, Linux libc /was/ GNU libc.
Without symbol versioning, as in the Linux libc days, we would have had to have suffered "flag day" upgrades, updating your entire OS for any ABI fixes to core (heavily depended upon) libraries.
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Defining the Rust 2024 edition
Wol
Defining the Rust 2024 edition