|
|
Subscribe / Log in / New account

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 29, 2025 15:04 UTC (Wed) by bluca (subscriber, #118303)
In reply to: Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ... by farnz
Parent article: Vendoring Go packages by default in Fedora

> The idea was that libraries would carefully curate their ABI, such that users never experienced an ABI break - no "soname bumps" or anything like that - as part of justifying the prices they charge developers to buy access to those libraries. The work distros have backed themselves into doing for all projects is the ABI stabilization work - with a few notable exceptions (glibc, GLib and a small number of others), upstream developers on open source projects don't bother keeping their ABI stable, as long as their APIs are stable enough for their intended consumers.

That's most definitely not true - some libraries are bad, but most are good on that front, and ABI transitions are relatively easy, with few exceptions especially compared to the grand total of number of libraries shipped.
And of course - in case there's a library that is badly maintained and doesn't bother with keeping ABI stable, that is simply a bad library, and not a candidate for being included in the first place

> But the original reason for dynamic linking is simply that consumers cannot recompile proprietary binaries, and want to receive binaries they can run without a "prepare" step. Good tooling can already rebuild all dependent packages on a version change (e.g. Open Build Service does this), and the other reason that people cared about dynamic linking back in the day is that downloading updates over dial-up was already painfully slow with statically linked packages.

That might surely have been the issue originally, but it's not today. Constantly rebuilding everything all the time costs a lot of compute power (hello, climate change anyone?), and while we in the western countries are blessed with symmetric fiber connections with no data caps, that is most definitely not the case everywhere in the world, and in many many countries metered 2g/3g/4g cellurar connections are the only option. And on top of that, disk space still matters in some cases: when building an initrd, you want that as small as possible even on a modern powerful laptop. And you want to save memory and latency by loading the DSO _once_ instead of every time for every program, which matters a lot of low-end/embedded devices.


to post comments

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 29, 2025 15:35 UTC (Wed) by farnz (subscriber, #17727) [Link] (14 responses)

And yet, despite all the costs you claim the distros are trying to avoid, they still do regular complete archive rebuilds to find FTBFS bugs caused by a dependency change (thus spending the compute time anyway), the size of a minimal install has skyrocketed (you used to be able to install Debian from 7 floppies, or 10 MiB, the "minimal" install image they offer is now 650 MiB or 65x larger), and the use of ASLR means that the DSO is no longer loaded "just once", but multiple times, because distros don't (based on looking at a Debian 12 system) carefully arrange their DSOs to minimise the number of pages with relocations in them.

On top of that, you only have to follow a distribution development list to see both announced and unannounced soname bumps, where a package has deliberately broken its ABI, along with bug reports because a package has silently broken its ABI. Distros have even set up language-specific tooling for C to generate ABI change reports, so that they can catch broken ABIs before users notice.

All of this is a lot of work, and it's largely based on tooling distros have built for C. It's not hugely surprising that other languages don't have the tooling yet - it took literal decades to get it built for C, and integrated so transparently into distros that it's now possible for package maintainers to not even be aware that it exists, right up until the package they maintain does a bad thing and the tooling reports bugs to them.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 29, 2025 18:37 UTC (Wed) by bluca (subscriber, #118303) [Link] (13 responses)

ABI transitions happen thorough a development cycle sure, but they are a known quantity, and libraries that get included do not break ABI every 5 minutes, otherwise they'd simply not be included. Some of the C++ ones are the worst offenders, like Boost, but their dev cycle is slow enough that it doesn't make that much of a difference in practice.

The most common C libraries that are used in any base system are very stable. glibc last broke compat what, 20 years ago? A very commonly used one with the most recent change was OpenSSL with version 3.0.
libacl, libpam, various util-linux libs, compression libs, selinux libs - those are all at soname revision 0, 1 or at most 2, and they are decades old.

It does happen of course, and there are processes for it, well-regulated and understood. And slow.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 14:41 UTC (Thu) by taladar (subscriber, #68407) [Link] (10 responses)

So basically what you are saying is that the problem is manageable because most C and C++ libraries don't change much any more anyway.

That isn't really a solution, it just means people avoid change because it has proven to be so painful.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 14:46 UTC (Thu) by josh (subscriber, #17465) [Link]

Exactly. Language packaging ecosystems are capable of handling problems like "I sent a patch to one of my dependencies to add a new API, they merged it and published a new version, and now I'm publishing a new version of my library using that new API", with a turnaround time in minutes in the limiting case. That doesn't mean every such version needs to be packaged by a Linux distribution, but it does set baseline expectations for what a system is capable of when designed for it.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 15:05 UTC (Thu) by bluca (subscriber, #118303) [Link] (8 responses)

No, that is very much not true, and objective data on that is so overwhelming I have no idea why you have to say it. Just check the changelog of glibc or libsystemd or any other system library if you don't believe me. The problem is manageable because developers of said libraries have learnt to do updates, bug fixes and add new features and interfaces without breaking backward compatibility, but apparently this looks like an alien concept in the golang/rustlang ecosystems.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 15:34 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

> but apparently this looks like an alien concept in the golang/rustlang ecosystems.

Because they don't wear the same rose-tinted glasses as you? Because in their world view this particular problem just doesn't exist? (After all, isn't that one of the selling points of Rust - whole classes of major problems in C just disappear because the concept doesn't make sense in the Rust world.)

It's like a bird trying to explain to a fish how you spread your wings to soar in the wind - the fish (one or two excepted) will have no idea what you're talking about.

Cheers,
Wol

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 16:12 UTC (Thu) by bluca (subscriber, #118303) [Link]

Breaking compatibility in public APIs affects all languages and ecosystems. What changes is just who pays the cost. Making sure it's not _developers_ who pay the cost of _developers_ breaking compat doesn't mean it's free.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 17:34 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Debian 8 from 10 years ago had about 43k packages. Debian 12 has about 64k packages. That's not a particularly fast growth.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 17:59 UTC (Thu) by mb (subscriber, #50428) [Link] (4 responses)

>bug fixes and add new features and interfaces without breaking backward compatibility,
>but apparently this looks like an alien concept in the golang/rustlang ecosystems.

Just because Rust chose different solutions than you prefer for the problem of backward compatibility, doesn't mean that they don't care.
Just because somebody else does something different from what you think would be the correct solution doesn't mean they are doing it wrong. It's quite possible that you are wrong. Not everybody except you is an idiot. Do you get that?

If you think backwards compatibility in the C ecosystem is so far superior, then where are C Editions? Where is the common practice in the C community to use Semantic Versioning?
These things work _really_ well in the Rust ecosystem. And they are either completely absent (Editions) or used extremely rarely (SV) in C.

Except from very few examples like glibc there are few projects in the C community that really care about backward compatibility to the extent possible with Rust Editions and Semantic Versioning. Most projects are stable because they are not changed much on the interface level anymore. glibc being a notable exception.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 18:36 UTC (Thu) by bluca (subscriber, #118303) [Link] (3 responses)

> Most projects are stable because they are not changed much on the interface level anymore. glibc being a notable exception.

This is absolute nonsense. Again, there is data on this, just look at it. Compare symbols exports vs SONAME revision of common system libraries. libsystemd0 alone added 213 new APIs just in the very latest release. SONAME revision is still 0.
Just because rustlang/golang library maintainers can't or won't maintain backward compatibility, it doesn't mean nobody else will.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 18:48 UTC (Thu) by mb (subscriber, #50428) [Link] (2 responses)

>Just because rustlang/golang library maintainers can't or won't maintain backward compatibility

Saying that Rust people don't maintain backward compatibility is:

>This is absolute nonsense

They do it in a very different way.
Like it or not.

There are more solutions to a problem than the Luca-way, you know?

I do understand that you either don't like or don't know what Rust people do.
But I'm just asking you politely to stop spreading this "they don't have/care about backwards compatibility" FUD.
I have ignored you for over half a year here in LWN comments and you are still busy spreading that nonsense. Why? This annoys me just as much as anti-systemd-FUD annoys you.

If you want Rust to have a stable soname/ABI, please come up with a proposal about *how* to do that.
Be constructive rather than destructive.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 19:41 UTC (Thu) by bluca (subscriber, #118303) [Link] (1 responses)

> I have ignored you for over half a year here in LWN comments and you are still busy spreading that nonsense. Why?

Because it's true, whether or not you like it. The standard library changes soname on every build. Backward incompatibility is programmatically baked in the build process itself.

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Jan 30, 2025 19:44 UTC (Thu) by corbet (editor, #1) [Link]

So are we sure that this conversation is going anywhere but in circles? Might it be time to conclude that there isn't more to be said (again)?

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Feb 7, 2025 18:17 UTC (Fri) by TheBicPen (guest, #169067) [Link] (1 responses)

> glibc last broke compat what, 20 years ago

August 2022. https://lwn.net/Articles/904892/

Don't just vendor - rebuild the ecosystem and persuade the vendor to work on software management ...

Posted Feb 7, 2025 18:34 UTC (Fri) by bluca (subscriber, #118303) [Link]


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds