Conill: The long-term consequences of maintainers’ actions [LWN.net]

Conill: The long-term consequences of maintainers’ actions

Posted Sep 17, 2021 21:01 UTC (Fri) by wahern (subscriber, #37304)
In reply to: Conill: The long-term consequences of maintainers’ actions by josh
Parent article: Conill: The long-term consequences of maintainers’ actions

How do distros handle backporting bug fixes, especially for dependencies through build systems like cargo? Does cargo let you patch source files locally? Or is none of this actually possible without, e.g., simply forking all the upstream repositories for a project and aliasing everything to your own forks?

to post comments

Conill: The long-term consequences of maintainers’ actions

Posted Sep 18, 2021 4:27 UTC (Sat) by josh (subscriber, #17465) [Link]

Distributions need to be self-contained and use local sources, including for dependencies. They shouldn't retrieve anything from the Internet during a build. Cargo has a "directory registry" mechanism that allows saying "if you'd get a crate from crates.io, get it from sources in this directory" instead.

(That said, we need a better mechanism for handling downstream patches, because right now distros just patch version 1.2.4 while pretending it's still 1.2.4. We need a way to create 1.2.4.1 instead.)

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 0:26 UTC (Sun) by robert_s (subscriber, #42402) [Link] (14 responses)

It's becoming increasingly unrealistic for distributions to diverge from the large statically-linked dependency sets dictated by golang, rust (and also java come to think of it) projects. On top of that, upstream projects are often hostile to downstream packagers making alterations like these.

But I think we're in for a shock the first time there's a really serious vulnerability in a widely used e.g. golang library. At that point, what do projects using that library do? Quietly bump their version, *maybe* make a note in the changelog, and move on? Or do like they should and release their own CVE? If they *don't*, what is the mechanism through which their users are supposed to get the nudge to upgrade? If they *do*, I'm certain the CVE process wouldn't be able to handle the avalanche effect from a widely used package. Do the users of the "recommended" distribution means (the official docker image, "just grab the binary"...) get the nudge somehow?

Or is the answer the increasingly familiar hand-wave of "just always run the latest version"? Because that might *sound* like an answer but it doesn't connect with the reality of running a system comprised of N thousand separate packages.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 3:39 UTC (Sun) by ilammy (subscriber, #145312) [Link] (13 responses)

> On top of that, upstream projects are often hostile to downstream packagers making alterations like these.

Is being unwilling to support hostile? It's understandable that upstream does not really want to support the entire combinatorial explosion of their package and all possible dependency versions, focusing on the latest ones where possible. Resources are scarce, their time is better spent elsewhere. From upstream viewpoint, if distro packagers need to support older versions of dependencies for whatever reasons they have, that’s their burden, not a responsibility of upstream.

It’s just that with modern languages it’s not unrealistic to have hundreds of transitive dependencies. In ye olden C dayes, you’d have just a handful of direct dependencies and an occasional transitive one here or there, with exceptions being rare. Now popping libraries like candies is easy, so that’s what everyone’s doing, without much regard over the stable maintenance burden they impose onto packagers by doing this. Developers use the latest version, it’s natural for them. Not so much for consumer users.

The question here is whether the distro packaging process is scalable enough. Arguably it isn’t. You can’t just take existing maintainer and say, “Look, you took care of ${software-package} and its dependencies. Now there are x10 more dependencies, you take care of them too, will ya?” So with the advent of software using more dependencies, you need to increase the maintenance team size too, so that each of those new mini-dependencies gets as much attention as a library got before, for the quality and speed of delivery to remain the same.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 13:59 UTC (Sun) by Vipketsh (guest, #134480) [Link] (12 responses)

> Resources are scarce

Why am I the only one who thinks this goes two ways ? An app or such not wanting to upgrade dependencies because it is work is reasonable as things stand today. But seemingly a dependency re-inventing itself causing tons and tons of cumulative resource burn for all downstreams is also reasonable, considering how no one complains about it.

Unless libraries keep some semblance of a stable public facing ABI things are going to be as bleak as the grand-parent post makes out. What's funny is that that exact situation has happened in the C world before: the fiasco that resulted when lots of projects are/were embedding zlib and a vulnerability was found therein. It took years to clean that mess up and instead of learning from that people are just setting up to the same situation all over again.

> upstream does not really want to support the entire combinatorial explosion of their package and all possible dependency versions

IMO, that's nothing but myth. The only versions (of dependencies and the app) that need to be supported are the ones that people actually use (with few exceptions what distributions ship) and there are not a lot of those. I also don't think upstream needs to be as forceful about versions as many seem to be. The linux kernel folks have the right mentality: promise to fix regressions, but at the same they freely make changes without checking them with some "combinatorial explosion of versions". If it breaks your code, it is up to *you* to report it and work with the upsteram devs to get it fixed. Don't report it and/or don't cooperate then it doesn't get fixed. Projects taking this attitude would go a long way for downstreams and I don't think it would cause *that* much more work for upstream.

> Developers use the latest version, it’s natural for them

Unfortunately, it goes the other way too and then causes just as much pain for the user. There are many projects that moved or are still moving from Python 2 to 3 at the last minute. Or maybe even past that.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 14:39 UTC (Sun) by ilammy (subscriber, #145312) [Link] (11 responses)

> It took years to clean that mess up and instead of learning from that people are just setting up to the same situation all over again.

Package management infrastructure is different now. Thinks like Cargo make updating dependencies easier. It’s not a quest of figuring out which of your dependencies exactly embed zlib and how to update it for their bespoke build system. Another thing is this static linking by default approach which makes maintaining ABI compatibility less of a concern: just rebuild everything when you need to upgrade something.

That said, API breakage is still a churn and can still deter applications from upgrading dependencies. No good way around that. Except for maybe waiting out for some more until things stabilize and stop breaking that often.

> The only versions (of dependencies and the app) that need to be supported are the ones that people actually use (with few exceptions what distributions ship) and there are not a lot of those.

The thing is, the version that people use may be different, sometime drastically. From distro user’s perspective the latest version is whatever their distro packages. From developer’s perspective it might be what *their* distro packages. Or maybe something closer to upstream.

> The linux kernel folks have the right mentality: promise to fix regressions

...in the next version of Linus’ tree, then maybe cherry-picked into Linux stable. Backporting the fix to 3.10-whatever version that RHEL ships is RHEL’s responsibility, not of the Linux kernel community. Same as resolving the issues from backports gone wrong that upstream kernel never had in the first place.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 15:31 UTC (Sun) by pizza (subscriber, #46) [Link] (3 responses)

> Package management infrastructure is different now. Thinks like Cargo make updating dependencies easier. It’s not a quest of figuring out which of your dependencies exactly embed zlib and how to update it for their bespoke build system. Another thing is this static linking by default approach which makes maintaining ABI compatibility less of a concern: just rebuild everything when you need to upgrade something.

A key difference is that Cargo only helps those already capable of recompiling the software. If you don't have the complete source code to _everything_ then you can't fix it yourself, period. You can't rely on a third party (eg distributions or some other system integrator) to update that one component and generate a new binary.

Rust (plus Go and all other static-link-only paradigms) makes you completely at the mercy of the software developer / vendor for all updates and fixes. That's what made the zlib thing so bad, and Cargo/etc won't change this -- even the relatively trivial "change your crate list to pull in a fixed version and recompile" actions represent more effort than has been historically demonstrated.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 15:47 UTC (Sun) by ilammy (subscriber, #145312) [Link] (2 responses)

> A key difference is that Cargo only helps those already capable of recompiling the software. If you don't have the complete source code to _everything_ then you can't fix it yourself, period.

But that’s already the case for FOSS distributions. They have all the source code available. They have the automatic build infrastructure. (Or well, should have it.) Rebuilding the world surely wastes time, electricity, bandwidth, storage space – more than ideally maintained dynamic linking ecosystem would – but it’s not impossible. Popularity of static-link-only paradigm shows that people are willing to rather accept some waste than maintain ABI compatibility.

> You can't rely on a third party (eg distributions or some other system integrator) to update that one component and generate a new binary.

But people rely on their distributions and vendors all the time to keep stuff updated.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 17:08 UTC (Sun) by pizza (subscriber, #46) [Link]

> But that’s already the case for FOSS distributions.

Much as I'd like it to be otherwise, FOSS distributions only ship a tiny portion of the software being written, and are not where most folks get their software. Indeed, this is gleefully called an advantage by the static-linked-world proponents ("cut out the middleman!") even though in practical terms this shifts F/OSS more like proprietary stuff.

But even putting aside proprietary software (which tends to fall more on the Apache OpenOffice side of the competency/responsiveness curve rather than LibreOffice side) the overwhelming trend is to push the F/OSS world into a app-store model with an all-in-one opaque blob per application that might as well be statically linked for all the technical ability an end-user has to modify things. The store itself has no ability to update anything; at best all it can do is scan for a known issue and prevent future downloads if a problem is found. Actually fixing things falls back to whomever "owns" that software's entry on the store. The software might as well be proprietary at that point.

We've been down this route before, more than once, and it has not gone well.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 22:09 UTC (Sun) by JanC_ (subscriber, #34940) [Link]

Automatic rebuilds would also trigger automatic updates everywhere much more frequently than they do now, and I can tell you that would upset pretty much anyone.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 15:48 UTC (Sun) by Vipketsh (guest, #134480) [Link] (6 responses)

> make updating dependencies easier

Granted, it's a easier to find dependencies of something, but that's still only part of the problem. If upgrading a dependancy is so easy as bumping a dependency version, there is no reason for projects to religiously want a single version either, but they do and they say it's for good reasons. The constant API changes that occur bring with it one of the issues that contributed to the zlib fiasco taking so long to solve: many projects embedding the library edited the interface and so finding the code was not enough -- extra work was needed to merge in the fix, same as if the new upstream's API changed.

> waiting out for some more until things stabilize

I wish this were more true, but that utopia never seems to come. For some reason, that I can not read out of any big picture, for GUI libraries (GTK & Qt) 25+ years of continuous development, 30-40 years of prior art and three or four major API changes were not enough for things to stabilise. In this time frame things didn't change that much either: we still use the same primitives (buttons, binary on/of switch, sliders, menus, etc.) as when GUIs started being a thing.

> it might be what *their* distro packages

What I'm saying is that there are not that many distributions and relevant versions of each. In other words, there is no "combinatorial explosion". Even between distros many are pretty similar in their package version choices (e.g. they often co-ordinate on Linux versions).

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 16:14 UTC (Sun) by mathstuf (subscriber, #69389) [Link] (5 responses)

> I wish this were more true, but that utopia never seems to come. For some reason, that I can not read out of any big picture, for GUI libraries (GTK & Qt) 25+ years of continuous development, 30-40 years of prior art and three or four major API changes were not enough for things to stabilise. In this time frame things didn't change that much either: we still use the same primitives (buttons, binary on/of switch, sliders, menus, etc.) as when GUIs started being a thing.

Alas, the world changes out from underneath us over time. The kernel is lucky in that hardware is pretty much set in stone once it ships, so it's just dealing with adding new things (in general). Qt, on the other hand, had to rework for things like moving widgets around to where they better belong (the widget APIs don't change that much, so it's mostly going from `QtGui` to `QtWidgets` or something. These things have improved usability because using Qt's PNG support no longer needs to bring in the GUI bits (Qt5 fixed this IIRC). Now with OpenGL being not *the* graphics API, but *an* API, more rework is necessary. I don't know what features, exactly, precipitated Qt6, but the Qt *developers* are still trustworthy (alas, I'm not so sure about their business unit :/ ).

I don't know. I admire how long Qt can put off decisions to break and at least they're well-advertised and not buried in some patch release.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 19, 2021 22:16 UTC (Sun) by JanC_ (subscriber, #34940) [Link] (4 responses)

Also useful if both the older & newer API are supported for a while, and preferably the newer API has all the features the older one has, so that people actually get some time to port things within a reasonable time…

Conill: The long-term consequences of maintainers’ actions

Posted Sep 20, 2021 5:27 UTC (Mon) by rodgerd (guest, #58896) [Link] (3 responses)

You say that, and yet even after maintaining Python 2 and 3 in parallel - for more than a decade! - with 6 years notice of end of life for 2, the team got entitled shrieking and abuse when they finally shut down 2.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 20, 2021 8:09 UTC (Mon) by farnz (subscriber, #17727) [Link]

Which was especially egregious, since all the team did was say that they would not release new versions of Python 2, nor would they look to see if Python 3 bugfixes applied to Python 2.

And there are things like Tauthon out there that someone who really wanted to stick to Py2 could use and help support; the complaining was largely because PSF Python would no longer support Python 2.

Conill: The long-term consequences of maintainers’ actions

Posted Sep 23, 2021 1:42 UTC (Thu) by JanC_ (subscriber, #34940) [Link] (1 responses)

I’m obviously not talking about dropping support for a past API after 10 years. But there are some that do that every single-digit number of weeks/months…

Conill: The long-term consequences of maintainers’ actions

Posted Sep 23, 2021 8:48 UTC (Thu) by NYKevin (subscriber, #129325) [Link]

I think the point is that *no* amount of notice will be universally acceptable. There's always somebody who's still using your ancient software (or hardware) that you told them to stop using years ago, because there's always some sector of the economy that doesn't monetarily benefit from upgrading just yet.

For example:

* Significant portions of the US financial system are or were recently running on EBCDIC-based systems, because that's the reason IBM gave for opposing the removal of trigraphs in C++17.
* In 2019, the US Air Force announced that they had figured out how to launch a nuclear missile without the use of floppy disks. 8-inch floppies, to be precise.
* Every single time Python is mentioned in any capacity in an LWN article, there's always a huge flame war in the comments over how terrible the 2-to-3 migration was, regardless of whether this actually has anything whatsoever to do with the contents of the article.