Adding Requests to the standard library [LWN.net]

Adding Requests to the standard library

Posted Apr 30, 2015 18:40 UTC (Thu) by Pc5Y9sbv (guest, #41328)
In reply to: Adding Requests to the standard library by pj
Parent article: Adding Requests to the standard library

I think it is a huge problem that open source programming language or application communities try to take over package management. Open source projects should try to be good "upstream" providers, but shouldn't pretend that there is no place for actual OS distributions who actually produce packages and coordinate their release to end users. The integration and coordinated release of updates is an editorial process that will not (I'd argue cannot) have a single solution point suitable for all users.

I think it is contrary to the public good to try to bypass the OS distribution package maintainers while not taking over the responsibility they have shouldered for all these years. Someone needs to do the less sexy work of back-porting security fixes and otherwise tending to the legacy software stacks out there, even when most upstream developers do not have the time nor interest to do that for each different user community with their different needs. You cannot simply state that everyone should chase a shared repository, updating bits of software every time a developer pushes out some changes. These are policy decisions, which do not have one globally correct answer.

Outside of a few high budget {S,I}aaS operations who can really curate their whole stack properly using internal engineering resources, there are really only a few choices: freeze an entire stack except for the small part you actively work on; face an endless churn where you try to track your third-party upstreams while they may abandon the versions of code you depend on at any moment when a critical problem arises; or rely on the OS distribution intermediary who can provide some happy medium where your third-party dependencies are kept patched in a backward-compatible manner.

It seems the community is groping around in the dark recently, rediscovering old black-box proprietary software delivery methods while not recognizing their drawbacks. Things like containers, statically linked applications, and other horribly stovepiped software modules are largely a regression for open source development. Unless you are one of those highly funded operations who can really curate a whole stack, these methods simply amount to abandonment-in-place of all your integrated third party elements.

to post comments

Adding Requests to the standard library

Posted May 1, 2015 16:01 UTC (Fri) by raven667 (subscriber, #5198) [Link] (30 responses)

> Someone needs to do the less sexy work of back-porting security fixes and otherwise tending to the legacy software stacks out there, even when most upstream developers do not have the time nor interest to do that for each different user community with their different needs.

That's the thing which would be great to change though, to help the upstream developers maintain a set of backported releases rather than punting to the distros to duplicate all of that integration work, especially when the patches come from the upstream themselves. Cut out the need for a middleman by fixing the problems which created the need.

Then the application developers can depend on one runtime maintained by the people who know most about it rather than one fork per distro.

> Things like containers, statically linked applications, and other horribly stovepiped software modules are largely a regression for open source development.

I think this is a reaction to inadequacies to application distribution in the Linux OS model where each version of each distro provides a unique ABI, which leads to 10+ common variations that an app developer is expected to support. It doesn't matter if the different Linux OS distros are 99.999% compatible with one another, if you use some reasonable number of shared libraries it is almost certain that some two common distros will have mutually incompatible ABIs. It only takes one library difference to screw it up for your application.

> Unless you are one of those highly funded operations who can really curate a whole stack, these methods simply amount to abandonment-in-place of all your integrated third party elements.

If you are highly funded enough you can have a dedicated release team for each OS to paper over all the tiny differences which are fatal to binary compatibility and take advantage of the libraries and infrastructure that each unique OS provides, or you can pick one stack to integrate with and make it a requirement.

This comes down to the Mythical Man Month, effectiveness goes down as the amount of coordination and communication needed to do work goes up, making every application developer coordinate with every Linux OS team and try to understand all of their quirks, is a huge amount of overhead that shouldn't be discounted.

Adding Requests to the standard library

Posted May 1, 2015 18:36 UTC (Fri) by dlang (guest, #313) [Link]

> I think this is a reaction to inadequacies to application distribution in the Linux OS model where each version of each distro provides a unique ABI, which leads to 10+ common variations that an app developer is expected to support. It doesn't matter if the different Linux OS distros are 99.999% compatible with one another, if you use some reasonable number of shared libraries it is almost certain that some two common distros will have mutually incompatible ABIs. It only takes one library difference to screw it up for your application.

you are blaming the distros for this, but it's really the developers that are driving this.

if you are developing a library, you should think about all your users, and not assume that everyone is always going to be upgrading to the latest version the minute you release it. you need to maintain backwards compatibility so that programs compiled against a prior version will keep working with a new version.

If you are developing an application, you need to evaluate the libraries that you choose to use, not just from the point of view of what features do they offer, but from the point of view of what their attitude towards stability is. If the library developers routinely break backwards compatibility, then they had better offer a _really_ great feature set to make up for this big drawback. And if you pick a library with a bad track record, you need to do so with the acceptance that this will cause you problems, and those problems are worth it.

Adding Requests to the standard library

Posted May 1, 2015 19:05 UTC (Fri) by dlang (guest, #313) [Link] (28 responses)

> That's the thing which would be great to change though, to help the upstream developers maintain a set of backported releases rather than punting to the distros to duplicate all of that integration work, especially when the patches come from the upstream themselves. Cut out the need for a middleman by fixing the problems which created the need.

The problem is that most of the upstreams don't have any interest in maintaining old versions with backported fixes. It's a lot of work and they've already released the fixed version. Why should they have to backport fixes across 10 years of development just because RedHat promised their paying users that fixes would be available that long? In the case of rsyslog, "supported versions" range from v3.22 to v8.9 depending on where you are getting it from.

The people who want the backports should be the ones who pay for them (either in money or in time). the software developers should avoid making changes that break existing users when they release new versions. Nobody is perfect, and occasionally there are really good reasons for breaking this compatibility, but if it's rare, it makes it far easier for users to just run the latest version and not have to worry about backports.

Remember when Firefox announced that they were moving to single release numbers and how there was a cry that it would break all distros because the distros couldn't stick with only shipping minor updates any longer? Firefox has done a good job of not breaking things with new releases and so it's basically a non-issue today and everyone just runs a current version (and many of the distros don't try to backport things, they just ship the new version). Users are happy, thins work.

more software should be treated this way, both from a "dont' break users" development point of view, and a "ship the latest" point of view. Shipping the latest of something that doesn't have a good "don't break users" development viewpoint is irresponsible on the part of the distros

Adding Requests to the standard library

Posted May 1, 2015 19:51 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (27 responses)

> The problem is that most of the upstreams don't have any interest in maintaining old versions with backported fixes. It's a lot of work and they've already released the fixed version.
It's a lot of work ON LINUX.

Making backports on Windows is extremely easy - if you simply compile it for the oldest supported OS version. For most of libraries even Win2000-level functionality is perfectly sufficient.

On Linux this is quite a bit more complicated.

Adding Requests to the standard library

Posted May 1, 2015 20:00 UTC (Fri) by dlang (guest, #313) [Link] (26 responses)

in spite of what you keep claiming, a lot of people are successfully doing exactly what you claim is so hard, they just compile against an older version of things and it 'just works'

you say
> For most of libraries even Win2000-level functionality is perfectly sufficient.

similar logic applies to most libraries on Linux, but if you insist on using the ones that aren't this way, who's fault is it?

Adding Requests to the standard library

Posted May 1, 2015 20:05 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (25 responses)

> in spite of what you keep claiming, a lot of people are successfully doing exactly what you claim is so hard, they just compile against an older version of things and it 'just works'
You don't only need to compile against the older versions. You have to compile against everything old.

For example, if you want to support RHEL5 (it's still not uncommon) then you have to compile with Ye Olde Glibc.

Then you want to use libjson which is happily packaged in RHEL7 and some Debians. How would you do that?

> similar logic applies to most libraries on Linux, but if you insist on using the ones that aren't this way, who's fault is it?
Let's put it this way - how do I get libffi5 on Debian 8?

Adding Requests to the standard library

Posted May 1, 2015 20:23 UTC (Fri) by dlang (guest, #313) [Link] (24 responses)

and if you want software to work on Win98 and Win 10 you also have problems.

Adding Requests to the standard library

Posted May 1, 2015 20:26 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (23 responses)

I have software that works perfectly fine on anything from WinXP (and probably Win2k, but who cares) to Windows 10. That's about 14 years of support.

RHEL5 was released in 2007.

Adding Requests to the standard library

Posted May 1, 2015 20:40 UTC (Fri) by dlang (guest, #313) [Link] (22 responses)

and people have posted here about binary-only linux software released in 1998 that still runs on the most recent distro releases.

so which is it? the fact that software that works over such time shows that things are good, or the fact that there is some software that breaks with the last release shows that everything is horrific?

you can find examples of both on any OS you pick.

You keep claiming that the situation on Linux is horrific, we had someone post on one of these threads that they had far less trouble supporting the variations on Linux than they had on supporting the variations on Windows.

Everyone knows you have problems on Linux and not on Windows, can you accept that other people find the opposite? And can you stop screaming about this on a montly basis?

Adding Requests to the standard library

Posted May 1, 2015 20:51 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (21 responses)

> and people have posted here about binary-only linux software released in 1998 that still runs on the most recent distro releases.
It will work. If it is completely statically linked or bundles everything above syscalls.

Adding Requests to the standard library

Posted May 1, 2015 21:22 UTC (Fri) by dlang (guest, #313) [Link] (20 responses)

> It will work. If it is completely statically linked or bundles everything above syscalls.

it will also work if it uses standard libraries that maintain good backwards compatibility (glibc, X11, etc)

Adding Requests to the standard library

Posted May 1, 2015 21:38 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

And this is pretty much the exhaustive list of commonly used libraries. Well, maybe also zlib.

Adding Requests to the standard library

Posted May 3, 2015 21:15 UTC (Sun) by paulj (subscriber, #341) [Link] (18 responses)

So basically only ancient, athena-era X11 apps.

Newer stuff, GNOME-era on, you're sunk. You need virtualisation/containers to run your old apps on the specific distro they were compiled for.

Adding Requests to the standard library

Posted May 3, 2015 23:03 UTC (Sun) by lsl (subscriber, #86508) [Link]

It's really not as bad as you make it sound.

http://upstream-tracker.org/

Quite a few solid libraries in there. Of course, if you want your program to run on RHEL5 you can't use library additions introduced yesterday. That seems to be the actual problem in some cases instead of missing backwards compat. It's the same on windows though: if you use features introduced with Windows 8 your program won't run on XP.

Adding Requests to the standard library

Posted May 4, 2015 3:34 UTC (Mon) by dlang (guest, #313) [Link] (16 responses)

> Newer stuff, GNOME-era on, you're sunk.

I believe that I have seen just a wee bit of displeasure at Gnome for the changes that they make from release to release ;-)

But in any case, is this the fault of the distro or the Gnome developers/maintainers? They are the ones who choose to make it so that it's hard or impossible to have both old and new versions of their libraries on the same system.

Adding Requests to the standard library

Posted May 4, 2015 3:37 UTC (Mon) by mjg59 (subscriber, #23239) [Link] (15 responses)

What's difficult about shipping the old GNOME libraries?

Adding Requests to the standard library

Posted May 4, 2015 3:48 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

my understanding is that the new versions use the same names as the old ones, so that if you want to install both, you have to rename one version (with the included hassle of changing everything that links to it to link to the new name)

I don't use Gnome, but I remember people complaining about this when Gnome 3 came out.

Adding Requests to the standard library

Posted May 4, 2015 3:50 UTC (Mon) by mjg59 (subscriber, #23239) [Link] (2 responses)

> my understanding is that the new versions use the same names as the old ones

Do you have any specific examples?

Adding Requests to the standard library

Posted May 4, 2015 3:58 UTC (Mon) by dlang (guest, #313) [Link] (1 responses)

no, as I said, go back and look at the discussions around the time that Gnome 3 was released and all the people who wanted to use the old version, but couldn't.

Adding Requests to the standard library

Posted May 4, 2015 4:00 UTC (Mon) by mjg59 (subscriber, #23239) [Link]

Those were discussions about whether it was possible to install multiple versions of binaries, not libraries.

Adding Requests to the standard library

Posted May 4, 2015 8:17 UTC (Mon) by paulj (subscriber, #341) [Link] (10 responses)

The old gnomemm library is difficult enough to ship that Fedora had to drop it, and along with that some (perfectly functional and very useful) applications stopped being available. E.g. referencer.

A container/chroot with the old libs is the easiest way to make that run (I tried building the old gnomemm myself from SRPMs - impossible due to build deps).

Adding Requests to the standard library

Posted May 4, 2015 10:55 UTC (Mon) by paulj (subscriber, #341) [Link] (9 responses)

Oh, and referencer is a free software app, with source. I spent 1.5 days going through it trying to update it to match the gnomemm → GLib/GTK+ changes (URI and GVFS stuff that moved to a 'similar but not quite the same' API in GLib), and still hadn't really made a dent of what was left. So I gave up and went with Zotero. That seemed less worse than relying on an app that needed fiddling with manually extracted old libs to keep running (and I'm happy with Zotero now).

ABI compatibility on desktop Linux - at least in the GNOME world - isn't even a joke.

Adding Requests to the standard library

Posted May 4, 2015 11:05 UTC (Mon) by paulj (subscriber, #341) [Link] (8 responses)

Oh, and I don't "blame" the GNOME people per se. It's sort of a wider failing. That the GNOME people have also been looking at doing their own distro-agnostic, package bundles suggests this too.

We need to get to a point where distros can guarantee forward compatibility. If a package installs on distro version N, it should install on all N+M, M>0.

Or, distros need to shrink to a core, and we look at higher-level ways of installing packages. As many of the programming languages already do for their library ecosystems - each differently though.

Adding Requests to the standard library

Posted May 4, 2015 14:33 UTC (Mon) by mathstuf (subscriber, #69389) [Link] (7 responses)

> We need to get to a point where distros can guarantee forward compatibility. If a package installs on distro version N, it should install on all N+M, M>0.

That's backwards compatibility. Forwards compatibility is "compile on N+M and run on N". *That* is not possible without versioned symbols and something to tell the linker to blacklist certain versions of symbols so that only specific ones are used. The solutions here[1] are way too verbose and finicky for sane maintenance (so you just end up building on Debian Etch'n'half and hoping for the best).

Adding Requests to the standard library

Posted May 4, 2015 14:34 UTC (Mon) by mathstuf (subscriber, #69389) [Link]

Oops, forgot the link: http://stackoverflow.com/questions/8823267/linking-agains...

Adding Requests to the standard library

Posted May 4, 2015 16:09 UTC (Mon) by paulj (subscriber, #341) [Link] (5 responses)

Uh, yes, I mean backward compatibility.

Point being, if you can't take (say) a package for SpiffyDistro 16 and install it on SpiffyDistro 20, you don't have a system that is going to be generally useful for computer users - because it isn't stable enough for vendors to target it. It might be useful for some niche subset of users.

It doesn't matter how it's done, through containers or what not, but that needs to work, if todays distro model is to survive.

Adding Requests to the standard library

Posted May 4, 2015 18:01 UTC (Mon) by dlang (guest, #313) [Link] (4 responses)

The thing is that excluding things that depend on "Desktop Environment" libraries, most apps and their libraries do continue to work on newer versions of a distro (and are pretty trivial to get working on different distros), there are a few notable exceptions (openssl, Python 3.x changes are a couple of noticeable ones), but overall things work well.

It's when you have things start tieing in to the Desktop Environments that things are more 'interesting'. Unfortunately this does include a large percentage of GUI apps, but not all of them, some GUI toolkits are more stable than others.

Adding Requests to the standard library

Posted May 4, 2015 18:16 UTC (Mon) by mjg59 (subscriber, #23239) [Link] (3 responses)

Again, which libraries are you talking about? libgnomemm vanished from Fedora because something lower down its build chain no longer built and nobody was willing to maintain it. The old binaries will still install.

Adding Requests to the standard library

Posted May 4, 2015 19:04 UTC (Mon) by paulj (subscriber, #341) [Link] (2 responses)

So why don't Fedora ship them then, if it's so easy to ship old libraries?

The end-user experience is that a perfectly good application disappeared.

Adding Requests to the standard library

Posted May 4, 2015 19:09 UTC (Mon) by mjg59 (subscriber, #23239) [Link] (1 responses)

It didn't build, and packages that don't build end up getting removed.

Adding Requests to the standard library

Posted May 4, 2015 19:36 UTC (Mon) by mjg59 (subscriber, #23239) [Link]

To be clear, I'm not arguing that this isn't a problem, just that the fault doesn't lie with the library authors.