|
|
Subscribe / Log in / New account

Point releases for the GNU C Library

By Jonathan Corbet
October 16, 2017
The GNU C Library (glibc) project produces regular releases on an approximately six-month cadence. The current release is 2.26 from early August; the 2.27 release is expected at the beginning of February 2018. Unlike many other projects, though, glibc does not normally create point releases for important fixes between the major releases. The last point release from glibc was 2.14.1, which came out in 2011. A discussion on the need for a 2.26 point release led to questions about whether such releases have a useful place in the current software-development environment.

The glibc 2.26 release is generally only found in relatively fast-moving distributions at this point. For most users, 2.26 has been without problems, but that is not true for everybody. There have been a few significant regressions in this release that have required fixes; one of those was seen as important enough that the question of creating a 2.26.1 point release was raised. Romain Naour subsequently brought that discussion to the libc-alpha mailing list. Having a point release containing important fixes would be helpful to downstream distributors that want to incorporate those fixes, he said.

Like many projects, glibc fixes bugs first in its repository trunk — the branch that will eventually become 2.27 at the moment. The project does not make point releases, but that does not mean it leaves users of its previous releases entirely out in the cold. A branch in the repository is created for each release and, at the discretion of the manager for that particular release, important fixes are backported and added to that "maintenance branch" post-release. So distributors have a stream of nicely backported fixes for each release; what they don't have are point releases that contain a well-defined set of those fixes.

Some glibc developers strongly question the need for those releases. Creating a formal release is a significant amount of work, involving running tests, writing release notes, creating tarballs, etc. A major release is an indication that a point of stability has been reached and the code is in a state that is meant to be ready for production use. The maintenance branches, instead, are meant to always be in such a state. A point release would just be an arbitrary marker placed in the stream of maintenance patches, so one might as well just use "git describe" output to name that point. Thus, Florian Weimer observed that "the difference between glibc 2.26.5 and glibc 2.26-40 seems rather minor to me". He suggested that all that was really needed was a special tag at the beginning of each release branch that would show up in the output of a git describe command.

Representatives from a few distributions made it clear that they felt differently, though. Arjan van de Ven noted that a release tag helps distributors and users to know what is in a given package:

It also leads to, say, a CVE be able to list "fixed in 2.26.5" and everyone (and all more importantly, all tools that we all use to cross reference our distros to CVE databases) will know if things are already fixed, or if someone needs to take a look for a fix to backport outside of the releases

Andreas Huettel, representing Gentoo, agreed that there is value in having a well-known reference point. Yann Morin, of Buildroot, also agreed, noting that these reference points would be especially helpful to smaller distributors that do not have the bandwidth to follow the maintenance branch closely.

In response to the discussion, Siddhesh Poyarekar, the 2.26 release manager, said that he would probably do a 2.26.1 release in the near future. He explicitly did not commit to doing any point releases beyond that one, though. He also said that a point release, if he creates one, should not be seen as a commitment by the project as a whole to create such releases in the future. Carlos O'Donell stressed that last point, saying that "we don't want to set a precedent that the glibc release manager is always going to do a point release".

O'Donell went on to say that the glibc community in general lacks the resources needed to put together regular point releases. But, he said, downstream distributors might be able to create those releases with "a little help" from the glibc developers. He has promised a proposal describing how that cooperation might work, but it has not been posted as of this writing. He has, however, come out in favor of Weimer's special-tag proposal.

One might be tempted to conclude from this discussion that the need for our release-based software-distribution conventions is waning over time. When every point in a project's history already has a unique name (its Git commit ID), and when growing use of continuous-integration techniques leads to a situation where repository contents can be run at arbitrary points with relatively high confidence, perhaps the release rigmarole can be done away with, at least for point releases. But anybody thinking so should consider the ways in which glibc is unique.

The C library sits between most applications and the kernel. While many users install their own application releases, and a certain number are happy to replace the kernel, it is relatively rare to install a new glibc release. The glibc project moves slowly, new releases tend to have little in the way of exciting new features, and glibc is a rather difficult component to replace so, for the most part, whatever is on the system is good enough for most users. Few people feel the need to be running whatever the latest release is, so they tend to care rather less about glibc point releases than about releases of other components in their systems.

Distributors, instead, pay a lot of attention to the C library. Once they have installed a specific version, they are likely to scrutinize any follow-on patches and will not necessarily accept every change that appears in the glibc maintenance branch. So some distributors, at least, may see little value in point releases for glibc, while others (the smaller ones, perhaps) see some use there.

In other words, glibc may be unique in its relative lack of need for regular point releases; the rest of the community is unlikely to want to trade in concise version numbers for random-seeming commit IDs. But, even in the corner of the community where glibc lives, some users would rather have the occasional well-defined point release, with the result that glibc 2.26.1 may make an appearance in the near future.


to post comments

Point releases for the GNU C Library

Posted Oct 17, 2017 4:39 UTC (Tue) by NightMonkey (subscriber, #23051) [Link] (3 responses)

"O'Donell went on to say that the glibc community in general lacks the resources needed to put together regular point releases."

If this major component of the Linux ecosystem is lacking resources to do regular point releases, can it be inferred that the glibc community as a whole is suffering from lack of interest? Do we have a lurking OpenSSL scenario waiting to bite us in glibc because of human resource starvation? Every multinational corporation on the planet likely has glibc somewhere. Can we pass the hat behind the corporate veils?

Point releases for the GNU C Library

Posted Oct 17, 2017 20:28 UTC (Tue) by nix (subscriber, #2304) [Link] (2 responses)

The glibc community as a whole is lively: it's safe to say that it is more active than it has ever been.

It's just that nobody wants to do the relatively boring scutwork of doing point releases when the decision about when to make them is totally arbitrary, since *every commit* on the stable branches is meant to be equally suitable for use.

Point releases for the GNU C Library

Posted Oct 18, 2017 8:15 UTC (Wed) by NAR (subscriber, #1313) [Link] (1 responses)

Then maybe have "patch Tuesdays" and do a point release every month (if there was any commit on the stable branch). Now, of course the problem would be that Fedora would use 2.27.1, Ubuntu would use 2.27.2, SuSE would use 2.27.3, Debian stable would be on 2.27.4, Debian testing on 2.27.5, Debian unstable on 2.27.6, the next Fedora on 2.27.7, the next Ubuntu on 2.27.8, etc...

Point releases for the GNU C Library

Posted Oct 19, 2017 7:30 UTC (Thu) by nim-nim (subscriber, #34454) [Link]

That just means a monthly point release is too frequent to be useful. I suspect the right balance between freshness and convergence on common versions (to share QA and field experience) would be around 4 per year, including major releases, and taking spring/autumn distro releasing into account (always do releasing about a month before those, and the two others mid-release).

As for numbering WillyTarreau posted a good automated recipe. Excellent software engineering as usual, Willy !

Point releases for the GNU C Library

Posted Oct 17, 2017 6:32 UTC (Tue) by pitrou (guest, #118306) [Link] (8 responses)

> Creating a formal release is a significant amount of work, involving running tests, writing release notes, creating tarballs, etc.

I'm a bit surprised by this sentence. Does this mean that tests otherwise don't get run on the maintenance branch?

Point releases for the GNU C Library

Posted Oct 17, 2017 8:56 UTC (Tue) by NAR (subscriber, #1313) [Link] (5 responses)

You still need to run the tests on the actual release (in case there's a bug in the release creating procedure).

Point releases for the GNU C Library

Posted Oct 17, 2017 10:20 UTC (Tue) by smurf (subscriber, #17840) [Link] (4 responses)

So? all of this can be automated.

If the maintainers don't want to do that, let some distro people step forward.

Point releases for the GNU C Library

Posted Oct 17, 2017 14:51 UTC (Tue) by NAR (subscriber, #1313) [Link] (3 responses)

Even if it is automated, it still takes time and effort. How many architectures and kernels does glibc support? The tests need to run all of them.

Point releases for the GNU C Library

Posted Oct 17, 2017 16:04 UTC (Tue) by zyga (subscriber, #81533) [Link] (2 responses)

Lots of automated is still zero of human effort.

I think it's very reasonable to question this and if it is indeed an infrastructure problem, solve it. The Linaro community tests countless kernel builds automatically 24/7. On ARM where it still is, arguably, not very easy. Given that a typical Linux distribution has automated builders for all the architectures they support it is not inconceivable to say that we can automatically build and test this on just about everything that matters.

If the problem is elsewhere it should be clearly stated so that interested parties can examine it and come up with a plan.

Point releases for the GNU C Library

Posted Oct 17, 2017 16:51 UTC (Tue) by k3ninho (subscriber, #50375) [Link] (1 responses)

> Lots of automated is still zero of human effort.
"You've got a nice lot of unaudited test output over here. It would be a real shame if the results you received somehow weren't the results you were expecting."

Tests nobody audits, in this case stuff taking zero human effort, aren't providing information to that question of "should I trust this software, and with what tasks?" They may be out of scope for the contemporary functionality, badly configured so irrelevant, or simply not yielding any or any meaningful output.

It's never zero human effort. An unchecked result isn't worth the time it took to run the tests. It's less effort than doing it all by hand, and can even be set up so that a clean run is trivial to confirm and a failing run includes clear diagnostic information about what broke, how and why, but it's literally never zero work.

We really are that short on community person-power that I'm writing this and not some tests. (Of this, I'm a bit ashamed.)

K3n.

Point releases for the GNU C Library

Posted Oct 22, 2017 15:17 UTC (Sun) by mb (subscriber, #50428) [Link]

>"You've got a nice lot of unaudited test output over here. It would be a real shame if the results you received somehow weren't the results you were expecting."

If your automated tests are failing and thus you need humans to look at them, you should better do so.
If your automated tests are passing, no human interaction is needed.

I don't see how this increases the human work. You will have to handle your failing tests anyway.

Point releases for the GNU C Library

Posted Oct 19, 2017 9:35 UTC (Thu) by swilmet (subscriber, #98424) [Link] (1 responses)

I also find it strange that releasing a new version is a significant amount of work.

“Release early, release often.”

“If it hurts, do it more often.”
https://martinfowler.com/bliki/FrequencyReducesDifficulty...

Point releases for the GNU C Library

Posted Oct 26, 2017 9:26 UTC (Thu) by Wol (subscriber, #4433) [Link]

"Difficulty" and "effort" are two completely different things.

Many moons ago, I wrote a Critical Path Analysis system. If I have, say, five people, and I have a bunch of jobs that require a person on site, there is a strict upper limit as to how often I can repeat those jobs.

Oh - and asking those same five people to work longer hours is counter-productive. It's called burn-out - squeezing more hours out usually gives you more billable hours for less work done.

Cheers,
Wol

Point releases for the GNU C Library

Posted Oct 17, 2017 10:55 UTC (Tue) by post-factum (subscriber, #53836) [Link]

It is like going systemd-way: https://github.com/systemd/systemd-stable

Arch has adopted this, for instance, and to my taste, this looks just fine.

Point releases for the GNU C Library

Posted Oct 18, 2017 9:56 UTC (Wed) by nim-nim (subscriber, #34454) [Link] (1 responses)

> One might be tempted to conclude from this discussion that the need for our release-based software
> distribution conventions is waning over time.

Yes, sure, we're all be better of with everyone distributing a different git commit, with non reproducible behaviours/bugs right and left and no possibility of shared QA, shared security audits, shared feedback. It works sooo well for Android to have a soup of commits and branches and not two systems with the same mix. One wonders why Google felt the need to bother with Trebble.

Instead of building sand castles with complex management engines that try to simulate dev behaviour and track what git commit is deployed where, requiring permanent network access to every possible git repo under the sun (since commit ordering can't be deduced from commit ids), massive info leak and non trivial cascading failures when part of the network is down or a bad commit is pushed somewhere, projects would be well inspired to automate the act of releasing a version (starting with adding a git release command).

Commit granularity is good and well for project developers and automated regression testing. It is not sufficient for actual deployment. The time scales, level of project knowledge, update frequencies, and level of risk are too different. Borking a dev build has little consequences, borking something that manipulates actual production data with actual real-world consequences is something else entirely. Just ask Equifax.

A release marks human intent and promise (the project reached a stability point, we don't intend to change it right and left in the near future, we are ready to accept problem reports on this state). There's no way for a purely automated git process to capture and convey this intent. A release is a synchronisation point between different individuals and organisations.

Besides, once you painstakingly deploy one of those git management sand castles, and have perfect visibility on your commit use, it is all perfectly useless. The amount of work needed to identify the security or functional properties of each commit is so huge no one bothers. If something fails it's your fault for drawing a bad ticket, no one is accountable and no one will bother analysing what's wrong in the commit mix you're using. You're usually asked to play commit lottery again with a new commit mix.

So doing a release is teeedious and a “rigmarole” and not needed by the devs themselves. Well, devs develop software. They can build better tools to make releasing less tedious. That's what Linux did when he got fed up with SCM state. Arguing releasing has no function is perfectly disingenuous and non productive.

Point releases for the GNU C Library

Posted Oct 18, 2017 15:55 UTC (Wed) by liw (subscriber, #6379) [Link]

Hear, hear. Well said.

Point releases for the GNU C Library

Posted Oct 19, 2017 5:01 UTC (Thu) by wtarreau (subscriber, #51152) [Link] (1 responses)

There exists another option for maintenance versions, which usually satisfies everyone. The principle is to use the number of patches from the maintenance version as the patchlevel. What's nice with this method is that it's only increasing so it's easy to say "this CVE was fixed in 2.26.73" implying that anything above that has the fix as well, and there's no extra work as automated scripts can produce the tarball by simply using "git describe" followed by "git log | wc -l". I'm already using this in haproxy and that's very convenient for everyone including users.

Point releases for the GNU C Library

Posted Oct 20, 2017 11:54 UTC (Fri) by nim-nim (subscriber, #34454) [Link]

Hi Willy,

Anyway the main problem is the lack of "release" notion at Git level, when devs want to use Git exclusively. One can try to workaround it but it will always be fragile and project-specific. I took the time to push the subject upstream.

https://public-inbox.org/git/1290947539.4254.150849603981...

I hope they'll do something.


Copyright © 2017, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds