LWN: Comments on "Point releases for the GNU C Library"

Point releases for the GNU C Library

Wol — Thu, 26 Oct 2017 09:26:11 +0000

"Difficulty" and "effort" are two completely different things.

Many moons ago, I wrote a Critical Path Analysis system. If I have, say, five people, and I have a bunch of jobs that require a person on site, there is a strict upper limit as to how often I can repeat those jobs.

Oh - and asking those same five people to work longer hours is counter-productive. It's called burn-out - squeezing more hours out usually gives you more billable hours for less work done.

Cheers,
Wol

Point releases for the GNU C Library

mb — Sun, 22 Oct 2017 15:17:27 +0000

>"You've got a nice lot of unaudited test output over here. It would be a real shame if the results you received somehow weren't the results you were expecting."

If your automated tests are failing and thus you need humans to look at them, you should better do so.
If your automated tests are passing, no human interaction is needed.

I don't see how this increases the human work. You will have to handle your failing tests anyway.

Point releases for the GNU C Library

nim-nim — Fri, 20 Oct 2017 11:54:54 +0000

Hi Willy,

Anyway the main problem is the lack of "release" notion at Git level, when devs want to use Git exclusively. One can try to workaround it but it will always be fragile and project-specific. I took the time to push the subject upstream.

https://public-inbox.org/git/1290947539.4254.150849603981...

I hope they'll do something.

Point releases for the GNU C Library

swilmet — Thu, 19 Oct 2017 09:35:56 +0000

I also find it strange that releasing a new version is a significant amount of work.

“Release early, release often.”

“If it hurts, do it more often.”
https://martinfowler.com/bliki/FrequencyReducesDifficulty...

Point releases for the GNU C Library

nim-nim — Thu, 19 Oct 2017 07:30:59 +0000

That just means a monthly point release is too frequent to be useful. I suspect the right balance between freshness and convergence on common versions (to share QA and field experience) would be around 4 per year, including major releases, and taking spring/autumn distro releasing into account (always do releasing about a month before those, and the two others mid-release).

As for numbering WillyTarreau posted a good automated recipe. Excellent software engineering as usual, Willy !

Point releases for the GNU C Library

wtarreau — Thu, 19 Oct 2017 05:01:01 +0000

There exists another option for maintenance versions, which usually satisfies everyone. The principle is to use the number of patches from the maintenance version as the patchlevel. What's nice with this method is that it's only increasing so it's easy to say "this CVE was fixed in 2.26.73" implying that anything above that has the fix as well, and there's no extra work as automated scripts can produce the tarball by simply using "git describe" followed by "git log | wc -l". I'm already using this in haproxy and that's very convenient for everyone including users.

Point releases for the GNU C Library

liw — Wed, 18 Oct 2017 15:55:24 +0000

Hear, hear. Well said.

Point releases for the GNU C Library

nim-nim — Wed, 18 Oct 2017 09:56:42 +0000

> One might be tempted to conclude from this discussion that the need for our release-based software
> distribution conventions is waning over time.

Yes, sure, we're all be better of with everyone distributing a different git commit, with non reproducible behaviours/bugs right and left and no possibility of shared QA, shared security audits, shared feedback. It works sooo well for Android to have a soup of commits and branches and not two systems with the same mix. One wonders why Google felt the need to bother with Trebble.

Instead of building sand castles with complex management engines that try to simulate dev behaviour and track what git commit is deployed where, requiring permanent network access to every possible git repo under the sun (since commit ordering can't be deduced from commit ids), massive info leak and non trivial cascading failures when part of the network is down or a bad commit is pushed somewhere, projects would be well inspired to automate the act of releasing a version (starting with adding a git release command).

Commit granularity is good and well for project developers and automated regression testing. It is not sufficient for actual deployment. The time scales, level of project knowledge, update frequencies, and level of risk are too different. Borking a dev build has little consequences, borking something that manipulates actual production data with actual real-world consequences is something else entirely. Just ask Equifax.

A release marks human intent and promise (the project reached a stability point, we don't intend to change it right and left in the near future, we are ready to accept problem reports on this state). There's no way for a purely automated git process to capture and convey this intent. A release is a synchronisation point between different individuals and organisations.

Besides, once you painstakingly deploy one of those git management sand castles, and have perfect visibility on your commit use, it is all perfectly useless. The amount of work needed to identify the security or functional properties of each commit is so huge no one bothers. If something fails it's your fault for drawing a bad ticket, no one is accountable and no one will bother analysing what's wrong in the commit mix you're using. You're usually asked to play commit lottery again with a new commit mix.

So doing a release is teeedious and a “rigmarole” and not needed by the devs themselves. Well, devs develop software. They can build better tools to make releasing less tedious. That's what Linux did when he got fed up with SCM state. Arguing releasing has no function is perfectly disingenuous and non productive.

Point releases for the GNU C Library

NAR — Wed, 18 Oct 2017 08:15:39 +0000

Then maybe have "patch Tuesdays" and do a point release every month (if there was any commit on the stable branch). Now, of course the problem would be that Fedora would use 2.27.1, Ubuntu would use 2.27.2, SuSE would use 2.27.3, Debian stable would be on 2.27.4, Debian testing on 2.27.5, Debian unstable on 2.27.6, the next Fedora on 2.27.7, the next Ubuntu on 2.27.8, etc...

Point releases for the GNU C Library

nix — Tue, 17 Oct 2017 20:28:09 +0000

The glibc community as a whole is lively: it's safe to say that it is more active than it has ever been.

It's just that nobody wants to do the relatively boring scutwork of doing point releases when the decision about when to make them is totally arbitrary, since *every commit* on the stable branches is meant to be equally suitable for use.

Point releases for the GNU C Library

k3ninho — Tue, 17 Oct 2017 16:51:38 +0000

> Lots of automated is still zero of human effort.
"You've got a nice lot of unaudited test output over here. It would be a real shame if the results you received somehow weren't the results you were expecting."

Tests nobody audits, in this case stuff taking zero human effort, aren't providing information to that question of "should I trust this software, and with what tasks?" They may be out of scope for the contemporary functionality, badly configured so irrelevant, or simply not yielding any or any meaningful output.

It's never zero human effort. An unchecked result isn't worth the time it took to run the tests. It's less effort than doing it all by hand, and can even be set up so that a clean run is trivial to confirm and a failing run includes clear diagnostic information about what broke, how and why, but it's literally never zero work.

We really are that short on community person-power that I'm writing this and not some tests. (Of this, I'm a bit ashamed.)

K3n.

Point releases for the GNU C Library

zyga — Tue, 17 Oct 2017 16:04:31 +0000

Lots of automated is still zero of human effort.

I think it's very reasonable to question this and if it is indeed an infrastructure problem, solve it. The Linaro community tests countless kernel builds automatically 24/7. On ARM where it still is, arguably, not very easy. Given that a typical Linux distribution has automated builders for all the architectures they support it is not inconceivable to say that we can automatically build and test this on just about everything that matters.

If the problem is elsewhere it should be clearly stated so that interested parties can examine it and come up with a plan.

Point releases for the GNU C Library

NAR — Tue, 17 Oct 2017 14:51:01 +0000

Even if it is automated, it still takes time and effort. How many architectures and kernels does glibc support? The tests need to run all of them.

Point releases for the GNU C Library

post-factum — Tue, 17 Oct 2017 10:55:49 +0000

It is like going systemd-way: https://github.com/systemd/systemd-stable

Arch has adopted this, for instance, and to my taste, this looks just fine.

Point releases for the GNU C Library

smurf — Tue, 17 Oct 2017 10:20:56 +0000

So? all of this can be automated.

If the maintainers don't want to do that, let some distro people step forward.

Point releases for the GNU C Library

NAR — Tue, 17 Oct 2017 08:56:52 +0000

You still need to run the tests on the actual release (in case there's a bug in the release creating procedure).

Point releases for the GNU C Library

pitrou — Tue, 17 Oct 2017 06:32:19 +0000

> Creating a formal release is a significant amount of work, involving running tests, writing release notes, creating tarballs, etc.

I'm a bit surprised by this sentence. Does this mean that tests otherwise don't get run on the maintenance branch?

Point releases for the GNU C Library

NightMonkey — Tue, 17 Oct 2017 04:39:50 +0000

"O'Donell went on to say that the glibc community in general lacks the resources needed to put together regular point releases."

If this major component of the Linux ecosystem is lacking resources to do regular point releases, can it be inferred that the glibc community as a whole is suffering from lack of interest? Do we have a lurking OpenSSL scenario waiting to bite us in glibc because of human resource starvation? Every multinational corporation on the planet likely has glibc somewhere. Can we pass the hat behind the corporate veils?