Systemd v261 released [LWN.net]

Systemd v261 has been released with a long list of changes, including a new cloud "Instance Metadata Service" (IMDS) subsystem, "boot secret" functionality for use on systems that lack a physical TPM, as well as support for the kernel's Live Update Orchestration (LUO) / Kexec Handover (KHO) systems when they are present and enabled. See the release notes for the full list of changes.

to post comments

All according to plans

Posted Jun 19, 2026 21:03 UTC (Fri) by bluca (subscriber, #118303) [Link] (27 responses)

Just 2 days late on the schedule. Keeping a 3 months release cadence is not exhausting at all.

All according to plans

Posted Jun 20, 2026 12:29 UTC (Sat) by Vorpal (guest, #136011) [Link] (26 responses)

From what I have heard in other projects (such as the Rust project or the Linux kernel), having a shorter release cycle is actually easier:

* There is less pressure and stress to meet deadlines, if the next release after is due in a couple of months anyway. Things can be done when they are done.
* The release process itself becomes easier since it will be routine and more likely to be more automated anyway.

The rust project also has editions, which only happen every few years, and that has been considered way more stressful from what I read, and people have been floating ideas about having smaller yearly editions instead, but so far nothing has come of it.

All according to plans

Posted Jun 20, 2026 14:28 UTC (Sat) by magfr (subscriber, #16052) [Link]

Actually the C++ experience is also a data point here. Everyone agrees that the 13 years from C++98 to C++11 was a disaster.
The current 3-year release cycle seems to work in that community.

All according to plans

Posted Jun 20, 2026 15:36 UTC (Sat) by ibukanov (subscriber, #3942) [Link] (1 responses)

Rust editions are not releases but just versions of Rust syntax/semantic. Critically older editions are still supported by the compiler and one can link against crates using different editions.

All according to plans

Posted Jun 22, 2026 7:43 UTC (Mon) by taladar (subscriber, #68407) [Link]

They are the only point for 3 years where certain changes can happen though which is probably more relevant to the stress factor. If you miss it you have to wait another 3 years.

All according to plans

Posted Jun 21, 2026 10:52 UTC (Sun) by farnz (subscriber, #17727) [Link] (1 responses)

The other big thing, IME, is that once you've got fast enough releases, you stop letting the release date slip so that an important feature or bug-fix can make it in. That change in attitude from "delay the release" to "delay the feature" is surprisingly helpful.

All according to plans

Posted Jun 21, 2026 12:01 UTC (Sun) by iabervon (subscriber, #722) [Link]

I think a key aspect is that, at a certain cadence, you can can say that work doesn't happen on a feature after the previous release comes out, and you're not losing a significant amount of development time from the schedule. This means that your release process doesn't have to predict how long it will take for a feature to be finished (because it doesn't get assigned to a release until it's already finished), and release engineering doesn't have to make any judgment calls.

All according to plans

Posted Jun 21, 2026 21:49 UTC (Sun) by ballombe (subscriber, #9523) [Link] (20 responses)

The downside is that you have more releases to support.

The opposite strategy is the RHEL strategy of reducing the number of concurrently supported versions.

All according to plans

Posted Jun 22, 2026 7:54 UTC (Mon) by taladar (subscriber, #68407) [Link] (19 responses)

On the other hand if releases are frequent and small the whole long term support use case sort of collapses because every single update to the next release is just a few changes so it is much harder to justify staying on old versions for fear of unpredictable amounts of breakage.

In fact I would go so far as to say that I have never dreaded updates as much as when I had to go from one release to another where both releases were years apart and that upgrade was one RHEL version to another. Hundreds of packages, most with new major versions or other significant changes, is not a fun experience if you try to do them all at once.

All according to plans

Posted Jun 22, 2026 9:00 UTC (Mon) by nim-nim (subscriber, #34454) [Link]

As you wrote yourself the key success factor is getting your users to update regularly instead of locking down a “good enough” version and then freaking out before accumulated changes when this locked version needs replacement. And avoiding drastic fell-good greenfield revolutions that invalidate user trust in the safety of updates. The steady stream of small controlled changes builds trust.

All according to plans

Posted Jun 22, 2026 16:42 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (17 responses)

On the other hand if releases are frequent and small the whole long term support use case sort of collapses

That depends a lot on who your users are. If users are other developers who are expected to be constantly updating their projects anyway, then change is just part of the landscape. But if you're building for end users who are just trying to get their work done, constant change is the enemy. Most end users would rather have their application work consistently than have to adapt to constant change, even if that change is supposedly an improvement.

If I'm working on a years-long project, it's critical that my software work consistently the whole time so my results at the beginning and end are truly comparable. Maybe there will be an improvement that's worth spending the time to redo my work, but you're going to have to sell me on it. Even people with less stringent requirements than mine- and I am working in an area where consistency is unusually important- the temporary loss of productivity adapting to change is a real cost. Developers need to be mindful that those costs don't disappear just because they've pushed them onto their users.

All according to plans

Posted Jun 23, 2026 9:14 UTC (Tue) by taladar (subscriber, #68407) [Link] (15 responses)

The problem is that LTS proponents pretend that those costs disappear when in reality you are actually paying them as a huge sum every once in a while and that sum is usually much, much higher thanks to something comparable to interest (also compare to the technical debt analogy) than the actual small costs of frequent change.

That is partially because you are adopting the changes at the wrong time (e.g. if everyone introduced a new network protocol version 3 years ago and you are only doing so now you have 3 years worth of problems the new version solves and interoperability issues with the part of the world that introduced it all roughly at the same time to each other), partially because adopting huge numbers of changes obscures the reasons things break for you when you do introduce changes and partially because change is not routine so you forgot lots of things since the last change and have to rediscover them.

All according to plans

Posted Jun 23, 2026 11:30 UTC (Tue) by pizza (subscriber, #46) [Link] (10 responses)

> The problem is that LTS proponents pretend that those costs disappear when in reality you are actually paying them as a huge sum every once in a while and that sum is usually much, much higher thanks to something comparable to interest (also compare to the technical debt analogy) than the actual small costs of frequent change.

I disagree.

Each update brings with it a fixed cost/overhead of re-validating your system/workflow. Doesn't matter if there are 1000 changes or just 10 -- you still have to pay that fixed cost, re-integrate, and test.

Yes, there are ways of reducing that overhead (ie automation), but those also come with their own implementation and validation costs. Like everything else, there's a cost/benefit curve in play, but in practice it's often *much* cheaper to do fewer larger updates than more frequent smaller ones,

FYI: My last two $dayjobs were in very heavily regulated spaces, with legally mandated support lifecycles that end at least a decade after the final unit was sold.. and that's after 5 years of development+certification before the first unit ever ships.

"When the weight of the paper equals the weight of the airplane, only then you can go flying." -- Donald Wills Douglas

All according to plans

Posted Jun 24, 2026 18:57 UTC (Wed) by rgmoore (✭ supporter ✭, #75) [Link] (9 responses)

Each update brings with it a fixed cost/overhead of re-validating your system/workflow. Doesn't matter if there are 1000 changes or just 10 -- you still have to pay that fixed cost, re-integrate, and test.

Exactly right. One of the systems I work with is used for regulated work, and we have to re-validate the system for almost any update. It's a pain even for the limited validation we can use if the update "shouldn't" affect our key application, and not just because it keeps our system out of production until we can finish the validation. We would never update our OS at all if our IT Security people let us get away with it. Our application software normally only gets updated when we replace the computer hardware, which would require a much more extensive validation anyway. In theory we might update earlier if the new version had some killer feature that would justify the pain of a full validation, but I don't remember that ever happening.

All according to plans

Posted Jun 25, 2026 7:42 UTC (Thu) by taladar (subscriber, #68407) [Link] (5 responses)

You are confusing the cost of the update with the cost of some overly bureaucratic validation process that is also designed under the assumption that each update changes everything. Of course a process working from that assumption would not be cheaper with an update that changes only a few things.

All according to plans

Posted Jun 25, 2026 9:30 UTC (Thu) by kleptog (subscriber, #1183) [Link] (3 responses)

> designed under the assumption that each update changes everything.

But changes that only affect a few things are rare. If you update glibc you have effectively updated everything because everything depends on it. So everything needs to be retested.

If you install an update for almost any package, that could change the behaviour of any program using that package, which could change the behaviour of any programs that depend on that, recursively. So you essentially need to retest everything for almost every change. and if you need to retest everything anyway, you may as well bunch the updates in larger blocks.

But you say that one tiny bugfix somewhere could not possibly break a system on the other side the world. Except yes it can. It's rare but it happens that (e.g) a tiny optimisation somewhere reveals a race condition somewhere else bringing down an entire system. Bureaucracy doesn't appear out of nowhere, some of those regulations are written in blood.

All according to plans

Posted Jun 25, 2026 11:34 UTC (Thu) by farnz (subscriber, #17727) [Link] (2 responses)

The difference is in the amount of validation needed with a "proper" LTS process.

If I have good ECOs for every change in the update, telling me precisely what changed and why, I can limit my testing to system interactions plus anything affected by the things changed in the ECO.

In contrast, if all I have is a standard changelog (like the Linux kernel provides), I have to retest everything, because I can't be sure that a change described as "fix r8152 hang during runtime PM" only touches the r8152 driver - it could (and probably does) touch USB core stuff as well.

My personal bias is therefore to say that frequent small releases, and no big releases, are the right thing for most projects; if you're not putting in the effort to put out a decent ECO for every change in your LTS branch that allows people to omit chunks of validation as "not necessary", then you're not saving those of your customers who care about reliability any effort, and you are adding work for yourself deciding which changes need backporting, which ones need splitting (so that the bug fix can be backported without the attached feature work) and which ones need rewriting for the LTS (to get a bug fix that doesn't depend on work that isn't going to be backported).

It's worth noting, too, that the longer the LTS lasts, the more changes that will need rewriting for the LTS, and thus the more work you're signing up for; and for this work to avoid a need for full downstream revalidation, you're going to have to produce decent explanations of what's changed so that downstream can just revalidate system interactions, not everything.

All according to plans

Posted Jun 26, 2026 9:52 UTC (Fri) by taladar (subscriber, #68407) [Link] (1 responses)

It should also be noted that a backport is still new, untested code that could fundamentally unintentionally break everything just as much as a new feature in a new version could. In fact, probably more so, because backports are often done by people less familiar with the code base.

Process for backporting without unintentional breakage

Posted Jun 26, 2026 12:26 UTC (Fri) by farnz (subscriber, #17727) [Link]

Indeed - this is why you need the process scaffolding all this so that the blast radius of the change is well-understood, and documented by the ECO-equivalent you release with the backported fix, along with the changelog entry that highlights the relevant risk in brief.

If you don't have that, then all you're doing is asserting that some changes are necessary, and thus that the risk of breakage must be accepted by users because, in your opinion, the risk they take on by not taking the change is unacceptably high.

Personally, I suspect we've ended up in the place we're in because we've taken the process needed for a proper LTS release, removed the bits that are hard work like ECOs, planning a change, comparing the change to the plan etc, and left ourselves with the bits that are easier to do, but not that helpful.

All according to plans

Posted Jun 25, 2026 17:34 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link]

Our validation process isn't based around the assumption that every update changes everything; it's based around the painful experience that sometimes updates have effects beyond what they were supposed to have. I would love to live in the paradise where every update came with a complete and accurate list of changes and never introduced any bugs, but our processes have to live in the real world. And as much as I like to complain about validation, it's still way easier than investigating a problem that slipped through because you didn't test thoroughly enough before moving to production. It may even be easier than digging through a list of changes to figure out which ones might affect our process.

Update verification

Posted Jun 25, 2026 9:39 UTC (Thu) by farnz (subscriber, #17727) [Link] (2 responses)

But how does that process become cheaper if, instead of a weekly release that covers everything I've done in the last week, I have a once every 3 years release that covers everything I've done, plus a weekly release that covers everything I think you "must take" as an update? Assume in both cases that I'm not following the sort of process that would let you significantly reduce the amount of validation you do per-update.

Similarly, how does it become more expensive if I'm offering a weekly update, but you only take an update every 3 years, as opposed to if I only offer a big update every 3 years, and you take it?

That's the underlying driver behind "let's not pretend we're doing proper LTS engineering, but instead do frequent releases"; your cost of taking the "must take" update weekly, and an additional "big bang" update every 3 years is the same as, or higher than, your cost of taking a smaller combined update weekly.

To actually do better involves imposing a lot of process so that you don't have to do much validation on each "must take" update, which is entirely doable (and is done in regulated scenarios), but isn't something that people generally want to do - it's why people pay for RHEL and similar, because they do follow those processes, which allows you to reduce validation on the "security" fixes RHEL provide.

Update verification

Posted Jun 25, 2026 21:40 UTC (Thu) by pizza (subscriber, #46) [Link] (1 responses)

> To actually do better involves imposing a lot of process so that you don't have to do much validation on each "must take" update, which is entirely doable (and is done in regulated scenarios), but isn't something that people generally want to do - it's why people pay for RHEL and similar, because they do follow those processes, which allows you to reduce validation on the "security" fixes RHEL provide.

And even if folks aren't willing to pay for that process, they're quite glad to reap the benefits from those who do. [1]

[1] Ie the many RHEL rebuilds that are only possible because a critical mass of organizations pays for "genuine" RHEL.

Update verification

Posted Jun 26, 2026 9:57 UTC (Fri) by taladar (subscriber, #68407) [Link]

The only situations where I use those RHEL clones is pretty much to support the customers who do want to run RHEL but are too cheap to provide proper build, test,... environments with licensing too.

The actual huge benefits come from early adopter distros since those are the ones finding the bugs that more "stable" distros then don't have to deal with.

For actual production use something that isn't quite bleeding edge but also isn't too far behind to get the benefits from new features and still supported upstream versions (e.g. like Debian stable during the full support phase) is much better than the 10 year LTS distros like RHEL.

All according to plans

Posted Jun 23, 2026 15:16 UTC (Tue) by edgewood (subscriber, #1123) [Link] (3 responses)

Many LTS customers are completely OK with that, because they never upgrade. Hardware is installed with the current LTS and configured for its purpose. Then toward the end of the support life or regulatory cycle, new hardware is purchased, the then-current LTS is installed, and the system is configured and tested to ensure it can take over. When it can, the old system is decommissioned.

There's no upgrade breakage because there's no upgrade: the old system stays working until there's confidence that the new one is working properly, even if that takes a long time due to all the changes.

If there is a compelling feature that new software would enable, the cycle can be sped up, but that mostly doesn't happen because the system has a purpose that it met when it was set up. If there's something new and compelling, the company will set up a new system with a new purpose.

All according to plans

Posted Jun 23, 2026 17:42 UTC (Tue) by farnz (subscriber, #17727) [Link] (1 responses)

Of course, if you never upgrade the system, why do you care about the version you install being an LTS, rather than one with the features you want?

In practice, though, people do get upgrade breakage on LTS systems - they take changes that are deemed "bug fixes", and those can break things, since the category "bug fix" is (like all human classifications) fuzzy around the edges, and includes things that break workflows, or that stop the system functioning.

And as soon as you start taking any changes at all (even "bug fixes only"), you get into the mess that people make mistakes; they can classify a "bug fix" as a "feature change" and not fix it, or classify a "feature change" as a "bug fix" and accidentally introduce an unwanted change into the LTS system. They can make mistakes and thus have a "bug fix" not fix the bug it's supposed to fix, or even introduce a regression.

It's not at all clear that a project that hasn't planned very carefully for LTS designs (with a decent test suite, change management processes including things like ECOs for every feature, not just change logs, a process for ensuring that bugs are fixed in all active versions - which probably involves adding more tests - and more) actually can do this well - there's a huge amount of work in doing LTSes properly, and just "slow down the release schedule" isn't nearly enough.

All according to plans

Posted Jun 23, 2026 21:19 UTC (Tue) by edgewood (subscriber, #1123) [Link]

I meant that they don't upgrade to new versions of their distribution. And yes, getting "bug fixes only" is tricky, that's why they pay so much for their enterprise distributions.

All according to plans

Posted Jun 24, 2026 0:29 UTC (Wed) by raven667 (subscriber, #5198) [Link]

I'm in this boat, we use very few VMs and mostly run on bare metal, the fixed cost of migrating data and retesting on an entirely new stack when the hardware is replaced at end of support lifecycle also covers testing the changes due to RHEL upgrades. It may be batching up a lot of change at once, but it's also a project that has a budget and can be scheduled, we know some amount of maintenance work will be necessary to accommodate changes, that balances against all the surprise breakage we don't have to worry about during the normal lifecycle, aside from a recent functional change of perl-DBD-MySQL being linked against Oracle libmysql instead of MariaDB libmariadb as it was previously, we very very very rarely see any negative outcome from regular patching and don't need to worry about bad surprises. Automated testing would be useful and better, sure, but it isn't necessary for survival as the breakage risk is low.

All according to plans

Posted Jun 23, 2026 10:49 UTC (Tue) by farnz (subscriber, #17727) [Link]

On the other hand, if I release weekly, but you want a change every 3 years, you can just pick up a release to suit your timetable; if I release every 3 years, and you have a weekly update window, you're not going to be able to get frequent releases from me.

Further, if you can keep up with weekly releases, you get to tell me when a recent change has damaged your productivity, and I'm likely to remember making it (since it merged in the last week) and be able to improve it. If you tell me that a change I made 2½ years ago has damaged your productivity, I might well have forgotten making that change, and also be unable to improve it quickly for you - and note this applies whether I release weekly, and you're telling me that you picked up a new release after 3 years, or whether I release once every 3 years, and you've gone from N-1 to N.