Frequent updates [LWN.net]

Frequent updates

Posted May 22, 2025 5:47 UTC (Thu) by geofft (subscriber, #59789)
Parent article: Recent disruptive changes from Setuptools

> One might argue that the simple fact that Setuptools is on version 78 points to a problem in itself — that Setuptools development is moving far too quickly. This is not entirely fair: about half of Setuptools major versions date to before the implementation of the new pyproject.toml-based system, with version 39 being released in March 2018. However, we still now see about five new major versions of Setuptools per year. Progress on updating legacy projects to modern standards, needless to say, has been far slower. While I'd prefer not to see Setuptools development artificially slowed, it would be nice to have breaking changes bundled together so that they occur less often.

This is intuitively appealing from the point of view of "Wow, I was just broken by an update," because you extrapolate from that to the belief that _every_ update might break you. But the whole point of frequent releases like this is the exact opposite: that almost no updates do break you.

Keep in mind that the question here is not how much breakage happens. That happens at its own pace regardless of version number. The question is what the experience is of having the breakage happen to you. If there are three breaking changes that affect you happening in the span of five years, again, it's intuitively appealing to say that you want to be disrupted once instead of three times. But actually dealing with all three changes at once is much more difficult. If you don't have a handle on what broke, you have much more code to bisect. If you are trying to adopt a new feature, you now have a roulette wheel that will with one-third probability impose three times the work on you before you are able to adopt that feature. And in any case, you still have to actually deal with all three changes, so you're not doing any less work.

If you really, really want to have coarser-grained updates, there are various ways to pull that off; you can set a global constraints file on setuptools and upgrade versions rarely, you can use some sort of frozen mirror, you can use the OS-packaged version of setuptools, etc. If you're getting all 78 versions of setuptools, that's because you're installing directly from PyPI, and presumably have decided that subjecting yourself to whatever version of everything is current on PyPI is worth doing.

The Linux kernel, for instance, is on its 74th release in the 2.6.x series. It has a little more whimsy in its actual version numbering scheme than setuptools' spartan integers, but it's not actually conveying anything different. It has an infamously strict rule on backwards compatibility. And yet people do not upgrade to every minor release and pay good money for the ability to stay on a minor release for a decade (or more). I don't think I've ever seen a proposal that the fact that the kernel is on version 6.14 means that kernel development is moving far too quickly, and it would be nice to have fewer mainline kernel releases, each with a heftier set of changes.

to post comments

Frequent updates

Posted May 22, 2025 11:46 UTC (Thu) by pizza (subscriber, #46) [Link]

> Keep in mind that the question here is not how much breakage happens. That happens at its own pace regardless of version number. The question is what the experience is of having the breakage happen to you. If there are three breaking changes that affect you happening in the span of five years, again, it's intuitively appealing to say that you want to be disrupted once instead of three times. But actually dealing with all three changes at once is much more difficult.

You forget that not everyone has the luxury of upgrading everything they have deployed in lock-step. Especially given that every [sub-[sub-[sub-]]]component they depend on has its own independent lifecycle.

...Case in point being setuptools iterating about 5x faster than actual python releases, and 10-25x faster than most deployment environments [1]. But more anecdotally, the lion's share of my time at $dayjob-1 was spent constantly cleaning up the messes from the impedence mismath between some rapidly-iterating bleeding-edge dependencies, others provided by EDA tool vendors [2], and everything in between.

[1] ie LTS/EL platforms with 2+ year release cadences and ~5yr standard support lifecycles
[2] More accurately, python code _generated_ by said EDA tools.

Frequent updates

Posted May 22, 2025 12:01 UTC (Thu) by aragilar (subscriber, #122569) [Link]

I would agree lots of small breaking changes tend to be easier to handle than a large one (as long as the effort to fix them is minimal, which is not always the case). I think though this all assume said changes are worth the pain, and maybe in some cases a major break is sometimes better because it forces a reset of how things are though of.

From what I've seen of setuptools issues, because of its long legacy, no matter what setuptools does, any changes setuptools makes to try to modernise will break large numbers of packages (and because wheels as distribution artefacts vs build caches are in many ways a least-common-denominator option, people keep relying on legacy distutils and setuptools features). This then means as soon as a package's packaging config is not kept up-to-date (even though it may need no other changes), issues arise. It also means the long tail of versions break in painful ways.

The obvious solution is to be picky about your dependency tree, and vendor and rewrite as needed, but if we're routing around issues with setuptools, at what point does it make sense to fork it (either freezing the old version, or making a more stable fork) rather than all the other packages?

Frequent updates

Posted May 22, 2025 14:06 UTC (Thu) by zahlman (guest, #175387) [Link]

The thing is that these kinds of breaking changes in Setuptools can break thousands of packages while still only impacting a very small percentage of packages. Both because of how big PyPI is, and because of ecosystem effects (when your package breaks, so do your dependents, transitively). If Setuptools bundled together multiple changes on the order of the one in version 78.0.0, it would IMHO be pretty unlikely that any given package had to deal with multiple of them at once. (Although the kinds of projects that are destined to be hit by multiple such changes eventually, might be better off just switching to a different build backend now.)

Also, if Setuptools had a slower release cadence for major versions, packages could think more seriously about setting a version upper-bound on the Setuptools version used to build them - since they could re-evaluate the decision less frequently.