LWN: Comments on "Fedora's modularity mess"

Fedora's modularity mess

cortana — Thu, 14 Oct 2021 08:26:14 +0000

The pain continues. Red Hat's otherwise quite nice Insights dashboard doesn't understand modules. It shows packages updates that would require module streams to be switched, or even disabled entirely.

What makes this particularly irritating is that enabling a module stream causes non-modular packages from enabled repositories to completely disappear when querying DNF. So you have Insights telling you that you can upgrade the softhsm package, but 'dnf list --showduplicates softhsm' only shows the package from the currently-enabled idm:DL1 module from the AppStream repo, and not the hidden 'ursine' module in EPEL.

Fedora's modularity mess

Wol — Thu, 12 Dec 2019 20:21:20 +0000

Especially when you have polar opposites

Cheers,
Wol

Fixed

kmweber — Thu, 05 Dec 2019 22:20:31 +0000

In the meantime, we'll just have to grin and bear it?

Fedora's modularity mess

rahulsundaram — Mon, 02 Dec 2019 17:49:12 +0000

> EPEL is packaged by Fedora maintainers using the libraries in Red Hat Enterprise as a base - so a third party repository for the main Enterprise distribution. Then you have Freshrpms / Livna / Elrepo / RPMFusion / whatever else Dag Wiers has contributed to :) (and whatever is today's hot repository).

If we are talking about RHEL here specifically, there are essentially two repositories in widespread use

Core - Commercially supported by Red Hat.
EPEL - Community supported

Packaging policy has nothing at all to do with this split clearly. This is in impact of RHEL being a commercial product

RPM Fusion is akin to non-free in Debian. Others got merged into RPM Fusion years and years back. It is far more popular in the Fedora space than within RHEL community. There isn't anything new or "hot" here. Again, the difference here is a combination of Fedora licensing policy and Red Hat is a commercial entity and doesn't want to take on liability of distributing patent encumbered code. Nothing in the RPM format or packaging policy is going to change any of this. It also has nothing to do with the modularity effort. So I am afraid I don't see the relevance

Fedora's modularity mess

amacater — Mon, 02 Dec 2019 17:01:30 +0000

kleptog has, essentially, made my point for me and clarified what I wanted to say but expressed badly.

Packaging policy can be applied equally to .debs and to RPM - but it isn't applied in the same way in most RPM based distributions and the IBM RPM ecosystem itself tends to depend on a mess of third party repositories, each from a small number of maintainers.

EPEL is packaged by Fedora maintainers using the libraries in Red Hat Enterprise as a base - so a third party repository for the main Enterprise distribution. Then you have Freshrpms / Livna / Elrepo / RPMFusion / whatever else Dag Wiers has contributed to :) (and whatever is today's hot repository).

Also you have the problem that the RPM ecosystem has historically only worked on a limited number of architectures so it's coming to ARM fifteen years late.

[And note, this suggestion from me is nothing new - I made a similar suggestion to rgb for the beginnings of the Beowulf project.]

Fedora's modularity mess

Conan_Kudo — Mon, 02 Dec 2019 15:58:17 +0000

RPM macros are "inspired" by an unholy combination between m4 and bash. There are some primitives that are bash-like and some that are m4-like. You can mostly get away from this if you write your logic in Lua (the other programming language supported by RPM), but most people don't.

Fedora's modularity mess

rahulsundaram — Mon, 02 Dec 2019 12:06:32 +0000

> There is nothing very special about the format, it's the Debian Packaging Policy that makes the difference

Indeed. So switching from RPM to Deb solves none of the problems and packaging policy can be applied in an identical way in an RPM based system. In fact, such packages already exist in Fedora and is not fundamentally different from software collections (which was more heavily used in RHEL and did not see much adoption in Fedora). The modularity group is just trying a different approach to solve it

Fedora's modularity mess

kleptog — Mon, 02 Dec 2019 09:50:53 +0000

There is nothing very special about the format, it's the Debian Packaging Policy that makes the difference. You start with the basic choice that "if you think people might want to parallel install packages they must have different names".

So you don't have postgresql:10 and postgresql:11, you have postgresql-server-10:10.11 and postgresql-server-11:11.6 and the corresponding client packages. The libraries libssl1.1, libssl1.0.2 and libssl1.0.0 are also distinct packages, the package name reflects the ABI presented. Then the whole discussion becomes much easier, dependencies are clear and installing multiple versions of a single package becomes unnecessary (although it is supported if the architecture is different).

To refer to the article, since libgit2-21 and libgit2-27 are co-installable the entire issue vanishes. It doesn't matter if the version 21 disappears from upstream, it remains on the user's system as long as they need it.

To make this work smoothly is of course a large part of the rest of the policy regarding file placement and versioning but it seems to me to solve all the issues this article is discussing. There is the choice that this scheme does not apply to dev packages for building, I think mostly because of (for example) lack of useful tooling support for dealing with three versions of the ssl.h header file being installed at the same time.

Fedora's modularity mess

snajpa — Sun, 01 Dec 2019 03:25:24 +0000

Hm, you're right, I wonder how I came across that bit of information. I've actually never bothered to validate that.

Hell, what do I know. And why do I care, I'll refrain from posting comments such as this in the future, esp. when I don't really care about the thing in question...

Sorry for that.

Fedora's modularity mess

rahulsundaram — Sat, 30 Nov 2019 23:50:22 +0000

You haven't cited a single unique feature of deb format that RPM format doesn't support. None of those solve the modularity goals. Gentoo slots and Nix does solve them in other ways but Debian doesn't. I would suggest starting with those for comparison points

Fedora's modularity mess

amacater — Sat, 30 Nov 2019 21:41:33 +0000

Debian _does_ have dependency resolution, strict packaging policy - and tracking of reverse dependencies. It's (almost) entirely reproducible. It generally works and is readily upgradable between major versions.

The modularity and fast moving packages - there's a working alternatives system and the stable/testing/unstable ecosystem generally means that newer versions are being worked on at the same time as stable remaining stable. Comparable in size to Fedora but having longer term support.
[Disclaimer: I've been using Debian since version 1.2 and Red Hat since 4.2 - I have to help support both but _still_ can't support developers wanting to use Red Hat for daily development with any degree of satisfaction.]

Fedora's modularity mess

rahulsundaram — Sat, 30 Nov 2019 21:02:32 +0000

> Or just admit that, of the surviving distributions, you were third to the party (after Slackware and Debian) and just ditch RPM for .deb - reproducible, documented - as Red Hat nearly did once before. ...

Slackware doesn't have a dependency resolver and .deb as a packaging format isn't all that different from .rpm and doesn't solve the modularity problem. So I don't see how that would help at all. Feel free to explain

Fedora's modularity mess

lkundrak — Sat, 30 Nov 2019 20:45:26 +0000

> Who of you actually knows that RPM spec files are m4 language macros?

What? No.

Fedora's modularity mess

amacater — Sat, 30 Nov 2019 20:04:20 +0000

Or just admit that, of the surviving distributions, you were third to the party (after Slackware and Debian) and just ditch RPM for .deb - reproducible, documented - as Red Hat nearly did once before. ...

Fedora's modularity mess

snajpa — Sat, 30 Nov 2019 19:42:48 +0000

You Red Hat guys should really take a hard, long and fresh look at Nix and figure out how do you go from the m4-hell you're in now to something like Nix or Guix. At their core, those technologies have solved what you guys will still be only aiming for 10 years from now.

Who of you actually knows that RPM spec files are m4 language macros?

So, you've been defining packages with a programming language all along.

Start doing it properly. You could get configuration management of those packages, auto-generated manual for all the configuration options these packages would provide and mainly, you could get the ability to install whichever versions you'd like on a single system.

And you would still be able to reproducibly manage that, which you can't now (Ansible doesn't even come close).

It would just solve all of the problems you're never going to solve up the stack. Fix the underlying actual problem that is the RPM's legacy, get it up-to-date with modern systems research and you'll be set for another 25 years as a clear market winner.

Fedora's modularity mess

epa — Fri, 29 Nov 2019 13:02:42 +0000

If the new version of X has been packaged in Rawhide then you can usually grab the source package and rebuild it. Or if not, you can take Fedora's current package for X and update it to the new upstream tarball. Either is usually pretty easy, and if not, well somebody has to do the packaging work no matter what kind of release cadence or modularity setup the distribution is using.

Fedora's modularity mess

Cyberax — Thu, 28 Nov 2019 07:19:44 +0000

The problem is not the library itself, but its reverse dependencies. There's no real way to encode which library version you want.

Fedora's modularity mess

Conan_Kudo — Thu, 28 Nov 2019 02:49:05 +0000

You actually can, RPM doesn't really care. Currently, DNF/YUM and friends encode a policy to forbid it for all but a few packages, but there's no reason we couldn't expand the scope...

Fedora's modularity mess

Conan_Kudo — Thu, 28 Nov 2019 02:47:52 +0000

Oi, ain't that the truth...

Fedora's modularity mess

dmnks — Wed, 27 Nov 2019 09:07:46 +0000

> The problem is versioning. You can't easily install two copies of the same RPM package.

Parallel installation is a solvable problem with RPM. There's no technical reason it couldn't work. In fact, Software Collections is a good example of an implementation that just adds a bunch of RPM macros and scriptlets to achieve exactly this.

But then, parallel installability is the one thing that Modularity is explicitly _not_ trying to solve :)

Fedora's modularity mess

Cyberax — Tue, 26 Nov 2019 20:01:12 +0000

The problem is versioning. You can't easily install two copies of the same RPM package.

Fedora's modularity mess

dmnks — Tue, 26 Nov 2019 14:34:21 +0000

> I think this stems from us struggling to clearly define what a "package" is. Is it a single binary/library, an application (comprising of multiple binaries), a software bundle (comprising of multiple related applications) or a bundle of bundles (a distribution)? Do we really need different concepts for each level in what appears to be a hierarchy with a clear set of rules and relationships?

Just to clarify, what I mean is that RPM packages have been historically used to deliver not only single programs and libraries, but also software bundles (as metapackages), so it seems logical to think of modules the same way.

Fedora's modularity mess

dmnks — Tue, 26 Nov 2019 14:06:01 +0000

This may sound rather naive, but I've always had the impression that we're just trying to reinvent the wheel here, "poorly". (Please take this with a grain of salt.)

Isn't this why we invented (RPM) packages and (YUM) repositories in the first place? Modules seem to share most of the same semantics; they deal with versioning, dependencies, upgrade paths (streams) etc. Why add another layer of complexity to that?

I think this stems from us struggling to clearly define what a "package" is. Is it a single binary/library, an application (comprising of multiple binaries), a software bundle (comprising of multiple related applications) or a bundle of bundles (a distribution)? Do we really need different concepts for each level in what appears to be a hierarchy with a clear set of rules and relationships?

Shouldn't we just take a step back and rethink the whole desire for "modules" in the simple context of packages and repositories first?

Fedora's modularity mess

jafd — Tue, 26 Nov 2019 09:40:39 +0000

It almost sounds as if the Modularity Team and the DNF committers live in two distinct ivory towers, perhaps a continent apart.

Fedora's modularity mess

mattdm — Tue, 26 Nov 2019 02:50:16 +0000

> Tons of fedora package bugs get no reply or an uninterested reply and gets autoclosed after a few years. This seems to be inventing a problem that doesn't exist.

I think the first part of that _is_ the problem.

Fedora's modularity mess

perennialmind — Mon, 25 Nov 2019 22:08:03 +0000

Correction: I did understate it, and what I'm trying to do is walk it back. :-)

Fedora's modularity mess

perennialmind — Mon, 25 Nov 2019 21:59:18 +0000

I don't mean to understate the complexity or the work. It would take similar effort to curate a "forward port" inverse of Debian Backports. I pictured opting in to a "FC31-29antiques" repository and pinning a set of packages to that moving target. At least then I get a hint of the complexity and relative age if I try pulling in a hodgepodge from "FC29-28antiques", "FC31-30antiques", "FC31-32backports", etc. For me, it would help set my expectations. It's not hard to see that maintaining Apache 2.2 for all eternity is infeasible, but it's hard to miss if I have to specify "Fedora9" to get it.

I'm used to managing repositories, and I like being able to fit with existing abstractions - if it's actually a good fit.

Fedora's modularity mess

sgallagh — Mon, 25 Nov 2019 19:24:43 +0000

> if they could make reaching across Fedora releases it would provide a lovely granularity of version selection.

That is in fact a core piece of the puzzle: When building a module, you select which releases of Fedora it will work on. Then on upgrade if you had indicated "I want to stick with PostgreSQL 10", you'll stay on the `postgresql:10` stream instead of moving to the `postgresql11` stream, even if it's the default stream for the next release.

The big devil in the details for upgrades is determining an appropriate heuristic for when it's okay to switch to the newer module default stream vs. stay on the previous one when upgrading. The closest we've come up with so far is the initial upgrade proposal: https://lwn.net/ml/fedora-devel/CAFoKQtx_rjF6wf_ArY8-hpfG...

I'm open to better ideas for the heuristics!

Fedora's modularity mess

langdonwhite — Mon, 25 Nov 2019 18:55:01 +0000

We always seem to leave out the use case for when Fedora is too slow.

We regularly get a new version of some tool/language/whatnot that misses the release window. Many people say "just wait 6 months for the next Fedora." Well, there a couple big problems with that. The obvious, "well, what if it can't get in then either" risk is pretty scary. However, the more subtle/less obvious problem is that many software projects are implemented and shipped in 6 months. That is a *lifetime* in web projects these days.

Without Modularity, I have to either choose to use the last rev of X or not use Fedora to develop my application.

Fedora's modularity mess

hkario — Mon, 25 Nov 2019 18:46:40 +0000

> That's a shame: if they could make reaching across Fedora releases it would provide a lovely granularity of version selection.

distribution is not just the packages, it's also all the integrations, consistent compilation options etc.
While installing FC29 package on FC31 should work in general, it's not guaranteed, let alone if it will work exactly as it did in FC29.

Fedora's modularity mess

langdonwhite — Mon, 25 Nov 2019 18:45:59 +0000

Most (if not all) of the Modularity Team members agree with you. However, that is not the way it is implemented in dnf at the moment.

Fedora's modularity mess

JFlorian — Mon, 25 Nov 2019 15:50:52 +0000

Indeed. I gave up trying to follow the ML and just *knew* that LWN would get after this eventually. Not disappointed. The wait was short and this gave a good overview.

Fedora's modularity mess

mathstuf — Mon, 25 Nov 2019 15:15:03 +0000

A way to mark a package as "maintained due to software stack" that gets a new dependency somewhere could raise a flag saying "I see you're depending on a package that was added just to satisfy dependencies. Do you want to also co-maintain that package?" to help reduce the burden for these would be good. Probably good to stop raising the flag once it has 2 or 3 co-maintainers already (or someone adopts it in a "I actually care about this package specifically" way).

Fedora's modularity mess

amacater — Mon, 25 Nov 2019 09:51:33 +0000

Finding the right terminology is always a grisly business ...

Fedora's modularity mess

bkircher — Mon, 25 Nov 2019 08:35:40 +0000

This. Thanks so much LWN for bringing a bit of sense and perspective to this mess.

Fedora's modularity mess

IanKelling — Mon, 25 Nov 2019 06:18:39 +0000

The more relevant longer requirements quote about not packaging
build-time dependencies:

> When a person decides that they want Fedora to carry a particular package
> and decides to do the work to accomplish this, it is not uncommon to
> discover that the package they care about has additional dependencies that
> are not yet packaged in Fedora. Traditionally, this has meant that the
> packager has needed to package up those dependencies and then continue to
> maintain them for anyone who may be using them for other purposes. This can
> sometimes be a significant investment in time and energy, all to support a
> package they don’t necessarily care about except for how it supports the
> primary package.

"needed to package up those dependencies" yes, of course, "and then
continue to maintain them for anyone who may be using them for other
purposes" Huh? I'm pretty sure that is not true at all. There definitely
is nothing in https://docs.fedoraproject.org/en-US/packaging-guidelines/
remotely like that. Tons of fedora package bugs get no reply or an
uninterested reply and gets autoclosed after a few years. This seems to
be inventing a problem that doesn't exist.

Yes, publishing build dependencies is hard, but its also a key part of
what makes a distro be useful, it enables people to contribute by
building and fixing software on their own machines and enables
derivatives.

Fedora's modularity mess

roblucid — Sun, 24 Nov 2019 08:45:32 +0000

The OpenSuSE distro would let me choose alternative repositories for tracking less stable versions of some projects, say web browser, desktop or kernel. A major update to new release would be handled by updating the base system entirely and then re-enabling the alternative repos after (assuming they were still required).
The case of developers building against a stable defined set of libraries would be handled by the build system, which uses an alternative root and an online package building service baselined against OS releases or Tumbleweed snapshots.
I struggle to understand what common cases the RHEL/Fedora can fulfill, in return for what appears impractically fragile complexity.
As a past sysadmin a key was reproducibility, developer systems which evolved in surprising ways, led to quality problems.

Typo reporting

tome — Sat, 23 Nov 2019 15:17:37 +0000

Holy moly, never saw that. It's bold even. Thanks Jake.

Fixed

mebrown — Sat, 23 Nov 2019 14:47:12 +0000

I see what you did there.

Typo reporting

jake — Sat, 23 Nov 2019 14:42:54 +0000

Also perhaps of interest is that it is prominently noted above every comment editing form:

Please do NOT post typos in the article as comments, send them to lwn@lwn.net instead.

jake