Fedora redesigns its modularity initiative

[Posted December 19, 2017 by corbet]

The Fedora Project has announced a number of changes to its modularity initiative after failing to meet its initial set of goals. "From an end-user’s perspective, Fedora will ship with two sets of repositories. One will be the traditional Fedora repositories (fedora, updates, and updates-testing) and the other will be a new set of repositories providing alternative and supplementary modules. We haven’t decided on a final name for these yet, so we will use the placeholder terms modular, modular-updates, and modular-updates-testing."

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 4:32 UTC (Wed) by jkingweb (subscriber, #113039) [Link] (4 responses)

Even after reading the source article I'm not sure what "modularity" means, exactly. Did I miss something?

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 10:23 UTC (Wed) by ballombe (subscriber, #9523) [Link]

Probably no. Modularity is a deep mathematical concept which was introduced by Taniyama and established by Wiles for rationale elliptic curves in his proof of Fermat last theorem. You cannot expect to understand it by reading a single article.
Especially one about a Linux distribution.

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 10:56 UTC (Wed) by swilmet (subscriber, #98424) [Link]

There was a LWN article about it:
https://lwn.net/Articles/680278/

There is also this documentation:
https://docs.pagure.org/modularity/

And this old wiki page:
https://fedoraproject.org/wiki/Modularity

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 11:08 UTC (Wed) by SomewhatAmazing (guest, #120306) [Link]

You're not alone in not fully understanding. It seems that ever time I find myself reading about Fedora's modularization efforts, Project Atomic, ostree, and other such efforts I find myself buckling under the load of not being fully familiar with the components and intricacies of Fedora's infrastructure and process.

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 21:19 UTC (Wed) by otaylor (subscriber, #4190) [Link]

At the heart of it, modularity is about breaking the idea that there is a single "Fedora N" package set, where every package is parallel-installable and there is exactly one version of each package. Modularity is supposed to allow one application to be based on libfoo version 2.3.x and apache-2.4, and another application to be based on libfoo version 3.1.x and apache-2.6.

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 6:12 UTC (Wed) by kloczek (guest, #6391) [Link] (1 responses)

As I wrote few times implementing this idea with only current rpm functionalities is waste of time.
Only package manager which can be used to manage variants is Solaris pkg, and first rpm must learn few tricks from pkg.

Fedora redesigns its modularity initiative

Posted Dec 24, 2017 19:23 UTC (Sun) by suckfish (guest, #69919) [Link]

Uh, rpm handles this just fine, although tools built on top of it like dnf do not.

While you mentioned Solaris pkg, you forgot self-extracting shell scripts containing tar-balls, which similarly make no attempt to constrain what hell a package can do to your system.

Doesn't matter how many times you repeat yourself...

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 7:58 UTC (Wed) by joib (subscriber, #8541) [Link] (8 responses)

Dear lazyweb, what is the relationship between the Fedora Modularity stuff and https://www.softwarecollections.org (SCL), which is already available at least on RHEL-based systems? Is modularity sort-of the "next generation" of SCL?

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 12:38 UTC (Wed) by cyperpunks (subscriber, #39406) [Link] (7 responses)

At least they try to solve the same problem. However, SCL seems to include low level stuff only, for example
various variants of GCC (5, 6 and 7) and MySQL (5.5, 5.6 and 5.7).

Modularity wants to solve more complex problems: for example offer both

WordPress 4.9 (which needs PHP 7 and MySQL 5.7) and
WordPress X.Y (which needs PHP 5 and MySQL 5.6)

and in one year offer

WordPress 5.0 (which needs PHP 7.2 and MySQL 8)

in the same distro, kind of multiple distros in the same distro, adding more modules
as time goes and still keep everything working as before.

This is hard problem.

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 19:24 UTC (Wed) by flussence (guest, #85566) [Link] (6 responses)

Gentoo's solution to this is called “slotted packages”, and it's not as hard as Fedora makes it out to be. In fact all those packages you listed are already installable in parallel here (except for mysql point releases, which I'm not sure of the need for).

Fedora redesigns its modularity initiative

Posted Dec 20, 2017 20:05 UTC (Wed) by cyperpunks (subscriber, #39406) [Link] (5 responses)

Can you explain how this works in Gentoo?

Fedora redesigns its modularity initiative

Posted Dec 23, 2017 21:00 UTC (Sat) by flussence (guest, #85566) [Link] (4 responses)

Packages that share the same base name but don't overwrite each other's files can have optional slot versions added, usually a number matching the upstream major version or .so version. For all intents and purposes they act as separate packages sharing the same base name, so for example you can install the "postgresql" package and get whatever the latest version of postgres is, but you can also ask for "postgresql:10" to keep version 10 without accidentally installing 11 when that's released.
Debian has something vaguely similar, but it's just a numeric suffix arbitrarily added to package names, so you end up with a lot of libkdefoo4 and 5 names, but also more than a few 6 or 2 or 5-5 with no correlation to the actual version number given by upstream.

There's also a sub-slot notation to indicate when different versions of a library would conflict but have incompatible ABIs, so that their dependents can be rebuilt automatically. Those are things like icu, ffmpeg/libav, mysql/mariadb, openssl/libressl. The combinatorial explosion of possibilities with packages like these is usually too much for other distros to keep up with and leads to flag days, CVEs going unfixed for weeks/months and other unpleasantness; they suffer from a lot of the same problems static linking does.

Fedora redesigns its modularity initiative

Posted Dec 24, 2017 2:10 UTC (Sun) by pabs (subscriber, #43278) [Link] (3 responses)

The other way to go is GoboLinux, every version of every package gets installed in a separate subdirectory and symlinks or bind mounts are used to create the FHS tree.

Fedora redesigns its modularity initiative

Posted Dec 24, 2017 21:14 UTC (Sun) by flussence (guest, #85566) [Link] (2 responses)

GoboLinux does some pretty neat things and I feel like its filesystem layout is better than the traditional Unix one in a lot of ways. In all the years it's been around, nobody else has bothered to try anything similar. Is it just inertia?

Fedora redesigns its modularity initiative

Posted Dec 31, 2017 10:49 UTC (Sun) by jospoortvliet (guest, #33164) [Link] (1 responses)

I guess it is a matter of the amount of pain solved vs amount caused by the new approach plus how much work migration is. The question is if all that motivates people to do the work ;-)

It is like anything in life, really... certainly some problems are solved but they are probably just not painful enough. I think the biggest factor might be the work of implementing it in a way that makes migration to the new way feasible. If it can't be done incremental but needs a flag day it is often not really possible. Why didn't you migrate to Gobo? Right, this one advantage isn't worth the migration cost. True for all of Fedora/Linux too (of course if we all wanted to migrate we collectively could improve gobo and reduce the cost).

My point: it most likely isn't a motivation or technical or emotional matter but a cost/benefit matter. And do count opportunity cost in: if we put these same efforts in something else wouldn't it help us forward more in the end?

cheap fork utopia

Posted Jan 4, 2018 10:01 UTC (Thu) by Garak (guest, #99377) [Link]

My longtime advice addressing this dynamic is to extremely reduce the cost of forking, and maintaining a fork of the entire distro. I think this can be done easily enough, but isn't because of an irrational narrative that I have heard a lot amongst LWN articles over the years. I admit the world is a big place, and perhaps the animosity towards the inefficiency aspects of various historical forks is more justified than I estimate. But it seems to me there are, or could be, or ought to be enough FOSS developers out there to easily sustain a comparatively vast multitude of forks. Lots of differently colored bikesheds for lots of different people would be the corollary. I.e. in this case, if it were trivial (tools provided, utterly simplified) to fork Fedora as a whole (or more realistically as a subset), and make any arbitrary (or even none) change/s, then we might see someone maintaining such FedoraForkModdedWithNewRootFSLayoutItch distro. Sure, lots of corner cases amongst the package library, so it starts as a fork that abandons the 90% of packages least popular. Then if it attracts the attention of people who try it out and find it worthwhile, that snowball/crowdsource/criticalmass momentum finishes the job, perhaps to the extreme of convincing the upstream to pull/merge the fairly radical change.

$0.02