Fedora SIG changes Python packaging strategy

Fedora's NeuroFedora special-interest group (SIG) is considering a change of strategy when it comes to packaging Python modules. The SIG, which consists of three active members, is struggling to keep up with maintaining the hundreds of packages that it has taken on. What's more, it's not clear that the majority of packages are even being consumed by Fedora users; the group is trying to determine the right strategy to meet its goals and shed unnecessary work. If its new packaging strategy is successful, it may point the way to a more sustainable model for Linux distributions to provide value to users without trying to package everything under the sun.

NeuroFedora SIG

The goal of the NeuroFedora SIG is to make it easy to use Fedora as a platform for neuroscience research. The SIG provides more than 200 applications for computational neuroscience, data analysis, and neuro-imaging, as well as a Fedora Comp-Neuro Lab spin with many applications for neuroscience preinstalled.

On June 6, Ankur Sinha started a discussion on the NeuroFedora issue tracker about changing the SIG's packaging strategy. He said that the group was no longer sure that providing native packages was the right way to serve the target audience:

A lot of our packages are python libraries but I'm seeing more and more users just rely on pip (or anaconda etc.) to install their packages instead of using dnf installed packages. I was therefore, wondering if packaging up Python libraries off PyPI was worth really doing.

Sinha said he had discussed the topic at Flock, which was held in early June, with Karolina Surma, a member of Fedora's Python SIG. Surma said that the Python SIG focuses on core Python packages, such as the Python itself, pip, and setuptools; Red Hat's Python developers had attempted to automatically convert all of the Python Packaging Index (PyPI) to RPMs in a Copr repository, but Surma said the resulting packages were not of sufficient quality to recommend. For Python libraries, it was up to packagers to determine if the use case required system packages.

Sinha tried to capture some of the reasons for and against providing native Python packages via the SIG. Since most of the upstream documentation refers to pip for installing modules from PyPI, Sinha thought it was unlikely that most users would turn to DNF for Python libraries. Pip also allows users to install different versions of libraries in virtual environments, which cannot be done with DNF. On the other hand, Sinha said that distribution packaging is important because it allows packagers to push out updates in a timely manner. "When installing from forges directly, there doesn't seem to be a way of users being notified if packages that they're using have new updates."

Benjamin Beasley said that the main reason to package libraries should be to support packaged Python-based tools and applications. He added that he saw value in having packages for command-line tools so that users could use them without having to care about what language the tools were written in or knowing how to use language-specific tools or repositories. Distribution packaging could be reserved for "software that is difficult to install, with things like system-wide configuration files, daemons, and awkward dependencies". If a Python library is not needed by other packages, on the other hand, there is not much justification for packaging it.

Testing and documentation versus packaging

Sinha asked if it even made sense to package applications that could be installed from PyPI and suggested that for some pip-installable packages it might be better for the SIG to provide documentation on installation rather than maintaining a package. For example, a pip package might require Fedora's GTK3 development package (gtk3-devel) to be installed in order to build and install correctly. He wondered if the SIG should package that application or just maintain documentation on how to install it with pip on Fedora.

After a few days, Sinha put out an update with his plan; he would create a list of Python packages of interest to the SIG, write a script to see if they could be successfully installed using pip, and orphan those that were not required to be native packages. He also asked again for input and specifically tagged SIG member Sandro Janke.

Two days later Janke appeared and agreed that it made sense to reduce the number of packages maintained by the SIG, as the current workload was challenging. He said that it would be a good idea to standardize on one tool (e.g. pip or uv) to recommend to users. "Supporting (and understanding) one tool well is better than supporting many tools half-heartedly". He was less convinced that the SIG should take responsibility for testing and documentation of Python modules that were not maintained by the SIG. "When it comes to local installation my first instinct is that the onus is with the user."

After gathering feedback from the SIG, Sinha floated a general guideline of "prioritizing software that could not be installed from an upstream index". He said he would post to Fedora's python-devel mailing list to gather feedback, but it was ultimately up to the NeuroFedora SIG since it would have to do the work.

On June 25, Sinha started that discussion on Fedora's python-devel list. To lighten its packaging load and better address user needs, Sinha said that the SIG had discussed a new strategy. The group would prioritize packaging software that was either difficult to install or completely unavailable on PyPI. If a package was installable with pip, then it would be dropped from Fedora.

Instead, the group would continue to test that software it cared about could be installed using pip; the SIG would provide documentation about software status and additional information required to get it working. NeuroFedora contributors would also report any problems to the respective upstreams and submit fixes when possible.

Michael Gruber said that the proposal was coming at just the right time. It would help to reduce the set of RPM-packaged applications, which would be beneficial when Fedora has to do a mass-rebuild of packages due to a Python change. Gruber also liked the idea of having a method to test if PyPI packages were installable on Fedora, and that might be beneficial to other language ecosystems as well.

Making a list

No one seemed to object to the plan, and Sinha wrote a script to find the leaf packages—those that are not required as dependencies for other packages—among the Python packages maintained by the SIG. He posted a list of more than 110 packages that the group could consider dropping. Beasley went through the list and provided a summary of packages that should be kept for various reasons, including some that were not Python packages but those that offered Python bindings. That brought the list down to nearly 70 packages, including a handful that had already been orphaned or retired for the upcoming Fedora 43 release.

One outstanding question is which versions of Python to test modules against. Each Fedora release has a default version of Python—version 3.13 in Fedora 42—but older and newer versions of Python are available from its repositories as well. For example, users can install Python 3.6, 3.9, 3.10, 3.11, 3.12, or 3.14 alongside the default version if they need an older or newer version for whatever reason. Janke noted that upstream maintainers are often slow to adapt to changes in the latest Python releases. He asked if the SIG should support the "bleeding edge" or the oldest default version in a supported Fedora release. As of this writing, that would be Fedora 41, which also uses Python 3.13.

On July 5, Sinha said that he had implemented a pytest-based checker to verify whether pip-installed packages install cleanly on a Fedora system. He also has an implementation of a searchable package list (sample screenshot here) that will display software that has been tested by the NeuroFedora SIG. The SIG has not deployed the changes yet, or finalized the list of packages to be dropped, but it seems likely this work will land in time for the Fedora 43 release in September.

At the moment, this strategy is limited to the NeuroFedora SIG, and will only impact a small percentage of Fedora packages. However, this hybrid approach of packaging the software that most benefits from being included in the distribution, while testing and documenting packages that can be installed from the language repository directly is something that might be worth examining on a larger scale and beyond the Python ecosystem.

The number of people interested in maintaining native packages for Fedora (and other distributions) has not kept up with the growth of software available in various language ecosystems. NeuroFedora's strategy might be a good way for Linux distributions to support developers and users while lessening the load on volunteer contributors.