A year of Python in Fedora

Distribution developers do a lot of work to keep a language ecosystem working well within the distribution. It is relatively thankless work that normally only becomes visible when there is a problem or complaint. But Miro Hrončok recently put together a look back at what the Fedora Python team did during 2020. While it is, obviously, Fedora-specific, it provides something of a look inside at the kinds of things that distribution teams work on.

In his announcement post to the Fedora devel mailing list, Hrončok said that he was inspired to create the report by a similar effort for the Cool Other Package Repo (Copr) team by Miroslav Suchý. That report was well-received, so Hrončok followed suit. Perhaps it will catch on with other Fedora teams as well as for other distributions.

Hrončok is on the Python maintenance team at Red Hat as well as a member of the Fedora Python special interest group (SIG). He noted that 2020 was special in the Python world because the Python 2 series reached its end of life on January 1 (notwithstanding the commemorative 2.7.18 release in April). Support for Python 2.7 from the core developers has ceased, but the Red Hat maintainers will still be making security updates to it for Red Hat Enterprise Linux for some time to come.

There were two releases of Fedora in 2020 (Fedora 32 at the end of April and 33 at the end of October) as well as a new Python 3.9 in early October. But Fedora 32 was released with the then-current Python, 3.8, which was released the previous October. Python 3.8 had missed being included in Fedora 31 "because the release schedules of Python and Fedora were a tad too misaligned to allow a safe upgrade so soon". Going forward, that alignment has been improved, so work on Python 3.9 for Fedora 33 began well in advance of the releases for both.

Retiring Python 2 led to some of the 2.7 compatibility features being dropped in Python 3.9. Getting an early jump on testing that release made Hrončok and Victor Stinner aware that many packages would not build due to those removals. They successfully lobbied the Python core developers to postpone those removals to Python 3.10 so that Python-using projects can have a bit more time to adapt to the new world. Meanwhile, work on Python 3.10, scheduled for October 2021 began in 2020 for Fedora, targeting Fedora 35, which is also due in October.

As can be seen, there are lots of moving parts to track and test. Another area that saw improvements in 2020 was the continuous-integration (CI) testing for the language. New architectures were added for both Fedora and RHEL CI testing for Python, including s390x, ppc64le, and aarch64. Testing upstream Python regularly has helped finding bugs in Python as well unrelated problems elsewhere in the distribution. Hrončok pointed to a bug in brk() for aarch64 and problems with building using GCC 10 on ppc64le as examples.

There is a truly eye-opening number of Python packages in Fedora; the language clearly makes up a big part of the distribution:

One interesting feature that came about in 2020 is a new marshalparser tool that can be used as part of the RPM-building process to ensure that compiled Python files (i.e. .pyc files) are bit-for-bit the same if they are built from the same source. The Python marshal serializer can produce different byte code under some circumstances, as described in a Bugzilla comment, so marshalparser can be used to fix that in places where it matters. This is obviously important for reproducible builds.

Fedora field-tested the use of the -fno-semantic-interposition GCC flag, which produced a significant performance boost for the language. That flag effectively shorts out the normal lookup of a function in a library by removing the default interposition mechanism that allows a library loaded using LD_PRELOAD to override a function. This precludes the use of LD_PRELOAD for libpython as a negative side-effect, but reduces pressure on the instruction cache of the CPU leading to a performance improvement of up to 27% depending on the workload. The change was made for Fedora 32 and is now part of the upstream CPython build process. It has also been backported to earlier versions of Python in RHEL 8 and CentOS.

A patch that most distributions had long carried to have Python install its modules to /usr/lib64 (rather than /usr/lib) has made it into the CPython mainline. Work from both Fedora and openSUSE developers means that those distributions, and others such as Gentoo and Debian, can stop carrying a patch that has been around for 16 years at this point.

These are only some of the highlights of the accomplishments that Hrončok documented in the report. He also speculated on the kinds of things that would be worked on in 2021:

The report gives a nice overview of the kinds of work that every Linux distribution does to integrate a language like Python into its ecosystem. Making two big projects work well together takes a lot of effort, but one of the goals for the Fedora Python SIG is to "ensure Fedora is the best operating system for a Python developer". Other distributions might beg to differ with that, but that just means there could be a friendly rivalry that can only lead to Python on Linux getting better. That benefits everyone.