Fedora discusses Flatpak priorities
Differences of opinion, as well as outright disputes, between upstream open-source projects and Linux distribution packagers over packaging practices are nothing new. It is rarer, though, for those disputes to boil over to threats of legal action—but a disagreement between the Open Broadcaster Software (OBS) Studio project and Fedora packagers reached that point in mid-February. After escalation to a higher authority, things have been worked out to the satisfaction of the OBS project, but some lingering questions remain. How Fedora should prioritize Flatpak repositories, how to handle conflicts between upstreams and Fedora packagers, and the mechanics of removing or retiring Flatpaks all remain open questions.
Flatpak
Flatpak has been in development for more than a decade, and its earliest origins date back to 2007. Originally it was called xdg-app, and is designed to bundle individual applications and their dependencies into an easy-to-install format that works on any Linux distribution.
It is more than just a packaging format, though. Flatpaks are sandboxed, which means that an application installed as a Flatpak can be limited or restricted entirely from accessing user files, graphics sockets, networking, devices, and so forth. They are also self-contained, so installing an application using a Flatpak won't disrupt other applications and libraries on a user's system. Flatpaks can be distributed as OStree repositories or Open Container Initiative (OCI) images.
Flatpak is meant to provide a number of benefits for projects, Linux distributions, and users over Debian packages, RPMs, and other distribution-specific formats. A project can target a single packaging format and distribute the software with the project's preferred defaults. Users don't have to depend on their distribution to package the application, or for distribution packagers to stay up to date with the most recent releases. The upstream can release major updates at whatever cadence it pleases, for example, and users don't have to wait for the next distribution major release for the upgrade.
Projects can also make use of runtimes that provide libraries and toolchains, such as the GNOME and KDE runtimes, which free developers from having to package all those dependencies or worry about things like "which version of GTK or Qt is available on the user's system?" Linux distributions, and their users, benefit from a wider selection of software without the need for the distribution to package everything under the sun.
Adoption of Flatpak has been on the upswing in the past few years. Some are even pushing for it to be the de facto method for installing software on the Linux desktop. Work on Flathub as a central repository for Flatpaks started in 2017, and it passed the one million active user mark last January. Flatpaks are being prioritized by GNOME as a way for distributing GNOME software, and it is the preferred and most expedient way to install applications on image-based distributions like Fedora's Atomic desktops, Bluefin, Aeon Desktop (based on openSUSE), and others because Flatpaks can be installed in the user's home directory and do not need to be installed system-wide. Fedora has its own Flatpak repository, which contains Flatpaks built from Fedora RPMs and distributed as OCI images. Fedora serves its Flatpak images and Linux container images from the same registry, and provides its own runtimes for applications rather than using the Freedesktop, KDE, or GNOME runtimes.
The complaint
OBS Studio is a popular open-source, multi-platform project for live streaming and screen recording. The project provides its own Flatpak package via Flathub and recommends installing that package prominently on its download page. Fedora enables the Flathub repository in the GNOME Software package-management application, but also provides OBS Studio as an RPM and as a Fedora-packaged Flatpak in the Fedora Flatpak repository. Users searching for OBS Studio using GNOME Software will be offered the Fedora Flatpak as the default, then the Fedora RPM, and finally the OBS Studio official Flatpak from Flathub.
OBS contributor Joel Bethke created
a ticket with the Fedora Flatpak special interest group (SIG) on
January 21. He complained that the OBS Studio Flatpak from Fedora
was "poorly packaged and broken
" which, in turn, led users to
complain to the upstream project because they thought they were
getting the official OBS Flatpak. The initial report did not specify
what those issues might be, but later in the thread Angaddeep Singh
pointed to a bug
filed with OBS Studio that complained about problems using the H.264
encoder, and a complaint
in the OBS forum that the Twitch chat and streaming interface
were missing. In both cases, users were using Fedora-packaged versions
of OBS and complaining upstream about problems that were not present
in the OBS builds. Bethke later
said that the project had received complaints specifically about
the Fedora Flatpak that included regressions due to the Qt version
bump, hardware acceleration not working, OpenH264 encoder failures,
and even failure to launch the application at all. He said that the
OBS Project's Flatpaks, and "in some cases, even the RPM
" did
not have the same issues. The reason Bethke requested
that the Flatpak either be removed, or that the Fedora Flatpak be
clearly identified as a third-party package, is that users were not
even aware they were using Fedora's instead of the OBS project's. He
also wanted an explanation why Fedora would publish its package at a
higher priority than OBS's official builds.
In response, Michael Catanzaro proposed
in a ticket for the Fedora
Workstation's working group that Fedora deprioritize or
remove Fedora Flatpaks from GNOME Software. He made the case that Fedora
Flatpaks have been "a significant source of quality problems and
have frankly been generally unsuccessful
". Users want Flatpaks
created by upstreams, but are confused by the way they are displayed in
GNOME Software. He recommended that the priority be changed to display
software from Flathub first, then Fedora Flatpaks, and finally Fedora
RPMs.
On February 4, Catanzaro posted an update
to his ticket with notes from the Workstation working group's
meeting (minutes). He said that the working group had not reached any final
conclusions, but had consensus on a few points. One was that
Fedora Flatpaks "are generally the worst software source for users
to install
", so they should not have the highest priority in GNOME
Software. He also acknowledged that there are problems with the user
interface for GNOME Software, because it was easy for users to be
confused about the source for software. However, he had some critical
words for the OBS Studio project's packaging: he preferred the RPM
version to the OBS Flatpak because OBS version "notably depends on an
EOL runtime that no longer receives security updates
".
Currently OBS Studio's Flatpak depends version 6.6 of the KDE/Qt
runtime, which reached end-of-life when the 6.8
runtime was made available in October 2024. The KDE project
only maintains branches with the two most recent versions of Qt
(currently 6.7 and 6.8), as well as long-term-support branches for Qt
5.15. This means that a 6.x version of the runtime has a shelf life of
about a year. According to the
list
of vulnerabilities for Qt, there seem to be a few vulnerabilities
that would impact the 6.6 runtime, but it's unclear if any would
present a real security problem for users of OBS Studio. Bethke replied
that OBS Studio had well-documented
reasons for not updating the KDE runtime due to regressions in
Qt. "We made the choice to have a functional application, report
the bugs upstream, and update the dependency once they have been
fixed.
" He also said that the project had the impression that the
concerns of OBS Studio as an upstream were being dismissed.
I have said this before, but I will say again it should not be upstream's responsibility to track and report on downstream packaging issues, and the fact that we have and it's been ignored has been frustrating. I still don't have a good understanding on why Fedora is so adamant about repackaging and redistributing Flatpaks wholesale without proper testing or validation.
He also pointed out that Yaakov Selkowitz, who owned the obs-studio
Flatpak package for Fedora, was maintaining hundreds of packages. "In what
world is a single person able to maintain, test, and support, that
many packages and applications?
" The Fedora Flatpak initiative, he
said, seemed poorly planned, badly implemented, and largely
unsustainable.
Catanzaro agreed
with several points that Bethke made, but ultimately said,
on February 12, that allowing a runtime to reach end of life
"is unacceptable and indicates terrible
maintainership
". That seemed to be the proverbial straw that broke
the camel's back, and prompted Bethke to issue
a formal request that Fedora remove any OBS Studio branding
from the distribution and threatened legal action if Fedora failed to
comply.
Resolution
On February 14 Fedora Project Leader (FPL) Matthew Miller stepped in to deescalate the situation, which brought Bethke back to the table to talk. Bethke said that, as an outsider, it was difficult to navigate Fedora's processes and groups when he didn't even know what some of the acronyms stood for or which groups to engage with.
I was asked to report things to several different locations by, supposedly, representatives of Fedora.
It was a frustrating process to even know who I should be talking to about these issues, and when the people who were talking to us decided to dismiss the issue and insult us instead, we were left with what we felt was no choice but to take action. We are always open to discussion as long as it is in good faith, and still are.
Fedora contributor Adam Williamson summarized the situation and sympathized with Bethke. He also noted that the request to assert OBS Studio's rights and remove the OBS Flatpak was being handled as a high-priority request. Unfortunately, he said, there was no current process for removing a Flatpak from Fedora. Indeed, based on the discussion in the ticket for Fedora release engineering there was confusion about whether it was actually possible to remove a Flatpak image from the index, and how to notify users of such an event. Following a meeting with the Flatpak SIG and Miller, Bethke withdrew the request to remove or rebrand the OBS Studio Flatpak on February 18, which reduced the urgency to find a solution. Bethke suggested that the ticket be used to track the technical issues with the Fedora Flatpak, and asked that Fedora make it clearer for upstream projects to report bugs.
Even though the OBS situation seems to be resolved, Fedora is left with a number of things to work through with regard to Flatpak. The first and most obvious, of course, is creating a policy and method for removing Flatpaks from its repository. It is somewhat surprising that the project rolled out a method of distributing packages without a plan to address the inevitable need to quickly remove a package from the repository when needed.
Priority
The next open question to tackle is the priority order for RPMs and various flavors of Flatpak in GNOME Software. When Fedora initially accepted the "unfiltered Flathub" proposal for Fedora 38, the Fedora Engineering Steering Council (FESCo) rejected the idea of prioritizing Flathub software over Fedora sources. (LWN covered this in 2022.) It's possible that attitudes have changed in the past couple of years, particularly given Flathub's increasing popularity and some of its newer practices for allowing projects to verify Flatpaks as being provided directly by the project. As Williamson suggested in the OBS Studio ticket, it may be time to rethink the role of distribution-specific Flatpaks:
The story we should be reading from all the comments on this from folks who find it inherently surprising that Fedora would offer its own flatpaks over ones from flathub is that flathub is successful: for a lot of people who buy into the flatpak/snap/appimage-style approach, flathub is their primary trusted source for flatpaks and it's where they assume their flatpaks will come from 'by default', especially if they've clicked through a "turn on third-party repos" dialog. And there are genuine reasons for this - in general the flatpaks from flathub work, people like that they can recommend them to others regardless of distribution and expect a consistent experience, and people like the involvement of upstream developers in many flathub flatpaks.
However, Catanzaro's effort to give Flathub packages top priority
does not seem to be finding much success. On February 18, he
said
that the Workstation working group had discussed the topic for a third
time, and it had still not reached consensus. "I'm no longer
confident that the Working Group will make any changes here.
" He
also
said that he planned to continue arguing to remove the Fedora
Flatpak repository—which would make Fedora RPMs the top priority,
followed by Flathub if it is enabled. But, he was beginning to fear
that the working group would not accept his proposal. On
February 25, he wrote
that the group had discussed it yet again, and he thought that the
group was evenly split on what to do. A possible option would be a
proposal that Fedora Flatpaks would need to be owned by the package's
RPM maintainer, which would result in "a drastically reduced set of
Flatpaks
".
Finally, it seems clear that there is work to be done to make it
easier for upstream projects to engage with Fedora when they have
questions or concerns about how their work is packaged. Justin Wheeler
said
that he had been working with the Fedora Council on an "explicit
statement on our 'upstream first' policy
" and had added
some language to specifically address collaboration and clear lines of
communication. However, that does not provide any concrete changes
that would help upstreams navigate and resolve their problems with
Fedora.
There is also the question of whether Fedora should honor requests from an upstream not to package software at all. Jordan Petridis wrote about an interview that Miller did recently where he discussed the OBS issue and an older kerfuffle about packaging the Bottles project. Petridis took issue with several of Miller's comments about Flathub's review processes, and complained that Fedora does not live up to its own standards around naming and branding that Miller discusses.
In the interview Miller talks about Fedora's branding rules and asking
that projects do not call their derivatives "Fedora". However, the
Bottles project asked Fedora not to package its software, or to call
it something else if it was going to package the software. After
initially considering the request and planning to drop it in
Fedora 38, the packager decided to go ahead and provide Bottles
RPMs anyway, since the upstream only provides Flatpak packages. And it
is still called Bottles in Fedora, several years later, albeit with a
disclaimer that it is not an official package and a link to
Fedora's bug tracker rather than upstream's. Petridis said that it was
wrong that it had to come to legal threats to get Fedora to "treat
application developers like human beings and get the Fedora packagers
and community members to comply
".
Fedora, and other distributions, might want to consider creating a simple and straightforward process for upstreams to report problems and discuss concerns—one that does not require navigating a twisty maze of SIGs, working groups, and multiple bug or issue trackers. Even then, however, upstreams may not wish to participate in the process at all and would prefer to be left to be the sole distributor of their software. At least under their own brand.
The changes and compromises that go along with distribution packaging are not always compatible with the vision that upstreams have for their software. This is especially true with applications like OBS Studio that make use of specialized hardware and have dependencies on multimedia codecs or other libraries that distributions like Fedora do not ship, or ship in modified form for legal reasons. Distributions should consider when it is appropriate to leave software distribution to an upstream project that is uninterested in—or even hostile to—having distributions provide native packages.
The landscape has changed dramatically since Linux distributions were at the center of the universe for free-software distribution. There is still immense value in the Linux distribution packaging model, but it is not a good fit for all cases. Recognizing that, and accepting it, might improve things (and reduce drama) for everyone involved.
