Fedora revisits the Git-forge debate

A seemingly straightforward question aimed at candidates for the in-progress Fedora elections led to a discussion on the Fedora devel mailing list that branched into a few different directions. The question was related to a struggle that the distribution has had before: whether using non-free Git forges is appropriate. One of the differences this time, though, is that the focus is on where source-git (or src-git) repositories will be hosted, which is a separate question from where the dist-git repository lives.

Background

The dist-git repository is the place where the distribution does most of its work to create and test packages, as described by Tomáš Tomeček back in May 2020. It works reasonably well for that, especially for those who are familiar with using it, but it is quite different than the Git repositories used by the upstream projects. It also has some seriously rough edges:

For some tasks, the workflow is just fine and pretty straightforward. But for the other, it’s very gruesome - the moment you need to touch patch files, the horror comes in. The fact that we operate with patch files, in a git repository, is just mind-boggling to me.

A blog post by Tomeček that introduces source-git gives a bit more information about dist-git. Each Fedora package has a dist-git repository that provides everything needed to create the binary RPM that gets shipped to users. That includes the RPM spec, source code, Fedora-specific patches, tests, and so on.

The dist-git repository has been (and still is) hosted on the Fedora Pagure code-hosting system, but maintaining Pagure had become something of a headache for the Red Hat Community Platform Engineering (CPE) team that manages the infrastructure for Fedora. Back in January 2020, that led to a gathering of requirements for what was needed in a Git-forge solution that would serve the needs of all of the distributions that Red Hat manages (Fedora, CentOS, and RHEL). The process ended acrimoniously after the botched announcement that CPE had chosen GitLab in what many in the Fedora community saw as a fait accompli. Roughly a year and a half later, a hosted GitLab service for Fedora was announced, which included some of the proprietary "ultimate tier" features, but Pagure would still be sticking around for the time being:

I know some of you are wondering what this means for Pagure.io and dist-git. The Community Platform Engineering (CPE) team will continue to run them in a supported mode as we do now. With the availability of GitLab, CPE and the Source Git SIG will continue to explore the FESCo [Fedora Engineering Steering Committee] feedback on what could make dist-git viable on GitLab in the future. We will also look for ways to integrate Gitlab into more of our tooling to provide options for the community. Right now though, this just adds another option for you to use. If you want to keep your work on Pagure, then by all means do.

The source-git effort (which is part of the Packit project) is a mechanism to make Git repositories for Fedora packages that mirror their upstream counterparts, to the point where the upstream developers will feel completely at home. It turns out that many Fedora packages were already being created that way, but in an ad hoc fashion. So source-git adds tooling to use the distribution-specific patches and build configuration (e.g. RPM spec files) to create a source RPM that can be used in dist-git to create the binary RPM for users. Currently, source-git is integrated with GitLab and GitHub, but not Pagure.

Question

It is against that backdrop Michael Catanzaro asked the candidates for FESCo and the Fedora Council about their thoughts on hosting source-git repositories: "do you support allowing Fedora src-git repositories to be hosted on gitlab.com, which a proprietary software git forge?" He noted that there are at least two options for an open-source solution, Pagure and the open-source version of GitLab, so he wanted to know where the candidates stood on that question. It turns out that the question is not entirely straightforward because it stems from something of a misunderstanding of the purposes of the source-git project—and how it interoperates with dist-git. It was also relatively easy to misunderstand what Catanzaro was actually asking, so the still unresolved question of where dist-git will be hosted was also re-litigated to a certain extent.

For example, FESCo candidate Fabio Valentini said that he was opposed to moving away from Pagure to "a proprietary solution from a vendor with an 'open core' business model", in part because it "sends the wrong message to the FOSS community". But he quickly realized that he had misunderstood the question; he has some concerns about source-git working with less popular forges, but is not completely opposed to hosting those repositories on GitHub and GitLab.

Fedora project leader Matthew Miller, who is not one of the candidates, took issue with the idea that Fedora might be sending the wrong message with its forge choices. There is a need to be pragmatic on certain choices (e.g. binary firmware) so that people can actually use the distribution on their machines, he said. Meanwhile, much of the open-source community has already chosen to use the Git* forges, so it makes sense to meet them where they "live":

More than that, though, if what we do sends a message... the message we are sending right now isn't working. People _aren't_ showing up in droves to help build Pagure. People aren't even showing up in significant numbers to _use_ Pagure outside of the Fedora space. And, again, that's not because Pagure isn't good — it is, with a great fundamental design where it's git all the way down. But our use of it hasn't changed the world, I don't see that improving. It doesn't _actually_ advance our mission and vision at all for us to be symbolically right if it doesn't change people's behavior, and... it doesn't.

Whenever this topic has come up, I've see lots of this, except people don't say the first part out loud: "Of course _I_ use GitHub for most of my stuff, because of all of the advantages — but Fedora, Fedora should never." We can't let ourselves be held by that.

I don't think Gitlab open core is ideal. But I think it's closer than GitHub. And I deeply believe that free software and real open source is _just plain better_ as a model, and I think they'll eventually realize that too. I think we could have a LOT more impact working WITH GitLab to move towards an all-open model than we will continuing on the current path.

In a response to Catanzaro's question, Miller made it clear that he believes that the answer to that "_has_ to be 'yes', and not just for GitLab but also GitHub". If the source-git repositories are going to be close to the upstream repositories, they need to be able to be on the same places that host those projects.

Beyond that, he segued into the question of dist-git as well. He noted the long list of open Pagure issues and said that the project is in an "emergency maintenance-only state". He pointed out that historically Fedora has been unable to set up and run an open-source GitLab instance, so that is not really a viable alternative either. He would much rather there be an open-source option, but does not really see one, though he hopes that GitLab could perhaps become that option some day. It is a matter of resource allocation, he said, and ended his message with a list of multiple other projects he would rather see the project focus on.

Catanzaro was somewhat surprised by that response, in part because he did not quite see how source-git and dist-git would work together.

src-git is exclusively used for downstream Fedora packaging, so I don't expect upstreams to be interested at all, unless upstream developers are also the Fedora packagers, right? I'm also assuming that direct commits to dist-git will be blocked if src-git is enabled -- because otherwise how would we keep them in sync and avoid breaking src-git? -- and the only way to commit to dist-git would be via src-git. Is that right too?

Miller said that there are quite a few upstream developers who are also Fedora packagers, so they will likely be interested in these source-git repositories. In addition, the Packit Service is a component that will keep the two repositories in sync so that contributors can use whichever of the two workflows they are most comfortable with. As David Cantrell put it: "No one is required to use it. If you are comfortable with using dist-git as you do now, that's fine."

Packit developer Hunor Csomortáni cautioned that source-git is still an experimental tool, though there is a goal to add it as an official option for Fedora packages:

All the work that is ongoing in this space is experimentation and custom tooling developed to support select packages. Yes, we (the Packit team) have a goal to propose to the Community to adopt the workflow as an alternative way of doing packaging work and allow source-git repositories to be hosted within Fedora realms, but that proposal has not been written yet.

For now, the question is only about where these experimental repositories can be hosted, but if source-git gets adopted as an option, it would be preferable to have both source-git and dist-git in the same forge, Csomortáni said. Currently there are "mixed signals" about where that would be, however. It would make sense to combine the repositories on a single forge for a few different reasons:

To have a better, less fragmented developer experience, lower maintenance and administration costs, and to ensure Community control over them, I think it would make sense to host official source-git repositories next to dist-git repos, in a sibling namespace, and be served by the same Git-forge.

Catanzaro was unconvinced by Miller's assertion that Fedora could not run its own open-source GitLab instance. "If GNOME and KDE and freedesktop.org and Debian and Purism can all do it, I'm pretty sure Fedora can too." Miller said that he would love to be proved wrong, but Stephen John Smoogen posted a message describing the problems that have cropped up when trying do so in the past. Those efforts have run aground on the same kinds of problems that plague all projects: "Getting an open source gitlab is not impossible. It just takes a lot of free time, systems and work done somewhere to make it happen."

No real conclusions came out of the discussion, but it would seem that at least the ideas behind source-git and its goals will be better understood. There is still the lurking problem of dist-git moving to the proprietary GitLab instance that CPE has set up; it is likely that the other open-source alternatives (Pagure or open-source GitLab) are not going to make the cut. That is understandably frustrating for many free and open-source software advocates in Fedora (and elsewhere), but also may be the least-bad option at this point. There are other important goals that the distribution has, in the eyes of some anyway, so putting energy into those things may be a better way for the project to advance—time will tell.