An uproar over the Fedora Git forge decision

We last looked in on the question of a Git forge for Fedora at the end of January—which seems like nearly a lifetime ago, but is, in truth, only around two-and-a-half months back. At that time, requirements were being gathered for an open decision-making process that would seemingly play out with lots of community participation. That is not at all what transpired, however, and much of the Fedora community feels that its needs have not been taken into consideration. There are a number of lessons that can be learned from all of this.

Announcement woes

After a lengthy requirements-gathering thread on the fedora-devel mailing list back in January, things went rather quiet until the March 28 posting of "CPE Weekly", which is a newsletter that covers the activities of the Red Hat Community Platform Engineering (CPE) team. That is the organization behind the Git forge effort; tucked into the end of the newsletter was the "announcement" that the team had chosen GitLab as the forge for Fedora and CentOS, while still continuing to run Fedora Pagure with community assistance for projects that want to use it as their Git host.

There was, it seems, a plan to announce the decision on the Fedora Community Blog (and on Blog.CentOS.org). But, as noted by CPE manager Leigh Griffin, that did not happen due to "unavailability and illness" of a volunteer who was going to do it, which meant the first mention of the decision ended up in the already scheduled newsletter. The net result, as Neal Gompa pointed out, was that "the delivery of this decision sucked".

Beyond that, though, Gompa went through the user stories that had been gathered as part of the decision-making process at great length; he said that many of them could not be satisfied with any open-source solution, so in some sense he is not surprised that CPE looked beyond Pagure. But many of the requirements identified also make it clear that the open-source GitLab Community Edition (CE) would not fulfill the needs listed, so he thinks that CPE is really aiming for the proprietary Ultimate/Gold edition.

As might be guessed, Griffin largely disagreed with much of Gompa's point-by-point analysis; he also said that no decision had been made on which of GitLab's offerings would be used. The requirements were gathered from multiple stakeholders within Red Hat, including Fedora, CentOS, Enterprise Linux (RHEL), and CPE itself, but were generally not really evaluated, just collected: "It was not our place to question valid use cases or requirements from our stakeholders."

Julen Landa Alustiza was surprised that the decision was made without following some of the steps that were promised. He pointed to the original blog post that announced the effort, which mentioned things like "a live video call and associated IRC meetings will be held and advertised in advance to discuss the requirements, talk about concerns and address any questions". None of those things happened, he said. Griffin agreed:

Others were surprised by how the process played out as well. David Kaufman wondered about the role of the Fedora Engineering Steering Committee (FESCo), Fedora Council, and the CentOS Governing Board in the decision. As Griffin pointed out, though, those organizations do not run the infrastructure in question—CPE does. As Miro Hrončok put it:

Fedora project leader Matthew Miller, who is also a member of the council, described the council's involvement:

He quoted Hrončok's caution, noting that the council is in the same position as FESCo: advisory-only. But he is optimistic that things will work out well:

While that conversation was ongoing, Griffin posted the (delayed) announcement of the decision, which set off another enormous thread. Many of the same complaints (and defenses) were heard, but the overarching unhappiness from multiple community members was perhaps best summed up by Hrončok:

Requirements lost

But it turns out that some of the requirements that the Fedora community thought it was providing to the process got watered down along the way. In particular, the first two items on the proposed requirements list, which were that the solution be self-hosting and also be free and open-source software (FOSS), were never quite communicated to CPE as requirements. Along with others, Kevin Kofler raised the point, noting that considering the GitLab-hosted solutions, which are proprietary, "should be an absolute no go". Michael Catanzaro agreed:

That communication failure was teased out in a sub-thread that discussed the staffing levels of the CPE team. Quite a ways down in a lengthy message from Adam Williamson, he pointed to the difference between the proposed list and the final summarized list that was used to make the decision:

Griffin noted that the list submitted to CPE is not the same as the one proposed (even though earlier Griffin had mistakenly said that it was). After a request from Williamson, Fedora Program Manager Ben Cotton posted the final list, which did not, in fact, contain the self-hosting and FOSS requirements. Williamson was understandably dismayed:

Cotton said that he "included a reminder that FOSS is always our strong preference where viable" when he sent the list, but he apologized for not making it into a "user story" as part of the requirements. In the previous message, he had also apologized for not posting the final list at the same time it was sent, which might have caught the problem earlier. As is perhaps not surprising at this point, Griffin acknowledged that CPE simply treated the FOSS and self-hosting items as a "strong preference" and not a requirement, but: "It will certainly influence our decision to opt for Gitlab CE for the Fedora preference to stay open source."

There was some additional unhappiness expressed when the next "CPE Weekly" was posted on April 4. It contained an apology for the communication breakdown and a request to continue engaging with the team:

Randy Barlow said that it was not the only thing that could have been done. "You can hold off on making a decision, and follow an open process instead." In addition, he was skeptical that the community could really benefit from engaging with CPE "when it has demonstrated that it does not meaningfully engage with the community". Chris Murphy agreed:

Griffin, however, clearly feels that the decision is the right one, no matter what flaws there were in the process. "Our choice of Forge that CPE will support is valid from the requirements we received". Part of the disconnect may be that Griffin and CPE see the council as the "engagement point" with Fedora, while the community has always been rather more freewheeling than that. In addition, FESCo has typically gotten involved in changes of this nature, as Hrončok, who is a FESCo member, noted in a comment on a council issue ticket:

That council issue resulted in an April 15 statement from the council about the Git forge decision. It expresses support for the decision made, while accepting blame for the communication woes. It also notes that the version of GitLab to be used has not been determined yet, but that "we are looking at the possibility of using GitLab CE for our dist-git specifically" no matter what version is used for other purposes.

The cynical among us can perhaps be forgiven for noting that all of the mistakes made fell pretty much perfectly to ensure that GitLab was chosen, while providing cover for CPE in making that decision. Even those who are not overly cynical may have some suspicions that the process was, to a certain extent, designed to arrive at that conclusion. As people suspected back when the process was announced in February, perhaps it was simply an exercise to justify dropping Pagure. For example, Williamson reflected on what he saw as the Kafka-esque nature of the process:

Once again, the pre-ordination of the decision was denied by Griffin, but even in that message there are hints that the Pagure option was quickly eliminated. It would seem that it might have been far easier and created much less rancor to simply state up front that Pagure was not viable under the requirements that were likely pretty well-known long before the gathering phase was started. Committing to a formalized process and then dropping the ball in various ways throughout that process has clearly left a bad taste in many mouths. Even unpopular decisions can be made directly and openly in fractious communities if the communication is handled well. When the communication is botched, however, all bets are off—even relatively popular decisions can go badly awry.