Fedora grapples with change

The Fedora Project is known for, among other things, having a well-defined set of processes for just about everything. It has extensive packaging guidelines that deal with the complexities of creating RPMs to install software, as well as processes for managing the legal questions that arise around shipping software. Fedora also has a well-defined change process for dealing with self-contained technical changes as well as major changes to the distribution, and other issues as they arise. At the moment, though, the project seems to be experiencing a sort of midlife crisis as it re-examines several of its change processes at once to determine if they are still effective.

Evolution

As a vendor-sponsored project, Fedora has always had to walk a difficult path in serving its users, contributor community, and corporate masters; what makes one party the happiest may be a source of angst for another. Red Hat's desire to trial technologies in Fedora does not always spark joy among contributors and users. Volunteer contributors may want to push ideas that are of little interest to Red Hat (at least at the time), or may even conflict with its choices. For example, Fedora's choice of Btrfs as the default filesystem contrasts with Red Hat's decision not to support it in Red Hat Enterprise Linux (RHEL).

Users want easy access to problematic software, such as patent-encumbered games or codecs, but shipping those could create expensive legal headaches for Red Hat. Attempts to make Fedora more user-friendly, say by setting the default for the $EDITOR environment variable to GNU nano, may not please some developers.

Fedora had little in the way of governance or policy when the project launched in 2003 as a kind of replacement for Red Hat Linux. Early on, Red Hat entertained the idea of creating a "Fedora Foundation" that would give the project some independence, and then pulled back from that in 2006. Some of the mechanisms that were put in place in anticipation of the foundation, such as a Fedora Board and Fedora Extras Steering Committee, evolved into the Fedora Council and Fedora Engineering Steering Committee (FESCo) over time.

There have been a number of times when there was friction between corporate and community priorities; but the project has more or less made it work over the years. It has done this by discussing the problems, finding some kind of consensus, documenting the policies that are hammered out along the way, and then using them for new decisions. See, for example, Tom Callaway's overview of how Fedora's legal policies were developed. Over the years, Fedora's governance and policies have become something of a model for other open-source projects.

The project's governance has had one constant: Red Hat has the final say. As former Fedora Project Leader (FPL) Max Spevack noted when Red Hat dumped the idea of an independent foundation:

Red Hat *must* maintain a certain amount of control over Fedora decisions, because Red Hat's business model *depends* upon Fedora. Red Hat contributes millions of dollars in staff and resources to the success of Fedora, and Red Hat also accepts all of the legal risk for Fedora. Therefore, Red Hat will sometimes need to make tough decisions about Fedora. We won't do it often, and when we do, we will discuss the rationale behind such decisions as openly as we can.

Just because Red Hat has power over Fedora does not mean that the company wants to use it, he said. Nor did it want to make all the important decisions about Fedora: effective community-driven decision making would be a direct measure of Fedora's success. "We aim to set the standard for open source innovation. A truly open Fedora Project is what makes that possible."

Sandbox

In the past year or so, though, the project's processes that have worked until now seem to be coming into question more and more often: usually when they seem to butt up against Red Hat's priorities. For example, last year's deliberation on Fedora's AI-assisted contribution policy that showed a disconnect between Red Hat's desire to experiment with AI in Fedora and contributors who wanted Fedora's policy to be much less friendly to AI.

In March, current FPL Jef Spaleta proposed a technology-innovation-lifecycle process he dubbed the "Fedora Sandbox" for "experimental features, components, output, process, or services".

The driver for Spaleta's sandbox idea seemed to be related to frustrations that Red Hat leadership had with pushing its experiments into Fedora; the project's existing processes, which involve compliance with Fedora's strict packaging and license policies as well as a need to gain community buy-in, could slow or stymie acceptance of things Red Hat was going to do for RHEL anyway.

Having to maintain those projects outside of Fedora, which ultimately serves as the foundation for a RHEL release, is a likely source of frustration for folks working on both—not to mention their managers and Red Hat leadership who (not unreasonably) just want features to land in RHEL on schedule to make customers happy.

Spaleta's sandbox process would have allowed experiments to be conducted in Fedora even if they broke "non legally binding" policies as long as they had a "reasonable path forward towards resolution" before being fully integrated into the project or distribution. He proposed that the sandbox would be in addition to the existing change processes and community initiatives that can be used to set longer-term goals for Fedora, such as the completed initiative to create the Fedora IoT project, or the current Git Forge initiative to replace the Pagure collaboration platform with the Forgejo-based Fedora Forge service. The sandbox proposal has not been accepted (or rejected) yet.

Initiative friction

Red Hat developer Gordon Messmer proposed an AI developer desktop initiative, on March 31, with an aim to "build a thriving community around AI technologies" within Fedora. That proposal met with some opposition from the community, with objections ranging from a dislike of AI to complaints about potentially changing Fedora's policy for out-of-tree kernel modules. Ultimately the proposal was discussed by the Fedora Council, initially approved, and then blocked at the last minute when council member Justin Wheeler changed his vote on May 8. Council member Miro Hrončok, likewise, also changed his vote on May 13 saying that "the Fedora community is not supportive of this initiative as is".

Another Red Hat idea, an automated approach to building an operating system called Project Hummingbird, was raised on the Fedora development list in April. It was greeted with some interest, as well as some confusion about how it might differ from other Fedora variants, but it didn't seem to face much opposition. It was never formally proposed, though McCarty said he planned to do so through Spaleta's sandbox initiative.

Instead, Red Hat bypassed public processes entirely. It asked the council in private for approval to use the Fedora trademark so that it could announce the project at Red Hat Summit on May 12. Fedora contributors outside of Red Hat were surprised and confused by the announcement. Michael Gruber, for example, wondered whether Hummingbird was legitimately a Fedora project or not. "There might be even some good ideas in there, but given how this started and how it is communicated I can put zero trust in this."

Red Hat employee and Fedora contributor Adam Williamson said it would have been ideal for Red Hat to make the request more openly, but he was happy that the company was trying to do the Hummingbird project within Fedora rather than doing an end run around the project. The more Red Hat has to do outside Fedora, he reasoned, the greater the risk that the company will question its funding of the project.

From inside Red Hat, especially if you don't work on Fedora, it is possible/tempting to look at Fedora as a source of strife. It's got all these people in it, with opinions, who aren't on the payroll! You can't tell them what to do or think or complain to their manager! They have eternal arguments about everything! You have to write a wiki page and convince some person you've never heard of that your idea is good! Who needs this?

So we're kind of constantly fighting a tendency for RH to just spin stuff up in channels it 100% controls, which tends to seem easiest at first then turn out to be a mess after a few years.

Christopher Klooz, however, worried that the funding justification meant that Fedora could be "step by step transformed into a corporate unit of RH". He added that it was already unclear "when the Council acts as agent of RH and when as agent of the community".

Williamson replied that he understood the point, but said that this was a "an ever-present tension" that Fedora has had to manage nearly as long as it's been around:

The way Fedora can "compete" is not by turning into the thing some parts of RH might superficially think they want - an upstream project that's open source in license terms but closely-controlled in governance - but by being the thing they need - a loosely-coupled upstream project with passionate and involved people who will make things awkward and uncomfortable at times but usually produce a better end result over the long term, and act as a useful check on RH's perceptions at times.

Red Hat employees who care about Fedora, he said, have to keep selling that vision and making sure that it works. The Hummingbird discussion petered out not long after Williamson's reply, but it appears that Fedora's decision-making bodies have been mulling over how things are done.

Hit pause

On July 1, Aoife Moloney, Fedora's "change wrangler", announced that the Fedora Council was proposing a pause to Fedora's community initiatives process. Existing initiatives would continue as planned, she said, though "the administrative framework around them may evolve".

The AI developer desktop, she said, had shown that the initiatives process had failed as a framework "where new ideas can surface, receive respectful feedback, and gain Council support for work that fits the project's present and/or future". The nature of the failure, however, was unspecified. One can imagine that the various participants in that discussion might well agree that something had not gone well, but what that something was would depend entirely on the observer's point of view.

The council, Moloney said, wanted to work out a new method of setting strategic direction "in an open, transparent way that more intentionally includes the community voice." The council recognized that it needed to be "better at being more open in our discussions and decision making" because much of the work that leads to proposals "happens under the radar before official approval processes kick in". At the moment the existing "approval pipeline" for initiatives involves the council performing trademark review, and then FESCo reviewing change proposals. That works well, she said, but misses "early and inclusive discussion for everyone across the project". Therefore, the council would be looking at the sandbox proposal closely as an alternative to initiatives or as a complement to some other process that it might develop.

The announcement phrased the pause as a proposal, but Moloney also closed the discussion on the AI developer desktop proposal saying that the council was now unable to consider it "as a result of halting the Community Initiatives process." The council would return to the question of "whether Community initiatives should be retired or revamped once this discussion has reached some kind of conclusion".

Changes to changes

Shortly before the council had made its announcement, FESCo member "Maxwell G" started a discussion on the Fedora development mailing list to gather feedback on how the changes process could be improved. He said he was not proposing anything specific, but wanted to throw out some ideas and see what other people thought.

For example, he wondered if it was time to move away from using Fedora's wiki for proposals, as well as the Wikitext formatting that goes along with it. The formatting often escapes the wiki and makes its way to other fora; for example, the Btrfs change proposal for Fedora 33 from 2020 has a combination of plain text, MediaWiki formatting, and HTML <span> tags that makes it unpleasant to attempt to decipher in any mail client.

He suggested that changes should be in Markdown or "some format that can easily be converted", and stored as text files in a Git repository rather than on the wiki. The change owners could file proposals as a merge request, which would be reviewed and merged by the change wrangler. He also floated the idea of moving away from the Discourse forum as the primary source of truth for change proposals; he also wanted to stop the practice of having discussions on changes take place both on the forum and Fedora's development list:

Cross-posting to Discourse was proposed as an experiment in fesco [ticket] #2989, but there was never a decision made about whether to stop or continue with the experiment. I find the fractured discussions between devel@ and Discourse hard to follow. I think the Discourse setup makes it easier for discussions to "accelerate" or become toxic or repetitive. I don't think it's any better at handling large threads.

The suggestion to consolidate discussions on the mailing list seemed popular. Michal Schorm agreed that splitting discussions between the mailing list and forum was bad, especially for the change owners who had to follow both. Björn Persson also agreed that the discussions needed to be consolidated, but he observed that it was much worse than just following a forum and mailing list discussion. "When I went through the Change process, the state of the Change was fractured over the wiki, Pagure, Bugzilla and even Gitlab." He counted ten different discussion fora and issue trackers, all in their own silos. "At least Pagure and Bugzilla are good at sending me email when someone writes a comment."

Kevin Fenzi, however, thought that the practice of using the forum for change requests should continue. He acknowledged that there were some problems with that, but it meant Fedora received feedback "from people who are not otherwise involved and sometimes [it is] very useful". He felt that it was important to hear from new voices and to accept feedback "in the place where they are".

Former Fedora program manager Ben Cotton, who had worked closely with the change process during his tenure, thought that the process should be modernized, but suggested that a better approach would be to think about what the process should look like at a high level, then think about how to implement it.

There was a suggestion from Martin Kolman that Fedora should build a simple web application to manage the change process. It would be less daunting for new contributors, he said, and could be designed to validate information before a change was submitted. The idea of another homegrown application to maintain, however, did not win supporters. Daniel P. Berrangé said that Fedora had "a long (and disappointing) track record" of building things and then being unable to support them in the long run. "I can't see it being a good use of resources to build & maintain a custom app for this."

Moloney, who works with changes as part of her job, weighed in on June 30. She said she had agreed with some of the suggestions, such as restricting change discussions to the mailing list, but asked that participants in the discussion ask themselves how much of hands-on involvement they had with the mechanics of the process "before suggesting ways to engineer a brand new way to do it". She also noted that there were many moving parts already, such as the move to Fedora Forge, and thought it might be a good idea to let the dust settle from those "and then see what we're missing".

Red Hat engineering manager Brendan Conoboy also spoke up with his view of Fedora's processes from the outside looking in. Overall, he thought that the change processes work pretty well for the Linux distribution. "Here's what doesn't work: changes that fit poorly inside existing processes." He cited the recent two-factor authentication discussion and the AI developer desktop proposal as examples. "The bar to clear to initiate change in a way every interest feels respected is tough to such a degree that even initiating a conversation is fraught."

Maxwell G pointed out that the AI discussion "hit a lot of pain points", some of which were procedural (such as the council failing to clearly announce its intent to vote on the proposal), but there were also social and political disagreements as well. In any case, he said, that was an initiative rather than a change proposal—and that process was under separate review by the council. Conoboy replied, "we knew the [AI] subject matter was controversial, and it seemed that following a documented process would provide a better framework for constructive feedback". Perhaps it did, he said, but it might have gone better if the idea had been socialized with FESCo first.

Outcome

After the discussion had wound down Maxwell G replied on July 7, with his summary of the conversation. His first takeaway was that he should have had a more detailed problem statement that had more context about what he was looking for from the discussion. He found that there was some support for a Git-based workflow, but the wiki-based workflow also had many supporters; many ideas were put forth, but no clear winners.

What was clear, however, was that the split between the mailing list and forum "creates a frustrating burden for Change Owners and other people trying to follow the discussion". The project had tried the "split-brain approach for development discussion" for three years, and it was not working. He said he would focus on solving that pain point first, and then would come back with ideas for a different wiki-based process at a later date.

Given that it is vacation season in much of the world, there is a good chance that the various efforts to modify Fedora's processes will see little movement in the next month or two. It seems likely, though, that there will be some shakeups to the project's processes once Fedora contributors return from holiday and pick up where these conversations have left off.