FESCo provenpackager sanction causes problems

The Fedora Engineering Steering Council (FESCo) has made a series of missteps in deciding to revoke a longtime Fedora contributor's provenpackager status. FESCo made the decision during a closed session, based on private complaints. It then publicly announced its decision, including the contributor's name, while only supplying a vague account of the contributor's actions. This has left the Fedora community with more questions than answers, and raised a number of complaints about the transparency of FESCo's process. In addition, the sequence of events has sparked discussions about package ownership, as well as when and how it's appropriate to push changes to packages that a developer doesn't own.

Package ownership

Sources for Fedora packages are currently hosted using Fedora's Git forge software, Pagure (though it is due to be replaced by Forgejo in 2025) at src.fedoraproject.org. The packages are managed using Distribution Git (dist-git), which is basically a Git implementation with additional data storage designed to hold content of source RPMs.

In Fedora, packages are typically owned by a maintainer who has packager status. This status is not granted automatically—users have to have a sponsor and learn the ropes to be added to the "packager" group. Once added to the group, they then have the ability to commit changes to the repository for packages they own. Then the package can be submitted to Fedora's Koji build system to be built for a release. Each package repository has a branch for each release that the package exists in, so a package might have f39, f40, f41, and rawhide branches for Fedora 39 through Fedora 41 and Rawhide, for example.

There are many scenarios where packagers may wish to modify packages they do not own. For example, when a person is packaging an update to an application they may need a newer version of a library it depends on. If the owner of the library package has not gotten around to updating it yet, it can be inconvenient to wait for its maintainer to make the update. This is a common scenario in a project where some contributors are paid to work on packaging, while others are volunteers who check in infrequently as time allows. Anyone can submit a pull request (PR) to modify a package, even if they aren't a Fedora packager at all, but only the maintainer (or co-maintainers), can commit those changes.

One exception to this is if a person has provenpackager status. The members of the "provenpackager" group have rights to commit changes to packages they do not own or maintain. To gain this status, a packager must submit a ticket to FESCo requesting the status and must receive at least three affirmative votes (and no negative votes) from provenpackager sponsors. For perspective, the list of packagers includes 2072 users, while there are only 156 members in the provenpackager group on src.fedoraproject.org.

Revocation

The episode began (at least publicly) on December 13, when FESCo member Josh Stone sent an announcement to fedora-devel that stated FESCo voted, after a private meeting, to revoke Peter Robinson's provenpackager status. Robinson has worn many hats within the project over the years, including a stint as part of Fedora's release-engineering team while employed by Red Hat. He has served on the now-defunct Fedora Board and its replacement, the Fedora Council. He left Red Hat in September and currently maintains more than 120 packages in Fedora as a volunteer. Stone wrote that multiple tickets had been opened with FESCo, privately, regarding Robinson's "packaging behavior":

In particular, on numerous occasions Peter has pushed uncommunicated updates to packages he has no prior relationship with, interfering with those packages' maintenance efforts. On at least a few occasions, this has resulted in other maintainers being forced to react to these changes with no coordination or notice.

The statement claimed that FESCo representatives had several conversations with Robinson and they had issued warnings to him, but he had continued to "use his provenpackager privileges in an unapproved manner, frequently causing additional work for other maintainers". As a result, FESCo had voted that his provenpackager status would be revoked, though he would still retain packager status. Stone said that the vote to revoke the status was seven in favor, and two against—but did not note how the individual members had voted.

People were quick to respond to the announcement with questions and requests for additional information. Gary Buhrmaster said that, to be transparent, FESCo should provide the specific conduct that led it to make its decision:

Facts are facts, and should not be in dispute, nor hidden from the community at large, should FESCo wish to maintain the full trust of that community.

Dominik Mierzejewski said that FESCo should supply an "exhaustive explanation with a compelling justification" for revoking Robinson's status, and to supply individual FESCo member votes before the current election closes on December 20.

FESCo member Fabio Valentini replied that FESCo had agreed to provide a public ticket with a summary of its deliberations "it just hasn't happened yet". That led others to wonder why FESCo had announced the decision before having the public ticket at the ready, and suggest that FESCo should hold back such announcements in the future.

First time

Valentini eventually responded that FESCo was still "coordinating internally" about supplying more information publicly. He prefaced his response by saying he was only speaking for himself, not all of FESCo. He reacted to some of the criticisms by saying that this was the first time that FESCo had been asked to revoke provenpackager status, so there was no process for doing so. As for timing, he said that the date of the announcement was chosen to be shortly after the decision was made and before FESCo members were offline for the holidays.

Robinson himself spoke up to say that the public announcement provided more "openness and transparency" than the email he had received personally from FESCo. (He linked to a screenshot of his reply to the email on Google Drive.) He said that none of the changes he has made to packages were meant to be malicious, but were made to improve the project and fix "actual problems users are experiencing".

He said that he had not been aware of any of the tickets that had been opened with FESCo, and could only remember one conversation with a FESCo representative about a change that he had made to the dav1d package, a cross-platform decoder for the AV1 codec, maintained by Valentini. In that conversation he had agreed that he should not have updated the .gitignore file in the repository, but that it had been done "without malice". Robinson added that the other changes to the package were made "using my architecture maintainers hat" in an attempt to solve numerous complaints about performance of video codecs on aarch64 for devices that do not have hardware acceleration.

Robinson's response also alluded to a history of disagreements with Valentini over maintenance of other packages. He questioned whether FESCo's process was unbiased, given that Valentini had been involved in the discussions and vote to remove him. "This feels like a kangaroo court to me." He said he looked forward to FESCo providing details of its conversations, since he had not received a reply directly.

"Broken trust"

Former Fedora Project Leader (FPL) Jared K. Smith expressed surprise "that this would happen with so little communication with Peter, and in such a public manner". He said he'd known Robinson for more than ten years, and always found him helpful and had not known him to abuse provenpackager privileges. People-related complaints always require a certain level of privacy, but "this doesn't pass the sniff test".

Stephen Smoogen said that Robinson can be difficult to deal with at times, but that his work was in service of the parts of the project he has been given charge over. Of the many people who had used provenpackager privileges in problematic ways, Robinson was the person he thought least likely to be dropped from the group. He hoped that those matters would be dealt with in a public setting in the future, or that information would at least be published before judgment was announced: "There have been a lot of things in the past that tested my temper in Fedora, but this is the first thing that has truly broken my trust." Simon de Vlieger also wanted to know why FESCo chose to publicly name Robinson in its announcement, and said it was "unnecessarily harmful".

Vit Ondruch was one of the few who spoke up in support of FESCo. He replied to the list with Robinson's commit to the Ruby package, which Ondruch owns, in 2014. Robinson said that it was when he was one of two release-engineering people working on Fedora, and the package was likely blocking the building of release artifacts. The example was not, he said, relevant or useful since a lot of things had changed since that patch was committed.

Ondruch replied with more detail, saying that Robinson's changes—including renumbering patches in the RPM spec file—were more than needed for part of the release process. He said he would "love to believe that this is a relic of the past, but some of the recent discussions" on the list did not support that. He did not elaborate about which discussions, and a quick search through recent fedora-devel discussions does not turn up any obvious examples. Given the age of his example, it's possible that his definition of recent extends farther into the past than one might expect. Ondruch also complained that Robinson didn't include "something like 'sorry'" for the decade-old offense so that "we could move on".

FESCo apologizes

FESCo member David Cantrell replied to the thread on December 16, and issued an apology on behalf of FESCo "for how we communicated the news". He said it was difficult to figure out how best to communicate "and we have made mistakes" by failing to make the facts available quickly. In some cases, he said, FESCo was dealing with situations where reporters wanted to remain anonymous.

He said that FESCo was "currently assembling" its facts so they could be shared publicly, but first wanted to discuss it with Robinson. FESCo is also discussing revising the provenpackager policies and its policies for handling situations related to provenpackagers. He reiterated that it was the first time FESCo had been asked to revoke those permissions.

Adam Williamson said that it struck him as problematic that FESCo was allowing anonymity in the process:

This is essentially a technical/process dispute, right? I see no indication that Peter has been accused of a particularly heinous crime or a CoC violation or anything like that. I'm having trouble seeing how anything that doesn't rise to that level could warrant a process involving anonymity for 'reporters' and behind-closed-doors FESCo discussions.

Valentini muddied the waters further by saying that none of the parties requested anonymity, but FESCo could not simply make the ticket public "because it also references a [Code of Conduct] (CoC) issue which *is* private and cannot be shared". Daniel P. Berrangé said he was surprised about the mention of a CoC issue, since that should be handled with confidentiality and by the CoC committee rather than FESCo.

FPL steps in

Current FPL Matthew Miller stepped in on December 17 to say: "Several things went very, very wrong in all this."

If the Proven Packager guidelines are ambiguous enough that different readings of them can lead to conflict this strong, we need to clarify them and make sure we have consensus on the [conferred] powers and duties.

In any case, FESCo should not be [adjudicating] Code of Conduct or behavioral allegations, and FESCo actions must not be used as a "shadow" CoC enforcement mechanism.

Miller said that he was not sure what needed to be done to make things right, but that Fedora's Council—which is Fedora's top-level community leadership body—would be working on immediate actions before the holiday, and longer-term actions in January.

Valentini, having been the one to drop the CoC comment in the first place, complained that he didn't understand where the idea came from that "this was basically a CoC issue that was raised to FESCo [...] it's just not true". He said that FESCo is not usurping CoC responsibilities, and the CoC mention was "only a minor side note" that should not have been mentioned at all. It is, to date, still unclear where CoC complaints factor into this situation, the nature of any complaints, and if Robinson was the subject of those complaints.

On December 18, Miller provided an update, saying that the Fedora Council had met to discuss the topic and had asked FESCo to put its decision on hold while the council investigates. The council would compile a report of what happened and when.

This is going to take some time (and would even without the holidays), and I appreciate your patience. The specific situation connects into broad questions about what "proven packager" should be — and even bigger ones about what it means to be a package maintainer in Fedora. I hope that we can improve how we collaborate and communicate overall. Once we have a full understanding, we will make recommendations on next steps, and possibly adjust Council policies.

Proven packager problems

Setting aside the personalities involved and FESCo's communication fumbles, this episode brings to the fore a common problem for community Linux distributions like Fedora and Debian. There is an inherent tension between individual package ownership and the collective responsibility and collaboration required for the entire project to work. Debian has had a number of discussions this year that have touched on ending single-person package maintainership, though little progress has been made in that area.

In the provenpackager thread, Richard W.M. Jones noted that there were times when a group of packages needed to be updated together and it was not feasible "to go through a months long asynchronous process where every package is a special flower". Jones said that Fedora needed "a bit less ownership and a bit more shared responsibility with packaging" and that packagers should just try to do the right thing.

Berrangé agreed with Jones that the notion of package ownership could give rise to problems. He said that packagers should be considered custodians of packages, rather than owners. "Fedora owns the package, maintainers are looking after it on behalf of Fedora." Packagers have personal preferences, but they should put those aside for the greater good of the project. If someone is following Fedora procedures, he said, "that should be considered fine, even if it doesn't align with personal preferences".

The provenpackager guidelines, he said, are too vague and open to interpretation. It was easy to see how differences of opinion would arise from the non-specific guidance in the policy:

Prior to making changes, provenpackagers should try to communicate with owners of a package in bugzilla, dist-git pull requests, IRC, matrix, or email.

Do provenpackagers need to try all five communication methods? How many times do they need to try? "Combine that with 'personal preferences' and it is surprising there are not more conflicts seen." The policy, he said, should have a default method that is considered sufficient to satisfy common scenarios.

It is, to say the least, not easy to provide policies that balance the needs of the many and the few. A project must find a way to avoid demotivating or hindering volunteers who do the bulk of the work on individual packages, while still making it possible for others to step in as needed. Giving gatekeeping powers over packages to individuals is, perhaps, necessary to ensure that volunteers will take responsibility to ensure the work gets done. But the single-maintainer model that has evolved can be at odds with long-term sustainability, contribute to maintainer burnout, and leave a vacuum when a packager suddenly steps away.

Managing the disagreements and conflicts when they arise between packagers requires clear policies and skillful diplomacy. In this instance, both seem to have failed. The provenpackager policy leaves too much room for individual differences of opinion. It also lacks a clear resolution process, and FESCo's attempt to create one on the fly—behind closed doors—has gone poorly. One hopes that Miller and the council will be able to achieve a better outcome.