FESCo provenpackager sanction causes problems
The Fedora Engineering Steering Council (FESCo) has made a series of missteps in deciding to revoke a longtime Fedora contributor's provenpackager status. FESCo made the decision during a closed session, based on private complaints. It then publicly announced its decision, including the contributor's name, while only supplying a vague account of the contributor's actions. This has left the Fedora community with more questions than answers, and raised a number of complaints about the transparency of FESCo's process. In addition, the sequence of events has sparked discussions about package ownership, as well as when and how it's appropriate to push changes to packages that a developer doesn't own.
Package ownership
Sources for Fedora packages are currently hosted using Fedora's Git forge software, Pagure (though it is due to be replaced by Forgejo in 2025) at src.fedoraproject.org. The packages are managed using Distribution Git (dist-git), which is basically a Git implementation with additional data storage designed to hold content of source RPMs.
In Fedora, packages are typically owned by a maintainer who has packager status. This status is not granted automatically—users have to have a sponsor and learn the ropes to be added to the "packager" group. Once added to the group, they then have the ability to commit changes to the repository for packages they own. Then the package can be submitted to Fedora's Koji build system to be built for a release. Each package repository has a branch for each release that the package exists in, so a package might have f39, f40, f41, and rawhide branches for Fedora 39 through Fedora 41 and Rawhide, for example.
There are many scenarios where packagers may wish to modify packages they do not own. For example, when a person is packaging an update to an application they may need a newer version of a library it depends on. If the owner of the library package has not gotten around to updating it yet, it can be inconvenient to wait for its maintainer to make the update. This is a common scenario in a project where some contributors are paid to work on packaging, while others are volunteers who check in infrequently as time allows. Anyone can submit a pull request (PR) to modify a package, even if they aren't a Fedora packager at all, but only the maintainer (or co-maintainers), can commit those changes.
One exception to this is if a person has provenpackager status. The members of the "provenpackager" group have rights to commit changes to packages they do not own or maintain. To gain this status, a packager must submit a ticket to FESCo requesting the status and must receive at least three affirmative votes (and no negative votes) from provenpackager sponsors. For perspective, the list of packagers includes 2072 users, while there are only 156 members in the provenpackager group on src.fedoraproject.org.
Revocation
The episode began (at least publicly) on December 13, when FESCo
member Josh Stone sent an announcement
to fedora-devel that stated FESCo voted, after a private meeting, to revoke
Peter Robinson's provenpackager status. Robinson has worn many hats within
the project over the years, including a stint as part of Fedora's release-engineering team while employed by Red Hat. He has served on the
now-defunct Fedora Board and its replacement, the Fedora Council. He
left Red Hat in September and currently maintains more than 120 packages in
Fedora as a volunteer. Stone wrote that multiple tickets
had been opened with FESCo, privately, regarding Robinson's "packaging
behavior
":
In particular, on numerous occasions Peter has pushed uncommunicated updates to packages he has no prior relationship with, interfering with those packages' maintenance efforts. On at least a few occasions, this has resulted in other maintainers being forced to react to these changes with no coordination or notice.
The statement claimed that FESCo representatives had several
conversations with Robinson and they had issued warnings to him, but he had
continued to "use his provenpackager privileges in an unapproved manner,
frequently causing additional work for other maintainers
". As a result,
FESCo had voted that his provenpackager status would be revoked, though he would
still retain packager status. Stone said that the vote to revoke the status
was seven in favor, and two against—but did not note how the
individual members had voted.
People were quick to respond to the announcement with questions and requests for additional information. Gary Buhrmaster said that, to be transparent, FESCo should provide the specific conduct that led it to make its decision:
Facts are facts, and should not be in dispute, nor hidden from the community at large, should FESCo wish to maintain the full trust of that community.
Dominik Mierzejewski said
that FESCo should supply an "exhaustive explanation with a compelling
justification
" for revoking Robinson's status, and to supply individual
FESCo member votes before the current election closes on
December 20.
FESCo member Fabio Valentini replied
that FESCo had agreed to provide a public ticket with a summary of its
deliberations "it just hasn't happened yet
". That led others to wonder why
FESCo had announced the decision before having the public ticket at the
ready, and suggest that FESCo should hold back such announcements in the
future.
First time
Valentini eventually responded
that FESCo was still "coordinating internally
" about supplying
more information publicly. He prefaced his response by saying he was
only speaking for himself, not all of FESCo. He reacted to some of the
criticisms by saying that this was the first time that FESCo had been
asked to revoke provenpackager status, so there was no process for
doing so. As for timing, he said that the date of the announcement was
chosen to be shortly after the decision was made and before FESCo
members were offline for the holidays.
Robinson himself spoke
up to say that the public announcement provided more "openness and
transparency
" than the email he had received personally from FESCo. (He
linked to a screenshot of his reply to the email on Google Drive.) He said that none of the changes
he has made to packages were meant to be malicious, but were made to
improve the project and fix "actual problems users are
experiencing
".
He said that he had not been aware of any of the tickets that
had been opened with FESCo, and could only remember one conversation with a
FESCo representative about a change
that he had made to the dav1d package, a
cross-platform decoder for the AV1 codec, maintained by Valentini. In that
conversation he had agreed that he should not have updated the .gitignore file in the repository, but that it had been done
"without malice
". Robinson added that the other changes to the
package were made "using my architecture maintainers hat
" in an
attempt to solve numerous complaints about performance of video codecs on
aarch64 for devices that do not have hardware acceleration.
Robinson's response also alluded to a history of disagreements with
Valentini over maintenance of other packages. He questioned whether
FESCo's process was unbiased, given that Valentini had been involved in the
discussions and vote to remove him. "This feels like a kangaroo court to
me
." He said he looked forward to FESCo providing details of its
conversations, since he had not received a reply directly.
"Broken trust"
Former Fedora Project Leader (FPL) Jared K. Smith expressed
surprise "that this would happen with so little
communication with Peter, and in such a public manner
". He said he'd
known Robinson for more than ten years, and always found him helpful and
had not known him to abuse provenpackager privileges. People-related
complaints always require a certain level of privacy, but "this doesn't
pass the sniff test
".
Stephen Smoogen said
that Robinson can be difficult to deal with at times, but that his work was
in service of the parts of the project he has been given charge over. Of the
many people who had used provenpackager privileges in problematic ways,
Robinson was the person he thought least likely to be dropped from the
group. He hoped that those matters would be dealt with in a public setting
in the future, or that information would at least be published before
judgment was announced: "There have been a lot of things in the past
that tested my temper in Fedora, but this is the first thing that has truly
broken my trust.
" Simon de Vlieger also wanted
to know why FESCo chose to publicly name Robinson in its announcement,
and said it was "unnecessarily harmful
".
Vit Ondruch was one of the few who spoke up in support of FESCo. He replied to the list with Robinson's commit to the Ruby package, which Ondruch owns, in 2014. Robinson said that it was when he was one of two release-engineering people working on Fedora, and the package was likely blocking the building of release artifacts. The example was not, he said, relevant or useful since a lot of things had changed since that patch was committed.
Ondruch replied
with more detail, saying that Robinson's changes—including
renumbering patches in the RPM spec file—were more than needed for
part of the release process. He said he would "love to believe that this
is a relic of the past, but some of the recent discussions
" on the list
did not support that. He did not elaborate about which discussions, and a
quick search through recent fedora-devel discussions does not turn up any
obvious examples. Given the age of his example, it's possible that his
definition of recent extends farther into the past than one might
expect. Ondruch also complained that Robinson didn't include "something
like 'sorry'
" for the decade-old offense so that "we could move
on
".
FESCo apologizes
FESCo member David Cantrell replied
to the thread on December 16, and issued an apology on behalf of FESCo
"for how we communicated the news
". He said it was difficult to
figure out how best to communicate "and we have made mistakes
" by
failing to make the facts available quickly. In some cases, he said, FESCo
was dealing with situations where reporters wanted to remain anonymous.
He said that FESCo was "currently assembling
" its facts so they
could be shared publicly, but first wanted to discuss it with
Robinson. FESCo is also discussing revising the provenpackager policies and its
policies for handling situations related to provenpackagers. He reiterated
that it was the first time FESCo had been asked to revoke those
permissions.
Adam Williamson said that it struck him as problematic that FESCo was allowing anonymity in the process:
This is essentially a technical/process dispute, right? I see no indication that Peter has been accused of a particularly heinous crime or a CoC violation or anything like that. I'm having trouble seeing how anything that doesn't rise to that level could warrant a process involving anonymity for 'reporters' and behind-closed-doors FESCo discussions.
Valentini muddied
the waters further by saying that none of the parties requested
anonymity, but FESCo could not simply make the ticket public "because it
also references a [Code of Conduct] (CoC) issue which *is* private and
cannot be shared
". Daniel P. Berrangé said he was
surprised about the mention of a CoC issue, since that should be handled
with confidentiality and by the CoC committee rather than FESCo.
FPL steps in
Current FPL Matthew Miller stepped
in on December 17 to say: "Several things went very, very
wrong in all this.
"
If the Proven Packager guidelines are ambiguous enough that different readings of them can lead to conflict this strong, we need to clarify them and make sure we have consensus on the [conferred] powers and duties.
In any case, FESCo should not be [adjudicating] Code of Conduct or behavioral allegations, and FESCo actions must not be used as a "shadow" CoC enforcement mechanism.
Miller said that he was not sure what needed to be done to make things right, but that Fedora's Council—which is Fedora's top-level community leadership body—would be working on immediate actions before the holiday, and longer-term actions in January.
Valentini, having been the one to drop the CoC comment in the first
place, complained
that he didn't understand where the idea came from that "this was
basically a CoC issue that was raised to FESCo [...] it's just not
true
". He said that FESCo is not usurping CoC responsibilities, and the
CoC mention was "only a minor side note
" that should not have been
mentioned at all. It is, to date, still unclear where CoC complaints factor
into this situation, the nature of any complaints, and if Robinson was the
subject of those complaints.
On December 18, Miller provided an update, saying that the Fedora Council had met to discuss the topic and had asked FESCo to put its decision on hold while the council investigates. The council would compile a report of what happened and when.
This is going to take some time (and would even without the holidays), and I appreciate your patience. The specific situation connects into broad questions about what "proven packager" should be — and even bigger ones about what it means to be a package maintainer in Fedora. I hope that we can improve how we collaborate and communicate overall. Once we have a full understanding, we will make recommendations on next steps, and possibly adjust Council policies.
Proven packager problems
Setting aside the personalities involved and FESCo's communication fumbles, this episode brings to the fore a common problem for community Linux distributions like Fedora and Debian. There is an inherent tension between individual package ownership and the collective responsibility and collaboration required for the entire project to work. Debian has had a number of discussions this year that have touched on ending single-person package maintainership, though little progress has been made in that area.
In the provenpackager thread, Richard W.M. Jones noted
that there were times when a group of packages needed to be updated
together and it was not feasible "to go through a months long
asynchronous process where every package is a special flower
". Jones
said that Fedora needed "a bit less ownership and a bit more shared
responsibility with packaging
" and that packagers should just try to do
the right thing.
Berrangé agreed with
Jones that the notion of package ownership could give rise to problems. He
said that packagers should be considered custodians of packages, rather
than owners. "Fedora owns the package, maintainers are looking after it
on behalf of Fedora.
" Packagers have personal preferences, but they
should put those aside for the greater good of the project. If someone is
following Fedora procedures, he said, "that should be considered fine,
even if it doesn't align with personal preferences
".
The provenpackager guidelines, he said, are too vague and open to interpretation. It was easy to see how differences of opinion would arise from the non-specific guidance in the policy:
Prior to making changes, provenpackagers should try to communicate with owners of a package in bugzilla, dist-git pull requests, IRC, matrix, or email.
Do provenpackagers need to try all five communication methods? How many
times do they need to try? "Combine that with 'personal preferences' and
it is surprising there are not more conflicts seen.
" The policy, he
said, should have a default method that is considered sufficient to satisfy
common scenarios.
It is, to say the least, not easy to provide policies that balance the needs of the many and the few. A project must find a way to avoid demotivating or hindering volunteers who do the bulk of the work on individual packages, while still making it possible for others to step in as needed. Giving gatekeeping powers over packages to individuals is, perhaps, necessary to ensure that volunteers will take responsibility to ensure the work gets done. But the single-maintainer model that has evolved can be at odds with long-term sustainability, contribute to maintainer burnout, and leave a vacuum when a packager suddenly steps away.
Managing the disagreements and conflicts when they arise between packagers requires clear policies and skillful diplomacy. In this instance, both seem to have failed. The provenpackager policy leaves too much room for individual differences of opinion. It also lacks a clear resolution process, and FESCo's attempt to create one on the fly—behind closed doors—has gone poorly. One hopes that Miller and the council will be able to achieve a better outcome.
