LWN: Comments on "FESCo provenpackager sanction causes problems"

Late to the conversation - this is key Fedora advantage

martin.langhoff — Mon, 30 Dec 2024 11:58:41 +0000

I've been a Debian and Fedora maintainer. And I've worked closely with Peter Robinson - some years ago.

Yes, provenpackagers cause some friction making needed changes fast, and any mistake they make gets amplified if the package maintainer of record has a strong sense of ownership. But that sense of ownership is in tension with moving things forward uniformly and quickly.

Fedora has it right – provenpackagers get sh_t done and most packages aren't unique flowers. I find this to be significantly better than Debian.

When I worked with Peter, I had this exact situation - every once in a while he'd fix something across a dozen packages, some of which I was the maintainer of record for. It was annoying, but it moved the ball forward every time.

In a few occasions, he made a mistake, so it was double annoying. I complained a bit, but the ball kept moving forward, so after complaining a bit, I helped mend the effort and... I didn't torch the person pushing for progress.

There are, indeed, technical solutions to social problems

smoogen — Sat, 21 Dec 2024 13:07:47 +0000

I agree, and I apologize that my sentence about Peter has blown up about it. I was trying to be clear that I know he can be rough around the edges and it got away from me.

There are, indeed, technical solutions to social problems

lkundrak — Fri, 20 Dec 2024 18:00:45 +0000

> I have no idea if Mr. Robinson really is "that guy,"

absolutely not

it's me, hi, I'm the problem, it's me

farnz — Fri, 20 Dec 2024 13:06:07 +0000

Fundamentally, no answer is satisfactory because it's all "we have private tickets, which you're not allowed to see, but trust us, they're damning".

There's an ordering problem here - first, you should have had the evidence ready to go public as easily as un-marking FESCo tickets as public. Second, probinson should have been granted read access to the tickets you're going to make public, and be given a chance to prepare his response - with you able to show via logs that he had indeed seen the tickets. And then, and only then, should you have made the announcement.

The trouble is that having done things backwards, you're in a bad place. You're effectively saying "trust us, our super-secret private information proves that you can trust us, but we can't show you it", and that's always a hard statement to make successfully, especially if people have emotional ties to Peter (friendship, for example).

There are, indeed, technical solutions to social problems

k3ninho — Fri, 20 Dec 2024 12:51:38 +0000

Valorising 'technically effective' over 'socially effective', which is sometimes hidden behind ideas of meritocracy, is a real pain when trying to manage people and coalesce teams to be perform greater than the sum of the parts. The '10x individual' idea is a heinous part of this, too, because it puts lower responsibility to be (or reward for being) a good colleague and team mate on someone who's not good at that, while binding everyone impacted by this individual's choices to a status level where they have to deal with working code now and little-to-no documentation or discussion.

I understand that those things have rare excellent projects started -- but sustained requires collaboration from people co-laboring with you.

One idea with Merge Requests is to set the expectation that a MR which passes the test suite ought to be approved, but you present the changes to the team as a teaching session documenting why you chose the approach and made the MR's design decisions. Because people get better at what they practice doing, you get gains in technical communication ... by practising technical communication.

K3n.

it's me, hi, I'm the problem, it's me

AdamW — Fri, 20 Dec 2024 02:49:25 +0000

Well, what you should have done is get all your ducks in a row properly.

Don't remove PP status till you're ready to announce the decision (because someone may notice and trigger a mess). Don't announce it until you're ready to justify it. If doing that requires work to hide something that *really really* needs to be private, do all that work first. Peter's been a PP for more than a decade and he hasn't burned the project down yet, whoever he might have annoyed a bit. There was no rush to remove his privileges immediately.

Once you hadn't done that, the first response by FESCo to people highlighting the issues should have been the response Matthew and the Council eventually had to do instead: apologize for the errors, and announce that the revocation was being suspended while all the ducks were put in the correct order.

It's possible the decision is entirely justified. The problem is nobody outside of FESCo can judge that, and it doesn't seem like there was any legitimate reason to delay disclosure of the reasons for the decision beyond disclosure of the decision itself.

There are, indeed, technical solutions to social problems

NYKevin — Fri, 20 Dec 2024 01:26:20 +0000

> I think that if you have a social community where the most effective people are also the rudest and hardest to work with, you should rethink your community structure and how it got to be that way.

This is not a Fedora-exclusive problem. Basically every internet community where "effective" has any sort of meaningful definition has exactly the same issue. There is nearly always "that one guy who rubs everyone the wrong way, but does good work too." (In my experience, the person does usually identify as male, but I'm sure female etc. examples can be found if you look hard enough. I use male pronouns for simplicity.)

The most common resolutions are:

1. Ban "that guy" from the community, and then argue over whether he should have been banned.
2. Don't ban "that guy," and then argue over the same.
3. "That guy" takes a hint and mellows out.

Most communities enter a holding pattern where (2) is the default answer right up until "that guy" manages to do something obnoxious enough to tip the moderators (or whatever they call themselves) over to doing (1) instead (frustratingly, this last straw often looks innocuous in isolation, which is why people continue to argue after the banhammer comes down). (3) is rare but probably does happen from time to time.

One other thing: Since I have no experience working with Fedora, I have no idea if Mr. Robinson really is "that guy," or if he just rubbed one or two people the wrong way once or twice (you are not "that guy" unless it's a long-term pattern of behavior and affects a lot of people). I do not want the above comment to be interpreted as a condemnation of him in particular - I'm just trying to describe a general trend I have observed in internet communities.

it's me, hi, I'm the problem, it's me

tchernobog — Fri, 20 Dec 2024 00:33:54 +0000

Very well said.

Additionally, the accused should be able to defend against accusations leveraged against them, before disciplinary action is taken.

I ask my kids "did you really hit your sister?" and "why?" before deciding whether to revoke television rights for the week. And sometimes accused and accuser swap...

Never mind adults and their complex courts of laws.

It's very hard to trust any process if even the accused is just shelled out punishment out of the blue fur something they don't feel responsible for. Would you trust your government if you had no hearing from a judge once being put to trial?

Even worse, supposing any accusations is founded, it takes away the ability of the perpetrator to learn from their punishment, instead than just righteously balking at the injustice that befell them.

FeSCO did not seem to hear the other side, met behind closed doors, had at least one person with a personal stake on the board that should have revised themselves, and publicly made an example of a person by publishing their full name publicly.

When all of this is wrong "trying to calm down people" because "it's a small punishment" is missing the fundamental point of social contract in a community based on trust among people.

it's me, hi, I'm the problem, it's me

ttuttle — Thu, 19 Dec 2024 21:43:52 +0000

As a former young child, it didn't work on me either.

Guess it's time to vote

jdulaney — Thu, 19 Dec 2024 21:34:07 +0000

I haven't voted in a fesco election in years; clearly that needs to change.

Guidelines, schmuidelines

cuviper — Thu, 19 Dec 2024 17:52:20 +0000

> Fedora should put this sanctioning on hold, if not revoke it outright, until such a process has been somewhat-agreed-on.

The article already mentioned the hold request from the Fedora Council, and that's been done via <https://pagure.io/fesco/issue/3305>.

it's me, hi, I'm the problem, it's me

hailfinger — Thu, 19 Dec 2024 17:37:45 +0000

Disclaimer: I don't use Fedora nor am I involved in this in any way, but I've seen similar situations unfold elsewhere. Enforcement action may be well-intentioned, but the (collateral) damage can be huge.

> I am also disappointed by how some FESCo members' efforts (including mine) to calm the waters have been received on the mailing list, and were often just interpreted in the worst possible way: "we are working on providing more details" is not enough, "we cannot share everything yet" results in being accused of backroom deals, but actually mentioning *why* it's not as easy as just un-marking the FESCo tickets as "private" is somehow even worse.

Try to take a birds-eye view on this: Some people in power make a decision, announce and enforce it. At the time of announcement/enforcement, they are not providing detailed reasoning nor transparency. People not in power fill the information vacuum with the explanation they think is most likely and start getting angry. People in power are still not providing transparency for reasons which can't be verified either. People not in power start getting angrier.

Something along the lines of "Trust us, we did the right thing, but we can't tell you our reasons (yet)" may be intended to calm the waters, but the only situation where this can even remotely work is parents taking care of very young children. If grown-up people realize they are treated like children, they get even angrier.

With adults, it's just politics. If elected officials can't justify their actions, the electorate will complain. Some officials have figured out that apologies go a long way in getting re-elected.

> I get that people are disappointed or even angry, but this looks like a situation in which *no* answer would satisfy some commenters on the mailing list.

A possible answer which may satisfy some commenters: an apology and a retraction of the decision until the reasoning can be made public completely.

Side note: If there is a process which may result in disciplining a person, that process should also afford at least a robust set of protections for the defendant. That's something which quite a few projects are lacking.

There are, indeed, technical solutions to social problems

geofft — Thu, 19 Dec 2024 17:12:09 +0000

A few things strike me as contributing to an inevitable conflict:

1. "Richard W.M. Jones noted that there were times when a group of packages needed to be updated together and it was not feasible 'to go through a months long asynchronous process where every package is a special flower'. Jones said that Fedora needed 'a bit less ownership and a bit more shared responsibility with packaging' and that packagers should just try to do the right thing."

2. "provenpackager" status exists specifically to override this sense of ownership, apparently without requiring second-person review from another provenpackager.

3. "Stephen Smoogen said that Robinson can be difficult to deal with at times, but that his work was in service of the parts of the project he has been given charge over." The actual email is harsher: "Yes, Peter is hard to get along with at times, and yes he can be brusque, pushy, and grumpier than any mule or cow I have had the 'pleasure' to work with, and I have been at the tail end of his tongue multiple times over the years in Fedora. Yet in all those years, I have also known that his work has been in service of the parts of the project he has been given charge of, be it architecture or package sets."

It seems like the community has ended up in a situation where individual package maintainers are encouraged to have an expectation of ownership and control over their packages, the project as a whole also has an expectation that certain trusted people will unilaterally override that ownership and control for the sake of forward progress in the project, and one of the people who has most beneficially exercised that ability (from the point of view that the exercise of that ability is a good thing) is someone who is unpleasant to work with. I don't think it's surprising that this would have blown up at some point.

In particular, given the simultaneous expectations of individual ownership and provenpackager overrides, the project has essentially engineered a situation where the most effective provenpackagers, the ones who most accomplish the goals of having that override capability, are the ones who are most willing to disregard other people's feelings and preferences and make opinionated changes. Someone who cares very much for respecting the preferences of the individual packager and working with them is simply going to exercise their override powers less.

Indeed, someone who is very effective at this interpersonal work probably doesn't need provenpackager status at all; they can just send patches or convince the maintainer that they can be trusted as a co-maintainer. (If the maintainer is unresponsive, this doesn't work, but that can be dealt with by a process that only applies to inactive maintainers, e.g. a shorter timeout for a maintainer to be removed and replaced, communicated clearly up front when someone offers to be a maintainer. The situations discussed in this article seem to be about provenpackager changes to packages that aren't unmaintained; e.g., the FESCo statement about "other maintainers being forced to react to these changes" implies that those other maintainers exist and are active enough to need to react.)

I think that if you have a social community where the most effective people are also the rudest and hardest to work with, you should rethink your community structure and how it got to be that way. To be clear, I don't think that rudeness or difficulty is an immutable characteristic of a person; your community may simply have told people who are capable of many interpersonal approaches that the effective approach is to be rude. I wholly agree with the point at the end of this article that the real issue here is about the provenpackager status and not about the individuals, and that Fedora (and Debian, and other projects) would benefit from rethinking them. Among other things, that same rudeness is probably dissuading new participants, and if you had more participants and a process for making the individual commitment lower, you wouldn't need the structures that lead to high-stakes conflict.

One specific technical thing that surprises me is that provenpackagers have access to commit without review. This was how we commonly did software development in the CVS/SVN days, before we had DVCSes that allow commit objects to be created but not applied to the mainline. I recall a number of upstream communities where getting access to the "commit bit" was an onerous and high-stakes process, and also where fractious disagreements arose from people making unreviewed commits that others didn't like. With the rise of GitHub and platforms with similar workflows, those problems don't seem to have scaled up proportional to FOSS development itself, and I think that's because the technical model both prevents an individual maintainer from acting unilaterally (if you require review for all changes) and also lessens the distinction between a maintainer and an ordinary contributor and so reduces the need for that level trusted access. (Another technical change that has worked for the better is CI that runs as part of a pull request or merge train; you don't need to argue about who broke the build, whether you should have tested on a specific platform, whether you conformed with style requirements, etc. if there are bots that enforce these for you and the configuration of those bots is expressed in code that can itself be changed by a normal pull request.) I assume Fedora's practice here predates the conversion to Git and I think this would be one of the first things to rethink.

it's me, hi, I'm the problem, it's me

decathorpe — Thu, 19 Dec 2024 15:49:46 +0000

While I agree that this has been handled and communicated poorly, I am quite confused by some responses from the community, given that

1. both the policy around packager / provenpackager responsibilies [0] [1] and the documentation for what changes are OK to be made using provenpackager privileges are quite clear (and not ambiguous at all, in my opinion) [2], and

2. it is (for now) undisputed that both policies have been violated numerous times in this case.

All information regarding this issue are being (or have already been) forwarded to the Fedora Council, and I am confident their investigation will eventually confirm that the FESCo decision was justified (if quite badly communicated).

I am also disappointed by how some FESCo members' efforts (including mine) to calm the waters have been received on the mailing list, and were often just interpreted in the worst possible way: "we are working on providing more details" is not enough, "we cannot share everything yet" results in being accused of backroom deals, but actually mentioning *why* it's not as easy as just un-marking the FESCo tickets as "private" is somehow even worse.

I get that people are disappointed or even angry, but this looks like a situation in which *no* answer would satisfy some commenters on the mailing list.

Side note: I also don't understand why some contributors are so agitated about loss the of provenpackager status here - they are *special* privileges that should be used in *exceptional* circumstances, and which should not be needed for normal workflows. Also, losing provenpackager group membership can happen as easily as ... just not contributing to Fedora for six months (for which there *is* an established process [1, last paragraph]), so I am confused why this is apparently considered such a draconian measure by some.

[0]: https://docs.fedoraproject.org/en-US/fesco/Package_mainta...
[1]: https://docs.fedoraproject.org/en-US/fesco/Provenpackager...
[2]: https://docs.fedoraproject.org/en-US/fesco/Who_is_allowed...

Guidelines, schmuidelines

smurf — Thu, 19 Dec 2024 15:43:59 +0000

> in 2014

Owch. My gut reaction to this one is "Grow up already".

Seriously, this should have been ample reason to request Ondruch to recluse himself from the current discussion.

> The provenpackager policy leaves too much room for individual differences of opinion.

That is not necessarily a problem. I'd rather have a loose set of "use your judgment dammit" guidelines than a five-page straightjacket that leads to rules lawyering instead of finding a compromise.

> It also lacks a clear resolution process,

… which is the actual problem, methinks.

Fedora should put this sanctioning on hold, if not revoke it outright, until such a process has been somewhat-agreed-on.