|
|
Subscribe / Log in / New account

A kernel code of conduct enforcement action

The Linux Foundation Technical Advisory Board (TAB) has decided to "restrict Kent Overstreet's participation in the kernel development process during the Linux 6.13 kernel development cycle" based on a recommendation from the Code of Conduct committee. In particular, the scope of the restriction will be to "decline all pull requests from Kent Overstreet" during the development cycle. Overstreet is the creator and maintainer of the bcachefs filesystem.

This action stems from a message Overstreet posted back in early September that was abusive toward another kernel developer; there is a fair amount of back-and-forth about the incident and the committee's attempts to extract a public apology from Overstreet in that thread. Overstreet has published a lengthy blog post describing his side of the story.


to post comments

A note

Posted Nov 23, 2024 2:12 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (20 responses)

Michal's gotten some hate mail because of this, so I'm going to repeat what I said in the blog post and ask people to please not do that. I doubt any will come by way of lwn (I hope), but I think it still worth saying.

The only reason I raised this as far as I did is because I think we do have some cultural issues worth talking about and seeing if I can address. I don't want individuals getting attacked; this was just a personal story to illustrate things that are not terribly atypical for LKML.

A note

Posted Nov 24, 2024 14:49 UTC (Sun) by lkundrak (subscriber, #43452) [Link] (11 responses)

Your message was hate mail, sir.

A note

Posted Nov 24, 2024 15:05 UTC (Sun) by pizza (subscriber, #46) [Link] (4 responses)

> Your message was hate mail, sir.

While I applaud your attempt to be polite and insulting at the same time, 'hate' in this context has legal definitions, nearly always containing explicit threats of some sort, usually tied in with some sort of protected category (eg gender, race, disabilities, that sort of thing)

This may be "insulting" but calling it "hate" massively belittles those who have, and still do, receive vitriolic missives that all-too-frequently escalate into physical harm.

A note

Posted Nov 24, 2024 15:21 UTC (Sun) by intelfx (subscriber, #130118) [Link]

> While I applaud your attempt to be polite and insulting at the same time

This is most definitely not something that ought to be applauded.

A note

Posted Nov 24, 2024 15:23 UTC (Sun) by lkundrak (subscriber, #43452) [Link]

My apologies if it sounded that way.

My criterion was not a legal one. What I considered "hate" were insults with zero bearing on technical merit, that I could only understand as stemming from personal animosity.

A note

Posted Nov 24, 2024 15:32 UTC (Sun) by pbonzini (subscriber, #60935) [Link] (1 responses)

lkundrak's point, as I understood it, was that the tone for the emails that Michal got was a direct consequence and emulation of the tone that Kent used on LKML. Therefore, if Kent applies the term "hate mail" to anything that his fanboys sent in Michael's direction, he should first of all own the fact that he was the first to send such "hate mail".

I'm more than willing to accept a legally imprecise use of the term considering that Kent's phrasing, word by word, *would* be considered hate speech in another context (for example if directed to an ethnic or gender minority).

A note

Posted Nov 24, 2024 19:48 UTC (Sun) by daroc (editor, #160859) [Link]

I think that very few of us reading this are lawyers, and that speculating about legal categories is unlikely to be helpful. I also don't see this branch of the conversation covering anything new, so let's leave it here.

A note

Posted Nov 24, 2024 22:29 UTC (Sun) by koverstreet (✭ supporter ✭, #4296) [Link] (5 responses)

If I screwed up, and a user lost data, and I was being dismissive of the problem, and they came swearing at me, I would not at all hold that against them - or if it was another engineer working on my code who lost a day due to my code being difficult to work on, same thing. In my book, at work, my responsibilities come before my personal feelings.

Obviously there has to be limits on this sort of thing; we can't be driving away and we can't be venting all the time. But on the "being nice" vs. "prioritizing our responsibilities" axis, I think the pendulum has swung a bit too far.

Certainly in bcachefs land I wouldn't mind hearing a bit more (well justified, high information content) rants and complaints.

A note

Posted Nov 25, 2024 0:19 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> But on the "being nice" vs. "prioritizing our responsibilities" axis, I think the pendulum has swung a bit too far.

I've been funding bcachefs via Patreon for quite some time. I like what you're doing, and I'm actually on your side in this particular argument.

However, it was not the heated discussion that caused the action, but a direct personal attack on an individual. This is not acceptable, and it's never a good idea anyway.

A note

Posted Nov 27, 2024 13:23 UTC (Wed) by Rudd-O (guest, #61155) [Link] (1 responses)

Kent was banned for insufficient public groveling — what in my estimation was a conspicuous exercise of power by the committee.

He had already sought to discuss and settle the matter with the individual. He also publicly acknowledged that he lost his cool.

Full context with receipts here:

https://www.youtube.com/watch?v=EeEo8Mg6fMg

I'm a long time ZFS user, but I think I will start contributing to Kent's Patreon now.

A note

Posted Dec 20, 2024 17:18 UTC (Fri) by tbird20d (subscriber, #1901) [Link]

That video contains a number of inaccuracies about the event, the COC, Dan Williams, Linus, Greg and the overall community. I wouldn't use it as a source of information about this situation.

A note

Posted Nov 26, 2024 1:25 UTC (Tue) by AdamW (subscriber, #48457) [Link] (1 responses)

Whether you're okay with other people swearing at you and telling you to get your head examined is up to you, yes.

Whether other people are okay with you doing it to them is up to *them*. Not you.

A note

Posted Nov 30, 2024 23:19 UTC (Sat) by raven667 (subscriber, #5198) [Link]

> Whether you're okay with other people swearing at you and telling you to get your head examined is up to you, yes.
> Whether other people are okay with you doing it to them is up to *them*. Not you.

To put a finer point on it, even if two people have the personal relationship and consent to speak more roughly with each other, as we sometimes do with close friends and colleagues, in a public forum that may not be appropriate behavior to model for the other participants as some will emulate that tone without context and others will quietly leave because they don't consent to be treated that way. The most senior and respected members of a forum set the tone in how they conduct themselves and what they accept from others.

one question

Posted Nov 25, 2024 15:34 UTC (Mon) by bmorel (guest, #138892) [Link] (3 responses)

Hello. I have only one question after reading a bit.

I do now know enough (nor am seriously interested into, really) of the drama to talk about it, but I have the impression that they changed the CoC so that they would be able to apply a sanction retro-actively? I do not know what the aim of the organisation is, but laws or rules being retroactively applied is one of the characteristics of terror-based government.

one question

Posted Dec 14, 2024 20:51 UTC (Sat) by jospoortvliet (guest, #33164) [Link] (2 responses)

It was more a matter of not having had to apply any sanctions before. Until now, when a developer resorted to name-calling or other unprofessional behavior during an argument, they would reflect on their actions when it was pointed out that their behavior was counterproductive and demotivating to others. They were generally mature enough to apologize afterward.

I'm sure not all people, when being asked or nudged to apologise publicly, appreciate that. But the kernel is a diverse and large community, and everybody has to adapt. Some people find it hard to recognise the effects of their own behavior and some are misbehaving because it strokes their ego to get away with it; or they feel a need to keep out others who they feel aren't up to their 'standards' by such behavior. Other people just always have to win, or get very stubborn when they encounter what they see as an authority. I can empathise with that last one, personally.

Whatever the reason, a single person who misbehaves can easily keep tens to hundres of people from engaging in the community - so, no matter how brilliant somebody is, if they can't play well with others the kernel will be better off without them.

Of course, the best solution would be for people to act like adults, learn and adapt - personal growth and all that. Or, if not, grumpily behave even if you don't agree. Sometimes you just have to take the loss. Again, I can emphatise.

I'm a little triggered by a piece in Kent's blog:

> But, my response was to say "no" to a public apology, for a variety of reasons: because this was the result of an ongoing situation that had now impacted two different teams and projects, and I think that issue needs attention - and I think there's broader issues at stake here, regarding the CoC board.

That really sounds like somebody who knows they are wrong but does not want to give in - does not want to 'lose' and thus tries to present it somehow as a "principled stand". And then, of course, we see him trying to present himself as the 'big guy' by telling everyone not to follow his example and also yell at the maintainer he yelled at. Expecting applause for telling everybody not to cut themselves on the knife you used to stab somebody? Hum, we're not in kindergarten.

Anyhow, that's why...

one question

Posted Dec 15, 2024 1:17 UTC (Sun) by intelfx (subscriber, #130118) [Link] (1 responses)

> Whatever the reason, a single person who misbehaves can easily keep tens to hundres of people from engaging in the community - so, no matter how brilliant somebody is, if they can't play well with others the kernel will be better off without them.

Yeah, like Viro and Torvalds and a few other senior maintainers who apparently get a free pass on any sort of verbal abuse, passive aggressiveness and all kinds of other underhanded behavior (cf. the recent Rust-for-Linux upheaval).

> Some people find it hard to recognise the effects of their own behavior and some are misbehaving because it strokes their ego to get away with it; or they feel a need to keep out others who they feel aren't up to their 'standards' by such behavior.

Yup, precisely.

I wish these stringent standards of conduct were applied to everybody equally, not just to the boogeymen du jour.

one question

Posted Jan 16, 2025 23:25 UTC (Thu) by jospoortvliet (guest, #33164) [Link]

I'd argue that a free pass wasn't given, and Linus has the receipts - https://lkml.org/lkml/2018/9/16/167 for example. I suppose Kent would benefit from such a thing, too.

I'm not saying they're perfect, or ever were. But saying they got no pushback, or that they ignored said pushback, is just not true.

A note

Posted Nov 26, 2024 7:08 UTC (Tue) by mstsxfx (subscriber, #41804) [Link] (2 responses)

Thanks for discouraging people from personal attacks Kent!

Let me just add couple of notes and clarifications.
- I didn't contact anyone to enforce the Code of Conduct in this case. I'm mentioning this because of some private accusations I've received about using the Code of Conduct to exert power.
- I'm fine with robust technical discussions and don't mind strong language directed at me. My cultural background is much closer to a direct and a potentially strong sounding wording than what would be considered much more politically correct by others. Personal attacks are neither productive nor they should be acceptable though. Any discussion stops there for me.
- Sometimes there's no single right solution, and the most clever one isn't always the best way forward. Differences of opinion between reviewers/maintainers and patch submitters are expected. Therefore different POV between reviewers/maintainers and patch submitters is normal and we should be able to sort those during a factual discussion. Our process puts more burden on submitter to defend their approach. That might sounds unfair but this is a generally recognized process by the community. On the other hand maintainers should be always fair, respectful and open to arguments.
- My response to Kent's public call out about feedback process is here https://lore.kernel.org/all/Zz7yqeI0RatH4ao5@tiehlicka/T/#u. I have tried to reference main entry points to discussions and I am open to hear specific feedback where my acting as a maintainer/reviewer could have been better or where it was unacceptable as Kent has concluded.
- I was in private discussions with Kent prior publishing his Patreon post. I have tried to clarify my thinking process in discussions that he has pointed out in his blog post. I felt like this could help to narrow down the disconnect. I also had some disagreement in the description of those discussions and asked him to put direct references to those discussions so that curious readers could follow them without an additional burden of digging those up from archives.
- The Code of Conduct committee contacted me last week for my opinion. I am not going to partially quote from that discussion but TL;DR summary would be that I do not insist on public apology nor I am a great proponent of enforced actions against individuals. I would much rather appreciate a private moderation while the discussion is tense and evolving in an unproductive way than punishment after the fact.
- I do not want to judge CoC cmte decision in this case, though, because I was not part of private discussions. It is really hard to make an opinion without having the full context IMHO.

Michal Hocko

A note

Posted Nov 26, 2024 8:16 UTC (Tue) by koverstreet (✭ supporter ✭, #4296) [Link] (1 responses)

Thanks Michal :)

Sorry I didn't end up doing the rewrite to incorporate your feedback - I ended up putting up what I had because at the last minute, when it seemed like things might go somewhere productive with the CoC, they didn't, so I felt I had to go with what I had, I'd pretty much lost energy to do more at that point.

Hoping we can get something done on PF_MEMALLOC_NORECLAIM for the Rust folks at some point - let me buy you a beer at LSF?

A note

Posted Nov 26, 2024 8:39 UTC (Tue) by mstsxfx (subscriber, #41804) [Link]

> Hoping we can get something done on PF_MEMALLOC_NORECLAIM for the Rust folks at some point

TBH I am not even sure what this means. So please try to define the problem (why does Rust support need this). All ideally on the mailing list.

> let me buy you a beer at LSF?

I am always open to talk.

About productivity

Posted Nov 30, 2024 1:56 UTC (Sat) by ReDress (guest, #174819) [Link]

Hello,

You talk about productivity. You did talk about productivity a few times. In the blog post and probably also here in the comments.

This seems like a valid concern to me since we're yet to see how the CoC ages. It will probably age well but at the same time, leaving the allowance that it leads to reduced productivity might be a good idea until it is proven and it is evident that it does not. At the same time, since the otherwise is not proveably true, it doesn't seem fair to blame the CoC.

That aside, ideas on how to avoid reduced productivity in kernel development might be great. Ideas on how to improve productivity regardless of the CoC would be great.

PS: Anarchy is not known to increase productivity.

Why only here ...

Posted Nov 23, 2024 2:24 UTC (Sat) by JMB (guest, #74439) [Link] (19 responses)

Well, I am following kernel news for a long time - and even though I would not use
that words (only in private ... yes - but never in public), it is not extraordinary for LKML.

I would really like to have explanation why excluding Russian kernel hackers and deleting
them from the maintainers file - without any personal communication - was not officially
dealt with by CoC - from my point of view this would be obligatory!
This was much more severe - especially if it is true that some of them were not include
in any list (which was reported by some news).

Giving the impression that Russians may deserve that treatment just by warfare of their
gouvenment is also not justifyable in an international environment - except they could change
those things or had been involved personally. I understand bad blood - but not such an abuse!

The way Rust people were blocked to give a talk - nothing happened - and those videos were
sent on exiting the kernel community - and CoC did ... nothing.
Sorry, but this is really bad - and will harm the kernel community.

And here it is on technical ground - and was treated on a much too high level - or
did that happen before? Why now - after so many much worse stories went viral.

From my perspective there is no justification for this suspension ... and I don't see
that a line is drawn and explained to really do justice - and help in improving the tone
to have more professional diskussions on LKML.
So from my perspective this was just a bad show ...

Additionally - why had been mitigations (some of them without positive effect)
entering in a hurry without -rc? - and bcachefs being experimental can not be worked
on when real user problems emerged and can only fixed by rewriting passages.
Code touching other regions should be limited to the merge window - of cause.
But I would understand big code changes if to help users of that FS.

So I would like Mr. Corbet to give some insight of why this is played on that level in
comparison with much severe problems seen on videos etc. - and other things
were not even brought to light.
And if one can bring this to perspective so CoC is not a bad show but helping
better understanding each other ... an being a more healthy community - this piece
is totally missing from my perspective.

If this should be fair - I would like to read about problematic behaviour and possible sanctions
before these are first used!
It is hard to get real information about the line which should be respected.
And thus it smells very fishy ... and I could cite a lot of other examples where CoC should
have interfered ... but nothing happened.

Why only here ...

Posted Nov 23, 2024 2:34 UTC (Sat) by cypherpunks2 (guest, #152408) [Link]

This seems somewhat rambling and doesn't make much sense. Are you asking why the TAB dealt with this but did not deal with other, unrelated events that in your view are worthy of TAB enforcement actions? The difference is that this was an action taken based on a heated exchange. It is qualitatively different from the other examples you provided, so they can't really be compared.

Why only here ...

Posted Nov 23, 2024 8:52 UTC (Sat) by patrick_g (subscriber, #44470) [Link] (4 responses)

> I would really like to have explanation why excluding Russian kernel hackers and deleting them from the maintainers file - without any personal communication - was not officially dealt with by CoC

Because that's not what happened?
As far as I know some kernel hackers were excluded because they were employed by firms under international sanctions. The nationality of the hackers are irrelevant. They can be Russian or Japanese or Norwegian, it doesn't matter. What matters is the fact that they were employed by a firm under international sanctions.
And Russian hackers working for firms NOT under sanctions were NOT excluded.

Why only here ...

Posted Nov 23, 2024 18:55 UTC (Sat) by geert (subscriber, #98403) [Link] (1 responses)

Why only here ...

Posted Nov 24, 2024 5:12 UTC (Sun) by HenrikH (subscriber, #31152) [Link]

that wasn't banning a single individual from sending in patches to the kernel. Tons of people suddenly trying to pretend being Linux wizards but somehow not knowing that the Maintainers file is for.

Why only here ...

Posted Nov 25, 2024 9:37 UTC (Mon) by rbtree (guest, #129790) [Link] (1 responses)

tbh it was really badly handled and it wasn't difficult to jump to certain conclusions because of complete lack of transparency and some conspicuous patterns in who were being kicked out: when you're being discriminated against all the time, you automatically begin taking everything personally and see things that aren't there — just ask any "minority" person you know well. The actual reasons for maintainers' termination were posted much later, and some remarks by Linus (in his usual style) also didn't help to calm things down. International cooperation is quickly falling apart, this is just the beginning.

Why only here ...

Posted Nov 25, 2024 13:18 UTC (Mon) by daroc (editor, #160859) [Link]

I think at this point this thread of the discussion is becoming off-topic; this article is about the recent CoC enforcement action in the kernel, not about the changes to the MAINTAINERS file.

Why only here ...

Posted Nov 23, 2024 9:33 UTC (Sat) by tytso (subscriber, #9993) [Link] (6 responses)

LSF/MM sessions are expected to be for discussions, not talks. And in the cast of the Rust session at the LSF/MM, most of the people in the room couldn't understand what they were trying to do with the Rust code that was on the slide. Unlike in lectures, where if the professor gives an incomprehensible talk, the audience just sits and lets it wash over them, it is expected that people stop the speaker and ask the speaker to clarify.

The problem was that the speakers chose an example which was too complicated; they may have been proud of the expressive power of Rust, but if the goal is to try to convince people that using Rust for file system code is a good idea, it might have been a tactical mistake. If the speakers had 2-3 hours to give.a tutorial on the Rust language; of its syntax and semantics, then as the climax of the talk, perhaps they could have shown off this super-clever construction. But they ran out of time, because they only had 30 minutes, and they pushed the audience into the deep end of the pool without any preparation --- is it any wonder that a lot of questions were asked and the discussion became tense?

So no, I don't think there was any Code of Conduct violation at the Rust session. It is not a CoC violation to have technical discussions; even tense technical discussions. It is fine to attack ideas; where it crosses a line is when people are attacked.

The technical points which Kent has raised, for example in his Patreon post, are really beside the point. Regardless of the technical arguments he might have been trying to make, it was the personal attack which went beyond the pale vis-a-vis the Code of Conduct. And while Kent might have been frustrated by the technical discussion, it didn't justify the public personal attack, and the way to repair the harm from a public attack is a public apology.

Finally, note that there have been other Code of Conduct violations over the past six years, and the Code of Conduct committee has investigated and stepped in to address them. The only reason why this situation has been exceptional is because it is the first time that an enforcement action has been needed; most of the time, coaching and education from the Code of Conduct committee is sufficient. See the Code of Conduct committee reports here: https://www.kernel.org/code-of-conduct.html

Why only here ...

Posted Nov 24, 2024 12:14 UTC (Sun) by rweikusat2 (subscriber, #117920) [Link] (5 responses)

> It is fine to attack ideas

Criticizing other people's ideas will usually be interpreted as the strongest conceivable form of personal attack and will usually cause them to go completely ballistic when (as they believe they are) retaliating in kind and this behaviour will even usually gain almost unanimous public support.

Why only here ...

Posted Nov 24, 2024 16:26 UTC (Sun) by tytso (subscriber, #9993) [Link] (3 responses)

Sorry, I can't agree with you. If someone interprets criticism of their ideas as the strongest form of personal attack, they are not going to survive long in Kernel development. We are human, and we all mistakes. If we put out something which is wrong, other people should and must point out that it is wrong. So for example, if someone sends out a buggy patch, and it gets NACK'ed, this is not a personal attack.

If someone suggests an idea that might work for their subsystem, such as for example mm, but some other subsystem maintainer (say, bcachefs) thinks it doesn't work for them, and the bcachefs maintainer attacks the mm subsystem proposal, this is not a personal attack. It is an attack on an idea, not a person. And by the way, Kent has done this a lot, himself, but this is not why he was called out. He was called out because of his attack on a persom, not on how he argued his ideas.

It's also not true that such behavior will gain "unanimous public support". Even on Kent's Patreon page, some of his supporters/funders called him out as going too far. I am hope that perhaps if he were to meditate on how he has started to lose the room even among what should be his strongest supporters, maybe he might figure out that his behavior is ultimately self-defeating.

Why only here ...

Posted Nov 24, 2024 18:33 UTC (Sun) by rweikusat2 (subscriber, #117920) [Link] (2 responses)

Many people tend to take criticism of their ideas personal, especially so if this criticism comes from people who have only a low or even no status at all in some well-established pecking order and targets ideas of people rather towards the top of it. The 'classic' (of some time ago) Linux kernel example was a guy who once wrote a device driver for the networking bits of an Intel IXP425 SoC which included a single ARM machine instruction in a place where this made sense (clz, by that time not already a gcc builtin). He got basically flamed to crisp for this for no technical reason at all and was, among other things, told that mere 'users' are not supposed to write their own device drivers without getting permission first¹.

I totally agree with your point: That's how it should be. But in practice, it frequently ain't so.

¹ As I needed such a driver at that time, I picked this up as a starting point but ended up rewriting most of it to make better use of the features of the hardware.

Why only here ...

Posted Nov 24, 2024 20:20 UTC (Sun) by marcH (subscriber, #57642) [Link] (1 responses)

> > Sorry, I can't agree with you. If someone interprets criticism of their ideas as the strongest form of personal attack, they are not going to survive long in Kernel development.

> I totally agree with your point: That's how it should be. But in practice, it frequently ain't so.

A lot of people are not comfortable working with open-source for various reasons, one of them being they have some level of emotional connection to their work and feel as you describe.

That's OK - there are plenty of closed-source jobs for them.

Why only here ...

Posted Nov 25, 2024 9:39 UTC (Mon) by rweikusat2 (subscriber, #117920) [Link]

The guy who exploded like a long dormant volcano because some nobody had dared to raise rational objections to one of his sacrosanct policy decisions was Russell King and I was only tangentially involved with that, ie, got only a minor share of the resulting torrent of abuse.

Why only here ...

Posted Nov 26, 2024 15:34 UTC (Tue) by nicbr (guest, #174696) [Link]

In the "regular", closed source software world, (good) ideas get shutdown regularly. That's just life in corporate environment. On top of that, you sometimes have to help fix issues with the new software, when you know that your shutdown-idea wouldn't have had those issues. Again, that's just life in the corporate world.

You can choose to leave, sure, but ime the grass is rarely greener somewhere else.

I obviously know that open source is supposed to be guided by higher principles than for-profit software, but accepting that some ideas can get rejected should never be considered a personal attack. Even if the rejection itself is virulent.

Why only here ...

Posted Nov 23, 2024 11:12 UTC (Sat) by Lennie (subscriber, #49641) [Link] (4 responses)

> I would really like to have explanation why excluding Russian kernel hackers and deleting
them from the maintainers file - without any personal communication

Sadly, because they were not allowed by the legal department.

Why only here ...

Posted Nov 24, 2024 14:36 UTC (Sun) by amarao (guest, #87073) [Link] (3 responses)

Yes, that is exactly the thing kernel is missing. Legal department. We also in desperate need to have HR department to do preliminary screening for potential opensource contributors. Some mandatory training in diversity and inclusivity is a must too. Also, fire drills. And, but of course, we need yearly performance reviews done by MBA-trained managers. And a good KPI system will allow to do more precise layoffs when needed.

Without all of it opensource is doomed to be a bunch of hackkers and nerds, and never be able to achieve enterprise enlightenment.

Why only here ...

Posted Nov 24, 2024 15:44 UTC (Sun) by pbonzini (subscriber, #60935) [Link] (2 responses)

The Linux kernel moves billions of dollars. That's definitely an amount where it's worth hiring a lawyer to advise you about international law.

Why only here ...

Posted Nov 24, 2024 16:07 UTC (Sun) by amarao (guest, #87073) [Link] (1 responses)

I understand that some companies are making billions out of Linux and pouring millions in it. I don't understand, why their voices should be the most important.

Why only here ...

Posted Nov 24, 2024 20:25 UTC (Sun) by pbonzini (subscriber, #60935) [Link]

I never mentioned the voices of any companies. Linux is hosted by Linux Foundation which, for better or worse, is based in the United States of America. Both Linux Foundation and Linux the open source project have to respect the laws of the USA(*) and, given the huge impact that the project has worldwide, that includes paying attention to international sanctions. How to do that is anybody's guess; Linux Foundation's lawyers gave their opinion, I can't judge if they're right but it's quite certainly more valuable than "just ignore the law".

(*) Things wouldn't be different for many other countries though

Why only here ...

Posted Nov 23, 2024 13:01 UTC (Sat) by daroc (editor, #160859) [Link]

> So I would like Mr. Corbet to give some insight of why this is played on that level in
comparison with much severe problems seen on videos etc. - and other things
were not even brought to light.

I am not Mr. Corbet, of course, but I will point out that LWN did in fact cover the situation that led to Wedson Almeida Filho retiring from the project [1], the removal of maintainers who work for sanctioned entities from the MAINTAINERS file [2], and, indeed, many other newsworthy events within the Linux community. It's true that we don't report on every mailing list fracas; this one is notable because it has caused a new response from the CoC committee, and because it may impact users of bcachefs.

If you have tips for stories that you think are particularly notable that we seem to have missed out on, you're welcome to email us at lwn@lwn.net.

[1]: https://lwn.net/Articles/987635/
[2]: https://lwn.net/Articles/995186/

Excessive action

Posted Nov 23, 2024 2:31 UTC (Sat) by cypherpunks2 (guest, #152408) [Link] (12 responses)

Wouldn't a better solution have been to enforce DNI (Do Not Interact) with Michal? This is not a community that people join for pleasure where temporarily expelling someone hurts no one other than the person being expelled. Blocking pull requests seems to me like one of the least efficient actions that could have been taken and the one that has the most adverse effects on kernel development. This isn't to say that the behavior should be ignored, but there are so many alternatives that seem to not have even been considered.

At least it's limited to only one development cycle.

Excessive action

Posted Nov 23, 2024 2:41 UTC (Sat) by dskoll (subscriber, #1630) [Link]

Yeah, it seemed out of proportion to me too, though I agree the language and tone was not acceptable. There was a public apology of sorts in the end in this message. But I guess it came after the sanction had already been decided.

Excessive action

Posted Nov 23, 2024 2:57 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (10 responses)

A DNI wouldn't have been appropriate, since we continued to talk about technical stuff.

I'm wondering why no one seems to be able to imagine just saying "hey, this seems tense, what can we do to help", _before_ taking some sort of enforcement action.

Right now, when things on the list get heated, it often ends up as a dogpile - and then if it gets to that point the CoC just picks whoever mouthed off the worst and slaps them.

Hardly conducive to keeping technical discussions on track, wouldn't you say?

Excessive action

Posted Nov 23, 2024 3:18 UTC (Sat) by cypherpunks2 (guest, #152408) [Link] (7 responses)

> Right now, when things on the list get heated, it often ends up as a dogpile - and then if it gets to that point the CoC just picks whoever mouthed off the worst and slaps them.

That is a problem. I am hopeful that this action will make enough waves that people rethink the best way to keep things from getting too heated, so long as it doesn't attract so much controversy that development is further derailed. I expect that people will recognize the issues with what happened and will agree with you. I have hope that people will have enough of a nuanced view to understand that it is okay to criticize the TAB's actions without defending or minimizing inappropriate behavior.

Excessive action

Posted Nov 23, 2024 3:23 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (6 responses)

Imagine if people were just nicer to each other. Not just used more nice words, but actually watched out for each other, as colleagues and friends.

But the people at the top set the tone, so I think that would have to start there.

Secondarily, I do think focusing too much on avoiding heated words and not enough on deeper issues of being able to make progress is a real issue as well, because it means being dismissive in an argument is the safest option.

When things get heated I find that there's often something interesting underneath.

Excessive action

Posted Nov 23, 2024 3:35 UTC (Sat) by cypherpunks2 (guest, #152408) [Link] (2 responses)

> Secondarily, I do think focusing too much on avoiding heated words and not enough on deeper issues of being able to make progress is a real issue as well

In the past there has been almost nothing done to stop a community from becoming toxic and abusive behavior was accepted from people who were also long-term contributors. People are swinging too far in the opposite direction to fix things, prioritizing polite discourse to such an extent that it can actually inhibit useful contributions. That's not to say that we should go back to the wild west past, but we shouldn't over-correct and create an atmosphere where people fear not personal attacks, but enforcement actions from up high.

Excessive action

Posted Nov 23, 2024 3:40 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (1 responses)

Yep, exactly. And if you'd seen some of what I've had to put up with lately, some of the people at the top never stopped being assholes.

And it's not just about striking a balance, that leads to very waffly "will we or won't we" enforcement that just drags things out, where the enforcers don't want to commit but everyone knows they're going to. The only way out is coming up with an idea of what we _do_ want to see.

Excessive action

Posted Nov 24, 2024 15:40 UTC (Sun) by brchrisman (subscriber, #71769) [Link]

I'm a little curious. Were you concerned that there had been private/off-list discussions relating to matters which were critical to your various goals/projects? I'm pretty spectacularly uninformed but interested in the context.

Excessive action

Posted Nov 23, 2024 16:02 UTC (Sat) by raven667 (subscriber, #5198) [Link] (2 responses)

> Imagine if people were just nicer to each other. Not just used more nice words, but actually watched out for each other, as colleagues and friends.

I think the CoC _is_ looking out for you and your colleagues by stepping in and giving you the opportunity to have a break from the stress, anxiety and subsequent anger that underlies the conflict, but your colleagues are ultimately not responsible for control of your feelings and choices in how you respond to frustrating situations, you need to figure out where the fear and anger come from and why it gets you off your game, I don't know you personally I hope you take the opportunity to decompress and come back with a fresh positive outlook. I'm not sure publicly debating randos in the comments on lwn is doing anything except aggravating a fresh wound, I don't think it's going to help you be the best collaborator, advocate and engineer for bcachefs that you clearly want to be.

Excessive action

Posted Nov 24, 2024 1:47 UTC (Sun) by cypherpunks2 (guest, #152408) [Link] (1 responses)

> I'm not sure publicly debating randos in the comments on lwn is doing anything except aggravating a fresh wound

As of the time this comment was posted, I haven't seen Kent debate anyone. In fact, the threads here seem calm and rational. Most people seem to be largely in agreement with him (i.e. sharing an understanding that his response was indeed rude but that the TAB's actions were excessive nonetheless). None of the comments here seem even remotely bellicose.

Excessive action

Posted Nov 24, 2024 13:10 UTC (Sun) by daroc (editor, #160859) [Link]

I think that it's possible to debate someone while being calm and rational, and I have seen a few comments push back against Kent. I do think you're right that everyone has remained polite and has had interesting things to say. I review the night's comments over breakfast each morning, and it's always pleasant when that means reading through a thoughtful discussion instead of needing to delete things or write admonishing comments.

Excessive action

Posted Nov 23, 2024 13:44 UTC (Sat) by zeroheure (guest, #165859) [Link]

> I'm wondering why no one seems to be able to imagine just saying "hey, this seems tense, what can we do to help", _before_ taking some sort of enforcement action.

Not exactly that, but the first reaction was Jonathan Corbet suggested "to calming down a bit". May be you missed his post ?

See https://lore.kernel.org/lkml/8734mitahm.fsf@trenco.lwn.net/

Excessive action

Posted Nov 24, 2024 19:53 UTC (Sun) by chexo4 (subscriber, #169500) [Link]

Right now, when things on the list get heated, it often ends up as a dogpile - and then if it gets to that point the CoC just picks whoever mouthed off the worst and slaps them.
Okay… have you considered, I don’t know, not being the one mouthing off the worst? You’re going to kill your own project, and none of it’s technical merits will save it from your behavior.

Kent's blog post

Posted Nov 23, 2024 6:39 UTC (Sat) by geofft (subscriber, #59789) [Link] (18 responses)

I appreciate Kent setting out his side of the story in a thoughtful way.

But I'm not sure I buy the argument that this sort of work inherently involves being acrimonious with each other in code reviews, even if you're otherwise chummy outside of that context.

"Filesystem work is a stressful job" is such an odd claim. Why? IIt's nothing like being a doctor or lawyer or general or diplomat or air traffic controller where situations can deteriorate in seconds and if you aren't doing your best work on zero notice, lives will be lost. It's not even like being an on-call sysadmin/SRE/etc. for a production server whose filesystem is misbehaving. The code in front of you isn't going to change until you run "git pull" and it's entirely at your discretion when to push, too. (Security vulnerabilities are an exception, but the post only talks about them in the context of designs that are known at the time of writing to lead to DoSes, not about a buffer overflow or speculation side channel being discovered and a race to patch it before it's exploited.) It may be a technically difficult and therefore mentally taxing job, but it should be no more stressful than being a sculptor or a carpenter or a poet.

I think Kent's examples actually argue that stress makes the job worse: if people hadn't been so acrimonious about the folio reviews, they might have kept conversing to the point of finding that they shared the same long-term vision without needing a third-party mediator to figure it out. In fact the email at issue here is another counterexample: there was nothing about the code itself nor the technical disagreement that required being acrimonious (as evidenced by Kent's levelheaded explanation of the issue in the blog post). There is nothing inherent about Kent nor his interlocutor that meant that conversations must get acrimonious. It was the increasing heat in the thread that made Kent, in his telling, lose his cool and send a message that is objectively beyond the pale (and, in my opinion, still merits an uncomplicated apology). So the answer is that the thread shouldn't have started to get hot in the first place, and people should have stepped in at the first sign of rancor.

And I do question the idea that people work best by yelling at each other over their code and then being friendly once it's over. In my experience there are some people who genuinely like intense arguments, and some who find them uncomfortable. There are people in this latter group who have nonetheless learned to perform the intense argument because they want to make some progress with people in the former group. But that does not at all mean they like doing so or that this process does not burn them out over time. If it really does work for a specific pair of people - they can yell at each other as much as they feel necessary over direct email without copying a list, over a call, or whatever, and then it's opt-in.

From the stories in the blog post it sounds like heated disagreement is often used not as a tool in itself but a way to push people to expand on technical detail when it isn't forthcoming - to push back against the spreadsheet of decisions without rationale, to cause people to realize that an uninvolved mediator was needed, etc. If this is indeed what's going on, then Linus and the TAB should step in here - maintainers should not be able to block patches without an explanation understandable to at least one of the patch submitter, Linus, or someone on the TAB (which is a _technical_ advisory board, despite its role in CoC enforcement). People who have long-term plans in mind should be encouraged to publicize them. And it seems odd to me to support bypassing a maintainer and landing the patch in Linus's or some other tree directly, while also letting the maintainer continue to hold that role. Either the maintainer had a justified objection, and bypassing will just cause the maintainer to pick up code they'll want to revert (as happened here), or the person should not actually be trusted with the ability to block patches.

I do agree with Kent's comments elsewhere in this comment section that this is an overall culture problem and kernel leadership could do a better job of setting the right tone from the beginning.

Kent's blog post

Posted Nov 23, 2024 7:24 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (14 responses)

> But I'm not sure I buy the argument that this sort of work inherently involves being acrimonious with each other in code reviews, even if you're otherwise chummy outside of that context.

The fight over whether XFS would have metadata checksumming split the team.

Kent's blog post

Posted Nov 23, 2024 22:25 UTC (Sat) by Heretic_Blacksheep (guest, #169992) [Link] (13 responses)

I'm about to say something my parents used to tell me often. People call it trite sometimes. But it's the basis of the principle of de-escalation. It's appropriate in this case. "You are not responsible for the actions of others, only your own. Just because everyone else is being a jerk to each other doesn't give you license to be one yourself." Be the solution, not the problem. It's the difficult path. The path less taken.

People can disagree without being abusive with each other. Just because the other side becomes abusive doesn't mean you must as well. People can lose an argument gracefully. There's no loss of face to not having one's ideas accepted or implemented - even if those ideas are rejected for what might appear to be silly reasons to you. Your apologies read as rationalizations rather than genuine apologies. You're doubling down on your behavior rather than altering it. It's coming to the point where the new community safeties are engaging because you're not altering your course.

Kent's blog post

Posted Nov 23, 2024 22:51 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (12 responses)

We'd all be worse off if everyone took that attitude.

Sometimes the fight is worth having.

Kent's blog post

Posted Nov 23, 2024 23:30 UTC (Sat) by geofft (subscriber, #59789) [Link]

Nothing in the comment you are replying to suggests the fight is not worth having. In fact, it is precisely because it is worth having that it is worth being respectful and kind during the fight, so that people aren't incentivized to step away from it.

Acrimony does not enable forward progress

Posted Nov 24, 2024 3:14 UTC (Sun) by NYKevin (subscriber, #129325) [Link]

Acrimony does not enable forward progress, it impedes it. The more acrimonious a thread becomes, the harder it is for everyone involved to reach a conclusion they can all accept. Your own blog post describes an example of this very process playing out.

Do you really think that those folks getting angry at each other *helped* them reach consensus?

Kent's blog post

Posted Nov 24, 2024 10:47 UTC (Sun) by gmgod (guest, #143864) [Link]

This is a fight worth having but it's in the middle of many other fights.

Yeah people are too hung up on apparences. They are denying other people's right to have opinions and feelings (especially sadness and anger), and their right to express them...

But in this case, ignoring the puritans, there is still a person that comes with grand claims and wants to change existing processes for himself while not giving the group time to acknowledge technical prowess nor intelligence. If you don't play by the rules, at least for a while, you'll have a hard time convincing anybody that they need be changed.

This comes atop of that. I cannot see it as anything other than a political move. CoCs are usually worded so subjectively that they cannot be used for anything else anyway.

I know it's frustrating to have excellent ideas and to see ways of improving things for everybody but with gatekeepers that prevent you from doing so for reasons they can't even articulate.

But at some point, everybody's human and that aspect has to be dealt with. No fights will be won if the group sees you as a disruptor, if the maintainers see you as demanding special casing and if people at large feel offended on behalf of others because of insinuations... I find it a shame that, having your wits, you've not yet understood that human dynamics, however stupid they might look, are a fact of life that you need to purposely navigate. Ignoring them won't serve you, nor your objectives, well.

Kent's blog post

Posted Nov 24, 2024 10:59 UTC (Sun) by RX14 (subscriber, #123970) [Link]

I agree the disagreement is worth having, that's all the better reason for it not to turn into a fight.

Kent's blog post

Posted Nov 24, 2024 20:04 UTC (Sun) by rra (subscriber, #99804) [Link] (3 responses)

> We'd all be worse off if everyone took that attitude.
> Sometimes the fight is worth having.

And this is exactly the problem: you are at present unable to see how to have the fight without being abusive. No one is arguing against having the fight, only about how the fight is conducted.

I'm just some random person on the Internet who has been in a lot of technical fights (in Debian, not in kernel development, so the context is admittedly different). You don't know me and you have no reason to take my advice. But my natural tendency is the same: I want to have a blunt discussion at the same level that the other person is discussing things, and if a point is particularly important to me or if something the other person says seems particularly outrageous, I want to drive home my point with force. I invent all sorts of justifications and reasons in my mind for why this time I need to put all of my emotion in the response, because some things DESERVE that emotion, damn it.

It doesn't help. It just doesn't. As much as I try to convince myself that this time for sure it will serve a purpose, it doesn't, not really.

The emotion is a crutch. The abuse is a crutch. It is your subconscious telling you that your argument is not working. The natural human reaction is to just apply more force, but applying more force causes damage and risks breaking all the components of the system, including you.

Roughly speaking, there are four reasons why my arguments aren't working in a discussion and I become tempted to apply more force. My argument may be bad: insufficiently precise, insufficiently detailed, insufficiently supported. The topic may be a matter of subjective judgment or future prediction and the other person fully understands my argument and simply disagrees. The other person may be simply wrong and completely unwilling to see it. Or I may be simply wrong and completely unwilling to see it.

In every case, adding the abuse and emotion doesn't help and usually makes it worse. When you apply more force in the argument, not only do you inspire MORE resistance, not less, in the other person, you also harden your own position. You make it less likely that you will be capable of realizing when you're wrong because now you're emotionally invested and you would also have to admit your emotion was unjustified. If the argument is bad, no amount of additional force applied to it will make the argument better. And if the debate is over subjective judgment, you will not convince other people to substitute your judgment for theirs by being abusive, nor should you be able to.

Pounding on the keys harder doesn't make the bug go away. You have to actually find the bug and fix it, and pounding on the keys harder doesn't help and usually hurts. It's incredibly difficult to let the emotion go and instead try to calmly understand which of those cases applies and how to move forward, but that's what works. Sometimes I've gotten away with throwing a big enough tantrum that someone else stepped in and did the work for me, but that's not the person I want to be, and based on your writing I don't think that's the person you want to be either.

I will grant that there are some exceptions for social problems where what's required is, essentially, a revolution or a threatened revolution and you need emotion to get people out into the streets and threaten violence, but I don't think that's the situation in kernel development and based on your blog post I don't think you think it is either.

Kent's blog post

Posted Nov 24, 2024 23:13 UTC (Sun) by raven667 (subscriber, #5198) [Link]

> Roughly speaking, there are four reasons why my arguments aren't working in a discussion and I become tempted to apply more force. My argument may be bad: insufficiently precise, insufficiently detailed, insufficiently supported. The topic may be a matter of subjective judgment or future prediction and the other person fully understands my argument and simply disagrees. The other person may be simply wrong and completely unwilling to see it. Or I may be simply wrong and completely unwilling to see it.

This is well said. I've found out something similar after making this mistake several times and have learned to recognize when I feel that righteous anger "how could they be so incompetent and stupid to not see X, Y, Z" that I'm the one who needs to sit down and re-evaluate before going off because most times I've gone down the other road and chose escalation I have later found out that _I_ misunderstood something fundamental and made an ass of myself requiring an apology. I've said a lot fewer dumb things that I need to walk back since then ;-)

Kent's blog post

Posted Nov 25, 2024 12:11 UTC (Mon) by beagnach (guest, #32987) [Link]

Very articulate any insightful. Thanks.

Kent's blog post

Posted Dec 23, 2024 10:22 UTC (Mon) by ras (subscriber, #33059) [Link]

One technique I use is write my email not as a reply to whoever I'm arguing with, but instead address it to the people silently following along. I try not to think about the person you are replying to at all. It ups the odds of ensuring you address the technical issues at hand, and almost eliminates the urge to inject an unwanted invective.

It's only worth doing when you've exchanged a few emails, so you've gone over a lot of ground but the temperature is rising. That is the time to re-read the exchange and re-imagine it from the point of view of a third party. Occasionally, you end up thinking "I'd be repeating myself and I don't want to bore these people", so I your email becomes "I have nothing more to add".

It has worked every time I've done it. Having the presence of mind to see through the red haze and apply it is not so certain.

Apologies are underrated

Posted Nov 24, 2024 22:26 UTC (Sun) by tome (subscriber, #3171) [Link] (3 responses)

An apology is a wonderful thing, especially for the person who makes it.

Apologies set the stage for a better future.

We apologize for offenses. An offense has a victim. The victim may very well have had it coming, but that has nothing to do with the apology. A genuine apology makes no excuses, makes no mention whatsoever of the victim's share of culpability. And it's good that it does not, because the circumstances that triggered the offense likely endure after the apology. An apology does nothing to diminish the force of inoffensive arguments made in the context of the offense. Quite the contrary, it removes a red herring.

Apologies are not easy. It takes courage to apologize. Apologizing shows courage.

An apology is disarming. It lets the victim and the offender come to their senses. It reduces strife and brings peace of mind to the apologizer.

There is no shame in apologizing, provided the apology is genuine. A genuine apology, made publicly, generates admiration for the apologizer, for honesty, sincerity, and good judgement.

It's not too late to apologize, Kent. You work hard and you argue hard. You deserve to enjoy the benefits of being a person who apologizes. You owe it to yourself to apologize.

Apologies are underrated

Posted Nov 26, 2024 17:47 UTC (Tue) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

I agree with everything you say about the value of an apology. The only problem is that a real apology has to come from a genuine acknowledgment of having made a mistake. Someone who feels like they've done something wrong will want to apologize because they feel bad about what they've done and hope to make things better by admitting fault. The main reason so many apologies are bad is because they're insincere; the person making the apology refuses to admit fault and is only apologizing because they hope to placate the person they're apologizing to. Of course it fails because it's missing the main thing the person receiving the apology wants: an admission of wrongdoing.

Apologies are underrated

Posted Nov 27, 2024 10:20 UTC (Wed) by anselm (subscriber, #2796) [Link] (1 responses)

Plus, many “apologies” tend to be not “I'm sorry for what I did” but rather various versions of “I'm sorry you didn't like what I did” or “I'm sorry that I got caught doing what I did”.

Apologies are underrated

Posted Nov 27, 2024 13:27 UTC (Wed) by Rudd-O (guest, #61155) [Link]

A.k.a. nonpologies.

Kent's blog post

Posted Nov 25, 2024 13:48 UTC (Mon) by nim-nim (subscriber, #34454) [Link] (2 responses)

> "Filesystem work is a stressful job" is such an odd claim. Why? IIt's nothing like being a doctor or lawyer or general or diplomat or air traffic controller where situations can deteriorate in seconds and if you aren't doing your best work on zero notice, lives will be lost.

But most those professions rely on stored data, usually in a filesystem, and data corruption can cause those deteriorating situations sometimes years later when some corruption condition is met. That is why working on data storage is stressful, mistakes can have long delayed effects and mistakes are not restricted to risk-free professions because everyone uses general-purpose storage.

General data storage is one part of IT where you can not just reset things and forget about past bugs. That *is* stressful for anyone that cares.

Kent's blog post

Posted Nov 30, 2024 23:01 UTC (Sat) by raven667 (subscriber, #5198) [Link] (1 responses)

> General data storage is one part of IT where you can not just reset things and forget about past bugs. That *is* stressful for anyone that cares.

Yes, but that stress doesn't have the *urgency* of the other examples, you can take months to figure out and discuss options before making a decision and changing the code, if things are getting heated you can afford the time to cool down and reflect before diving in again, that is a significant and substantial difference between examples such as surgeon or air traffic controller.

Kent's blog post

Posted Dec 1, 2024 17:11 UTC (Sun) by marcH (subscriber, #57642) [Link]

Yes, there's usually no urgency element to it, it's a different sort of stress. But "filesystems are stressful" is not an "odd claim".

How did it all start?

Posted Nov 23, 2024 10:36 UTC (Sat) by dottedmag (subscriber, #18590) [Link] (18 responses)

Was CoC involved because the (presumably) injured side raised an issue, or are they policing all interactions now, trying to quell all "unprofessional" infractions, even victimless?

How did it all start?

Posted Nov 23, 2024 15:32 UTC (Sat) by pbonzini (subscriber, #60935) [Link] (17 responses)

There is no such thing as a victimless interaction when one party asks the other to have their brain checked. Language can have a chilling effect to the discussion which makes the whole community a victim.

It is not a matter of being "professional", whatever that means, it's a matter of being civil and it's perfectly fine to take action when the boundaries of civility are exceeded; no matter how thick the skin of the other person.

How did it all start?

Posted Nov 23, 2024 16:58 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (15 responses)

If we've got a maintainer pushing for an approach that will cause CVEs and being dismissive of criticism, that's just not going to end well.

How did it all start?

Posted Nov 23, 2024 18:49 UTC (Sat) by pbonzini (subscriber, #60935) [Link]

That's not a justification. Period.

How did it all start?

Posted Nov 23, 2024 20:33 UTC (Sat) by tux3 (subscriber, #101245) [Link] (13 responses)

Kent, from reading here and lurking #bcache, I get the impression that you care about the underlying message, and not as much how it's being said. I think you would agree an insult wrapped in euphemisms and verbosity is still an insult, right?

But that other layer to the message is **very** important, to most people, and especially the other way around. Adding flame and personal attacks to something that could have just been a strong, cold disargeement over substance starts fights, when it doesn't discourage or scare people away.

You might have a point when you want to talk about deeper issues. But please also consider taking a bit more time to sleep on it, to let cooler words prevail. Even if all you care about is the code. It does end up making a difference.

How did it all start?

Posted Nov 23, 2024 20:43 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (12 responses)

I did quite a lot of patient explaining before things got to this point, yet we still had a mm maintainer adamantly pushing for an approach to GFP_NOFAIL that was a security risk; this after a long period of dealing with frustrating axe grinding.

So I'm not really sure what you really expect.

Curious how it's always the person who gets frustrated who needs to "work on himself". Are we here just to be nice to each other, or is shipping working code important too?

How did it all start?

Posted Nov 23, 2024 20:58 UTC (Sat) by tux3 (subscriber, #101245) [Link] (9 responses)

I'm sorry if that's coming across wrong, I don't mean to pin all the blame on any single person.
I do think you have a point about the axe grinding, and it might do some good if you can start that deeper discussion about being put in frustrating situations.

The unfair part is that it's much easier to call out a bright line violation than slow, continuous axe grinding.
There does have to be some point where there are bright line rules, still.

For what it's worth, I've been a happy bcachefs user for some time now, I've peeked a little bit inside. I think I might have sent a trivial patch at some point. So I can tell the code is excellent. And I certainly can understand if you're getting a little tired of unrequited advice... Happy to stop here, if I'm bothering you. Much sympathies.

How did it all start?

Posted Nov 23, 2024 21:01 UTC (Sat) by koverstreet (✭ supporter ✭, #4296) [Link] (8 responses)

thanks, that's all I've been trying to get across

How did it all start?

Posted Nov 24, 2024 0:00 UTC (Sun) by himi (subscriber, #340) [Link] (7 responses)

I feel like the core of the problem is that we have a pair of widely accepted ethical principles clashing here: the idea that it's unacceptable to be mean or nasty in interpersonal communications (particularly in professional or technical contexts), and the idea that it's unacceptable to knowingly make stupid or unreasonable arguments (particularly when you're in a position of authority that gives particular weight to those arguments).

Knowingly making stupid or unreasonable arguments is trolling. It can be a difficult thing to identify, particularly in the context of arcane technical debates and engineering discussions - making a convincing argument that someone you're dealing with is genuinely trolling can take a lot of work, and it's common for there to be disagreement about whether it *is* trolling or not, even after putting in that effort. But being difficult to prove conclusively doesn't stop it from being harassment - as much a form of harassment as intemperate language.

I don't know if Kent would necessarily agree with this characterisation, but my reading is that he feels he was being trolled, and by someone in a position with enough authority to nack his patches (adding significantly to the impact of the trolling). Regardless of whether his responses were within the bounds of acceptable behaviour, trolling should be treated the same as any other claim of harassment, and it seems like a large part of the problem here is that Kent feels that the trolling he's experienced is being downplayed or ignored, while his intemperate language is being emphasised and ultimately punished.

"Don't be nasty to people" is an easy thing to put in a code of conduct, and it's relatively easy to identify breaches and enforce consequences for those breaches.

"Don't troll" is also pretty easy to put in a code of conduct - in fact it's already one of the examples of unacceptable behaviour in the kernel code of conduct:

Examples of unacceptable behavior by participants include:
- The use of sexualized language or imagery and unwelcome sexual attention or advances
- Trolling, insulting/derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others’ private information, such as a physical or electronic address, without explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting

Whether reporting incidents to the Code of Conduct Committee would achieve anything I don't know.

How did it all start?

Posted Nov 24, 2024 0:15 UTC (Sun) by koverstreet (✭ supporter ✭, #4296) [Link] (6 responses)

It's not even trolling, people can be arguing in counterproductive ways without it necessarily being in bad faith. You don't want to punish people _more_.

And, I think asking the committee for intervention even more would just turn things into a circular firing squad. Why can't we ask them to just take a slightly more nuanced approach when they do intervene?

The answers I've heard were all along the lines of "But that's work and we don't want to be bothered!". Well, yes, that does take more work than just being an authoritarian who smacks people, but it's the more community minded thing to do.

How did it all start?

Posted Nov 24, 2024 0:50 UTC (Sun) by tytso (subscriber, #9993) [Link] (5 responses)

The nuance is personal attacks are not OK. Technical disagreements are fine, and are never an excuse for someone to carry out a personal attack.

I think the mm developers' points were actually valid, and I will note that other file systems don't have a problem, or at least not as much of a problem, as bcachefs seems to have with how the mm subsystems handles thing. I've certainly had my disagreements with Michal in the past, but *I* never felt the need to resort to personal attacks, and we eventually worked out a compromise (in fact, it was over __GFP_NOFAIL)

With respect to the mm/bcacefs thread, it certainly smeed to me that both sides seemed to be talking past each other, and I could see how both sides might feel that the other was ignoring their clearly, obviously, valid points. Sometimes the answer to that is to try to hash things out over a video conference; or perhaps in person at an event like LSF/MM. Or perhaps by trying your best to restate the other party's argument in as charitable way possible, and stating where you agree, and then trying to identify the precise place where you disagree. But **never** via a personal attack.

Is that sufficiently nuanced?

How did it all start?

Posted Nov 24, 2024 1:18 UTC (Sun) by koverstreet (✭ supporter ✭, #4296) [Link] (4 responses)

No, Michal was wrong, we're not going to be killing processes that use GFP_NOFAIL incorrectly. Or are you defending that?

And you've been following me around with comments that are distinctly personal and not technical every time testing comes up, so - let's please stick to what I asked last night.

How did it all start?

Posted Nov 25, 2024 4:55 UTC (Mon) by zorro (subscriber, #45643) [Link] (3 responses)

Why is killing processes that use that flag incorrectly such a big deal? Why not fix that code and move on?

How did it all start?

Posted Nov 25, 2024 5:49 UTC (Mon) by koverstreet (✭ supporter ✭, #4296) [Link] (2 responses)

Because it's not really "incorrect" if they have an error path, and a great many GFP_NOFAIL uses do have error paths.

They generally do an emergency shutdown of the filesystem, i.e. it's not an error that should ever happen in normal operation, but it won't take down the entire machine.

But if you kill the process, then you do take down the entire machine - because the only reason you use GFP_NOFAIL is if you're in a context where you can't unwind (without doing that emergency shutdown), and the reason you can't unwind is because you're holding locks.

So if you kill the process, those locks never get released, and the entire machine quickly grinds to a halt.

How did it all start?

Posted Nov 25, 2024 18:38 UTC (Mon) by mstsxfx (subscriber, #41804) [Link] (1 responses)

> Why is killing processes that use that flag incorrectly such a big deal? Why not fix that code and move on?

Let me just reference https://lore.kernel.org/all/ZtV6OwlFRu4ZEuSG@tiehlicka/T/#u which is trying to explain the reasoning.
GFP_NOFAIL is documented as never failing:

* %__GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
* cannot handle allocation failures. The allocation could block
* indefinitely but will never return with failure. Testing for
* failure is pointless.
* It _must_ be blockable and used together with __GFP_DIRECT_RECLAIM.
* It should _never_ be used in non-sleepable contexts.
* New users should be evaluated carefully (and the flag should be
* used only when there is no reasonable failure policy) but it is
* definitely preferable to use the flag rather than opencode endless
* loop around allocator.
* Allocating pages from the buddy with __GFP_NOFAIL and order > 1 is
* not supported. Please consider using kvmalloc() instead.

while some callsites might have a error path one would have to review all existing users and add error handling where it is missing.

So now there are not really many options how to deal with GFP_NOFAIL with the _current_ semantic.
1) drop __GFP_NOFAIL and get back to the situation that existing callers will revert to their endless loop around the allocator or simply rely on GFP_KERNEL effectively never fails semantic for non-costly (order < 4) allocations.
2) change the existing semantic and make it almost never fail - with a good definition of "almost"
3) keep the strict never fail semantic that is currently documented.

Each of the model has its cons and pros. 1 and 2 would require reviewing all existing users and that is not really trivial task. Option 3 needs to be really careful about unsupportable allocation requests (e.g. non sleeping requests would require essentially a busy loop around allocator without any upper bound, or higher order request could simply put the machine down through OOM killer storm because of the memory fragmentation).

The crux of the disagreement was in how to deal with impossible. Failing a (by definition) non-failing allocation can result in security issues (silent memory corruptions as this is not guaranteed to OOPS). Killing the allocation process might leave locks behind with similar effect but the fact could be easier to observe in the kernel log (in a sense a guaranteed OOPs). Or keep retrying indefinitely and make the system unusable.

What makes this decision making really hard is wide variety of allocator users. I am not really happy about the existing gfp flags semantics we have because it is really hard to understand unless you take the deep tour into the implementation. This results in misunderstanding and wrong use of the allocator. I have seen that countless times and I am definitely not blaming authors of the incorrect code. But seeing all this I am really concerned about adding more confusion like GFP_NOFAIL that can actually fail.

I still believe that this kind of discussion is not completely insane.

Michal Hocko

How did it all start?

Posted Nov 26, 2024 8:50 UTC (Tue) by mchehab (subscriber, #41156) [Link]

> So now there are not really many options how to deal with GFP_NOFAIL with the _current_ semantic.
> 1) drop __GFP_NOFAIL and get back to the situation that existing callers will revert to their endless loop around the allocator or simply rely on GFP_KERNEL effectively never fails semantic for non-costly (order < 4) allocations.
> 2) change the existing semantic and make it almost never fail - with a good definition of "almost"
> 3) keep the strict never fail semantic that is currently documented.

It sounds to me that maybe it is time to deprecate GFP_NOFAIL and replace it by other flag(s) that may have a better defined semantics.

How did it all start?

Posted Nov 24, 2024 6:34 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

> So I'm not really sure what you really expect.

A very simple guide: just don't insult people. That's it. Calling the _ideas_ "stupid" is borderline, but is acceptable in the LKML. Calling people "stupid" is not.

How did it all start?

Posted Nov 24, 2024 9:39 UTC (Sun) by NYKevin (subscriber, #129325) [Link]

> So I'm not really sure what you really expect.

If direct engagement is not working, the next step is to escalate. Not escalate as in "escalate the argument," but escalate as in "appeal to a higher forum." That might be Linus, the TAB, or somebody else, I don't pretend to be an expert on Linux politics and procedure. The point is, there has to be somebody who is The Decider when discussion on LKML proves fruitless. That person is probably different depending on the scope of the disagreement (e.g. if two developers disagree about something that only affects subsystem XYZ, then the XYZ maintainer is probably The Decider for that particular disagreement, but code that touches multiple subsystems might need a Decider who is higher up the chain).

I also want to stress that escalation is a process, not an event. You don't just shoot off an angry email to the next person up the chain as soon as somebody disagrees with you. If I were going to escalate a technical issue, in my professional life, I would start by writing up a short document explaining my view and their view, where we seem to agree, where we seem to differ, and what aspects we each seem to consider more important. I would then politely ask the other person to review that document and either agree with it or recommend changes, and I would (probably) allow them to freely edit the sections which represent their views. During this process, it is entirely possible that the disagreement would be resolved, and no actual escalation would need to take place. Since I work at a big tech company, my next step would be to show the document to my manager and ask them to figure out what to do with it, but I imagine the process in a community-run system like LKML will differ. Presumably, you circulate the document publicly for a while, then when it's in good shape, you show it to whoever The Decider happens to be and ask them to make a decision.

There are a few failure modes to watch out for:

* If the other person gives a non-answer, such as just stating that my document is "wrong" or a "waste of time" without further information (or if they ghost me altogether), I would note their objection (and/or non-participation) in the document, ask them if they want to elaborate on it, and then proceed with the escalation anyway.
* If the other person is The Decider and there is no higher authority... look, sooner or later, somebody has to lose one of these arguments. If you've already escalated all the way to the top and Linus/the TAB/whoever still thinks you're wrong, it's time to fold and either live with the decision or discontinue the affected work. If you feel really strongly, you could fork the kernel or carry an out-of-tree patch.
* If the other person resorts to insults or other aggressive behavior, don't respond in kind. Ignore them or report them, as you see fit, and continue to discuss the substantive issue without regard for the insults.

My big tech company has an HR department, so these failure modes are generally less of an issue for me (they don't do this "CoC" stuff in professional tech, they just tell you to cut it out or you're fired). They are included mostly for completeness.

How did it all start?

Posted Nov 24, 2024 10:49 UTC (Sun) by gmgod (guest, #143864) [Link]

I feel offendee by what you said.

Bound to happen

Posted Nov 23, 2024 11:33 UTC (Sat) by leandro (guest, #1460) [Link] (3 responses)

These were, and still are to a great measure, meritocratic communities. People aged, got some half-baked prudence and, in the absence of traditional ethical structures such as the Protestant ethos that made the substract of the cultures the initial developers came from, created these soviets and their constitutions which perpetuate the illness of corporate structures and inject it into our communities.

Our civilisation is dying, it would be delusional to think the free software culture would survive it, save a miracle. The next civilisation will hopefully learn from our foibles, but that would be a miracle too.

Bound to happen

Posted Nov 23, 2024 22:05 UTC (Sat) by dvdeug (guest, #10998) [Link]

Traditional ethical structures such as the Protestant ethos that says "But I tell you, if you are angry with a brother or sister, you will be judged. If you say bad things to a brother or sister, you will be judged by the council. And if you call someone a fool, you will be in danger of the fire of hell."? Haverings about civilization don't help discussion much.

If we want the best kernel at the end of the day, you've got to take care of your contributors, and get new contributors to replace the ones that inevitably die or quit. Just saying "meritocratic" doesn't deal with the problem that some very skilled people don't enjoy arguments and won't get involved in a project with a lot of them. It doesn't do anything with the problem that he who yells the loudest and longest is not necessarily the most correct.

Meritocracy is a scam

Posted Nov 24, 2024 3:23 UTC (Sun) by NYKevin (subscriber, #129325) [Link] (1 responses)

"Meritocracy" has never been a coherent, well-defined system. It's literally the equivalent of saying "we'll hire the best people," except in Latin. The only thing that changed is that we all started to figure that fact out.

Meritocracy is a scam

Posted Nov 25, 2024 16:29 UTC (Mon) by farnz (subscriber, #17727) [Link]

Note that early definitions of the term "meritocracy" were in a dystopian context, describing how you could pervert a system intended to appoint the best people to the jobs into one where you appoint the politically well-connected.

Non-sense

Posted Nov 23, 2024 15:01 UTC (Sat) by rbranco (subscriber, #129813) [Link] (21 responses)

Projects become victims of their own success when they can take maintainers for granted. We've seen it Python, Linux, etc. The *BSD crowd should take note and avoid this like the plague.

Refusing pull requests is nuts. What if the code contains a security fix? Paying a fine would be better. A win-win.

Enough with first world problems!

BSD

Posted Nov 23, 2024 15:13 UTC (Sat) by corbet (editor, #1) [Link] (5 responses)

The "*BSD crowd" is actually an interesting case in point. Have a look at the origin stories for some of those BSD variants and think about whether that's what you really want for Linux, or if it's maybe better for us to figure out how to get along with each other?

BSD

Posted Nov 23, 2024 15:18 UTC (Sat) by rbranco (subscriber, #129813) [Link] (4 responses)

The forks were exclusively about technical and not political issues. And they cross-pollinate each other to a extent.

BSD

Posted Nov 23, 2024 16:15 UTC (Sat) by raven667 (subscriber, #5198) [Link] (1 responses)

> The [BSD] forks were exclusively about technical and not political issues

That's an absurd distinction, the software projects aren't run by robots, they are run by human people and forks are created when people can't compromise and collaborate successfully on a shared vision. When people value different things that is a political disagreement not a technical one, miscatagorizing these things is going to lead to future misunderstanding of the fundamental nature of conflicts when collaboration breaks down.

BSD

Posted Nov 23, 2024 16:26 UTC (Sat) by rbranco (subscriber, #129813) [Link]

> That's an absurd distinction, the software projects aren't run by robots, they are run by human people and forks are created when people can't compromise and collaborate successfully on a shared vision. When people value different things that is a political disagreement not a technical one, miscatagorizing these things is going to lead to future misunderstanding of the fundamental nature of conflicts when collaboration breaks down.

When everything is political, nothing is. The core differences were 100% technical and the forks were a win-win for everyone... Linux has multiple distributions for mostly technical reasons, but now we seem to afford to fork over petty politics.

BSD

Posted Nov 23, 2024 17:01 UTC (Sat) by marcusb (guest, #16598) [Link] (1 responses)

> The forks were exclusively about technical and not political issues

At least in the case of OpenBSD forking from NetBSD, that doesn't jive with history as I remember it.

See http://mail-index.netbsd.org/netbsd-users/1994/12/23/0000.html and https://www.theos.com/deraadt/coremail.html.

BSD

Posted Nov 23, 2024 18:19 UTC (Sat) by malmedal (subscriber, #56172) [Link]

Yes, from conversations in USENIX and LISA in the 90s I gathered that splits From 386BSD to NetBSD and FreeBSD and why FreeBSD and NetBSD could not work together all were 100% personality based. Also some acrimony directed towards BSDI.

I think a major difference is that with Linux nobody questions that Linus Torvalds made the thing, while with the BSDs there were several people who felt that they were the true inheritor of the BSD tradition.

Non-sense

Posted Nov 23, 2024 15:32 UTC (Sat) by tuna (guest, #44480) [Link] (14 responses)

As far as I understand, Ken Overstreet can post patches on the mailing list as any other user. It is only his pull requests that will not get "pulled" by Linus T.

Non-sense

Posted Nov 23, 2024 16:47 UTC (Sat) by rbranco (subscriber, #129813) [Link] (13 responses)

I don't follow the mailing lists because I don't have time for drama and the sheer amount of content these days, but I think using another account is not allowed for "obvious" reasons. Posting as another user just doesn't make sense.

Is being pulled by Linus a privilege? That is also a very warped view.

Not being allowed to contribute for X amount of time is just crazy. But I guess this is the new normal these days and nothing surprises me anymore.

Non-sense

Posted Nov 23, 2024 21:30 UTC (Sat) by ballombe (subscriber, #9523) [Link] (9 responses)

Indeed, from a legalistic stand point this is ridiculous.

Linus himself is not under sanction, so the TAB has no basis to force him to do or not anything.
Linus could pull Kent tree and the TAB would not be able to do anything about it, or the TAB could rescind the sanction, and Linus could still refuse to pull Kent tree.

The TAB is just pretending to have authority it does not have.

Non-sense

Posted Nov 24, 2024 11:22 UTC (Sun) by Sesse (subscriber, #53779) [Link] (4 responses)

What makes you think the TAB has to _force_ Linus to do anything? My understanding is that he's simply agreed to abide by their decisions.

Non-sense

Posted Nov 24, 2024 14:13 UTC (Sun) by ballombe (subscriber, #9523) [Link] (3 responses)

This is exactly my point: the decision is Linus alone. The TAB is only playing around.

Non-sense

Posted Nov 24, 2024 19:16 UTC (Sun) by kleptog (subscriber, #1183) [Link] (2 responses)

That's a rather odd way of putting it. That's like saying a judge's ruling is irrelevant because the police could simply choose to ignore it.

That is very common in separation of powers situations. One party (the TAB) formulates a decision, and another party (Linus) is tasked with enforcing it. Linus could in theory refuse to implement, but that would trigger an immediate constitutional crisis. The question is if that's worth it.

Whichever way you look at it, the TAB is the one that renders the decision.

Similar to how in a constitutional monarchy the monarch has in theory the ability to ignore the will of parliament. In practice if they ever did that the resulting constitutional crisis would destroy one or both institutions.

Non-sense

Posted Nov 27, 2024 13:30 UTC (Wed) by Rudd-O (guest, #61155) [Link] (1 responses)

> That's like saying a judge's ruling is irrelevant because the police could simply choose to ignore it.

Yes, exactly. If the police refuse to abide by the judge's order, then the judge's order is irrelevant for all intents and purposes connected to reality.

"He has made his law. Let him enforce it."

Non-sense

Posted Nov 28, 2024 14:29 UTC (Thu) by foom (subscriber, #14868) [Link]

"If", sure.

But in practice judges' rulings DO get enforced, and so do the TAB rulings (at least this time).

Non-sense

Posted Nov 24, 2024 22:17 UTC (Sun) by ianmcc (subscriber, #88379) [Link] (3 responses)

The TAB is part of the Linux Foundation, answerable to The Linux Foundation Board of Directors. The Linux Foundation employs Linus. That gives them all the authority they need, unless Linus wants to die on this hill and quit the Linux Foundation. Why would he do that?

Non-sense

Posted Nov 25, 2024 0:15 UTC (Mon) by sashal (✭ supporter ✭, #81842) [Link] (2 responses)

The TAB has no "jurisdiction" over Linus. For that matter, the TAB doesn't really have any power or control over anything in the community. The TAB's "power" comes from the trust that the community has put in it by electing the current TAB members.

It would make sense that Linus will choose to respect the TAB's judgment and opinion around community issues such as this, right? This is why the TAB is around to begin with. There's no crazy LF board of directors thing playing out behind the scenes, this is just a group of folks (Linus included) doing what they thing is best for the future of the community.

No one wants to be dealing with this, this is not why folks signed up to do kernel work, and ideally this is the last time that there is a CoC related action in kernel-land.

Non-sense

Posted Nov 25, 2024 11:02 UTC (Mon) by ianmcc (subscriber, #88379) [Link] (1 responses)

Sure, there is not likely anything going on behind the scenes, simply because Linus complied with the ruling of the TAB.

But in a hypothetical, if Linus had refused that decision, or subsequently chooses to accept code from Kent, then it puts the TAB and the LF Board in a very difficult position. That would ultimately the Board's decision to resolve, and it could go in a variety of ways, eg, the Board might overrule the TAB or terminate the TAB completely, or they might part ways with Linus.

https://wiki.linuxfoundation.org/tab/start

Non-sense

Posted Nov 25, 2024 17:05 UTC (Mon) by sashal (✭ supporter ✭, #81842) [Link]

Hypothetically everything is possible :)

I wouldn't say that the relationship between the parties you've mentioned are as strenuous as you might think.

Again, this is an example of everyone working together to agree on a course of action and implementing it. No one was forced into anything. This is purely for the benefit of the community.

Non-sense

Posted Nov 24, 2024 14:54 UTC (Sun) by lkundrak (subscriber, #43452) [Link] (1 responses)

> Is being pulled by Linus a privilege?

I always thought it very much is? You generally are asking someone to do something for you which you don't have a right to demand.

You really are not entitled for me to pull your code into my tree, or for Linus to pull Kent's code.

Non-sense

Posted Nov 26, 2024 7:44 UTC (Tue) by ianmcc (subscriber, #88379) [Link]

Off topic, but I have to say, taken out of context,

> > Is being pulled by Linus a privilege?

Is absolutely hilarious.

Non-sense

Posted Nov 24, 2024 16:34 UTC (Sun) by tuna (guest, #44480) [Link]

As far as I understand Kent Overstreet can send in patches to the LKML like any other person using his own name etc and Linus T or any maintainer can choose to apply them to their tree which will later be pulled by Linux T for the next RC of Linux.

The only thing that has changed is that Linus T will not do a pull of Kent O's tree for this release cycle of Linux.

Form over substance ; weird asymmetry

Posted Nov 23, 2024 21:53 UTC (Sat) by sheepdestroyer (guest, #54968) [Link]

Being technically wrong and unproductive during a technical argument should not warrant being insulted.

But It would surely suffice to prevent a *demand* for a public apology ; or at least not before the insulted party having issued an apology for being wrong and uncooperative in the first place.

A blame only for not being nice/polite/SFW enough during a technical argument, regardless of who was technically right, is putting form over substance.

Forced apologies are harmful

Posted Nov 24, 2024 1:37 UTC (Sun) by cypherpunks2 (guest, #152408) [Link] (35 responses)

Reading through the thread again, I think undue weight has been put on the necessity of issuing a public apology. A forced apology functions as a form of public humiliation and, whether or not that is the intention, it has that effect. Psychological research has shown time and time again that forced apologies foster resentment and distrust, especially in an environment where one will be punished for not making an inauthentic apology.

Forced apologies are harmful

Posted Nov 24, 2024 12:46 UTC (Sun) by cytochrome (subscriber, #58718) [Link] (26 responses)

That seems to be a very abuser-centic perspective, Where is the consideration for the victim?

Forced apologies are harmful

Posted Nov 24, 2024 14:10 UTC (Sun) by cypherpunks2 (guest, #152408) [Link]

It's quite unnecessary to describe an escalation that results in an insult as involving an abuser and a victim. One can be in the wrong, can be combative, and can even be downright rude without being an abuser if there is not a pattern of behavior showing malice or wanton disregard for another's peace.

The harm caused by forced apologies is harm to all individuals involved, not just the person who will receive a bruised ego. An inauthentic apology is absolutely a disservice to the person on the receiving end. I was coming at this from the perspective of a recipient. Healing comes from mutual understanding, not from retribution against your interlocutor via public humiliation. It just so happens that a forced apology is an easy way to give the outward appearance of mutual understanding without actually achieving it, and to those not emotionally invested in the situation, the mien of understanding is just as good as the real thing.

Apologies should come from the heart, and to say so does not diminish the harm caused by callous behavior nor justify insulting others.

Forced apologies are harmful

Posted Nov 26, 2024 8:29 UTC (Tue) by ssmith32 (subscriber, #72404) [Link] (22 responses)

A forced apology doesn't help the person who was insulted, unless the victim is a (little bit of a) sadist who reaps psychological benefits from seeing another person coerced into doing something they do not want to do, and would be humiliated by.

However, it may help the person who did the insulting if they can be nudged to summon the willpower to put their pride aside in order for everyone to get back to insulting each other's specific ideas, instead of insulting the organ with which they create their ideas.

In short, this all seems to be taking place with actions and processes that assume everyone is operating at the emotional maturity level of a cloistered adolescent.

If a grown person can't stop from calling someone they work with an idiot in the middle of a debate about error handling, they lack the impulse control I would expect of an adult.

Conversely, if you're deeply hurt by someone you work with calling you an idiot, and consider yourself a "victim", I also would consider you emotionally immature. And you probably shouldn't take mass transit... or drive.. or ride a bike. All of which could result in people who are clueless calling you an idiot (e.g. I've been told to "get the f** off the road" and had cars swerve at me for being in a left turn lane on a bike... in areas where it would not only be illegal, but unsafe to pedestrians for me to be otherwise). So I ride on, leaving them to stew in traffic and ignorance.

If you sideline the person due to their poor impulse control and move on, that's fine. But you shouldn't be deeply hurt or regard yourself as a "victim".

I do wonder if all these people who talk how rough and tumble the world of kernel development is, and needs to be, has ever sat down with an MMA fighter, let alone boxed their friends. One of the calmest dudes I knew was an MMA instructor - who had no trouble apologizing and never called anyone an idiot in anger. But nor did he get particularly offended by insults. He certainly had no desire to fight someone over juvenile name-calling, whether physically or otherwise. Unless he was getting paid to ;)

Forced apologies are harmful

Posted Nov 26, 2024 9:10 UTC (Tue) by Wol (subscriber, #4433) [Link] (2 responses)

> A forced apology doesn't help the person who was insulted, unless the victim is a (little bit of a) sadist who reaps psychological benefits from seeing another person coerced into doing something they do not want to do, and would be humiliated by.

Apologies for mentioning it again, but there is also the little problem of cultural clash. I could never get involved in a - fact based - discussion of race on Groklaw, because what was acceptable English for me was unacceptable for GL's American, but what was acceptable to them was totally unacceptable for me. What do you do? An apology from me for offending them is meaningless, but they're not going to apologise for excluding me from the conversation.

Or I had a conversation with a co-worker (many moons ago). I got very stressed and agitated, and their manager complained to my manager about my "threatening behaviour". When I mentioned this to my co-worker she was shocked - she hadn't felt at all threatened (nor should she - it was obvious my stress-symptoms weren't aimed in her direction at all).

The only time an apology is going to work is when the apology-giver understands WHY an apology is necessary. I've learnt that - I use apologies to build trust. But if I'm going to "I'm sorry you're offended over nothing", I'm going to be extremely pissed off if I'm strong-armed into it. But if I realise the recipient is a shrinking violet - well it plays into my hands.

Cheers,
Wol

Forced apologies are harmful

Posted Dec 14, 2024 21:06 UTC (Sat) by jospoortvliet (guest, #33164) [Link] (1 responses)

Keep in mind that there are other people in the conversation - watching. I hope that, if you have kids, you try and model adult behavior - teaching them, for example, that you are able to reflect on your behavior, take the consequences it has on others into account and apolgise.

I think this is also a factor in the kernel community, as it is in any work space. People learn good behavior from each other - and if you misbehave, saying sorry in public shows others that 1. everybody makes mistakes and 2. just say sorry and we can all move on. That's a good, healthy way of working together. Conversely, being a stubborn d*** and getting away with it shows people that bullying gets you what you want. Don't be surprised if the result is a ton of bullying.

I'm not saying this is the only factor, just that it is one to consider. I do apologise sometimes for things I genuinely feel I didn't really do - to show a good example to others. What's the worst that can happen? I'm an adult and can (usually) handle a bruised ego; and being the bigger person never hurts.

Forced apologies are harmful

Posted Dec 14, 2024 22:48 UTC (Sat) by Wol (subscriber, #4433) [Link]

> I'm not saying this is the only factor, just that it is one to consider. I do apologise sometimes for things I genuinely feel I didn't really do - to show a good example to others. What's the worst that can happen? I'm an adult and can (usually) handle a bruised ego; and being the bigger person never hurts.

Exactly - and that builds trust!

Two further examples, and I very much get the impression some people in the kernel community might benefit from learning this ... bear in mind I'm borderline autistic.

And in both cases, the other party was a woman. Like many women, they naturally feel intimidated in the presence of a man - especially someone slightly "bull in a china shop" like me. But fortunately, they both felt able to tell me they felt I was talking down to them. I was shocked (I shouldn't have been). Cue a bit of trust building - apologise for making them feel this way (no blame! Whether I intended it or not, that was the consequences of my actions, I don't need to feel guilty, what's wrong with a *genuine* "I'm sorry I made you feel that way"?).

Then a little bit of psychology - if I treat you as an equal, I will feel you are treating me the same even if you aren't. If you feel inferior, you will feel that way even if I don't feel I'm treating you that way. But those two conversations were brilliant for me. And as far as I can tell for them too. They now know they can push back at me, and I'll respect them for it. They know my words are advice not orders. I've learnt to say my piece and *step* *back* (I think one of my - female of course - workers keeps doing daft things, but I advise and walk away, she needs to find out from OTHER people that I'm right :-) it'll harm ME if I push matters.

I still mess up, of course, but the result is a good working relationship. They know I'll say my piece - they know they can say their piece! (and they do) - but importantly they know that - at the end of the day - I *won't* interfere. They know I want them to get it right, but they know I want them to get it right THEIR way (and I know if they do it MY way, it's going to be a mess, anyway! :-)

Cheers,
Wol

Forced apologies are harmful

Posted Nov 27, 2024 0:52 UTC (Wed) by interalia (subscriber, #26615) [Link] (8 responses)

> If you sideline the person due to their poor impulse control and move on, that's fine. But you shouldn't be deeply hurt or regard yourself as a "victim".

I feel you're reading too much into the word "victim", I look at it as being casually factual. If someone steals from me, I'm a victim of theft; if someone punches me, I'm a victim of assault. This isn't some deep assertion of victimhood on my part, but something I think you'd agree is trivially true. If someone yells an insult at me then I'm a victim of verbal abuse, in the literal sense of the sentence "Person A verbally abused person B".

It's just factual that misconduct of the above kinds has a perpetrator and a victim (and sometimes bystanders/witnesses), and I was the victim. If the misconduct was a personal insult that doesn't mean I am saying I was deeply hurt or traumatised by it, but I'm also not going to deny it. Misconduct happens with widely varying severity and trauma but at low trauma levels there isn't suddenly a lack of a victim, and I don't feel people should decline to use the word.

Forced apologies are harmful

Posted Nov 27, 2024 13:17 UTC (Wed) by nye (subscriber, #51576) [Link] (7 responses)

> This isn't some deep assertion of victimhood on my part, but something I think you'd agree is trivially true.

I disagree with this really quite strongly.

Pushing somebody to breaking point so you can get the satisfaction of watching them lose their cool does not in any sensible definition make you a victim. I don't think *this* instance was actually deliberate trolling, but it *very frequently* is.

Forced apologies are harmful

Posted Nov 27, 2024 14:03 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

> I disagree with this really quite strongly.

PLEASE don't confuse victim with perpetrator. READ WHAT THE GP WROTE.

Cheers,
Wol

Forced apologies are harmful

Posted Nov 28, 2024 10:11 UTC (Thu) by nye (subscriber, #51576) [Link] (1 responses)

Wol, you are an enraging troll whose value to LWN is consistently profoundly negative. Exactly the kind of person I'm talking about, in fact.

The only value you provide at all is that people can now pay to mute you - maybe this has caused an increase in subscriber counts? Please consider this the next time you try yelling at somebody because you don't understand them and are too narcissistic to consider that perhaps the misunderstanding is yours.

(I rewrote this post four times and this is the most polite I could manage. You don't want to know what was in the first draft.)

not so shouty as he seems

Posted Nov 28, 2024 15:22 UTC (Thu) by sdalley (subscriber, #18550) [Link]

Well, knowing from some of Wol's other posts over the years that he's a fundamentally decent guy, I often read him with interest and just interpret his shouty-looking capitals as, in his case, a slightly bold alternative font.

Forced apologies are harmful

Posted Nov 27, 2024 16:42 UTC (Wed) by interalia (subscriber, #26615) [Link] (3 responses)

> Pushing somebody to breaking point so you can get the satisfaction of watching them lose their cool does not in any sensible definition make you a victim.

None of that was said in my comment or implied, as far as I can tell.

But consider that saying "victim of assault" does not have to imply "was morally right", and that victims and perpetrators are not an exclusive-or situation... nothing stops people from being victims sometimes but perpetrators at other times or even at the same time e.g. two people in a fistfight might well be both the victim of assault and also a perpetrator of it.

Forced apologies are harmful

Posted Nov 28, 2024 10:23 UTC (Thu) by nye (subscriber, #51576) [Link] (2 responses)

> None of that was said in my comment or implied, as far as I can tell

That's what I took from "If someone yells an insult at me then I'm a victim of verbal abuse" as an absolute statement.

I don't agree that yelling an insult is necessarily abuse, and I don't agree that the word "victim" can be used to describe this without stretching it so far that it has no meaning any more.

> saying "victim of assault" does not have to imply "was morally right"

Sure, but it does strongly and unavoidably imply that the person doing it was morally wrong.

It sounds like you mean the word to be essentially equivalent to "recipient", but if that's really what you want it to mean, would you say that I'm a victim of my Amazon delivery? I think most people would not.

(This isn't just a semantic aside BTW, but something that I think is on topic - I feel like I'm perhaps the only person in this thread who sees Kent's mail as the rash but understandable impulse of somebody pushed past breaking point, even if that wasn't the intent.)

Forced apologies are harmful

Posted Nov 28, 2024 15:06 UTC (Thu) by interalia (subscriber, #26615) [Link]

> That's what I took from "If someone yells an insult at me then I'm a victim of verbal abuse" as an absolute statement.

In my original comment I was using myself in an abstract hypothetical where someone yelled an insult at me, so to me it seemed rather odd that suddenly the reason for the insult in my hypothetical was because I had pushed someone else to breaking point so I could get them to lose their cool.

I wrote some text replying to the rest of it but deleted it, I get some of your point but don't care enough to hash that out further so will leave it there. Thanks for the discussion.

Forced apologies are harmful

Posted Dec 3, 2024 5:18 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

> Sure, but it does strongly and unavoidably imply that the person doing it was morally wrong.

So now we're going to run LKML on the theory of "two wrongs make a right?"

You cannot abuse people, even if they purportedly "deserve" it. If someone is misbehaving, then you either ignore it or escalate it to the CoC people (which may entail drafting a policy to actually prohibit the conduct in the first place, but that's a detail). You are not entitled to simply assume that someone is engaging in bad faith and assign a punishment of verbal abuse on the spot.

Forced apologies are harmful

Posted Nov 27, 2024 1:40 UTC (Wed) by pizza (subscriber, #46) [Link] (9 responses)

> In short, this all seems to be taking place with actions and processes that assume everyone is operating at the emotional maturity level of a cloistered adolescent.

The point of forcing apologies is an overt demonstration of *power*, nothing more.

Forced apologies are harmful

Posted Nov 27, 2024 2:17 UTC (Wed) by dskoll (subscriber, #1630) [Link] (8 responses)

But unless a committee charged with enforcing a code of conduct can demonstrate power, it's completely useless.

Forced apologies are harmful

Posted Nov 27, 2024 4:35 UTC (Wed) by pizza (subscriber, #46) [Link] (7 responses)

> But unless a committee charged with enforcing a code of conduct can demonstrate power, it's completely useless.

Yet counter-intuitively, every time they exercise said power they actually weaken it.

...The only power they have is to exclude folks from "their" sandbox, but the harsh reality is that the overwhelming majority of the folks working on Linux are doing so in non-mainline vendor-controlled sandboxes.

Forced apologies are harmful

Posted Nov 27, 2024 17:54 UTC (Wed) by dskoll (subscriber, #1630) [Link] (6 responses)

> Yet counter-intuitively, every time they exercise said power they actually weaken it.

How so? I'm not following. Given what happened, do you think the action of the committee will make it more likely, less likely or about the same that someone else will make a public personal attack against someone? I think it will make it less likely, in which case the action had its intended effect.

Forced apologies are harmful

Posted Nov 27, 2024 19:55 UTC (Wed) by pizza (subscriber, #46) [Link] (5 responses)

> How so? I'm not following.

The only power they have is that of exclusion.. but it's purely performative act by a paper tiger. They can't actually "fire" anyone; they can't prevent anyone from working on Linux, or anyone else from using said work or otherwise collaborating with them.

After all, these excluded folks have countless other places to accomplish their (still continuing) work, and downstream distributors of Linux overwhelmingly routinely apply non-mainline patchsets.

Another question with complicated ramifications -- what's going to happen when there's high priority CVE but the maintainer behind the subsystem/etc in question is still in timeout? Does someone else have to submit the work of the excluded person for a fix to land?

If you're going to suspend or otherwise kick people out of your sandbox, you'd better have a plan to replace them. After all, ripping out entire drivers/subsystems will be a blatant, fragrant "breaking userspace". Is that really a "better for the community" outcome?

> Do you think the action of the committee will make it more likely, less likely or about the same that someone else will make a public personal attack against someone

I think all of this is just performative BS. When nearly everyone out there is already running a non-mainline fork, all this will demonstrate is how little power you actually have.

Does the COC decision work? (was Forced apologies are harmful)

Posted Nov 27, 2024 22:31 UTC (Wed) by dskoll (subscriber, #1630) [Link] (4 responses)

The only power they have is that of exclusion.. but it's purely performative act by a paper tiger

Actually, no. People hate being excluded. It is actually a serious punishment; this is hard-coded into our DNA from our days of surviving in small groups in a hostile prehistoric environment. It's why banishment has been a serious punishment for thousands of years across many different cultures.

I think all of this is just performative BS. When nearly everyone out there is already running a non-mainline fork, all this will demonstrate is how little power you actually have.

So you didn't answer my question, but I'm assuming you think it won't make any difference. I disagree. There's a lot more cachet to having your code in the mainline kernel rather than a fork, and I think the action will make people think twice before embarking on public personal attacks.

Does the COC decision work? (was Forced apologies are harmful)

Posted Nov 28, 2024 1:06 UTC (Thu) by pizza (subscriber, #46) [Link] (2 responses)

> Actually, no. People hate being excluded. It is actually a serious punishment; this is hard-coded into our DNA from our days of surviving in small groups in a hostile prehistoric environment. It's why banishment has been a serious punishment for thousands of years across many different cultures.

Are you seriously saying that being "Excluded from working on the mainline by the linux foundation" matters one iota when (1) your salary isn't being paid by said linux foundation, and (2) your actual employer is overwhelmingly likely to be shipping their own non-mainline fork?

Seriously.

Does the COC decision work? (was Forced apologies are harmful)

Posted Nov 28, 2024 1:58 UTC (Thu) by mjg59 (subscriber, #23239) [Link]

Yes? Once you're above a trivial level of divergence maintaining a fork becomes increasingly difficult, and vastly reduces the enthusiasm of external contributors. Getting something into mainline eases maintenance burden and increases the probability that other people will help make your solution better.

Does the COC decision work? (was Forced apologies are harmful)

Posted Nov 28, 2024 14:09 UTC (Thu) by dskoll (subscriber, #1630) [Link]

Are you seriously saying that being "Excluded from working on the mainline by the linux foundation" matters one iota when (1) your salary isn't being paid by said linux foundation, and (2) your actual employer is overwhelmingly likely to be shipping their own non-mainline fork?

Yes, that's exactly what I'm saying. It might not matter financially in the short-term, but it does impact your career (public meltdowns are never good when you are job-seeking) and it matters personally. Additionally, any employer shipping its own kernel would be much happier if the employee was not excluded from working on the mainline; this greatly simplifies maintenance of any fork.

If it really "didn't matter one iota", then this thread would have two comments on it instead of dozens/hundreds.

Does the COC decision work? (was Forced apologies are harmful)

Posted Nov 28, 2024 3:06 UTC (Thu) by intelfx (subscriber, #130118) [Link]

> I think the action will make people think twice before embarking on public personal attacks.

I wish it would. But that’s not going to happen as long as these actions are only carried out against the boogeymen du jour, while Real Kernel Maintainers™ (starting with Linus) continue to insult their peers (see the sister thread about comm[]) at every opportunity and do so with absolute impunity. Rules for thee and not for me.

Forced apologies are harmful

Posted Nov 27, 2024 13:33 UTC (Wed) by Rudd-O (guest, #61155) [Link] (1 responses)

> That seems to be a very abuser-centic perspective, Where is the consideration for the victim?

The victim can decide whether a public apology is needed. It's categorically not the purview of the committee to decide for the victim. If the victim has not requested such a thing, then it's transparently evident that the committee only demands that for humiliation / flaunting power purposes.

Forced apologies are harmful

Posted Dec 20, 2024 18:34 UTC (Fri) by tbird20d (subscriber, #1901) [Link]

>The victim can decide whether a public apology is needed. It's categorically not the purview of the committee to decide for the victim.

I strenuously disagree. Victims are often in the least favorable position to request an apology. They may not even recognize the value
to the community that a public apology would provide.

> If the victim has not requested such a thing, then it's transparently evident that the committee only demands that for humiliation / flaunting power purposes.
It's not transparently evident to me. All CoC deliberations I've been privy to have been about communicating to the rest of the
community the boundaries of acceptable behavior, and getting those involved back to productive activity as smoothly and quickly as possible.

Forced apologies are harmful

Posted Nov 24, 2024 15:26 UTC (Sun) by mattdm (subscriber, #18) [Link] (7 responses)

I agree with this. Resentment, distrust, and insincerity are bad enough, but pragmatically the effect is often to center the reported party's focus around their objection to being forced to apologize, rather than the actual behavior and its impact.

The goals should be: ongoing community health, restitution and repair for the victim if necessary and possible, and a sincere change in future behavior. A genuine apology is a good step towards all of those, but it needs to come from self-reflection, recognition of common community purpose, understanding that apology alone isn't sufficient — and acceptance that this goes two ways: even a total, heartfelt apology doesn't get forced forgiveness.

Otherwise, you just have playground situation where parents make a bully stand and mumble "sorry" and the victim say "I forgive you" — and the same thing the next day, and the next.

Forced apologies are harmful

Posted Nov 25, 2024 0:19 UTC (Mon) by sashal (✭ supporter ✭, #81842) [Link] (6 responses)

In my mind, asking for a public apology isn't done to "humiliate" a party, but rather it's a checkpoint that the party at fault understood what went wrong and makes an attempt at amending things and makes a promise to not repeat the behavior.

I don't think that the offended party necessarily has to accept the apology, and I can see cases where the offended party may choose not to interact with an individual anymore.

The apology is mostly for the benefit of the community.

Forced apologies are harmful

Posted Nov 25, 2024 7:04 UTC (Mon) by tytso (subscriber, #9993) [Link] (5 responses)

I agree that it's mostly for the benefit of the community, but I'd go farther than that. If someone writes a proper, well-written apology (see the resources at [1] and [2]), it can actually be beneficial for the person writing the apology, even if it wasn't written with completion sincerity.

[1] https://www.health.harvard.edu/blog/the-art-of-a-heartfel...
[2] https://www.npr.org/2023/01/25/1150972343/how-to-say-sorr...

If one follows the four key steps of a real apology as described in [1]

* Acknowledge the offense
* Explain what happened (WITHOUT excusing the behavior)
*. Express remorse
* Offer to make amends

... it can't help but affect the writers' thinking. It's a lot more than just saying, "Sorry! I'm forced to say it. Satisfied?"

Also of note from [2]:

McCarthy adds that a bad apology can even make things worse.

"It's akin to the cover-up being worse than a crime, if you make an
apology that says, you know, 'You shouldn't even have a white sofa,'
or, 'You shouldn't have been standing there,'" she said.

So an apology which is buried in a long message explaining how the victim was wrong and unreasonable in many different ways, is not really a real apology, or is at best a really a bad apology.

Forced apologies are harmful

Posted Nov 25, 2024 8:51 UTC (Mon) by cypherpunks2 (guest, #152408) [Link] (4 responses)

This is absolutely correct if the apology is genuine, but a forced apology is not a heartfelt apology. You cannot force someone to apologize from the heart, even if people are very good at faking it under pressure.

Forced apologies are harmful

Posted Nov 25, 2024 13:57 UTC (Mon) by Wol (subscriber, #4433) [Link] (1 responses)

I think I've surprised my boss on a few occasions (and it pays off, he's almost always on my side if I do get involved in something I shouldn't).

Step up, take a measured look at what went wrong (and DON'T assume you're in the right - you could both be, or you could both be in the wrong ...).

Then ask yourself "how do I offer an olive branch?". Looking for an olive branch benefits YOU, and if the other guy won't accept it, more fool them. Fault is rarely one-sided, it's easy to find something to proffer, and when it's not expected it places you firmly on the moral high ground IN OTHER PEOPLES' EYES.

Cheers,
Wol

Forced apologies are harmful

Posted Nov 25, 2024 14:10 UTC (Mon) by pizza (subscriber, #46) [Link]

> Then ask yourself "how do I offer an olive branch?". Looking for an olive branch benefits YOU, and if the other guy won't accept it, more fool them.

More fool them, but you're still the one that gets punished.

Forced apologies are harmful

Posted Nov 25, 2024 17:11 UTC (Mon) by sashal (✭ supporter ✭, #81842) [Link] (1 responses)

This is just my own personal opinion: The community isn't necessarily interested in a "heartfelt" apology.

The point in asking for an apology is really an attempt at clarifying that a certain behavior in unacceptable in our community, and the apology part is acceptance and understanding of these terms.

Is it even better if the apology is real and true? Sure!

Forced apologies are harmful

Posted Nov 25, 2024 18:16 UTC (Mon) by marcH (subscriber, #57642) [Link]

The more important word in "public apology" is not "apology".

A very difficult topic

Posted Nov 25, 2024 8:55 UTC (Mon) by jd (guest, #26381) [Link]

On the one hand, doing nothing damages Linux and the community. On the other, whatever one did, it would not be possible to avoid damaging Linux and the community.

I don't see that the code of conduct committee had any good options and I suspect that they did what they believed would do the most good with least harm - but it's never possible to be sure if that's how it will play out.

BTRFS and Facebook

Posted Nov 25, 2024 12:44 UTC (Mon) by snajpa (subscriber, #73467) [Link] (13 responses)

There's a reason why we're running OpenZFS and not BTRFS. I'm not on LKML neither am I planning to, when I see this community's approach to filesystems and containers. Fail and fail and nope I have better things to do in life than arguing with heavily opinionated but outside-of-linux-never-tried-anything devs. Easy for them to view Btrfs as sufficient.

And no, that Facebook uses it isn't an argument. With their army of developers, they could be using whatever, straight from git master, no need to care about anything because they'll just fix the problems they encounter. What they don't encounter they obviously don't fix but hey Facebook uses it, it's a good filesystem!

Whatever. Community built on allowed toxicity is dispelling a rather newcomer for trying to use that toxicity to improve the community. Best of luck.

BTRFS and Facebook

Posted Nov 25, 2024 12:52 UTC (Mon) by pizza (subscriber, #46) [Link]

> Whatever. Community built on allowed toxicity is dispelling a rather newcomer for trying to use that toxicity to improve the community. Best of luck.

"improvements" in the eyes of the beholder, that is.

'how can anyone govern a country with 246 varieties of cheese? ' -- Charles de Gaulle

BTRFS and Facebook

Posted Nov 25, 2024 14:53 UTC (Mon) by jd (guest, #26381) [Link]

I am sympathetic to your argument. Facebook is not maintaining btrfs for the betterment of the community as a whole but rather for themselves.

Those old enough will remember that this is precisely why PGCC failed utterly against EGCS and why the original branch of GCC failed against both of them.

In the end, you design for the community at large or into a brick wall.

I'm not going to get into the whole good/bad thing, but it seems obvious that the corps have lost sight of why contributing to Linux was useful.

BTRFS and Facebook

Posted Nov 26, 2024 15:06 UTC (Tue) by paulj (subscriber, #341) [Link]

It's worth bearing in mind that all (to a degree of approximation with near negligible error) the BTRFS storage at Facebook is not holding data that matters. Essentially all BTRFS storage at Facebook is disposable - if there are errors, automated systems will just pull the machine "offline" (i.e., not part of any cluster anymore), automated systems will apply 1 or more remediating actions - up to and including re-imaging. Nearly all BTRFS storage at Facebook is not really put under that much stress, and Facebook's systems are designed to be tolerant to regular failures (cause at Facebook's scale, rare errors happen regularly). Further, where BTRFS is holding data that matters, still no single BTRFS is significant. Etc.

BTRFS and Facebook

Posted Nov 27, 2024 13:34 UTC (Wed) by Rudd-O (guest, #61155) [Link] (1 responses)

ZFS for the win.

We CAN have nice things.

BTRFS and Facebook

Posted Nov 28, 2024 3:53 UTC (Thu) by cypherpunks2 (guest, #152408) [Link]

ZFS on Linux is great, but it has three serious issues (two of which are Linux-specific):
  • The ARC cache has all its memory marked as non-reclaimable (this is where the myth that ZFS is bloated comes from) and it is up to the ZFS driver to notice when memory pressure is growing and shrink the cache accordingly, otherwise the system can OOM before the ARC shrinks. The patents on ARC have expired so in theory Linux can replace its native LRU cache with an ARC-based one, but I don't see that happening any time soon.
  • Also due to being poorly integrated with the mm subsystem, data that is read from mmap'd files is duplicated in the LRU-based page cache as well as the ARC cache which reduces reliability (especially due to in-memory checksumming) and increases memory overhead.
  • Being a CoW filesystem, ZFS is particularly susceptible to fragmentation. ZFS's problem is that it has no ability to defragment, not even offline. The only way to simulate defragmenting is by doing a full snapshot send/receive.
There are other, smaller issues such as unlink() operations being particularly slow and the CDDL license pushing all development out-of-tree, but those are not too problematic and are pretty much unavoidable anyway. The other issues are more severe. That doesn't mean that ZFS is a bad filesystem, to the contrary I think it is the best filesystem for Linux for reliability as its reliability features go far beyond merely checksumming all data and metadata. But we should all keep in mind that there is room for improvement, especially on Linux.

BTRFS and Facebook

Posted Nov 27, 2024 20:26 UTC (Wed) by tuna (guest, #44480) [Link] (7 responses)

If you don't like Linux and how it is developed it is totally reasonable to run something else. Maybe some BSD system works better for you?

BTRFS and Facebook

Posted Nov 27, 2024 22:19 UTC (Wed) by snajpa (subscriber, #73467) [Link] (6 responses)

thank you for the suggestion, but nope, I'm good with our 44 patches on top of Linux :)

some of them do stuff like consider CDDL compatible with GPL (I mean, sue me if that's how you wish to be remembered)... some of them replace the need for ugly lxcfs... some of them make containers feel more like what containers should be

I'm pretty happy with it :) no arguments, no bikeshedding needed to get to a point where I'm pretty happy with our deployment... as for your attempt to demonstrate why I won't engage, if you enjoy such fallacies as when I have stuff to criticize, your first impulse is to tell me to go away... well, thank you for that demonstration

BTRFS and Facebook

Posted Nov 28, 2024 10:40 UTC (Thu) by Wol (subscriber, #4433) [Link] (5 responses)

> some of them do stuff like consider CDDL compatible with GPL (I mean, sue me if that's how you wish to be remembered)

So long as you're the end user, who's going to sue? Who will have standing? Who will even have a case!

If you're the end user, the GPL is irrelevant to you (unless you were involved in illegalities whilst acquiring the program). I guess the same is true of CDDL.

So yep. Some random Joe could sue you. They could sue you for anything. Doesn't mean they have a case.

Cheers,
Wol

BTRFS and Facebook

Posted Nov 28, 2024 11:08 UTC (Thu) by snajpa (subscriber, #73467) [Link] (4 responses)

> So long as you're the end user, who's going to sue? Who will have standing? Who will even have a case!

We do redistribute tho, you can download the iso and give it a try - https://vpsadminos.org - and it's all on GitHub.

Most of the arguments that still make the circles whenever ZFS pops up are just... FUD. It all starts with the rumor that supposedly engineers at Sun wanted a licence that is incompatible with GPL, well... people just took that as the base truth and it's done forever, when in fact... a source of that claim is a person who was supposed to be close to the process, but the guy who was her boss at the time says the exact opposite, the engineers wanted something BSD like and so the CDDL is a result of a corporate process, while the supposed GPL incompatibility was never a goal.

It was never tried in court, same as the EXPORT_SYMBOL_GPL and this "it combines in memory so it is a derivative work" stuff. My impulse whenever this comes up would be to fork Linux, publish it under a different name, with all this nerds-be-playing-lawyers-or-otherwise-gatekeep-because-NIH stuff gone. When I see how they treated Kent, that impulse is higher than ever. And AFAIK I would still be within my rights to do that, without creating any limitations on anyone's other's rights.

This nerd says that in OpenZFS case, CDDL is effectively practically compatible with GPL, because in spirit they truly are and that's the only thing that matters in the end, NIH or not.

BTRFS and Facebook

Posted Nov 28, 2024 11:23 UTC (Thu) by snajpa (subscriber, #73467) [Link]

it was already pretty offtopic when I first commented, so if anyone would like to argue with me about it further, I'm reachable via mail - snajpa at snajpa.net ;)

BTRFS and Facebook

Posted Nov 29, 2024 14:01 UTC (Fri) by paulj (subscriber, #341) [Link] (1 responses)

> It all starts with the rumour that supposedly engineers at Sun wanted a licence that is incompatible with GPL

Former Sun engineer here. I was not senior enough to be one of the engineers in the licence discussions for what became OpenSolaris, but the senior engineer in my group was. And he who told me that Sun management had wanted the GPL, but a number of very prominent Solaris engineers said "hell no". A similar account has been related by, IIRC, one or both of Simon Phipps and Danese Cooper (I forgot which). That account further describes that a group of said prominent senior engineers had threatened to quit Sun if management proceeded with the GPL.

So, this is something that has been described by multiple different people here on LWN - at least 1 of whom was in those meetings.

The one bit that is misdescribed there is that Sun wanted a licence "incompatible with the GPL". That part is not - TTBOMK - correct. At least, that was not a deliberate initial goal TTBOMK. For Sun management wanted the GPL itself, AIUI. However, various senior Sun engineers had a strong dislike of the GPL - most of them came from the BSD world. They did not want the GPL at all. They actually wanted Solaris to be placed under the BSD licence! (Is how my senior colleague told it to me anyway). A permissive licence was never going to fly for management.

So the constraints were (here I mix in /my summarisements/ with the facts I know of above):

- Management wanted to go open-source, as did engineering.
- Management were not prepared to totally give away Solaris (i.e. no BSD licence), they wanted the strongest possible licence, i.e. copyleft
-- Schwarz had written favourably about the GPL
-- I believe management's initial idea was to use the GPL
- Engineering hated the GPL and absolutely refused to go with that (I don't think there's a rational reason for this, just cultural factors from BSD leanings)
- So... management decided to come up with some new licence
-- Possibly legal also liked the idea of being able to write a new licence, and weren't fans of the GPL
--- Though, some in Sun legal did help with input to the GPLv3 drafting process
- We got the CDDL

I don't know if anyone spotted the compatibility issues. There was extensive consultation about the CDDL within Sun prior to the decision. We were all - within the Solaris engineering group - given the opportunity to review it and give comments internally. I remember looking at it, I don't remember thinking about compatibility or seeing the issue, unfortunately. Surely someone had thought about it, one way or another though.

I think - with hindsight, applying my own speculative analysis - the decision to go with the CDDL, and in particular to *not* use the CDDL, is all about senior Sun engineers wanting to rescue their baby, Solaris. They could see the Sun setting, and that it could well go down. They wanted to extract their beloved software, Solaris, which they had spent years and years on, from the setting Sun before it was too late. To be able to take Solaris and keep working on it, and start new businesses around it, after Sun was gone. And I think a key thing for them to achieve that rescue was they either wanted unfettered access to the code (i.e. BSD) OR failing that they wanted to ensure it would be distinct from Linux.

That's just my subjective analysis. Cause it's the only thing that can make some sense of what happened. Going with a GPL-incompatible licence made no sense for Sun Microsystems, Inc. as a business. All it achieved was bad press and bad blood - despite Sun management being favourable towards the GPL. Sun released nearly all of Java under the GPL in '07. So clearly there wasn't an issue with upper-level management with the GPL. Yet Solaris went with the CDDL. It made no sense at all, except in the context that a bunch of high-profile Sun engineers just hated the GPL.

BTRFS and Facebook

Posted Nov 29, 2024 14:12 UTC (Fri) by paulj (subscriber, #341) [Link]

Oh, I should revise something. Initially I wanted to dispute this:

> Sun wanted a licence that is incompatible with GPL

At least, if you read "Sun management" for Sun. That said, based on what I wrote you /could/ argue there /may/ have been some subset of Sun (likely in Solaris engineering) who wanted that. However, I have _no_ direct evidence for that - I have not read any account of this. All I know is that a group of leading Solaris engineers vehemently opposed GPLing Solaris - I have read that account from either Phipps or Cooped (an account reported on on LWN) and I heard the exact same thing well before they had put that online straight from the mouth of someone who was there.

BTRFS and Facebook

Posted Dec 3, 2024 5:39 UTC (Tue) by NYKevin (subscriber, #129325) [Link]

> And AFAIK I would still be within my rights to do that, without creating any limitations on anyone's other's rights.

At least under US law, the only plausible reason for this *not* being allowed would be one or both of 17 USC 1201 ("Circumvention of copyright protection systems") and 1202 ("Integrity of copyright management information"). GPL version 3 expressly waives 1201 (and equivalents in other countries), but version 2 predates the whole concept by several years, so it does not have such a waiver. You can probably still argue that GPL version 2 either implicitly waives that law, or at least that it gives rise to equitable defenses against a supposed violation.

As for 1202, it is probably waived by the modification rights given in both versions of the GPL, but I'm not sure how specific that waiver needs to be in order to be effective. If doing this professionally, it might be wise to consult a lawyer.

GFP_NOFAIL and error handling

Posted Dec 1, 2024 18:26 UTC (Sun) by marcH (subscriber, #57642) [Link] (1 responses)

Apologies for posting an "off-topic" technical comment in a CoC discussion (could not resist).

I have a very shallow understanding of the technical debate but I found it very interesting that it is about... error paths: always the least designed area with the worst implementations and least test coverage. I only skimmed the discussion yet I was impressed by the numbers of "what ifs". The best antidote to "what ifs" is test code: either the test passes or not; no hypothetical. Sometimes the test is wrong? Fine: fix it first and then resubmit your change. But thoroughly testing error paths tends to be very hard hence seldom done. Add a bit of concurrency on top and testing becomes even harder.

Obviously, this is all happening in C: one of those ancient languages that gives the developer absolutely zero tool or any sort of help to handle errors.

From https://www.patreon.com/posts/116412665
> Besides dramatic accusations of breakage - theoretical in this case, from a developer with a history of introducing silent data corruption bugs into code I've written...

Again, these "theoretical breakages" and "silent data corruptions" become much less drama when reproducing them is as easy as pushing some test buttons. If the test is correct and fails then the change gets immediately reverted with a warning not to forget to run the whole test suite next time. Test suites don't have difficult personalities or bruised egos.

Unfortunately, you too rarely become a rock star or get a promotion by writing difficult test code. Not even by writing production code easier to test for that matter.

GFP_NOFAIL and error handling

Posted Dec 1, 2024 18:35 UTC (Sun) by marcH (subscriber, #57642) [Link]

> Unfortunately, you too rarely become a rock star or get a promotion by writing difficult test code. Not even by writing production code easier to test for that matter.

Forgot to say: at least now you may be able to make some money by finding vulnerabilities in untested error paths. That's some form of progress I guess.


Copyright © 2024, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds