Reducing kernel-maintainer burnout
Ted Ts'o started off the session by saying that kernel maintainers end up having to do all of the tasks that nobody else working on a given subsystem wants to take on. These can include patch review, release engineering, testing, and responding to security reports. The expectations placed on maintainers have gone up over time, and kernel maintainers are feeling the pressure as a result.
![[Ted Ts'o]](https://static.lwn.net/images/conf/2023/lpc/TedTso-sm.png "Ted Ts'o")
Darrick Wong, Ts'o continued, broke
down the aspects of the maintainer job nicely when he stepped
down as the XFS maintainer. Ts'o is uncertain, though, about how well
that has worked to get others to step up and tackle some of those jobs.
Martin Petersen asserted that the real problem is that people are sending too many patches, but Dave Airlie strongly disagreed. As the DRM subsystem maintainer, he is "processing more changes than anybody" without having to touch a single patch. The way to handle this problem is to build up a structure with people who are able to take on the various tasks needed. The filesystem layer, he said, is more important than graphics; why doesn't it have more people than the DRM layer does?
Josef Bacik answered that building up that structure is hard; he has been trying with various people for the last three years. One developer simply couldn't do the work, another was unable to bring things to a conclusion. The filesystem problem space is complicated, he said, and finding people who can work in that area is hard.
Steve Rostedt said that part of the problem may be documentation; when he runs into bugs, he can't find documents describing how things work. Christoph Hellwig suggested writing down problems as they are encountered. Christian Brauner said that there is extensive documentation about the filesystem layers, but it tends to be hard to understand.
Airlie said that the trick is for maintainers to leave voids for others to fill. Thomas Gleixner, though, brought up the example of the generic interrupt subsystem. There is currently one person maintaining it, even though "if it breaks, the whole world breaks". There are a lot of people sending patches, but nobody showing any desire to maintain it. Airlie said that, if there are 100 people sending patches, there may be five who can be convinced to help maintain the subsystem; Gleixner answered that he sees a lot of "random drive-by names" that clearly have no intention of sticking around.
The need for reviewer help in particular came up; Linus Torvalds jumped in to say that reviewing is boring, so it is unsurprising that people don't want to do it. He keeps seeing huge patch sets being sent out once each week with little seeming to happen with them. A few tweaks, perhaps, before the next version is sent, but no real resolution. What is really needed, he said, is to find ways to get away from the email patch model, which is not really working anymore. He feels that way now, even though he is "an old-school email person".
Bacik said that the Btrfs developers are using GitHub to track things; it is good at showing the work that is outstanding, so he can see what has been languishing. That has improved the subsystem's throughput significantly. There are, he said, tools out there that "make the tasks we hate easier", and that every other project uses to get their work done. Gleixner, though, expressed skepticism that adopting another tool would solve the problem. He has a patch-tracking system that works well enough, he said; the real solution is to teach managers that, with proper engineering, work gets done sooner (and life for maintainers is easier).
Ts'o said that he never knows how long a patch submitter will be around, so it's never clear whether time spent to educate them will be worthwhile. He also said that, while asking submitters to fix existing technical debt in order to get their work merged is asking too much, maintainers can take a stand against adding more debt. Hellwig said that a developer trying to contribute code is often the best opportunity to get some cleanup done. Bacik said that the Btrfs community has been guiding developers that way for a long time and has learned how to do it well; he admitted that he should be writing their approach down. Maintainers should, he said, take a bigger role in teaching others.
Gleixner said that a lot of useful information for contributors has indeed been written down. Dave Chinner, though, worried that pointing contributors to that documentation can come across as an impersonal brush-off. That is why he often takes the time to write a long response to contributors needing guidance. Rostedt said that today's developers have different needs. He asked how many in the room had started kernel work because their employers had told them to; no hands were raised. That is not the case for many of today's new developers, he said.
"Being a maintainer is a part of our identity", Airlie said; it is likely how we got our current job and is not something that we readily let go of. Brauner added that people tend to hold onto power for dear life. Wolfram Sang said that he likes reviewing, but can get no support for doing that work; Dan Williams said that developers tend not to understand just how much social capital they can get from doing good reviews. Bacik said that the group was taking an overly simple view of the reviewing task; many developers hesitate to do reviews because they don't want to be seen as having missed something if a bug turns up. It was widely agreed that nobody should feel that way, since no one can be expected to catch everything. How to communicate that to the community as a whole is unclear, though.
Torvalds said that a Reviewed-by tag mainly means that the reviewer will be copied on any bug reports; developers should add those tags liberally, he said. Gleixner added that maintainers make fools of themselves every other day. Hellwig said that he has been trying to review code outside of his comfort area; it takes a couple of times before he feels that he understands well enough to offer a Reviewed-by tag. Rostedt, though, raised the issue of bare Reviewed-by tags offered without any discussion, which can be a sign of somebody trying to game the system and get into the statistics. Bacik said that, if the maintainer does not know the reviewer, their Reviewed-by tag means nothing.
Torvalds said that some subsystems are setting their requirements for contributors too high, making it hard for new developers to come in. Chinner added that the kernel's culture can be off-putting and not inclusive, making people fight to get their changes in. Bacik agreed, saying that there is no arbiter in the community; he said that Torvalds wants developers to figure things out for themselves, so disagreements over changes often end up as big battles. He would like to move to a system that is more encouraging of efforts to find solutions.
Torvalds said that, while the community gets a lot of new contributors, it doesn't tend to get many new maintainers. The contributor and maintainer roles should be separated, he said. Chinner said that becoming a maintainer is often seen as a promotion for developers who do good work; Torvalds answered that people often see maintainers as some sort of "super developer", but they are really just managers. He took a moment to thank Konstantin Ryabitsev for the b4 tool, which has made life much easier for maintainers; the attendees responded with applause.
As this part of the session came to a close, Williams said that part of the pay for reviewing work is autonomy within a subsystem, but that the community doesn't actually provide that autonomy. Instead, maintainers hold onto all of the decision power. Airlie answered that the DRM subsystem has done well with distributing that power among a number of developers.
A support group
![[Steve Rostedt]](https://static.lwn.net/images/conf/2023/lpc/SteveRostedt-sm.png "Steve Rostedt")
A related session was run by Rostedt, who started by saying that he has
heard a lot of maintainers and developers complaining about burnout. There
are many things that could be done about this problem, but often all that a
tired developer really needs is somebody to talk to. He is proposing the
creation of a list of developers who are willing to lend an ear when the
need arises. These developers would have no power, they would just be
there to provide support and advice when a problem arises.
Torvalds answered that if he wanted to talk to somebody, he wouldn't go to a kernel developer. Bacik, though, said that he is willing to do some basic support work. He can talk well with developers who are at the same level in the community, but his ability to get others to listen to him is not great. He suggested that Torvalds should take less of a laissez-faire approach to the development community and help solve problems more often.
Chinner asked what problem Rostedt was trying to solve; Rostedt answered that many developers feel isolated and that they could benefit from a support group, but they don't know who to talk to. Bacik said that, with the developers he works with, he knows that problems can be worked out. But perhaps developers who lack that assurance could use some support.
Williams asked whether the inability for developers to see each other for a couple of years contributed to problems; many people seemed to think that it did.
At the end of this session, Torvalds said that about half of the emails he receives are private, rather than copied to the mailing lists. Developers are always welcome to send him a note when they are having problems; he has often had long discussions with developers about conflicts. Ts'o said that individual subsystems often have a decision maker who can bring conflicts to an end, resolving disputes by decree if they have to. The community lacks that resource for cross-subsystem issues, though.
The next step will be for Rostedt to propose an addition to the kernel's
process documentation describing this support group.
| Index entries for this article | |
|---|---|
| Kernel | Development model |
| Conference | Kernel Maintainers Summit/2023 |
Posted Nov 24, 2023 17:08 UTC (Fri)
by james (subscriber, #1325)
[Link] (6 responses)
Jon, when was the last time you had a full week's vacation?
(Obviously, you don't have to answer that — but is this a wider problem?)
Posted Nov 25, 2023 1:23 UTC (Sat)
by beagnach (guest, #32987)
[Link]
Posted Nov 25, 2023 15:47 UTC (Sat)
by rsidd (subscriber, #2582)
[Link] (2 responses)
I'm not sure I would enjoy being cut off from everything for a week.
Posted Nov 27, 2023 8:42 UTC (Mon)
by taladar (subscriber, #68407)
[Link] (1 responses)
Posted Dec 10, 2023 11:01 UTC (Sun)
by sammythesnake (guest, #17693)
[Link]
On the other hand, what works for one person is certainly not guaranteed to work for another. I know I find being "off work" (in a very *very* different role (!)) takes really quite some effort and adaptation - I get bored of not having something to do! Perhaps that's the case for some maintainers (and people like rsidd)... I suspect that the nature of the maintainer role probably needs complete breaks from time to time to actually *feel* off the hook, though.
As a practical matter, there needs to be somebody who can take on some of the role during those breaks for things that can't wait for one reason or another, as well as making good calls about whether the break needs interrupting. Getting a proper break from the responsibility is somewhat undermined by having to worry about what's waiting upon return!
Posted Nov 30, 2023 16:04 UTC (Thu)
by andy_shev (subscriber, #75870)
[Link] (1 responses)
Posted Nov 30, 2023 16:51 UTC (Thu)
by geert (subscriber, #98403)
[Link]
Posted Nov 25, 2023 5:08 UTC (Sat)
by vasvir (subscriber, #92389)
[Link]
Posted Nov 25, 2023 6:27 UTC (Sat)
by wtarreau (subscriber, #51152)
[Link] (1 responses)
In haproxy originally I was the only one to merge others' patches (like in Linus' current position) and I became a bottleneck. Some of the main developers suggested that I should let a few more people commit to the project, and that significantly unclogged everything. We're about 4 committers active daily. We certainly do merge a bit more bugs because now there are less reviews for such people who can directly push, but that's only just a bit because when you're alone on the critical path, you don't spend your time reviewing everything and you tend to just trust and merge, which is not much better. However we probably multiplied by 3 or 4 our patch bandwidth capacity, so overall I think the bug rate per patch has significantly decreased. Of course sometimes it can create some frictions because some disagree on something that was just merged. And so what ? It can later be fixed or even reverted and that's all; we just insist a lot that merged patches must not break bisects and that if CI is broken it must be immediately fixed, possibly via a revert.
And interestingly, there are affinities that developed between certain developers and certain committers. That's normal, some interactions may feel smoother to those who want a more human experience or more efficient for those who just expect more direct wording. Also some committers specialized in certain areas over time, and other developers know how to more efficiently interact with them. But at least now all these committers perform some level of review (since they're responsible for what they merge) or sometimes trust the submitter for some patch sets.
I know that some Linux subsystems already have multiple committers. I think that this should be encouraged much more, first among those already doing reviews, and maybe till the point it ultimately reaches the top level. I'm sure it would attract more people into review work and get more maintainers from current reviewers-only. This would also mechanically reduce the number of steps between the moment a patch is produced and the moment it is merged, because all this patch manipulation requires time from someone, and that's all the chain that suffers to move patches from one tree to another. With less intermediaries involved, there can be a better experience for the patch's author and less work for other maintainers.
That's also a nice way to start to engage into getting more people up to speed to replace aging developers (which will also become a problem once the kernel reaches 40 years old and most developers who were originally students just want to retire). There's a less difficult transition when one person leaves a group of 10 than when this person is alone.
Posted Nov 25, 2023 15:10 UTC (Sat)
by Paf (subscriber, #91811)
[Link]
I agree wholeheartedly - I think there is a general discouragement cycle that occurs when reviewing doesn’t seem to result in anything, because patches still languish unmerged or lose the attention of the author or…. So there can be this lengthy disconnect between effort and result, which I think really discourages people from reviewing, then they don’t bother so patches languish…. In short, I guess, frictions matter and can add up. I think reducing them can create a virtuous cycle where people will review more if they see it has a point. (Not that it doesn’t have a point today, but if the connection between action and result is stronger, etc, etc.)
Posted Nov 25, 2023 15:16 UTC (Sat)
by karim (subscriber, #114)
[Link] (40 responses)
I haven't done kernel work in ages. But even back then (20 years ago) it was hard to keep track of patches and discussions on the lkml. I've had the chance to play in many ecosystems since I stopped being active on lkml and out of everything I've seen I think the kernel developer community might want to take a look at Gerritt (https://www.gerritcodereview.com/). It's notably used for Android but also for many other projects (see https://en.m.wikipedia.org/wiki/Gerrit_(software) ).
One of the benefits of Gerritt is that any random Joe Developer on the internet can easily see at-a-glance what the state of a given patch's acceptance is and what exactly is being objected to, line by line. The barrier to entry for anyone wanting to keep track of a patch, let alone do an external review, is way much higher for the Linux kernel.
In fact, come to think of it, I wonder if the medium chosen for review (i.e. email) is actually contributing to both isolating developers from external help and making their job harder. Then again I haven't maintained an active role in lkml in almost 20 years. So maybe I'm out of touch.
Posted Nov 25, 2023 16:27 UTC (Sat)
by wtarreau (subscriber, #51152)
[Link] (35 responses)
Browser-based tools are extremely inefficient for intensive use and try hard to compensate for the UI deficiencies by implementing features that are supposed to make your life a bit easier, but in practice they only ease contributors' lives at the expense of the person in charge of the final merge. Ah and when you want to perform minor edits before merging a patch, it becomes so difficult compared to just clicking on a merge button that it's not surprising to see the abysmal quality of commit messages on projects mostly developed through browsers like many of those hosted on github/gitlab for example.
One particularity of the linux kernel project that I think makes it unique is that it has thousands of maintainers. All of them regularly need to interact with others because changes often step on other areas, and they can't always be authoritative when responding to a contributor because of this. There's a very long people dependency chain that's often visible in discussions where developers discuss what part should go through which tree for example. What is needed there is lower latency in exchanges and/or a lower number of exchanges (i.e. more rights to many maintainers). If more maintainers were able to push their changes instead of imploring someone else to queue them for the next 3 months without being able to count on them, it would certainly change a lot of things for their activities!
Posted Nov 25, 2023 16:45 UTC (Sat)
by laurent.pinchart (subscriber, #71290)
[Link] (6 responses)
This is also one of my major pain points with web-based UIs *for reviews*. They are all terrible compared to real text editors.
When discussing e-mail vs. web-based UIs, two separate discussions seem to always be mixed together. I believe the vast majority of developers, including kernel developers and maintainers, don't claim that mail clients are good for *tracking* the status of patches. That why we have patchwork and other similar tools, and git forges do a reasonable job there too compared to mail clients. When it comes to *reviewing* code, however, web-based UIs are just terrible.
> it's not surprising to see the abysmal quality of commit messages on projects mostly developed through browsers like many of those hosted on github/gitlab for example
I'm appalled that after so many years neither github nor gitlab has implemented the ability to comment on a commit message. I agree this is likely one of the factors that explain why so many projects have terrible commit message hygiene. Surely this can't be that complex to implement compared to the zillions of features that those forges provide, which makes me wonder if they have consciously decided not to provide the feature, have never tried their own dog good, or if github/gitlab are also developed with bad commit message hygiene.
Posted Nov 25, 2023 18:49 UTC (Sat)
by Sesse (subscriber, #53779)
[Link] (3 responses)
I used to feel the same. But web-based UIs have a huge advantage over email: It's easy to see which comments have been addressed, and which ones still need your attention. It's like having one little email thread per comment, which is really nice—so it's the advantage of something like Patchworks, but on a much finer scale.
(I've mainly used some Google-internal tools and Gerrit; both are fine by me. But I'm not a kernel maintainer.)
Posted Nov 26, 2023 9:58 UTC (Sun)
by adobriyan (subscriber, #30858)
[Link] (2 responses)
Yes! The ability to point specific line and write comment near it (in a monospace font!) is what's needed.
> (I've mainly used some Google-internal tools and Gerrit; both are fine by me. But I'm not a kernel maintainer.)
Gerrit is cool (unless coworkers are slacking on reviewing your diffs and then it is not cool anymore).
Posted Nov 26, 2023 17:13 UTC (Sun)
by khim (subscriber, #9252)
[Link] (1 responses)
Gerrit uses Markdown so you write in a monospace font like this: Here's just some random example. Where, ironically, enough, you may see discussion about commit message with embedded code snippets in a monospaced font. I haven't looked for that, just clicked on some change with large number of unadressed comments. Not sure how well Gerrit would scale to the Linux kernel scale, but if there are limited number of reviewers it works very nicely: each “subdiscussion” is attached to some piece of code and there's status to know if it's addressed or not, you may easily see how many complains are not addressed. The biggest issue is that often when you address the comment and rewrite the code gerrit couldn't find the point to attach it in the new version of patch and then it just shows it at the top of the file. Then you open the previous version of patch and look on comments on the left and code on the right. Not ideal, but not sure that part is handled by e-mail any better.
Posted Nov 26, 2023 17:40 UTC (Sun)
by excors (subscriber, #95769)
[Link]
Posted Nov 26, 2023 16:59 UTC (Sun)
by khim (subscriber, #9252)
[Link]
In Gerrit one may add comments to the commit message using the same approach used for code: select the piece and add inline comment. Most likely that. The assumption is that you would only used Git for code snapshots and all the metadata goes into Web interface instead. I suspect that's even conscious decision to cause lock-in effect.
Posted Nov 27, 2023 10:13 UTC (Mon)
by fschrempf (subscriber, #160781)
[Link]
GitLab being an open-source project one could argue that simply no one ever cared enough to step up and implement the feature. The same argument kernel maintainers would use for some kernel related feature requests. Looking at the 6-year-old issue at https://gitlab.com/gitlab-org/gitlab/-/issues/19691 it looks like there has been some interest over the years but not enough for someone to actually do the necessary work.
And yes, a lot of projects developed on GitLab/GitHub suffer from bad commit message hygiene which is a pity in most cases, but some project development workflows are designed a lot differently than the kernel one and don't even really allow (or care?) to have a clean Git history (e. g. due to a lot of merge commit and bot commit noise).
Posted Nov 25, 2023 18:21 UTC (Sat)
by laf0rge (subscriber, #6469)
[Link]
I still maintan a number of open source software projects these days within the osmocom (open source mobile communications) project. Given my kernel background, we've originally started with a email + patch based system in 2008, added patchwork slightly later and have migated over to gerrit a number of years ago.
I have to say to me as a maintainer, reviewing and processing patches in gerrit is making my life much easier than an e-mail based workflow. Don't get me wrong, I am an e-mail person, I dislike GUIs, work all day on command line interfaces and I do read/write my e-mail in mutt. But still, gerrit is in my opinion a big benefit for the maintainer, unlike it was claimed earlier that it is a benefit for the patch submitter. I'd actually argue the other way around: For a random developer it's much easier to send a patch by e-mail than to install the git hooks for inserting the Change-ID and learning how to push stuff to gerrit.
I also don't get the comments made about "comments not being in-line with the code" or "linear comments". In gerrit, you are placing your comments right at the specific line[s] of code that you are commenting on, and the entire thread of comments is attached to that line of code.
I think a lot of the opinions against code review tools comes from the fact that most people have only seen github/gitlab/gitea style pull requests, which I as a maintainer consider a significantly inferiror tool the way how gerrit works. Gerrit doesn't know "pull requests", it knows individual patches. It tracks a single patch over any number of revisions. It can show you easily not just the current version of the patch, but the changes compared to any of the previous versions of it. It keeps code review at the specific location within the code. It allows you to track if a given comment raised in review is considered resolved or not. It warns you if you try to apply a patch that still has unresolved comments, etc.
For people who want to review / comment from the command line, there's a command line tool called "gerrit review" (see https://gerrit-documentation.storage.googleapis.com/Documentation/2.8/cmd-review.html) so you don't have to open your browser if you don't want to. I'm personaly not using it, but several of the osmocom developers do use it.
Posted Nov 25, 2023 20:07 UTC (Sat)
by roc (subscriber, #30627)
[Link] (16 responses)
Github PRs exist as branches you can pull, which makes applying them to a tree very easy. The branch is named right at the top of the PR! On the other hand, it is not obvious how to apply patches from a pile of email messages to a tree, although I take your word for it that those "in the know" have some way to do it.
Likewise, most people's email clients make very poor code editors, but I take your word for it that those "in the know" have all developed email-based workflows that do incorporate good code editors for when you want to include code in review comments.
It makes sense to prioritize maintainer needs over contributor needs in a project like the kernel, where many people *have* to contribute to the kernel to get their work done so there's little downside to making that process arbitrarily difficult to get started with. And therefore it makes sense that you would never want to change processes that all the maintainers already know. But the gap between what most people use and how the kernel process works keeps growing.
Posted Nov 25, 2023 20:33 UTC (Sat)
by viro (subscriber, #7872)
[Link] (15 responses)
In any case, you are supposed to read and review the stuff in the branch; that's, er, somewhat more mentally taxing effort than any of the above.
As for the editors, I'm yet to find one in a browser that wouldn't be atrocious; used to be possible to have the damn thing start xterm and a sane editor in it, but that got broken. And yes, I'm using a browser to type that comment in - and hating every minute of it. For anything that requires minimally non-trivial manipulations of text (like, you know, search and replace with regular expressions, or pipe a block of text through a filter - trivial things like that) it would be extremely painful.
Posted Nov 25, 2023 22:47 UTC (Sat)
by laurent.pinchart (subscriber, #71290)
[Link] (2 responses)
I stopped counting the number of times I pressed ctrl-w to delete the word I had just typed in a browser-based text editor. Just thinking about it makes me want to scream.
Posted Nov 26, 2023 2:59 UTC (Sun)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
Posted Nov 26, 2023 17:13 UTC (Sun)
by gray_-_wolf (subscriber, #131074)
[Link]
But those (as far as I know) do not (and cannot) work on privileged pages (about:config, addons.mozzila.org, ...). Which is annoying.
What did work for me is to just build my own firefox with modified key bindings: https://git.sr.ht/~graywolf/gecko-dev/commit/7648188f9751... . I assume it is possible without the compilation, but I found now documentation for it. And I am rebuilding anyway...
It can be combined with following in .config/gtk-3.0/gtk.css to get actual C-w working everywhere:
@binding-set text-entry
entry { -gtk-key-bindings: text-entry; }
Offtopic: I also prefer SHIFT+INSERT to paste PRIMARY instead of CLIPBOARD, so https://git.sr.ht/~graywolf/gecko-dev/commit/46b121a7a550... is nice as well (albeit bit hackish).
Posted Nov 26, 2023 13:03 UTC (Sun)
by kleptog (subscriber, #1183)
[Link] (8 responses)
Maybe it's just me, but the idea I actually need to apply a patch to a local branch to review it seems, well, suboptimal. After all, I may be managing several projects, and many patches and having to manage all those branches just seems like unnecessary work. Compared to something like Gerrit where the patch is already applied, I can simply view the change and move through the diff and make comments directly just seems much easier. Patches that don't apply cleanly, or fail the build step are already filtered out for you.
With email patches, rebase failures are made the problem of the maintainer rather than the submitter, and that just seems wrong. If I actually want to try out the patch, I can just pull the version from Gerrit directly.
> For anything that requires minimally non-trivial manipulations of text (like, you know, search and replace with regular expressions, or pipe a block of text through a filter - trivial things like that) it would be extremely painful.
Why on earth would you want search/replace while entering a comment? If you're actually want to propose a change to the patch itself, you can edit it within the browser itself (VSCode can be embedded anywhere AIUI)
Posted Nov 27, 2023 4:58 UTC (Mon)
by viro (subscriber, #7872)
[Link]
If patches are trivial and don't need me to look outside of context diff, sure, I can reply directly to email. What's the benefit of web-based anything in that respect, anyway? For anything more involved I very much prefer to be able to do it in a real editor, where I can poke around the tree, etc. Can you do the same in browser? As in, something in the patch gets me to look for the places where this field of this structure is ever accessed, to verify that their locking is, indeed, sufficient. Or something like "their change seems to remove a bunch of calls of that primitive; how many are left afterwards? If there's none, this and that could've been done much simpler and cut down the complexity of patch series big way... <greps> Umm... only two callers left; let's look around and see if they can be massaged out of existence as a preliminary step - looks like that might be a serious win overall". Or writing an explanation that would be longer than a few lines, with e.g. pseudo-code snippets in it. Or any number of other things, really...
As for rebase failures - huh? If I don't know which version the series is based on, how the hell could I review it in the first place, not to mention pulling it in? If the patches do not apply, you ask the submitter what are they supposed to be against; if they can't tell that, you don't want to deal with the series anyway. Oh, and throwaway branches are not a problem to remove, you know...
Posted Nov 27, 2023 12:19 UTC (Mon)
by farnz (subscriber, #17727)
[Link]
FWIW, I use local branches for review all the time with web-based review systems like Gerrit and GitHub PRs. It's just convenient to have the full power of my local setup available for reviewing the code, looking at what things outside the change are doing (which in turn allows me to suggest helpful follow-ups). git makes it trivial to check out a branch or commit and look at it, to try out changes I'm going to suggest (complete with building and/or testing them).
And git worktree add makes it trivial to create a new working copy for the duration of one review, which I can then delete after I'm done.
Posted Nov 27, 2023 18:43 UTC (Mon)
by wtarreau (subscriber, #51152)
[Link] (5 responses)
This question is funny! This happens to be all the time: just because your ideas develop as you review while you're commenting. You start by suggesting something and you change your mind in the middle for something better, so you just want to s/foo/bar/g to replace all the variables prefixes. And similarly, wanting to just indent a block to put an "if" clause around is super frequent when reviewing. All things that browsers make impossible to do without having to press space 100 times (since tab will get you out of the box and if you're unlucky you'll even press "Post" while touching the space bar). I, too, despise every single minute I type in a browser. These tools are entirely designed to make you consume contents, not produce them.
Posted Nov 29, 2023 12:26 UTC (Wed)
by kleptog (subscriber, #1183)
[Link] (4 responses)
I guess we mean different things by reviewing. Reviewing for me does not involve writing any code, the idea is to provide the submitter enough information to fix it themselves and resubmit.
Posted Nov 29, 2023 14:42 UTC (Wed)
by wtarreau (subscriber, #51152)
[Link]
It's often much more efficient (and friendly) to write "I think maybe a better approach could be something around this:" and give the start of an example, than "no, still not what I'm expecting, try again". Look at reviews done on LKML, a non-negligible number of them suggest code snippets to discuss around. This is particularly true for bug fixes, where it's often a tradeoff and multiple approaches need to be explorated. That's precisely one of the reasons some tools are quite bad at this, when they don't allow the reviewer to express what they're thinking about, and cause many round trips.
Posted Nov 29, 2023 20:34 UTC (Wed)
by viro (subscriber, #7872)
[Link]
Posted Nov 30, 2023 11:48 UTC (Thu)
by farnz (subscriber, #17727)
[Link]
I often find myself wanting to quickly confirm that a suggestion I have is workable before I make it - I'm not infallible, but I can quickly write a small amount of code (ignoring error checking, ripple effects outside the area I'm changing, good style etc) to see what my suggestion will do to the code I'm reviewing. If I like what I see, I'll make the suggestion, while if I see that it's actually obfuscating the code, I'll not suggest it.
Posted Dec 4, 2023 14:02 UTC (Mon)
by Hattifnattar (subscriber, #93737)
[Link]
And a reason or a suggestion often involve, surprise-surprise, code.
Reviewing is just a conversation on a specific topic. The topic being code, it's entirely reasonable that the conversation will include code fragments...
Posted Nov 26, 2023 13:41 UTC (Sun)
by nevets (subscriber, #11875)
[Link] (1 responses)
Posted Nov 27, 2023 5:08 UTC (Mon)
by viro (subscriber, #7872)
[Link]
Posted Dec 26, 2024 12:00 UTC (Thu)
by alx.manpages (subscriber, #145117)
[Link]
Actually, you can optimize that a little bit:
In neomutt(1)/mutt(1), you can press '|' within a message to pipe it to a command, which can be 'git am -s'. So, when reviewing patches for a project, I cd(1) into the worktree where I apply patches from contributors, then read mail, and from mail directly `|git am -s`, omitting the "save" step. You can even do that with encrypted mail (with a configuration line to tell it to decrypt while piping).
I expect other MUAs should have similar abilities.
BTW, another advantage of email:
You can use PGP to sign and/or encrypt the email.
Posted Nov 26, 2023 16:28 UTC (Sun)
by johannbg (guest, #65743)
[Link] (1 responses)
Well you first have to become a contributor prior to becoming a maintainer, which is often the path which existing maintainers use to evaluate contributors and then later if approved offer/ack them as fellow maintainers right so my question to you is...
During your career as a kernel maintainer how many contributors have you approached and asked if they want to become a maintainer? When doing so what did you use as an incentive for that person to become a maintainer and what was the contributors response to that?
The thing is either existing maintainers aren't reaching out to prospects and offer them to become maintainers or existing contributors see no value in becoming maintainers in the first place.
If there is no value in the role of a maintainer then that role is effectively becoming obsolete ( some sort of structural change is needed ).
Posted Nov 28, 2023 16:17 UTC (Tue)
by wtarreau (subscriber, #51152)
[Link]
Accepting a co-maintainer is nice for everyone. Less stress for both, particularly for the person, knowing that you're still there for complicated stuff, and it allows that person to start to take some liberties they wouldn't otherwise have attempted to take. *this* factor is important because there is not a single way to do things, and your way might have been good in the past and no more suitable. A fresh mind can come with a different approach, and being a maintainer, feel a bit more legitimate (or at least feel like there's less risk their work gets rejected).
If I had to do it again, I would definitely do it. The most stressful for me is to know I'm on a critical path and slowing someone else down. This is negative participation and is not acceptable. Sometimes it's better to have other people with a bit less experience of certain points and more availability than just rely on experts that are never available. Then experts can say in the background and appear only when needed.
Posted Nov 27, 2023 23:12 UTC (Mon)
by pj (subscriber, #4506)
[Link] (7 responses)
Have you considered finding an external editor plugin for your browser? I'm fairly sure they're available for most browsers, and generally let you right-click on a textarea and send it to an editor... and then replace the existing text when you save from the editor.
Posted Nov 27, 2023 23:42 UTC (Mon)
by viro (subscriber, #7872)
[Link] (4 responses)
If anyone could share pointers towards a working setup that would allow e.g. to launch an xterm with vi in it, the contents of the area copied there and copied back after you are done editing it, I would be really grateful.
Posted Nov 28, 2023 5:48 UTC (Tue)
by Cyberax (✭ supporter ✭, #52523)
[Link] (2 responses)
Well, it kinda is.
Your only solution right now is to use a localhost-bound HTTP server, which is not such a bad idea. Chrome might also gain support for UNIX domain sockets, WHATWG has approved an RFC for it last year.
You also can try to create a fake service using Chrome debug API, but it's even _worse_.
Posted Nov 30, 2023 21:35 UTC (Thu)
by comex (subscriber, #71521)
[Link] (1 responses)
https://developer.chrome.com/docs/extensions/mv3/nativeMe...
Posted Dec 1, 2023 6:13 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Dec 6, 2023 16:42 UTC (Wed)
by lacos (guest, #70616)
[Link]
"external editor revived" <https://github.com/Frederick888/external-editor-revived> and "toggle line wrap" <https://git.kiszka.org/togglelinewrap.git> for thunderbird.
Posted Dec 2, 2023 0:15 UTC (Sat)
by jschrod (subscriber, #1646)
[Link] (1 responses)
I'm using Emacs server already, that part is covered.
I didn't find such a plug-in - but maybe my search foo is not good enough and I used the wrong search terms. Posted Nov 25, 2023 20:08 UTC (Sat)
by b7j0c (guest, #27559)
[Link]
Not to mention the frustration of different clients - I found that using Fastmail, I had to run dos2unix on attachments to make a patch merge-able. Not the end of the world, but one more hazard to deal with.
SourceHut has an interesting patch UI layered on top of email, coupled with the "hut" command line tool that allows patches to be merged directly from the message list id. Even this misses the conversational element of Pull Requests as implemented by Github, Gitea etc.
The ForgeJo (fork of Gitea used at Codeberg) folks are trying to create a forge-neutral format for messages like Pull Requests etc, which would alleviate concerns about being locked in to a particular forge...being JSON, these messages would at least amenable to modern web UIs.
Posted Nov 27, 2023 17:40 UTC (Mon)
by paulj (subscriber, #341)
[Link] (1 responses)
The only example I know of is "git appraise" - https://github.com/google/git-appraise . Perhaps it's the only one cause it's good enough, and we all should get behind it?
In a similar vein, distributed bug tracking in git. "git bug" - https://github.com/MichaelMure/git-bug - seems one of the best developed tools in this space?
Posted Nov 27, 2023 17:43 UTC (Mon)
by paulj (subscriber, #341)
[Link]
Like how 'ledger' format became a de facto standard for "plain text accounting".
Perhaps the git 'appraise' and 'bug' tools should be the basis for that?
Posted Jan 9, 2024 22:48 UTC (Tue)
by ceplm (subscriber, #41334)
[Link]
Posted Nov 26, 2023 9:33 UTC (Sun)
by johannbg (guest, #65743)
[Link] (11 responses)
While the kernel development remains spreader like infectious disease across random corners of the internet people wont bother to participate and contribute what ( little ) free time they have to do so.
In other words solve the fragmentation problem in the kernel community, move the development to the place where people actually reside these days, re-roll the dices for charisma for some of it's maintainer, slow down the development cycle for the maintainers health and bobs your uncle.
Posted Nov 26, 2023 10:42 UTC (Sun)
by b7j0c (guest, #27559)
[Link] (3 responses)
I would recommend something like a Gitea or Forgejo instance managed by the Linux Foundation. Yes, this would require contributors to maintain an account on such a service, but if you can't be bothered to register an account it is unlikely you are serious about making contributions anyway.
Posted Nov 26, 2023 12:29 UTC (Sun)
by johannbg (guest, #65743)
[Link] (2 responses)
Problem 1.
Problem 2.
Bear in mind in exercises like these, the amount of energy is the same, it's just a matter where it's being spent, how efficiently and more importantly *who* is spending it.
With a self hosting approach like you propose with [Insert tool like Gitea] then you are putting all the cost,time and effort onto the Linux Foundation, which means you need to sell that idea to Finance,HR,Infrastructure at LF ( which arguably will be a hard sell ) on top of having to convince everyone to consolidate to LF.
In addition to that ( which relates to problem 2 ), you need to sell people the idea of having to sign-up to participate in the community which reduces the likelihood of participation and excludes drive by contribution.
With something like github all the energy is being spent by github, no additional load whatsoever on LF or any vendor related to the community and there is nothing to sell because people already exist there ( Github is the largest and the most *financially* stable of those ) .
Yes the downside is loss of control to certain aspect and *everyone* will have to adapt to new workflow but I would consider that a little price to pay for a survival of a community.
Posted Nov 26, 2023 16:31 UTC (Sun)
by b7j0c (guest, #27559)
[Link] (1 responses)
Posted Nov 26, 2023 16:40 UTC (Sun)
by johannbg (guest, #65743)
[Link]
Posted Nov 27, 2023 6:30 UTC (Mon)
by roc (subscriber, #30627)
[Link] (6 responses)
If there was a genuine competitor that Linux had to compete with for contributors, things would be very different.
Posted Nov 27, 2023 10:22 UTC (Mon)
by johannbg (guest, #65743)
[Link] (5 responses)
So if by some stroke of luck *any* employer in *any* field finds an employee born 2000 or later that can just think for itself, is not glued to the phone all the time and shows up on right time. That employer found a keeper.
That translates to in the F/OSS world, if he did not stumble to your project by accident, made a PR, he's a keeper.
The point being the need is there regardless if the kernel community admits needing it or not because that contributor pool is not growing bigger by any means, nor any smarter thus is nowhere near the expectation or standards existing maintainers have and expect.
Everyone is fighting for those contributors/people that have the smarts that finds it's way all the way down to the keyboard ( from head to hands ) because that resource is becoming scarce, fast...
Posted Nov 27, 2023 11:03 UTC (Mon)
by mb (subscriber, #50428)
[Link]
Posted Nov 27, 2023 13:01 UTC (Mon)
by karim (subscriber, #114)
[Link] (3 responses)
Posted Nov 27, 2023 16:30 UTC (Mon)
by khim (subscriber, #9252)
[Link] (2 responses)
I suspect that you both are right. Take a look on quote from Aristotle just above your message. People also like to boring earlier Socrates: The children now love luxury; they have bad manners, contempt for authority; they show disrespect for elders and love chatter in place of exercise. I don't know why mb brought that quote but it makes me queasy: does he understand context for these quotes or not? Does he want to read and laugh “oh, yeah, wise people told the same thousands of years ago… humanity is still around, everything would be fine… just relax”. Or maybe he wants to support johannbg? It's really hard without him saying anything. Let me remind you something. Both Socrates and, later, Aristotle lived in a failing former hegemon state: state that had rich history, which was, many years before, a full-blown hegemon of Ancient Greece… but was clearly past it's zenith of power and on the road of the irreversible decline! Do they forget about that or just simply don't know? Note that Athen's greatest achievements in the philosophy and culture happened after that period which wouldn't have been possible if all young people were awful. Some were quite bright, indeed, but most of the great people in that period weren't born in Athens anymore. I think what we are observing now is similar phenomena: Western Civilization is collapsing, the majority of young people are useless for mankind, but there are still enough bright people who may still do many great things.
Posted Nov 28, 2023 19:01 UTC (Tue)
by ghodgkins (subscriber, #157257)
[Link]
"There is a great tendency among the children of today to rebel against restraint, not only that placed upon them by the will of the parent. But against any restraint or limitation of what they consider their rights [...] this fact has filled well minded people with great apprehensions for the future." - Rev Henry Hussmann
No "failing former hegemon state" or "irreversible decline" to see here. Despite Fr. Hussmann's great apprehensions, America made out alright in the 20th century. Indeed, by my math, the children he was complaining about here led America through a period of unprecedented prosperity, technological advancement, and hegemonic power post-WW2.
Posted Nov 28, 2023 19:29 UTC (Tue)
by mb (subscriber, #50428)
[Link]
Half of the population has an IQ less than 100.
Posted Nov 26, 2023 15:18 UTC (Sun)
by flussence (guest, #85566)
[Link] (3 responses)
Pardon the snark. The point is they *used* to. The process got all this way under the assumption that people believed in a noble cause enough to volunteer their time to it.
What changed between then and the 2020s is that the demographic who are here to do things for the greater good are no longer the majority in certain areas of FOSS space. The punk bar is now something very different. The magic that sustained this ecosystem has gone elsewhere. I do have hope it can be fixed, but it probably requires lots of outside specialist help.
Posted Nov 26, 2023 17:30 UTC (Sun)
by khim (subscriber, #9252)
[Link] (1 responses)
I don't see how that may be fixed. Linux started as a volunteer project where people were not paid. Today the majority of contributors come from commercial companies and for many of them kernel development is a regular job. Now that people who started as free volunteers and later got job in commercial companies (do we even have maintainers who work on kernel without that being either their full-time job or part-time job mentioned in their work contract?… maybe is some fringe areas like M68K support, I guess…) tell to these developers that do work for money and expect that they would do something for free… why should they? How does that work? It's time to admit that kernel is commercial project and start acting accordingly. Hordes of volunteers couldn't arrive if most people work on kernel to earn money! For them if job doesn't make it's easier to earn money then it's not worth doing.
Posted Nov 27, 2023 12:43 UTC (Mon)
by mfuzzey (subscriber, #57966)
[Link]
But while most kernel developers are now paid Linux hasn't fallen into the trap of giving their employers too much technical say and becoming a corporate project. No manager of a kernel developper can say "I know it's half baked but merge it now beacuse marketing wants it" and Linux largely maintains the "it'll ship when it's ready" and "longterm maintability is important" mindsets that are missing in most commercial projects.
Companies paying kernel developpers do get some say in the "what" (ie if they're interested in some particular area of the kernel, say power management or networking, they can hire specialists in those areas and ask them to work on that rather than other parts of then kernel) but they don't get a say on the "how" (ie what implementation is acceptable, that lies with the maintainers and, ultimately, Linus).
Of course what works for Linux (by virtue of being a "commons" on which most higher level stuff is now built) probably doesn't work for a lot of other OSS projects which still struggle to find developpers.
Posted Nov 26, 2023 21:05 UTC (Sun)
by rra (subscriber, #99804)
[Link]
There are approximately 100x more projects that I want to volunteer my time towards than I have available time. I suspect this is true of everyone who has the ability to volunteer time with no compensation, which is itself a tiny fraction of humanity. Free software and open source have expanded far beyond the scale that is supportable with volunteer labor without some sort of vast change to our surrounding societies that gives people far more leisure time.
The "it's a noble cause and people should volunteer their time for free to support it" model also looks very different when you're in your twenties than when you're in your forties or fifties, particularly if you have kids.
I don't think that original model can be fixed, absent some sort of hard-to-imagine change where we all only work four hours a week for our existing salaries. I don't think it *should* be fixed absent such a post-scarcity SF scenario. We need something new, or at least new to this type of work.
We have followed a path that has been followed by many other technologies and developments in history: something that was sustainable through the leisure activities of relatively rich people has now expanded into something large enough that it needs to be prioritized by society as a whole. That means people need to be supported by society in some way while they work on it (through salary or whatever other mechanism). The problem is not going to shrink in the future back to a scale supportable by leisure activity.
Posted Nov 27, 2023 11:56 UTC (Mon)
by taladar (subscriber, #68407)
[Link]
Maybe there needs to be a systematic approach to this problem first, list all the requirements in a process- and tool-agnostic way first (e.g. "make sure contributions to the subsystem are free of obvious bugs, free of intentional security holes, match the coding style, are documented, have tests, use the new and not the deprecated API for x,...", "test integration of all contributions within the subsystem/with contributions to other subsystems/...", "see later who reviewed a particular contribution", "allow contributors to see if their contribution is making progress/needs a new revision/...") and then discuss appropriate options to achieve the goal via processes, tools, manually by a specific role or automation,...
The kernel has created new tools in the past, perhaps this or adapting existing tools more heavily than the typical user might be appropriate here again.
Posted Nov 27, 2023 15:02 UTC (Mon)
by Nikratio (subscriber, #71966)
[Link] (1 responses)
Not the fault of the writer if this was indeed how the session ran, but if so it must have been rather frustrating.
Posted Nov 30, 2023 0:48 UTC (Thu)
by nevets (subscriber, #11875)
[Link]
Posted Nov 28, 2023 22:45 UTC (Tue)
by sramkrishna (subscriber, #72628)
[Link]
Also Ted Tso looks burned out in that photo. :)
Darrick needs to come for beers. :-)
One maintainer has either not taken a complete week away from the kernel for way too long, or is very good at covering his breaks.
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Torvalds: Reducing kernel-maintainer burnout
What is really needed, he said, is to find ways to get away from the email patch model, which is not really working anymore. He feels that way now, even though he is "an old-school email person".
Those are clickbait headline news.
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Deleting unused lines in email client before reply is waste of time after all.
> The ability to point specific line and write comment near it (in a monospace font!) is what's needed.
Reducing kernel-maintainer burnout
Here is how I would do that:
```
if (foo) {
...
} else {
...
}
```
I think your example is a non-public link, but the same review is public here instead.
Reducing kernel-maintainer burnout
> I'm appalled that after so many years neither github nor gitlab has implemented the ability to comment on a commit message.
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
{
bind "<Control>w"
{
"delete-from-cursor" (word-ends, -1)
};
}
textview { -gtk-key-bindings: text-entry; }
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Stop the fragmentation that has happen in the community and make it whole again, which provides single point of entry for *everyone* to partake in every aspect of the kernel development, which in turn will make it easier for anyone to join the community and start participating.
Increase the ( likelihood of ) participation in the kernel community which in turn will reduce the load on existing contributors ( more contributors == less load on everyone ).
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Does it have enough resources man power wise to run it?
Has LF's HR managed to clone Konstantin? ;)
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
(Aristotle)
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
>but there are still enough bright people who may still do many great things.
Has always been like that.
You are just getting old and grumpy.
Reducing kernel-maintainer burnout
> I do have hope it can be fixed, but it probably requires lots of outside specialist help.
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout
Reducing kernel-maintainer burnout