|
|
Subscribe / Log in / New account

Toward a kernel maintainer's guide

By Jonathan Corbet
November 26, 2018

Linux Plumbers Conference
"Who's on Team Xmas Tree?" asked Dan Williams at the beginning of his talk in the Kernel Summit track of the 2018 Linux Plumbers Conference. He was referring to a rule for the ordering of local variable declarations within functions that is enforced by a minority of kernel subsystem maintainers — one of many examples of "local customs" that can surprise developers when they submit patches to subsystems where they are not accustomed to working. Documenting these varying practices is a small part of Williams's project to create a kernel maintainer's manual, but it seems to be where the effort is likely to start.

In theory, Williams said, kernel maintenance is a straightforward task. All it takes is accumulating patches and sending a pull request or two to Linus Torvalds during the merge window. In this ideal world, subsystems are the same and there is plenty of backup to provide continuity when a maintainer takes a vacation. In the real world, though, the merge window is a stressful time for maintainers. It involves a lot of work juggling topic branches, a lot of talking to people (which is an annoying distraction), and the fact that Torvalds can instinctively smell a patch that is not yet fully cooked. Maintenance practices vary between subsystems, and there is no backup for the maintainers in many of them. It is hard for a maintainer to take a break.

Kernel maintainers, he said, are a gang of opinionated people. They don't always agree on things, but the good news is they don't have to. So why would we want a maintainer's handbook? The idea is to create a reference [Dan Williams] manual for both maintainers and contributors, a collection of "tribal knowledge" and best practices rather than a set of rules. There is a lot of good advice for maintainers to be found in email discussions, but nobody has, yet, gone to the effort to capture that information and present it in a useful form.

Another way of putting it, he said, is that there is a fair amount of pain in the community, and he would like to try to alleviate it. Contributors feel the pain of trying to get a maintainer to do something; maintainers, instead, feel the pain of simply trying to hold everything together. He noted that he, too, is guilty of doing things as a maintainer that have caused him stress as a contributor; it is easy to unintentionally make the process harder for others. By addressing some of those pain points, Williams hopes he can help to create a better experience for all involved.

For example, one painful experience for contributors is getting silence in response to patches sent to a maintainer. Different maintainers exhibit different latencies, so it is hard to know when to press further. One way to address this problem might be for maintainers to advertise an equivalent to a service-level agreement (SLA) documenting the response time they agree to provide. Associated with the SLA could be information like a set of trusted reviewers who could stand in for the maintainer for many review tasks, the location of the subsystem's test suite, and more. By setting the contributor's expectations, the guide should make their life easier; they will know when to resend a patch.

Another part of the guide would concern itself with preventable maintainer mistakes. There have been a lot of lectures on the proper use of Git posted by Torvalds (and others) over the years; it should be collected and put into a place where maintainers can find it before they make a mistake. Torvalds, Williams said, provides great explanations of how things should be done "after the storm passes"; he does so patiently, repeatedly, as the same mistakes are discovered anew. Why, Williams asked, isn't this information written down anywhere?

As Williams looked into the creation of a kernel maintainer's guide, he discovered that one already exists; it was created by Tobin Harding in 2017 and hasn't been changed since. His first objective is to add subsystem profiles to this guide; the profile is meant to tell contributors how to work with the subsystem. It would include information like:

  • Whether the subsystem accepts pull requests or, instead, requires that all submissions be posted as patches to a mailing list.
  • The last day before the merge window that new features can be posted and the last day that any new features could actually be merged. This "last day" is likely to be expressed in terms like "when -rc5 comes out".
  • What the requirements are for Reviewed-by or Acked-by tags on patches and whether the maintainer is allowed to merge unreviewed patches.
  • Whether the subsystem has a test suite and where it can be found.
  • A list of trusted reviewers for the subsystem.
  • The "resend cadence" for the subsystem — how long should a contributor wait before resending a patch?
  • The time zone(s) in which the maintainers operate, which would be a hint for when contributors could expect a response to an email.
  • The maintainer's opinion on trivial cleanup patches.
  • Whether the maintainer trusts off-list patch reviews. These often take the form of a Reviewed-by tag from somebody who works for the same company as the submitter; not all maintainers put much faith in such tags.

If nothing else, this list is an interesting overview of how different kernel maintainers approach their job. It drew the obvious question ("why do we have these differences in the first place?") from the audience, but there isn't really an answer beyond "it has always been that way".

The session wound down with some unfocused discussion on the details of the subsystem profiles. Should they be listed in the MAINTAINERS file? Should they include the maintainer's expectations on the documentation of new features? Answers to those questions will have to await the conclusion of the wider discussion, which is ongoing as of this writing. But, as Mel Gorman noted at the end of the talk, the work that has been done so far is a useful enumeration of the problem space, which is a good start.

[Thanks to the Linux Foundation, LWN's travel sponsor, for supporting my travel to the event.]

Index entries for this article
KernelDocumentation
ConferenceLinux Plumbers Conference/2018


to post comments

Toward a kernel maintainer's guide

Posted Nov 26, 2018 19:59 UTC (Mon) by l1k (subscriber, #112260) [Link]

FWIW, a document that might be called a "subsystem profile" exists for the PCI subsystem, but isn't part of the tree yet, it was only posted to the list so far.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 2:54 UTC (Tue) by unixbhaskar (guest, #44758) [Link] (1 responses)

Sounds interesting though. But, in practice "that's the way it is" is very difficult to get rid of and it will take way more time than one can imagine. And if it does, it will be a damn good thing to happen too.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 10:25 UTC (Tue) by k3ninho (subscriber, #50375) [Link]

>"why do we have these differences in the first place?"
...should always draw the answer that we're adults working together in the varied ways that different people do.

Even if you find a common structure to the workflow, each person will deliver it in their own unique way, from their perspective on the tasks at hand or their understanding of the problem and its solutions to how they work through each task -- the challenge is to share enough information among a team responsible for common success so that people can make progress.

Cat herding teaches us that cats are going to be cats, so let's work with that to communicate and interact in ways which get the best from each kitteh.

K3n.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 7:28 UTC (Tue) by marcH (subscriber, #57642) [Link] (3 responses)

> Torvalds, Williams said, provides great explanations of how things should be done "after the storm passes"; he does so patiently, repeatedly, as the same mistakes are discovered anew. Why, Williams asked, isn't this information written down anywhere?

Why indeed?

I almost never answer generic questions directly; I just hate repeating myself too much. I update or submit a documentation update instead and answer with a one-line pointer to the update.

https://en.wikipedia.org/wiki/Don%27t_repeat_yourself

> It would include information like:

As a (former) "consumer" of code freshly merged by maintainers, I would recommend adding the following information:

- where is the git repo where maintainer X shares code he or she just merged, on its way up(stream)
- short "map" of said git repo, notably which git branches are "volatile" versus not (in other words: does using the cherry-pick "-x" option make sense or not)

These seem to be highly maintainer-dependent.

https://chromium.googlesource.com/chromiumos/docs/+/maste...
https://public-inbox.org/git/70ccb8fc-30f2-5b23-a832-9e47...

> It drew the obvious question ("why do we have these differences in the first place?") from the audience, but there isn't really an answer beyond "it has always been that way".

"awareness" is always the first step and documentation achieves that (and more)

Toward a kernel maintainer's guide

Posted Dec 15, 2018 5:12 UTC (Sat) by djbw (subscriber, #78104) [Link] (2 responses)

"Don't Repeat Yourself" (DRY) is a good principle, but when the document is not written yet, please do repeat until someone takes the initiative and writes it down.

The details on the git repo and a repo map are a good idea. They also dovetail with the recent addition of a kernel.org patchwork-bot that uses a map of the maintainer's git repo to determine when to mark patches "Accepted" vs "Mainline".

Toward a kernel maintainer's guide

Posted Dec 16, 2018 1:24 UTC (Sun) by marcH (subscriber, #57642) [Link] (1 responses)

> "Don't Repeat Yourself" (DRY) is a good principle, but when the document is not written yet, please do repeat until someone takes the initiative and writes it down.

The point I tried to make: it's barely more effort to wrap an email answer / "upgrade" it to a documentation patch (submitted by... email?). So little more that it pays off as soon as the second time the same question comes again.

Now of course I forgot the whole documentation system has to be implemented in the first place: https://xkcd.com/974/
Cheap shot sorry, I can imagine there's a real effort to get all this started and fair play Dan.

PS: I'm considering my next Linux distro to be Arch purely based on their... wiki.

Toward a kernel maintainer's guide

Posted Dec 16, 2018 2:06 UTC (Sun) by djbw (subscriber, #78104) [Link]

Ah, it's a fair point, especially how long it has taken to find time to get this going, i.e. just start collecting links to lore.kernel.org archives of these examples. I do think there is some value to writing a preface, but probably less than I think. I'll start with that with some rough categories to gather the links I know about and worry about longer winded curation later.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 11:21 UTC (Tue) by mfuzzey (subscriber, #57966) [Link] (5 responses)

I think not all differences are equal.

Having different coding style rules for different parts of the kernel seems to be mostly a pain for everyone involved and I don't see any reasonable justification other than inertia.

On the other hand having somewhat different *processes* in terms of git trees, review rules, cycle time etc for different subsystems makes a lot more sense since maintainers do have different workflows and the "best" workflow is certainly also very dependent on the size of the subsystem (in terms of patch volume and number of developers).

Toward a kernel maintainer's guide

Posted Nov 27, 2018 15:29 UTC (Tue) by bfields (subscriber, #19510) [Link] (1 responses)

Having different coding style rules for different parts of the kernel seems to be mostly a pain for everyone involved and I don't see any reasonable justification other than inertia.

I can think of a few.

It takes effort to reach consensus: is a flame war on Christmas-tree declaration ordering really such a high priority?

People have more fun if they're given a little leeway.

The original Documentation/CodingStyle was 193 lines. Looks like currently Documentation/process/codingstyle.rst is 1079 lines. I wonder if people read it.

Toward a kernel maintainer's guide

Posted Nov 30, 2018 11:30 UTC (Fri) by ale2018 (guest, #128727) [Link]

Questions about reasons for doing x or y should always be welcome. In practice, they sound so marginal that readers just skip. Walking through the code together seems to be gone for good, or perhaps it's just not supported by the tools?

Toward a kernel maintainer's guide

Posted Nov 27, 2018 16:01 UTC (Tue) by rweikusat2 (subscriber, #117920) [Link] (2 responses)

> Having different coding style rules for different parts of the kernel seems to be mostly a pain for everyone involved and I don't see > any reasonable justification other than inertia.

Empirically, this model has worked exceedingly well for a long time, hence, there's no reasonable justification to change it now just because that's unfamiliar to certain people. The only "pain" involved here is if style conventions are being enforced (or sometimes enforced, depending on the social situation) which aren't documented anywhere: There's no way someone who isn't already familiar with whatever petty regulations are considered important can divine knowledge about them. Hence, trainwrecks are guaranteed to occur: Someone's being annoyed that things were done differently despite "everybody knows that THIS is the proper colour for a bikeshed!", someone else finds himself suddenly stepping on hidden landmines despite painstakingly trying to do everything "in the right way".

As the paragraph above probably does (and certainly should) suggest, ideally, there would be no (or at least as few as possible) petty regulations at all. That different conventions about these can successfully coexist suggests that they aren't really necessary.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 16:56 UTC (Tue) by mfuzzey (subscriber, #57966) [Link] (1 responses)

> Empirically, this model has worked exceedingly well for a long time, hence, there's no reasonable justification to change it now just because that's unfamiliar to certain people.

Yes, that's what I meant by inertia

>The only "pain" involved here is if style conventions are being enforced (or sometimes enforced, depending on the social situation) which aren't documented anywhere

Obviously if they aren't being enforced there is no pain :)
But if they are enforced, even if documented, it is a pain to those that work in multiple subsystems whereas those that only work in one probably never notice.

>As the paragraph above probably does (and certainly should) suggest, ideally, there would be no (or at least as few as possible) petty regulations at all. That different >
>conventions about these can successfully coexist suggests that they aren't really necessary.

Indeed, I completely agree with this. I wasn't suggesting that there should be a great "christmas tree debate" kernel wide. But that a rule should either be considered sufficiently important to be in the global CodingStyle or it should not exist.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 22:59 UTC (Tue) by rweikusat2 (subscriber, #117920) [Link]

> Indeed, I completely agree with this. I wasn't suggesting that there should be a great "christmas tree debate" kernel wide.
> But that a rule should either be considered sufficiently important to be in the global CodingStyle or it should not exist.

I don't think that's realistic, especially considering that some maintainers enforce coding style regulations which contradict what's written in the CodingStyle text (Patrick "This function is used only once. Get rid of it" McHardy being an example I personally encountered). OTOH, expecting outside contributors (or prospective outside contributors) to read throuhg a few years of a large number of high-volume mailing lists in order to discover "undocumented preferences" of over 1000 people is also not realistic. That's nothing but a camouflaged and guarded "No entry"-sign.

There must be a way for someone to determine which guidelines to follow, ie, a document listing them. Otherwise, one can only try to imitate how the other code is written and that's not a trusty guide: Some "style regulations" are actually not about style at all, they mandate functional differences, even to the point of requiring that code works in an unlogical way. Eg, some people are strongly convinced that all loops ought to be pre-test loops. I don't think so because the initial state is often known and testing a condition known be either true or false seems absurd to me. Nevertheless, I'd gladly follow such a rule if someone considered it important. But there's no way to infer this from code.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 15:31 UTC (Tue) by kdave (subscriber, #44472) [Link] (1 responses)

I've written 2 wiki pages about process and good practices, and another oriented to coding practices, regarding the btrfs development. They're incomplete, there are namely the recurring topics or things that I'd like people to know before contributing. This should supplement the well known CodingStyle and SubmittinPatch documents, yet many of style or subsystem coding practices get pointed out way too often. This makes me think that nobody reads the pages, unless pointed to. I for one welcome the initiative to gather and document the maintainers' pracitces and preferences so it's more discoverable.

Toward a kernel maintainer's guide

Posted Nov 27, 2018 19:19 UTC (Tue) by nilsmeyer (guest, #122604) [Link]

Since the wiki is disconnected from the code, it may be better to add this Documentation to the source tree.

Toward a kernel maintainer's guide

Posted Nov 28, 2018 21:41 UTC (Wed) by ncm (guest, #165) [Link]

The problem with most coding conventions is that many entries legislate superstitions, often long-debunked ones.

The most pervasive and harmful of these is that programmers get confused by seeing more than one style. Appeals to this myth block improvements. In fact it is always easy to tell which of two styles in a file is the new one, and which is legacy. The useful unit of consistency is the equivalent of a paragraph -- a function, or a block in a bigger function.

The second is that any rule is as good as any another -- that all that matters is consistency. Not having objective evidence for one or other is not the same as that no such evidence is possible. Usually it would be way more work to collect than the difference is worth to one project, but there are plenty of projects conducting the experiment. So, sometimes any rule is better than no rule, but then comes time to step to a better rule. Take it, because it will make a difference.

Toward a kernel maintainer's guide

Posted Nov 29, 2018 16:44 UTC (Thu) by bored (subscriber, #125572) [Link] (1 responses)

If the goal is to remove the no trespassing signs, and reduce the amount of bikesheding, then the solution is an authoritative automated style test and appropriate shaming of people complaining about style violations not enforced by said test.

Humans are error prone, and don't generally enforce rules consistently ("do as I say, not as I do"), no amount of subsystem specific style guides/etc is going to change that unless its programmatically enforced.

Toward a kernel maintainer's guide

Posted Nov 29, 2018 17:53 UTC (Thu) by karkhaz (subscriber, #99844) [Link]

Yes, I felt like I was missing something during this whole discussion. The example that keeps coming up is reverse Christmas tree, but that's surely something that can be added to checkpatch.pl or a similar per-subsystem script. Why do maintainers ever need to think about this stuff? Same thing for the suggestion of a maintainer SLA, which will immediately go out-of-date as soon as it's written down---why not an automatic tool that displays maintainers' mean email response time over the past few months, the variance, as well as any recent break in responses that might indicate that they're currently on holiday or otherwise indisposed?

There are probably a few tiny cases where an automated tool wouldn't suffice or should be overridden, but I don't see why a lot of this work can't be replaced with a very small shell script.

Local customs as signalling mechanism

Posted Dec 3, 2018 19:20 UTC (Mon) by ecree (guest, #95790) [Link] (4 responses)

I have long suspected that the reverse xmas tree rule is a "no brown M&Ms clause"⁽¹⁾. I.e. it's there to give maintainers an easy way to tell whether a submitter has taken the effort to learn and follow the subsystem's rules and practices. One could also look at this as a kind of shibboleth.
If this theory is correct, then making these "local customs" more publicly visible could undermine their purpose, and drive maintainers to impose ever more obscure requirements in an arms race against the people documenting them.

[1] https://en.wikipedia.org/wiki/Van_Halen#Contract_riders

Local customs as signalling mechanism

Posted Dec 16, 2018 0:05 UTC (Sun) by marcH (subscriber, #57642) [Link] (3 responses)

> If this theory is correct, then making these "local customs" more publicly visible could undermine their purpose,

I doubt Dan or anyone else will ever surface a "reverse Christmas tree rule" at the very *top* of any new documentation, it'll be buried somewhere in the middle as it should. So gathering and structuring this sort of documentation will only make it look *more* like a [Van Halen] contract that is finally possible to read entirely and will make complying with some "brown M&Ms clause" easier if any.

Local customs as signalling mechanism

Posted Dec 16, 2018 0:47 UTC (Sun) by excors (subscriber, #95769) [Link] (2 responses)

Surely the position of the rule in the document should depend entirely on the length of the rule, rather than attempting any sort of logical ordering based on the meanings of the rules.

Local customs as signalling mechanism

Posted Dec 16, 2018 1:12 UTC (Sun) by marcH (subscriber, #57642) [Link] (1 responses)

I normally refrain from posting this type of comments on LWN but sorry can't resist this time: ROFL.

Lame excuse: the previous comment "forgot" a smiley.

Local customs as signalling mechanism

Posted Dec 17, 2018 11:02 UTC (Mon) by sdalley (subscriber, #18550) [Link]

Well, yes indeed. Absence of the smiley amplified the LOL by enhancing the image of the rule being uttered with the utmost seriousness by some po-faced pompous twit.
[pauses to collect myself]


Copyright © 2018, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds