The trouble with upstreaming

Upstreaming agony

Ted Ts'o started off the session by referring to a Kernel Summit talk held earlier in the week where a couple of Chrome OS developers reviewed their efforts to get closer to the mainline. He referred specifically to slide 10, titled "Upstreaming agony", from the presentation slide deck. That slide talks about how some maintainers are far more responsive than others, how some require massive unrelated changes, and more; "oh sure we can apply this two liner… *after* you rewrite the subsystem". This variability, Ts'o said, can be off-putting for new developers and can be the reason why many developers choose not to participate in the kernel community at all. What, he asked, can be done to improve this situation?

Linus Torvalds answered that there have been similar discussions at many maintainer gatherings; part of the problem is just that not all people are the same, but it is also true that subsystems are different from each other. The memory-management subsystem affects everything, he said, and will thus be harder to change than some random driver. We do have pain points, and some maintainers do make things too hard, but he doesn't see a productive way forward to solve the problem. "Naming and shaming" of problematic maintainers might help, he suggested.

Dave Airlie mentioned some of the the problems that vendors experience; bringing up a new system-on-chip (SoC) can involve upstreaming dozens of drivers, each of which has a different path — involving different acceptance criteria — into the mainline. That does not make things easy. Arnd Bergmann responded that, for Arm-based SoCs, there is a workaround to this problem in the form of his arm-soc tree. Drivers can go in via that path.

I could not resist the urge to point out that, in previous discussions on this topic, the group had decided to create a maintainer's handbook with subsystem profiles where each subsystem's quirks can be documented. The maintainers handbook exists now, but the section for subsystem profiles is nearly empty. So we still have almost no documentation to help patch submitters work with specific subsystems. Torvalds suggested going the other way, whereby developers would document their problems working with specific subsystems. Maintainers don't think about these things, he said, but patch submitters have no choice.

I expressed my lack of enthusiasm for the task of handling patches that add criticism of other maintainers.

Chris Mason reminded the group of the discussion at the end of the previous session, which touched on developers who are happier working within their companies than with the upstream community. Even people seen as core developers can feel that way. Getting code upstream is an uncertain process, Mason said; it makes the patches better, but at the cost of making life harder for developers. It would be good to find ways to add more certainty to the process. At the Monday discussion on folios, developer Matthew Wilcox said that he did not know which tests to run to show the performance impact of his work. When such a long-time core developer doesn't know that, we have done something wrong, Mason said.

Torvalds answered that, when it comes to memory management, there are no benchmarks, just "thousands of different things" that developers run. He mentioned the Phoronix benchmarks, saying that they have found a "ridiculous" number of problems. The benchmarks are not great, but what is great is that there is a person who follows up and is tracking performance over time. He gets annoyed at the syzbot tests, he said, because they are run on massive Intel machines that will show a performance regression just because a cache line moved. For performance testing, that is not hugely helpful. [Note that Torvalds said "syzbot", but it seems likely he meant to refer to the Intel 0day tests].

Al Viro added that a report of a regression is interesting, but even more interesting is the standard deviation of the results. He has seen many claims of improvement that, once one looks at the data, turn out to be "basically noise". Torvalds noted that the syzbot tests do at least provide the standard deviation with their results.

Thomas Gleixner said that the community can document formal criteria, but it's much harder to document expectations with regard to code design. The fact that there are different criteria in different areas makes that harder. And, in any case, people don't listen to that sort of guidance and just "cram things into corners". Mason asked if the tone of some of the guidance given to developers might make things harder; sometimes they can feel like they are getting beaten up all the way through the review process. Gleixner replied that he mostly asks questions, and he spends time explaining issues to inexperienced developers. He gives up, though, when dealing with ten-year veterans who won't listen. Mark Brown added that the community has discussed tone many times before; it often comes down to differences in culture.

At the close of this mostly inconclusive session, Ts'o brought back the point that many developers want to work outside of the community because it is easier. It is not always clear how to fix that; many areas that developers can work on affect a huge variety of users. It is easy to make a patch that works well for one workload but which trashes others. In large companies, developers are rewarded for solving that company's problems; the rewards for contributing back to the community and improving the kernel for everybody are harder to find. We would do well, he said, to find ways to point out and reward developers who have made the kernel generally better.

A common problem, he continued, is the case of the first developer for a new hardware feature being asked to make a new framework that works in the general case. The user interrupts patch set is currently experiencing that, he said; this is new Intel feature, but developers are asking how it will work for other architectures. That is a hard question to answer when other CPU manufactures haven't said whether or how they will make that feature available. We need to go easier on the first developer to a new area, he said.

At that point the room fell silent and the discussion moved on to the next topic.