Scaling the KVM community

The scalability of Linus Torvalds was a recurring theme during Linux's early years; these days maintainer struggles are a recognized problem within open-source communities in general. It is thus not surprising that Sean Christopherson gave a talk at Open Source Summit Europe (and KVM Forum) with the title "Scaling KVM and its community". The talk mostly focused on KVM for the x86 architecture—the largest and most mature KVM architecture—which Christopherson co-maintains. But it was not a technical talk: most of the content can be applied to other KVM architectures, or even other Linux subsystems, so that they can avoid making the same kinds of mistakes.

The problem

The KVM hypervisor is not small: it covers 6 architectures and each often supports various "flavors" of virtualization. It consists of about 150,000 lines of code, plus 90,000 lines of tests split between the kernel self tests and the kvm-unit-tests project. Every year over 150 contributors add over a thousand commits. These numbers are expected to grow, and Christopherson identified three areas that need to scale to accommodate KVM's growth:

• Development: KVM needs to support new architectures as they appear, new features of existing architectures, and new virtualization use cases.
• Maintenance: a broader set of architectures, features and use cases means more patches to review and more code to maintain
• Validation: a growing amount of code means more things that can break

It is important to note that scaling development and maintenance does not necessarily require more developers and more maintainers. The existing developers and maintainers can scale better if they can be more efficient, for example if fewer bugs are introduced. In fact, adding more developers and maintainers without improving the validation efforts will make development and maintenance slower and more expensive. It is worth noting that, whenever Christopherson referred to maintainers throughout the talk, he also included in the group people that are listed in the MAINTAINERS file as "reviewers", as well as those who are regularly triaging and fixing bugs.

In order to improve the situation, he said, the first step is to recognize the current problems in the community; even agreeing on the existence of problems would be a successful result of his presentation. In order to identify the problems, he took inspiration from the metrics used for computer networks and proposed four characteristics that can make KVM a successful community: low latency, high efficiency, better monitoring, and more durability.

Metrics

Low latency means less time between writing the code and having it reviewed and merged. High latency is perhaps the biggest problem in KVM x86, which is due to the time spent waiting for reviews and the number of "pings" that have to be sent to get the maintainers' attention. This leads to developer frustration; it also impacts the schedule of downstream consumers of KVM, making it harder for them to stay close to the Linux mainline and to work upstream first.

Over the last decade, the number of people maintaining the code base has been roughly flat, but the number of contributors has doubled. A common approach to increasing the number of maintainers is to define sub-components, but according to Christopherson the KVM code for each architecture is too small for this to be feasible. Experience also shows that splitting subsystems too finely leads to duplicated code, as was the case when code to support Intel and AMD processors was developed more independently. He thinks it is unlikely that the number of maintainers will grow in the near future.

The number of non-merge commits to KVM over the last two years has nearly doubled. This is mostly thanks to Google (his employer) moving aggressively to an upstream-first approach and merging patches that had been in its internal fork for years. However, it is unlikely that the activity will go back to the pre-2020 levels once all of those features are merged. Plenty of other changes are in the pipeline, and those that have been dropped due to lack of review resources could come back, for example address space isolation or virtual-machine introspection. This puts a high demand on developers and maintainers to do their work efficiently.

Developer efficiency is badly affected by uncertainty. In order to improve efficiency, both developers and maintainers need to know what they can expect of each other. If developers do not know which tests they need to run, and on which architectures, they will send more flawed patches; without clear rules for which branch to develop against, they will have to respin them more frequently in case of conflicts. On the other hand, maintainers need to be clear on the state of accepted patches. The time between a maintainer accepting a patch and the corresponding commit showing up in a public tree should be short, Christopherson said, because that is another source of developer uncertainty.

Efficiency not only means less time spent on the implementation of a new feature, but also less code to write in order to keep feature parity across architectures. RISC-V is the newest architecture to gain a KVM port, and it has the opportunity to generalize code written for other architectures (for example, hypervisor page-table management) instead of duplicating it. If this happens, the number of architecture-independent patches will grow in the future, after having remained almost flat for more than 10 years; it will thus reduce the number of architecture-dependent patches as well.

Monitoring is required to ensure that KVM does not acquire new bugs, and that those which slip through are fixed quickly. KVM's wealth of test code is a great asset for the project, but it doesn't help if the tests are not run so that the same bug is encountered, debugged, and even fixed by multiple users and developers.

Looking at KVM x86 commits, about 20-25% of the commits have a "Fixes" tag attached to the commit message. That percentage has started to grow at the same time that the number of overall commits accelerated. This is likely not a bad thing: "Fixes" tags have become more prevalent in general in kernel development over the last few years, and the increased development pace included fixing a lot of old bugs.

Still, "upstream KVM is woefully behind on the continuous-integration train", Christopherson said. Most of the continuous testing of KVM comes from bots that test the kernel at large, and the timing of the tests is not consistent enough for it to be relied upon as a measure of KVM's health. Testing could be done at a minimum on the public trees, but it could be done at a finer grain for each patch submission before it makes to an official queue.

Durability is just another name for stability: a stable hypervisor means that developer and maintainer time is not spent constantly fixing KVM, especially not fixing bugs that were introduced several years ago and stayed unnoticed for a long time.

Improving durability is not necessarily a simple matter, as it may entail a change in the developers' mindset. In the past, KVM has often adopted a "good enough" approach, but does close only count in horseshoes or does it also count in KVM? This approach worked when KVM had relatively few features, but eventually these shortcuts even started to interact in unexpected ways with each other or with changes made to guest operating systems. In fact, on x86 there is an API to disable "quirks", which are processor behaviors that are present in KVM virtual machines but not on actual hardware.

Even if a shortcut solution remains working, the community tends to forget the details due to attrition. Of all the people who have contributed 20+ commits to KVM for at least three years, only five are still active. Good-enough code imposes a long-term penalty on the stability of a project. Unless the community diligently documents these shortcuts, knowledge of them will be lost and will have be relearned the hard way—not just what KVM does, but why it behaves that way.

How to improve?

Having enumerated the issues that can jeopardize the well-being of the KVM development community, Christopherson proceeded to describe five ways to fix these problems. Many of these improvements are not limited to KVM, and could be applied more generally to other Linux subsystems or even other open-source projects.

The first is to document everything that developers need to work efficiently. This had been a constant theme during the first part of the talk: patch lifecycle, testing requirements for developers, expectations around flaky tests, and deviations from architectural specifications had already been mentioned as things that should be explicitly written down. He also proposed documenting the key dates within the Linux release cycle where patches will be reviewed and accepted, so that maintainers do not feel obliged to squeeze in patches at the last minute and break things. Removing uncertainty removes friction by avoiding the perception that maintainers are ignoring certain developers or features.

The second aspect of the solution is testing. Diligent testing on the part of the developers can be a major contributor to maintainer efficiency: many issues with submitted patches are caught when the maintainers run kvm-unit-tests or the KVM self tests because the developers had never done so before submission. New features should come with associated tests, which has been enforced more and more strictly over the past few years; but rather than write minimal tests to appease the maintainers, developers should use tests to find bugs themselves, using brute force when possible and even introducing bugs purposely to verify that the tests catch them. Tests can also serve as documentation and point out edge cases in the code.

It is important that the tests are easy to write and do not require too much boilerplate. While developers are unlikely to ever be excited about writing tests, the framework should not get in the way. This is especially true of the KVM self-tests framework, which allows describing test scenarios precisely but can also be daunting to approach.

The third point is sharing. More effort has to be put into sharing code across architectures. Common problems should be solved once, by consolidating code instead of duplicating it. That can only be done if maintainers are familiar with multiple architectures and suggest sharing the code from the beginning. Christopherson proposed that I, as the overall KVM maintainer, reduce my own x86 responsibilities and focus on this issue, so that the next generation of maintainers can be trained to operate on a code base where cross-architecture work is the norm. This transition was in fact already in progress at the time of the talk.

The fourth improvement is automation of both integration and developer testing in order to catch bugs as soon as possible. This reduces the "downtime" of KVM, the amount of time where the top of the development tree is broken and no new changes can be merged. While rare, these events happen and make it harder for contributors to do their work.

After the talk, Christopherson was asked how to get developers to follow his suggestions and automate their work. His answer was that the community does not only include kernel developers, and there are people that are knowledgeable about setting up automation. They can do so in a way that can help everyone in the community. He also remarked on how reinventing the test-automation wheel seems to be a "rite of passage" for Linux developers; he suggested instead that experienced developers try to share methods and scripts for testing, or handy Git aliases, even though they are unlikely to ever be part of the upstream kernel sources.

His final suggestion is to adapt to the changing circumstances. Sharing a focus on durability of the code and the community means a different approach to development of new features. Christopherson stressed that, as a rule, new features should be implemented according to the hardware specifications and without making assumptions about the guest's behavior. While there will always be exceptions due to features that are not easily (or not at all) virtualizable, following the hardware specifications relieves the developers from having to document said assumptions. Despite all the flaws that processor manuals have, they are way ahead of KVM documentation, which can then be restricted to the differences between real hardware and virtual machines.

On top of this, developers should speak up when an issue arises and propose improvements to the process and documentation so that it does not happen again. Even if their proposal is not accepted, they will at least get an explanation as to why things are the way they are.

Conclusions

KVM's growth in size and complexity has certainly been challenging to the community and the maintainers; Google's upstreaming of the previously-internal KVM patches helped make this clear to Christopherson. Addressing the challenges described in the talk will make it easier for KVM developers to add support for new technologies and new use cases of virtualization—and to make them available to private users and cloud-computing providers worldwide.