Some 5.19 development statistics

The 5.19 kernel was released, after a one-week delay to deal with the fallout from the Retbleed mitigations, on July 31. By that time, 16,399 commits (15,134 non-merge and 1,265 merges) had found their way into the mainline repository, making this development cycle the busiest since 5.13 (16,030 non-merge changesets and 1,157 merges). Tradition dictates that now is the time for a look at where the changes in 5.19 came from, and we've learned not to go against tradition.

Individual contributors

Work on 5.19 was contributed by 2,086 developers; that is a new record, beating the 2,062 who contributed to 5.13. Of those developers, 278 made their first kernel contribution during this development cycle. The removal of a number of old drivers and an unloved architecture took 301,000 lines of code out of the kernel repository, but that effort was overwhelmed by the 1,105,000 lines of code that were added, for a net growth of 804,000 lines of code.

The top contributors to 5.19 were:

Most active 5.19 developers

By changesets Krzysztof Kozlowski 211 1.4% Christoph Hellwig 193 1.3% Ville Syrjälä 175 1.2% Matthew Wilcox 151 1.0% Jakub Kicinski 130 0.9% Geert Uytterhoeven 123 0.8% Mark Brown 118 0.8% Masahiro Yamada 105 0.7% Arnd Bergmann 104 0.7% Martin Kaiser 102 0.7% Kuniyuki Iwashima 101 0.7% Christophe Leroy 96 0.6% Minghao Chi 96 0.6% Biju Das 94 0.6% Andy Shevchenko 90 0.6% Marek Vasut 89 0.6% Miaohe Lin 87 0.6% Dmitry Baryshkov 87 0.6% Ping-Ke Shih 81 0.5% Pavel Begunkov 79 0.5% Jason A. Donenfeld 79 0.5% Jack Xiao 79 0.5%

By changed lines Hawking Zhang 222682 18.1% Huang Rui 185566 15.1% Martin Habets 44361 3.6% Jakub Kicinski 34636 2.8% Ping-Ke Shih 29871 2.4% Huacai Chen 21159 1.7% Bjorn Andersson 15738 1.3% Christoph Hellwig 14024 1.1% Leo Liu 11632 0.9% Haijun Liu 11006 0.9% Fabio M. De Francesco 9561 0.8% Ian Rogers 8691 0.7% Imre Deak 7937 0.6% Zhengjun Xing 7508 0.6% Arnd Bergmann 7424 0.6% Leon Romanovsky 6573 0.5% Mark Brown 6502 0.5% Cezary Rojewski 6492 0.5% Peter Ujfalusi 6463 0.5% Veerasenareddy Burru 5652 0.5% Manivannan Sadhasivam 5614 0.5% Jack Xiao 5215 0.4%

The top contributor of changesets in 5.19 was Krzysztof Kozlowski, who focused mostly on devicetree fixes. Christoph Hellwig continues to rework code all over the kernel, and found the time to remove the h8300 architecture as well. Ville Syrjälä contributed a large number of changes to the Intel i915 graphics driver, Matthew Wilcox continues the folio work, and Jakub Kicinski worked extensively in the networking subsystem.

In the lines-changed column, as has become traditional, Hawking Zhang and Huang Rui outdid everybody else with the addition of hundreds of thousands of lines of machine-generated amdgpu header files. Martin Habets added the "siena" network driver, Kicinski removed a number of old network drivers while taking a break from his other work, and Ping-Ke Shih added support for Realtek 8852ce network adapters.

The lists of top testers and reviewers will look familiar to those who have been following these articles:

Test and review credits in 5.19
Tested-by Daniel Wheeler 94 8.4% Bean Huo 29 2.6% Nathan Chancellor 29 2.6% Geert Uytterhoeven 27 2.4% Heiko Stuebner 26 2.3% Nícolas F. R. A. Prado 23 2.1% Michael Riesch 21 1.9% Marek Szyprowski 20 1.8% Arnaldo Carvalho de Melo 19 1.7% Gurucharan 18 1.6% Sedat Dilek 18 1.6% Giuseppe Scrivano 18 1.6%	Reviewed-by Christoph Hellwig 246 2.9% Hawking Zhang 220 2.6% Rob Herring 164 2.0% AngeloGioacchino Del Regno 149 1.8% Krzysztof Kozlowski 144 1.7% David Sterba 123 1.5% Darrick J. Wong 103 1.2% Bard Liao 102 1.2% Andy Shevchenko 102 1.2% Stephen Boyd 101 1.2% Jani Nikula 98 1.2% Ranjani Sridharan 88 1.1%

Many of the test credits continue to accrue to people who are seemingly working as part of their employer's internal quality-assurance process, though there appear to be fewer of those than in previous cycles. On the review side, this was a 70-day development cycle; both Christoph Hellwig and Hawking Zhang thus reviewed at least three patches for each of those days. Hellwig's reviews are widespread, while Zhang's are focused on amdgpu patches by AMD developers. It is good to see that there are developers who are evidently reviewing patches as part of their job.

A look at the report credits — along with who is including the Reported-by: tags in their fixes — also shows the evolution of an ongoing story:

Report credits in 5.19
Reporter kernel test robot 207 17.0% Zeal Robot 134 11.0% Abaci Robot 53 4.4% Syzbot 49 4.0% Dan Carpenter 44 3.6% Hulk Robot 37 3.0% Stephen Rothwell 27 2.2% Rob Herring 19 1.6% Guenter Roeck 12 1.0% Geert Uytterhoeven 11 0.9% Marek Szyprowski 11 0.9% Nathan Chancellor 8 0.7% Sudip Mukherjee 8 0.7%	Credited by Minghao Chi 93 7.6% Jiapeng Chong 31 2.5% Lv Ruyi 24 2.0% Yang Li 22 1.8% Krzysztof Kozlowski 20 1.6% Eric Dumazet 19 1.6% Yang Yingliang 16 1.3% Paul E. McKenney 14 1.1% Masahiro Yamada 14 1.1% Hans de Goede 14 1.1% Linus Torvalds 13 1.1% Randy Dunlap 12 1.0% Mario Limonciello 12 1.0%

We are evidently in the midst of the robot wars and most of us never even noticed; a full 40% of the report credits are going to robots at this point. If one looks at which developers are adding Reported-by tags to their patches, the picture becomes clearer: the top four reporters work for the companies that run the Zeal and Abaci robots (ZTE and Alibaba, respective). It is reasonably clear that these developers are developing and running their own robots to find bugs, then crediting those robots with the reports.

Companies

The employer numbers are relatively steady and boring. A total of 245 employers supported work on 5.19, with the most active being:

Most active 5.19 employers
By changesets Intel 1645 10.9% (Unknown) 1135 7.5% Linaro 862 5.7% AMD 837 5.5% Red Hat 792 5.2% (None) 653 4.3% Google 624 4.1% Meta 528 3.5% SUSE 462 3.1% Huawei Technologies 446 2.9% NVIDIA 421 2.8% Oracle 414 2.7% (Consultant) 385 2.5% Renesas Electronics 348 2.3% Arm 281 1.9% MediaTek 235 1.6% Qualcomm 232 1.5% IBM 230 1.5% Pengutronix 208 1.4% NXP Semiconductors 195 1.3%	By lines changed AMD 465548 37.9% Intel 80061 6.5% Linaro 59759 4.9% Meta 53080 4.3% Xilinx 45774 3.7% (Unknown) 37529 3.1% Realtek 36049 2.9% Google 30767 2.5% NVIDIA 30524 2.5% MediaTek 29215 2.4% Red Hat 27048 2.2% Loongson 23819 1.9% (None) 22890 1.9% (Consultant) 22322 1.8% SUSE 16983 1.4% Qualcomm 14455 1.2% Oracle 13815 1.1% Arm 12806 1.0% IBM 12339 1.0% Renesas Electronics 10812 0.9%

Perhaps noteworthy here is the slow but steady decline of Red Hat, which was the top employer for many years. The picture looks a little different if one considers non-author signoffs, though:

Non-author signoffs in 5.19

Individual Greg Kroah-Hartman 932 6.5% David S. Miller 785 5.5% Alex Deucher 704 4.9% Mark Brown 656 4.6% Andrew Morton 525 3.7% Jakub Kicinski 422 2.9% Jens Axboe 296 2.1% Mauro Carvalho Chehab 282 2.0% Bjorn Andersson 273 1.9% Kalle Valo 272 1.9% Borislav Petkov 230 1.6% Martin K. Petersen 225 1.6% Michael Ellerman 207 1.4% Arnaldo Carvalho de Melo 200 1.4% Shawn Guo 195 1.4% David Sterba 176 1.2% Rafael J. Wysocki 166 1.2% Geert Uytterhoeven 152 1.1% Vinod Koul 148 1.0% Catalin Marinas 145 1.0%

By employer Linaro 1959 13.6% Red Hat 1854 12.9% Intel 1445 10.1% Meta 1056 7.4% Linux Foundation 1037 7.2% Google 930 6.5% AMD 786 5.5% SUSE 748 5.2% Qualcomm 416 2.9% NVIDIA 352 2.5% Arm 339 2.4% IBM 313 2.2% (Consultant) 307 2.1% (None) 304 2.1% Oracle 260 1.8% Huawei Technologies 202 1.4% (Unknown) 160 1.1% Renesas Electronics 156 1.1% Cisco 140 1.0% Broadcom 112 0.8%

A developer who signs off on a patch that they did not write is (usually) the maintainer who accepts the patch and sends it upstream. The above tables, thus, offer an approximate picture of who our most active maintainers are. About half of the patches merged into the mainline kernel are going through the hands of maintainers working for just five companies. On one hand, that shows a potentially concerning concentration of power in a relatively small number of employers. On the other, this is the list of companies that are most willing to pay for maintainers to do their jobs — a good thing, given that the kernel project is short of maintainers overall.

When bugs were introduced

When a commit fixes a bug, it will often contain a Fixes: tag indicating the commit that first introduced that bug. This information is useful for a number of reasons, including deciding how far back a fix needs to be backported in the stable kernels. But it can also give an indication of how long bugs have been in the kernel. The 5.19 cycle saw the addition of 2,541 commits with Fixes: tags; 712 of those (28%) referred to other 5.19 commits. Those bugs never made it into a mainline release, but the rest did. Looking at tags referring to previous releases gives this result:

As one might expect, many of the bugs fixed in 5.19 were introduced in recent releases; 268 of them came from 5.18. What is perhaps more surprising is the long tail of references back to earlier releases; only 2.6.21, 2.6.28, and 2.6.32 are missing from the plot because they had no commits that were fixed in 5.19. It can be surprising to see that there is any code left from those early development cycles at all; that code exists, though, and it still contains some bugs.

The spike at 2.6.12 may seem strange, but remember that the Git history begins then; all of the Fixes: tags pointing to 2.6.12 name commit 1da177e4c3f4, which was the initial commit that started the whole thing off. They are, thus, referring to bugs that were introduced sometime before early 2005. Almost all of those fixes are dealing with data-race issues that were seen as less problematic on the hardware of that era.

The curious can look at the full list of 5.19 fixes, which contains pointers to the fixed commits.

One can also use Fixes: tags to get a sense for when bugs are introduced during the development cycle. In this case, the results are:

-rc	5.19		All time
-rc1	656	4.7%	66,154	7.3%
-rc2	7	2.1%	1,512	3.5%
-rc3	6	2.0%	1,179	3.3%
-rc4	13	2.8%	987	3.1%
-rc5	6	2.7%	924	3.6%
-rc6	4	0.9%	863	3.5%
-rc7	15	3.3%	755	3.8%
-rc8	5	1.9%	275	3.9%
-rc9	—		32	2.2%
final	—		472	3.8%

The 5.19 numbers should be taken with at least one grain of salt; as we have seen above, the fixes for 5.19 commits will be wandering into the kernel over the next decade or so. That makes 5.19 appear, probably falsely, to be better than the kernel history as a whole; getting a complete picture for this cycle will require some patience. Beyond that, the Retbleed fixes were merged for 5.19-rc7; there were numerous fixes needed for those, which explains the elevated rate at -rc7.

In general we see, as we might expect, that most bugs enter the kernel during the merge window, whether one looks at absolute numbers or as a percentage of total commits. After that, the bug rate drops, but remains roughly the same through the development cycle. In theory, as the final release gets closer, developers should be more careful and only push the most important and well-tested commits. In the real world, late-cycle patches are just as likely to be buggy as those that came earlier, and patches that enter the mainline after the last -rc release seem to be especially risky.

On to 6.0

In the 5.19 announcement, Linus Torvalds let it be known that the next kernel would probably be named 6.0. As usual, the major-number bump has no special meaning for the kernel; it's just another release with a lot more changes in it. As of this writing, 12,325 non-merge changesets are waiting in linux-next, suggesting that 6.0 will not be as busy a cycle as its predecessor. Come back in early October for the details on how it played out.

Index entries for this article
Kernel	Releases/5.19

---

to post comments