Leading items

Welcome to the LWN.net Weekly Edition for December 17, 2020

This edition contains the following feature content:

• Speeding up CPython: an unusual proposal with the potential to significantly speed the execution of Python programs.
• Changing CentOS in mid-stream: the CentOS distribution enters another new chapter.
• Reducing page structures for huge pages: a trick to reduce memory overhead on large systems.
• Statistics from the 5.10 kernel development cycle: our usual look at the provenance of the code that went into the 5.10 release.
• A new release for GNU Octave: an overview of Octave and the 6.1 release.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Speeding up CPython

Python, at least in the CPython reference implementation, is not a particularly speedy language. That is not at all surprising to anyone who has used it—the language is optimized for understandability and development speed, instead. There have been lots of efforts over the years to speed up various parts of the interpreter, compiler, and virtual-machine bytecode execution, though no comprehensive overhaul has been merged into CPython. An interesting new proposal could perhaps change that, though it is unclear at this point if it will take off.

Mark Shannon posted a message about his ideas to the python-dev mailing list in late October. He noted that CPython is slow, "yet little is done to fix it"; he would like to change that. He has a four-phase plan to get there on the technical side, along with some ideas on how to fund the effort. Beyond that, he had some thoughts on what was different from earlier efforts:

He pointed those interested to a GitHub repository where he had placed a few documents describing the plan. The overall idea is for each phase to increase CPython performance by 50%, which results in a 5x speedup at the end of all four phases. The intent is for these improvements to apply widely: "Performance should improve for all code, not just loop-heavy and numeric code." He described a tiered execution model that would separate code into tiers based on how frequently it runs; each tier would be handled differently:

The funding plan is rather creative and different, but also seems to be drawing some scrutiny—and perhaps controversy. Each phase would require roughly $500,000; half of that would be used to pay for the development and the other half would be held by the organization that is coordinating the funding for use in the maintenance of CPython. He suggests the Python Software Foundation (PSF) as the obvious choice for that role.

In order to reduce the risks of a contractor for a specific phase being unable to deliver the specified performance increase, payment would only be made on delivery. For the first phase, Shannon has working code, so he is essentially suggesting that the PSF raise the funds to engage him as the contractor for that phase. But that approach raised the eyebrows of Steven D'Aprano:

He went on to ask what happens to the code if the PSF decides not to pursue the idea. Shannon did not directly address that question, but he is trying to negotiate a business deal of sorts:

Chris Angelico wondered about Shannon's plan for phase two, which is envisioned as a grab bag of fairly small improvements: "you're proposing a number of small tweaks in stage 2 and expecting that (combined) they can give a 50% improvement in overall performance?" But Shannon pointed out that compounding helps here: "20 tweaks each providing a 2% is a 49% speedup."

Also, as D'Aprano noted, each phase stands on its own. No commitment would need to made for phase two until phase one has been delivered (and been paid for). Then if Shannon was unable to achieve the 50% performance increase for phase two, as Angelico worries, Shannon would not be paid for that work. There is some wiggle room in the proposal, D'Aprano pointed out, so that getting a 49% increase would still result in payment, perhaps on "some sort of sliding scale".

Angelico also asked if the $2 million might be better spent on improving PyPy. Shannon said that there is a difference in what partial success would mean for the different projects. The main hurdle for PyPy as a replacement for CPython is getting the C extensions to work; a partial solution to that problem only goes so far:

Paul Moore was generally in favor of the proposal if someone could be found to fund it. But it is important to ensure that the usual development workflow, including code review, pull requests, and so forth, is going to be followed, he said. While Shannon did not directly reply to that message, elsewhere in the thread he made it clear that he plans to handle these changes in the normal fashion. In fact, both he and Moore said that some of the money might be used to pay for code review.

Gregory P. Smith was favorably inclined toward the idea as well. Shannon is effectively proposing a path toward bringing ideas from his earlier HotPy (2) project to CPython "with an intent to help maintain them over the long term", Smith said. He believes the terms are quite reasonable, with regard to both the money and the risk protections; it is a good fit for CPython:

The Python steering council also weighed in. Thomas Wouters posted a lengthy reply in early November that relayed some of the council discussion of the idea. The council is not opposed to the idea at all, but does caution that there are a lot of details that need to be worked out before it could make a serious effort to find an entity or entities to fund phase one. There is also a question of prioritization:

Shannon replied that he had not yet made a proposal to the council but was glad to get the feedback. He agreed that maintenance is an important priority, which is why he included money for that in his proposal, but that it will be easier to justify performance improvements to potential funding entities:

He also said that the total amount of code being added to CPython is not that large and that he hopes to offset any complexities in the implementation with better organization of the code base.

It is an ambitious proposal that was generally met with approval, even if the "show me the money" funding model is somewhat unorthodox. It is not entirely clear where things go from here, but it would seem a bit sad to miss out on an opportunity to make such a large difference in CPython performance. For many Python-using companies, $500,000 (or even $2 million) is not much of a stretch financially—gathering a few of them together should make that even easier. With luck, some discussions are already underway.

Comments (38 posted)

Changing CentOS in mid-stream

For years, the CentOS distribution has been a reliable resource for anybody wanting to deploy systems with a stable, maintained Linux base that "just works". At one point, it was reported to be the platform on which 30% of all web servers were run. CentOS has had its ups and downs over the years; for many, the December 8 announcement that CentOS will be "shifting focus" will qualify as the final "down". Regardless of whether this change turns out to be a good thing, it certainly marks the end of an era that began in 2004.

How we got here

CentOS was born out of an effort to build and distribute packages from the RHEL source provided by Red Hat. The initial CentOS release — CentOS 3.1 (based on the RHEL 3 release), came out in March 2004. There was also a CentOS 2 release (based on RHEL 2), but that showed up two months later. CentOS quickly attracted attention from users looking for a relatively stable system during a time when distributors were doing their best to separate free "community" distributions from the paid-for "enterprise" variants. LWN first mentioned CentOS in February 2004, and looked more closely in early 2005.

CentOS proved to be the stable base it promised to be; that CentOS 2 release, for example, was supported until June 2009 and CentOS 3 had support until November 2010. There were some challenges, though; also in 2009, project co-founder Lance Davis disappeared, along with his control over the project's domain name and bank account. That situation was eventually worked out, happily, but not before the project survived some significant turbulence, forcing it toward more transparency in its governance and finances.

The project also had trouble making timely releases in 2009, a problem that would resurface the following year — and often thereafter. Creating a CentOS release is not a simple or particularly fun job, so volunteers have often proved hard to come by. In 2011, this problem caused the project to fall dangerously behind on security updates while trying to produce the CentOS 6.0 release — a problem that would plague the project for much of the year. In 2012, Oracle tried to use the update delays as a way to market Oracle Linux to CentOS users.

At the beginning of 2014, Red Hat acquired the CentOS project, taking ownership of the trademarks and hiring several CentOS developers. At the time, a governing board was set up, and Red Hat promised that the project would be run in a "public, open, and inclusive" way. A small fuss over version numbers raised concerns about how a post-acquisition CentOS would be run but, for the most part, things continued as before, just on a more solid footing. The project announced a rolling release at the end of that year.

After that, CentOS appeared to settle into a steady routine of relatively predictable releases; the project mostly managed to stay out of the news. The Scientific Linux distribution, another popular RHEL-based offering, stopped making new releases in 2019, with sponsor Fermilab stating that it would switch to CentOS 8 instead. With support promised through May 2029, that seemed like a good bet. CentOS had become boring, predictable, and reliable.

CentOS Stream

In September 2019, a blog post by Red Hat CTO Chris Wright introduced CentOS Stream, describing it this way:

This new offering was parallel to the existing CentOS distribution, Wright said; "this means that nothing changes for current users of CentOS Linux and services". And nothing did change for most of those users, who paid little attention to the CentOS Stream product.

That story changed on December 8, though, as can be seen in another blog post from Wright: "we’ve informed the CentOS Project Governing Board that we are shifting our investment fully from CentOS Linux to CentOS Stream". No longer would CentOS Stream be a parallel offering; now it is the only offering. He added that:

To drive home the point that the deal has truly been altered, Red Hat announced that the end-of-life date for CentOS 8 had been moved forward to the end of 2021, taking away over seven years of support that had once been promised for that release. CentOS users who are concerned that CentOS Stream might not be suitable for production use are encouraged to "contact Red Hat about options".

To a first approximation, CentOS Stream looks a lot like CentOS, but the relationship between the two distributions and RHEL differs. While traditional CentOS is derived from official RHEL releases, CentOS Stream gets (most) updates before they go into RHEL. It is, in effect, a rapidly changing snapshot of what the next RHEL release might look like. A package that might update once in CentOS, after being released in RHEL, may update several times in CentOS Stream before becoming an actual RHEL package.

This new arrangement is better for some users. Anybody who wants to ensure that their code will work with the next RHEL release can test on CentOS Stream and get an early warning of any problems. Users who would like to get fixes more quickly may also like CentOS Stream better. For many production users, though, the prospect of "updates several times daily" lacks appeal. These users tend to want their platforms to be stable with minimal changes; a system that looks a lot like the beta test for RHEL is not what they had in mind. The unhappiness in the CentOS user community is thus easy to understand. "Furthering enterprise Linux innovation" wasn't their goal for CentOS.

Now what?

Most users understand that Red Hat was never under any obligation to maintain a stable platform for them while asking nothing in return. But many of them had made commitments based on the promise of support for CentOS 8 into 2029; Red Hat's decision to back away from that promise feels like a betrayal of trust. Had Red Hat left the CentOS 8 schedule unchanged but announced that there would be no CentOS 9 release, the level of complaining would have been much lower.

To many, the decision also makes a mockery of the allegedly open nature of CentOS project governance. This decision was handed down from a Red Hat corporate office and the CentOS board had, as this message from longtime CentOS contributor and board member Johnny Hughes makes clear, no real prospect of changing it. The nature and goals of the CentOS project (as seen by its users) have been fundamentally changed by corporate fiat, and the "community" had almost no say in the matter.

So now CentOS 8 users have one year in which to implement any changes they will make in response to the end of support. (Note that the CentOS 7 support schedule remains unchanged, with support going through June 2024). Some may find that CentOS Stream works well enough for them in the end. It is, after all, unclear how unstable this distribution will actually be; presumably all packages added to it will have been through the RHEL quality-assurances processes. CentOS Stream could prove to be mostly a marketing artifact — a way of making CentOS look more dangerous to drive RHEL subscription sales without actually making CentOS worse.

Many users may not want to take that chance, though. Some may sign up for RHEL subscriptions, but most are likely to go looking for alternatives. There is a lot of talk about creating a new community-based, CentOS-like project to rebuild RHEL as before. These efforts might succeed, but it is worth remembering how CentOS struggled before it got proper financial support. Creating this kind of distribution involves a lot of tedious, detailed work, and volunteers to do that work will still be hard to find.

There are other existing community-based distributions that offer a long-term stable platform, of course. These include Debian, Ubuntu LTS, openSUSE Leap, and various others. Most of these are likely to be entirely suitable for a wide range of production deployments. Each has its own quirks, but so does CentOS.

An alternative that some may consider is to give up on the idea of an "enterprise" distribution altogether. These distributions were created in an era when one would buy an actual, physical computer, deploy it somewhere, and expect it to do its job for years. Such systems still exist, but it is increasingly common to just order up new virtual machines when there is a job to be done, and to discard them when they are no longer useful. The value offered by long-term-stable distributions in this new world is less clear. Many systems that are deployed on CentOS might actually be better off using distributions with near-mainline kernels and more current software.

It will take time for the dust to settle from this change, and Red Hat may have burned some community trust that will prove hard to get back. But the community should also remember that CentOS has been an immensely valuable gift from actors across the free-software community, with Red Hat playing a large part in its making and support. We may or may not like this change, but a "thank you" to Red Hat for its part in the creation of 15 years (so far) of CentOS is entirely appropriate.

Comments (204 posted)

Reducing page structures for huge pages

Kernel development is a constant exercise in reducing overhead; any resources taken by the kernel are not available for the workload that users actually want to run. As part of this, the page structure used to manage memory has been kept as small as possible. Even so, page structures typically take up just over 1.5% of the available memory, which is too much for some users. LWN recently looked at DMEMFS as one approach to reduce this overhead, but that is not the only work happening in this area. Two developers are currently working independently on patches to reduce the overhead associated with huge pages in particular.

Memory maps and tail pages

The aforementioned page structure holds most of what the kernel knows about every page of memory in the system: how it is used, its position on various lists, the location of its backing store, etc. There is, thus, one page structure for each physical page in the system; in common configurations, that means one 64-byte structure for every 4096-byte page.

In the early days of Linux, the kernel would allocate a simple array of page structures that was just large enough to represent the memory installed in the system; that worked because physical memory itself looked like a simple array of pages. Since then, though, life has become rather more complex for the memory-management subsystem. Nodes in NUMA systems have distinct ranges of memory with, possibly, large gaps between them. Memory can be plugged into a system (or removed from it) at run time. Virtualized guests can have memory injected into them (or removed) while they run as well. As a result, the simple, linear model of memory no longer works.

The kernel has used a few different models for the memory map over time; see this article for the full history. The preferred model in current times (for 64-bit systems) is called "sparsemem-vmemmap"; it uses the system's memory-management unit (MMU) to restore the illusion of a simple, linear map (called the "vmemmap"). Specifically, each architecture reserves a portion of the kernel address space for this map; the x86-64 architecture, for example, puts it at 0xffffea0000000000 when four-level page tables are in use. Whenever memory is added to the system (including memory "added" when the kernel discovers it at boot time), a suitable number of page structures is allocated and the set is mapped into the vmemmap. Discontiguous chunks of memory can thus be made to appear to be contiguous, simplifying a number of low-level management functions.

The end result looks vaguely like this:

On a system with 4096-byte pages and a 64-byte struct page, one page of memory will need to be allocated and mapped into the vmemmap array for every 64 page structures. Once that is done, the page structure for any given page can be found by using its page-frame number as an offset from vmemmap_base (on x86 systems).

Compound pages complicate the situation. A compound page is formed when a group of adjacent pages is grouped together into a larger unit. The most common use is for huge pages — larger page sizes implemented by the system's CPU and MMU. The x86-64 architecture, for example, implements 2MB and 1GB huge pages; there can be significant performance benefits from using them. Whenever a huge page is created from a set of single ("base") pages, the associated page structures are changed to reflect the compound page that they now represent.

The first base page in a compound page is called the "head" page, while all of the others are called "tail" pages. A 2MB huge page is thus made up of one head page and 511 tail pages. The page structure for the head page is marked as a compound page, and represents the whole set. The page structures for the tail pages, instead, contain only a pointer to the head page. (That is not quite true; the first couple tail pages have some metadata about the compound page, but that can be ignored here).

Thus, of the 512 page structures associated with a 2MB huge page, 511 are essentially identical copies of a sign saying "look over there instead". Those structures take up eight pages of memory in their own right, seven of which represent only tail pages and contain identical data.

Trimming wasted page structures

Both patch sets described here take the same approach to saving memory. The first out the gate was this set from Muchun Song, currently in its eighth revision. Song realized that there was no compelling reason to keep all of those pages full of identical page structures around, especially given the virtual mapping already used by the vmemmap mechanism. Consider a more compact version of the diagram above:

Here, one 2MB huge page is represented by those eight pages of page structures, almost all of which correspond to tail pages and just point to the structure for the head page.

In a kernel with Song's patch set applied, that structure is changed. Since seven of those eight pages all contain identical pages, they can be replaced with a single page instead; that one page can be mapped seven times to fill out the vmemmap array as before:

As far as the rest of the kernel is concerned, nothing has changed; the page structure for any given tail page looks the way it did before. But six pages of duplicated data can now be given back to the system for other uses for as long as the compound page continues to exist. In other words, 75% of the memory overhead for this compound page has just been eliminated.

The savings for 1GB huge pages are even more dramatic; 4094 of the 4096 pages representing tail pages can be eliminated. General-purpose systems tend not to use many 1GB huge pages, but there are situations, including some virtualization workloads, where they are useful. It's not surprising that the prospect of saving nearly 16MB of overhead for each 1GB huge page is attractive to hosting providers.

Huge pages do not necessarily remain huge forever; they can be returned to the kernel or split up for a number of reasons. When that happens, the full set of page structures must be restored. In Song's patch set, this work is deferred to a workqueue so that the necessary pages can be allocated in a relatively relaxed setting. This work adds some compute-time overhead to both the allocation and freeing of huge pages; Song included a set of benchmark results and concluded that this overhead is "not significant". There is also overhead added because the patch set disables the use of huge pages for the vmemmap itself; that will evidently be rectified if and when the core patch set is accepted.

The alternative patch set is from Joao Martins. It implements the same idea, eliminating most of the pages containing page structures for tail pages. While Song's patch set is focused on main memory, though, Martins's work is specifically for nonvolatile RAM. This memory is always brought online as a separate operation, so there is no need to free existing pages out of the vmemmap; instead, new devices are brought online formatted as huge pages from the beginning. That simplifies the code by eliminating all of the logic to change the vmemmap on the fly, but at the cost of reducing the applicability of the technique.

One incidental advantage of treating nonvolatile RAM this way, though, is that it eliminates the need to initialize large numbers of page structures when a new memory array is attached. That speeds the process of making that memory available to the system considerably. Use of huge pages also significantly accelerates the task of mapping this memory into kernel space, an operation that happens frequently when the DAX direct-access subsystem is in use.

Both patch sets appear to have advantages and disadvantages. The problem at this point is that the memory-management developers are highly unlikely to be interested in merging both of them. So either one will have to be chosen over the other, or the two patch sets will need to be somehow combined into one that meets everybody's needs. That sort of reconciliation can take time; the process of merging low-level memory-management trickery is not the fastest either. So this work is probably not arriving anytime in the near future, but in the longer term it may well lead to a lot of memory saved on Linux systems.

Comments (4 posted)

Statistics from the 5.10 kernel development cycle

Linus Torvalds released the 5.10 kernel on December 13 at the end of a typical nine-week development cycle. At that point, 16,174 non-merge changesets had been pulled into the mainline; that makes 5.10 a larger cycle than 5.9, but it falls just short of the record set by 5.8, which ended with 16,308 changesets. For the most part 5.10 is just another routine kernel release, but there are a couple of interesting things to be seen in the overall statistics.

A total of 1,971 developers contributed to 5.10 — again, just short of the record set by 5.8. Of those developers, 252 (just under 13%) made their first contribution in 5.10; that is the lowest number seen since 5.6. The most active 5.10 developers were:

Most active 5.10 developers

By changesets Krzysztof Kozlowski 491 3.0% Mauro Carvalho Chehab 378 2.3% Christoph Hellwig 265 1.6% Pierre-Louis Bossart 116 0.7% Lee Jones 116 0.7% Randy Dunlap 115 0.7% Allen Pais 112 0.7% Jonathan Cameron 107 0.7% Maxime Ripard 103 0.6% Dave Airlie 98 0.6% Lad Prabhakar 97 0.6% Andy Shevchenko 87 0.5% Chris Wilson 85 0.5% Evan Quan 84 0.5% Colin Ian King 84 0.5% Andrii Nakryiko 82 0.5% Vladimir Oltean 80 0.5% Alex Deucher 79 0.5% Qinglang Miao 77 0.5% Kees Cook 70 0.4%

By changed lines Sudeep Dutt 26779 3.5% Mauro Carvalho Chehab 22741 2.9% Corentin Labbe 19091 2.5% Fabio Estevam 16757 2.2% Christoph Hellwig 14286 1.9% Cezary Rojewski 14106 1.8% Chandan Uddaraju 9357 1.2% Daniel W. S. Almeida 8012 1.0% Mike Travis 7873 1.0% Andrii Nakryiko 7455 1.0% Oded Gabbay 6989 0.9% Hans Verkuil 6839 0.9% Larry Finger 6758 0.9% Vadym Kochan 6382 0.8% Krzysztof Kozlowski 6371 0.8% Mauro Rossi 6227 0.8% Jonathan Marek 6106 0.8% Marc Kleine-Budde 6049 0.8% Jin Yao 5811 0.8% Jiaxin Yu 5464 0.7%

The author contributing the most changesets to 5.10 was Krzysztof Kozlowski, who made cleanups and small improvements all over the Arm and driver subsystems — at a rate of almost eight per day, seven days per week. Mauro Carvalho Chehab did a lot of work across the media, documentation, and staging subsystems. Christoph Hellwig's work included significant rewrites across the filesystem and block layers, along with the set_fs() removal. Pierre-Louis Bossart did a lot of work in the sound subsystem, and Lee Jones contributed a large number of warning fixes.

Sudeep Dutt contributed a single patch to 5.10, but that patch removed the drivers for Intel "many integrated core" (MIC) devices, deleting enough code to land at the top of the "lines changed" column. Corentin Labbe resurrected the Zoran MJPEG capture driver in the staging tree, and Fabio Estevam removed a bunch of old Arm board files.

The busiest testers and bug reporters this time were:

Test and report credits in 5.10
Tested-by Hoegeun Kwon 86 7.3% Chanwoo Choi 79 6.7% Stefan Wahren 79 6.7% Thierry Reding 33 2.8% Aaron Brown 31 2.6% Arnaldo Carvalho de Melo 24 2.0% Nikolay Borisov 23 1.9% Nick Desaulniers 21 1.8% Yoshihiro Shimoda 16 1.4% Jonas Karlman 16 1.4% Douglas Gilbert 16 1.4% Marek Szyprowski 15 1.3% Srinivas Kandagatla 13 1.1% Sedat Dilek 13 1.1% Daniel Thompson 12 1.0%	Reported-by kernel test robot 191 15.1% Hulk Robot 190 15.0% Syzbot 97 7.7% Dan Carpenter 40 3.2% Stephen Rothwell 23 1.8% Randy Dunlap 20 1.6% Qian Cai 17 1.3% Naresh Kamboju 14 1.1% Julien Grall 8 0.6% Alexei Starovoitov 7 0.6% Rob Herring 7 0.6% Marek Szyprowski 7 0.6% Colin Ian King 7 0.6% Geert Uytterhoeven 6 0.5% Lars-Peter Clausen 6 0.5%

The top three testers show an interesting pattern: their Tested-by tags all appear together on the same patches, all targeting the vc4 DRM driver. On the report side, we see that nearly 38% of all credited bug reports come from automated testing systems. Note that, since these tags appear in patches, they indicate reports that actually resulted in some sort of fix; that is a lot of bugs that won't be around to affect users later on.

A total of 228 companies (that we know of) supported work on the 5.10 kernel, an increase relative to recent past releases. The companies supporting the most work were:

Most active 5.10 employers

By changesets Huawei Technologies 1434 8.9% Intel 1297 8.0% (Unknown) 1075 6.6% (None) 954 5.9% Red Hat 915 5.7% Google 848 5.2% AMD 698 4.3% Linaro 670 4.1% Samsung 570 3.5% IBM 521 3.2% NXP Semiconductors 439 2.7% Facebook 422 2.6% Oracle 414 2.6% SUSE 410 2.5% (Consultant) 404 2.5% Code Aurora Forum 313 1.9% Arm 307 1.9% Renesas Electronics 283 1.7% NVIDIA 262 1.6% Texas Instruments 218 1.3%

By lines changed Intel 96976 12.6% Huawei Technologies 41049 5.3% (Unknown) 40948 5.3% Google 39160 5.1% NXP Semiconductors 35898 4.7% (None) 30998 4.0% Red Hat 30467 3.9% Code Aurora Forum 29615 3.8% Linaro 29384 3.8% Facebook 27479 3.6% BayLibre 24159 3.1% AMD 23343 3.0% (Consultant) 19905 2.6% IBM 18312 2.4% MediaTek 15893 2.1% Arm 13390 1.7% Texas Instruments 11814 1.5% SUSE 11063 1.4% Oracle 10542 1.4% NVIDIA 10481 1.4%

The presence of Huawei at the top of the "by changesets" column may be a bit of a surprise, though something similar happened in 5.8. As was the case then, Chehab's work obviously helped to drive that number, but it was also the result of 94 other developers working for Huawei who contributed at least one patch to 5.10. Huawei has built up a significant kernel-development operation. Beyond that, these results are mostly as one would expect.

Another difference in this cycle can be seen by looking at the non-author signoffs in the merged commits. Applying a Signed-off-by tag to a patch that one did not write is usually done by maintainers who are applying patches to be sent upstream; looking at these signoffs thus give an indication of who the gatekeepers to the kernel are. For 5.10, the results look like this:

Non-author signoffs in 5.10

Developers David S. Miller 1074 7.0% Greg Kroah-Hartman 780 5.1% Mark Brown 707 4.6% Alex Deucher 609 4.0% Jakub Kicinski 486 3.2% Mauro Carvalho Chehab 485 3.1% Kalle Valo 453 2.9% Andrew Morton 423 2.7% Jens Axboe 329 2.1% Alexei Starovoitov 317 2.1% Hans Verkuil 315 2.0% Martin K. Petersen 289 1.9% Michael Ellerman 245 1.6% Vinod Koul 239 1.6% Shawn Guo 197 1.3% Paolo Bonzini 196 1.3% Borislav Petkov 187 1.2% David Sterba 176 1.1% Herbert Xu 175 1.1% Will Deacon 172 1.1%

Employers Red Hat 2198 14.3% Linaro 1588 10.3% Facebook 1214 7.9% Intel 1143 7.4% Google 1082 7.0% Linux Foundation 863 5.6% Huawei Technologies 788 5.1% SUSE 710 4.6% AMD 642 4.2% Code Aurora Forum 508 3.3% IBM 418 2.7% Oracle 407 2.6% (None) 392 2.5% NVIDIA 364 2.4% Cisco 315 2.0% Arm 296 1.9% Qualcomm 239 1.6% (Consultant) 221 1.4% Texas Instruments 187 1.2% Samsung 138 0.9%

Seeing David Miller, the maintainer of the networking subsystem, at the top of this list is traditional, and he occupies this position for 5.10 as well. Miller, unfortunately, abruptly dropped out of the kernel community just after the 5.9 release due to a health issue, so this table reflects his work done prior to the 5.10 merge window. After that, Jakub Kicinski took over networking maintenance, which explains his appearance on the list (and Facebook's relatively high position). This change will be more strongly felt in the 5.11 release; Kicinski currently has signed off nearly 900 patches in linux-next.

Happily, Miller is recovering and has started applying networking patches again. Meanwhile, this episode turned into an unplanned test of the community's response when one of its most active maintainers is no longer able to do that work. That response appears to have gone well, with the flow of networking patches into the mainline and linux-next continuing at a strong pace. There appears to have been little disruption in the networking community overall.

It would be nice to believe that all important kernel subsystems are as well prepared for an abrupt maintainer change, but that is almost certainly not the case. So the kernel community might not pass the next test in such good form. That said, the numbers this time around show that kernel development is still going strong; kernels are released on a predictable schedule, participation across the community is high, and there are still numerous new developers making their debut in each release. 2020 was a difficult year, but the kernel community has gotten through it with, it seems, relatively little trouble.

Comments (10 posted)

A new release for GNU Octave

On November 26, version 6.1 of GNU Octave, a language and environment for numerical computing, was released. There are several new features and enhancements in this release, including improvements to graphics output, better communication with web services, and over 40 new functions. We will take a look at where Octave fits into the landscape of numerical tools for scientists and engineers, and recount some of its long history.

What is GNU Octave?

In the words of its manual:

Octave is free software distributed under the GPLv3. The program was first publicly released in 1993; it began as a teaching tool for students in a chemical engineering class. The professors, James B. Rawlings and John G. Ekerdt, tried to have the students use Fortran, but found that they were spending too much time trying to get their programs to compile and run instead of working on the actual substance of their assignments. So the professors created early versions of Octave as an easier alternative. The name is a tribute to the late Octave Levenspiel, a former professor of Octave's principal original author, John W. Eaton. Octave became part of the GNU project in 1997.

To understand Octave's subsequent development requires some awareness of another tool, called MATLAB, which was also created as a Fortran alternative. MATLAB has an interesting origin story: it was created to enable students to use Fortran's linear algebra routines without having to write Fortran programs. Its authors later turned it into a commercial product, which became quite successful. Now this powerful (and expensive) system is in use by millions of engineers and scientists all over the world. This created a market of people doing numerical computing who became used to MATLAB's syntax, but lost access to it after leaving their employers or institutions; beyond that, some people were also interested in free-software alternatives.

There are now several options, at different levels of compatibility. Octave, written in C, C++, and Fortran, soon adopted the goal and policy of being a fully compatible replacement for MATLAB. According to the Octave Wiki, any differences between Octave and MATLAB are considered to be bugs, "in general", and most existing MATLAB scripts will work unmodified when fed to Octave, and vice versa. The exceptions are scripts that exploit extensions to the syntax introduced in Octave, or that use recent features in MATLAB that have not yet been implemented in Octave.

Generations of engineers were raised with MATLAB, and its tendrils extend far, exerting its influence even over the recently developed language Julia, which I wrote about here recently. MATLAB and Octave deliberately confine their attention to numerics, with a particular focus on matrix manipulations, whereas Julia is a true general-purpose language. But Julia, like MATLAB, indexes arrays starting from one rather than zero, and its syntax for dealing with arrays is partly designed to provide the comfort of familiarity to refugees from MATLAB, without attempting to be a clone of the earlier language.

The latest version of Octave is available as a package in Debian unstable, but users of more conservative distributions may have to build the latest release. For example, the latest version available for Ubuntu is 5.2 (released in January 2020). Octave is a large system with many dependencies, and users may desire an alternative to compiling that still gets them a recent version. This page provides a detailed rundown of various methods of installation for several Linux distributions, as well as a detailed guide to compiling Octave. Octave is available for Windows and macOS, and there are packages for both FreeBSD and OpenBSD as well.

Using GNU Octave

When octave is started in the terminal it brings up an interactive prompt. The user can type in expressions, and the results are printed immediately. In other words, operation is similar to gnuplot, Maxima, Julia, and other programs that offer a read-eval-print loop (REPL) style of interaction. The REPL provides tab-completion of function and variable names that works similarly to Bash tab-completion on the command line. It also offers the usual command history and editing through readline.

If one adds the --gui flag when invoking Octave, a graphical environment speedily appears on the screen, instead of a simple REPL prompt. This is replete with the customary inscrutable array of icons whose purpose is revealed in tooltips when they are hovered over, and the standard hierarchical menu bar at the top. One of these menu items promises "help", leading to the inbuilt copy of the manual, as illustrated below. Having a complete, hyperlinked manual locally available is convenient. It allows searching through an index of functions and the content of any particular page, but not through the entire text of the manual.

Although the manual is quite thorough, it concerns itself almost entirely with the Octave language, as it might be used from the Octave REPL or when writing scripts in any text editor. It contains no help for the GUI that has shipped with Octave since version 3.8 as an experimental feature, and fully supported in 4.0.0; neither how to use it or what, if any, additional powers it affords the user. There is also nothing significant in the Octave Wiki.

Fortunately, everything that the GUI offers can be discovered with a bit of experimentation. In the following illustrations, I've reduced the size of the GUI window considerably in order to produce more compact screen shots; in normal operation, the user has much more room. I'm going to use the Octave GUI to solve the following system of equations:

    x + 2y + 3z = 2
    3x + 2y + z = 2
    2x + 2y + 2z = 2

The layout of the environment can be seen below:

On the left is a panel containing three small segments. From top to bottom, these are the file browser; the workspace, showing a list of all currently defined variables, and some information about their types, dimensions, etc.; and the command history.

On the right is the main panel, which can contain different displays, controlled by the tabs along the bottom. The layout can be arranged to suit the taste of the user. Each panel can be resized, and any panel can be split off into a separate window.

The editor is where the user can write and change Octave scripts. It offers syntax highlighting and command completion. As the user types, scrollable boxes appear with all possible completions. The editor also provides a typical debugger, with the ability to set breakpoints and step through the script.

My little three-line script defines a 3x3 matrix of coefficients, stored in the variable a, the vector of the right hand side of the system, stored in b, and finally invokes the Octave function linsolve(), which is supposed to return the solution as a vector of three values for the three variables.

The script in the panel is run by selecting the "run" menu item or by clicking the gear with the yellow triangle. I've done this, and the command tryingOctave has appeared in the command history panel. This is the name I gave to my script. But where is the output?

A new variable, called ans, has appeared in the workspace panel, showing that something has happened. Switching to the command window tab, shown in the next figure, shows the output of each command in the script. This panel is a REPL that behaves the same as invoking Octave from the command line, including the usual readline functionality.

Above we can see the matrix of coefficients nicely printed, the vector of the right-hand side of the equation system, and the results, calculated by linsolve(), for the three variables.

If the variable editor panel is selected, a double-click on a variable in the workspace displays it in a spreadsheet-like layout as shown below. This provides random access to elements of arrays, which can be directly altered.

So it appears that the GUI offers significant functionality, belying its somewhat homely appearance; it amounts to an IDE for Octave.

Here is one more illustration, demonstrating Octave's integrated graphics and another of its built-in functions, fft(), which implements a fast Fourier transform (FFT). The Fourier transform decomposes a periodic function into a sum of sines and cosines of different frequencies, and the FFT is an algorithm for doing so quickly. Of course, we humans can perform Fourier transforms in real time in our heads—literally. The inner ear Fourier transforms sound and sends the different frequencies along separate nerve bundles to the brain.

To test this routine, I'll choose a function for analysis where the Fourier transform is known analytically: the square wave. In the first line of this script, I've defined one cycle of a square wave in the variable sw, by appending a vector of 64 1s to a vector of 64 0s, to create one period defined by 128 numbers. The second line calculates its FFT and stores it in tsw, which should now contain the spectrum: the amplitudes of the frequencies in the original function. For a real-valued function, the spectrum is stored in elements 2 through N/2 of the result, so we extract them from tsw and plot the result using the bar() function to create a bar graph. Note that Octave's indexing syntax follows MATLAB in using parentheses.

The result, showing the (un-normalized) spectrum, appears in a graph window. Octave has always used gnuplot for plotting, and plans to continue to include that as an option indefinitely. However, recent versions also include an OpenGL plotting backend, which is more integrated with some Octave GUI functions that, for example, can create GUI elements for user interaction with Octave scripts.

In these examples I've used just a couple of the huge number of built-in functions for Octave. In addition to these, there is an official repository containing a vast number of third-party packages covering almost any topic in numerical analysis that one can think of.

The new release

Version 6.1.0 is a major release with a long list of user-visible changes. Many of the changes are specifically for the purpose of improving MATLAB compatibility, including keeping up with changes to MATLAB behavior.

Some of the more interesting enhancements and changes include new functions to facilitate communicating with web services, for use in scripts that analyze data retrieved over the web. There are also several improvements to the numerical nitty-gritty, such as improved random-number generation and numerical integration. This release also brings bug fixes and improvements to graphing, including font smoothing, control over legends, drawing of axes, and placement of superscripts and subscripts in labels. There is also more complete integration of Unicode throughout the system. The package system has been improved, several functions that had been deprecated in the 4.x series have now been removed, and about 40 new ones added.

Getting help

Octave is quite well documented. The manual, in particular, is well done and complete. Octave has extensive help within the program itself, which is available by typing "help". In addition to the manual, there is an Octave Wiki that contains a FAQ and other useful information.

There are many books available about GNU Octave. These cover the range from general introductions to guides for researchers in specific disciplines. Many of these books feature both Octave and MATLAB, which is an indication of the functional interchangeability of the two systems. There are Discourse forums for general help, for Octave developers, and general community discussions, as well as the help-octave mailing list with archives that go back to 1992, and an #octave IRC channel. In addition, the project maintains a list of firms offering paid support and consulting in the use of Octave.

Advice

I think the appeal of Octave should be apparent from the two examples in the previous section. It is particularly noteworthy that the linear algebra and signal processing routines were ready to use; I didn't have to type any import statements or figure out where things were. This is one way that Octave beats both Python and Julia: so much is just ready to go.

I also didn't need to write any loops or create any dummy variables, due to Octave's powerful array syntax. Just a few succinct statements solved a non-trivial problem in both cases. And when I wanted a plot, I just asked for one. Again, no need to import libraries (or wait for them to pre-compile). A complete manual is at one's fingertips, and everything in the manual can be used immediately, with no fuss and no waiting. Octave provides an impressively efficient path to the solution to a numerical problem.

It is a great time to be a scientist or engineer with an interest in computation and a desire to use free software. There is a wide variety of mature and powerful software available, with options that are general in scope and others that are focused on particular disciplines. This pleasant situation may, however, create a paralysis of choice. Although the software is free, there is an investment of time and mental space in choosing any particular tool over the others; so one wants to choose rationally.

For those who are already trained in MATLAB and who seek a tool for numerical exploration and modeling, Octave is the obvious choice. For those who may need a more general-purpose language, or whose sights are set on performing large-scale simulations, the best choice for most people today is probably Julia. However, Python may appeal to those who want to work in certain areas, such as machine learning, where it is still the most popular language, and where there are highly developed libraries.

Comments (5 posted)