Kernel development

Kernel release status

The current development kernel is 4.6-rc7, released on May 8. Quoth Linus: "Nothing particularly scary, and the more people who test this out, the more confident we can be that the final 4.6 is all good. So please take a moment to try it out."

Stable updates: 4.5.4, 4.4.10, and 3.14.69 were released on May 11.

Comments (none posted)

Quotes of the week

— Al Viro

— Greg Kroah-Hartman

Comments (2 posted)

Some numbers from the 4.6 development cycle

With the 4.6-rc7 release on May 8, the 4.6 kernel is nearly ready for release. It wouldn't be a proper development cycle without a set of statistics showing where the code came from this time around. With the 4.6 kernel we have seen some old names return to the lists of the top contributors, but, for the most part, it looks like a typical — if busy — cycle.

The 4.6 merge window, with 12,172 non-merge commits, was the busiest in the history of the kernel project. By current appearances, though, 4.6 will not be setting the record for the busiest development cycle overall. The 13,354 commits merged up to 4.6-rc7 are a lot, but it seems unlikely that the total when 4.6-final comes out will exceed the 13,694 merged for 4.2, much less the 13,722 merged for 3.15. So the record set for 3.15 will, at the beginning of June, have held for a full two years.

Those commits were contributed by 1,661 developers, which is a new record, beating the 1,625 seen contributing to 4.3. 283 of those developers made their first commit to the kernel during this development cycle. The most active 4.6 developers were:

Most active 4.6 developers

By changesets Arnd Bergmann 204 1.5% Chaehyun Lim 197 1.5% Oleg Drokin 165 1.2% Jes Sorensen 131 1.0% Amitoj Kaur Chawla 117 0.9% Christian König 107 0.8% Dennis Dalessandro 100 0.7% Mauro Carvalho Chehab 97 0.7% Al Viro 93 0.7% Peter Hurley 91 0.7% Linus Walleij 86 0.6% Geert Uytterhoeven 86 0.6% Laxman Dewangan 85 0.6% Andy Lutomirski 84 0.6% Marc Zyngier 84 0.6% Laurent Pinchart 84 0.6% Namhyung Kim 81 0.6% Mike Rapoport 81 0.6% Bhaktipriya Shridhar 80 0.6% Tomi Valkeinen 78 0.6%

By changed lines Faisal Latif 28327 4.3% Dennis Dalessandro 16034 2.4% Mike Marshall 15434 2.3% Shraddha Barke 14126 2.1% Oleg Drokin 7935 1.2% Josh Poimboeuf 7917 1.2% Andrew Duggan 6640 1.0% John Crispin 5914 0.9% Tomi Valkeinen 5893 0.9% Archit Taneja 5704 0.9% Andrew F. Davis 5513 0.8% Jiri Pirko 5410 0.8% Santosh Shilimkar 5338 0.8% Alexandre Bounine 5191 0.8% Yuval Mintz 5054 0.8% James Simmons 4619 0.7% Laurent Pinchart 4534 0.7% Mauro Carvalho Chehab 4533 0.7% Sudeep Dutt 4387 0.7% Marc Zyngier 4230 0.6%

Longtime kernel developer Arnd Bergmann found his way to the top of the "by changesets" column mainly through his ongoing work to eliminate warnings from the ARM kernel builds. Chaehyun Lim continues to work on the wilc1000 staging driver, Oleg Drokin did a lot of cleanups in the Lustre filesystem, Jes Sorensen continues to clean up the rtl8xxxu staging driver, and Amitoj Kaur Chawla made cleanups throughout the driver subsystem.

On the "lines changed" side, Faisal Latif got to the top of the list with the addition of the i40iw InfiniBand driver. Dennis Dalessandro worked on various RDMA drivers, Mike Marshall added the OrangeFS filesystem, and Shraddha Barke removed some unloved drivers from the staging tree.

The number of employers seen supporting kernel work remains steady at just over 200; this count has not changed significantly since the early 3.x days. The most active employers in this cycle were:

Most active 4.6 employers

By changesets Intel 2009 15.0% (Unknown) 1358 10.2% Red Hat 1043 7.8% (None) 647 4.8% Linaro 588 4.4% Outreachy 413 3.1% Samsung 390 2.9% SUSE 364 2.7% Renesas Electronics 336 2.5% IBM 312 2.3% AMD 307 2.3% ARM 253 1.9% Google 247 1.8% (Consultant) 238 1.8% Texas Instruments 230 1.7% Oracle 191 1.4% Code Aurora Forum 175 1.3% Atmel 166 1.2% NVidia 159 1.2% Huawei Technologies 147 1.1%

By lines changed Intel 131992 20.0% (Unknown) 43271 6.5% Red Hat 40589 6.1% (None) 29098 4.4% Linaro 19259 2.9% IBM 17455 2.6% Outreachy 17445 2.6% Omnibond Systems 17122 2.6% Texas Instruments 16779 2.5% Code Aurora Forum 14484 2.2% Renesas Electronics 14222 2.2% Google 14183 2.1% AMD 13581 2.1% SUSE 12304 1.9% Samsung 12107 1.8% Oracle 11441 1.7% Mellanox 10971 1.7% ARM 10811 1.6% Facebook 9642 1.5% (Consultant) 8086 1.2%

As usual, little has changed in this table. Lots of cleanup patches made their way into the staging tree as part of the application process for Outreachy internships; this time around, some of the Outreachy interns are employing Coccinelle scripts and increasing their productivity accordingly. One name that might be unfamiliar is Omnibond Systems — the company behind OrangeFS. Otherwise, it's mostly the names one usually sees in this place.

Reviewing Reviewed-by

Finally, let's take a quick look at Reviewed-by counts. A developer can offer a Reviewed-by tag after having reviewed a patch in depth; these tags are considered a sign that the patches in question have been closely examined and are of high quality. They are also there to credit the developers who spend time reviewing code — an important objective, since review bandwidth is one of the community's most limited resources. In the 4.6 development kernel, a total of 3,645 Reviewed-by tags were applied to patches; the most prolific sources of them were:

Most active 4.6 reviewers
Alex Deucher	155	4.3%
Mike Marciniszyn	141	3.9%
Ira Weiny	133	3.6%
Dennis Dalessandro	130	3.6%
Christoph Hellwig	98	2.7%
Hannes Reinecke	91	2.5%
Johannes Thumshirn	82	2.2%
Oleg Drokin	77	2.1%
Daniel Vetter	72	2.0%
Ville Syrjälä	68	1.9%
Krzysztof Kozlowski	64	1.8%
Christoffer Dall	54	1.5%
Christian König	53	1.5%
Thomas Gleixner	49	1.3%
Chris Wilson	47	1.3%
Dean Luick	44	1.2%
Maarten Lankhorst	41	1.1%
James Simmons	40	1.1%
Chunming Zhou	37	1.0%
Laurent Pinchart	36	1.0%

This table makes it clear that different developers have different views of how Reviewed-by should be used. The top reviewer is Alex Deucher, the maintainer for the AMD graphics drivers. Alex certainly reviews a lot of patches; in this case, though, most of the patches with his Reviewed-by tags also carried his Signed-off-by tags — applied when he accepted the patches for merging. It is normally understood that maintainers should review patches before applying them; most maintainers do not add a Reviewed-by tag for that work, but Alex does.

Out of the next three reviewers on the list, Mike Marciniszyn, Ira Weiny, and Dennis Dalessandro, only Mike appears in the MAINTAINERS file. They do have something in common, though, in that they all work for Intel. Patches generally come out of Intel with Reviewed-by tags from these developers (among others), often all three together, applied before the code has been seen by the rest of the world. We are clearly seeing the results of some sort of internal Intel review policy at work. There is no real way to know how thorough this review process is, other than to observe that Intel contributes a lot of patches and problems are rare.

The fifth entry in the list, Christoph Hellwig, typifies the sort of review that was envisioned when this tag was created. Nobody who has had Christoph's attention applied to a patch of theirs will have missed the fact that a serious review has taken place. The work of the other four reviewers named above may well be just as thorough and just as valuable, but, since it's not public review, there is no way to know for sure.

Of course, almost all of the review work that does take place in public still does not result in a Reviewed-by tag being applied to the patches involved. So it could be argued that this tag has never really achieved its purpose of documenting and crediting the review work that is being done.

The overall picture created by these numbers, though, is one of a development process that continues to function like a relatively well-tuned machine. The number of contributors continues to increase, the patch flow is steady, and there do not appear to be many process-scalability issues in sight.

Comments (4 posted)

Transparent huge pages in the page cache

As was discussed at the recent Linux Storage, Filesystem, and Memory-Management Summit, there is interest in adding transparent huge page (THP) support to filesystems, particularly tmpfs but ultimately others as well. Huge pages are important when mapping large files into a process's virtual memory as they reduce the per-access overhead. Supporting huge pages transparently is essential as the tradeoffs between huge and non-huge pages depend on system-level parameters such as the amount of free memory and its level of fragmentation. Individual applications are not able to assess these parameters and so cannot be expected to make appropriate decisions.

The level of interest in THP for filesystems is such that two competing patch sets have emerged that provide similar functionality with quite different approaches. The discussions at the summit were largely about engineering issues, including amount of testing, general maintainability, and level of intrusiveness into other parts of the kernel. While these are appropriate for that forum, they can leave the technically minded wondering what the deep technical differences are, what these new "team pages" are, and whether they are — or are not — a good idea.

Before delving into those differences, it will be useful to understand how the requirements for THP support in filesystems differ from those for the existing THP support for anonymous memory. The different requirements primarily come from the fact that files can be accessed concurrently and in different sized units, and particularly how this can cause the file to grow.

Anonymous memory is only accessed by memory mapping (i.e. with mmap()) and the size of this mapping is usually fixed on allocation. Sharing between processes only happens as the result of a fork() call and, for the process-private mappings that support THP, huge pages will only remain shared as long as they are unchanged. So every mapping of an anonymous transparent huge page will be the same size.

Memory used for file-backed data can be accessed by direct reads and writes and can be mapped concurrently by multiple processes, which may request differently sized mappings. Preventing one process from getting a huge mapping because some other process has only mapped one or two (small) pages is not nearly so acceptable as it might be with anonymous memory. For this reason, THP for filesystems must be able to support concurrent mapping as both huge and non-huge pages.

Unlike anonymous memory, space in files isn't allocated by mapping it: that happens as the result of write() or fallocate() calls. The allocated size may not be a multiple of the huge-page size, and the space allocated may not even be contiguous if holes are left. If huge pages were only used once a file was large enough to completely fill them, then a file that was not initially allocated to its final size would need to be migrated from individual pages to huge pages before a huge-page mapping would be possible. This would often be unnecessary work. So THP for files must allocate huge pages before the file is known to be big enough to fully utilize them.

Finally, a file may be used without being mapped into process memory at all, while anonymous memory is always mapped. So any changes to a filesystem to support transparent huge page mapping must not negatively impact normal read/write performance on an unmapped file.

The two patch sets approach these challenges from two different directions. One seems to say "we know how to do filesystems, let's build on that to enable huge mappings". The other starts with "we know how to do huge mappings" and proceeds to "how can we adjust those to work with filesystems?" Both patch sets support only tmpfs. This allows for valuable simplifications but means they cannot easily be assessed as fully general solutions.

Page teams: extending filesystems toward huge pages

The first patch set is in use at Google and is being championed by Hugh Dickins. It starts with current filesystem use of the page cache and asks "how can we encourage huge page mappings?". This involves allocating a "high order" physically contiguous set of pages when possible, annotating them so that the memory-management code can easily detect that a huge-page mapping will work, and allowing the set to be broken into individual pages when, but only when, necessary.

The allocation step is relatively straightforward: when allocating memory in the page cache, if there are not currently any allocated pages in the region that a huge page would fill, request a huge page. If that works, good. If not, just fall back to individual small pages.

The annotation step is where the term "team pages" comes in; the large allocation is referred to as a "team" or "team of pages." The first page of the high-order set of pages, and every page in that set that has been instantiated in the file, gets a new page flag: PG_team. This makes it easy, when looking at a page, to know if it is part of a larger team allocation. By itself this is not quite enough to allow a huge page to be mapped since, when that mapping happens, any "holes" within that page must be instantiated as they could be written to. So every small page must get PG_team set. This instantiation happens the first time that a huge mapping is attempted, and an extra page flag — PG_checked — is set on the head page. From then on, a huge mapping can be created after just checking for PG_team and PG_checked.

Finally, it must be possible to break up the team when free memory is short. There are two sides to this. If a team has never been mapped as a huge page, it may have some unallocated pages (PG_team not set) that are invisible to memory management (as they are not part of the file), but must be made visible so they can be released when memory pressure is high. Pages in the team that are part of the file will be tracked on an LRU (least recently used) list so that the oldest pages can be written out to swap should that become necessary. Either of these will require disbanding the team before individual pages are written or freed.

Unallocated pages are marked in the file's radix tree using a tag, SHMEM_TAG_HUGEHOLE, which reuses the PAGECACHE_TAG_DIRTY tag used by normal filesystems. All files with "huge holes" are kept in a new LRU list that is scanned by a shrinker when memory reclaim tries to recover memory. The basic approach of this shrinker is, quoting Dickins, to "find [the] least occupied huge page in [the] older half of shrinklist, and migrate its cachepages into the most occupied huge page with enough space to fit, again chosen from [the] older half of shrinklist."

Rather than just freeing some scattered pages, it attempts to free a whole huge page. This generally helps reduce fragmentation of free space, so there are more likely to be huge pages free the next time one is needed.

When dismantling a team of pages that has no holes, it is necessary to get a page lock on the head page first in order to avoid racing with an attempt to map the whole team into an address space. Consequently, attempting to disband a team when a tail page has reached the top of the LRU list is problematic since the reclaim process is not a good time to lock some other page. The page-teams patch set addresses this by refusing to write a tail page to swap before the head page has been written. When the head page is processed, it disbands the team, writes out that head, and thus frees all the tail pages to be written when their time comes. As pages tend to be written out in the same order as in the file, this usually works quite well.

When it doesn't work, though, the very large number of pages that refuse to be evicted (up to 511 for every 512) can distort the active and inactive page counts that are used to guide page eviction. To address this, slightly creative accounting is used to not count the tail pages, but instead to count the head pages as though they were multiple pages. This means that, as long as the head page is active, the whole team is accounted as active, and the memory reclaim code will be keen to transition more pages from active to inactive so they can then be freed.

While this approach seems to have minimal impact on filesystems by allowing them to continue to use pages in the page cache much as they already do, there are problems. As already noted, the PAGECACHE_TAG_DIRTY tag is being reused to record where in the radix tree there are holes that can be reclaimed. This is not a problem for tmpfs as its pages don't have a distinction between "clean" and "dirty". Anything that is in memory can be written to swap and then removed from memory, it need never be marked "clean". It is a problem for most other filesystems, though, so some other way of finding "huge holes" would need to be found.

Another conflict is that, to achieve the creative accounting mentioned, the head page holds a count of the number of allocated tail pages. This number is stored in the same place in the page description structure that most filesystems use to store a pointer to their own private data. This is not a problem for tmpfs, which doesn't use the private field, but would be for other filesystems. And then there is that fact that the head page of a team imposes a particular meaning on the PG_checked flag, which would interfere with file systems that use that flag themselves.

It is quite possible that these issues can all be addressed with sufficient ingenuity. The patch set is deliberately aimed at supporting only tmpfs, not on general filesystem support, to keep the complexity manageable. But they are issues to keep in mind when considering the long term future of THP support in the page cache.

THP-enabled tmpfs using compound pages

The second patch set under consideration comes from Kirill Shutemov. It starts with the premise that the memory-management code already knows how to map compound pages into a process's address space, because they are used for anonymous THP and for files in the hugetlbfs filesystem. It tries to make such compound pages (collections of pages that largely behave like a single page) usable by a filesystem, particularly tmpfs.

Having individual compound pages, rather than teams of small pages, means that the active/inactive LRUs just see the one page rather than multiple related pages on different LRUs, so the creative accounting needed for page teams is irrelevant here. However, when memory is mapped with MAP_LOCKED, it must be accounted as "unevictable" rather than active or inactive. If just part of a compound page is mapped and locked, some extra care is needed, once again, to get that accounting right.

Unlike the page team approach, compound pages provide no obvious mechanism for keeping track of the free space that might be at the end of a compound page after the end of the file. The current patch set doesn't address this at all, treating that space as in-use until the compound page is disbanded for some other reason, typically to write it out to swap. There are suggestions that this will be addressed in a future patch set. Similarly this patch set makes no effort to support page-sized holes within a compound page.

Processes expect to be able to map individual small pages. If the region they want to map is within a compound page, this becomes more complex: it is necessary to find the compound page that contains the target region and then map part of that. The current approach for finding that compound page stores all the small pages within the compound page in the page-cache radix tree. This is similar to how page teams operate, but doesn't make best use of having a single compound page. This solution is expected only to be interim. Matthew Wilcox, one of the DAX developers, has some patches that enhance the radix tree to be able to store a single entry covering a large range of addresses. This will help with DAX, which uses huge, and even larger, pages and would help with mapping compound pages too.

Allowing, eventually, only one radix tree entry per huge page is a real benefit, and there are other related benefits. Any operation that works a page at a time could fairly easily operate on a huge page at a time, thus reducing per-page overhead. Andrea Arcangeli, architect of the original huge-page support in Linux, sees this as a key issue, noting that:

In terms of compatibility with other filesystems, it is quite clear that a traditional filesystem would need a lot of change to work with compound pages. A filesystem will typically attach some private data to each page, possibly a list of buffer_heads describing the filesystem blocks in that page. With compound pages that private data structure will now have to describe a much larger range of blocks. A linear search as currently used would no longer be appropriate. On the positive side, at least there is somewhere to store that private data, which is not currently the case with team pages.

Six of one, half a dozen of the other

The two approaches are clearly different and they both have costs and benefits. Probably the two key questions are: which provides the lowest overheads for large scale operations, and which integrates more easily with other filesystems. Neither patch set is really sufficiently mature to provide answers to those questions and it wouldn't be at all surprising if each question is answered best by a different approach. Continuing on without a clear decision does not seem likely to be in the best interests of Linux in the short term, but making one now does not seem at all easy. I'm certainly glad that it isn't my responsibility to make that call.

Comments (3 posted)

Two approaches to x86 memory encryption

Techniques for hardening the security of running systems often focus on access to memory. An attacker who can write (or even read) arbitrary memory regions will be able to take over the system in short order; even the ability to access small regions of memory can often be exploited. One possible defensive technique would be to encrypt the contents of memory so that an attacker can do nothing useful with it, even if access is somehow gained; this type of encryption clearly requires hardware support. Both Intel and AMD are introducing such support in their processors, and patches to enable that support have been posted for consideration; the two manufacturers have taken somewhat different approaches to the problem, though.

Intel's Software Guard Extensions

Intel's offering is called "Software Guard Extensions," or SGX; details can be found on the SGX web page. SGX is built around the idea of creating "enclaves" of protected code and data. One or more ranges of physical memory are set aside as the "enclave page cache"; the contents of that memory (whether data or code) are only accessible to code that is, itself, located within the enclave. That code is callable from outside the enclave, but only via a set of entry points defined when the enclave is set up.

Memory within the enclave is encrypted using an engine built into the processor itself; the key that is used is generated at power-on and is not available to any running code. As a result, according to Intel's page, the contents of the enclave are "protected even when the BIOS, VMM, OS, and drivers are compromised". Those will certainly be appealing words for anybody who has despaired of ever preventing the compromise of any of those components.

This mechanism seems to be aimed at protecting relatively small ranges of memory; its overhead is apparently too high to do more than that. So, for example, one might load a private key and the code to sign data with that key into an enclave. Thereafter, it will be possible for an application to use the key to create signatures, but nobody can gain access to the key itself, even if the kernel itself is compromised.

The SGX patch set, posted by Jarkko Sakkinen, creates a new device (/dev/sgx) which supports a number of ioctl() calls to control the feature. Interestingly, there are no capability checks on the ioctl() calls themselves; anybody who can open the device can set up enclaves — and the default permissions allow access to everybody. A new enclave is created with SGX_IOCTL_ENCLAVE_CREATE, and pages of data are added to it with SGX_IOCTL_ENCLAVE_ADD_PAGE. Things are then made ready to run with SGX_IOCTL_ENCLAVE_INIT. That last operation requires passing in an initialization token containing a hash of the enclave data and an appropriate signature. There does not appear to be a way to delete an enclave once it has been established.

The "appropriate signature" part turns out to be a bit of a sticking point, in that said signature must come from Intel itself. In other words, it is not currently possible for the owner of a system to set up and use an enclave without getting Intel's approval and agreeing to a set of conditions. That means, as Ingo Molnar put it, that "it only allows the execution of externally blessed static binary blobs". Needless to say, there is no shortage of opposition to that idea in the kernel community; Ingo went on to say:

The discussion on the list suggests that Intel does plan to eventually make the feature work without the need for third-party signatures, but it is not clear when that might happen. Meanwhile, there is another roadblock, in that the patches do not actually work — one cannot actually run code in a protected enclave under Linux. Instead, enclaves can only run in the "debug mode," where it's possible to read and manipulate data inside the enclave from the rest of the system. That, obviously, detracts from the utility of the feature. It's not entirely clear why this limitation is in place.

Jarkko had wanted to place the code into the staging tree, where developers could play with it until it was actually made to work, but there is no visible community interest in going along with that plan. So the SGX patches are going to have to wait for some time until (1) they actually work, and (2) the hardware can support enclaves signed by arbitrary parties.

Secure Memory Encryption

AMD's technology is called "Secure Memory Encryption" (or SME); a description can be found in this white paper [PDF]. The patch set from Tom Lendacky adding support for SME came out, by some coincidence, one day after the SGX patches were posted.

SME is, in a sense, a simpler mechanism. Rather than establishing enclaves, a system with SME simply marks a range of memory (even all of memory) for encryption by setting a bit in the relevant page-table entries. The memory controller will then encrypt all data stored to those pages using a key generated at power-on time; all data read from the range will be transparently decrypted. No code running on the processor (not even the kernel) has access to the encryption key. Enabling encryption is said to slightly increase memory latency, but the white paper suggests that the performance impact will normally be quite small.

The SME approach, thus, will not protect memory from an attacker who has compromised the kernel; from the point of view of a running system, the memory might as well not be encrypted at all. Instead, it is intended to protect against cold-boot attacks, snooping on the memory bus, and the disclosure of transient data stored in persistent-memory arrays.

SME can be used as the base for another feature, though, called "Secure Encrypted Virtualization" (SEV), where each virtual machine gets its own key. At that point, the value of the feature will, if it functions as advertised, be significantly higher; it can protect virtual machines from each other, keeping their contents secure even if one of them manages to compromise the host system. Indeed, it should be able to protect virtual machines from the hypervisor itself. In a world where everything seems to be moving toward virtual machines running on shared cloud infrastructure, this kind of protection would be a useful enhancement to the security of the cloud as a whole.

The current patch set only implements SME, though, leaving SEV for the future. If the system is booted with the mem_encrypt=on command-line parameter, encryption will be enabled for all of physical memory, with a few exceptions. Video memory, for example, should not be encrypted; the device tree loaded at boot, if any, will also need to be accessed in the clear. Beyond that, encrypted memory will be used throughout, with most of the system being entirely unaware that the feature is in use.

The SME patches do not have the signature-related issues that came up with SGX, but this feature was not universally acclaimed either. Andy Lutomirski detailed several ways that he thinks the feature could be broken before concluding: "But I guess it's better than nothing." Paolo Bonzini added that the SEV feature is "very limited unless you paravirtualize everything" and worried that it is being oversold as a general mechanism for the securing of virtual machines. He suggested that the work should maybe not be merged in its current form.

Tom acknowledged that the technology has limitations in its current form, saying:

There have been no statements from the x86 maintainers regarding whether the SME patches would be merged, but it would not be surprising if their approach were similar to the one they have taken with the SGX patches: they will be seriously considered when the full functionality is present and that "vision" has been implemented.

The end result of all this discussion is that we will probably not see support for either manufacturer's memory-encryption technology in a near-term kernel. But the direction that hardware development is taking offers some encouragement. Those of us who have despaired of ever truly securing our software may well be right; we need levels of defense that come into play when the software has failed. Done right, hardware-based defenses can come to the rescue here without taking away our power to secure and control our own systems. Once the hardware reaches that point, Linux will certainly be able to take advantage of those capabilities.

Comments (24 posted)