LWN Weekly Edition
Front pageSecurity
Kernel development
Distributions
Development
Linux in the news
Announcements
->One big page
This page
Previous weekFollowing week
| |||||||||||||
Kernel developmentBrief items Kernel release statusThe 2.6.29 merge window is open, so there is no development kernel release as of this writing. Quite a bit of work has been merged for 2.6.29; see the separate article below for details.The current stable 2.6 kernel is 2.6.28, released by Linus on December 24. Some of the highlights of this kernel are the addition of the GEM GPU memory manager, the ext4 filesystem is no longer "experimental", scalability improvements in memory management via the reworked vmap() and pageout scalability patches, moving the -staging drivers into the mainline, and much more. See the excellent KernelNewbies summary for lots more details about 2.6.28. Says Linus: "In fact, even _if_ you have friends or family, leave them to their endless toil over that christmas ham or turkey, and during the night, when they're asleep, you can give them that magical present of a newly updated computer. When they wake up tomorrow morning, tell them how you saw Santa crawl down the chimney with his USB stick in hand, updating the OS of all good boys and girls."
Kernel development news Quotes of the week
The software design moral: Everything is shit and will attempt to
kill you when you're not looking.
-- Matthew Garrett
I don't believe "auto-destroy my music collection" is a sane
default.
-- Alan Cox
BTW, the current influx of higher-complexity filesystems certainly
worries me a little.
-- Christoph Hellwig
Can you post the patch, so that we can see if we can find some
silly error that we can ridicule you over?
-- Linus Torvalds (Thanks to Jeff
Schroeder)
There's a lot of stuff here, as can be seen by the final diffstat
number:
-- Greg Kroah-Hartman
779 files changed, 472695 insertions(+), 26479 deletions(-)and yes, it's all crap :)
I will just note wryly that it used to be that I could compile 0.9x
kernels on a 40 MHz 386 machine in 10 minutes. Some 15 years
later, it still takes roughly the same amount of time to compile a
kernel, even though computers have gotten vastly faster since then.
Something seems wrong with that....
-- Ted Ts'o
2.6.29 merge window, part 1As of this writing, some 6500 non-merge changesets have been accepted for the 2.6.29 development cycle. There is the usual set of new device drivers, combined with a number of important core kernel changes.As of this writing, user-visible changes include:
Changes visible to kernel developers include:
The merge window opened on December 28; if the usual two-week pattern holds, changes should be accepted through January 11. Tune in next week for an update on the final patches merged for the 2.6.29 kernel.
The future for grsecurityUsing an out-of-tree kernel patch has several downsides but, as long as the patch is maintained and updated with the kernel, it is workable. If the developers lose interest—or funding—it suddenly becomes a much bigger problem for users. That scenario may be about to play out for users of the grsecurity tool as a recent release comes with a warning that it could be the last. Users of grsecurity are, unsurprisingly, worried about the future of the security tool, but calls for its inclusion in the mainline are not likely to be successful. Over time, largely because of the efforts of others outside of the grsecurity project, various pieces of grsecurity (and the associated PaX project) have been added to the kernel. But, there are a number of reasons that the full grsecurity patch is not in the mainline; the most basic is that the developers seem unwilling or uninterested in following the normal path to inclusion. The grsecurity patch implements a number of security features that are useful, particularly for web servers or servers that provide shell access to untrusted users. One of the major features is role-based access control (RBAC), which is an alternative to the traditional UNIX discretionary access control (DAC) or the more recent mandatory access control (MAC) provided by SELinux and Smack. The aim of RBAC is create a "least privilege" system, where users and processes have only the minimum necessary privilege to accomplish their task. grsecurity also includes hardening of the chroot() system call, to eliminate privilege escalation and other vulnerabilities from within a "chroot jail". In addition, there are a number of other miscellaneous features like auditing and restricting /proc information, all of which are listed on the grsecurity features page. Another major component of grsecurity is the PaX code, which restricts memory use so that various exploits, such as buffer overflows and other code execution vulnerabilities, are blunted or eliminated. It does this by making data pages non-executable using—or emulating—the "no execute" (or NX) bit. PaX restricts mprotect() to not allow pages that are both writable and executable to avoid code injection as well. PaX also adds much more aggressive address space layout randomization (ASLR) than is currently used by Linux. PaX is developed separately from grsecurity, by the anonymous "PaX Team", then incorporated into grsecurity by developer Brad Spengler. The project has been around for a long time; grsecurity started in 2001, while PaX began in 2000. There are numerous satisfied users and grsecurity has been used in distributions such as NetSecL and Hardened Gentoo, but it has never made it into the mainline. Gabor Micsko recently posted a request on linux-kernel for Linus Torvalds to reconsider grsecurity:
The common opinion of the developers of grsecurity, PaX and their users
is that acceptance of the code into the kernel would be the best
solution for saving the project, beside finding another long-term
sponsor.
Torvalds replied that much of what was in grsecurity and PaX was "insane and very annoying and invasive code." He then went on to explain some of the history:
The apparent inability (and perhaps more importantly - total
unwilling[n]ess) from the PaX team to be able to see what makes sense in a
long-term general kernel and what does not, and split things up and try to
push the sensible things up (and know which things are too ugly or too
specialized to make sense), caused many PaX features to never be merged.
Much of it did get merged over the years (mostly because some people spent the time to separate things out), but no, we're not going to suddenly start merging code like that just because the project is in trouble. None of the basic issues have been solved. A perfect example of the unwillingness to work with the kernel hackers is embodied in the decision not to implement RBAC as a Linux Security Module (LSM). For better or worse, LSM is the mechanism used to implement access control in the kernel. Conceptually, it is a good fit for the grsecurity RBAC code. It might require additional LSM hooks, but working on getting those hooks added is the right approach. There was some uncertainty about LSM at one time, but it clearly is the way forward today. There may also be an issue with the PaX code, in that anonymous contributions to the kernel are not allowed. Presumably Spengler, or some other interested hacker, could sign off on that code, but it cannot be accepted directly from "PaX Team". To the extent grsecurity and PaX have been proposed for inclusion, they have always been presented as a single monolithic patch. There has never been an attempt to break the patch up into logical chunks that can be accepted or rejected on their individual merits. So far, that has not occurred even after the project lost its sponsor. But waiting until the last minute is not going to work. As Robert Hancock puts it:
Saying to the kernel developers "here, throw this huge blob of code into
your kernel because otherwise we're taking our ball and going home" is not
how it works.
If there is value in the existing code, interested users and developers need to work within the kernel process to get it accepted. To do that, one must identify the useful pieces and proceed from there. Valdis Kletnieks suggests:
Probably the best way to proceed would be for the stakeholders to come to some
agreement on which parts are the "sane stuff" (which could be an interesting
food fight), split those parts out, and submit them for inclusion as standalone
separate patches.
This is yet another example of the perils of out-of-tree code. By all accounts, there are satisfied grsecurity users who may well be left behind if the grsecurity project fails to find sponsors by the end of March. They can, of course, continue running the grsecurity-enhanced kernels they currently have, but may not be able to take advantage of upcoming kernel advances. Perhaps the stakeholders will gather together and continue updating grsecurity for newer kernels, but that still leaves the underlying problem. They would be better served spending at least part of their time working with the kernel hackers to get as much of grsecurity and PaX as possible merged into the mainline.
Btrfs aims for the mainlineThe Btrfs filesystem has been under development for the last year or so; for much of that time, it has been widely regarded as the most likely "next generation filesystem" for Linux. But, before it can claim that title, Btrfs must stabilize and find its way into the mainline kernel. Btrfs developer Chris Mason has been saying for a while that he thinks the code will come together more quickly if it is merged relatively soon, even if it is not yet truly ready for production use. General experience with kernel development tends to support this position: in-tree code gets more review, testing, and fixes than out-of-tree code. So the development community as a whole has been reasonably supportive of a relatively early Btrfs merge.In our last Btrfs episode, Andrew Morton suggested that a 2.6.29 merge be targeted. Chris would like that happen; to that end, he has posted a version of Btrfs for consideration. Unsurprisingly, that posting has already increased the amount of attention being paid to this code, with the result that Chris quickly got a list of things to fix. Most of those have now been addressed, but there are a few remaining issues which could still impede the merging of Btrfs in this development cycle. This article will look at the potential roadblocks. One of those is the user-space API. Btrfs brings with it a whole set of new ioctl() calls, none of which have been seriously reviewed or even documented. These calls perform functions like creating snapshots, initiating defragmentation, creating or resizing subvolumes, adding devices to the volume set, etc. Interestingly, there has been no real complaint about the volume-management features of Btrfs in general. But the interface to features like that needs close scrutiny; normally, user-space APIs cannot be broken once they are merged into the mainline. There has been some talk of making an exception for Btrfs, since there is little chance of systems becoming dependent on a specific interface before Btrfs is production-ready. Still, once distributions start shipping Btrfs tools - to help testers if nothing else - an API change would cause pain. Any potential for this kind of pain would make API changes very hard to do. So Linux may well end up being stuck with the early Btrfs API. Given that at least one developer thinks that this API needs a serious rework, this issue could turn out to be a serious roadblock indeed. Then, there is the issue of the special-purpose locking primitives used in Btrfs. To understand this discussion, it's worth looking at the locking function used within Btrfs:
int btrfs_tree_lock(struct extent_buffer *eb)
{
int i;
if (mutex_trylock(&eb->mutex))
return 0;
for (i = 0; i < 512; i++) {
cpu_relax();
if (mutex_trylock(&eb->mutex))
return 0;
}
cpu_relax();
mutex_lock_nested(&eb->mutex, BTRFS_MAX_LEVEL - btrfs_header_level(eb));
return 0;
}
The lock in question is a mutex, but it is being acquired in an interesting way. If the lock is held by another process, this function will poll it up to 512 times, without sleeping, in the hope that it will become available quickly. Should that happen, the lock can be acquired without sleeping at all. After 512 unsuccessful attempts, the function will finally give up and go to sleep. Chris justifies this behavior this way:
Btrfs is using mutexes to protect the btree blocks, and btree
searching often hits hot nodes that are always in cache. For these
nodes, the spinning is much faster, but btrfs also needs to be able
to sleep with the locks held so it can read from the disk and do
other complex operations.
For btrfs, dbench 50 performance doubles with the unconditional spin, mostly because that workload is almost all in ram. For 50 procs creating 4k files in parallel, the spin is 30-50% faster. This workload is a mixture of disk bound and CPU bound. That kind of performance increase seems worth going for. In fact, it reflects a phenomenon which has been observed in other situations as well: even when sleeping locks are used, performance often improves if a processor spins for a while in the hope that a contended lock will become available. If the lock can be acquired without sleeping, then the overhead associated with putting the process to sleep and waking it up can be avoided. Beyond that, though, there is the fact that the process seeking to acquire the lock is probably well represented in the CPU's cache. Allowing that process to continue to run will, if the lock can be acquired quickly, almost certainly lead to better system performance. For this reason, the adaptive realtime locks patch was developed last year, though it never found its way into the mainline. In response to the Btrfs discussion, Peter Zijlstra proposed a spinning mutex patch which is intended to provide the same benefits as the special Btrfs locking function, but for more general use and without the addition of magic constants. In Peter's patch, an attempt to acquire a contended lock will spin for as long as the process holding that lock is actually running on a CPU. If the lock holder goes to sleep, any process trying to acquire the lock also goes to sleep. The heuristic seems to make sense, though detailed benchmarks have not been posted. The patch was received reasonably well, though Linus has insisted that some changes be made. So a more general spinning mutex may well find its way into the mainline. Whether it will go in for 2.6.29 is not clear, though. Developers tend to like their core locking primitives to be reasonably well tested; merging something which was developed toward the end of the merge window could be a hard sell. Until something like that happens, Chris is uninterested in removing his special locking function:
But, if anyone working on adaptive mutexes is looking for a coder,
tester, use case, or benchmark for their locking scheme, my hand is
up. Until then, this is my for loop, there are many like it, but
this one is mine.
Finally, there is the question of the name. Some reviewers have suggested that the filesystem should be merged with a name which makes it clear that it's not meant for production use - "btrfsdev," for example. Chris is resistant to that idea, noting that, unlike existing filesystems, Btrfs is known to be new and has no reputation for stability. He has stated his willingness to make the change, though, if it is truly considered to be necessary. Bruce Fields pointed out that calling it "Btrfs" from the beginning could possibly burn future developers who boot an old kernel (with a non-production Btrfs) after switching to a newer, production-ready version of the filesystem. All of this adds up to an uncertain fate for Btrfs in 2.6.29; there are a fair number of open issues and it's late in the merge window. Of course, Btrfs could be merged after 2.6.29-rc1; since it is a completely new subsystem, it won't cause regressions. But if Linus concludes that there are enough loose ends in the current Btrfs code, he may just decide to give it one more development cycle before bringing it into the mainline. So, while nobody seems to doubt that Btrfs will go in, the question of when remains open. (With any luck, we hope to have an authoritative article on Btrfs for this page in the near future, once the author - you know who you are! - gets it written. Stay tuned.)
Patches and updates Kernel trees
Build system
Core kernel code
Development tools
Device drivers
Filesystems and block I/O
Memory management
Networking
Architecture-specific
Security-related
Virtualization and containers
Miscellaneous
Page editor: Jonathan Corbet |
|||||||||||||