Kernel development

Kernel release status

The current development kernel is 4.0-rc6, released on March 29. According to Linus: "Things are calming down nicely, and there are fixes all over. The NUMA balancing performance regression is fixed, and things are looking up again in general. There were a number of i915 issues and a KVM double-fault thing that meant that for a while there I was pretty sure that this would be a release that will go to rc8, but that may be unnecessary."

Stable updates: 3.19.3 was released on March 26; 3.14.37 and 3.10.73 followed on the 27th.

Comments (none posted)

Quotes of the week

— Frederic Weisbecker

— Borislav Petkov

Comments (13 posted)

Attaching file descriptors to processes with CLONE_FD

The classic Unix system design has many advantages, but friendliness to library code is not always one of them. The problem, at its core, is the use of global structures and mechanisms that cannot be manipulated independently by an application and any libraries it may use; the file descriptor table and signal handling vectors are a couple of examples. A recently posted patch set is trying to address a little piece of this problem, but may point the way toward a larger long-term change in how processes are managed in Linux.

Consider a library that needs to create a child process with fork() and be notified when that process exits. The library cannot call wait() without blocking the entire calling process and, possibly, interfering with the reaping of any child processes created elsewhere in the application. Receiving asynchronous notifications with the SIGCHLD signal has a similar problem: there can only be one SIGCHLD handler, so catching it in the library clashes with the application's use of that signal. The need to avoid this kind of conflict with application code can greatly limit what can be done within a library.

Josh Triplett's CLONE_FD patch set is an attempt to solve that particular problem. It adds a new flag, CLONE_FD, to a variant of the clone() system call (called clone4()); if that flag is present, a successful clone4() will return a file descriptor referring to the process it just created. There is little that can be done with that descriptor by default, but, if the CLONE_AUTOREAP flag is also set, the system's behavior changes somewhat.

In particular, when a process created with CLONE_AUTOREAP exits, it will be cleaned up immediately rather than sitting in the "zombie" state until the parent process gets around to calling wait(). This behavior can be useful in its own right if the parent process is not concerned about when the child exits. If the parent does care about its child's exit, though, it can create a process file descriptor with CLONE_FD; that descriptor will become readable when the process exits. In particular, it will be possible to read a structure like:

    struct clonefd_info {
	uint32_t code;   /* Signal code */
	uint32_t status; /* Exit status or signal */
	uint64_t utime;  /* User CPU time */
	uint64_t stime;  /* System CPU time */
    };

With this mechanism in place, a library function can create a process and wait for it to complete without interfering with process management anywhere else in the program. It thus solves the problem of having to share the machinery for managing process exit information by removing that sharing and, essentially, turning a running process into a sort of "file" that can be tracked and managed exclusively by the code that knows about it.

There was one tiny little problem in the way of implementing this solution, though: there is no room for additional flags for the clone() system call. So Josh had to create a new one, which he called clone4(). The intended interface, as presented by the C library, is:

    int clone4(uint64_t flags, size_t args_size, struct clone4_args *args,
               int (*fn)(void *), void *arg);

The flags field holds the flags to the operation: the traditional clone() flags and the new ones described above. As with clone(), the child process will call fn(arg) after its creation. The args argument looks like this:

    struct clone4_args {
	pid_t *ptid;
	pid_t *ctid;
	unsigned long stack_start;
	unsigned long stack_size;
	unsigned long tls;
	int *clonefd;
	unsigned clonefd_flags;
    };

Most of these fields match arguments passed to the clone() call; they have just been moved into a separate structure. The clonefd and clonefd_flags fields are new, though; the first is a location to store the created file descriptor when CLONE_FD is passed; the second can hold the O_CLOEXEC and O_NONBLOCK flags to be applied to that file descriptor.

The size of the clone4_args structure must be passed separately in args_size. If fields must be added to this structure in the future, the passed size will allow the kernel to determine which version of the structure is in use and respond accordingly.

For completeness, the actual system call (invoked from the C library wrapper) looks like this:

    int clone4(unsigned flags_high, unsigned flags_low,
               unsigned long args_size,
               struct clone4_args *args);

At this level, clone4() behaves like fork(), returning into both the parent and child processes (but with different return values).

This functionality is useful enough, but the patch description also includes this intriguing note:

Process IDs (PIDs) are subject to a narrow race condition: a process could exit and be replaced by another using the same PID. In a system where there are many processes running and there is a lot of fork activity, the probability of short-term PID reuse is significant, especially if, as is usually the case, the range of available PIDs has not been increased beyond the traditional 32,768. A file descriptor that is attached to a process at creation time is not subject to this race, though; it should thus be safer for other processes to operate on.

Adding file-descriptor-oriented process-management system calls is a future exercise, though; for now, the CLONE_FD functionality solves the immediate problem. This patch has been through two rounds of review so far; some significant changes have been made, but the core functionality seems to be uncontroversial. One more review round seems likely to happen, though; after that, this feature could conceivably be added as early as the 4.2 development cycle.

Comments (51 posted)

Statistics from the 4.0 development cycle

The 4.0 kernel development cycle is heading toward its close; the final release can be expected on, mostly likely, April 12 (or possibly April 19 if a late regression turns up). This release will be notable for the new major version number, of course, even though the shift to 4.x does not mean anything more than "the minor numbers were getting too big." It may also be notable for being one of the slower development cycles in the last couple of years, though one should bear in mind that "slow" is used in a relative sense here.

To be specific, as of this writing, the 4.0 cycle has seen the addition of just over 10,000 non-merge changesets to the mainline repository. 1,403 developers have contributed to this release cycle so far; they have added 403,000 lines of code and removed 222,000 for a net growth of 181,000 lines. The list of the most active developers looks a bit different than it has in the last few cycles:

Most active 4.0 developers

By changesets Lars-Peter Clausen 179 1.8% Takashi Iwai 172 1.7% H Hartley Sweeten 153 1.5% Rickard Strandqvist 111 1.1% Antti Palosaari 95 0.9% Thierry Reding 94 0.9% Geert Uytterhoeven 88 0.9% Ian Abbott 88 0.9% Maxime Ripard 85 0.8% Michael S. Tsirkin 82 0.8% Marcel Holtmann 79 0.8% Ben Skeggs 77 0.8% Arnd Bergmann 76 0.8% Laurent Pinchart 75 0.7% Rasmus Villemoes 75 0.7% Al Viro 71 0.7% Trond Myklebust 71 0.7% Andy Shevchenko 66 0.7% Krzysztof Kozlowski 64 0.6% Christophe Ricard 62 0.6%

By changed lines Ben Skeggs 23587 4.8% Hans Verkuil 16433 3.3% Thomas Petazzoni 10642 2.1% Tero Kristo 9341 1.9% Hariprasad Shenai 8810 1.8% Michal Kazior 7878 1.6% H Hartley Sweeten 6925 1.4% Laurent Pinchart 6803 1.4% Dudley Du 5399 1.1% Takashi Iwai 5137 1.0% Antti Palosaari 4913 1.0% Boris Brezillon 4666 0.9% Christoph Hellwig 4365 0.9% Arnd Bergmann 3974 0.8% Rusty Russell 3963 0.8% Tony Lindgren 3960 0.8% Rickard Strandqvist 3921 0.8% Magnus Damm 3771 0.8% Andrzej Pietrasiewicz 3697 0.7% Maxime Ripard 3664 0.7%

For once, Hartley Sweeten's work on the Comedi drivers did not put him at the top of the "by changesets" list; that place was taken by Lars-Peter Clausen, who worked mostly in the audio and media driver trees. Takashi Iwai's work remains entirely within the audio subsystem tree. Below Hartley, Richard Strandqvist cleaned up a lot of dead code throughout the tree, while Antti Palosaari did a lot of work in the media driver subsystem.

In the "lines changed" column, Ben Skeggs carried out a massive renaming of symbols in the Nouveau driver; "nouveau" became "nvkm" in the parts of the code that make up the direct-rendering kernel module. Hans Verkuil continues to do work throughout the media subsystem; the bulk of his changed lines, though, took the form of removing some old, unloved drivers. Thomas Petazzoni added a number of framebuffer drivers to the staging tree, Tero Kristo cleaned up and enhanced the TI OMAP clock subsystem, and Hariprasad Shenai did a bunch of work on the cxgb4 network/InfiniBand drivers.

For some years, work on the staging tree has tended to dominate these two lists, but that is not the case for 4.0. Indeed, this is one of the slowest development cycles for the staging tree in general, as can be seen in the plot below:

The slow traffic in the staging tree explains much of the relative slowness of the 4.0 development cycle in general.

There are 197 employers who are known to have supported development of the 4.0 kernel. The most active of these were:

Most active 4.0 employers

By changesets Intel 1164 11.6% (None) 864 8.6% (Unknown) 712 7.1% Red Hat 703 7.0% SUSE 463 4.6% Linaro 401 4.0% Samsung 361 3.6% (Consultant) 336 3.3% Free Electrons 251 2.5% IBM 226 2.2% Renesas Electronics 188 1.9% Freescale 165 1.6% Google 154 1.5% Vision Engraving Systems 153 1.5% Primary Data 149 1.5% AMD 142 1.4% Texas Instruments 139 1.4% Oracle 137 1.4% Qualcomm 120 1.2% ARM 119 1.2%

By lines changed Intel 49062 9.9% Red Hat 46588 9.4% (None) 34171 6.9% Samsung 23835 4.8% Free Electrons 22943 4.6% Texas Instruments 22395 4.5% (Unknown) 22369 4.5% Cisco 19438 3.9% Linaro 17553 3.5% SUSE 11764 2.4% Renesas Electronics 10796 2.2% (Consultant) 10719 2.2% Chelsio 10439 2.1% IBM 10355 2.1% Code Aurora Forum 9697 2.0% Tieto 8115 1.6% ARM 7316 1.5% Vision Engraving Systems 6925 1.4% Qualcomm 6850 1.4% AMD 6781 1.4%

In the 3.19 development statistics article, we noted that developers with no affiliation (volunteers) were holding steady at about 11% of the total changeset contribution. The accompanying suggestion that the decline in volunteer developers could be ending may have been premature, though; in 4.0, only 8.6% of the (relatively low) changeset count came from volunteers. That is the lowest point since 3.10 and the second-lowest ever.

Otherwise, there is not much that jumps out from the above table; corporate support for kernel development doesn't change a whole lot from one development cycle to the next.

As of this writing, there are about 6,500 non-merge changesets in the linux-next tree, putting linux-next in a place similar to where it was at this point in the 3.19 development cycle. That suggests that, unless things change, 4.1 will be another relatively slow cycle — though, it bears repeating, the incorporation of over 10,000 changes in a development cycle lasting less than three months is not all that slow. Even if we are not setting records at the moment, it seems clear that the kernel development community remains strong and active.

Comments (none posted)

XFS: There and back ... and there again?

In a thought-provoking—and characteristically amusing—talk at the Vault conference, Dave Chinner looked at the history of XFS, its current status, and where the filesystem may be heading. In keeping with the title of the talk (shared by this article), he sees parallels in what drove the original development of XFS and what will be driving new filesystems. Chinner's vision of the future for today's filesystems, and not just of XFS, may be a bit surprising or controversial—possibly both.

History

In the early 1990s, "before I knew who SGI was", Chinner said, storage was exceeding capacities that could be addressed with 32 bits. The existing filesystem for SGI's Unix, IRIX, was the Extent File System (EFS) that only supported 32-bit addresses. At that time, 64-bit CPUs with large-scale multiprocessing were coming on, but systems with hundreds of disks were not performing well.

XFS came about to fix those problems. The "x" originally just meant "undefined". This new undefined filesystem, xFS, had to support a number of features: fast crash recovery, large filesystems with both sparse and contiguous large files, and filesystems and directories that could hold huge numbers of files. The filesystem would need to support terabyte and petabyte capacities in the foreseeable future at that time, he said.

Chinner showed a graph of the lines of code changed in XFS over time that went all the way back to the beginning and up through commits from the previous week. Because the full commit history for XFS is available, graphs like that can be made, he said. Another way to use the history is to track bugs back to their introduction. The oldest bug anyone has found in recent times was 19 years old, he said.

The first production release of XFS was in December 1994 in conjunction with the release of IRIX 5.3. In mid-1996, XFS became the default filesystem in IRIX 6.2. The on-disk format had already gone through four versions. Even those early versions had the "feature mask" that allows XFS filesystems to turn on or off individual filesystem features. Those can be set for a particular filesystem at mkfs time; the mask will be consulted for various types of operations. It is a feature of XFS that makes it particularly flexible.

The "IRIX years" of the late 1990s saw various feature additions. Hardware RAID array support was added in 1997, which resulted in XFS gaining the ability to align its allocations to match the geometry of the underlying storage. By 1999, the original directory structure was showing its age, so version 2 directories were added, increasing directory scalability to tens of millions of files. Chinner has personally tried 350 million files in a directory, but he doesn't recommend it; running ls on such a directory takes twelve hours, he said with a laugh.

Shift to Linux

Shortly thereafter, feature development in XFS on IRIX slowed down. SGI had shifted its focus to large NUMA machines and distributed IRIX; development moved from XFS to the new CXFS cluster filesystem. But SGI also had a new product line that was based on Linux, which didn't have a filesystem with the features of XFS. So a team was formed in Melbourne, Australia to port XFS to Linux.

In 2000, XFS was released under the GPL. The first stable release of XFS for Linux came in 2001. And, in 2002, XFS was merged into the mainline for Linux 2.5.36.

One of the unfortunate side effects of XFS and other filesystems being merged into Linux was the proliferation of what Chinner called "Bonnie++ speed racing". People were using the filesystem benchmarking tool with little knowledge, which resulted in them "twiddling knobs" in the filesystem that they did not understand. The problem with that is that Google still finds those posts today, so those looking for better XFS performance are finding these old posts and following the instructions therein, which leads to the never-ending "noatime,nodiratime,logbufs=8 meme" for XFS mount options.

In the early 2000s, the Linux version of XFS started to diverge from the IRIX version with the group quotas feature. The version 2 log format was added in 2002, which helped make metadata performance much better. Also in 2002 came inode cluster delete and configurable sector sizes for XFS. In 2004, the 2.4 and 2.6 development trees were unified and the feature mask was expanded. XFS had run out of bits in the feature mask, he said, so the last bit was used to indicate that there is an additional feature mask to be consulted.

A "major achievement" was unlocked in 2004, when SUSE shipped full XFS support in SUSE Linux Enterprise Server (SLES) 9. It was a validation of all the work that SGI had done on the filesystem, he said. Someone from the audience spoke up to say that SLES 8 had shipped with XFS support in 2001, which seemed to surprise Chinner a bit.

The mid-2000s saw lots of misinformation spread about XFS (and other filesystems) during the "filesystem wars", Chinner said. His slides [PDF] contain several quotes from that time about "magical" features, including large capacitors in SGI power supplies that caused XFS not to lose data on power failure, zeroing of data after an unlink operation so that undelete was not possible, and the zeroing of all open files when there is an unclean shutdown. None of those were true, but they have become part of the XFS lore.

More features came about in 2005 and 2006. Extended attributes were added into the inode, for example, which resulted in an order-of-magnitude performance improvement for Samba atop XFS. That time frame was also the crossover point where Linux XFS started outperforming IRIX XFS. On Linux 2.6.16, XFS achieved a 10GB/second performance on a 24-processor Altix machine, he said.

A few years further on, the "O_PONIES wars" began. Chinner pointed to a Launchpad bug about an XFS file data corruption problem under certain conditions. The workaround was to use fdatasync() after renaming the file, but that would require changes in user space. The bug was closed as a "Won't fix", but none of the XFS developers were ever even consulted about it. It turns out that it actually was a bug in XFS that was fixed a year later.

Shift to the community

"Sad days" arrived in 2009, when the SGI XFS engineering team was disbanded. The company had been losing money since 1999, he said. The community stepped in to maintain the XFS tree while SGI reorganized. After that, SGI intermittently maintained XFS until the end of 2013; he didn't mention that most of the work during that time was being done by developers (like himself) working at other companies. Chinner said he went away to the racetrack for a weekend and came back to find out that he had been nominated to take over as maintainer.

Since the shutdown of the XFS team, development on the filesystem has largely shifted to the wider community, he said. Development did not slow down as a result, however; if anything, it accelerated. But all of that work is not just about adding code, a lot of code has been removed along the way, as well.

He rhetorically asked if XFS is "still that big, bloated SGI thing"? He put up a graph showing the number of lines of code (LOC) in XFS for each kernel release. The size of the XFS codebase started dropping around 2.6.14, bottomed out in 3.6, rose again until 3.15 or so, and has leveled off since. The level (just under 70,000 LOC) is lower than where the graph started with 2.6.12 (roughly 75,000 LOC). The line for Btrfs crossed that of XFS around 3.5 and it is at 80,000+ and still climbing; it hasn't leveled off, Chinner said.

He listed the top developers for XFS, with Christoph Hellwig at the top, followed closely by Chinner himself. He called out early XFS developers Adam Sweeney and Doug Doucette, noting that they had done "tons of work" in a fairly small number of years. He quoted Isaac Newton ("If I have seen further than others, it is by standing upon the shoulders of giants.") and said that XFS came about "not because of me", but from the work of all of those others on the list.

Recent and ongoing developments in XFS include sparse inode chunk allocation to support GlusterFS and Ceph, unification of the quota API, and reverse mapping for the internal B-trees (more reasons for doing so cropped up at the recently completed LSFMM Summit, he said). In addition, reflink support for per-file snapshots has been added, defragmentation improvements have been made, and support for the direct access block layer (DAX) has been added. There is "lots going on at the moment", he said.

The near future

In among working on all of that, Chinner has been trying to plot out the next five years or so of XFS development. He did a similar exercise back in 2008 and posted some documents to XFS.org. Over the years since then, all of the features in those documents have been ticked off the list, though reverse B-trees, which are 95% complete, are "the last bit". So, now is the time to plan for the future.

There are known storage technologies that are pulling filesystem developers in opposite directions, he said. Shingled magnetic recording (SMR) devices and persistent memory are going to change things radically. The key for developers of existing filesystems is to figure out "good enough" solutions to allow those filesystems to work on SMR and persistent memory devices. Eventually, someone will come up with a filesystem that specifically targets SMR devices or persistent memory that "blows everything else away".

Over the next five or more years, XFS needs to have better integration with the block devices it sits on top of. Information needs to pass back and forth between XFS and the block device, he said. That will allow better support of thin provisioning. It will also help with offloading block clone, copy, and compress operations. There are more uses for that integration as well, including better snapshot awareness and control at the filesystem layer.

Improving reliability is another area where XFS will need more work. Reconnecting orphaned filesystem objects will become possible with the B-tree reverse mapping abilities. That will also allow repairing the filesystem while it is online, rather than having to unmount it to do repairs. Proactive corruption detection is also on the list. When corruption is found, the affected part of the filesystem could be isolated for repair. These features would make XFS "effectively self-healing", Chinner said.

There is a "fairly large amount of work" on the list, he said. While he is targeting five years, he can't do it alone in that time period. If it is just him, it will probably take six or seven years, he said with a grin.

Farther out

But we also need to be thinking a little further ahead. Looking at the progression of capacities and access times for "spinning rust" shows 8GB, 7ms drives in the mid-1990s and 8TB, 15ms drives in the mid-2010s. That suggests that the mid-2030s will have 8PB (petabyte, 1000 terabytes) drives with 30ms access times.

The progression in solid-state drives (SSDs) shows slow, unreliable, and "damn expensive" 30GB drives in 2005. Those drives were roughly $10/GB, but today's rack-mounted (3U) 512TB SSDs are less than $1/GB and can achieve 7GB/second performance. That suggests to him that by 2025 we will have 3U SSDs with 8EB (exabyte, 1000 petabytes) capacity at $0.1/GB.

SSDs have lots of compelling features (e.g. density, power consumption, performance) and are cost-competitive. But persistent memory will be even denser and faster, while still being competitive in terms of cost. It is at least five years before we see pervasive persistent memory deployment, however. Chinner did caution that memristors are a wild card that could be "a game changer" for storage.

Those projections indicate that focusing on spinning rust would be largely misplaced. XFS (and other filesystems) have to follow where the hardware is taking them. SSD storage will push capacity, scalability, and performance much faster than SMR technologies will. Though "most of the cat pictures" on the internet will likely be stored on SMR drives before too long, he said with a chuckle.

8EB is about as large as XFS can go. In fact, the 64-bit address space will be exhausted for filesystems and CPUs in the next 10-15 years. Many people thought that the 128-bit addresses used by ZFS were "crazy", but it doesn't necessarily look that way now.

By 2025-2030, XFS will be running up against capacity and addressability limits. It will also be hitting architectural performance limits in addition to complexity limits from continuing to support spinning rust disks. The architecture of XFS will be unsuited to the emerging storage technologies at that time.

All of those projections place a hard limit on the development life of XFS. It doesn't make sense to do a large-scale rework of the filesystem to support SMR devices, for example. SMR will be supported, but the project is "not going to turn everything upside down for it". There is a limited amount of life left in the filesystem at this point and SMR technologies will be outrun by other storage types.

From Btrfs, GlusterFS, Ceph, and others, we know that it takes 5-10 years for a new filesystem to mature. That implies we need to start developing any new filesystems soon. For XFS, there is the definite possibility that this 5-7 year development plan that he is working on will be the last. In 20 years, XFS will be a legacy filesystem. That's likely true for any current filesystem; he may not be right, he said, but if he is, we may well be seeing the last major development cycle for all of the existing filesystems in Linux.

[I would like to thank the Linux Foundation for travel support to Boston for Vault.]

Comments (46 posted)