Kernel development

Brief items

Kernel release status

The 3.5 kernel was released on July 21; see Linus's announcement for some low-level details. Headline features in 3.5 include the CoDel queue management algorithm (a piece of the solution to the bufferbloat problem), the seccomp filters sandboxing mechanism, autosleep (an alternative to Android's opportunistic suspend mechanism), the uprobes user-space probe subsystem, the contiguous memory allocator, the new kcmp() system call, metadata checksumming in the ext4 filesystem, and a lot more. See the KernelNewbies 3.5 page for more information.

Stable updates: 3.0.38 and 3.4.6 were released on July 19 with the usual set of important fixes. The 3.2.24 update is in the review process as of this writing; its release can be expected any time.

Comments (none posted)

Quotes of the week

-- Arjan van de Ven

-- Mike Galbraith

Comments (none posted)

CRtools 0.1 released

The OpenVZ blog has the announcement of the release of CRtools 0.1. "It is our ultimate goal to merge all bits and pieces of OpenVZ to the mainstream Linux kernel. It's not a big secret that we failed miserably trying to merge the checkpoint/restore [CPT] functionality (and yes, we have tried more than once). The fact that everyone else failed as well soothes the pain a bit, but is not really helpful. The reason is simple: CPT code is big, complex, and touches way too many places in the kernel. So we* came up with an idea to implement most of CPT stuff in user space, i.e. as a separate program not as a part of the Linux kernel. In practice this is impossible because some kernel trickery is still required here and there, but the whole point was to limit kernel intervention to the bare minimum. Guess what? It worked even better that we expected. As of today, after about a year of development, up to 90% of the stuff that is needed to be in the kernel is already there, and the rest is ready and seems to be relatively easy to merge."

Comments (36 posted)

Kernel development news

3.6 merge window part 1

Linus traditionally waits for a day or so after a major release before beginning to merge patches for the next cycle, but, with 3.6, he started right in. As of this writing, some 4,300 non-merge changesets have been pulled into the mainline; much of the activity thus far has been from the networking and ARM subsystems. Significant user-visible changes include:

• The perf events subsystem now has support for the "uncore" performance measurement unit on Intel Nehalem and Sandy Bridge CPUs.

• The x86 architecture now supports the reboot=bios and reboot-cpu command-line options on 64-bit processors (as well as on 32-bit, which has been supported for a long time)

• "Suspend to both" support allows the system to be suspended after writing a hibernation image to disk. Then, should power run out before the suspended system is resumed, it can be restarted from the disk image instead.

• The CANFD extension to the controller area network (CAN) protocol is now supported.

• Numerous netfilter modules have gained proper namespace support. The netfilter user-space connection tracking helper infrastructure has also been merged.

• The Bluetooth layer now has "three-wire UART" support, enabling Bluetooth operations over serial port connections.

• The TCP small queues patch set, another piece of the solution to the bufferbloat problem, has been merged.

• The TCP fast open protocol extension has been merged. TCP fast open is a patch out of Google that reduces the overhead of TCP connection setup, hopefully making protocols like HTTP go faster.

• A long effort to remove the IPv4 routing cache from the networking subsystem has come to its conclusion. David Miller wrote:
  The ipv4 routing cache is non-deterministic, performance wise, and is subject to reasonably easy to launch denial of service attacks. The routing cache works great for well behaved traffic, and the world was a much friendlier place when the tradeoffs that led to the routing cache's design were considered.

  What it boils down to is that the performance of the routing cache is a product of the traffic patterns seen by a system rather than being a product of the contents of the routing tables.

  The replacement code simplifies the networking subsystem and, hopefully, gives better performance on high-volume systems.

• New hardware support includes:
  • Processors and systems: Freescale BSC9131RDB reference boards, Altera SOCFPGA Cyclone V systems, Marvell Armada 370 and Armada XP boards, TI OMAP5 processors, TI EVMC6678LE evaluation boards, and Freescale (Motorola) Coldfire 5251/5253 and 5441x processors.

  • Audio: TI Isabelle audio ICs, ST-Ericsson AB8500 codecs, Dialog DA732x audio codecs Wolfson Micro WM5102 and WM5110 audio controllers, and ST STA529 audio amplifiers.

  • Input: Lenovo ThinkPad USB keyboards with trackpoint and Roccat Savu gaming mice.

  • Miscellaneous: Samsung S2MPS11 voltage regulators, Maxim 77686 voltage regulators, TI/National Semiconductor LP8720/LP8725 voltage regulators, Dialog Semiconductor DA9052 PMICs, Honeywell Humidicon HIH-6130/HIH-6131 humidity sensors, Wolfson Micro WM831x and WM832x PMICs, and NVIDIA Tegra20 APB DMA controllers.

  • Networking: RealTek rt3290 WiFi controllers, Sony PaSoRi contactless reader NFC controllers, Atmel RF230/231 radio transceivers, Broadcom BCM8706 and BCM8727 PHYs, and Asix AX88172A USB 2.0 Ethernet interfaces.

Changes visible to kernel developers include:

• The obsolete static_branch() interface has been removed in favor of static_key_true() and static_key_false(). Some information on this interface can be found in this article.

• Some initial work has been done to separate the dynamic tick code from the idle task, setting the ground for stopping the timer tick on non-idle CPUs.

• The power domains subsystem has seen some integration with the cpuidle code to handle situations where devices share power lines with CPU cores.

• The VFS layer has seen some significant changes. There is a new atomic_open() inode operation that combines the process of looking up, possibly creating, and opening a file into a single, atomic operation. The whole "open intents" mechanism has been removed. Numerous other operations have had prototype changes. The deferred fput() changes have been merged, simplifying the process of cleaning up file structures.

• The PowerPC architecture now supports the jump label mechanism.

• The NLMSG_NEW() and NLMSG_PUT() macros have been removed from the netlink interface.

• The input subsystem has a new interface for the creation of user-space drivers; see Documentation/hid/uhid.txt for details.

• There is a new grouping mechanism for I/O memory management units intended to help enable safe device access to virtualized guests.

This merge window can be expected to last until sometime around August 4, so there is quite a bit of code that can be expected to find its way into the mainline before the -rc1 release happens. See next week's Kernel Page for coverage of the continuation of the 3.6 merge window.

Comments (2 posted)

The UAPI header file split

Patches that add new software features often gain the biggest headlines in free software projects. However, once a project reaches a certain size, refactoring work that improves the overall maintainability of the code is arguably at least as important. While such work does not improve the lives of users, it certainly improves the lives of developers, by easing later work that does add new features.

With around 15 million lines of code (including 17,161 .c files and 14,222 .h files) in the recent 3.5 release, the Linux kernel falls firmly into the category of projects large enough that periodic refactoring is a necessary and important task. Sometimes, however, the sheer size of the code base means that refactoring becomes a formidable task—one that verges on being impossible if attempted manually. At that point, an enterprising kernel hacker may well turn to writing code that refactors the kernel code. David Howell's UAPI patch series, which has been proposed for inclusion during the last few kernel merge windows, was created using such an approach.

The UAPI patchset was motivated by David's observation that when modifying the kernel code:

He went on to elaborate that this problem of "inclusion recursion" in header files typically occurs with inline functions:

As is the way of such things, a small itch can lead one to thinking about more general problems, and how to solve them, and David has devised a grand nine-step plan of changes to achieve his goals, of which the current patch set is just the first step. However, this step is, in terms of code churn, a big one.

What David wants to do is to split out the user-space API content of the kernel header files in the include and arch/xxxxxx/include directories, placing that content into corresponding headers created in new uapi/ subdirectories that reside under each of the original directories. As well as being a step toward solving his original problem and performing a number of other useful code cleanups, David notes that disintegrating the header files has many other benefits. It simplifies and reduces the size of the kernel-only headers. More importantly, splitting out the user-space APIs into separate headers has the desirable consequence that it "simplifies the complex interdependencies between headers that are [currently] partly exported to userspace".

There is one other benefit of the UAPI split that may be of particular interest to the wider Linux ecosystem. By placing all of the user-space API-related definitions into files dedicated solely to that task, it becomes easier to track changes to the APIs that the kernel presents to user space. In the first instance, these changes can be discovered by scanning the git logs for changes in files under the uapi/ subdirectories. Easing the task of tracking user-space APIs would help many other parts of the ecosystem, for example, C library maintainers, scripting language projects that maintain language bindings for the user-space API, testing projects such as LTP, documentation projects such as man-pages, and perhaps even LWN editors preparing summaries of changes in the merge window that occurs at the start of each kernel release cycle.

The task of disintegrating each of the header files into two pieces is in principle straightforward. In the general case, each header file has the following form:

    /* Header comments (copyright, etc.) */

    #ifndef _XXXXXX_H     /* Guard macro preventing double inclusion */
    #define _XXXXXX_H

    [User-space definitions]

    #ifdef __KERNEL__

    [Kernel-space definitions]

    #endif /* __KERNEL__ */

    [User-space definitions]
  
    #endif /* End prevent double inclusion */

Each of the above parts may or may not be present in individual header files, and there may be multiple blocks governed by #ifdef __KERNEL__ preprocessor directives.

The part of this file that is of most interest is the code that falls inside the outermost #ifndef block that prevents double inclusion of the header file. Everything inside that block that is not nested within a block governed by a #ifdef __KERNEL__ block should move to the corresponding uapi/ header file. The content inside the #ifdef __KERNEL__ block remains in the original header file, but the #ifdef __KERNEL__ and its accompanying #endif are removed.

A copy of the header comments remains in the original header file, and is duplicated in the new uapi/ header file. In addition, a #include directive needs to be added to the original header file so that it includes the new uapi/ header file, and of course a suitable git commit message needs to be supplied for the change.

The goal is to modify the original header file to look like this:

    /* Header comments (copyright, etc.) */

    #ifndef _XXXXXX_H     /* Guard macro preventing double inclusion */
    #define _XXXXXX_H

    #include <include/uapi/path/to/header.h>

    [Kernel-space definitions]

    #endif /* End prevent double inclusion */

The corresponding uapi/ header file will look like this:

    /* Header comments (copyright, etc.) */

    #ifndef _UAPI__XXXXXX_H     /* Guard macro preventing double inclusion */
    #define _UAPI__XXXXXX_H

    [User-space definitions]

    #endif /* End prevent double inclusion */

Of course, there are various details to handle in order to correctly automate this task. First of all, sometimes the script should produce only one result file. If there is no #ifdef __KERNEL__ block in the original header, the original header file is in effect renamed to the uapi/ file. Where the header file is disintegrated into two files, there are many other details that need to be handled. For example, if there are #include directives that are retained at the top of the original header file, then the #include for the generated uapi/ file should be placed after those #include directives (in case the included uapi/ file has dependencies on them). Furthermore, there may be pieces of the original header that are explicitly not intended for kernel space (i.e., they are for user-space only)—for example, pieces governed by #ifndef __KERNEL__. Those pieces should migrate to the uapi/ file, retaining the guarding #ifndef __KERNEL__.

David's scripts handle all of the aforementioned details, and many others as well, including making corresponding changes to .c source files and various kernel build files. Naturally, no scripting can correctly handle all possible cases in human-generated files, so part of the current patch set includes pre-patches that add markers to "coach" the scripts to do the right thing in those cases.

Writing scripts to automate this sort of task becomes a major programming project in its own right, and the shell and Perl scripts (.tar.xz archive) to accomplish the task run total more than 1800 lines. (Using scripting to generate the patch set has the notable benefit that the patch set can be automatically refreshed as the relevant kernel source files are changed by other kernel developers. Given that the UAPI patches touch a large number of files, this is an important consideration.)

Knowing the size of those scripts, and the effort that must have been required to write them, gives us a clue that the scale of the actual changes to the kernel code must be large. And indeed they are. In its current incarnation, the UAPI patch series consists of 74 commits, of which 65 are scripted (the scripted changes produce commits to the kernel source tree on a per-directory basis). Altogether, the patches touch more than 3500 files, and the diff of the changes amounts to over 300,000 lines.

The scale of these changes brings David to his next problem: how to get the changes accepted by Linus. The problem is that it's impossible to manually review source code changes of this magnitude. Even a partial review would require considerable effort, and would not provide ironclad guarantees about the remaining unreviewed changes. In the absence of such reviews, when Linus received David's request to pull these patches in the 3.5 merge window, he employed a time-honored strategy: the request was ignored.

Although David first started working on these changes around a year ago, Linus has not to date directly commented on them. However, back in January Linus accepted some preparatory patches for the UAPI work, which suggests that he's at least aware of the proposal and possibly willing to entertain it. Other kernel developers have expressed support for the UAPI split (1 and 2). However, probably because of the magnitude of the changes, getting actual reviews and Acked-by: tags has to date proved to be a challenge. Given the impossibility of a complete manual review of the changes, the best hope would seem to be to have other developers review the conceptual approach employed by David's scripts, possibly review the scripts themselves, perform a review of a sample of the changed kernel source files, and perform kernel builds on as many different architectures as possible. (Aspiring kernel hackers might note that much of the review task on this quite important piece of kernel work does not require deep understanding of the workings of the kernel.)

Getting sufficient review of any set of kernel patches, let alone a set this large, is a perennial difficulty. Things at least took a step forward with David's request to Linus to have the patches accepted for the currently open 3.6 merge window, when Arnd Bergmann provided his Acked-by: for the entire patch series. Whether that will prove enough, or whether Linus will want to see formal agreement from additional developers before accepting the patches is an open question. If it proves insufficient for this merge window, then perhaps a rethink will be required next time around about how to have such a large change accepted into the mainline kernel.

Comments (12 posted)

Who wrote 3.5

Now that the 3.5 Linux kernel has been released, it's time for the traditional look at who wrote it. Here we'll try to summarize who did all of the work that went into this release.

Fastest-changing kernel ever

The 3.5 kernel was released one day faster than the 3.4 kernel was, in 62 days. The last time a kernel was released this quickly was back in 2005 with the 2.6.14 kernel release (61 days).

In those 62 days, the kernel developers crammed in a record-breaking 176.73 changes per day (7.36 changes per hour.) This is the fastest-changing kernel that has been recorded since I started keeping track of this development metric back in the 2.5 kernel release series.

These changes resulted in the following overall changes:

Changes in 3.5
571987	lines added
358836	lines removed
135848	lines modified

The kernel is still increasing at a pretty constant 1.37% growth in the number of lines and files, which is similar to the growth rate of the past three kernel releases.

Individual contributions

1,195 different developers contributing patches to the 3.5 kernel; those developers worked for at least 194 different companies. The names of the contributing developers are pretty familiar to those who track these statistics:

Most active 3.5 developers

By changesets Greg Kroah-Hartman 239 2.2% Axel Lin 191 1.7% Mark Brown 187 1.7% H. Hartley Sweeten 135 1.2% David S. Miller 131 1.2% Daniel Vetter 130 1.2% Al Viro 128 1.2% Stephen Warren 121 1.1% Tejun Heo 112 1.0% Eric Dumazet 105 1.0% Hans Verkuil 102 0.9% Paul Mundt 102 0.9% Johannes Berg 102 0.9% Shawn Guo 102 0.9% Thomas Gleixner 98 0.9% Dan Carpenter 86 0.8% Sam Ravnborg 84 0.8% Chris Wilson 79 0.7% Trond Myklebust 74 0.7% Eric W. Biederman 73 0.7% Jiri Slaby 73 0.7% Arnaldo Carvalho de Melo 71 0.6% Artem Bityutskiy 68 0.6% Hans de Goede 68 0.6% Takashi Iwai 64 0.6%

By changed lines Paul Gortmaker 44000 5.7% Viresh Kumar 20425 2.7% Steven Rostedt 14615 1.9% H. Hartley Sweeten 13083 1.7% Dave Airlie 12217 1.6% Sakari Ailus 10835 1.4% Dong Aisheng 10574 1.4% Sonic Zhang 10494 1.4% Paul Walmsley 10084 1.3% Ben Skeggs 10000 1.3% Rob Herring 9886 1.3% Sascha Hauer 9602 1.3% Stephen Warren 9365 1.2% Parav Pandit 8846 1.2% Nicholas Bellinger 8704 1.1% Linus Walleij 8496 1.1% Shawn Guo 7797 1.0% David S. Miller 7445 1.0% Phil Edworthy 7189 0.9% Sam Ravnborg 6752 0.9% Hans Verkuil 6718 0.9% Alexander Shishkin 6668 0.9% Tejun Heo 6579 0.9% Greg Kroah-Hartman 6524 0.9% Vladimir Serbinenko 6451 0.8%

In the quantity category (remember, we don't judge quality), I did a large number of cleanup patches removing old USB logging macros from the system, which resulted in the majority of my changes in the 3.5 kernel. Axel contributed a great number of regulator driver fixes and enhancements, and Mark Brown did the majority of his work in the sound system-on-a-chip drivers area. H. Hartley Sweeten has been working on cleaning up the Comedi (data acquisition) drivers to get them ready to move out of the staging area of the kernel. This work has him showing up in these statistics for the first time. And rounding out the top five is David Miller with a large number of networking core and driver patches.

Along with H. Hartley Sweeten, Daniel Vetter is also a newcomer to the "top changesets" list. His contributions came from numerous changes and enhancements to the Intel graphics drivers. Although Hans Verkuil is also a name that might not be familiar to many, his contributions to the Video4Linux drivers and core code show he is a core contributor to a subsystem that many users rely on every day.

Considering the statistics in lines changed, Paul Gortmaker leads by virtue of the fact that he deleted all of the old Token Ring drivers from the kernel. Viresh Kumar did a lot of SPEAr processor and driver work, adding numerous new drivers for the platform. Steven Rostedt did a large amount of development on ftrace and ktest (a kernel-testing tool). H. Hartley Sweeten did the aforementioned Comedi driver cleanup work, and Dave Arlie made major changes in the area of graphics drivers.

Reviewing the work

All kernel patches are reviewed and "Signed-off-by" a subsystem maintainer before they are committed to the Linux kernel. The developers with the most sign-offs for the 3.5 kernel were as follows:

Developers with the most signoffs (total 20391)
Greg Kroah-Hartman	1216	6.0%
David S. Miller	922	4.5%
Mauro Carvalho Chehab	605	3.0%
Mark Brown	549	2.7%
John W. Linville	493	2.4%
Linus Torvalds	424	2.1%
Andrew Morton	373	1.8%
Daniel Vetter	268	1.3%
Dave Airlie	255	1.3%
Al Viro	197	1.0%
Axel Lin	191	0.9%
Trond Myklebust	173	0.8%
Arnaldo Carvalho de Melo	165	0.8%
James Bottomley	164	0.8%
Artem Bityutskiy	157	0.8%
Kyungmin Park	156	0.8%
Samuel Ortiz	154	0.8%
Linus Walleij	153	0.8%
Ingo Molnar	150	0.7%
Wey-Yi W Guy	146	0.7%
Thomas Gleixner	139	0.7%
Stephen Warren	136	0.7%
H. Hartley Sweeten	135	0.7%
Shawn Guo	131	0.6%
Paul Mundt	128	0.6%

I ended up doing the most sign-offs for this kernel release because of many changes in the staging and USB subsystems. David Miller follows with his work in the networking and networking driver trees, as well as in the IDE drivers. Mauro is the maintainer of the Video4Linux subsystem, Mark Brown is the maintainer of the embedded sound drivers, and John Linville is the maintainer of the wireless driver subsystem.

These numbers reflect the picture of what has been happening in the past few kernel releases, with the majority of changes happening in the staging and networking areas of the kernel.

Who sponsored this work

Here is the list of the companies who sponsored the developers doing the work for this kernel release, and the number of changes attributed to them:

Top changeset contributors by employer
(None)	1343	12.3%
Red Hat	1123	10.2%
Intel	1061	9.7%
(Unknown)	860	7.8%
Linaro	519	4.7%
Novell	440	4.0%
Texas Instruments	313	2.9%
IBM	282	2.6%
Linux Foundation	279	2.5%
Google	265	2.4%
Samsung	251	2.3%
Oracle	204	1.9%
Renesas Electronics	201	1.8%
MiTAC	191	1.7%
NVIDIA	188	1.7%
Wolfson Microelectronics	187	1.7%
(Consultant)	160	1.5%
NetApp	153	1.4%
Vision Engraving Systems	135	1.2%
Qualcomm	121	1.1%

Longtime readers of this series of articles will notice that Linaro has appeared in the top 5 kernel developer companies by number of contributions for the first time. This is due to the increased number of patches Linaro has been contributing, as well as the organization's wish to have the member company employees' contributions be counted as coming from Linaro, instead of the member company itself, as we had previously been doing.

A newcomer to the top 20 companies is Vision Engraving Systems, thanks to the Comedi development work from H. Hartley Sweeten. With his work, hopefully this subsystem can move out of the staging area of the kernel in a future release.

Other than the large jump from Linaro, the other companies in the top 25 are well known. Even NVIDIA—despite Linus's well-publicized, and in my opinion well-deserved, criticism of its Linux graphics driver development efforts—continues to be a large contributor to the kernel in the area of embedded processor support for its products. Texas Instruments, Samsung, MiTAC, Wolfson Microelectronics, Qualcomm, Renesas, and Nokia are also primarily focused in the embedded Linux area, showing the wide range of ongoing company support for Linux in embedded systems.

Work continues as usual

With the 3.5 kernel release, the number of contributors remains as high as previous releases, the rate of contributions is greater than ever (as measured by number of patches per day), and the rate of increase in the size of the kernel code remains the same as it has been for the past year. This shows that the kernel development community is still growing, and maintaining its incredibly rapid development cycle, ensuring that Linux remains the largest software engineering project ever.

Comments (18 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.5 released . (July 21, 2012)

• Greg KH: Linux 3.4.6 . (July 19, 2012)

• Steven Rostedt: 3.4.4-rt14 . (July 20, 2012)

• Steven Rostedt: 3.2.23-rt37 . (July 20, 2012)

• Greg KH: Linux 3.0.38 . (July 19, 2012)

• Steven Rostedt: 3.0.36-rt58 . (July 22, 2012)

Core kernel code

• Michel Lespinasse: augmented rbtree changes . (July 20, 2012)

• Ankit Jain: Make io_submit non-blocking . (July 25, 2012)

• Srivatsa S. Bhat: [RFC PATCH 0/6] CPU hotplug: Reverse invocation of notifiers during CPU hotplug . (July 25, 2012)

Development tools

• Steven Rostedt: [GIT PULL][v3.6] ftrace: Allow kprobes to work with ftace . (July 22, 2012)

• Jiri Olsa: perf: Add backtrace post dwarf unwind . (July 22, 2012)

Device drivers

• Johannes Winkelmann: hwmon: add support for Sensirion C1 sensor . (July 23, 2012)

• Laurent Pinchart: Renesas SH Mobile DRM driver . (July 23, 2012)

• Javier Martin: media: coda: Add driver for Coda video codec. . (July 23, 2012)

• Arun Kumar K: update MFC v4l2 driver to support MFC6.x . (July 23, 2012)

• Jiang Liu: provide interfaces to access PCIe capabilities registers . (July 25, 2012)

• Ming Lei: firmware loader: introduce cache/uncache firmware . (July 25, 2012)

• Alex Williamson: VFIO driver for v3.6 . (July 25, 2012)

Documentation

• Cyrill Gorcunov: kcmp.2 manpage and prctl.2 update . (July 23, 2012)

• Christoph Hellwig: XFS status update for June 2012 . (July 25, 2012)

Filesystems and block I/O

• Zheng Liu: ext4: extent status tree (step 1) . (July 23, 2012)

• Kent Overstreet: Bcache v15. (July 25, 2012)

• Cyrill Gorcunov: procfs fdinfo extension, new round . (July 25, 2012)

• Stanislav Kinsbursky: Lockd: grace period containerization . (July 25, 2012)

Memory management

• Mel Gorman: Memory management performance backports for -stable V2 . (July 23, 2012)

• Glauber Costa: memcg kmem limitation - slab. . (July 25, 2012)

Networking

• Michael S. Tsirkin: tun zerocopy support . (July 22, 2012)

• Yuchung Cheng: TCP Fast Open client . (July 23, 2012)

Architecture-specific

• Andres Salomon: OLPC: create a generic OLPC EC driver . (July 20, 2012)

• Matt Fleming: x86, efi: Handover Protocol . (July 23, 2012)

• dinguyen@altera.com: [RFC PATCHv4 0/1] arm: socfpga: Add initial support for Altera's SOCFPGA HW . (July 23, 2012)

• Cyril Chemparathy: Introducing the TI Keystone platform . (July 25, 2012)

• Kelvin Cheung: (Updated) MIPS: Add board support for Loongson1B . (July 25, 2012)

Virtualization and containers

• Yoshihiro YUNOMAE: virtio-trace: Support virtio-trace . (July 25, 2012)

Miscellaneous

• Pavel Emelyanov: Checkpoint-restore tool v0.1 . (July 23, 2012)

• David Howells: UAPI header file split . (July 23, 2012)

• Ian Kent: autofs 5.0.7 release . (July 25, 2012)

Page editor: Jonathan Corbet
Next page: Distributions>>