LWN.net Weekly Edition for March 19, 2009

Developments in the GCC world

As GCC nears its 4.4 release, there are a number of criteria that need to be met before it can be released. Those requirements—regressions requiring squashing—have been met, but things are still stalled. A number of issues were raised with the changes to the runtime library exemption that have caused the release, and a branch that will allow new development into the GCC tree, to be delayed until that is resolved. In the meantime, however, GCC development is hardly standing still, there are numerous interesting ideas floating around for new features.

Changing the runtime library exemption was meant to allow the creation of a plugin API for GCC, so that developers could add additional analysis or processing of a program as it is being transformed by the compiler. The Free Software Foundation has long been leery of allowing such a plugin mechanism because they feared that binary-only GCC plugins of various sorts might be the result. In January, though, the FSF announced that it would change the exemption—which allows proprietary programs to link to the GCC runtime library—in order to exclude code that has been processed by a non-GPL "compilation process". It is a bit of license trickery that will only allow plugins that are GPL-licensed.

Shortly after the new exception was released, there were some seemingly substantive issues raised on the gcc-devel mailing list. Ian Taylor neatly summarized the concerns, which break down into three separate issues:

• Code that does not use the runtime library and its interfaces at all might not be interpreted as included in the definition of an "Independent Module", which would then disallow it from being combined with the GCC runtime libraries. The code that fell outside of the "Independent Module" definition would not be affected directly, but combining it with other, compliant code that did use the runtime library would be disallowed.

• There are questions about whether Java byte code should be considered a "high-level, non-intermediate language". It is common to generate Java byte code using a non-GCC compiler, but then process it with gcj.

• There is also a hypothetical question about LLVM byte code and whether it should be considered a "high-level, non-intermediate language" as well.

Definitions of terms makes up the bulk of the runtime library exemption, so it is clearly important to get them right. The first issue in Taylor's summary seems like just an oversight—easily remedied—but the last two are a little more subtle.

By and large, the byte code produced as part of a compiler's operation is just an intermediate form that likely shouldn't be considered a "high-level, non-intermediate language", but Java and LLVM are a bit different. In both cases, the byte code is a documented language, somewhat higher level than assembly code, which, at least in the case of LLVM, is sometimes hand-written. For Java, non-GPL compilers are often used, but based on the current exemption language, the byte code from those compilers couldn't be combined with the GCC runtime libraries and distributed as a closed source program. Since LLVM is GPL-compatible, there are currently no issues combining its output with the GCC runtime, but Taylor is using it as another example of byte code being generated by non-GCC tools.

In addition to laying out the issues, Taylor recommends two possible ways forward. One of those is to clarify the difference between a compiler intermediate form and a "high-level, non-intermediate language". The other is to expand the definition of an eligible compilation process to allow any input to GCC that is created by a program that is not derived from GCC. Trying to make the former distinction seems difficult to pin down in any way that can't be abused down the road, so the second might be easier to implement. After all, the GCC developers can determine what kinds of input the compiler is willing to accept.

This may seem like license minutiae to some—and it is—but it is important to get it right. The FSF has chosen to go this route to prevent the—currently theoretical—problem of proprietary GCC plugins, so they need to ensure that they close any holes. As Dave Korn pointed out in another thread, releasing anything using an unclear license could create problems down the road:

Meanwhile, GCC developers have been working on reducing the regressions so that 4.4 can be released. Richard Guenther reported on March 13 that there were no priority 1 (P1) regressions, and less than 100 overall regressions, which would normally mean that a new branch for 4.4 would be created, with 4.5 development being added to the trunk. But, because of the runtime library exception questions, Richard Stallman asked the GCC Steering Committee (SC) to wait for those to be resolved before branching.

The delay has been met with some unhappiness amongst GCC hackers. Without a 4.4 release branch, interesting new features are still languishing in private developer branches. As Steven Bosscher put it:

Bosscher suggested releasing with the old exemption for 4.4 and fixing the problems in the 4.5 release. While that could work, it would seem that Stallman and the SC are willing to give FSF legal some time to clarify the exemption. In the end, though, the point is somewhat moot as there is, as yet, no plugin API available.

As part of the discussion of the new runtime library exception, Sean Callanan sparked a discussion about a plugin API by mentioning some of the plugins his research group had been working on. That led to various thoughts about the API, including a wiki page for the plugin project and one for the API itself. Diego Novillo has also created a branch to contain the plugin work.

The basic plan is to look at the existing plugins—most of which have implemented their own API—to extract requirements for a generalized API. In addition to the plugins mentioned by Callanan, there are others, including Mozilla's Dehydra C++ analysis tool, the Middle End Lisp Translator (MELT), which is a Lisp dialect that allows the creation of analysis and transformation plugins, and the MILEPOST self-optimizing compiler. Once the license issues shake out, it would appear that a plugin API won't be far behind.

There are other new features being discussed for GCC as well. Taylor has put out a proposal to support "split stacks" in GCC. The basic idea is to allow thread stacks to grow and shrink as needed, rather than be statically allocated at a particular size. Currently, applications that have enormous numbers of threads must give each one the worst-case stack size, even when it might go unused during the life of that thread. So, this could reduce memory usage, thus allowing more threads to run, but it would also alleviate the need for programmers to consider stack size for applications with thousands or millions of threads.

Another feature is link-time optimization (LTO), which is much further along than split stacks. Novillo put out a call for testers of the LTO branch in late January. There are a number of optimizations that can be performed when the linker has access to information about all of the compilation units. Currently, the linker only has access to the object files that are being collected into an executable, but LTO would put the GCC-internal representation (GIMPLE) into a special section of the object file. Then, at link time (but not actually implemented by the linker), various optimizations based on the state of the whole program could be performed. The kinds of optimizations that can be done are outlined in a paper [PDF] on "Whole Program Optimizations" (WHOPR) written by a number of GCC hackers including Taylor and Novillo.

While it is undoubtedly disappointing to delay GCC 4.4, hopefully the license issues will be worked out soon and the integration of GCC 4.5 can commence. In the interim, work on various features—many more than are described here—is proceeding. The FSF has always had a cautious approach to releases—witness the pace of Emacs—but sooner or later, we will see GCC 4.4, presumably with a licensing change. With luck, six months or so after that will come GCC 4.5 with some of these interesting new features.

Comments (41 posted)

Campsite offers plug-and-play freedom of the press

The non-profit Media Development Loan Fund (MDLF) released a major upgrade to its online journalism content management system (CMS) Campsite last week. Campsite 3.2 brings a flexible new plugin system and several improvements to search, templating, and content editing. MDLF describes Campsite as a CMS tailor-made for newspaper publishers — many of whom cannot afford expensive commercial products or the IT support required to heavily customize general-purpose CMS packages. Many of MDLF's target organizations are independent media in countries in transition, but the system is used in newsrooms all over the globe.

Campsite is deployed by more than 70 publications, many in Central and Eastern Europe near MDLF's Center for Advanced Media in Prague (CAMP), the office from which Campsite takes its name. But the software is also popular in Latin America, such as at El Periodico in Guatemala City, Guatemala.

MDLF's mission is to support independent journalists and media organizations, so that they are "strong enough to hold governments to account, expose corruption and drive systemic change." Founded in 1996, it provides funding to independent media in 23 countries, made possible through private donations and public grants. MDLF describes tools as the key investments for independent media, including printing presses, radio and television transmitters, and software. Campsite and the other CAMP projects grew out of MDLF's need to provide low-cost, open source software for new media outlets.

The feature set caters to the needs of professional news publications, which CAMP's Head of Research and Development Douglas Arellanes described as an "organic" relationship. "Campsite has been around since 2000, and nearly all of its features have come on the basis of real-world implementations."

Arellanes says journalists and editors on deadlines have better things to do than worry about CMS management, and that is the key difference that sets Campsite apart. He personally likes Wordpress and has great respect for the project, noting that:

Campsite's back-end allows an organization to replicate the newspaper workflow: authors can create and edit stories, submitting them to the editors when ready; editors can alter them, schedule them to run at predetermines times, change their visibility, move them between sections, and ultimately approve their publication. The system also handles administrative tasks like managing subscriptions, tracking article views, and moderating reader comments. A single back-end can also run multiple publications with different rules, schedules, layouts, and subscription lists and policies.

It can handle paid or unpaid subscriptions, supports embedded multimedia in articles, can integrate with the Phorum web forum package, and is multilingual from the ground up. The interface is available in 14 languages, and every individual article can be translated, whether as a one-off special, side-by-side in a single publication, or with entire sections or publications in separate languages.

Lead developer Mugur Rus said Campsite takes security very seriously. The administration interface can run over SSL and uses fine-grained role based privileges on all accounts. The system also uses CAPTCHA for comment forms, logs all events, and can use email notification to alert system administrators. Rus says Campsite itself has only been cracked once, and that it uses the standard security features of its free software based platform.

A peek at 3.2

Campsite is written in PHP and is designed to run on Apache servers using MySQL. The manual cites Apache 2.0.x, PHP 5.0, and MySQL 5.0 as the minimum version dependencies, and requires ImageMagick to handle graphics. In addition, you must run PHP as an Apache module, not as CGI, and there is a short list of required PHP directives to set up in the installation's php.ini file. Campsite runs on Linux, FreeBSD, Windows, and Mac OS X servers. No current Linux distributions are known to include Campsite, although from time to time users have shared their own home-brewed packages.

New in version 3.2 are improvements to search functionality, content editing, and site templating. The search improvements include an "advanced" search mode and increased support for non-ASCII languages that were problematic in earlier revisions. The story editor now uses the WYSIWYG TinyMCE component, with which administrators can customize the available markup features by privilege level. The Smarty-based templating system now supports functions, and developers have begun migrating the Campsite administration interface to Yahoo's open source AJAX interface library YUI.

The most significant feature is the debut of a plugin architecture that can extend the functionality of a publication but remain integrated with the core Campsite story content. For example, one of the three default plugins in Campsite 3.2 is a blogging module. Arellanes observed that most newspaper sites are just beginning to implement staff blogs, and that when they do they are typically stand-alone deployments of existing blog systems that sit isolated from the rest of the publication. Using Campsite's blog plugin, however, content is accessible via the same topic tags and search, whether it is a news article or a blog post.

The other two default plugins implement reader polls and a "live" interview system — in which readers can ask questions, an administrator can approve or reject them. The interviewee can then respond to them and have the answers posted automatically.

3.2 also uses a simpler installation process, inspired by Wordpress's five-minute install experience. Users now need only to expand the tar archive of the latest Campsite build, put the site contents into the folder they desire on their Web host, and follow the step-by-step setup process through the Web installer.

Moving forward

Arellanes said to expect a quick turnaround for the next stable release of Campsite, focusing on cache performance and overall speedups, and implementing a content API that will permit Campsite sites to make their story content available to outside users for aggregation or content mashups.

He also hopes to see more development from third-party coders on plugins using the new plugin API. "The open source model, as it concerns Campsite, has meant that we're really growing in terms of the number of features that are being contributed from non-core developers. We expect this trend to really pick up now that we've got a plugin architecture to work with."

The press release for Campsite 3.2 notes that independent media in developing countries have long operated on limited funds that preclude the expensive CMS solutions preferred by other organizations — the very situation that drives MDLF's software projects. But it also points out that newspapers in the "developed" world are facing a financial crisis of their own. Consequently, an open source CMS like Campsite makes more sense than ever.

Comments (5 posted)

Better than POSIX?

One might well think that, at this point, there has been sufficient discussion of the design decisions built into the ext4 "delayed allocation" feature and the user-space implications of those decisions. And perhaps that is true, but there should be room for a summary of the relevant issues. The key question has little to do with the details of filesystem design, and a lot to do with the kind of API that the Linux kernel should present to its user-space processes.

As has been well covered (and discussed) elsewhere, the delayed allocation feature found in the ext4 filesystem - and most other contemporary filesystems as well - has some real benefits for system performance. In many cases, delayed allocation can avoid the allocation of space on the physical medium (along with the associated I/O) entirely. For longer-lived data, delayed allocation allows the filesystem to optimize the placement of the data blocks, making subsequent accesses faster. But delayed allocation can, should the system crash, lead to the loss of the data for which space has not yet been allocated. Any filesystem may lose data if the system is unexpectedly yanked out from underneath it, but the changes in ext4 can lead to data loss in situations that, with ext3, appeared to be robust. This change looks much like a regression to many users.

Many electrons have been expended to justify the new, more uncertain ext4 situation. The POSIX specification says that no persistence is guaranteed for data which has not been explicitly sent to the media with the fsync() call. Applications which lose data on ext4 are not using the filesystem correctly and need to be fixed. The real problem is users running proprietary kernel modules which cause their systems to crash in the first place. And so on. All of these statements are true, at least to an extent.

But one might also argue that they are irrelevant.

Your editor recently became aware that Simson Garfinkel's Unix Hater's Handbook [PDF] is available online. To say that this book is an aggravating read is an understatement; much of it seems like childish poking at Unix by somebody who wishes that VMS (or some other proprietary system) had taken over the world. It's full of text like:

But behind the silly rhetoric are some real points that anybody concerned with the value of Unix-like systems should hear. Among them are the "worse is better" notion expressed by Richard Gabriel in 1991 - the year the Linux kernel was born. This charge states that Unix developers will choose implementation simplicity over correctness at the lower levels, even if it leads to application complexity (and lack of robustness) at the higher levels. The ability of a write() system call to succeed partially is given as an example; it forces every write() call to be coded within a loop which retries the operation until the kernel gets around to finishing the job. Developers who cut corners like that are left with an application which works most of the time, but which can fail silently in unexpected circumstances. It is far better, these people say, to solve the problem once at the kernel level so that applications can be simpler and more robust.

The ext4 situation can be seen as similar: any application developer who wants to be sure that data has made it to persistent storage must take extra care to inform the kernel that, yes, that data really does matter. Developers who skip that step will have applications which work - almost all the time. One could well argue that, again, the kernel should take the responsibility of ensuring correctness, freeing application developers from the need to worry about it.

The ext3 filesystem made no such guarantees, but, due to the way its features interact, ext3 provides something close to a persistence guarantee in most situations. An ext3 filesystem running under a default configuration will normally lose no more than five seconds worth of work in a crash, and, importantly, it is not prone to the creation of zero-length files in common scenarios. The ext4 filesystem withdrew that implicit guarantee; unpleasant surprises for users followed.

Now the ext4 developers are faced with a choice. They could stand by their changes, claiming that the loss of robustness is justified by increased performance and POSIX compliance. They could say that buggy applications need to be fixed, even if it turns out that very large numbers of applications need fixing. Or, instead, they could conclude that Linux should provide a higher level of reliability, regardless of how diligent any specific application developers might have been and regardless of what the standards say.

It should be said that the first choice is not entirely unreasonable. POSIX forms a sort of contract between user space and the kernel. When the kernel fails to provide POSIX-specified behavior, application developers are the first to complain. So perhaps they should not object when the kernel insists that they, too, live up to their end of the bargain. One could argue that applications which have been written according to the rules should not take a performance hit to make life easier for the rest. Besides, this is free software; it would not take that long to fix up the worst offenders.

[PULL QUOTE: There is a case to be made that this is a situation where the Linux kernel, in the interest of greater robustness throughout the system, should go beyond POSIX. END QUOTE] But fixing this kind of problem is a classic case of whack-a-mole: application developers will continually reintroduce similar bugs. The kernel developers have been very clear that they do not feel bound by POSIX when the standard is seen to make no sense. So POSIX certainly does not compel them to provide a lower level of filesystem data robustness than application developers would like to have. There is a case to be made that this is a situation where the Linux kernel, in the interest of greater robustness throughout the system, should go beyond POSIX.

The good news, of course, is that this has already happened. There is a set of patches queued for 2.6.30 which will provide ext3-like behavior in many of the situations that have created trouble for early ext4 users. Beyond that, the ext4 developers are considering an "allocate-on-commit" mount option which would force the completion of delayed allocations when the associated metadata is committed to disk, thus restoring ext3 semantics almost completely. Chances are good that distributors would enable such an option by default. There would be a performance penalty, but ext4 should still perform better than ext3, and one should not underestimate the performance costs associated with lost data.

In summary: the ext4 developers - like Linux developers in general - do care about their users. They may complain a bit about sloppy application developers, standards compliance, and proprietary kernel modules, but they'll do the right thing in the end.

One should also remember that ext4 is still a very young filesystem; it's not surprising that a few rough edges remain in places. It is unlikely that we have seen the last of them.

As a related issue, it has been suggested that the real problem is with the POSIX API, which does not make the expression of atomicity and durability requirements easy or natural. It is time, some say, to create an extended (or totally new) API which handles these issues better. That may well be true, but this is easier said than done. There are, of course, the difficulties in designing a new API to last for the next few decades; one assumes that we are up to that challenge. But will anybody use it? Consider Linus Torvalds's response to another suggestion for an API extension:

Application developers will be naturally apprehensive about using Linux-only interfaces. It is not clear that designing a new API which will gain acceptance beyond Linux is feasible at this time.

Your editor also points out, hesitantly, that Hans Reiser had designed - and implemented - all kinds of features designed to allow applications to use small files in a robust manner for the reiser4 filesystem. Interest in accepting those features was quite low even before Hans left the scene. There were a lot of reasons for this, including nervousness about a single-filesystem implementation and nervousness about dealing with Hans, but the addition of non-POSIX extensions was problematic in its own right (see this article for coverage of this discussion in 2004).

The real answer is probably not new APIs. It is probably a matter of building our filesystems to provide "good enough" robustness as a default, with much stronger guarantees available to developers who are willing to do the extra coding work. Such changes may come hard to filesystem hackers who have worked to create the fastest filesystem possible. But they will happen anyway; Linux is, in the end, written by and for its users.

Comments (141 posted)

Page editor: Jonathan Corbet

Security

Tin Hat: secured by running from RAM

Security measures generally fall into one of two categories. Either they are reactive like anti-virus software and respond to intrusions as they arise, or they are architectural like file permissions, and built to prevent intrusions in the first place. Tin Hat Linux, a project undertaken by D'Youville College in Buffalo, New York, falls firmly into the architectural category. By running a security-hardened operating system entirely in RAM — preferably while using encrypted drives for storage — Tin Hat attempts to deter attackers even if they have physical access to the system. The result is a distribution that trades off a challenging install and a slow system boot in favor of high security and fast applications, a mixture of weaknesses and strengths that may put off as many users as it attracts.

Anthony G. Basile, Chair of Information Technology at D'Youville College, says, "Tin Hat was started during a modern operating system course I taught. We started looking at security issues regarding the kernel and then worked outward. Eventually we came up with the idea of building our own distro to include these ideas."

Basile explains, "This is how we picture someone using Tin Hat. They boot off the CD (or pen drive). Then use a key file from a pen drive, plus passphrase, to unencrypt their drives. They do their work and then shutdown."

Preparing to use Tin Hat

Before you use Tin Hat, ideally you should have a working familiarity with Gentoo Linux — or, more specifically, with the Hardened Gentoo project, which Tin Hat is specifically based upon. Without this background knowledge, you may find yourself scrambling to understand some aspects of Tin Hat, particularly the instructions for encrypting drives. You also need a machine with at least four gigabytes of RAM, and preferably eight, since 3GB will be taken up by the operating system.

Understand, too, that the x86 and AMD64 images on the site are not live CDs; instead they are what the project refers to as "prebuild images." That is, they are not usable versions of the distribution in themselves, but the tools that you use to create the ISO image you will boot.

This installation process is very hands-on, and has little similarity to that of any modern major distribution. For this reason, you should also read the project's thorough but slightly chatty Quick Start documentation. Much, if not all of the documentation, is available after a prebuild image loads, but you should understand what you are undertaking before you start, and possibly research any concepts with which you are unfamiliar.

In fact, you may want to print out the Quick Start, or display it on another computer to assist you while you work, particularly if you are not at least vaguely familiar with basic administrative commands like ifconfig or common configuration files like xorg.conf. Some familiarity with filesystem encryption won't hurt, either.

Building and installing an image

The purpose of the prebuild CDs is to create a workable image customized for a specific machine. To use a prebuild, burn a CD from an image downloaded from the project site, then use it to start the system on which you plan to use Tin Hat. Either the CD will open a GNOME desktop or else you will have to enable video from the shell prompt, an option that seems most likely if you are using a large widescreen monitor.

Assuming you have some knowledge of GNU/Linux systems, you should have few problems building a custom image if you follow the steps in the Quick Start. The process consists of ten steps, the first five of which are basic configuration: Changing the default passwords that come with the prebuild CDs, and setting up video, networking, sound, and printing. Much of this hardware should be detected by default, but, if any is not, the Quick Start gives you detailed instructions on the steps you should take, as well as some basic suggestions for troubleshooting.

With the sixth step, the process becomes more esoteric. Since RAM is in short supply, you need to be sure that the amount of memory allotted to tmpfs, the RAM-based filesystem, still leaves you with at least a gigabyte or so to run programs. More, of course, is desirable.

In the next two steps, you have the option of installing and upgrading applications, and configuring a new kernel. These steps ensure that the image is current, but, between the two of them, can take well over an hour — likely more — to complete.

These steps are followed by the optional preparation of an encrypted filesystem on either a hard drive or a flash drive that you can use to store your personal files when using Tin Hat. As the Quick Start explains, because the entire operating system is loaded into RAM, you can encrypt an entire device, and have no need for an unencrypted /boot partition. "Encrypting the entire block device this way," the Quick Start suggests, "makes it difficult for an attacker to know whether he is looking at encrypted data or random noise" — although in practice, a competent cracker would probably assume encryption.

Tin Hat favors the use of loop-aes, but also supports the use of dm-crypt, although, presumably because of dm-crypt's emphasis on external drives, its use is not recommended for hard drives. The Quick Start, however, does suggest using dm-crypt's cryptsetup to prepare a flash drive before encrypting it. Fortunately, the detailed instructions are enough to guide even relative newcomers to cryptography through the necessary steps.

The final step is to build the ISO. In this step, you use shell scripts stored in /home/thuser/SAVE to remove entries from system logs, and build the image so that you can either burn it or install it on a flash drive.

Booting into Tin Hat

Because Tin Hat does not run from a hard drive, its boot time is about twice that of most distributions if you are booting from a flash drive. If you are booting from CD, the time can be up to four times slower than a distribution on a hard drive, depending on the speed of your CD drive. Most of this boot time is required to uncompress the squashfs filesystem of the image and copy it into RAM memory.

However, when Tin Hat finally loads, you are compensated for the slow boot time. True, Tin Hat's lightly branded GNOME desktop is unremarkable, and there is little to say about the software except that the selection and the currency is what you chose to make it when you created the image you are using. But, because applications run from RAM rather than the hard drive, they open and close at fractions of the usual speeds. For some, this speed may be almost as important a reason for running Tin Hat as security.

Running in RAM also has the advantage of reducing the inconvenience of at least one security feature. Although Tin Hat includes almost twenty services, only a third are started at boot time, in keeping with the basic security premise of minimizing them. On a system running from a hard drive, starting each additional service would generally involve a small but noticeable delay. However, because Tin Hat runs from RAM, the delay is no longer noticeable — even though you do still have to manually start most services.

But running in RAM also creates other problems. If you find that you neglected to include essential software in the image, you must either create and burn another image or install the extra software each time you boot into Tin Hat. You must remember, too, to save your work to a hard drive or removable device, or else you lose it. For full security, you should save to an encrypted filesystem, although the choice is yours

Even more significantly, if you are running the minimum 4 gigabytes, you are more or less confined to basic productivity and Internet use; Tin Hat itself uses three of those gigabytes, and does not use a swap file, simply killing programs when it runs out of memory. While for many people, this functionality is probably enough, some might find these limitations too high a price to pay for a system that disappears when you turn it off. But, if the system gets infected, you can do a completely clean reboot.

The limits of Tin Hat

Tin Hat is unquestionably more secure than the average distribution, although its installation is rough and ready by modern standards. However, as the project itself admits, Tin Hat is only a "baby step" towards a completely secure system. In fact, the project site suggests that a completely secure system is likely impossible.

For one thing, some compromises in security are necessary just to allow the X Windows System and GNOME to run. More importantly, Tin Hat has yet to provide protection against cold boot attacks, in which data in RAM is recovered even after rebooting, or msramdmp, an application that can recover the tmpfs root file system. Nor can any security measures, no matter how ingenious, protect against social engineering attacks or simple human carelessness, such as leaving a system unattended.

Still, within these limitations, Tin Hat remains an intriguing possibility for the security-conscious. For others, the usefulness of Tin Hat depends on whether they are willing to endure the assorted inconveniences in return for greater peace of mind. Many, I suspect, are not.

Comments (1 posted)

New vulnerabilities

avahi: denial of service

Package(s):	avahi	CVE #(s):	CVE-2009-0758
Created:	March 16, 2009	Updated:	September 29, 2010
Description:	From the Mandriva advisory: A security vulnerability has been identified and fixed in avahi which could allow remote attackers to cause a denial of service (network bandwidth and CPU consumption) via a crafted legacy unicast mDNS query packet (CVE-2009-0758).
Alerts:	Ubuntu USN-992-1 2010-09-29 Debian DSA-2086-1 2010-08-04 CentOS CESA-2010:0528 2010-07-14 Red Hat RHSA-2010:0528-01 2010-07-13 SuSE SUSE-SR:2010:002 2010-02-01 Gentoo 200904-10 2009-04-08 Mandriva MDVSA-2009:076 2009-03-13

Comments (none posted)

evolution-data-server: multiple vulnerabilities

Package(s):	evolution-data-server	CVE #(s):	CVE-2009-0547 CVE-2009-0582 CVE-2009-0587
Created:	March 16, 2009	Updated:	May 25, 2010
Description:	From the Red Hat advisory: Evolution Data Server did not properly check the Secure/Multipurpose Internet Mail Extensions (S/MIME) signatures used for public key encryption and signing of e-mail messages. An attacker could use this flaw to spoof a signature by modifying the text of the e-mail message displayed to the user. (CVE-2009-0547) It was discovered that Evolution Data Server did not properly validate NTLM (NT LAN Manager) authentication challenge packets. A malicious server using NTLM authentication could cause an application using Evolution Data Server to disclose portions of its memory or crash during user authentication. (CVE-2009-0582) Multiple integer overflow flaws which could cause heap-based buffer overflows were found in the Base64 encoding routines used by Evolution Data Server. This could cause an application using Evolution Data Server to crash, or, possibly, execute an arbitrary code when large untrusted data blocks were Base64-encoded. (CVE-2009-0587)
Alerts:	SuSE SUSE-SR:2010:012 2010-05-25 SuSE SUSE-SR:2010:011 2010-05-10 SuSE SUSE-SR:2010:006 2010-03-15 Debian DSA-1813-1 2009-06-08 SuSE SUSE-SR:2009:010 2009-05-12 Mandriva MDVSA-2009:078 2009-03-23 Fedora FEDORA-2009-2784 2009-03-18 Fedora FEDORA-2009-2792 2009-03-18 CentOS CESA-2009:0355 2009-03-18 CentOS CESA-2009:0354 2009-03-18 Ubuntu USN-733-1 2009-03-16 CentOS CESA-2009:0358 2009-03-17 Red Hat RHSA-2009:0358-01 2009-03-16 Red Hat RHSA-2009:0355-01 2009-03-16 Red Hat RHSA-2009:0354-01 2009-03-16

Comments (none posted)

ffmpeg: several vulnerabilities

Package(s):	ffmpeg, ffmpeg-debian	CVE #(s):	CVE-2008-4610 CVE-2009-0385
Created:	March 17, 2009	Updated:	December 17, 2009
Description:	From the Ubuntu advisory: It was discovered that FFmpeg did not correctly handle certain malformed Ogg Media (OGM) files. If a user were tricked into opening a crafted Ogg Media file, an attacker could cause the application using FFmpeg to crash, leading to a denial of service. It was discovered that FFmpeg did not correctly handle certain malformed 4X movie (4xm) files. If a user were tricked into opening a crafted 4xm file, an attacker could execute arbitrary code with the privileges of the user invoking the program.
Alerts:	Mandriva MDVSA-2009:335 2009-12-17 Mandriva MDVSA-2009:319 2009-12-05 Mandriva MDVSA-2009:297-1 2009-12-05 Mandriva MDVSA-2009:297 2009-11-13 Mandriva MDVSA-2009:298 2009-11-13 Mandriva MDVSA-2009:299 2009-11-13 Debian DSA-1782-1 2009-04-29 Debian DSA-1781-1 2009-04-29 Fedora FEDORA-2009-3433 2009-04-09 Fedora FEDORA-2009-3428 2009-04-09 Slackware SSA:2009-098-03 2009-04-08 Gentoo 200903-33 2009-03-19 Ubuntu USN-734-1 2009-03-16

Comments (none posted)

glib: attacker-supplied code execution

Package(s):	glib	CVE #(s):	CVE-2008-4316
Created:	March 13, 2009	Updated:	April 24, 2009
Description:	From the rPath advisory: Previous versions of glib contain a vulnerability in the base64 encode and decode functions which may result in executing attacker-supplied code when processing large strings. This vulnerability is present only through applications that accept user-supplied strings and process them with the base64 encode or decode functionality of glib.
Alerts:	Gentoo 200904-02 2009-04-03 Mandriva MDVSA-2009:085 2009-04-02 Fedora FEDORA-2009-2688 2009-03-13 Slackware SSA:2009-086-02 2009-03-30 Mandriva MDVSA-2009:080 2009-03-26 Red Hat RHSA-2009:0336-01 2009-03-24 Debian DSA-1747-1 2009-03-20 Ubuntu USN-738-1 2009-03-16 rPath rPSA-2009-0045-1 2009-03-12 Fedora FEDORA-2009-2657 2009-03-13 SuSE SUSE-SA:2009:026 2009-04-24

Comments (none posted)

gst-plugins-base: arbitrary code execution

Package(s):	gst-plugins-base0.10	CVE #(s):	CVE-2009-0586
Created:	March 17, 2009	Updated:	July 13, 2009
Description:	From the Ubuntu advisory: It was discovered that the Base64 decoding functions in GStreamer Base Plugins did not properly handle large images in Vorbis file tags. If a user were tricked into opening a specially crafted Vorbis file, an attacker could possibly execute arbitrary code with user privileges.
Alerts:	Gentoo 200907-11 2009-07-12 SuSE SUSE-SR:2009:009 2009-04-21 CentOS CESA-2009:0352 2009-04-08 Red Hat RHSA-2009:0352-01 2009-04-06 Mandriva MDVSA-2009:085 2009-04-02 Ubuntu USN-735-1 2009-03-16

Comments (none posted)

icu: content protection mechanism bypass

Package(s):	icu	CVE #(s):	CVE-2008-1036
Created:	March 12, 2009	Updated:	April 17, 2009
Description:	ICU has a content protection mechanism bypass vulnerability. From the Red Hat alert: A flaw was found in the way ICU processed certain, invalid, encoded data. If an application used ICU to decode malformed, multibyte, character data, it may have been possible to bypass certain content protection mechanisms, or display information in a manner misleading to the user. (CVE-2008-1036)
Alerts:	Debian DSA-1762-1 2009-04-02 Ubuntu USN-747-1 2009-03-26 Red Hat RHSA-2009:0296-01 2009-03-12 rPath rPSA-2009-0064-1 2009-04-17

Comments (none posted)

libpng: memory leak

Package(s):	libpng	CVE #(s):	CVE-2008-6218
Created:	March 13, 2009	Updated:	August 17, 2011
Description:	From the CVE entry: Memory leak in the png_handle_tEXt function in pngrutil.c in libpng before 1.2.33 rc02 and 1.4.0 beta36 allows context-dependent attackers to cause a denial of service (memory exhaustion) via a crafted PNG file.
Alerts:	openSUSE openSUSE-SU-2011:0915-1 2011-08-17 Mandriva MDVSA-2010:133 2010-07-15 Debian DSA-1750-1 2009-03-22 Gentoo 200903-28 2009-03-15 rPath rPSA-2009-0046-1 2009-03-12

Comments (none posted)

libsoup: arbitrary code execution

Package(s):	libsoup	CVE #(s):	CVE-2009-0585
Created:	March 16, 2009	Updated:	May 13, 2009
Description:	From the Red Hat advisory: An integer overflow flaw which caused a heap-based buffer overflow was discovered in libsoup's Base64 encoding routine. An attacker could use this flaw to crash, or, possibly, execute arbitrary code. This arbitrary code would execute with the privileges of the application using libsoup's Base64 routine to encode large, untrusted inputs. (CVE-2009-0585)
Alerts:	SuSE SUSE-SR:2009:010 2009-05-12 Mandriva MDVSA-2009:081 2009-03-27 Debian DSA-1748-1 2009-03-20 CentOS CESA-2009:0344 2009-03-17 Ubuntu USN-737-1 2009-03-16 Red Hat RHSA-2009:0344-01 2009-03-16

Comments (none posted)

mldonkey: path traversal

Package(s):	mldonkey	CVE #(s):	CVE-2009-0753
Created:	March 13, 2009	Updated:	March 24, 2009
Description:	From the Debian advisory: It has been discovered that mldonkey, a client for several P2P networks, allows attackers to download arbitrary files using crafted requests to the HTTP console.
Alerts:	Gentoo 200903-36 2009-03-23 Fedora FEDORA-2009-2758 2009-03-16 Fedora FEDORA-2009-2703 2009-03-16 Debian DSA-1739-1 2009-03-13

Comments (none posted)

mod_security: denial of service

Package(s):	mod_security	CVE #(s):
Created:	March 13, 2009	Updated:	March 18, 2009
Description:	From the Fedora advisory: Security fixes for potential denials of service when using PDF XSS protection as well as when parsing multipart requests. See the release announcement for more details.
Alerts:	Fedora FEDORA-2009-2654 2009-03-13 Fedora FEDORA-2009-2686 2009-03-13

Comments (none posted)

Opera: multiple vulnerabilities

Package(s):	opera	CVE #(s):	CVE-2008-5178 CVE-2008-5679 CVE-2008-5680 CVE-2008-5681 CVE-2008-5682 CVE-2008-5683
Created:	March 17, 2009	Updated:	March 18, 2009
Description:	From the Gentoo advisory: Multiple vulnerabilities were discovered in Opera: * Vitaly McLain reported a heap-based buffer overflow when processing host names in file:// URLs (CVE-2008-5178). * Alexios Fakos reported a vulnerability in the HTML parsing engine when processing web pages that trigger an invalid pointer calculation and heap corruption (CVE-2008-5679). * Red XIII reported that certain text-area contents can be manipulated to cause a buffer overflow (CVE-2008-5680). * David Bloom discovered that unspecified "scripted URLs" are not blocked during the feed preview (CVE-2008-5681). * Robert Swiecki of the Google Security Team reported a Cross-site scripting vulnerability (CVE-2008-5682). * An unspecified vulnerability reveals random data (CVE-2008-5683). * Tavis Ormandy of the Google Security Team reported a vulnerability when processing JPEG images that may corrupt memory (CVE pending).
Alerts:	Gentoo 200903-30 2009-03-16

Comments (none posted)

perl-MDK-Common: privilege escalation

Package(s):	perl-MDK-Common	CVE #(s):
Created:	March 13, 2009	Updated:	March 18, 2009
Description:	perl-MDK-Common has a privilege escalation vulnerability. From the Mandriva alert: The functions used to write strings into shell like configuration files by Mandriva tools were not taking care of some special characters. This could lead to some bugs (like wireless keys containing certain characters not working), and privilege escalation. This update fixes that issue by ensuring proper protection of strings.
Alerts:	Mandriva MDVSA-2009:072 2009-03-10

Comments (none posted)

TMSNC: arbitrary code execution

Package(s):	TMSNC	CVE #(s):	CVE-2008-2828
Created:	March 12, 2009	Updated:	March 18, 2009
Description:	TMSNC, a Text-based client for the MSN instant messaging protocol, has a buffer overflow vulnerability that can possibly used for the execution of arbitrary code when processing an instant message.
Alerts:	Gentoo 200903-26 2009-03-12

Comments (none posted)

weechat: denial of service

Package(s):	weechat	CVE #(s):	CVE-2009-0661
Created:	March 18, 2009	Updated:	April 6, 2009
Description:	The weechat IRC client has a bug in its color-handling code which can cause an out-of-bounds read and subsequent crash.
Alerts:	Gentoo 200904-04 2009-04-04 Fedora FEDORA-2009-2859 2009-03-20 Fedora FEDORA-2009-2861 2009-03-20 Debian DSA-1744-1 2009-03-18

Comments (none posted)

wesnoth: multiple vulnerabilities

Package(s):	wesnoth	CVE #(s):	CVE-2009-0366 CVE-2009-0367
Created:	March 12, 2009	Updated:	March 18, 2009
Description:	The game wesnoth has multiple vulnerabilities. From the Debian alert: CVE-2009-0366 Daniel Franke discovered that the wesnoth server is prone to a denial of service attack when receiving special crafted compressed data. CVE-2009-0367 Daniel Franke discovered that the sandbox implementation for the python AIs can be used to execute arbitrary python code on wesnoth clients. In order to prevent this issue, the python support has been disabled. A compatibility patch was included, so that the affected campagne is still working properly.
Alerts:	Debian DSA-1737-1 2009-03-11

Comments (none posted)

yaws: denial of service

Package(s):	yaws	CVE #(s):	CVE-2009-0751
Created:	March 16, 2009	Updated:	March 18, 2009
Description:	From the Debian advisory: It was discovered that yaws, a high performance HTTP 1.1 webserver, is prone to a denial of service attack via a request with a large HTTP header.
Alerts:	Debian DSA-1740-1 2009-03-14

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current 2.6 development kernel is 2.6.29-rc8, released by Linus on March 12. It has the usual set of fixes, and the patch rate does seem to be slowing down. "In fact, it seems to be stabilizing to the point where I'm hoping that we're approaching a final 2.6.29, and this might be the last -rc. We'll have to see." The short-form changelog is in the announcement, or see the full changelog for all the details.

As of this writing, nearly 200 changesets have been merged since the 2.6.29-rc8 release. They are mostly fixes, there is also a new mascot logo, a driver for ServerEngines BladeEngine 2 adapters, and a driver for the "Dave" Ethernet controller found on the Qong Board FPGA.

The current stable 2.6 kernel is 2.6.28.8, released on March 16; the 2.6.27.20 update was released at the same time. Both contain an even longer than usual list of important fixes.

Comments (none posted)

Kernel development news

Quotes of the week

-- Andrew Morton

-- Patrick McHardy

-- Linus Torvalds seeks to restart the checkpoint discussion

Comments (1 posted)

Ts'o: Delayed allocation and the zero-length file problem

Here's a posting from Ted Ts'o on ext4, delayed allocation, and robustness. "What’s the best path forward? For now, what I would recommend to Ubuntu gamers whose systems crash all the time and who want to use ext4, to use the nodelalloc mount option. I haven’t quantified what the performance penalty will be for this mode of operation, but the performance will be better than ext3, and at least this way they won’t have to worry about files getting lost as a result of delayed allocation. Long term, application writers who are worried about files getting lost on an unclean shutdown really should use fsync."

Comments (232 posted)

Garrett: ext4, application expectations and power management

Here is Matthew Garrett's contribution to the ext4 debate. Not for anybody who is offended by profanity. "Dear filesystem writers - application developers like writing lots of tiny files, because it makes a large number of things significantly easier. This is fine because sheer filesystem performance is not high on the list of priorities of a typical application developer. The answer is not 'Oh, you should all use sqlite'. If the only effective way to use your filesystem is to use a database instead, then that indicates that you have not written a filesystem that is useful to typical application developers who enjoy storing things in files rather than binary blobs that end up with an entirely different set of pathological behaviours."

Comments (164 posted)

The kernel gets a new logo

As everybody knows, only important fixes will be merged into the mainline kernel at this late stage of the development cycle. One of the fixes merged by Linus on March 17 was a high-resolution SVG image of "Tuz," the mascot of the 2009 linux.conf.au conference. Tuz, in his new home at Documentation/logo.svg, serves to remind the world of the difficulties faced by the Tasmanian devil and how the linux.conf.au attendees supported the effort to save this species from extinction.

Comments (15 posted)

Unioning file systems: Architecture, features, and design choices

A unioning file system combines the namespaces of two or more file systems together to produce a single merged namespace. This is useful for things like a live CD/DVD: you can union-mount a small, writable file system on top of a read-only DVD file system and have a usable system without needing to transfer the system files from the DVD to the root file system. Another use is to export a single read-only base file system via NFS to multiple clients, each with their own small, writable overlay file system union mounted on top. Another use might be a default set of files for a home directory.

Unioning file systems of various sorts have been around since at least the Translucent File Service (or File System), written around 1988 for SunOS. BSD has had union mounts since 4.4 BSD-Lite, around 1994, and Plan 9 implemented union directories in a similar time frame. The first prototype of a union-style file system for Linux was the Inheriting File System (IFS), written by Werner Almsberger for Linux 0.99 in 1993. It was later abandoned, as Werner writes, because "I looked back at what I did and was disgusted by its complexity." Later implementations for Linux include unionfs (2003), aufs (2006), and union mounts (2004), as well as FUSE prototypes and implementations of various forms. So how is it that in 2009, Linux still does not have an in-tree unioning file system?

The short answer is that none of the existing implementations meet the kernel maintainers' standards of simplicity, clarity, and correctness. What makes it so difficult to write a unioning file system that meets these standards? In this week's article, we'll review general unioning file system concepts, and common design problems and solutions. In a subsequent article, we'll review the top contenders for a mainline kernel unioning file system for Linux, as well as unioning file systems for other operating systems.

Unioning file systems implementation guidelines

First, let's define what a useful unioning file system implementation would look like. The basic desire is to have one read-only file system, and some sort of writable overlay on top. The overlay must persistently store changes and allow arbitrary manipulation of the combined namespace (including persistent effective removal of parts of the read-only namespace - "whiteouts"). The implementation should be fast enough for use as a root file system, and should require little or no modification of underlying file systems. It should be implemented in the kernel; FUSE-based file systems have many uses but aren't appropriate for many use cases (root file systems, embedded systems, high (or even moderate) file system performance, etc.).

A successful unioning file system will be an improvement (in terms of disk space used, cost of code maintenance, deployment time, etc.) over the alternatives - copying over all the files up front, clever division of file systems into multiple mount points, or writing an entire new on-disk file system. If we gain all the features of a unioning file system, but complicate the VFS code too much, we'll have a union file system at the cost of a slow, unmaintainable, buggy VFS implementation. If a union file system includes as much or more code than the underlying file system implementations, you start to wonder if supporting unioning in underlying file systems individually would be more maintainable.

One alternative to unioning file systems is the copy-on-write block device, often used for virtualized system images. A COW block device works for many applications, but for some the overhead of a block device is much higher than that of a unioning file system. Writes to a file system on a COW block device will produce many duplicated blocks as the bitmaps, journals, directory entries, etc. are written. A change that could be encapsulated in a few bytes of directory entry in a unioning file system could require several hundred kilobytes of change at the block level. Worse, changes to a block device tend only to grow; even with content-based block merging (not a common feature), two file systems which are logically nearly identical may differ by many megabytes at the block level. An important case is one in which you delete many files; with a unioning file system this will decrease the space needed to store differences. With a COW block device, used space will increase.

One problem that should NOT be solved by unioning file systems (and file systems in general) is source control. Modern source control systems are quite good compared to the crud we had to live with back in the 1990's. Source control software back then was so buggy and slow that it actually seemed like a good idea to implement parts of it in the kernel; indeed, some commercially successful source control systems still in use today require explicit file system support. Nowadays, many fast, reliable source control systems are available and it is generally agreed that complex and policy-heavy software should be implemented outside of the kernel. (Those who disagree are welcome to write gitfs.)

General concepts

Most unioning file systems share some general concepts. This section describes branches, whiteouts, opaque directories, and file copy up.

Branches are the various file systems that are unioned together. Branch access policies can be read-only, writable, or more complex variations which depend on the permissions of other branches. Branches are ordered or stacked; usually the branch "on top" is the writable branch and the branch "on the bottom" is read-only. Branches can sometimes be re-ordered, removed, added, or their permissions changed on the fly.

A commonly required feature is that when a particular directory entry is deleted from a writable branch, that directory entry should never appear again, even if it appears in a lower branch. That is, deleting a file named "x" should result in no file named "x" in that directory afterward, even if a file named "x" exists in a lower branch. Usually this is implemented through a combination of whiteouts and opaque directories. A whiteout is a directory entry that covers up all entries of a particular name from lower branches. An opaque directory does not allow any of the namespace from the lower branches to show through from that point downwards in the namespace.

When a file on a read-only branch is altered, it must be copied up to some writable branch. Copy up is usually triggered either when the file is opened with write permissions, or when the first write to the file's data or metadata occurs.

Common design problems

Unioning file systems often run into the same difficult design problems. Often, these problems only have a few obvious options with built-in tradeoffs, and unioning file systems can be characterized to some degree by which set of tradeoffs they choose. In this section, we'll review some of the top design problems and their common solutions.

Namespace pollution: Often, whiteouts, opaque directories, persistent inode numbers, and any other persistent file system data are stored in specially-named files. These files clutter up the available namespace and produce unexpected behavior. Some minor namespace pollution has been acceptable in UNIX as long as it is restricted to the top-level directory (think "/lost+found"), but namespace pollution on a per-directory or per-file basis is generally frowned on. Solutions to this problem include various ways of shuffling around the namespace pollution - moving it to a single subdirectory per directory or file system or storing it in an external file system - or not creating namespace pollution in the first place (which generally requires support from the underlying file system).

Whiteouts and opaque directories: While the concepts of whiteouts and opaque directories are fairly general, the implementation varies. The most common option is to create a directory entry with a special name, such as ".wh.<filename>" which indicates that the corresponding directory entry should never be returned. This can cause clashes with user file names and prevent stacking one union over another. It also makes directory removal time-consuming. An "empty" directory can have many thousands of whiteout entries which must be deleted before the rmdir() can complete. Sometimes directory removals are pushed off to a special work thread to improve rmdir() latency.

Another option is to create a special file or directory entry type to mark whiteout directory entries, and give whiteout directory entries a reserved inode number. The name in the directory entry itself is the same as the name being whited out and does not appear in directory listings. This form of whiteouts requires support from the underlying file system to store the necessary flags and from the file system utilities to accept the special entries. One more option is to make whiteout entries be hard links to special whiteout files, or symbolic links to reserved targets. The hard link solution requires handling the case of exceeding the maximum link count on the target file.

Directories can be marked opaque with either a flag in the directory inode (again requiring support from the underlying file system) or with a directory entry with a reserved name, like ".opaque".

Timing of directory copies: When a file is created on a writable branch in a directory that exists only on another branch, the entire directory path to that file must be present on the writable branch. The path can be created on-demand when the target file is copied, or each directory may be preemptively copied to the writable branch as it is opened. Creating paths on demand introduces complexity, locking issues, and additional latency on file writes. Creating paths on directory open simplifies code and improves latency on file write, but uses up additional (often negligible) space on the writable branch.

Kernel memory usage: Unioning file systems often introduce new data structures, extra copies of data structures, and a variety of cached metadata. For example, a union of two file systems may require three VFS inodes to be allocated for one logical object. The most common implementation of whiteouts and duplicate entry removal requires reading all directory entries from all branches into memory and constructing a coherent view of the directory there. If this cache is maintained across system calls, we have to worry about regenerating it when underlying branches change. When it is not cached, we have to reallocate memory and remerge the entries repeatedly. Either way, a lot of kernel memory must be allocated.

Code cleanliness: One of the main points of a unioning file system is to reuse existing file system code, with the expected benefits in ease of maintenance and future development in that code base. If the implementation is excessively complex or intrusive, the net effect will be a reduction in ease of maintenance and development. The sheer number and variety of Linux file systems (on-disk, in-memory, and pseudo) demonstrates the benefit of clean and simple file system interfaces.

Stack depth: The Linux kernel has a limited available stack depth. Calling internal file system functions from unexpected code paths or, worse yet, recursively, can easily result in exceeding the kernel stack limit. Some unioning file systems are implemented as stacked or layered file systems, which inherently add to stack depth.

Number of branches: Memory usage is often proportional to the number of branches in use. Branches may be limited to a compile-time maximum, but allocating enough memory for the maximum is prohibitive. Dynamic allocation and reallocation as branches are added and removed can be complex and introduce new opportunities for failure.

Coherency of changes: A common feature request is to allow modification of more than one branch at once. This requires some method of cache coherence between the various parts of the file system. Usually this method is absent or only partially correct. Users often request direct access to the file systems making up the branches of the union (instead of access through the unioning file system), a situation particularly difficult to deal with.

Dynamic branch management: Users often would like to add, remove, or change the policies of branches in a union while the union is still mounted. In trivial cases, this is a simple matter of parsing mount options and manipulating data structures, but may have major effects on any cache coherency implementation.

readdir() and friends: One of the great tragedies of the UNIX file system interface is the enshrinement of readdir(), telldir(), seekdir(), etc. family in the POSIX standard. An application may begin reading directory entries and pause at any time, restarting later from the "same" place in the directory. The kernel must give out 32-bit magic values which allow it to restart the readdir() from the point where it last stopped. Originally, this was implemented the same way as positions in a file: the directory entries were stored sequentially in a file and the number returned was the offset of the next directory entry from the beginning of the directory. Newer file systems use more complex schemes and the value returned is no longer a simple offset. To support readdir(), a unioning file system must merge the entries from lower file systems, remove duplicates and whiteouts, and create some sort of stable mapping that allows it to resume readdir() correctly. Support from userspace libraries can make this easier by caching the results in user memory.

Stable, unique inode numbers: NFS exports and some applications expect inode numbers of files not to change between mounts, reboots, etc. Applications also expect that files can be uniquely identified by a combination of the device id and the inode number. A unioning file system can't just copy up the inode number from the underlying file system since the same inode is very likely to be used on more than one file system. Unique (but not stable) inode numbers can be implemented fairly trivially, but stable inode numbers, require some sort of persistent state mapping files from the underlying file system to assigned inode numbers.

mmap(): mmap() is always the hard part. A good way to sound smart without knowing anything about file systems is to nod sagely and say, "What about mmap()?" mmap() on a file in a union file system is hard because the file may suddenly disappear and be replaced with another (copy up of a file on write from another process) or can be changed without the cooperation of the unioning file system (direct writes to the file systems backing branches). Some unioning file systems permit forms of mmap() which are not correctly implemented according to the POSIX standard.

rename() between branches: rename() is a convenient system call that allows atomic file renames on the same file system. Some unioning file systems try to emulate rename() across different branches. Others just return EXDEV, the error code for an attempted rename() across different file systems. Most applications can cope with the failure of rename() and fall back to a normal file copy.

File system implementation differences: Different file systems have different rules about permitted file names, allowed characters, name length, case sensitivity, extended attribute names and sizes, etc. The unioning file system wants to translate from one file system's rules to another. Some union file systems just restrict the types of file systems they support rather than implement complex translation code for uncommon use cases.

Multiple hard links: A file on a lower branch may have multiple hard links; that is, multiple paths point to the same inode. When a file with multiple hard links on a read-only branch is altered, strict UNIX semantics require finding all the other paths to that file and copying them to the writable branch as well. Unfortunately, UNIX file systems generally don't provide an efficient way to find all the paths to an inode. Some union file systems keep a list of inodes with undiscovered alternate paths and copy them over when a new path is accessed, others just ignore the problem. It's an open question as to how many applications depend on the correct behavior of hard links when used in this manner.

Permissions, owner, and timestamps: These attributes are often calculated using using a combination of underlying file system permissions, options specified on mount, the user and umask at time of mount, and branch management policies. Exact policies vary wildly.

Feature creep: Sometimes, the features provided by unioning file systems go beyond the actual needs of 99.9% of unioning use cases. This may be the result of fanciful user requests, or a "just because we can" approach - users only ask for two or three branches, but it's possible to support thousands of branches, so let's do it, or the code is structured such that all combinations of X, Y, and Z features are trivial to implement, even though users only want X and Y or Y and Z. Each feature may seem nearly free, but often ends up constraining the implementation or providing new opportunities for bugs. Focus is important.

Summary

Union file systems are inherently difficult to implement for many reasons, but much of the complexity and ugliness come from solving the following problems: whiteouts, readdir() support, stable inode numbers, and concurrent modifications to more than one branch at a time. Few of these problems have obviously superior solutions, only solutions with different tradeoffs.

Coming next

In the next article, we will review BSD union mounts, unionfs, aufs, and Linux union mounts. For each of these unioning file systems, we'll look at specific implementation details, including kernel data structures and their solutions to the various union file system problems described in this article. We'll also review the issues surrounding merging each solution into the mainline Linux kernel. We'll wrap up with general analysis of what does and doesn't work for unioning file systems, from both a purely technical and a software engineering point of view.

To be continued...

Acknowledgements

Many discussions with developers helped with the background of this article, but this article in particular benefited from discussions with Erez Zadok, Christoph Hellwig, and Jan Blunck.

Comments (26 posted)

A look at ftrace

There are quite a variety of tracing options for Linux, with SystemTap being the most prominent, but that particular solution has yet to become easily usable, at least partly due to its many dependencies on user-space configuration and tools. Another choice, which originally came from work on realtime Linux, is ftrace. Ftrace is a self-contained solution, requiring no user-space tools or support, that is useful for tracking down problems—not only in the kernel, but in its interactions with user space as well.

The name ftrace comes from "function tracer", which was its original purpose, but it can do more than that. Various additional tracers have been added to look at things like context switches, how long interrupts are disabled, how long it takes for high-priority tasks to run after they have been woken up, and so on. Its genesis in the realtime tree is evident in the tracers so far available, but ftrace also includes a plugin framework that allows new tracers to be added easily.

In a suitably configured kernel—one with CONFIG_FUNCTION_TRACER enabled—accessing ftrace is done through the debug filesystem. Typically, it is mounted this way:

    # mount -t debugfs nodev /debug

That creates a /debug/tracing subdirectory which is used to control ftrace and for getting output from the tool.

One chooses a particular tracer to use by writing to /debug/tracing/current_tracer—possibly after querying the tracers available by reading /debug/tracing/available_tracers. On a recent Fedora rawhide kernel, checking the tracers available results in:

    # cat /debug/tracing/available_tracers
    power wakeup irqsoff function sysprof sched_switch initcall nop

Tracing is then enabled or disabled by writing to /debug/tracing/tracing_enabled. Using the function tracer would be done as follows:

    # echo function >/debug/tracing/current_tracer
    # echo 1 >/debug/tracing/tracing_enabled
    ...some commands or activity to trace...
    # echo 0 >/debug/tracing/tracing_enabled

This produces a trace of each kernel function called, along with a timestamp, which will allow a kernel hacker to see what is going on inside the kernel.

Output from ftrace can be read from one of several files in the tracing directory:

• trace - contains human-readable output of the trace

• latency_trace - output from the same trace, but organized so that system latency issues can be diagnosed, also in human-readable format

• trace_pipe - contains the same output as trace, but is meant to be piped into a command. Unlike the other two, reading from this file consumes the output.

The ring buffer used by ftrace is a fixed size (governed by the buffer_size_kb file), so trace or latency_trace entries will eventually be overwritten.

Here are examples of the output from a function trace. The header helps to decode the various fields in the trace. This is what the output from trace looks like (heavily edited for brevity's sake):

    # tracer: function
    #
    #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
    #              | |       |          |         |
		bash-3330  [000]   147.799029: sys_open <-syscall_call
		bash-3330  [000]   147.799030: do_sys_open <-sys_open
		bash-3330  [000]   147.799030: getname <-do_sys_open
		bash-3330  [000]   147.799031: kmem_cache_alloc <-getname

While this is latency_trace output (similarly edited):

    # tracer: function
    #
    function latency trace v1.1.5 on 2.6.29-0.215.rc7.fc11.i586
    --------------------------------------------------------------------
     latency: 0 us, #120119/5425477, CPU#0 | (M:desktop VP:0, KP:0, SP:0 HP:0 #P:2)
	-----------------
	| task: -0 (uid:0 nice:0 policy:0 rt_prio:0)
	-----------------

    #                  _------=> CPU#            
    #                 / _-----=> irqs-off        
    #                | / _----=> need-resched    
    #                || / _---=> hardirq/softirq 
    #                ||| / _--=> preempt-depth   
    #                |||| /                      
    #                |||||     delay             
    #  cmd     pid   ||||| time  |   caller      
    #     \   /      |||||   \   |   /           
	bash-3330    0.... 3531221us : sys_open (syscall_call)
	bash-3330    0.... 3531222us : do_sys_open (sys_open)
	bash-3330    0.... 3531222us : getname (do_sys_open)
	bash-3330    0.... 3531223us : kmem_cache_alloc (getname)

Each line in the two output formats shows one function call, with one level of backtrace, along with the process name, process id, and which CPU the call was made on. The latency_trace output also provides information on whether interrupts have been disabled, whether a reschedule has been called for, whether its running in an interrupt context, and whether preemption has been disabled. The timestamp for the latency_trace output is relative to the start of the trace in microseconds; the space after the time, and before the colon, is a field that gets set to either '!' or '+' to call attention to especially long delays (in the example output it is always blank). Unfortunately, the header is a little misleading for where the "delay" and "caller" fields point.

The sysctl kernel.ftrace_enabled governs whether function entry is recorded as part of the trace. It can be turned on by using either of the following commands:

    # sysctl kernel.ftrace_enabled=1 
    or 
    # echo 1 >/proc/sys/kernel/ftrace_enabled

Without that, many of the tracers are essentially pointless.

Half-a-dozen different tracers are described in the voluminous Documentation/ftrace.txt that comes in the kernel source tree. In addition to the function tracer, there is the sched_switch tracer that tracks and reports on context switches in the kernel, showing process wakeups along with when they get scheduled. Each trace entry has a timestamp along with the priorities and states of the affected processes.

The nop tracer is not a tracer at all, but can be used to clear whatever tracer is currently active. There are another seven tracers in the mainline that have yet to make it into the documentation. In addition, there are even more tracers that are targeted for the 2.6.30 kernel.

There are four tracers that look for the maximum time spent in a particular state, recording that maximum time (in trace_max_latency) along with a trace of the functions called during that state. irqsoff looks for the longest time spent with interrupts disabled, while preemptoff looks for the maximum time spent with preemption turned off. Combining the two gives the preemptirqsoff tracer. The final tracer, wakeup looks for the maximum latency for the highest priority process to get scheduled after it has been woken up.

Each of those helps by reducing the amount of trace data that a kernel hacker needs to wade through. For kernels that are configured with CONFIG_DYNAMIC_FTRACE, there is another way to filter the trace data. Dynamic ftrace allows the user to select kernel functions that are either included or excluded from the tracing information collected. The available_filter_functions file lists the functions that can be filtered, while writing function names to set_ftrace_filter or set_ftrace_nofilter will include or exclude those functions. Those lists can be appended to by using the standard >> shell redirection operator. In addition, there is support for wildcards so that:

    echo 'hrtimer_*' >/debug/tracing/set_ftrace_filter

will only gather tracing information from high-resolution timer functions.

Other tracers available in the mainline kernel (i.e. what will become 2.6.29) include:

• power - traces the CPU power state transitions

• function_graph - similar to the function tracer, but traces both function entry and exit, which allows a call graph to be generated

• sysprof - periodically (governed by sysprof_sample_period) generate stack traces for the running process or kernel thread

• initcall - traces the entry and exit of initialization function calls during system boot

• branch - traces branch prediction and execution

• hw-branch-tracer - uses the Branch Target Stack (BTS) feature of x86 CPUs to trace branch execution

• mmiotrace - traces memory-mapped I/O

The production of human-readable output directly from the kernel recently led to some hard questions from Andrew Morton: "Why on earth do we keep on putting all these pretty-printers and pretty-parsers into the kernel? I mean, how hard is it for userspace to read a text file, do some basic substitutions and print it out again?" But, others argue that is precisely because of the ease of getting a readable trace directly from the kernel which makes ftrace so useful.

Morton's argument is that the fixed format is more difficult for scripts to parse and that the output is English-only. He contends that some kind of API would simplify things. Pointing to the latency_trace output as an example, he said:

Rather than needing some kind of user-space tool to interpret the ftrace output, a kernel hacker can quickly get what they need from the kernel itself. As Ingo Molnar pointed out, there may be no build environment available on the target machine, so having the trace output available directly is useful: "Self-sufficient kernel instrumentation is a key concept to usability". Further, he said, the output has been designed to be scriptable, but if that is not sufficient, there are options to produce raw output. Overall, he sees kernel pretty-printing as a needed feature:

The options that Molnar refers to are accessed via the trace_options file, which lists the current options settings when read, or it can be written to change options. Three separate options: raw, hex, and bin (along with several to determine which fields are included) control the format of the output. These can produce output in formats that may be easier for some tools to use. Not requiring any tool to get a useful trace, though, is seen as a very important part of ftrace, at least by some.

There are lots more options and features of ftrace than are covered here. The aforementioned ftrace.txt document is a good place to start for those interested in more details.

Beyond those tracers already in the mainline, there are another handful that are being readied for 2.6.30. This includes a new event tracer, which allow tracing various scheduling, locking, and interrupt handling events. Continuous statistics gathering for workqueues and branch prediction is included as well.

Ftrace is a useful tool that can provide excellent diagnostic information from within a running kernel. It is not a general-purpose tool, nor is it geared to people unfamiliar with the innards of the kernel, but it certainly has its uses. There is quite a bit of activity going on with ftrace these days, with numerous patches and enhancements floating around on linux-kernel. While it may not reduce Dtrace envy directly, it is a tool that many kernel hackers are using today.

Comments (10 posted)

Where 2.6.29 came from

We are very much creatures of tradition here at LWN. So, as the 2.6.29 development cycle nears its end, tradition drives us to take a look at where the code came from in this development cycle.

As of March 17, 11,610 non-merge changesets have been folded into the 2.6.29 kernel. These patches added an amazing 1,228,000 lines of code and documentation, while removing 401,000; the 2.6.29 kernel will have 827,000 more lines than its predecessor. Some 1420 1166 developers took part in this cycle. Your editor, sensing that this number represents a record, decided to look back at previous kernels:

Release	Developers
2.6.22	885
2.6.23	854
2.6.24	950
2.6.25	1124
2.6.26	1027
2.6.27	1022
2.6.28	1078
2.6.29	1166

It would seem that there is a clear trend here: the kernel development community has grown significantly over the last couple of years. The number of employers represented by these developers (175) has grown a little, but the uncertainties involved in associating developers with employers argue against reading too much into that particular number. Suffice to say that quite a few companies are supporting kernel development work.

Here are the individual developer statistics:

Most active 2.6.29 developers

By changesets Chris Mason 671 5.8% Takashi Iwai 173 1.5% Jaswinder Singh Rajput 158 1.4% Stephen Hemminger 154 1.3% Mike Frysinger 150 1.3% Christoph Hellwig 143 1.2% Ben Dooks 142 1.2% Alexey Dobriyan 138 1.2% Ingo Molnar 133 1.1% Rusty Russell 127 1.1% Steven Rostedt 110 0.9% Mauro Carvalho Chehab 109 0.9% Mark Brown 108 0.9% Sam Ravnborg 108 0.9% David S. Miller 107 0.9% Greg Kroah-Hartman 105 0.9% Harvey Harrison 101 0.9% David Howells 100 0.9% Russell King 93 0.8% Paul Mundt 87 0.7%

By changed lines Greg Kroah-Hartman 280883 20.8% Luis R. Rodriguez 71604 5.3% Chris Mason 69935 5.2% Daniel Krueger 56534 4.2% David Kiliani 41371 3.1% David Daney 18767 1.4% Solomon Peachy 17386 1.3% Robert Love 15262 1.1% Sujith 14703 1.1% Inaky Perez-Gonzalez 14388 1.1% David S. Miller 13422 1.0% Jesse Barnes 13036 1.0% Christoph Hellwig 12548 0.9% Michael Hennerich 12334 0.9% Subbu Seetharaman 12285 0.9% Jaswinder Singh Rajput 11651 0.9% Rusty Russell 10878 0.8% Ben Dooks 10809 0.8% David Schleef 10325 0.8% Mark Brown 9945 0.7%

Chris Mason comes out on top of the "changesets" category as a result of the Btrfs merge. It is a significant body of code, to be sure, but the changeset count is as high as it is because the entire Btrfs development history was merged. So we're seeing rather more than three months worth of work there. Takasi Iwai did a great deal of work in the ALSA subsystem, and in the Intel HDA driver in particular. Jaswinder Singh Rajput contributed quite a few patches of the cleanup variety. Stephen Hemminger's work consisted mainly of changing the network driver API, then fixing a long list of broken drivers. And Michael Frysinger contributed a lot of changes to the Blackfin architecture.

If one looks at the number of lines changed, the picture is a little different. As with 2.6.28, Greg Kroah-Hartman fed large amounts of crap (his word) into the -staging tree; that code does not retain the original author information within the git repository (though, of course, credits in the code itself are unchanged). Luis Rodriguez did a lot of work on Atheros wireless drivers and the cfg80211 subsystem; much of this work was associated with regulatory compliance support. Daniel Krueger achieved his place on the list by contributing a single patch: the Systec Electronic openPOWERLINK network stack. David Kiliani is another one-patch wonder; his was a driver for Meilhaus ME-IDS data collection cards. Daniel and David's patches both went into the -staging tree. So, three of the top five code contributors put their work in by way of -staging.

The associated employer statistics look like this:

Most active 2.6.29 employers

By changesets (None) 1612 13.9% (Unknown) 1378 11.9% Red Hat 1229 10.6% Oracle 992 8.5% IBM 749 6.5% Intel 686 5.9% Novell 632 5.4% (Consultant) 370 3.2% Analog Devices 282 2.4% Fujitsu 212 1.8% (Academia) 204 1.8% Renesas Technology 165 1.4% Nokia 163 1.4% Vyatta 154 1.3% Parallels 149 1.3% Simtec 138 1.2% Atheros Communications 131 1.1% AMD 130 1.1% Wolfson Microelectronics 104 0.9% SGI 100 0.9%

By lines changed Novell 306183 22.7% (Unknown) 197224 14.6% Atheros Communications 96202 7.1% Oracle 93846 7.0% (None) 92811 6.9% Red Hat 77087 5.7% Intel 62265 4.6% SYS TEC electronic GmbH 56534 4.2% Analog Devices 44659 3.3% IBM 40560 3.0% (Consultant) 28983 2.1% Cavium Networks 18767 1.4% Renesas Technology 16946 1.3% Nokia 11951 0.9% Simtec 10886 0.8% Broadcom 10314 0.8% Wolfson Microelectronics 10147 0.8% Freescale 8520 0.6% Chelsio 7738 0.6% QLogic 7322 0.5%

The employer numbers tend not to change radically from one release to the next; many of the same companies show up every time. New appearances this time include Vyatta (which supports Stephen Hemminger's work) and some companies (Simtec, SYS TEC, Cavium Networks) which contributed support for their own products.

The number of patches with Reviewed-by tags remains relatively small - less than 5% of the total. The top credited reviewers this time around are:

Developers with the most reviews
James Morris	64	20.2%
Dave Chinner	51	16.1%
Christoph Hellwig	39	12.3%
Andrew Morton	14	4.4%
Eric Sandeen	12	3.8%
Daisuke Nishimura	11	3.5%
KOSAKI Motohiro	10	3.2%
Matthew Wilcox	8	2.5%
WANG Cong	7	2.2%
Zhu, Yi	5	1.6%
KAMEZAWA Hiroyuki	5	1.6%
Eric Anholt	4	1.3%
Pekka Enberg	4	1.3%
Tomas Winkler	4	1.3%
Paul Menage	4	1.3%
Mike Christie	4	1.3%
Grant Grundler	4	1.3%

These numbers remain an artifact of how the reporting of reviews is done; certainly there is more code review than this going on. The same is true for reporting and testing credits. For 2.6.29, the numbers are:

Most credited 2.6.29 reporters and testers
Reported-by credits Randy Dunlap 13 3.8% Ingo Molnar 7 2.0% Li Zefan 6 1.7% Alexander Beregalov 5 1.5% Stephen Rothwell 5 1.5% Stefan Richter 4 1.2% Johannes Berg 4 1.2% Eric Sesterhenn 4 1.2% Kamalesh Babulal 4 1.2% Larry Finger 3 0.9% Linus Torvalds 3 0.9% Andrew Morton 3 0.9% Guennadi Liakhovetski 3 0.9% Huang Ying 3 0.9% Daisuke Nishimura 3 0.9% Meelis Roos 3 0.9% Geert Uytterhoeven 3 0.9%	Tested-by: credits Hin-Tak Leung 14 5.2% Mike Frysinger 7 2.6% Larry Finger 7 2.6% Ingo Molnar 6 2.2% Herton Ronaldo Krzesinski 6 2.2% Li Zefan 5 1.9% Daisuke Nishimura 4 1.5% KAMEZAWA Hiroyuki 4 1.5% Andrew Patterson 4 1.5% Meelis Roos 4 1.5% KOSAKI Motohiro 3 1.1% Stephen Gildea 3 1.1% Robert Jarzmik 3 1.1% Serge Hallyn 3 1.1% Eric Sesterhenn 3 1.1%

All told, 2.6.29 was one of the most active development cycles yet, with vast amounts of code finding its way into the kernel and a record number of developers participating. The development community might be justified in taking a rest after this much work, but the kernel process, it seems, never stops. There is already a lot of work waiting for the 2.6.30 merge window to open, at which point the whole cycle will start anew.

(Thanks, as always, to Greg Kroah-Hartman for his help in assembling these statistics.)

Comments (3 posted)

RCU: The Bloatwatch Edition

Introduction

Read-copy update (RCU) is a synchronization mechanism that was added to the Linux kernel in October of 2002. RCU improves scalability by allowing readers to execute concurrently with writers. In contrast, conventional locking primitives require that readers wait for ongoing writers and vice versa. RCU ensures read coherence by maintaining multiple versions of data structures and ensuring that they are not freed up until all pre-existing read-side critical sections complete. RCU relies on efficient and scalable mechanisms for publishing and reading new versions of an object, and also for deferring the collection of old versions. These mechanisms distribute the work among read and update paths in such a way as to make read paths extremely fast. In some cases (non-preemptable kernels), RCU's read-side primitives have zero overhead. RCU updates can be expensive, so RCU is in general best-suited to read-mostly workloads.

Although Classic RCU's memory footprint has been acceptable, hierarchical RCU has a memory footprint that is considerably larger. I was surprised to find that this additional memory consumption did not greatly concern the embedded Linux people I talked to, but then again, I certainly do not know everyone in the embedded Linux community, and the complexity of both hierarchical RCU, preemptable RCU, and the dynticks interface made the thought of a near-minimal kernel-capable RCU with a classic RCU API more interesting. In addition, a recent show of hands at linux.conf.au indicated that there is still interest in running Linux on small-memory single-CPU systems. Discussions with Josh Triplett identified an attractive optimization for the uniprocessor case, and so “RCU: The Bloatwatch Edition” was born.

Contents:

1. Review of RCU Fundamentals
2. Design of Tiny RCU
3. Tiny-RCU Code Walkthrough
4. Testing
5. Memory Footprint

These sections are followed by concluding remarks and the answers to the Quick Quizzes.

Review of RCU Fundamentals

This section is quite similar to its counterpart in the description of hierarchical RCU. People familiar with RCU semantics may wish to proceed directly to the next section.

In its most basic form, RCU is a way of waiting for things to finish. Of course, there are a great many other ways of waiting for things to finish, including reference counts, reader-writer locks, hashed locks, events, and so on. The great advantage of RCU is that it can wait for each of (say) 20,000 different things without having to explicitly track each and every one of them, and without having to worry about the performance degradation, scalability limitations, complex deadlock scenarios, and memory-leak hazards that are inherent in schemes using explicit tracking.

In RCU's case, the things waited on are called "RCU read-side critical sections". An RCU read-side critical section starts with an rcu_read_lock() primitive, and ends with a corresponding rcu_read_unlock() primitive. RCU read-side critical sections can be nested, and may contain pretty much any code, as long as that code does not explicitly block or sleep. If you abide by these conventions, you can use RCU to wait for pretty much any desired piece of code to complete.

Quick Quiz 1: But what if I need my RCU read-side critical section to sleep and block?

RCU accomplishes this feat by indirectly determining when these other things have finished, as has been described elsewhere for Classic RCU and realtime RCU.

In particular, as shown in the following figure, RCU is a way of waiting for pre-existing RCU read-side critical sections to completely finish, including memory operations executed by those critical sections.

However, note that RCU read-side critical sections that begin after the beginning of a given grace period can and will extend beyond the end of that grace period.

The following section gives a very high-level view of how the Tiny RCU implementation operates.

Design of Tiny RCU

The key restriction that enables a smaller and simpler RCU implementation is CONFIG_SMP=n, which means that any time the sole CPU passes through a quiescent state, a grace period has elapsed. In principle, the scheduler could simply invoke all pending RCU callbacks on each context switch, but in practice it is wise to maintain a degree of isolation between the scheduler and RCU. An arms-length relationship allows RCU callbacks to invoke appropriate scheduler functions (e.g., wake_up()) without potential deadlock issues. Therefore, the RCU core runs in softirq context.

This uniprocessor approach also simplifies the data structure, so that each flavor of RCU (rcu_ctrlblk and rcu_bh_ctrlblk) has the following structure:

  1 struct rcu_ctrlblk {
  2   long completed;
  3   struct rcu_head *rcucblist;
  4   struct rcu_head **donetail;
  5   struct rcu_head **curtail;
  6 };

The ->completed field is required only for the rcu_batches_completed() and rcu_batches_completed_bh() interfaces used by the RCU torture tests. The ->rcucblist field is the header for the list of RCU callbacks (rcu_head structures), the ->donetail field references the next pointer of the last rcu_head structure in the list whose grace period has completed, and the ->curtail field references the next pointer of the last rcu_head structure in the list.

The following figure shows a sample callback list that has two callbacks ready to invoke and a third callback still waiting for a grace period (or, equivalently on a uniprocessor system, for a quiescent state):

Quick Quiz 2: But we should be able to simply execute all callbacks at each quiescent state! So why bother with separate ->donetail and ->curtail sublists?

A block diagram of tiny RCU appears to the right; see this page for a full-size version.

The blue rectangles in this diagram represent functions making up tiny RCU, the blue rectangles with rounded corners represent data structures within tiny RCU, and the red rectangles represent other parts of the Linux kernel. Solid arrows represent function-call interfaces, while dashed arrows represent indirect invocation. The RCU read-side and list-manipulation primitives are not shown, nor are the interfaces specific to the "rcutorture" test module.

Like classic and hierarchical RCU, tiny RCU's grace-period detection is driven by context switches, the scheduling-clock interrupt (scheduler_tick()), and the idle loop. The scheduling-clock interrupt handler and idle loops contain code as follows:

    if (rcu_pending(cpu))
	rcu_check_callbacks(cpu, 0);

So if rcu_pending() indicates that the RCU core has any work to do, rcu_check_callbacks() is invoked, which in turn checks to see if the CPU is currently in a quiescent state, invoking either or both of rcu_qsctr_inc() and rcu_bh_qsctr_inc() as appropriate. These in turn invoke rcu_qsctr_help(), which, if there are RCU callbacks present, updates the callback lists to indicate that their grace period has elapsed and returns 1 to tell the caller to invoke raise_softirq(). At some later time, rcu_process_callbacks() will be invoked from softirq context, which, via __rcu_process_callbacks(), invokes all callbacks in rcu_ctrlblk and rcu_bh_ctrlblk.

Tiny RCU also interfaces to dynticks, albeit in a slightly different way than do the classic, preemptable, and hierarchical RCU implementations. Because there is but a single CPU, entering dynticks-idle mode is both a quiescent state and a grace period, allowing rcu_qsctr_inc() to directly invoke raise_softirq(). This direct invocation in turn means that rcu_needs_cpu() can simply return zero, since any callbacks are processed on the way into dynticks-idle state.

The call_rcu() and call_rcu_bh() interfaces queue callbacks via the __call_rcu() helper function. The synchronize_rcu() interface is a special case that will be described in the code walkthrough in the next section.

Interestingly enough, the general shape of the above block diagram applies to all RCU implementations. Of course, the processing is more complex in the CONFIG_SMP case, particularly in the __rcu_process_callbacks() area.

Tiny-RCU Code Walkthrough

This code walkthrough starts with tiny RCU's update-side API, then discusses the grace-period machinery, and finally the dynticks interface.

Update-Side API

The code for synchronize_rcu() is as follows:

  1 void synchronize_rcu(void)
  2 {
  3   unsigned long flags;
  4 
  5   local_irq_save(flags);
  6   rcu_ctrlblk.completed++;
  7   local_irq_restore(flags);
  8 }

This code merely increments the ->completed field with interrupts disabled. This works because synchronize_rcu() may only be invoked when it is legal to block, in other words, synchronize_rcu() cannot be called from within an RCU read-side critical section. Therefore, anywhere synchronize_rcu() may legally be invoked is guaranteed to be a quiescent state. Because quiescent states are also grace periods on uniprocessor systems, as noted by Josh Triplett, any call to synchronize_rcu() is automatically a grace period.

Quick Quiz 3: If all calls to synchronize_rcu() are automatically grace periods, why isn't synchronize_rcu() simply an empty function?

Quick Quiz 4: If all calls to synchronize_rcu() are automatically grace periods, why doesn't synchronize_rcu() process any pending RCU callbacks?

The following shows the code for the call_rcu() group of functions:

  1 static void __call_rcu(struct rcu_head *head,
  2            void (*func)(struct rcu_head *rcu),
  3            struct rcu_ctrlblk *rcp)
  4 {
  5   unsigned long flags;
  6 
  7   head->func = func;
  8   head->next = NULL;
  9   local_irq_save(flags);
 10   *rcp->curtail = head;
 11   rcp->curtail = &head->next;
 12   local_irq_restore(flags);
 13 }
 14 
 15 void call_rcu(struct rcu_head *head,
 16         void (*func)(struct rcu_head *rcu))
 17 {
 18   __call_rcu(head, func, &rcu_ctrlblk);
 19 }
 20 
 21 void call_rcu_bh(struct rcu_head *head,
 22      void (*func)(struct rcu_head *rcu))
 23 {
 24   __call_rcu(head, func, &rcu_bh_ctrlblk);
 25 }

Lines 1-13 show the code for __call_rcu(), which is a common-code helping function. Lines 7-8 initialize the specified rcu_head structure. Line 9 disables interrupts (and line 12 restores them), so that lines 10-11 can enqueue the callback undisturbed by interrupt handlers that might also invoke call_rcu().

Lines 15-19 and lines 21-25 are simple wrappers implementing call_rcu() (which enqueues the callback to rcu_ctrlblk) and call_rcu_bh() (which enqueues the callback to rcu_bh_ctrlblk), respectively. The callbacks are enqueued to the last segment of the queue, in other words, to the portion still waiting for a grace period to end.

Grace-Period Machinery

The lowest-level grace-period machinery is supplied by the rcu_qsctr_inc() family of interfaces that report passage through a quiescent state. These functions are implemented as follows:

  1 static int rcu_qsctr_help(struct rcu_ctrlblk *rcp)
  2 {
  3   if (rcp->rcucblist != NULL &&
  4       rcp->donetail != rcp->curtail) {
  5     rcp->donetail = rcp->curtail;
  6     return 1;
  7   }
  8   return 0;
  9 }
 10 
 11 void rcu_qsctr_inc(int cpu)
 12 {
 13   if (rcu_qsctr_help(&rcu_ctrlblk) ||
 14       rcu_qsctr_help(&rcu_bh_ctrlblk))
 15         raise_softirq(RCU_SOFTIRQ);
 16 }
 17 
 18 void rcu_bh_qsctr_inc(int cpu)
 19 {
 20   if (rcu_qsctr_help(&rcu_bh_ctrlblk))
 21         raise_softirq(RCU_SOFTIRQ);
 22 }

Lines 1-9 show rcu_qsctr_help(), which is a common-code helper function for rcu_qsctr_inc() (lines 11-16) and rcu_bh_qsctr_inc() (lines 18-22), which in turn report ``rcu'' and ``rcu_bh'' quiescent states, respectively.

Within rcu_qsctr_help, lines 3-4 check to see if there are any RCU callbacks within the specified rcu_ctrlblk structure that are still waiting for their grace period to complete, and, if so, line 5 updates the pointers so as to mark them as being ready to invoke, and line 6 returns non-zero in order to tell the caller to cause the callback-processing code to start running. Otherwise, line 8 returns zero to tell the caller that it is not necessary to process callbacks.

The rcu_qsctr_inc() function invokes rcu_qsctr_help() twice, once for rcu and once again for rcu_bh, and, if either indicates that callback processing is required, it also invokes raise_softirq(RCU_SOFTIRQ) to cause callback processing to be performed.

Quick Quiz 5: Why not simply directly call rcu_process_callbacks()?

The rcu_bh_qsctr_inc() function invokes rcu_qsctr_help(), but only for rcu_bh. If the return value indicates that callback processing is required, raise_softirq(RCU_SOFTIRQ) is invoked to make this happen.

Dynticks Interface

The tinyrcu dynticks interface is refreshingly simple, because uniprocessor systems need not worry about the dynticks-idle state of the nonexistent other CPUs. The code is as follows:

  1 void rcu_enter_nohz(void)
  2 {
  3   if (--dynticks_nesting == 0)
  4     rcu_qsctr_inc(0);
  5 }
  6 
  7 void rcu_exit_nohz(void)
  8 {
  9   dynticks_nesting++;
 10 }
 11 
 12 void rcu_irq_enter(void)
 13 {
 14   rcu_exit_nohz();
 15 }
 16 
 17 void rcu_irq_exit(void)
 18 {
 19   rcu_enter_nohz();
 20 }
 21 
 22 void rcu_nmi_enter(void)
 23 {
 24 }
 25 
 26 void rcu_nmi_exit(void)
 27 {
 28 }

The rcu_enter_nohz() function is shown on lines 1-5. Line 3 decrements the dynticks_nesting global variable, and, if the result is zero, the CPU has entered dynticks-idle mode, which is a quiescent state. (The dynticks_nesting global variable is initialized to 1.)

Quick Quiz 6: Why is only rcu_qsctr_inc() invoked? What about rcu_bh_qsctr_inc()?

The rcu_exit_nohz() function is shown on lines 7-10. This function simply increments the dynticks_nesting global variable.

As can be seen on lines 12-20, the rcu_irq_enter() and rcu_irq_exit() functions are simple wrappers around rcu_exit_nohz() and rcu_enter_nohz(), respectively. Please note that entering an interrupt handler exits dynticks-idle mode from an RCU viewpoint, and vice versa. This often-confusing relationship is maintained in the names of these functions.

Lines 22-28 show rcu_nmi_enter() and rcu_nmi_exit(), both of which are empty functions. Because we are running on a uniprocessor, we can safely ignore NMI handlers. The reason this is safe is that there cannot be any quiescent states on this CPU within an NMI handler, and there are no other CPUs to execute concurrent quiescent states.

Finally, the rcu_needs_cpu function (not shown) simply returns non-zero, indicating that RCU is always prepared for a given CPU to go into dynticks-idle mode.

Testing

Running the script included in the hierarchical RCU article shows that the CONFIG_NO_HZ kernel config parameter is important, and further investigation identifies the CONFIG_PREEMPT parameter as well.

This results in a nice small set of testing scenarios:

1. !CONFIG_PREEMPT and !CONFIG_NO_HZ
2. !CONFIG_PREEMPT and CONFIG_NO_HZ
3. CONFIG_PREEMPT and !CONFIG_NO_HZ
4. CONFIG_PREEMPT and CONFIG_NO_HZ

This is a welcome contrast to the situation for hierarchical RCU.

Memory Footprint

Simpler code means smaller memory footprint, as can be seen in the following table:

	Memory use
Implementation	text	data	bss	total	filename
Classic RCU	363	12	24	399	kernel/rcupdate.o
	1237	64	124	1425	kernel/rcuclassic.o
				1824	Classic RCU Total
Hierarchical RCU	363	12	24	399	kernel/rcupdate.o
	2344	240	184	2768	kernel/rcutree.o
				3167	Hierarchical RCU Total
Tiny RCU	294	12	24	330	kernel/rcupdate.o
	563	36	0	599	kernel/rcutiny.o
				929	Tiny RCU Total

Tiny RCU is about twice as small as Classic RCU (almost 900 bytes saved), and more than three times smaller than Hierarchical RCU (more than 2KB saved).

Conclusion

As RCU's capabilities have grown, so has its code size. This Bloatwatch edition of RCU forces this trend in reverse, producing a very small implementation via the !CONFIG_SMP restriction. This implementation also provides a minimal Linux-kernel-capable implementation that may provide a good starting point for people wishing to learn about RCU implementations.

Acknowledgements

I am grateful to Josh Triplett for the conversation that got this project started, to Ingo Molnar and David Howells for their encouragement to see this through, to Will Schmidt for his help getting this to human-readable state, and to Kathy Bennett for her support of this effort.

This work represents the view of the authors and does not necessarily represent the view of IBM.

Answers to Quick Quizzes

Quick Quiz 1: But what if I need my RCU read-side critical section to sleep and block?

Answer: A special form of RCU called "SRCU" does permit general sleeping in SRCU read-side critical sections. However, it is usually better to rework your RCU read-side critical section to avoid sleeping and blocking. One other exception is preemptable RCU, which allows RCU read-side critical sections to be preempted and to block while acquiring what in non-RT kernels would be spinlocks.

Back to Quick Quiz 1.

Quick Quiz 2: But we should be able to simply execute all callbacks at each quiescent state! Why to we need to separate ->donetail and ->curtail sublists?

Answer: Recall that the callbacks are not invoked directly from the scheduler, but rather from softirq context. It would be illegal to invoke callbacks that were registered after the quiescent state but before softirq commenced execution. Such callbacks could be registered from within irq handlers by invoking call_rcu(), and these irq handlers could be invoked between the time that the quiescent state occurred and the time that the softirq handler started executing.

Back to Quick Quiz 2.

Quick Quiz 3: If all calls to synchronize_rcu() are automatically grace periods, why isn't synchronize_rcu() simply an empty function?

Answer: If synchronize_rcu() were an empty function, then rcutorture would incorrectly conclude that RCU was not working, as it would appear to rcutorture that grace periods were never ending. Furthermore, synchronize_rcu() must at least prevent the compiler from reordering code across the call to synchronize_rcu().

Back to Quick Quiz 3.

Quick Quiz 4: If all calls to synchronize_rcu() are automatically grace periods, why doesn't synchronize_rcu() process any pending RCU callbacks?

Answer: Because there currently appears to be no need to do so, given that callbacks will be processed on the next context switch or the next scheduling-clock interrupt from either user-mode execution or from the idle loop. Should experience demonstrate that these are insufficient, then one approach would be to add a raise_softirq(RCU_SOFTIRQ) statement to synchronize_rcu().

Back to Quick Quiz 4.

Quick Quiz 5: Why not simply directly call rcu_process_callbacks()?

Answer: Deferring execution to the softirq environment disentangles RCU callback invocation from the scheduler, permitting the callbacks to freely invoke things like wake_up(). In addition, for rcu_bh, quiescent states might occur with locks held, and the RCU callbacks might well need to acquire these locks, potentially resulting in deadlock. In both cases, deferring to the softirq environment ensures a clean state for the callback.

Back to Quick Quiz 5.

Quick Quiz 6: Why is only rcu_qsctr_inc() invoked? What about rcu_bh_qsctr_inc()?

Answer: Because rcu_qsctr_inc() includes an implicit rcu_bh_qsctr_inc(), as can be seen from the code in the previous section.

Back to Quick Quiz 6.

Comments (4 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 2.6.29-rc8. (March 13, 2009)

• Thomas Gleixner: 2.6.29-rc7-rt1. (March 14, 2009)

• Greg KH: Linux 2.6.28.8. (March 17, 2009)

• Greg KH: Linux 2.6.27.20. (March 17, 2009)

Core kernel code

• Arun R Bharadwaj: timers: Framework for migration of timers. (March 16, 2009)

• Christopher Brannon: /dev/time for Linux, inspired by Plan 9. (March 16, 2009)

• David Brownell: handle_threaded_irq(). (March 18, 2009)

• Gautham R Shenoy: sched: Extend sched_mc/smt_framework. (March 18, 2009)

Development tools

• Peter Zijlstra: generic software counters -v2. (March 14, 2009)

• Paul Mackerras: perfcounters: provide a way to read the current value of interrupting counters. (March 17, 2009)

• Paul Mackerras: perfcounters: add an mmap method to allow userspace to read hardware counters. (March 17, 2009)

• Frederic Weisbecker: Syscalls tracing. (March 14, 2009)

• Mathieu Desnoyers: TIF_KERNEL_TRACE thread flags. (March 15, 2009)

• Steven Rostedt: [GIT PULL] memory saving ring buffer for tip/tracing/ftrace. (March 16, 2009)

• Akinobu Mita: introduce debug_kmap_atomic. (March 16, 2009)

• Larry Woodman: mm tracepoints. (March 17, 2009)

Device drivers

• Nicholas A. Bellinger: : lio-core-2.6.git updated to v2.6.29-rc8. (March 14, 2009)

• Rafael J. Wysocki: PM: Rework suspend-resume ordering to avoid problems with shared interrupts (updated 2x). (March 14, 2009)

• Bartlomiej Zolnierkiewicz: IDE subsystem status report. (March 14, 2009)

• Jing Huang: bfa: Brocade BFA FC SCSI driver (overview). (March 15, 2009)

• Anton Vorontsov: FSL eSDHC support. (March 16, 2009)

• Kevin Hilman: new memory_accessor interface and usage. (March 16, 2009)

• Robert Love: Open-FCoE fixes and features for 2.6.30 merge window. (March 18, 2009)

Documentation

• Greg KH: The Linux Staging tree, what it is and is not.. (March 18, 2009)

Filesystems and block I/O

• Jens Axboe: Per-bdi writeback flusher threads. (March 12, 2009)

• Tejun Heo: block: cleanup patches, take#2. (March 16, 2009)

• J. R. Okajima: AuFS2 source files. (March 16, 2009)

• Theodore Ts'o: ext4: Add support for data=alloc_on_commit mode. (March 17, 2009)

• Mark Fasheh: Ocfs2 updates for 2.6.30. (March 18, 2009)

• swhiteho@redhat.com: GFS2 Pre-pull patch posting. (March 18, 2009)

• Boaz Harrosh: exofs for kernel 2.6.30. (March 18, 2009)

• Dan Williams: Asynchronous raid6 acceleration (part 1 of 2). (March 18, 2009)

Memory management

• Mel Gorman: Cleanup and optimise the page allocator V3. (March 16, 2009)

• Nitin Gupta: : compressed in-memory swapping. (March 17, 2009)

Networking

• Cyrill Gorcunov: introduce netfilter conntrack protos pernet functionality v2. (March 16, 2009)

• Patrick McHardy: : First release of nftables. (March 18, 2009)

Architecture-specific

• Paul Mundt: What's in sh-2.6.git for 2.6.30?. (March 18, 2009)

• monstr@monstr.eu: Microblaze linux support. (March 18, 2009)

Security-related

• James Morris: What's coming in the security subsystem. (March 18, 2009)

• Serge E. Hallyn: [RFC PATCH] CAP_FS_MASK: add some long-missing capabilities to cap_fs_mask. (March 18, 2009)

Virtualization and containers

• KAMEZAWA Hiroyuki: memcg softlimit (Another one) v4. (March 12, 2009)

• Vivek Goyal: IO Controller. (March 12, 2009)

• Jeremy Fitzhardinge: 2.6.30 Xen core updates. (March 14, 2009)

• Jeremy Fitzhardinge: Xen dom0 core changes. (March 14, 2009)

• Jeremy Fitzhardinge: Xen dom0 apic changes. (March 14, 2009)

• Jeremy Fitzhardinge: Xen dom0 hardware access support. (March 14, 2009)

• Sukadev Bhattiprolu: /proc/pid/checkpointable. (March 17, 2009)

Benchmarks and bugs

• Rafael J. Wysocki: 2.6.29-rc8: Reported regressions from 2.6.28. (March 15, 2009)

• Rafael J. Wysocki: 2.6.29-rc8: Reported regressions 2.6.27 -> 2.6.28. (March 15, 2009)

Miscellaneous

• Kay Sievers: udev 140 release. (March 16, 2009)

• Hans de Goede: libv4l release: 0.5.9. (March 16, 2009)

• Karel Zak: util-linux-ng v2.15-rc1. (March 18, 2009)

Page editor: Jonathan Corbet

Distributions

News and Editorials

VectorLinux 6.0

VectorLinux never really gets the attention it deserves. I've tested several releases over the years and I always walk away impressed. Vector has original tools, stability, and high performance (at least on my hardware). Version 6.0 was released on February 21 and Vector shows really well.

VectorLinux is based on the venerable Slackware Linux distribution but most of the packages are built from source by the developers. They use Slackware packages for some of the core system primarily as a time saver. Vector developers have been working hard to automate their build system so they can be less reliant on Slackware. Nearly 90% of their distribution is original work this release. Even if Slackware were to suddenly disappear, VectorLinux would continue to be developed. The original work in Vector sets it apart from other derivatives, some of which can be seen in the system tools and new graphical installer.

The Graphical Installer

For years Vector's installer approximated Slackware's famous ncurses interface, but the first thing you'll notice in this release is the new graphical installer. It was written from scratch with Gambas, but resembles the Red Hat/Fedora Anaconda in appearance and operation. It walks the user through installation and configuration much like any other Linux installer. There isn't an individual package selection step, but one can choose from six main categories such as Development, Kernel source, or X11.

The process proceeded fairly smoothly on my HP Pavilion laptop with one exception. Following the installation step a dialogue box appeared stating that the next step would configure the bootloader. I clicked <Okay> to close the pop-up and <Next> on the main screen. With no visual feedback, the button did not depress, I clicked <Next> a couple more times. I came to believe the installer had frozen. However, it had not. For some reason Lilo is used instead of the more common GRUB, and while my back was turned Lilo was installed over the existing GRUB in the MBR. So it appears my clicks registered with the back-end even though no visual feedback was received. Had I known I would have chosen to skip the bootloader install in favor of adding VectorLinux the existing menu.lst. It was also necessary to add "noapic" to Vector's kernel boot parameters in order to prevent X11 from freezing the system when starting X. I rarely need to apply this option with more recent Linux releases using Xorg 7.3 or newer. It's only fair to state that the original button issue may not happen on a machine with more RAM.

The Interface & Software

VectorLinux is available in several formats and packages. The Deluxe and SOHO enhanced versions are available at the Vector CD Store for around $20. These contain extra packages such as KDE, OpenOffice.org, multimedia and other software, bundled on the CDs for convenience. There are installable Live CDs available for download as well as the Standard Version.

I've had the honor of testing the enhanced versions in the past, but I chose to review the no-cost Standard Edition this release. The Standard Edition, targeting general desktop users, features the Xfce and LXDE desktops along with many other applications. VectorLinux includes security and other package updates. Users can track these from the news page, which is also available as an rss feed.

In my opinion both desktops are decorated attractively and tastefully. They are similar for consistency. Handy links, plugins, and applets reside in the panel. I find the menus tidy and easy to navigate.

The Standard version software stack consists of graphic applications like Inkscape, GQview and The GIMP; multimedia apps such as XMMS, MPlayer, VLC, and Xine; and AbiWord, Scribus, and Gnumeric for office tasks. For internet browsing you can choose from Firefox, Opera, Dillo, or SeaMonkey. There is also Pidgin for instant messaging and Deluge to download bittorrent files. KDE and lots of other software are available for install using the standard Slackware package manager or Gslapt front-end. Multimedia plugins and codecs are included as well as extra device drivers. Beware though as some of these are proprietary or patent encumbered (in places where software patents are recognized).

Under the hood is Xorg Server 1.4.2, GCC 4.2.3, and Linux kernel 2.6.27.12. The i586 optimized kernel is built from scratch and includes several popular patches. These include Bootsplash, which allows what I think is a pretty boot sequence. TuxOnIce (formerly suspend2) is used for a suspend-to-disk implementation. This out-of-tree implementation is alleged to be more reliable and to have a better user interface.

The VectorLinux Tools

The Vector tools consist of small lightweight utilities that make up in function for what they may lack in glitz. Most hardware is auto-detected and auto-configured these days with any distribution, but if you'd like to customize your settings you could start with VASM. VASM is Vector's control center, which is a wrapper that conveniently places most of their tools under one interface.

In the VectorLinux Control Center you will find broad categories listed that when clicked will bring up tools that will help you configure that portion of the system. For example, under Hardware you will find tools to view or configure the hardware devices of your computer. One of these is ALSA Sound, which brings up the ALSA Configurator that will attempt to detect and allow one to setup their sound card if needed. Keyboard Map will allow one to change the default keymap. Mouse provides for the configuration of tablets, pointing pens, and joysticks as well as serial, usb, and PS/2 mice. Mounter offers a choice in removable device management tools from either the more commonly used HAL or Vector's own VL-HOT. Hardware Set lets you choose which subsystems to start at boot such as PNP, ACPI, or Wireless.

The Network category offers configurations for various networking tasks such as the firewall through the Firestarter setup wizard. Another lets users set or change the machine's hostname and DNS, while yet another helps to setup dial-up modems. The Wireless icon starts the wireless network configuration tool. I found Wicd to be reliable in establishing connections.

Under XWindows you can configure the Display Manager (graphical login preference), Boot Mode (text or graphical, server or desktop) and Video Configuration. Setups for CUPS, SAMBA, and Init Runlevels can be found under Service.

Within the Filesystem panel CFdisk can be started, filesystem formats can be done, partitions can be added to /etc/fstab, and the Bootloader can be installed or restored. Also found is the Backup System. With this tool one can schedule regular daily backups as a cronjob or initiate manual backup or restore operations. Vector has original BASH scripts backupsys and backupsync which can perform full or incremental functions.

Utilities include vcpufreq, Vpackager, Hardinfo, VLFreeMem, and Gslapt. Vcpufreq is a simple but effective means to set the current or default clock speed of your CPU. This is especially useful for saving battery life of mobile computers or reducing heat and wear. Vpackager allows users to build software packages. Hardinfo packs a lot of system information within a small tidy application and also allows the user to compare their system performance to others. VLFreeMem frees up memory by dropping the page, inode, and dentry caches when clicked by echoing "3" into the /proc/sys/vm/drop_caches file.

Gslapt is the front-end to Slackware's package management system. It functions much like Synaptic or any other graphical package manager. You can search for the software of interest, mark it for installation, and click to apply. It, as well as its dependencies, will be installed with some nice graphical feedback to keep you informed. It will also track and install package and security updates as found on the preconfigured Vector software repositories.

Closing Thoughts

VectorLinux strives to provide a high-performance, easy-to-use, stable Linux desktop distribution. It always performs very well for me on any machine tested. I was impressed with its out-of-the-box support for my Broadcom wireless ethernet chip that usually requires Ndiswrapper and Windows drivers. The included NVIDIA graphic drivers and multimedia support were a nice convenience. It does everything I require with no hassles. As usual, I enjoyed using VectorLinux and wonder why it isn't more popular.

Comments (3 posted)

New Releases

Jaunty Alpha 6 released

The sixth alpha release of Ubuntu's Jaunty Jackalope (v9.04) is available for testing. The following editions are available for download: Ubuntu Desktop, Server, Netbook Remix, MID, Ubuntu Education Edition, Kubuntu, Xubuntu, UbuntuStudio and Mythbuntu. Click below for more information.

Full Story (comments: none)

Distribution News

Debian GNU/Linux

Debian and keyboard mapping copyrights

The debian-legal list often serves as an early warning system for interesting copyright issues. Currently, the project is discussing the fate of a French Dvorak keyboard mapping which is evidently subject to a copyright claim by one Francis Leboutte, who says it can only be used under a Creative commons noncommercial, no-derivatives license. That license, being non-free, would exclude the this keyboard mapping (which is implemented entirely in free software) from the Debian distribution. But there is a clear question here: can a keyboard layout be copyrighted at all? Click below for the debian-legal discussion; this issue has also come up in the freedesktop.org bug tracker.

Full Story (comments: 24)

Debian Policy 3.8.1 released

Debian Policy 3.8.1 is out with the first round of Policy changes for the squeeze development cycle. "The next time you upload a package, please check it against the new version of Policy and update Standards-Version in debian/control accordingly."

Full Story (comments: none)

Section changes in the archive

Joerg Jaspert looks at some changes in the Debian archives as the new development cycle gets underway. "As Lenny is finally released, and we are early in the cycle for squeeze, now is the best time to do some long-needed changes to our archive. Much of what we are currently doing is not visible to you as a user of the archive, but this change is."

Full Story (comments: none)

Changes in ftpmaster

Joerg Jaspert reports on some changes to Debian's FTPMaster team. They are still looking for new people to join the team. Click below for the requirements.

Full Story (comments: none)

Kicking off Squeeze: initial tidbits from the RMs

The Release Managers have some additional information on the squeeze kickoff. For those not following along at home, squeeze is currently Debian-testing and it will be the new version Debian when it's ready.

Full Story (comments: none)

Fedora

Fedora Board Recap 2009-03-10

Click below for a recap of the March 10 meeting of the Fedora Advisory Board (click below). Topics include Trademark Followup and What is Fedora?.

Full Story (comments: none)

Gentoo Linux

Initial Council Meeting Summary for 12 March 2009

A summary of the March 12, 2009 meeting of the Gentoo council is available (click below). Topics include EAPI-3 Proposals, Technical Agenda Items and more.

Full Story (comments: none)

SUSE Linux and openSUSE

openSUSE-GNOME Team meeting minutes from 05-March-2009

The minutes from the openSUSE GNOME team meeting for March 5, 2009 are available (click below). Topics include 11.2 planning, random bugs, GNOME:Factory Status and mailing list threads.

Full Story (comments: none)

New Distributions

SYS

SYS is an easy to install and use distribution. There are plenty of applications for beginners and advanced users. This non-commercial distribution aims to make Linux and open software easy for all kinds of users. SYS packages can be used for Slackware and VectorLinux too. More information about SYS can be found on LinuxQuestions.

Full Story (comments: none)

Distribution Newsletters

Ubuntu Weekly Newsletter #133

The Ubuntu Weekly Newsletter for the week ending March 14, 2009 is out. "In this issue we cover: Jaunty Jackalope Alpha 6 released, Ubuntu Testing Day: Notify-OSD, MOTU Council News, Hug Day: March 19th, Ubuntu Florida Rocks Florida Linux Show, gmail filters for bug email, Inside Launchpad AJAX sprint, Ubuntu Forums Beginners Team, Ubuntu Forums Interview: Connor Imes, Canonical QA Desktop Automation Sprint, Ubuntu Women project status, Ubuntu Drupal 6.2.0 released, Ubuntu Podcast #21, Server Team Meeting Minutes, US Teams Meeting Minutes, and much, much more!"

Full Story (comments: none)

OpenSUSE Weekly News/62

This issue of the OpenSUSE Weekly News covers openSUSE Trademark Guidelines Released, Ken Yap: How to clone a VirtualBox Linux VM to a real machine (v0.9), 11.2 Roadmap and Fixed Release Cycle for openSUSE, Rupert Horstkötter: OSF Status Report #3 and several other topics.

Comments (none posted)

Fedora Weekly News #167

The Fedora Weekly News for the week ending March 15, 2009 is available. "This week Announcements links to some helpful "Fedora Classroom IRC Logs" including one on using mock. PlanetFedora big-ups several posts including David Lutterkort's explanation of how config file manipulation can be simplified using XPath to query Augeas. Marketing notes the latest meeting log. QualityAssurance reports that the last "Test Day" for Intel graphics chipsets was valuable and advertizes the next for XFCE. Developments summarizes the "Provenpackager Re-Seed" and watches aghast as the "Mono Conflagration Jumps to Blog". Artwork listens to some ideas about a "Theme Song". Virtualization reports the Xen "dom0 Kernel: Better, Still not Ready" and KVM "Snapshot Support Discussed"."

Full Story (comments: none)

DistroWatch Weekly, Issue 294

The DistroWatch Weekly for March 16, 2009 is out. "This week we follow up on our introduction to Logical Volume Management with a walk-through guide on using it with some of the most popular Linux distributions. In the news section, Slackware presents a massive first update of its 'current' tree since version 12.2 was released three months ago, Ubuntu packages Plymouth, the kernel-based mode-setting splash program for 'Karmic Koala', openSUSE battles to restore its main download server after a hardware fault, Tiny Core presents a new version of its revolutionary 10 MB desktop distro, and Debian delivers the first glimpses of 'Squeeze', the distribution's next stable release. The issue also focuses on the ext4 file system after reports of possible data loss - is it safe to use it? Finally, two links to interviews with Debian and Ubuntu developers, as well as TuxRadar's nostalgic look at the best desktop environments of 2000."

Comments (none posted)

Newsletters and articles of interest

High-security, RAMdisk Linux rev'd (DesktopLinux)

DesktopLinux looks at Tin Hat Linux. "An interesting physical security-focused Linux distribution was upgraded a couple of days ago. Tin Hat Linux reportedly takes a Vista-like five minutes to boot, because its whole filesystem is decrypted and loaded from an optical drive onto a RAMdisk (tmpfs). But after that, it's likely Puppy-fast!"

Comments (none posted)

Tiny Core: A Linux desktop in just 10MB (Tectonic)

Tectonic takes a look at Tiny Core Linux. "In what must be one of the smallest Linux versions ever, Tiny Core Linux is a portable Linux desktop that is just 10MB in size. With the 10MB disk image in hand users can run it from a CD, USB drive or just as a minimal hard drive installation."

Comments (none posted)

Page editor: Rebecca Sobol

Development

Stretching and squishing audio files with rubberband

The Rubber Band Library and the associated rubberband audio processing utility have been developed by Breakfast Quay, creators of the dssi-vst VST audio plugin adapter. The software is dual-licensed, with GPLv2-licensed source code and a commercially licensed edition. The Rubber Band description states:

The features document discusses the capabilities of the library in more detail and the Rubber Band Technical notes explains some of the underlying software methods:

See the Rubber Band API documentation for more information on the library's components. Version 1.3 of Rubber Band Library was announced on March 16, 2009, no new features were added but a number of build and runtime bugs were fixed.

The source code for rubberband can be downloaded and built, or a pre-compiled executable is available for the curious (and trusting). The Usage document explains the various options available for rubberband. A simple example batch run such as: rubberband -T1.5 infile.wav outfile.wav takes infile.wav and produces outfile.wav with a tempo that's 1.5 times faster, but with the same pitch. There are a number of additional options that can be used to select the other types of audio conversions and to fine-tune the processing methods. For the curious, a number of example audio files are available for listening to.

A number of interesting uses for rubberband come to mind. The software could be used in radio production for making those annoying compressed legalese notices that show up at the end of pharmaceutical ads. It could be used to greatly speed up the time it takes to listen to audio books and podcasts, or the producers of those files could use it for compressing their sound files to reduce bandwidth usage. Recordings can be pitch-shifted to correct the speed problems that can be caused by older analog recording equipment. Those who are learning a new language could use the software to slow down the speed of the foreign speech. The software could be useful for producing musical sound effects or amusing answering machine messages.

Rubberband addresses a fairly narrow range of audio processing needs, but gives the user more control when compared to built-in pitch-shifting and/or tempo-shifting functions found in software such as the popular Audacity and Ardour audio editors. It makes a useful addition to a collection of open-source audio processing utilities.

Comments (11 posted)

System Applications

Database Software

PostgreSQL 2009-03-16 security update

Security updates dated 2009-03-16 for the PostgreSQL DBMS have been announced. "The PostgreSQL Project today released minor versions updating all active branches of the PostgreSQL object-relational database system, including versions 8.3.7, 8.2.13, 8.1.17, 8.0.21 and 7.4.25. This release fixes a denial of service issue with encoding conversion, and all users should update their installations at the next reasonable opportunity."

Full Story (comments: none)

PostgreSQL Weekly News

The March 15, 2009 edition of the PostgreSQL Weekly News is online with the latest PostgreSQL DBMS articles and resources.

Full Story (comments: none)

Interoperability

Samba 3.3.2 is available

Version 3.3.2 of Samba has been announced. This is a bug fix release, see the release notes for details.

Comments (none posted)

Virtualization Software

TightVNC: 1.3.10 released (SourceForge)

Version 1.3.10 of TightVNC has been announced. "TightVNC is a free remote control package derived from the popular VNC software. With TightVNC, you can see the desktop of a remote machine and control it with your local mouse and keyboard, just like you would do it sitting in the front of that computer. Version 1.3.10 fixes various bugs, includes a number of impovements and introduces a version of TightVNC for U3 flash drives."

Comments (none posted)

Web Site Development

Midgard 8.09.4 released

Version 8.09.4 of the Midgard web framework is out. "The Midgard Project has released the fourth maintenance release of Midgard 8.09 Ragnaroek LTS. Ragnaroek LTS is a Long Term Support version of the free software content management framework. The 8.09.4 "Kaiku" release focuses on API and architecture cleanups in order to ease transition from Midgard 1.x series API to Midgard 2.x APIs."

Full Story (comments: none)

Midgard2 9.03.0 beta 1 released

Version 9.03.0 beta 1 of the Midgard2 web framework is out. "The first beta of Midgard2 9.03 is targeted at web framework and desktop developers. It provides a comprehensive set of content repository APIs that can be used to build replicated information applications that share their information using a common storage layer and replication tools."

Full Story (comments: none)

Pyjamas 0.5 released

Version 0.5 of Pyjamas, a web widget set and Python-to-Javascript compiler has been announced. "The 0.5 release is a significant functionality update. Additions have been made to the core python-to-javascript compiler such as support for exceptions (try / except), lambda, and a debug option to track and print run-time stack traces, in cases where the deployment of a javascript script debugger is difficult or impossible (IE6 running under Wine). Also, the code-generator has undergone a reorganisation, and now has much better support for local and global variable, function, class and module scopes."

Full Story (comments: none)

web2py 1.59 released

Version 1.59 of web2py a Python-based web framework, has been announced. "New features In 1.59: - DB2 support. - new Service interface to expose functions in CSV, XML, JSON, JSONRPC, XMLRPC and AMFRPC (Flash//Flex). - optional Amy editor (instead of edit_area) has keyboard shortcuts and autocompletion."

Full Story (comments: none)

Desktop Applications

Audio Applications

Sonic Visualiser: v1.5 now available (SourceForge)

Version 1.5 of Sonic Visualiser has been announced. "Sonic Visualiser is an application for inspecting and analysing the contents of music audio files. It combines powerful waveform and spectral visualisation tools with automated feature extraction plugins and annotation capabilities. Version 1.5 of Sonic Visualiser is now available. This release contains a small number of new features and a larger number of bug fixes over the previous 1.4 release."

Comments (none posted)

Desktop Environments

GNOME 2.26 released

GNOME 2.26 is out. "Released on schedule, to the day, GNOME 2.26 builds on top of a long series of successful six months releases to offer the best experience to users and developers." There's lots of new stuff in this release, including a new disc burning application, better multiple-monitor support, and, believe it or not, a volume control widget which is actually integrated with the underlying (pulseaudio) sound system. See the release notes for details.

Full Story (comments: 3)

GNOME Software Announcements

The following new GNOME software has been announced this week:

• Accerciser 1.6.0 (bug fixes and translation work)
• Anjuta 2.26.0 (bug fixes and translation work)
• atk 1.26.0 (bug fixes, documentation and translation work)
• at-spi 1.26.0 (translation work)
• Brasero 2.26.0 (bug fixes, documentation and translation work)
• Cheese 2.26.0 (bug fix and translation work)
• Clutter 0.9.2 (bug fixes and code cleanup)
• CrunchyFrog 0.3.4 (new features, bug fixes and translation work)
• Dasher 4.10.0 (bug fixes and translation work)
• Deskbar-Applet 2.26.0 (translation work)
• Ekiga 3.2.0 (new features, bug fixes and translation work)
• Evince 2.26.0 (bug fixes and translation work)
• Evolution 2.26 (new features and bug fixes)
• Eye of GNOME 2.26.0 (bug fixes and code cleanup)
• gbrainy 1.1 (new features, bug fixes and translation work)
• GCalctool 5.26.0 (bug fixes, documentation and translation work)
• Gdl 2.26.0 (bug fixes)
• gedit 2.26.0 (new features and bug fixes)
• GENIUS 1.0.6 (new features, bug fixes and translation work)
• Glade 3.6.0 (bug fixes and translation work)
• GLib 2.20 (new features)
• glibmm 2.20.0 (documentation work)
• Glom 1.10.0 (bug fixes and performance improvements)
• gnome-applets 2.26.0 (bug fixes and translation work)
• gnome-control-center 2.26.0 (bug fixes and translation work)
• Gnome Games 2.26.0 (bug fixes and translation work)
• gnome-keyring 2.26.0 (bug fixes and code cleanup)
• gnome-mag 0.15.5 (translation work)
• GNOME Media 2.26.0 (new features, bug fixes and translation work)
• GNOME Nettool 2.26.0 (bug fixes, code cleanup and translation work)
• GNOME Power Manager 2.26.0 (translation work)
• gnome-settings-daemon 2.26.0 (build fixes and translation work)
• GNOME Utilities 2.26.0 (bug fixes and translation work)
• GOK 2.26.0 (translation work)
• GTK+ 2.16.0 (new features)
• Gtk2-Perl 2.26.0 (new features)
• gtk-engines 2.18.0 (translation work)
• Libgda 3.99.14 (new feature and bug fixes)
• Libgda 4.0.0 (new features)
• libgnomekdb 2.26.0 (bug fixes)
• libvtemm 0.20.0 (new feature)
• Metacity 2.26.0 (bug fixes and translation work)
• Metacity 2.27.0 (bug fixes, code cleanup and translation work)
• MonoDevelop 2.0 Beta 2 (bug fixes and performance improvements)
• moserial 1.0.5 (bug fixes)
• moserial 1.0.7 (new features and translation work)
• mousetweaks 2.26.0 (translation work)
• Orca 2.26.0 (new features, bug fixes and translation work)
• Pango 1.24.0 (new features, bug fixes and documentation work)
• PyGooCanvas 0.14.0 (new feature and documentation work)
• Sabayon 2.25.0 (bug fixes and translation work)
• seahorse 2.26.0 (new features and bug fixes)
• seahorse-plugins 2.26.0 (bug fixes and code cleanup)
• Sphinx 0.6 and 0.5.2, beta 1 (new features and bug fixes)

You can find more new GNOME software releases at gnomefiles.org.

Comments (none posted)

KDE Commit-Digest

The February 22, 2009 edition of the KDE Commit-Digest is out. The content summary says: "Experimentation with recording presentations in Okular. Mobipocket format support added to Okular, Strigi, and the thumbnailer. Ability to configure gestures in the "Hotkeys" KControl module. Start of a metadata editor and other general work in Plasmate. Support for multiple collections, and HTML emails in the LionMail Plasma widget. A "maintenance" tab, with reorganised status displays and operations added to the folder properties dialog in KMail. Initial check-in of Qt QObject - GTK GObject bridge..."

Comments (none posted)

KDE Software Announcements

The following new KDE software has been announced this week:

• 2ManDVD 0.6.1 (bug fixes)
• 2ManDVD 0.6.2 (bug fixes)
• digiKam 0.9.5 (bug fixes and translation work)
• digiKam 0.10.0 (first KDE4 release)
• FlvToMp3 1.1.2 (translation work)
• GamCat 0.27 (unspecified)
• Kipi-plugins 0.2.0 for KDE 4 (unspecified)
• KRadio4 for KDE 4.2 snapshot-2009-03-18 (new feature and bug fixes)
• KShowmail 3.3.1 (bug fix and translation work)
• Navigation plasmoids set. 0.4 (new features)
• NixieClock 0.5 (unspecified)
• rekonq 0.0.4 (new features and bug fixes)
• YouTube Service 0.1 (initial release)

You can find more new KDE software releases at kde-apps.org.

Comments (none posted)

Xorg Software Announcements

The following new Xorg software has been announced this week:

• xf86-video-ati 6.12.0 (new features and bug fixes)

More information can be found on the X.Org Foundation wiki.

Comments (none posted)

Desktop Publishing

AsciiDoc 8.4.1 released

Version 8.4.1 of AsciiDoc has been announced, it features a new Python API. "AsciiDoc is an uncomplicated text document format for writing articles, documentation, manuals, books and UNIX man pages. AsciiDoc files can be translated to HTML, XHTML and DocBook (articles, books and refentry documents) using the asciidoc(1) command. DocBook can be post-processed to presentation formats such as HTML, PDF, DVI, roff and Postscript using the a2x toolchain wrapper and readily available Open Source tools."

Full Story (comments: none)

LyX 1.6.2 announced

Version 1.6.2 of the LyX document processor has been announced. "This is the second maintenance release in the 1.6.x series. The release fixes a large number of major and critical bugs that were reported by users of LyX 1.6.0 and 1.6.1. Most notably, three bugs that might lead to dataloss were detected and resolved. Furthermore, a number of crashes,file conversion and LaTeX output problems are fixed."

Full Story (comments: none)

Electronics

GNU Radio release 3.2rc1 is available

Release 3.2rc1 of GNU Radio, a software-defined radio platform, is out. "This release has had a large number of updates since 3.2rc0, touching a variety of components--please use the Trac website to peruse the revision log on the trunk for details. It incorporates all development trunk updates and bug fixes up to r10622 (today)."

Full Story (comments: none)

Open Arms (IEEE Spectrum Online)

The IEEE Spectrum Online magazine has a detailed article about the state of prosthetics for amputees. The perception is that the state of the art for replacement limbs has advanced greatly, but the reality is somewhat different. The article's author lost his arm in Iraq and is now working on development of prosthetic limbs as part of a free software (and hardware) project. "The MyOpen project on the Open Prosthetics site is working on ways to create a product that can serve both a niche and a mass market. MyOpen, an open-hardware signal-processing board that will be compatible with APL’s arm, will also serve as a universal controller for video games. We want creative people to get their hands on this device and push it to the limits. We want this device to enhance the experience of playing video games, and in doing so, push prosthetics innovation into the 21st century." (Thanks to Evgeny Stambulchik)

Comments (3 posted)

Encryption Software

pyDes: 2.0.0 released (SourceForge)

Version 2.0.0 of pyDes has been announced. "A pure python module which implements the DES and Triple-DES encryption algorithms. Triple DES is either DES-EDE3 with a 24 byte key, or DES-EDE2 with a 16 byte key. This release updates the pyDes module to be compatible with both Python 2 and Python 3 interpreters."

Comments (1 posted)

Graphics

Why glxgears is slower with Kernel Modesetting (and why it doesn't matter)

Fedora QA team member Will Woods looks at recent changes to add kernel modesetting to Fedora in a post on his blog. People were noticing a sizeable decrease in the frame rate of glxgears, believing that it was a good general measure of 3D performance. "[...] glxgears is rendering an insanely simple scene - so simple that the actual 3D rendering time is basically zero. So the only thing glxgears really tests is the performance of glXSwapBuffers() - basically, how fast we can push render buffers into the card. This operation is slower with DRI2, but - roughly speaking - unless it was an order of magnitude slower (e.g. glxgears drops from 1000FPS to under 100FPS) it wouldn't make any real difference." One of the tests he recommends for 3D performance is the always amusing Extreme TuxRacer.

Comments (34 posted)

GUI Packages

Urwid 0.9.8.4 released

Version 0.9.8.4 of Urwid, a console-based user interface library, has been announced. "This is a maintenance release that adds compatibility with Python 2.6 and fixes a number of bugs."

Full Story (comments: none)

wxWidgets 2.8.10 released

Version 2.8.10 of wxWidgets has been announced. "The wxWidgets team is pleased to announce a new wxWidgets release. The latest stable release in the 2.8 series contains a number of bug fixes, and upgrading is recommended for all wxWidgets users."

Comments (none posted)

Multimedia

Elisa Media Center 0.5.32 released

Version 0.5.32 of Elisa Media Center has been announced. "New features include a brand new picture slideshow with cool transition effects."

Full Story (comments: none)

Music Applications

First version of alsa-midi-latency-test released

The initial release of alsa-midi-latency-test is available. "The console application aims to become a Linux drop-in-replacement for the Windows application "MidiTest"".

Full Story (comments: none)

Canorus 0.7 beta released

Version 0.7 beta of Canorus, a free music score editor, has been announced. "After more than a year of an active development Canorus team is proud to announce immediate release of Canorus version 0.7beta."

Full Story (comments: none)

Qtractor 0.4.0 released

Version 0.4.0 of Qtractor, an audio/MIDI multi-track sequencer application, has been announced. "It's been quite a while since last time (Fluffy Doll on Xmas:). However, Qtractor is back again on track and this time with great news. And the big news are that this pet is leaving its rusty four-on-the-floor cage and spreading it's musical genre targets. Still a bedroom/home-studio sequencer though, but not for the techno-boy/girl only anymore--if one may trump about it, it's getting a general-purpose sticker now. And what makes it like just that? One long due feature, now stroked by implementation lightning: Tempo/Time-signature Map. Or in other words: project sessions may now have multiple parts with different tempo (BPM) and/or time-signatures."

Full Story (comments: none)

QuteCsound 0.4 released

Version 0.4 of QuteCsound has been announced. "I'm pleased to announce QuteCsound version 0.4. This version incorporates new features and bug fixes arising from the 3 previous Release Candidate releases. QuteCsound is a simple frontend for Csound featuring a highlighting editor with autocomplete, interactive widgets and integrated help."

Full Story (comments: none)

Science

Papywizard 2.0.0 released

Version 2.0.0 of Papywizard has been announced, it features a switch to PyQt and more. "Papywizard is a free panohead control software, mainly developped for the Merlin/Orion astronomic mount but usable for other panoheads, as long as it is possible to talk to them (hardware/software). The project is developped with the support of Kolor company, which develops the famous Autopano Pro stitcher software."

Full Story (comments: none)

PyTables 2.1.1 released

Version 2.1.1 of PyTables has been announced. "PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. This is a maintenance release, so you should not expect API changes."

Full Story (comments: none)

Video Applications

MotionTrackOSC 0.2 released

Version 0.2 of MotionTrackOSC has been announced. "MotionTrackOSC is a small program that does motiontracking on either a camera image or a video file. The tracked motion is output to a client via osc."

Full Story (comments: none)

Web Browsers

Firefox 3.1 Beta 3 is now available for download

Version 3.1 Beta 3 of Firefox has been announced. "Firefox 3.1 Beta 3 is now available for download. This milestone is focused on testing the core functionality provided by many new features and changes to the platform scheduled for Firefox 3.1."

Full Story (comments: none)

Miscellaneous

QLC: 3.0 is out (SourceForge)

Version 3.0 of QLC has been announced. "The Q Light Controller (QLC) is an X11/Linux application to control DMX or 0-10V lighting systems like dimmers, scanners and other lighting effects. Our goal is to replace expensive and rather limited hardware lighting desks with free software. This release includes (so far) Windows & Linux binaries as well as full sources."

Comments (none posted)

Roundup Issue Tracker release 1.4.7

Version 1.4.7 of Roundup Issue Tracker, an issue-tracking system with command-line, web and e-mail interfaces, has been announced. "1.4.7 is primarily a bugfix release which contains important security fixes".

Full Story (comments: none)

Roundup Issue Tracker release 1.4.8

Version 1.4.8 of Roundup Issue Tracker has been announced, it includes regression and bug fixes as well as some new features. "Roundup is a simple-to-use and -install issue-tracking system with command-line, web and e-mail interfaces. It is based on the winning design from Ka-Ping Yee in the Software Carpentry "Track" design competition."

Full Story (comments: none)

Task Coach: Release 0.72.2 available (SourceForge)

Version 0.72.2 of Task Coach has been announced. "Task Coach - Your friendly task manager. Task Coach is a simple open source todo manager to manage personal tasks and todo lists. Often, tasks and other things todo consist of several activities. Task Coach is designed to deal with composite tasks. This is a bugfix release."

Comments (none posted)

Languages and Tools

C

GCC 4.4.0 Status Report

The March 13, 2009 edition of the GCC 4.4.0 Status Report has been published. "The trunk is still in stage 4 which means it is open under the usual release branch rules. Thus the trunk is open for regression and documentation fixes only. We have been asked by the SC to not branch for now but wait for the wording for the new runtime license to arrive from the FSF and that being put in place. We currently have 82 serious regressions, which is below 100 and zero P1 regressions."

Full Story (comments: none)

Caml

Caml Weekly News

The March 17, 2009 edition of the Caml Weekly News is out with new articles about the Caml language.

Full Story (comments: none)

Java

OpenSwing: 1.9.5 released (SourceForge)

Version 1.9.5 of OpenSwing has been announced. "OpenSwing is a component library that provides a rich set of advanced graphics components and a framework for developing java applications based on Swing front-end. It can be applied both to rich client applications and Rich Internet Applications. In this release: Rebuild whole project. The distribution includes also Jar file libraries, source files, javadoc, licence and readme.txt installation instructions."

Comments (none posted)

Perl

kurila 1.18 released

Version 1.18 of Perl Kurila has been announced. "Perl Kurila is a dialect of Perl. With Perl 5 as ancestor it has a solid base, but without the backwards compatibility requirement of Perl 5, it is free to evolve into new directions."

Full Story (comments: none)

Parrot 1.0.0 released

Version 1.0.0 of the Parrot virtual machine has been released. Parrot is somewhat associated with Perl (and Perl 6 in particular), but it is intended to be able to run any dynamic language. More information can be found on parrot.org (or, until parrot.org becomes responsive again, Wikipedia).

Full Story (comments: 9)

Python

Concurrence Framework 0.3 released

Version 0.3 of Concurrence Framework has been announced. "Concurrence is a framework for creating massively concurrent network applications in Python. It takes a Lightweight-tasks-with-message-passing approach to concurrency. The goal of Concurrence is to provide an easier programming model for writing high performance network applications than existing solutions (Multi-threading, Twisted, asyncore etc)."

Full Story (comments: none)

Cython 0.11 released

Version 0.11 of Cython has been announced. "Cython is a language that makes writing C extensions for the Python language as easy as Python itself. Cython is based on the well-known Pyrex, but supports more cutting edge functionality and optimizations. The Cython language is very close to the Python language, but Cython additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code."

Full Story (comments: none)

IMDbPY 4.0 and IMDbPYKit 1.0

Version 4.0 of IMDbPY and version 1.0 of IMDbPYKit are out. "IMDbPY is a Python package useful to retrieve and manage the data of the IMDb movie database about movies, people, characters and companies. With this release, support for keywords and top250/bottom100 lists; also, it's now possible to export any data in XML. Many bugs were fixed and other minor improvements are present. Alongside IMDbPY 4.0, the 1.0 release of IMDbPYKit is available. IMDbPYKit is the new web interface of choice of IMDbPY; mostly developed by H. Turgut Uyar, it's based on the Paste Webkit toolkit and it has a lot of nice feature (e.g.: it can output data in both XML and HTML)."

Full Story (comments: none)

Python-URL! - weekly Python news and links

The March 11, 2009 edition of the Python-URL! is online with a new collection of Python article links.

Full Story (comments: none)

Tcl/Tk

Tcl-URL! - weekly Tcl news and links

The March 11, 2009 edition of the Tcl-URL! is online with new Tcl/Tk articles and resources.

Full Story (comments: none)

Tcl-URL! - weekly Tcl news and links

The March 17, 2009 edition of the Tcl-URL! is online with new Tcl/Tk articles and resources.

Full Story (comments: none)

Cross Assemblers

gputils 0.13.7 released

Version 0.13.7 of gputils, a collection of tools for Microchip PIC microcontrollers, has been announced. "Added support for all processors supported by MPLAB 8.20 (except eeprom16 and related). Added support for "LIST M=?" directive. Fixed several bugs."

Comments (none posted)

Cross Compilers

Arduino 0014 released

Version 0014 of Arduino, a development system for the Arduino board, has been announced, it includes minor bug fixes. "Arduino is an open-source electronics prototyping platform based on flexible, easy-to-use hardware and software. It's intended for artists, designers, hobbyists, and anyone interested in creating interactive objects or environments. Arduino can sense the environment by receiving input from a variety of sensors and can affect its surroundings by controlling lights, motors, and other actuators." See the release notes for more information.

Comments (none posted)

SDCC 2.9.0 RC2 released

Version 2.9.0 RC2 of SDCC, a C cross compiler for Intel 8051, Maxim 80DS390, Zilog Z80 and Motorola 68HC08 microprocessors, has been announced.

Comments (none posted)

Version Control

bzr 1.13 released

Version 1.13 of bzr has been announced. "This release includes bug fixes and a few performance and feature improvements. GNU Changelog output can now be produced by ``bzr log --format gnu- changelog``. Debug flags can now be set in ``~/.bazaar/bazaar.conf``. Lightweight Checkouts and Stacked Branches should both be much faster over remote connections. The Bazaar team is happy to announce availability of a new release of the bzr adaptive version control system."

Full Story (comments: none)

GIT 1.6.2.1 released

Version 1.6.2.1 of the Git distributed version control system has been announced, it includes a number of bug fixes.

Full Story (comments: none)

Miscellaneous

BugzillaMetrics: 0.9.7 released (SourceForge)

Version 0.9.7 of BugzillaMetrics has been announced, it includes new features and bug fixes. "BugzillaMetrics is a tool for the evaluation of change request metrics. Its main characteristics are the specification of metrics by the users, an event driven algorithm, transparent execution optimization and an abstraction of its data sources."

Comments (none posted)

Page editor: Forrest Cook

Linux in the news

Recommended Reading

How the Linux kernel works (TuxRadar)

TuxRadar takes a simplified look at the Linux kernel. Targeted at users and application programmers, rather than system-level programmers, the article gives an overview of what the kernel does, with a bit about how it does it. "All the other pieces you find in a Linux distribution - the Bash shell, the KDE window manager, web browsers, the X server, Tux Racer and everything else - are just applications that happen to run on Linux and are emphatically not part of the operating system itself. To give some sense of scale, a fresh installation of RHEL5 occupies about 2.5GB of disk space (depending, obviously, on what you choose to include). Of this, the kernel, including all of its modules, occupies 47MB, or about 2%."

Comments (16 posted)

The Free Beer Economy (Linux Journal)

Doc Searls ponders the word free on Linux Journal. "Why is FREE! the world's best-selling noun, verb, adjective and adverb, yet so hard to credit as a foundation for business in the Internet Age? And what will happen when business folk finally grok the abundant opportunities that FREE! provides?"

Comments (5 posted)

Companies

Kodak releases ScanTWAIN document scanner for Linux

Kodak has announced the release of their Linux-compatible ScanTWAIN document scanner software. "Emphasizing support for open source and building on its previous participation in the OpenUSB Project, Kodak today announced at FOSE 2009 in Washington, D.C., the availability of a robust, open source and production quality scanning application for use on the LINUX Operating System called ScanTWAIN. ScanTWAIN functionality will support duplex scanning at rated speed on automatic document feeder (ADF) based scanners, like those available from Kodak. The ScanTWAIN scanning application is available via the SourceForge website and is issued under GPL, Gnu Public License, version 2."

Comments (20 posted)

Discouraging Software Patent Lawsuits (Red Hat Magazine)

Red Hat VP and Assistant General Counsel Rob Tiller covers Red Hat's stance on its patent applications related to the AMQP specification. "Recently we've seen some surprising comments about Red Hat's stand on software patents and, in particular, about one of its patent applications related to the AMQP specification. It looks like clarification is called for. Our views and our position, as expressed in our work for patent reform, our Patent Promise, and our work with the AMQP project, have not changed."

Comments (1 posted)

Linux Adoption

Gendarmerie saves millions with open desktop and web applications (OSOR)

The EU Open Source Observatory reports from a talk by Xavier Guimard of the French Gendarmerie which, he says, has saved millions of Euro by moving to Linux-based desktops. "According to Guimard the move to open source has also helped to reduce maintenance costs. Keeping GNU/Linux desktops up to date is much easier, he says. 'Previously, one of us would be travelling all year just to install a new version of some anti virus application on the desktops in the Gendarmerie's outposts on the islands in French Polynesia. A similar operation now is finished within two weeks and does not require travelling.'"

Comments (17 posted)

Legal

Patents Being Abused To Put Your Life In Danger (TechDirt)

TechDirt takes a look at how the lawyers for Priority Dispatch Corporation are abusing the patent system. "Bob Austin, who for many years has worked in major metropolitan fire and EMS departments, had the idea of creating an open source medical dispatch system. Such [an open source] system would have numerous benefits. Beyond being a free system, it also would allow best practices to easily bubble up in a way that actively would help save lives. If another EMS department could improve on the system, they easily could do so and contribute it back to the community." The Priority Dispatch lawyers claimed possible patent infringement for ten patents. See Wikileaks for more information. (Found on LinuxMedNewsArticle)

Comments (none posted)

Interviews

Fellowship interview with Georg Greve (FSFE Fellowship)

Here's a Free Software Foundation Europe Fellowship interview with Georg Greve. "Georg Greve is the founder of the Free Software Foundation Europe and has served as its president since the beginning in 2001. Marking the eight birthday of the FSFE I asked him some questions on his own background and the history of the FSFE for a special birthday edition of the Fellowship interview series."

Comments (none posted)

Resources

Linux tip: Finding rootfs during boot (IBM developerWorks)

IBM developerWorks shows how to work with Linux boot problems. "As a Linux® administrator, you may encounter rootfs errors like cannot mount rootfs and kernel panic when you try to reboot a server after attaching volumes from external storage or even after installing a new Linux operating system.. This article outlines the Linux booting process on an x86 platform, shows why this problem happens, and offers four tips to avoid it or fix it."

Comments (none posted)

Linux services without mains power (Tectonic)

Stevan Lockhart explains how he set up an off-grid Linux-powered home and business in Scotland. "Originally from Cape Town, Stevan Lockhart now lives in the north west of Scotland in a house that is off the electricity grid. Using a wind generator, some solar panels and free and open source software he and his wife both run their businesses off just 20W of power. Here Stevan explains the software and hardware decisions that made this lifestyle possible."

Comments (6 posted)

The seven best Linux Foundation contest videos (Computerworld)

As the Linux Foundation's "We're Linux" video contest has entered the judging phase, Steven J. Vaughan-Nichols lists his seven favorite entries. "I'm not a judge on the committee that will decide the winner, but I do know a little bit about both Linux and marketing. So, here are my seven favorite picks in the contest. I tried, I really did, to cut the list to five, but I couldn't do it. It was hard enough to get to seven."

Comments (3 posted)

Reviews

KDE4 Version of Digikam Photo Management (KDE.News)

KDE.News takes a look at the newest release of Digikam. "Photographers in the Free world rejoice! On behalf of the Digikam developer team, Gilles Caullier has announced the first KDE 4 release of Digikam, the photo management application. Digikam is a full-fledged tool that covers the whole digital photography workflow, from downloading the pictures off of your camera (and in the new version even capturing photos from inside digikam) to sorting, editing, rating and tagging (including geolocation and editing of raw photos in the new version) and of course also publishing your photos. Digikam now also allows for importing images from Social Media such as Facebook and SmugMug."

Comments (none posted)

The Squirrel portable shell and scripting language (developerWorks)

Here's an overview of the Squirrel shell on developerWorks. "If you don't want to commit to the idiosyncrasies of the Fish shell, the Bash shell, the Z shell, the Windows operating system's cmd.exe, or some other shell scripting language, try the Squirrel Shell. The Squirrel Shell provides an advanced, object-oriented scripting language that works equally well in UNIX, Linux, Mac OS X, and Windows systems. You can write a script once and run it anywhere."

Comments (9 posted)

Miscellaneous

How to Lie with Maps: When Open Source and National Security Collide (Linux Journal)

Linux Journal covers an effort to blur Google maps to preserve (US) national security. "Could a skilled bad guy glean a lot of information from Google Earth or similar system? Sure. Just like you or I can plot out a route to maximize our walking time or minimize our commute. I would argue that I could get better information from a 7.5 minute quadrangle because it would also include the ability to generate points that I could enter into my GPS for turn-by-turn directions. But until we actually go through the motions, it is just static data. And I would suggest that the attackers in Mumbai, once they planned their route on paper, walked, rode and photographed any number of things that were not available publicly."

Comments (10 posted)

Page editor: Forrest Cook

Announcements

Non-Commercial announcements

The ACTA treaty is secret for "national security"

James Love reports on the latest attempt to get the text of the proposed ACTA treaty from the US government. "The agreement, misleadingly named the Anti-Counterfeiting Trade Agreement, or ACTA, is thought to cover a wide range of intellectual property enforcement issues -- including standards for granting injunctions for alleged infringement of patents or copyrights, damages, seizures of goods in transit, surveillance of Internet digital file transfers, searches of personal property, and a dozen other topics." The answer: the treaty, despite being available to governments and lobbyists, is classified for "national security" reasons and we cannot see it. "Change," it seems, only goes so far.

Comments (none posted)

CPMPPSO call for submissions

A call for submissions has gone out for the Common Persistent Model Patterns for Performance and/or Scalability Optimization reports. "ODBMS.ORG, a vendor-independent non-profit group of high-profile software experts lead by Prof. Roberto Zicari, today announced the opening of a call for submissions for Common Persistent Model Patterns for Performance and/or Scalability Optimization. ODBMS.ORG invites both vendors and Application architects, Enterprise architects, Developers who use databases to submit implementation techniques (database design patterns) which are generally useful for all adopters. The best submissions will be published in a new series of reports in ODBMS.ORG. All submissions will be published under free software licenses." Submissions are due by May 29, 2009.

Full Story (comments: none)

FFII: EPO seeks to validate software patents without the European Parliament

The FFII has sent out an advisory warning that the European Patent Office is trying to establish software patents via an indirect route. "The President of the European Patent Office (EPO), Alison Brimelow, has asked the Enlarged Board of Appeal (EBA) to decide on the interpretation of the European Patent Convention (EPC) regarding the exclusion of software from patentability. The EBA is replacing the European Parliament in order to validate software patents EU-wide without the need of a debate." There is an opportunity to comment on the upcoming decision through the end of April.

Full Story (comments: 9)

Audacity: GSoC call for students

The Audacity audio editor project has announced a preliminary call for students for the 2009 Google Summer of Code. "GSoC offers student developers a $4,500 stipend to write code for open source projects. We're asking interested students to contact us now to discuss their proposals. We're looking for projects that will move us towards a new stable 1.4 Release later this year. If you are an eligible student, or know one who might be interested, please review our draft documentation."

Comments (none posted)

XMMS2: GSoC call for students

The XMMS audio player project has announced its application for the 2009 Google Summer of Code. "We have just applied for Google Summer of Code 2009. Hopefully we will be included in the program for the fourth year in a row. Get involved in our ideas and community now!"

Comments (none posted)

Resources

EFF launches search tool for uncovered government documents

The Electronic Frontier Foundation has announced a search tool for uncovered government documents. "In celebration of Sunshine Week, the Electronic Frontier Foundation (EFF) today launched a sophisticated search tool that allows the public to closely examine thousands of pages of documents the organization has pried loose from secretive government agencies. The documents relate to a wide range of cutting-edge technology issues and government policies that affect civil liberties and personal privacy."

Full Story (comments: none)

Calls for Presentations

Gran Canaria Desktop Summit Call for Participation

A call for participation is out for the Gran Canaria Desktop Summit. The Summit will also co-host GUADEC, the GNOME Conference and Akademy, the KDE conference. The Desktop Summit runs from July 4 to July 10 2009, in Las Palmas, Gran Canaria, Spain.

Full Story (comments: none)

Japan Linux Symposium call for participation

The first annual Japan Linux Symposium will be held October 21-23 in Tokyo, immediately after the 2009 Kernel Summit. The call for participation is now open, with submissions due by May 1. JLS looks to be an interesting event, bringing together developers from around the world in a country which is rapidly becoming a major contributor. Talks are sought on a wide variety of topics; see the CFP for details.

Comments (none posted)

ToorCamp 2009 call for participation

A call for participation has gone out for ToorCamp 2009. "ToorCamp is the United State's first ever full-scale hacker camp. Modelled after the camps in Holland and Germany, ToorCamp will focus on all of the technology topics that ToorCon has become famous for but will expand out into other areas of society. ToorCamp will offer 2 days of talks on many different topics -- Security, Internet, Emerging Technologies, Hardware Hacking, and Privacy are just some of the areas we will be covering. ToorCamp will also feature 2 days of hands-on workshops on a multitude of different skills that you may have never found yourself interested in learning about before. Blacksmithing, Lock Picking, Orienteering, Logic Design, Archery -- These are just a few of the topics you can expect." The event takes place on July 2-5, 2009 in a Titan-1 missile silo near Moses Lake, WA. Submissions are due by April 3.

Full Story (comments: none)

YAPC::NA::09 just $99 for the next day days + CFP

YAPC::NA::09 tickets are discounted until March 25. The event takes place in Pittsburgh, PA on June 22-24. "The very first Yet Another Perl Conference was held in '99. To celebrate the 10th anniversary of that event, we are offering a special YAPC|10 admission price of $99 — but only for 10 days! After that, the price goes up to $125." Also, a call for proposals has been announced for the event, submissions are due by April 24.

Comments (none posted)

Upcoming Events

FUDCon and LinuxTag reminders

A reminder has been posted for FUDCon and LinuxTag. "I'm here to remind everyone about the relevant details for FUDCon Berlin 2009 and LinuxTag 2009.
https://fedoraproject.org/wiki/FUDCon:Berlin_2009
https://fedoraproject.org/wiki/LinuxTag_2009
LINUXTAG: Berlin, Germany. June 24 - 27. FUDCON: Berlin, Germany. June 26 - 28."

Full Story (comments: none)

Preliminary PGCon 2009 schedule released

The preliminary schedule for PGCon 2009 has been announced, the event takes place on May 19-22 in Ottawa, Canada. "PGCon is the premiere PostgreSQL conference. It is simply the best place to meet other PostgreSQL people and learn more. The talks are absolutely amazing, not to mention the great social atmosphere."

Full Story (comments: none)

PyCon 2009 hosts cross-community summit meetings

PyCon 2009 will host some cross-community summit meetings. "Python programmers won't be the only ones coming to PyCon this month. PyCon 2009 is hosting two new summit events that will draw key developers and strategists not only from Python, but also from the Perl, Ruby, Java, .NET, and JavaScript communities. The Virtual Machine Summit on March 25, sponsored by Sun Microsystems, and Python Language Summit on March 26, both at PyCon 2009 in Chicago, are invitation-only events that will bring cross-discipline groups together to discuss and strategize on challenges that are common across their projects."

Full Story (comments: none)

Events: March 26, 2009 to May 25, 2009

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
March 23 March 27	iPhone Bootcamp	Atlanta, Georgia, USA
March 23 March 27	ApacheCon Europe 2009	Amsterdam, The Netherlands
March 23 April 3	Google Summer of Code '09 Student Application Period	online, USA
March 24 March 26	UKUUG Spring 2009 Conference	London, England
March 25 March 29	PyCon 2009	Chicago, IL, USA
March 27 March 29	Free Software and Beyond The World of Peer Production	Manchester, UK
March 28	Open Knowledge Conference 2009	London, UK
March 31 April 2	Solutions Linux France	Paris, France
March 31 April 3	Web 2.0 Expo San Francisco	San Francisco, CA, USA
April 3 April 4	Flourish Conference	Chicago, IL, USA
April 3 April 5	PostgreSQL Conference: East 09	Philadelphia, PA, USA
April 6 April 7	Linux Storage and Filesystem Workshop	San Francisco, CA, USA
April 6 April 8	CELF Embedded Linux Conference	San Francisco, CA, USA
April 8 April 10	Linux Foundation Collaboration Summit	San Francisco, CA, USA
April 14	OpenClinica European Summit	Brussels, Belgium
April 15	Linuxwochen Österreich - Krems	Krems, Austria
April 16 April 17	Nordic Perl Workshop 2009	Oslo, Norway
April 16 April 18	Linuxwochen Austria - Wien	Wien, Austria
April 16 April 19	Linux Audio Conference 2009	Parma, Italy
April 20 April 23	MySQL Conference and Expo	Santa Clara, CA, USA
April 20 April 24	samba eXPerience 2009	Göttingen, Germany
April 20 April 24	Perl Bootcamp at the Big Nerd Ranch	Atlanta, GA, USA
April 20 April 24	Cloud Slam '09	Online, Online
April 22 April 25	ACCU 2009	Oxford, United Kingdom
April 23	Linuxwochen Austria - Linz	Linz, Austria
April 23 April 24	European Licensing and Legal Workshop for Free Software	Amsterdam, The Netherlands
April 23 April 26	Liwoli 2009	Linz, Austria
April 25	Linuxwochen Austria - Graz	Graz, Austria
April 25	Festival Latinoamericano instalación de Software libre	All Latin America, All Latin America
April 25	Grazer Linux Tage 2009	Graz, Austria
April 25 April 26	LinuxFest Northwest 2009 10th Anniversary	Bellingham, Washington, USA
April 25 May 1	Ruby & Ruby on Rails Bootcamp	Atlanta, Georgia, USA
April 27	OSDM 2009	Bangkok, Thailand
May 4 May 6	EuroDjangoCon 2009	Prague, Czech Republic
May 4 May 6	SYSTOR 2009---The Israeli Experimental Systems Conference	Haifa, Israel
May 4 May 7	RailsConf 2009	Las Vegas, NV, USA
May 4 May 8	JavaScript/Ajax Bootcamp at the Big Nerd Ranch	Atlanta, Georgia, USA
May 5	Linuxwochen Austria - Salzburg	Salzburg, Austria
May 6 May 8	Embedded Linux training	Maynard, USA
May 6 May 9	Libre Graphics Meeting 2009	Montreal, Quebec, Canada
May 7	NLUUG spring conference	Ede, The Netherlands
May 8 May 9	Linuxwochen Austria - Eisenstadt	Eisenstadt, Austria
May 8 May 9	Erlanger Firebird Conference 2009	Erlangen-Nürnberg, Germany
May 8 May 10	PyCon Italy 2009	Florence, Italy
May 11	The Free! Summit	San Mateo, CA, USA
May 13 May 15	FOSSLC Summercamp 2009	Ottawa, Ontario, Canada
May 15	Firebird Developers Day - Brazil	Piracicaba, Brazil
May 15 May 16	CONFidence 2009	Krakow, Poland
May 16 May 17	YAPC::Russia 2009	Moscow, Russia
May 18 May 19	Cloud Summit 2009	Las Vegas, NV, USA
May 19	Workshop on Software Engineering for Secure Systems	Vancouver, Canada
May 19 May 21	Where 2.0 Conference	San Jose, CA, USA
May 19 May 22	PGCon PostgreSQL Conference	Ottawa, Canada
May 19 May 22	php|tek 2009	Chicago, IL, USA
May 19 May 22	SEaCURE.it	Villasimius, Italy
May 21	7th WhyFLOSS Conference Madrid 09	Madrid, Spain
May 22 May 23	eLiberatica - The Benefits of Open Source and Free Technologies	Bucharest, Romania
May 23 May 24	LayerOne Security Conference	Anaheim, CA, USA

If your event does not appear here, please tell us about it.

Audio and Video programs

Fosdem 2009 videos released

Conference videos from Fosdem 2009 are available. "In our tradition of releasing embedded Linux conference videos, we are happy to release 9 videos from the embedded room at the Fosdem 2009 conference in Brussels last month".

Full Story (comments: none)

Page editor: Forrest Cook