LWN.net Weekly Edition for August 4, 2011

Emacs and the GPL

The Free Software Foundation (FSF) prides itself on being the champion of free software and its licensing, so a recent misstep with Emacs source code has to be a little embarrassing. But it is just that, a misstep, and one that was quickly addressed in two separate ways. While it may be tempting to lambaste the FSF and the Emacs developers for the oversight (and many have done so), it really should be seen as a lesson for projects and companies to be ever more vigilant regarding the licensing of their code. That may cause some to question whether complying with free software licenses is harder than it needs to be, but that would be ignoring the huge benefits that the GPL and free software bring to the table.

The problem stems from the inclusion of CEDET (a Collection of Emacs Development Environment Tools) into Emacs 23.2. CEDET has parsers for various languages that are based on Bison grammar files that get translated into Elisp (an Emacs dialect of Lisp). The grammar files were not included with Emacs 23.2 and 23.3, but, in addition, the Lisp output from the Bovine and Wisent parser generator tools used to do the translation had been hand-edited. That meant that even having the grammar files was insufficient to reproduce the Lisp code that got shipped with those versions of Emacs.

Needless to say, shipping code without providing source in the "preferred form of the work for making modifications" is not at all GPL-compliant. That's not really a legal issue for the Emacs project because the FSF owns the copyright to the code and is thus not required to follow the GPL. It certainly sets a bad precedent, however, and one that Richard Stallman and others wanted fixed in short order. That fix took two paths.

To start with, Chong Yidong created new versions of the two affected Emacs releases. Those included the grammar files and a script to apply the changes made to the Lisp output. That means that anyone distributing 23.2a or 23.3a would be in compliance with the GPL.

But that left those who had already distributed the earlier versions in a bit of a bind, as they were in technical non-compliance with the GPL for doing so. Stallman solved that problem with a grant of permission that essentially absolves those distributors of any copyright violation and "extends to versions with unrelated modifications, provided those versions comply with GNU GPL version 3 in all other respects". He also notes that the GPLv3 makes the process a little easier:

The version numbers aren't quite right in his explanation above (though they are in the grant), but his point is an interesting one. The GPLv3 has explicit language that reinstates the distribution right for violators if they cease the violation (which the permission grant effectively does) and aren't contacted by the copyright holder in 60 days. The picture for GPLv2 would have been a bit murkier because it lacked the automatic reinstatement of the license. Though, obviously, the FSF could have explicitly forgiven the earlier violations had Emacs been released under GPLv2.

Clearly Stallman is a stickler for adhering to the letter of the GPL—as he should be—and the problem was resolved less than a week after it came to his attention. The fact that it took more than a year for the problem to be recognized is unfortunate, but it stemmed from a misunderstanding about the "preferred form". It seems that Chong believed that the Lisp source code, which was distributed, was a reasonable interpretation of providing the source—which it is. Where it falls down, of course, is that it isn't the form that a programmer would want to use if they were to make modifications to CEDET.

There was some unhappiness expressed in the thread on emacs-devel regarding the coverage of the issue, specifically that which appeared at LWN. While there may have been some intemperate comments in the thread, it's a little strange to hear that coverage of the issue should have been out-of-bounds. As Stephen J. Turnbull points out, it's important for word to get out about the problem:

Turnbull's footnote notes that it is extremely unlikely that it would go that far, but his point is well taken. Whether one likes the coverage (and comments) or not, it is important for the word to get out to downstream distributors so that they can decide if they want to update to 23.2a or 23.3a. It would be well-nigh impossible for the project to find and contact all of the different distributors who have shipped the non-compliant code. The Linux and free software media have a role to play, though the coverage here and elsewhere still won't reach every downstream of course.

Turnbull also makes a larger point:

There is a cost associated with using free software, and complying with the license is the biggest part of that cost. Proprietary licenses have their costs as well, of course, and those can be partially measured in dollars. Complying with a thick stack of legalese in order to license some bit of proprietary code may seem easier, at least partly because companies are used to those kinds of licenses, but probably isn't. In the end, free software is an enormous bargain, in general, because the amount of work required to comply is far outweighed by the usefulness of the code. It may take a bit of a different mindset, but the value gained is enormous.

Beyond that, the GPL also provides a level playing field for contributors, so that individuals and companies can contribute to a common code base and know that no one has the upper hand. No one can take the code proprietary or make changes and distribute them without sharing those changes back. Copyright assignment (or licensing) may muddy the water to some extent, but, for many projects, the GPL provides the needed cover for competitors to collaborate.

In the end analysis, this was a minor problem that was addressed fairly quickly. The grammar files were always available at the CEDET site, and the handmade changes were pretty trivial overall. That said, it is important that it was found and fixed, not least because the FSF is the poster child for GPL compliance. The silver lining is that other projects are likely to remember this mistake and keep an eye out for similar problems in their own code. If suffering a little public embarrassment results in fewer GPL mistakes (and violations) down the road, the FSF's goals will have been well-served in the process.

Comments (23 posted)

Web-based feed reading with rsslounge

Finding an open source RSS feed reader is not a hard task. Finding an open source feed reader that compares with Google Reader is another story. After using Newsbeuter for a while, I eventually slipped back to using Google Reader. Having a web-based reader is just too convenient for those of us who use two, three, or more computers regularly. After a bit of hunting, I found rsslounge, a GPLed and web-based feed reader that gives Google Reader a strong run for its money.

Written by Tobias Zeising, rsslounge requires PHP 5.2.4, MySQL, and a web server. The web server needs to support .htaccess files, but otherwise it doesn't appear to be too picky. I chose Apache, but you should be able to use lighttpd or nginx if you are willing to add a few re-write rules.

Installing rsslounge is a five-minute task for anybody passingly familiar with setting up a PHP-based application. If you've ever installed WordPress, Drupal, or something similar, the rsslounge installation is easy as pie. The only word of caution when installing rsslounge: users should be sure to create an administrative account if it's going to be on a public network. Without such an account, anyone can come in and change the settings of the installation and/or add and delete feeds. Note that rsslounge is designed for a single user, and not for multiple users with separate accounts.

Since I've been using Google Reader for some time, I decided to start testing rsslounge by importing my Google Reader feeds. This was easy and took about 30 seconds for rsslounge to parse the file and start fetching feeds. It took about another minute before rsslounge had fetched the feeds and started displaying them. I should note that rsslounge did a good job of importing the feeds and preserving the categories/folders from Google Reader. This is sometimes a problem when importing Google Reader's OPML into another application, and, if you have a lot of feeds, having to re-categorize them can be a bit of a hassle.

The layout for rsslounge is similar to Google Reader: The list of feeds is displayed on the left-hand side and the articles are displayed on the right. Each item in a feed is displayed with the time stamp for the item, its headline or title, and then the name of the site that it originated from. On the left-hand side, rsslounge displays the site names, site icons, and unread count (if any). To read an item, clicking on the headline will expand the article, or clicking on the site name will open the link in a new window. Like Reader, you can "star" items to save them for re-reading. One thing that I'm not crazy about with rsslounge is that there appears to be no way to expand items by default. That is, Google Reader gives the option of just skimming headlines, or skimming full posts. With rsslounge, you simply skim headlines and have to expand each item when you want to read it.

If you like to skim feeds without touching the mouse, rsslounge supports a short list of keyboard shortcuts. You can navigate within feeds pretty easily, though switching feeds using the keyboard does not yet seem to be supported. So, for example, if you're reading through your uncategorized feeds and want to switch to a category, there's no keyboard shortcut for that.

Though rsslounge may not be at par with Google Reader's keyboard shortcuts, it does make up for that with a few features that you don't get with Google Reader. For example, rsslounge allows you to browse through feeds by date. Up in the left-hand corner is a small calendar icon (which is easy to miss) that will expose a monthly calendar. You can then choose to see a feed, category, or all feeds by date. For instance, if you want to see only feeds from July 16, you can click the All Items category and then select the 16th from the calendar.

Managing feeds is also far more pleasant in rsslounge, at least in this users' opinion. If a feed's URL changes, in Google Reader you're stuck unsubscribing and then re-subscribing using the new URL. In rsslounge you simply click on its edit icon, change the source for the feed, and you're good to go.

The settings for feeds are more fine-grained than with Google reader as well. If you subscribe to a lot of RSS feeds that are for images rather than text, such as feeds from photo blogs like deviantArt or Flickr, rsslounge has a special feed type for those. If you select the image type for a feed, then rsslounge displays a grid of thumbnails instead of a list of headlines.

If you have some feeds that take priority over others, you can increase their priority so that they'll be shown first. You can also, of course, decrease a feed's priority so that you only see it after you've read everything else. Another thing that many users will appreciate is the option to filter feeds. You can provide a regular expression in the "Edit Feed" dialog that will be used to match items from a given Atom or RSS feed. If the entry doesn't match the expression, it won't be fetched. It doesn't seem that rsslounge provides support for filtering out entries that match an expression, unfortunately.

The only downside that I've found so far with rsslounge is a paucity of documentation and information on the project in general. The wiki for rsslounge on Google Code has a brief set of installation instructions, and there's the keyboard shortcuts document, and that's really about it. It appears to be a one-man show, though there is a forum for users to ask questions. However, the forum is a bit underpopulated.

Other web-based alternatives

A few years ago, there was a project released by Fastladder, free web-based RSS and Atom feed reader service. It claims to be faster than Google Reader and Bloglines, with a "transaction speed close to that of humans", whatever that means. Called OpenFL (at least according to its README, the Google Code project is just "fastladder"), it's a Ruby on Rails project that can be installed on Linux, Windows, or Mac OS X. However, it looks like the project is more or less dead as an open source effort. The last updates shown on Google Code date back to April 2009, and most of the code hasn't been touched since February 2008. There's also a fork on GitHub, which is also a bit long in the tooth — the last updates are from May 2010. However, the code is out there, so if anyone would like to pick it up it's easily forkable on GitHub.

For non-defunct projects, there's also Tiny Tiny RSS, which also requires PHP and a web server, but gives the option of MySQL or PostgreSQL. It requires a bit more setup and tweaking than rsslounge, but also offers more options. For instance, it can be configured to use SimplePie for parsing feeds instead of Magpie, and has a multi-user mode. Nathan Willis has a good write-up on Linux.com about Tiny Tiny RSS that goes into detail on setting it up.

Overall, rsslounge looks to be close to a drop-in replacement for Google Reader. It does lack a few features, but it's a good substitute for Google Reader without the need to give up control of your data or privacy.

Comments (21 posted)

Getting a better Glimpse at unstable applications

Developers from the Elementary OS project have crafted a new method that allows users to install and run sandboxed versions of unstable applications more safely inside their main Linux environment. Dubbed Glimpse, the system uses a chroot environment to isolate the unstable code and Another Union FS (aufs) to safely provide the chrooted binary with read access to the host filesystem. Just as importantly, Glimpse provides a straightforward mechanism for users to automatically install the unstable applications they wish to test, using pre-configured OS "profiles."

It is certainly already possible to test pre-release upstream applications before one's distribution provides them directly, of course, but the Glimpse team found the available options unsatisfying. Testing a new release in a virtual machine isolates it from real world data — which makes it difficult to get a practical feel for editing, file management, and a host of other tasks. Compiling from source deprives one of regular updates, and often introduces the time-consuming task of compiling pre-release library dependencies as well (which may also introduce instabilities to the rest of the system).

Launchpad offers an Apt-based middle ground through its personal package archives (PPAs), which quite a few development teams use to provide public testing or "daily build" repositories, but these too often require updating library packages with the resulting hit to system stability. In contrast, Glimpse provides the user with an easy-to-use launcher that can start up pre-release applications, but keeps all of the packages involved sandboxed from the host OS. The applications run at native speed, and can read local files, but cannot modify or delete them on the underlying filesystem.

Glimpse inside

The Glimpse framework makes heavy use of the Ubuntu Customization Kit (UCK) tools, which are designed to add and replace individual packages within bootable ISO images, but provide extra utilities as well — including uck-remaster-chroot-rootfs, which executes a package with chroot providing an alternate root directory.

The Glimpse workflow begins with installing a distribution "profile" package. A profile consists of Bash scripts to install, update, and purge the test OS, and scripts to install and purge specific applications. Profiles are distribution-specific, not only because they start with specific ISO releases as the base system, but also because they rely on remote Apt repositories to provide package updates, include distribution-specific tweaks to enable PulseAudio compatibility and other miscellany, and provide a custom "launcher list" of testable applications.

When run, Glimpse loads a list of the installed OS profiles. The first time a profile is used, Glimpse downloads the base ISO of the OS to be tested (in the current profiles, this is a bootable live CD image), or allows the user to locate a local copy on the filesystem (if one exists). It then uses UCK tools to "remaster" the image adding any external unstable packages users wish to test. Subsequently, Zsync pulls in any updates to the base ISO, which prevents the need to re-download the entire image if an updated alpha or beta image is released. Together, these tools handle dependency resolution in the same fashion that Apt would for a full installation.

When an installed profile is run, Glimpse opens up a second window with basic profile-control buttons: "Launch an app," "Update sandbox," "Purge file modifications," and "Remove sandbox." The "Launch an app" button opens up a window with the pre-configured launcher list defined in the profile. These launchers are merely a pre-selected list of .desktop launchers available within the sandboxed OS. Aufs is used to mount an overlay filesystem on top of specific directories (/home/$USER, /var/run, /proc, /dev/pts, /tmp, /sys, and /dev/shm). The overlay filesystem provides read-only access to the host OS underneath. That permits the user to work with his or her real files, but isolates any changes within the sandbox. When the user clicks on a launcher, the binary is executed with uck-remaster-chroot-rootfs.

Thus far, the only two profiles available are Ubuntu 11.10 and Elementary OS 0.2 (which is an Ubuntu derivative). The Elementary OS profile includes launchers for nine Elementary OS applications not found in upstream Ubuntu, such as the BeatBox audio client and Marlin file manager. A terminal is also in the list, however, which makes it possible to launch the Elementary OS version of any other application (which could be useful to test compatibility or new features).

Of course, since Elementary OS is such a new distribution — the first release was in March of 2011 — the Ubuntu 11.10 profile will certainly attract a larger set of users. Unfortunately, at the moment it is considerably less polished: it cannot initiate the application launcher window of the Elementary OS profile, and there are several bugs filed indicating that the launcher and filesystem mounting scripts are incomplete. Nevertheless, interested Ubuntu users can still make good use of the Elementary OS profile to test PPA-provided unstable packages thanks to the fact that Elementary OS 0.2 uses Ubuntu 11.10 as its base ISO, and provides a terminal in its Glimpse application launcher list.

Glimpse in practice

Glimpse itself is available as a PPA via Launchpad. The main package and the two official OS profiles are in separate packages. Zsync and UCK are dependencies, as are the GTK+ shell-script interface Zenity and a special purpose library named Gaffel that Glimpse's developers wrote in order to write better GTK dialogs for Glimpse.

Where it works, Glimpse offers a compelling user experience: there is no lag, the sandboxed applications run at full speed, and are virtually indistinguishable from natively-installed programs. The main difference is the process for launching them: the Glimpse UI must be launched first, then the proper OS profile selected, then the launcher list, then, finally, the application — with each step spawning a separate window. For launching a single application, it would be much nicer to roll those steps together (perhaps consolidating them into a sub-menu of the desktop environment's Applications menu, just to provide some level of distinction between the sandboxed and un-sandboxed applications). Then again, it could be argued that testing "unstable" code is not a process most users will engage in on a long-term, regular basis.

The Glimpse sandboxing method does have its limitations. First, no matter what distribution is used for the profile (and no matter how many libraries within the sandbox UCK loads in order to launch the app), applications running in Glimpse sandboxes are always run using the host OS's kernel. That could cause problems if the application in question needs a driver or filesystem module unavailable in the host OS kernel, but it is probably unavoidable. Second, Glimpse uses aufs to mount overlay versions of several key directories, but for one reason or another, it does not mount the /usr/share hierarchy, which has the side effect of making the host OS's window manager themes unavailable. If you use a theme not provided in the default install, your sandboxed applications will look (and perhaps even behave) differently.

Glimpse also allows only one copy of each OS profile to be installed. The profiles and their associated files (the ISO image, "remastered" update packages, and filesystem overlays) are stored in ~/.glimpse, each in its own subdirectory by distribution name, rather than by some unique key or ID, and the scripts enumerate and access them by name. While this is good enough for the general use-case (testing pre-release applications), Glimpse would be useful for some additional scenarios if multiple copies of each profile could be installed and compared.

The system also lacks a UI for adding additional application launchers and additional Apt repositories to an installed profile. Here again, for testing purposes, it is easy enough to open a terminal and run add-apt-repository to enable a third-party PPA, but as long as we are dreaming of game-changing application testing, we'd might as well dream big. Finally, as is frequently the case with new utilities, there is a shortage of documentation. The code is reasonably well-documented (and it is Bash), but I eventually gave up trying to hack the incomplete Ubuntu 11.10 profile into shape with only the Elementary OS profile for guidance.

From a security standpoint, developer Sergey Davidioff cautions that exploits breaking out of the chroot environment would then break through Glimpse as well. This could result in a privilege escalation attack, because Glimpse must be run with root privileges in order to mount aufs filesystems. The trouble is that Glimpse launches the tested applications with uck-remaster-chroot-rootfs, which was designed to execute system configuration scripts when remastering an ISO image. So although it launches the selected application in a chroot environment, it also launches that application as root. Chroot alone does not provide a security "jail." Accidental vulnerabilities are a constant threat of course, but a malicious attacker could replace a "daily build" PPA package and thus trick users into running it in Glimpse.

Glimpse of the future

Davidoff frames his introductory blog post about Glimpse as a project that will help Elementary OS users alpha-test new software for the distribution (although, for the time being, Glimpse is not yet an official sub-project of Elementary OS). There have been other tools designed to do the same task, such as Arkose and Ubuntu Test Drive. Arkose is designed to build a static sandbox around a single application, however, which means that users must repeat the process for every application they test. Ubuntu Test Drive runs the "tested" application on a remote server, and tunnels the GUI back to the client machine using the NX protocol.

Between the two, Arkose is definitely the closer match for Glimpse, and with some additional code it might be able to provide a comparable experience, with multiple applications and support for daily (or frequent) package updates. However, it would probably be shortsighted to focus solely on Glimpse's ability to track a full distribution during the alpha-testing phase. A simple scheme to run chroot-sandboxed applications with aufs-based filesystem access has other useful applications, too.

For example, I occasionally wonder whether or not some of the dust-covered binaries of long-abandoned applications in my desk drawer (such as IBM ViaVoice for Linux or The Sims) would still run — and perhaps run better on today's faster processors. To test them out, I would need to procure older, not newer, copies of a bunch of system libraries. But if I could sandbox them with Glimpse, it would be far easier to work with than running a full distribution in a virtual machine, and the odds of getting the audio to work correctly would be better as well.

If multiple profiles were supported, a range of testing scenarios become possible that today would have to be done in a virtual machine. Naturally, since Glimpse does not provide access to a sandboxed kernel, it is no replacement for a full virtual machine environment — but at the application level, it is interesting to speculate. A great many Unix graybeards will fall back on "chroot it" as the answer to how to test an untrusted program, but it has never been presented with a friendly GUI veneer on the outside. Perhaps now it has.

In the short term, Glimpse needs to complete its Ubuntu 11.10 OS profile; this will attract a significantly large user and developer base that could extend the tool further. A way to launch sandboxed application without running them as root is critical to attracting more than the casual user — although here again the exposure to attack involves planting a backdoor or exploiting a bug in an upstream application. There are bugs filed for encrypted home directory support and other features, but in the medium term the next big hurdle will be getting contributors to develop profiles for other distributions — openSUSE and Fedora are mentioned in Davidoff's blog post and, although much of the logic remains the same, supporting RPM and Yum instead of Apt and Debian packages will be a challenge, and adding support for other distribution's ISO remastering tools could be a task unto itself.

Whatever direction Glimpse takes in the future, for today it is an intriguing tool for users and developers alike. Most distributions are on six-to-eight-month release cycles, which makes it conceivable to run "unstable" applications while waiting for the next version to hit the file server. Whether testing or living on the bleeding-edge is the goal, Glimpse does make it simple to get started.

Comments (11 posted)

Page editor: Jonathan Corbet

Security

Password storage on Android devices

A number of Android device owners are upset to learn that their trusty products store sensitive credentials — such as email account passwords — in plain text. The bug tracker entry has amassed an impressive number of votes, but as the attached discussion reveals, any proposed solution butts up against consumers' conflicting expectations — such as that the device continue to check for messages as long as it is turned on, yet never inconvenience the user by demanding a password.

The root of the trouble is that POP, SMTP, IMAP, and other email protocols may require the client to provide the account password on every connection attempt. Thus, in order for the phone to check email silently in the background, the OS must have access to the password in plaintext. Google's default email application does just this, storing account passwords in the clear inside a SQLite database. Google's Gmail-specific client application, which uses a different, token-based authentication protocol, does not require the password on each connection, but the token itself may serve just as well. Discovery of that password "flaw" spawned a tremendous public outcry in the comments section of Android bug #10809.

Of course, Android is not alone in this regard. Apple's iOS and other mobile platforms also store passwords on the device in plain text, and a client application on any other computing platform faces the same challenge: Thunderbird, PINE, Evolution, KMail, etc. — even IM clients — all send the account password to the server when left running unattended. Consequently, they must either store the password in plain text, ask for it on every connection attempt, or encrypt it and require the user to enter a different passphrase to unlock the storage location.

Analyzing the problem

The difference is that mobile phone users are unaccustomed to logging into their phones with a username and password, and even if they do, they tend to leave their phones on (and thus leave themselves "logged in") 24 hours a day. A password store decrypted at login time would be readily available to a pickpocket, even if it is stored securely when the phone is deactivated. The pickpocket attack reveals another challenge to mobile password security: there is a difference between security measures that protect user credentials on "disk" when the phone is off, and measures that protect the credentials during an active session.

Android will be implementing its own form of full-disk encryption with 3.0 ("Honeycomb"), which will offer protection against extracting information from the flash storage card. But this will not help any users on earlier versions of Android, nor does it protect against attackers who steal running phones.

Google cannot force third-party application developers to store passwords wisely, but many commenters on bug 10809 feel that the company ought to provide a more secure option for the set of default Android applications. The discussions on the bug tracker and on external sites like Android Central make for interesting reading.

Some suggested encrypting (or obfuscating) the password file itself, and storing the key to that file elsewhere, such as on a "keychain" stored on the device. But as Android developer Andy Stadler pointed out, this does not improve security, it only moves the weak link to a different file. The password safe must be decrypted for the application to use it, which means the password to the password safe must itself be stored in plain text for unattended access. As he put it, there is "a simple test: if you can boot up the device and it will begin receiving email on your configured accounts, then the passwords are not truly secure. They are either obfuscated, or encrypted with another key stored somewhere else."

Others suggested not storing the passwords locally at all, but storing only a hash of the password instead. This approach does not work either, as user hyperhacker noted, because hash functions are one-way, and the original password is what must be sent to the server. Thus in practice either the user will be prompted for the password to validate the hash value (in which case the stored hash is clearly useless), or else the "hash" function is actually an encryption function, and the scheme is no different from the other encrypt-the-password-file proposals mentioned above.

Inconvenient security

The "executive summary" of the situation is that either the OS has access to a password in plaintext without user intervention, or else the user must be prompted to supply a password that is stored nowhere on the device. The real question is how much inconvenience the user will tolerate.

A scheme that asks the user for a boot-time passphrase to unlock a password safe is not very secure because of how long phone sessions typically last. As user mahashel put it in a related bug report:

One alternative would be to re-prompt the user for a passphrase whenever the device is awakened from an idle or sleep state. User aigarius argued that this would prevent the device from checking email in the background. "If you 'tie' it to the unlock code, then you can only check the email when the phone is unlocked, which is clearly quite useless." This is not true in the strictest sense: the OS could "forget" the password only when the user begins the unlock procedure — that way the background process could continue to check messages and activate alerts, and a pickpocket would still be locked out — but it still requires the user to enter a passphrase, PIN code, gesture, or some other form of credential every time he or she picks up the phone to do anything. That level of inconvenience seems to be anathema to most consumers.

User solinym suggested biometrics might be better received among consumers than passwords, with a caveat: "If you think key rotation is a PITA, you haven't tried replacing your fingerprints." He also raised the question of hardware security, tamper-resistance in particular, suggesting that the hardware required to prevent hardware key-extraction would add little to the cost of modern smartphones.

Gertvdb suggested storing a "master password" on the phone's SIM card, because SIM cards are more secure by default, they limit the number of attempts to enter the PIN code, and they have storage capacity. "The phonebook / SIM SMS message store should have enough space, even on 8k SIM cards. (And doesn't seem to be used by Android) (Some compression might make sense, to reduce the amount of storage needed)." Of course, not all phones use SIM cards, and on those that do, denial-of-service is a risk, as an attacker (perhaps a casual one just snooping around) could exceed the allowed number of PIN guesses and lock out the phone owner, even if they do not gain access to the data on the device.

As is typical of security debates, the talk returned time and time again to whether or not providing some form of weak security was worthwhile (to prevent casual attackers), or whether, in the absence of a foolproof solution, all other solutions give users a "false sense of security" and are "a complete waste of time" for developers to attempt.

Eventually a number of suggestions arose that amount to offering user-selectable "security levels," through which security-conscious users could enable strong encryption of their credentials at the cost of automated access, and the more laissez-faire could dispense with it. User danthanos called for a public API providing a secure storage service:

Tangential issues

Google has not responded with a game plan. Stadler, who marked the bug as "medium" importance last September, also commented "rest assured, I am not closing this bug. We recognize that this is causing concern for some users, and we're going to look at identifying steps that can make your data more secure" — which remains the last public word on the subject from the Android team.

The full-disk encryption debuting in Honeycomb is unquestionably a step in the right direction, because it does offer protection against exposing one's password in the event of a memory card theft. Intriguingly enough, however, another commenter reported that Android already has an internal "Credential Storage" API that is not documented for public use. Several third-party Android discussion forums have also brought up this API, so it appears to be in the wild already, but there is no official discussion of it. It appears to be used by applications for storing VPN certificates and other credentials not chosen by the user, but without a published API it is not likely to be useful to developers (and abusing it may break important things as well).

Android does have a "remote wipe" function (as asked about by hyperhacker in the bug discussion thread) for administrators to activate in case of theft, although it must be set up in advance. Needless to say, a remote wipe is hit-or-miss: if activated before an attacker accesses the phone, it works, if activated afterward it does no good at all. Remote wipe can also be used in ways that users may not expect.

In the meantime, security-focused users and developers can always turn towards third-party Android Market tools to offer additional layers of protection. There are stand-alone applications for bulk password storage, although these do not automatically sign-in to email (or other services). There are applications that provide password-protected access to the application as a whole, and there are PGP clients and directory encryption utilities galore. Even with all of those add-ons activated, however, it will still be up to the user to make the final choice: would you rather have your phone check your email in the background (and store your account password in plaintext), or have your phone keep your password safe (and force you to supply a password for access). There's just no way to have it both ways.

Comments (18 posted)

Brief items

Security quotes of the week

-- Mike Masnick notes a lack of irony detection at the FBI

-- Richard Clayton reports on the ineffective blocking in a "Site Blocking" report

-- ShareMeNot

Comments (1 posted)

WASP: The Linux-powered flying spy drone that cracks Wi-Fi & GSM networks (Geek.com)

Geek.com takes a peek at the Wireless Aerial Surveillance Platform (WASP), which is a repurposed US Army unmanned aerial vehicle (UAV or drone) outfitted to intercept WiFi and GSM. It will be demonstrated at the upcoming Black Hat and DEFCON security conferences. "If you happen to see this yellow drone flying above your neighborhood you’d be right to be concerned. WASP is equipped with the tools to crack Wi-Fi network passwords made possible by an on-board VIA EPIA Pico-ITX PC running BackTrack Linux equipped with 32GB of storage to record information. BackTrack offers a full suite of digital forensics and penetration testing tools making it a good fit for this setup. [...] WASP can also act as a GSM network antenna meaning it will be able to eavesdrop on calls/text messages made over that network by any phone deciding to connect through it."

Comments (none posted)

New vulnerabilities

clamav: denial of service

Package(s):	clamav	CVE #(s):	CVE-2011-2721
Created:	July 28, 2011	Updated:	November 7, 2011
Description:	From the Ubuntu advisory: It was discovered that the hash processing code in libclamav improperly handled messages with certain hashes. This could allow a remote attacker to craft a document that could cause clamav to crash, resulting in a denial of service.
Alerts:	Fedora FEDORA-2011-15119 2011-10-29 Fedora FEDORA-2011-15076 2011-10-29 Gentoo 201110-20 2011-10-23 openSUSE openSUSE-SU-2011:0940-1 2011-08-24 Fedora FEDORA-2011-10053 2011-08-02 Fedora FEDORA-2011-10090 2011-08-02 Mandriva MDVSA-2011:122 2011-08-13 Pardus 2011-106 2011-08-08 Ubuntu USN-1179-1 2011-07-28

Comments (none posted)

erlang: predictable seeds used for random number generation

Package(s):	erlang	CVE #(s):	CVE-2011-0766
Created:	August 1, 2011	Updated:	November 26, 2012
Description:	From the CVE entry: The random number generator in the Crypto application before 2.0.2.2, and SSH before 2.0.5, as used in the Erlang/OTP ssh library before R14B03, uses predictable seeds based on the current time, which makes it easier for remote attackers to guess DSA host and SSH session keys.
Alerts:	Fedora FEDORA-2011-9657 2011-07-23 Fedora FEDORA-2011-9598 2011-07-23 Mageia MGASA-2012-0338 2012-11-23

Comments (none posted)

foomatic: arbitrary code execution

Package(s):	foomatic	CVE #(s):	CVE-2011-2697
Created:	August 1, 2011	Updated:	September 23, 2011
Description:	From the Red Hat advisory: An input sanitization flaw was found in the foomatic-rip print filter. An attacker could submit a print job with the username, title, or job options set to appear as a command line option that caused the filter to use a specified PostScript printer description (PPD) file, rather than the administrator-set one. This could lead to arbitrary code execution with the privileges of the "lp" user.
Alerts:	CentOS CESA-2011:1109 2011-09-22 Ubuntu USN-1194-1 2011-08-22 Mandriva MDVSA-2011:125 2011-08-14 CentOS CESA-2011:1109 2011-08-14 openSUSE openSUSE-SU-2011:0892-1 2011-08-12 Scientific Linux SL-foom-20110801 2011-08-01 Red Hat RHSA-2011:1109-01 2011-08-01 Debian DSA-2380-1 2012-01-04 Gentoo 201203-07 2012-03-05

Comments (none posted)

foomatic: arbitrary code execution

Package(s):	foomatic	CVE #(s):	CVE-2011-2964
Created:	August 2, 2011	Updated:	August 22, 2011
Description:	From the Scientific Linux advisory: An input sanitization flaw was found in the foomatic-rip print filter. An attacker could submit a print job with the username, title, or job options set to appear as a command line option that caused the filter to use a specified PostScript printer description (PPD) file, rather than the administrator-set one. This could lead to arbitrary code execution with the privileges of the "lp" user.
Alerts:	Ubuntu USN-1194-1 2011-08-22 Fedora FEDORA-2011-9575 2011-07-22 Fedora FEDORA-2011-9554 2011-07-22 Mandriva MDVSA-2011:125 2011-08-14 openSUSE openSUSE-SU-2011:0892-1 2011-08-12 Red Hat RHSA-2011:1110-01 2011-08-01 Scientific Linux SL-foom-20110801 2011-08-01 Debian DSA-2380-1 2012-01-04 Gentoo 201203-07 2012-03-05

Comments (none posted)

kernel: multiple vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2011-1479 CVE-2011-1927 CVE-2011-2484 CVE-2011-2493 CVE-2011-2495 CVE-2011-2498
Created:	August 2, 2011	Updated:	September 14, 2011
Description:	From the openSUSE advisory: CVE-2011-1479: A regression in inotify fix for a memory leak could lead to a double free corruption which could crash the system. CVE-2011-1927: A missing route validation issue in ip_expire() could be used by remote attackers to trigger a NULL ptr dereference, crashing parts of the kernel. CVE-2011-2484: The add_del_listener function in kernel/taskstats.c in the Linux kernel did not prevent multiple registrations of exit handlers, which allowed local users to cause a denial of service (memory and CPU consumption), and bypass the OOM Killer, via a crafted application. CVE-2011-2493: A denial of service on mounting invalid ext4 filesystems was fixed. CVE-2011-2495: The /proc/PID/io interface could be used by local attackers to gain information on other processes like number of password characters typed or similar. CVE-2011-2498: Also account PTE pages when calculating OOM scoring, which could have lead to a denial of service.
Alerts:	Oracle ELSA-2011-2037 2011-12-15 Red Hat RHSA-2011:1813-01 2011-12-13 Ubuntu USN-1285-1 2011-11-29 Ubuntu USN-1281-1 2011-11-24 Ubuntu USN-1279-1 2011-11-24 Ubuntu USN-1256-1 2011-11-09 Ubuntu USN-1245-1 2011-10-25 Ubuntu USN-1244-1 2011-10-25 Ubuntu USN-1243-1 2011-10-25 Ubuntu USN-1242-1 2011-10-25 Ubuntu USN-1241-1 2011-10-25 Ubuntu USN-1240-1 2011-10-25 Ubuntu USN-1239-1 2011-10-25 Scientific Linux SL-kern-20111020 2011-10-20 Ubuntu USN-1236-1 2011-10-20 CentOS CESA-2011:1386 2011-10-21 Red Hat RHSA-2011:1386-01 2011-10-20 Scientific Linux SL-kern-20111005 2011-10-05 Red Hat RHSA-2011:1350-01 2011-10-05 Ubuntu USN-1218-1 2011-09-29 Ubuntu USN-1216-1 2011-09-26 CentOS CESA-2011:1212 2011-09-22 Debian DSA-2310-1 2011-09-22 Ubuntu USN-1212-1 2011-09-21 Ubuntu USN-1208-1 2011-09-14 Ubuntu USN-1205-1 2011-09-13 Ubuntu USN-1204-1 2011-09-13 Ubuntu USN-1203-1 2011-09-13 Ubuntu USN-1202-1 2011-09-13 Ubuntu USN-1253-1 2011-11-08 Ubuntu USN-1201-1 2011-09-13 Red Hat RHSA-2011:1253-01 2011-09-12 Debian DSA-2303-2 2011-09-10 Scientific Linux SL-kern-20110906 2011-09-06 Debian DSA-2303-1 2011-09-08 Red Hat RHSA-2011:1212-01 2011-09-06 SUSE SUSE-SU-2011:0984-3 2011-09-02 SUSE SUSE-SU-2011:0984-2 2011-09-02 SUSE SUSE-SA:2011:038 2011-09-01 Scientific Linux SL-kern-20110823 2011-08-23 Red Hat RHSA-2011:1189-01 2011-08-23 Fedora FEDORA-2011-11103 2011-08-18 Ubuntu USN-1193-1 2011-08-19 SUSE SUSE-SU-2011:0899-1 2011-08-12 SUSE SUSE-SA:2011:034 2011-08-12 Ubuntu USN-1186-1 2011-08-09 openSUSE openSUSE-SU-2011:0861-1 2011-08-02 openSUSE openSUSE-SU-2011:0860-1 2011-08-02 Ubuntu USN-1379-1 2012-02-28 Ubuntu USN-1380-1 2012-02-28 Ubuntu USN-1383-1 2012-03-06 Ubuntu USN-1386-1 2012-03-06 Ubuntu USN-1387-1 2012-03-06 Ubuntu USN-1394-1 2012-03-07 Oracle ELSA-2012-0150 2012-03-07

Comments (none posted)

libsoup: directory traversal

Package(s):	libsoup	CVE #(s):	CVE-2011-2524
Created:	July 29, 2011	Updated:	March 23, 2012
Description:	From the Red Hat advisory: A directory traversal flaw was found in libsoup's SoupServer. If an application used SoupServer to implement an HTTP service, a remote attacker who is able to connect to that service could use this flaw to access any local files accessible to that application via a specially-crafted request. (CVE-2011-2524)
Alerts:	Debian DSA-2369-1 2011-12-21 Fedora FEDORA-2011-9820 2011-07-31 Pardus 2011-108 2011-09-05 Fedora FEDORA-2011-9763 2011-07-31 Ubuntu USN-1181-1 2011-07-28 Scientific Linux SL-libs-20110728 2011-07-28 Red Hat RHSA-2011:1102-01 2011-07-28 openSUSE openSUSE-SU-2011:0875-1 2011-08-05 Mandriva MDVSA-2012:036 2012-03-23

Comments (none posted)

libwebkit: arbitrary code execution

Package(s):	libwebkit	CVE #(s):	CVE-2011-1774
Created:	August 2, 2011	Updated:	August 3, 2011
Description:	From the CVE entry: WebKit in Apple Safari before 5.0.6 has improper libxslt security settings, which allows remote attackers to create arbitrary files, and consequently execute arbitrary code, via a crafted web site. NOTE: this may overlap CVE-2011-1425.
Alerts:	SUSE SUSE-SU-2011:0857-1 2011-08-02

Comments (none posted)

mapserver: SQL injection

Package(s):	mapserver	CVE #(s):
Created:	August 2, 2011	Updated:	August 21, 2012
Description:	MapServer versions 4.x, 5.x and 6.x are potentially vulnerable to SQL injection, particularly if OGC protocols are enabled and with layers connecting to an SQL RDBMS backend, either natively or via OGR. See the MapServer advisory for details.
Alerts:	Fedora FEDORA-2011-9555 2011-07-22 Fedora FEDORA-2011-9517 2011-07-22 Mageia MGASA-2012-0230 2012-08-21

Comments (none posted)

selinux-policy: unspecified vulnerabilities

Package(s):	selinux-policy	CVE #(s):
Created:	August 2, 2011	Updated:	August 3, 2011
Description:	From the Scientific Linux advisory: These updated selinux-policy packages include a number of bug fixes and enhancements. Space precludes documenting all of these changes in this advisory.
Alerts:	Scientific Linux SL-seli-20110721 2011-07-21

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The 3.1 merge window is still open. See the summary below for what has been merged in the last week.

Stable releases: The 2.6.35.14 longterm kernel was released on August 1.

The 2.6.39.4 stable kernel was released on August 3: "Please note, this is the LAST release of the 2.6.39 kernel series. All users of 2.6.39 should be moving to 3.0 right now. This tree is now end-of-life, please plan accordingly."

The 3.0.1 stable kernel is currently in the review process. It can be expected to be released on or after August 5.

Comments (1 posted)

Quotes of the week: Merge window grumpiness edition

-- Linus Torvalds (additional grumpiness contained within)

-- Linus Torvalds

-- Linus Torvalds

Comments (none posted)

Kernel development news

3.1 merge window part 2

Around 1400 non-merge changesets have been pulled into the mainline since last week's merge window article. That makes for 6844 changesets since the 3.0 release at the time of this writing. Linus Torvalds is still on vacation, and the merge numbers are a bit down compared to previous releases, so there may be more to come. Significant user-visible changes include:

• The LIO iSCSI target has been merged. This was somewhat controversial, first because the competing SCST target implementation was pushed aside in favor of LIO, but also because there were questions about whether the CHAP authentication should be done in the kernel or in user space. The version that got merged does that authentication in the kernel over the objections of James Bottomley.

• pNFS now supports IPv6.

• eCryptfs now has support for encrypted keys.

• md now has support for bad block management.

• tools/power/cpupower has been added with tools to monitor power management for multiple architectures, and is eventually slated to replace the Intel-specific tools in tools/power/x86.

• dm now supports separate metadata devices for better fault handling and array sanity checking.

• New hardware supported includes:

  • Audio/video: Cirrus Logic CS421x codecs, Xceive XC4000 silicon tuners, ATMEL Image Sensor Interface (ISI), Endpoints SE401 USB cameras, Marvell Armada 610 integrated camera controllers, NXP TDA18271c2 silicon tuners, TI DM644x video processing back-end (VPBE) displays, Samsung S5P family video devices, OmniVision OV5642 camera sensors, Micronas DRX-K DVB-C/T demodulators.

  • Miscellaneous: Freescale MMA8450 accelerometers, InvenSense MPU3050 tri-axial gyroscope sensors, Kionix KXTJ9 accelerometers, Xilinx watchdog timers, ADP1653 LED flash controllers, Digital Devices Octopus bridge devices, Maxim MAX1668 and compatible temperature sensors, NTC NCP15WB473, NCP18WB473, NCP21WB473, NCP03WB473, and NCP15WL333 temperature sensors, National Semiconductor LM95245 dual temperature sensors, National Semiconductor LM25066, LM5064, and LM5064 power management, monitoring, control, and protection sensors, Maxim MAX8998/LP3974 PMIC battery chargers, Maxim MAX8997/MAX8966 PMIC battery chargers, TI TPS95612 power management chips, TI TPS65912 power regulators, AnalogicTech AAT2870 backlights, AnalogicTech AAT2870 power regulators, Cirrus Logic EP93xx DMA controllers.

Changes visible to kernel developers include:

• Multiple ARM SoCs and device drivers have added device tree support.

• A watchdog timer driver core has been added.

• The SLUB slab allocator no longer requires locks on the fast path for architectures that support cmpxchg.

• EFI non-volatile storage can now be used as a pstore backend to persistently store log messages or other information.

One notable patch set that will not be merged this time around is the Native Linux KVM tool. Torvalds decided that he needed more convincing before merging it:

The normal two-week merge window would nominally expire on August 5, but Torvalds's vacation may alter that somewhat (in either direction). We'll pick up any significant merges in an update next week should it be warranted. Stay tuned ...

Comments (none posted)

SKB fragment lifetime tracking

In May, LWN examined the "stable pages" patch, whose intent is to be sure that pages under I/O cannot be modified (by the kernel or user space) until the I/O completes. Block I/O is not the only context in which this kind of problem arises, though; memory which has been given to the network stack should also be kept stable until the transmission is complete. Unfortunately, it is hard to know when the network stack is truly done with a page, leaving the system open to possible data corruption problems.

Ian Campbell described how things can go wrong in June. Imagine a page full of data to be written to a file on an NFS-mounted filesystem. The NFS code will put together a network I/O operation as represented by an sk_buff structure (an "SKB") and pass it into the network stack for transmission. Perhaps the server is slow or the network is noisy; one way or another the acknowledgment from the remote NFS server is slow in coming - slow enough that the network stack decides to retransmit the request. While the data sits in the retransmission queue (perhaps already handed to the interface driver), the ACK arrives from the server. The network stack will tell the NFS client that the operation has completed. The page used to contain the data to be written could then be rewritten with some other data - even though that retransmission is still waiting to go out. The result could be a (re)transmission of corrupted data. This problem is especially acute for O_DIRECT writes - where the application is waiting for the end of the operation - but it can come up in other situations as well.

SKBs can have a destructor function, so one would think that it would be possible to just wait until the network stack finishes with the structure before releasing the relevant page(s). But the network stack works in strange and mysterious ways, and the fact that it has finished with an SKB does not imply that it is finished with the pages of data referenced by that SKB. The "cloning" of SKBs happens often in the network stack, and pages of data can actually move between SKBs. The networking code manages the page reference counts directly, so there is no danger of the data pages being put to some other use entirely by the system. But that is not helpful to higher layers, which have no way to know when it's safe to signal the completion of an operation.

Fixing this problem requires a significant set of changes to the low-level SKB-handling code. Ian's patch series starts by defining a set of helper functions for the tracking of references to pages from SKBs. Current networking code calls get_page() and put_page() directly; after patching, all of those calls have been wrapped by functions like skb_frag_ref(). Quite a few changes are required to get the networking core and in-tree drivers to use these functions.

Once that is done, the patch series introduces the concept of a "fragment destructor" for SKBs:

    struct skb_frag_destructor {
	atomic_t ref;
	int (*destroy)(void *data);
	void *data;
    };

The low-level functions that add fragments to SKBs are modified to take an additional destructor argument. The destructor is always optional; code which does not need to use destructors can simply pass a null pointer instead.

At this point, it's a relatively simple matter for the accessor functions added earlier in the series to increment and decrement the reference count whenever there are destructors present. When the reference count (ref) drops to zero, the provided destroy() function will be called. Putting the reference counter in the destructor is a useful optimization: in the absence of destructors, the overhead of maintaining the reference count can be skipped. Also worth noting is the fact that multiple fragments in an SKB can share the same destructor; in this case, the destroy() function will only be called when the networking code has finished with all of those fragments.

One other optimization is that, in the presence of a destructor, the network code will no longer increment and decrement the reference counts associated with the pages in the fragments. In this situation, the calling code is assumed to hold a reference for the duration of the operation, so separate reference counting at that level is not needed.

The final step is to make use of this capability. The internal kernel_sendpage() function gains an extra parameter to hold a pointer to the destructor, should the caller want to use one. The sunrpc code is changed to not signal completion of operations until the networking code indicates that it is done with the associated memory. And that solves the problem - for NFS at least; there should be no more troubles with pages being reused while they are still under network I/O. There are other places in the kernel which can - and presumably will - make use of this functionality in the future; this work was originally motivated by problems encountered in the implementation of zero-copy I/O for Xen clients. Ian suspects that subsystems like iSCSI could also benefit from this mechanism.

The patch set seems to have been relatively well received. There will be another posting at some point reorganizing some of the work, but there does not appear to be a need for significant changes at this point. So this feature seems likely to appear in the 3.2 kernel.

Comments (2 posted)

Meet the Lockers

Our story begins with an observation made by Jon Corbet in a recent article about delays in the release of Linux-3.0:

While undeniably true, it is not clear that this note of gloom is entirely justified by the immediate circumstances - after all the bug was found and fixed to everyone's satisfaction in a little over 24 hours. However it is clear that it wasn't just this event that triggered the observation. Earlier we read:

What was once "approachable" is now "difficult to understand". If these words were used to describe a life-partner, we might be looking to choose between a divorce settlement or counseling, though often it is a healthy dose of understanding that is really needed.

The human touch

The allusion to human relationships is not as far fetched as it might seem. In his excellent book "The Maths Gene", Keith Devlin writes about Wim Klein, "a famous calculating wizard who in the days before electronic computers once held a professional position with the title 'computer'".

Apparently Klein was known to say: "Numbers are friends to me." and referring to 3844 he says:

The point Devlin is leading to is summed up thus:

The idea is that we can treat abstract ideas more like friends than tools and understand them holistically instead of just functionally. If we get to know their preferences and peculiarities we can side step the fact that they are sometimes difficult to understand and once again find them approachable.

One more quote from that recent LWN article will start us on the path to understanding what it means to be friends with abstractions. In a parenthetical comment concerning an assignment of NULL to d_inode, Jon writes:

This is just the sort of thing that suggests a holistic "friendly" relationship rather than a more distant formal one. Something just doesn't seem right, "He doesn't usually behave this way". Being able to make that sort of observation helps you focus in and ask incisive questions. You should then either be able to help your friend, or else understand them at a deeper level. We'll come back to this particular observation later, but first I want to demonstrate the effectiveness of this sort of friendship in a context that all experienced C programmers should find very familiar.

"For": he's a jolly good fellow

A simple and common construct in C programs is the "for" loop. It is extremely flexible and can be used like a "while" loop, can iterate over a set of numbers, can walk a linked list, and can perform many more tasks. A common form that you might expect to see quite often is:

    for (i = 0; i < count; i++)

This will execute the body of the loop "count" times assigning "i" to each value from "0" to "count - 1" in turn. There are similar constructs that are almost as common and familiar such as

    for (i = 0; i <= final; i++)

or

    for (i = 1; i <= count; i++)

which are really just minor variations on the theme. But if you see

    for (i = 0; i <= count; i++)

a seasoned C programmer, familiar with the ways of the "for", will immediately think that something is odd. Not necessarily wrong, but certainly odd. This might be an error, but it could also be that "count" has been misnamed and is really a "final" rather than a "count" sort of value (an ordinal rather than a cardinal). Or maybe the loop really has a good reason for executing "count + 1" times. At this point the friendship has served a valuable purpose by identifying a possible issue, but must give way to a more detailed analysis to determine if there really is a problem.

We can make this very concrete by examining the Linux kernel source code (v3.0) which is often a good source of interesting code fragments. A search for the above pattern can be achieved with:

    git grep 'for.*(.*= *0.*;.*<=.*c[ou]*nt.*;.*)'

which results in:

    arch/ia64/kernel/cpufreq/acpi-cpufreq.c:	for (i = 0; i <= data->acpi_data.state_count; i++)
    drivers/gpu/drm/radeon/evergreen_cs.c:	for (i = 0; i <= pkt->count; i++, idx++, reg += 4) {
    drivers/gpu/drm/radeon/r100.c:	for (i = 0; i <= pkt->count; i++, idx++) {
    drivers/gpu/drm/radeon/r100.c:	for (i = 0; i <= (pkt->count + 1); i++, idx++) {
    drivers/gpu/drm/radeon/r100.c:	for (j = 0; j <= count; j++) {
    drivers/gpu/drm/radeon/r600.c:	for (j = 0; j <= count; j++) {
    drivers/gpu/drm/radeon/r600_cs.c:	for (i = 0; i <= pkt->count; i++, idx++, reg += 4) {
    drivers/i2c/algos/i2c-algo-pcf.c:	for (i = 0; i <= count; i++) {
    drivers/staging/rts_pstor/rtsx_transport.c:	for (i = 0; i <= buf_cnt / (HOST_SG_TBL_BUF_LEN / 8); i++) {
    fs/ufs/balloc.c:		for (pos = 0; pos <= uspi->s_fpb - count; pos++) {
    scripts/docproc.c:	for (i=0; i <= symfilecnt; i++)

A closer examination of each of these is left as an exercise for the reader. The chance of finding a genuine bug is very high, though it is likely that the bugs are not very serious or they would have already been fixed.

The success in finding a bug even when we weren't really looking for one is evidence of the value of building strong relationships with important abstractions as we can learn to recognize early warning signs. Unfortunately the building of strong relationships can be time-consuming and so is often neglected, being left to those with a passion rather than those who simply have a task to do.

In human relationships a friendship can blossom more quickly if a mutual friend acts to introduce two parties and start them out on a sound footing. The same can work for abstractions. While much has been written about RCU ("Read-Copy-Update" which - from the name "rcu-walk" seems to be the center-point of the aforementioned bug), it has largely been of a formal style. The documentation in linux/Documentation/RCU/ and on Wikipedia is, not unlike a résumé, suitable for an introduction to a potential employer, but not so much as an introduction to a future friend. What follows is an attempt to present RCU together with other related technologies in a somewhat different style aimed at building friendships - what you might call the human-interest perspective.

A family affair

While it is wrong to judge someone based on their parents, knowing their family background can certainly lead to a deeper understanding. So we will start by introducing RCU's parents who are, for the purpose of this tale, Sir Spinlock and Dame Refcount - both recently honored for services to computer engineering.

Sir Spinlock is somewhat "old school" and a stickler for doing everything "just right". When he protects a data structure he does it "properly" (as he says) and ensures everything is consistent, stable and completely up-to-date. His wife, Dame Refcount, feels that her husband takes a rather short-term view of things and, like many mothers, just wants to be sure that her charges are safe and can be found when needed (comments about apron strings are generally not appreciated).

The gentleman and his lady had another child much earlier (too early some say) who was named "Rwlock". When write access is needed, he is much like his father becoming a short term exclusive lock. When read access is wanted he is more like his mother, keeping count of all the different readers. However on the whole he seems to have received the worst of both worlds, lacking the long term connections of his mother, or the simple fairness available to his father. Master Rwlock doesn't get out much these days.

In some ways Rwlock is of the same generation as his parents as none of them really seem to understand the problem of cache line bouncing. Sir Spinlock likes to say that "Cache lines belong in banks" and seems to think he is being awfully funny, but it just makes the younger generation feel awkward, smile weakly, and try to change the subject.

That younger generation are the twins - RCU and his sister Seqlock. They both grew up in the shadow of Sir Spinlock's dominance of multiprocessing and each reacted in their own way. They have both learned the importance of stability but were culturally more focused on the appearance than the reality, and technologically more inventive in how to provide it.

RCU explains that he agrees with his dad's mantra of "Always consistent, always stable" but finds that the "always up-to-date" idea is over-rated. He himself finds that he is usually running late for something or other and doesn't see that it has hurt him all that much. RCU is a big fan of the photo-copiers and realized early that the best way to preserve the appearance of stability was not to change the original at all but to make a copy and work on that. So while RCU always provides stable data, there might be something newer that he is working on that he won't let you see just yet. RCU's real skill is understanding use-by dates. What you get might not be perfectly fresh, but it will never become stale.

His sister, Seqlock, takes a very different approach to the same problem learning from her favorite piece of technology - the digital camera. Like many of us she just takes lots and lots of photos and picks the ones she likes. "All of them are stable", she says, "and it isn't too hard to tell when one isn't consistent - just throw those away". You might not think it is so easy, but then this is her special skill - seeing the blur in the photo and taking another. She is happy to help and tell you when a new photo is needed.

Another notable contrast between the twins is reflected in their mother's pet names for them: "replace" and "in-place" (she sometimes calls them "The place twins" which they find very irritating). RCU insists on updating things by replacing them so he is happiest with relatively small objects without too much linkage which would need to be re-threaded. He "doesn't work on books", he says, "just collections of pages, so don't try looking at two pages at once!". When he does have these small objects he can replace them with lightning speed, so others hardly even notice anything has changed.

Seqlock expects people to take a photo if they are interested so she just updates things when ever she likes, where ever they are - just being sure to add enough "blur" so that if she is caught "in the act" another photo will be taken. This means she can handle much bigger containers than her brother, and containers with lots of connections. Even a wide-angle lens can notice the blur and take a second photo.

A chip off the old lock

When we look more closely at RCU to see how he achieves his flashy performance we can see echoes of both his parents at work. When he was younger he found this mildly embarrassing but since they were both knighted he has found a new respect for his heritage.

From his father, RCU appreciates the value of a quick change when no-one else can see and has refined this down to a single pointer update which modern hardware guarantees to be atomic. When he is really trying to impress he uses the "xchg()" atomic operation to replace an old structure with a new structure without even needing any locking for an update (though there are some costs to this and it is mostly just a party trick - but see net/ipv4/cipso_ipv4.c:cipso_v4_req_setattr()).

This observation about the centrality of the pointer is crucial to understanding RCU's sleight-of-hand. It is easy to get distracted by the "rcu_read_lock() / rcu_read_unlock()" calls and think that they provide some sort of atomicity, but they don't. Rather they come from his mother's side of the family as we will see shortly. As the atomicity is constrained to a single pointer, two separate pointer accesses - even in the one rcu_read_lock()ed section - will not be coherent. This means it is very important not to dereference the same pointer twice (you might get two different values) and if you need to read two separate pointers, be very careful about assuming any relationship between the two values - there might be one but you need to understand the rest of the code to be sure. The most common case of dereferencing multiple pointers in the one rcu_read_lock()ed region is when following a linked list. Each pointer found leads directly to the next. In this case you can be sure that each consecutive pair will be ordered if there is ordering in the list, but you cannot be sure you will see all entries that are in the list when you started, or that you have seen all the entries that are in the list when you finish.

Now that we know that friends don't let RCU-friends assume coherence outside of a single protected structure, it should be easy to see that something looks wrong in kernel/cgroup.c:cgroup_attach_proc(). The relevant code snippet - very well commented which helps a lot - is:

	rcu_read_lock();
	if (!thread_group_leader(leader)) {
		/*
		 * a race with de_thread from another thread's exec() may strip
		 * us of our leadership, making while_each_thread unsafe to use
		 * on this task. if this happens, there is no choice but to
		 * throw this task away and try again (from cgroup_procs_write);
		 * this is "double-double-toil-and-trouble-check locking".
		 */
		rcu_read_unlock();
		retval = -EAGAIN;
		goto out_free_group_list;
	}

where:

	#define thread_group_leader(p)	(p == p->group_leader)

is defined elsewhere.

This code is assuming that rcu_read_lock() will provide coherence between this access of "leader->group_leader" here and other accesses to something in the "leader" structure later on in "while_each_thread()". While this is not totally impossible, as a well placed "synchronize_rcu()" between updates could lead to some coherence, it is not the usual way RCU is used and questions need to be asked. A discussion with the author of this code suggests that what is really needed is a read_lock() on "tasklist_lock".

So just like with the case of the "for" loop the fact that something "doesn't look right" isn't a guarantee that it is wrong but it is certainly an indication that the code is worth checking.

From his mother, RCU learned the value of counting to keep track of things. While his mum has a counter in each individual object, RCU tracks the whole system state (or at least that part entrusted to him) with a single counter (though admittedly it is spread across all CPUs). The details of getting this just right are are rather tricky, but as mentioned earlier, that is his real skill.

Sometimes he thinks this one skill is over-rated and finds a real kinship with the artists and musicians he knows who can only find work in copying the ancient masters or teaching unwilling children to play. In RCU's case the skill that most people pay for is not the "quick change, always stable" trick that he is so proud of, but the "never stale" promise he has to make to achieve this. Among the several hundred times that RCU is employed by the Linux kernel, a substantial majority simply use RCU as an extra, cheap, reference count to stop things from going stale until they are really not needed. In a number of cases this is very explicit in a "put" function. For example kernel/audit_tree.c has:

static inline void put_tree(struct audit_tree *tree)
{
	if (atomic_dec_and_test(&tree->count))
		call_rcu(&tree->head, __put_tree);
}

which makes it very clear where RCU stands - one step below his mother (we note that she isn't complaining, only him).

Even when RCU does get to use his quick-change trick it is usually to simply replace an old value with a new value - no real copying is done. Like most of us, he hardly ever uses his middle name.

Some reasonably simple examples of true RCU where the old value is copied and updated are in fs/afs/security.c:afs_cache_permit() and drivers/md/linear.c:linear_add(). In both these cases an array needs to be made bigger, so a new structure is allocated and RCU is used to slip it into place without anyone noticing.

Others, possibly more genuine (as they aren't just for a size change), can be found where "list_replace_rcu()" or "hlist_replace_rcu()" are used (about a dozen places in Linux-3.0). For example in net/ipv4/fib_trie.c:fib_table_insert(), RCU is used to update several fields in a "fib_alias" atomically.

Seeking Seqlock's secret

Returning to Miss Seqlock and examining more closely her relationship with her parents we find that she is very close to her father and depends on him to provide exclusion from other writers when writing to a protected data structure (though in the Linux kernel she has a more independent alter ego named "Seqcount" who leaves the write-side exclusion to be taken on by whoever is available).

On the other hand she shows her rebellious side by not requiring a strict bracketing of get/put calls. Sir Spinlock's calls must always pair a spin_lock() with spin_unlock() and RCU must always follow rcu_read_lock() by rcu_read_unlock(). Dame Refcount almost always pair "increment" with "decrement" though sometimes when there is only one reference left she won't bother decrementing but just throws the object away. Seqcount - possibly inspired by her mother, only pretends to need balance, on the read side at least. Her standard pattern is to balance "read_seqbegin()" with "read_seqretry()" as is shown quite eloquently in, for example, fs/nfs/nfs4state.c:nfs4_copy_stateid():

	do {
		seq = read_seqbegin(&state->seqlock);
		memcpy(dst, &state->stateid, sizeof(*dst));
	} while (read_seqretry(&state->seqlock, seq));

We have begin, copy, and retry. But it doesn't have to be like that. A simple variation can be seen in fs/cifs/dir.c:build_path_from_dentry() where three different error cases can return from the middle of the loop. In any other locking scheme the error path would need to unlock or release whatever was being held, but not with Miss Seqlock. When she is in charge you can check-out any time you like.

While this example copies the entire protected structure that is not necessary and would be prohibitive when Seqlock is protecting a large or complex structure. Rather it is only necessary to copy the pieces of interest. If they prove interesting you might copy some more bits and some more, each time only checking "read_seqretry()" to make sure the entire picture you are forming is valid (and not blurred). This is not yet common in Linux with the new scalable dcache being the only advanced user. This first exposure may yet gain more friends for Seqlock, though, and we can expect to see more of her as time passes.

It's all about teamwork

We have already noted that Seqlock usually depends on her dad for help with write-side exclusion and that RCU often helps out his mum. The co-operation doesn't end there. The parents have long worked together (it was in the middle of an "atomic_dec_and_lock()" that he proposed) and the twins carry on that tradition. Some times the family members take turns in protecting a structure and sometimes they share out responsibility for different fields, so Sir Spinlock might protect some fields in a structure while RCU guarantees that others won't change except by copying the whole.

As we now turn back to look at the context of the bug that started this discussion which was in the new path name lookup code we find that all four (main) members of the family are making significant contributions and working together.

As the earlier article on this topic outlines, Sir Spinlock and Dame Refcount are used for the "always works but sometimes slow" ref-walk option, while RCU and Seqlock help out for the "often works, always fast" rcu-walk option. This allows us to compare and contrast them in a real-life context.

Where ref-walk uses reference counts on one dentry after another to keep each dentry active, rcu-walk uses RCU to keep an effective reference count on all dentrys. Where ref-walk uses spinlocks to get a stable view of each dentry, rcu-walk uses seqlock to avoid using an unstable view.

To really understand this bug some degree of friendship with VFS is needed too, but fortunately not much. Central to the issue is the "d_inode" field of a dentry. It is fairly special because it doesn't change much. The only valid changes are from NULL to a valid inode pointer, and from a valid inode pointer back to NULL. And that second one is special too - it is only allowed to happen if there are no other references to the dentry.

The result of this in the old dcache (before rcu-walk) is that you don't need any locking for read access to d_inode, only a counted reference. If it is NULL, then you know the file doesn't exist. If it is not NULL, then the value is the file's inode and it won't change - the reference you have on the dentry becomes an implied reference on the inode.

As rcu-walk largely mapped spinlocks to seqlocks and refcounts to RCU you might assume that an RCU reference is sufficient to make accessing d_inode safe but it isn't quite that simple. To get a new reference on a dentry you need to be holding a spinlock (hence dget_locked() is the function to use). So while you only need a reference to access d_inode, you need to have previously taken a spinlock. This doesn't map directly into RCU usage as getting an RCU reference (with rcu_read_lock()) can never require a spinlock. So in rcu-walk, both Seqlock and RCU are used to protect d_inode. RCU holds a global reference so you can safely copy the d_inode value, and then later Seqlock will tell you if the copy you made is still valid.

It seems that there is room for some confusion on whether just a reference, or also a lock, is need to access d_inode and we see some of that in the patches proposed. The first patch proposed by Linus seems to treat d_inode like an RCU-protected field which should not be changed (as observed in our earlier parenthetical comment), and so removed the code that cleared it before the last RCU-reference was dropped. While it may have fixed the problem it didn't fully explain it as the field is meant to be Seqlock protected.

Linus's second patch added some Seqcount handling which seemed to acknowledge that the field was meant to be Seqlock protected, but also moved the access to d_inode towards the end of the loop. That loop shows off Seqlock's unbalanced nature quite nicely. It has a read_seqcount_begin() but had no read_seqretry(). Linus added a read_seqretry() which resulted in balance but that didn't really address the issue.

As mentioned earlier a "Seqlock protected region" is somewhat vague as it can have several read_seqretry() calls with each being used to validate what has just been copied. While a Seqlock protected region is, in some sense, still active when we enter this loop, it is no longer meant to protect d_inode. d_inode was copied out and validated in an earlier call to _d_lookup_rcu() (from do_lookup()). The current Seqlock protection will only be used if this inode is a directory. So in this loop there is no Seqlock protection for d_inode. The only reason it is still safe to access it at the end of the loop is because we know we have a mount point, and we know a mount point is pinned in such a way that d_inode cannot change (as the comment explains). At the top of the loop we don't have that certainty.

So it seems that the problem child here is Seqlock, but in fairness it isn't really her fault. She looks like a lock, but doesn't bracket things at all the same way that more traditional locks do and so those familiar with the older generation can get confused about exactly what she is protecting.

If we back-track a little and find out how the reference to d_inode came to not be properly protected we find commit 62a7375e5d77d654695 which moves the access to d_inode from the end of that loop to near the top. This looks like it should be safe as a Seqlock seems to be active, but as we now know, that can be deceiving. The reason for that move is probably worth exploring, but not here. This article is about friendship and friendship will rarely tell us how to fix a problem - we usually need to visit a specialist for that. But a friendship with data structures and locking mechanisms can help us identify which code is worthy of a closer inspection just as we have seen in this exploration.

In this case a friend would need to have known that d_inode requires Seqlock protection, and to have fully understood the rather independent approach that Seqlock takes to protected regions. Maybe Seqlock just needs some more friends.

Invitation

It has been said that given enough eyeballs, all bugs are shallow. For this to be true the eyes must also be able to focus and recognize (and even be open). Hopefully by gaining more friends who are able to recognize its faults, the Linux kernel can avoid the stigma of being difficult to understand and can once more be seen as approachable. Would you like to be friends?

Exercises:

1. Examine each unusual "for" loop found above and determine if each is a bug, an interesting usage, or an uninteresting false-positive.

2. "struct super" uses a secondary reference count "s_count" which provides similar functionality to that provided by RCU. Determine if any flavor of RCU can actually be used to achieve the same result. Outline the costs and benefits of such a change.

3. Understand why commit 62a7375e5d77d654695 moved the d_inode access as it did and so determine if commit 59430262401bec02 has undone some good work.

4. Examine the usage of RCU in jbd2_dev_to_name and list any features that "don't seem right". A deeper analysis is not required as this code was recently removed from the kernel.

[The author would like to thank Paul McKenney for providing valuable insights into the character of RCU.]

Comments (8 posted)

Patches and updates

Kernel trees

• Thomas Gleixner: 3.0-rt4 . (August 2, 2011)

• Thomas Gleixner: 3.0-rt6 . (August 2, 2011)

• Andi Kleen: The longterm Linux 2.6.35.14 kernel is released . (August 1, 2011)

• Greg KH: Linux 2.6.39.4 . (August 3, 2011)

Build system

• David Woodhouse: Enable 'make CONFIG_FOO=y oldconfig' . (August 2, 2011)

Core kernel code

• Jan H. Schönherr: sched: Enforce order of leaf CFS runqueues (and list cleanup) . (August 2, 2011)

Development tools

• Robert Richter: perf, x86: Implement AMD IBS . (August 2, 2011)

Device drivers

• Rafael J. Wysocki: PM / Domains: Allow generic PM domains to have multiple parents . (August 2, 2011)

• Timur Tabi: drivers/misc: introduce Freescale Data Collection Manager driver . (August 2, 2011)

Filesystems and block I/O

• Glauber Costa: Per-superblock dcache limitation . (August 2, 2011)

• Andi Kleen: Updated direct IO optimization patchkit v2 . (August 2, 2011)

Janitorial

• Paul Gortmaker: introduce export.h; reduce module.h usage . (August 2, 2011)

Memory management

• Christoph Lameter: SLUB: [RFC] Per cpu partial lists V3 . (August 2, 2011)

Architecture-specific

• H. Peter Anvin: Direct support for the x86 RDRAND instruction . (August 2, 2011)

Security-related

• Mike Waychison: kinit: Add drop_capabilities support. . (August 3, 2011)

Virtualization and containers

• Frederic Weisbecker: cgroups: Task counter subsystem (was: New max number of tasks subsystem) . (August 2, 2011)

Miscellaneous

• Matt Fleming: efilinux release 0.8 . (August 2, 2011)

• Bob Pearson: crc32: add slice by 8 algorithm . (August 3, 2011)

Page editor: Jonathan Corbet

Distributions

A look at OpenWrt

There are numerous Linux distributions focused on small network routers; most of them are ultimately derived from the WRT54G source that was extracted from LinkSys some years ago. One popular choice, Tomato, was reviewed here at the beginning of 2010. Tomato is not a hugely active project, though (one minor release in the 18 months since that review) and is not fully free software. Users wanting a more active, more free distribution are likely to settle on OpenWrt; your editor recently picked up a new router with the idea of playing with this distribution.

Unlike many router distributions, OpenWrt makes a point of supporting a wide range of hardware; see the "table of hardware" for the full list. The table is actually recommended reading for anybody thinking of getting a new router; beyond saying whether a specific router is supported, it provides a concise summary of the features of each.

The router used for this review is a Netgear WNDR3700v2; it is a nice device which runs OpenWrt well - but only for those who are willing to spend a bit of time at it. This product is quite new, so OpenWrt support is in an early state; there are no stable releases with WNDR3700v2 support, and probably will not be for a little longer (it seems that the upcoming 10.03.1 release will include this support). The OpenWrt WNDR3700v2 page describes in great detail how to get things going. Fortunately, much of the information found there (including advanced material like hardware mods) can be ignored at the beginning.

A running OpenWrt instance feels a lot like any other Linux distribution; one can use ssh to run a shell on the device and move around in the usual ways. But it does not take long for the differences to become apparent. One of the first of those is that OpenWrt uses its own packaging system, a tool called opkg. Its operation is easy enough to learn for anybody who is familiar with other package managers, but it is sufficiently different from the alternatives to require a bit of manual reading to get started.

OpenWrt has also taken an interesting approach to configuration; everything is meant to be found in a single directory (/etc/config), and in a single format. One learns quickly that there are some exceptions - /etc/hosts and /etc/ethers, for example, but the bulk of OpenWrt configuration fits within that framework. There is a set of tools making it easy for scripts to manipulate configuration files without breaking them, and another set which seems to implement the configurations by running the relevant daemons with large numbers of command line arguments. Configuring OpenWrt can lead one into a vast maze of obscure options, but, at least, one need not worry about application-specific configuration syntax.

Most of the time. Configuring OpenWrt as a simple router is not particularly difficult; in many situations it may simply work out of the box with no tweaks required. When one gets into the more advanced capabilities it gets rather more complicated. OpenWrt is aimed, after all, at users who are sufficiently motivated to wipe the factory firmware from their device and run something else; one can be sure that it gives those users the ability to tweak almost anything. Would you like, say, 16 options for the behavior of each LED on the front panel? OpenWrt is there for you, but you need to be prepared to spend some time reading the wiki to figure out how to do it.

One of the places where Tomato excels is with its web-based administration screens; unfortunately, that's also the part of Tomato that is non-free. OpenWrt has traditionally been for those who like to configure their routers with vi. That situation has not changed greatly, but the various efforts around creating a browser-based management interface seem to have coalesced around a system called "LuCI". LuCI makes many of the basic tasks around configuring (and monitoring) a router easy, and it makes some of them quite hard. It lacks the cleanness, focus, and ease of Tomato's interface, but it makes it up in volume. One cannot change every aspect of the router's configuration with LuCI, but it looks like the LuCI developers made their best effort toward that goal.

LuCI has an overview screen providing a quick view of the status of the router: load average, free memory, status of the interfaces, DHCP clients, wireless users, etc. There are realtime bandwidth-usage plots that are quite similar (suspiciously so) to those produced by Tomato, but without as much flexibility. Just about every aspect of how the network interfaces work can be configured through LuCI. For a wireless interface, for example, one can deal with mundane parameters like SSID, channel, encryption, and whether the interface is considered to be on the LAN or part of the outside world (or somewhere in between). But one can also tweak the transmit power, MAC address filtering, and many other things. It's all there, but one ends up digging through lots of tabs to find everything; sometimes it turns out to be easier just to go into /etc/config and edit the file directly.

LuCI also provides a basic interface to the package manager, allowing the installation and removal of specific packages. It turns out that there is a lot of stuff in OpenWrt that one can install if desired. As one might imagine, just about any tool which relates to networking is present, but there's more to it than that. Anybody with a desire to run apache, asterisk, cdparanoia, cups, erlang, git, lame, or about 2,000 others need only install the relevant packages. This is part of the power of OpenWrt, of course; it turns a cheap router into a low-power Linux box capable of doing any of a number of things never anticipated by the manufacturer.

Of course, one needs to have enough space and computing power in the router to actually run that stuff. The WNDR3700v2 has all of 16MB of flash; one is not going to be installing an office suite there. The OpenWrt developers have seemingly put a lot of effort into maximizing the resources (especially storage) available on the router, with some interesting implications.

In most installations, squashfs is used for the base filesystem. Squashfs is very good at compression, so the base image is about as small as it can be. The problem, of course, is that squashfs is also a read-only filesystem; the compression techniques used force that limitation. Needless to say, installing new packages onto a read-only filesystem - or even just saving configuration information - can be a challenge. To enable filesystem changes, OpenWrt sets aside a portion of flash for a separate JFFS2 filesystem which is then overlaid over the base filesystem using mini_fo. There is talk of switching to overlayfs instead, but that has not yet happened.

This arrangement can lead to some counter-intuitive results. Removing a package from the system, for example, can decrease the amount of free space available. Space on the squashfs filesystem cannot be recovered; it can only be papered over by adding notes to the overlay filesystem. There are tools for a suitably-motivated user to build a custom image with exactly what is needed, but, for most users, the OpenWrt distribution will remain mostly as it was when it was installed.

There is one important implication here: routinely updating packages is not something that OpenWrt users do. At even a low level of package churn, the available storage space would likely run out in a hurry. To make things worse, the bootloader requires the kernel to be placed in flash ahead of the first filesystem. Since (1) the kernel is not stored in the filesystem itself, and (2) a change in kernel size requires relocating the filesystems that come after it, it's really not possible to update the kernel on an OpenWrt installation. OpenWrt does make it relatively easy to update the entire distribution - configuration files are preserved - but that is not something administrators will do often.

That adds up to a bit of a scary situation. Routers are often in the position of being fully exposed to the Internet and having full access to the local network. They are an obvious target for attack, especially when they are capable of running anything that can be put onto a Linux system. The net is not full of stories of exploitable OpenWrt vulnerabilities, but the possibility always exists, especially if some of the more complex or obscure packages are installed. It seems solid, but your editor would sleep much better if OpenWrt had a better way to install updates and a process for dealing with security issues.

The flexible and widely-ported nature of OpenWrt make it, like Debian, a useful base for others to build their own distributions on. The stock firmware on the WNDR3700v2 is, in fact, derived from OpenWrt - a fact which presumably aided the developers working on that port. Comcast, a big US Internet provider, has a special OpenWrt distribution for those participating in its IPv6 tests. The openwrt-robe project repurposes routers for the development of in-vehicle systems. CoovaAP is a derivative intended for the easy deployment of public WiFi hotspots. PacketProtector, instead, uses OpenWrt to create a "unified threat management" device. The bufferbloat community will soon be releasing a debloated OpenWrt variant for testing. There are many more; OpenWrt has become a base distribution that has made a lot of creativity possible.

In summary: OpenWrt shows some of the best of what our community is capable of. This community has created a distribution which makes a variety of commodity routers into much more powerful devices than their manufacturer may have intended, and they have created a base with which many other things can be done. All that is lacking is just a bit more support from the vendors; wouldn't it be nice if routers just came with an OpenWrt variant with full access to the package repository and updates? Unfortunately, vendors still see themselves as selling black boxes rather than general-purpose devices. But, as long as we can put a proper distribution onto these devices ourselves, that will be good enough.

Comments (39 posted)

Brief items

Distribution quote of the week

-- Josh Boyer (joins the Fedora kernel team)

Comments (none posted)

Release for CentOS-6.0 Minimal

CentOS-6.0/minimal is available for i386 and x86_64 architectures. "The minimal install iso media is an alternative install to the main CentOS-6.0 distribution and comes with a trimmed down, preselected rpm list. However, it still runs off the standard installer, with all the regular features that one would expect from the main distribution, except the rpm selection screen has been disabled. Running an install from this media will not allow you to change the rpms selected for install." These images do contain the complete recovery and rescue environment found in the main distribution.

Full Story (comments: none)

Oracle Linux 5.7

Oracle has announced the release of the seventh update to Oracle Linux 5. Three sets of kernel packages are available with this release, including the Unbreakable Enterprise Kernel (the default), a Red Hat compatible Kernel, and a Red Hat compatible Kernel with bug fixes added by Oracle.

Comments (none posted)

Red Hat Enterprise Linux Extended Update Support 5.4 - End Of Life

The extended update support for Red Hat Enterprise Linux 5.4 has ended. "Note: This does not impact you unless you are subscribed to the Extended Update Support (EUS) channel for Red Hat Enterprise Linux 5.4."

Full Story (comments: none)

Scientific Linux 6.1

Scientific Linux has released version 6.1 for i386 and x86_64. "Scientific Linux release 6.1 is based on the rebuilding of RPMS out of SRPMS's from Enterprise 6 Update 1, both Server and Client." More information can be found in the release notes.

Comments (none posted)

Distribution News

Debian GNU/Linux

Release Update: Goals, Arches, Rolling, Removals

Neil McGovern has an update from the Debian release team, including release goals, architectures, 0-day NMU policy, improving the experimental repository, the package removal process, and a look at what is due in the next update.

Full Story (comments: none)

DebConf11 ends as another success for the Debian Project

DebConf11, this year's annual Debian Conference was held in Banja Luka, Republika Srpska, Bosnia and Herzegovina. Click below for a conference summary. "The Debian Conference was attended by over 400 contributors from over 70 countries ranging as far as New Zealand, Taiwan and Brazil. Beside the original scheduled 78 sessions, about 30 additional sessions where scheduled during the conference." DebConf12 will be held in Managua, Nicaragua.

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• DistroWatch Weekly, Issue 416 (August 1)
• openSUSE Weekly News 186 (July 30)
• Ubuntu Weekly Newsletter, Issue 226 (July 30)

Comments (none posted)

First Look at Jeoss, a Lean and Mean Ubuntu-Based Server Distribution (Linux.com)

Carla Schroder takes a look at Jeoss, a compact, install-everywhere Ubuntu-based server distribution. "Jeoss is maintained by Patrick Massotta, and is based on Ubuntu 8.04 LTS, the Hardy Heron. It is not virtualized, but rather trimmed down to the naked essentials for a lean, mean efficient server. It uses a modified Ubuntu linux-image-2.6.24-27-server kernel. It differs from the Ubuntu server kernel in three significant ways: it is low-latency, it includes i586 instructions, and it does not require a PAE-enabled CPU. (PAE, Physical Address Extension, is a slick hack for 32-bit CPUs to access more than 4 GB RAM.) So Jeoss can be used with lower-power and non-PAE CPUs like Pentiums and AMD Geode processors, which are popular on single-board computers."

Comments (none posted)

Page editor: Rebecca Sobol

Development

The road to Enlightenment

The Enlightenment window manager, first released in 1997, is finally reaching its long-awaited 0.17 release. Work on this release began almost ten years ago, and involved a huge rewrite (or three) of the window manager, plus the creation of several libraries, the core of which are called Enlightenment Foundation Libraries, or EFL.

Enlightenment is well-known in the Linux community mainly for being extremely lightweight and for displaying impressive graphical effects even in its earlier incarnations from more than a decade ago. It originally started as an attempt to provide a more graphically stimulating environment for Unix systems and was even used as the window manager for GNOME before Metacity. Enlightenment 0.16 was released in 2000 and is still actively maintained, but the majority of the developer community switched focus to the new version and the development of the EFL. Due to the attention to performance by the community, led by Carsten Haitzler, the EFL have found a niche in embedded systems.

Enlightenment 0.17, or E17, has been stable for quite some time and is used by many, but since it was never released officially, support for it on distributions is quite limited so far. The Enlightenment community has lately been engaged in releasing version 1.0 of the EFL, which happened in January. Currently, work is ongoing on two fronts:

• Readying E17 for a release, which involves closing a few remaining issues

• Preparing EFL 1.1, which will fix several bugs and have much improved documentation.

What are the foundation libraries?

During the course of creating E17, a bunch of libraries were devised (and quite a few were later deprecated) to ease the development of the new window manager. Given the attention to performance during the development of the libraries, they have now become a major alternative for use in programming embedded systems.

The first library is Eina, which is a library used by most of the others that implements basic types, provides memory pools for allocation with several different policies, and provides other common services. It has optimizations such as "stringshares" which allocate memory only once for each string, thus avoiding increased memory usage due to string duplication in the various layers of the program.

Eet is a very simple library for data serialization, encoding, and access. It's used as a really fast database in several applications. Ecore, as it name implies, provides the main infrastructure for applications such as the main loop, allowing EFL programs to deal with both user and system generated events. It also offers other low-level services, such as threads and network connections.

Evas, a stateful canvas for drawing is one of the primary reasons why the EFL are so fast, by employing optimizations first introduced in its predecessors (imlib1 and imlib2). In addition, Evas has faster rescaled JPEG loading by using image compression techniques such as macroblocks and all the SSE and NEON multimedia extension optimizations you could ask for.

And finally Edje, a powerful theming library that allows for rich graphical UIs, which was one of the inspirations for Qt's QML (itself based on the previous QEdje). It provides a secure and sandboxed UI, and also has support for scripting in Embryo (based on the language PAWN) and Lua.

There are quite a few more libraries available on the Enlightenment SVN, but the ones above are the most important. There is also work being done on a new widget toolkit called Elementary, which uses all the other libraries as its base and will be the basis of future work after the release of E17. Elementary is already quite mature and should see its own 1.0 release soon.

The libraries in question are heavily optimized, and provide support for several different architectures and graphical backends such as X11, OpenGL, SDL, DirectFB, and Windows XP and CE, among others.

Where the EFL are being used

Currently there are various and diverse projects using the EFL as base. From the recently announced Ordissimo, which is a Linux distribution targeting the elderly population, to home automation systems like the one created by Calaos and featured in a video. Millions of set top boxes were deployed by Free.fr using EFL, and recently a big player in the appliances industry has started selling a refrigerator with a touchscreen interface developed using the new Elementary toolkit.

Canonical sponsored an EFL version of its netbook-launcher targeting ARM devices that have no 3D acceleration (the official version being based on Clutter and depending on OpenGL). It had as many graphical effects, if not more, than did its counterpart and also showcased how powerful Edje is by allowing completely different UIs to be displayed by only changing the theme. The most recent big player to start supporting the EFL is Samsung, which is not only providing patches to the libraries and creating new widgets, but has also engaged with ProFUSION in improving the documentation available for programmers and in developing a new WebKit port to the Enlightenment libraries. Those interested in learning more about this new port should check out the information available on the official wiki. The LiMo foundation, an industry consortium dedicated to creating a Linux-based operating system for mobile devices, has also announced that the EFL will be an integral component of the LiMo 4 platform.

Status of E17 and helping out

E17 is currently quite stable, and a few distributions are already shipping packages, albeit not always as up-to-date as one could hope. This is one of the things that is planned for the impending release: having better engagement with distributions so that E17 becomes available to a much larger audience.

There are several different modules and plugins available, including support for making it a tiling window manager and also for compositing (both in software and OpenGL). Due to the focus on optimizations mentioned before, one can pretty much bet that E17 will work really well even on older hardware (or newer hardware with CPU limitations). Moreover there is also support to spread rendering to multiple cores, so that applications written using the EFL (E17 included) can scale easily from running on a 200Mhz MIPS to an eight-core Xeon running at 2.3Ghz per core.

Regarding applications, work has begun on a few basic applications: Eve (a web browser based on WebKit-EFL), Enjoy (a music player), Envision (document reader), and EPhoto (photo manager). They are at differing levels of completion, but right now they are mostly being developed as showcases of how powerful the EFL are. Each application will have UIs targeting different devices (such as netbooks, desktops, tablets, and TVs) that completely change how the user interacts with them, but without changing the underlying code. Code for all of these applications is also on the SVN repository, but please be aware that they are mostly still fairly basic (and need more contributors).

Those who wish to test it can find instructions here, alongside information for specific distributions.

There are plenty of things to do for those interested in joining the Enlightenment community, but one of the things needed the most right now is for people to write documentation and to do testing and bug reporting. There is lots of low-hanging fruit available, so those interested in getting started should join the IRC channels #e (for users) or #edevelop (for developers) on Freenode and ask for more information. People who can help contributing with packaging for any distribution are also very welcome. There are also a couple of mailing lists used for discussion and there is more information on the official site.

Overall, Enlightenment has come a long way in recent years to become a stable and viable alternative to the more well-known libraries such as Qt and GTK+, especially for the embedded space. Documentation, which was severely lacking before, is being improved and should now be enough for newcomers to get going. With the release of EFL 1.0, one can only hope that 2011 might still see an E17 release being announced.

Comments (8 posted)

Brief items

Development quotes of the week

-- Andy Wingo looks at Google's V8 JavaScript engine (part of a series of articles)

-- Part of a panel description for a session at the upcoming Desktop Summit featuring Mark Shuttleworth, Michael Meeks, Bradley Kuhn, and moderated by Karen Sandler

Comments (none posted)

Collectfs: a trash collecting userspace file system for Linux

Collectfs, which is a filesystem in user space (FUSE) for Linux that seamlessly protects files from being clobbered accidentally, has been released. "It is not intended as a replacement for revision control or backups. The intention is to protect you during the between-times, when you're not covered by these other tools. [...] Any file that is overwritten by remove (unlink), move, link, symlink, or open-truncate is relocated to a trash directory (mount-point/.trash/). Removed files are date-time stamped so that edit history is maintained (a version number is appended if the same file is collected more than once in the same second)."

Comments (none posted)

digiKam 2.0.0 has been released

The digiKam photo management tool has released version 2.0.0. "The road to version 2.0.0 took more than a year of heavy development. The team proudly announces the first release of the new generation of digiKam. This version features long awaited face detection and recognition, image versioning support, XMP metadata sidecar files support, big improvements in tagging and marking photos, reversed geo-tagging and many other improvements, including a total of 236 fixed bugs." LWN recently reviewed the release.

Comments (none posted)

Genius 1.0.13 release

The Genius calculator application has made its "yearly tradition" release. "In any case, Genius is one of the oldest GNOME projects, it has been the original GNOME calculator before I got wild ideas about it doing absolutely everything. It is programmable, has a powerful language and handles many fun features including support for matrices, rational numbers, and nice 2D and 3D plotting."

Full Story (comments: none)

KDE 4.7 is released

KDE 4.7 has been released with new features like improvements to Phonon, social semantic desktop components, a new integrated instant messaging application, and more. "In addition to the many new features described in the detailed release announcements, KDE contributors have closed over 12,000 bug reports since the last major releases of KDE software in January 2011. As a result, KDE software is more stable than ever before."

Full Story (comments: none)

Google open sources LevelDB

Google has announced the release of LevelDB, "a fast persistent key-value store", under a BSD license. "LevelDB is a C++ library that can be used in many contexts. For example, LevelDB may be used by a web browser to store a cache of recently accessed web pages, or by an operating system to store the list of installed packages and package dependencies, or by an application to store user preference settings. We designed LevelDB to also be useful as a building block for higher-level storage systems. Upcoming versions of the Chrome browser include an implementation of the IndexedDB HTML5 API that is built on top of LevelDB."

Comments (9 posted)

LibreOffice 3.4.2 for enterprise users

The Document Foundation (TDF) has announced the release of LibreOffice 3.4.2. This version has been declared enterprise ready, unlike previous versions that were aimed at testers and early adopters. "LibreOffice 3.4.2 fixes the majority of the most-important bugs identified by users in the previous version, and can be deployed for production needs by most enterprises."

Full Story (comments: 23)

Newsletters and articles

Development newsletters from the past week

• Caml Weekly News (August 2)
• PostgreSQL Weekly News (July 31)
• Tahoe-LAFS Weekly News (July 31)

Comments (none posted)

CouchDB and SQLite creators introduce UnQL, a NoSQL query language (The H)

The H reports on the introduction of UnQL (UNstructured Query Language, pronounced "Uncle"), which is meant to be the SQL of the NoSQL movement. "CouchDB creator Damien Katz and SQLite creator Richard Hipp have been working on UnQL to create a higher level query language for NoSQL document databases. Katz says the specification 'stems from our belief that a common query language is necessary to drive NoSQL adoption in the same way SQL drove adoption in the relational database market'. Hipp notes that 'Unql builds upon our experience with SQL, supplementing that language with syntax and concepts appropriate for the unstructured, self-describing data formats of post-modern applications'."

Comments (36 posted)

Jokosher: A Completely Kosher Audio Multitool (LinuxInsider)

LinuxInsider reviews Jokosher, a multitrack audio editor. "Jokosher offers a solid list of features. Chief among its best assets is the interface. For example, the design has musicians in mind, so the terminology in the menus has a familiar ring. Of course, non-musicians might have to come up to speed with musicology phrases to reach a comfort zone with Jokosher."

Comments (6 posted)

Kde 4.7 Released And In The Wild (Linux Journal)

Michael Reed looks at KDE SC 4.7 on Linux Journal. "If you are a 4.6 user who has just upgraded, don't expect to be aware of major changes the first time you reboot. Some of the core applications have been updated, but most of the work has gone into improving the underlying frameworks. The applications themselves have been shifted to a greater reliance on Akonadi, the PIM storage framework and NEPOMUK, the semantic information database."

Comments (63 posted)

Page editor: Jonathan Corbet

Announcements

Articles of interest

MPEG LA: 12 companies own patents essential to Google's VP8 codec (ars technica)

Ars technica reports on "progress" in forming a WebM/VP8 patent pool. Twelve companies have submitted an unknown number of patents that MPEG LA (the MPEG licensing authority) has deemed essential to implementing Google's currently royalty-free video codec. "With so many companies submitting patents of their own to MPEG LA, formation of a patent pool becomes much more likely, and the chance that WebM will retain its royalty-free position shrinks accordingly. That isn't yet a foregone conclusion, however—though the companies have come forward and made initial submissions, they may decide that it's not worth forming a patent pool for some reason. Even if they do, the decision to enforce their patents against VP8 users is separate; MPEG LA doesn't enforce patents (it has sued companies, but for breach of contract, not patent infringement), and so it would be up to the individual members of the pool to take legal action against infringers."

Comments (29 posted)

Bob Young on Lulu and collaborative innovation (Opensource.com)

Opensource.com has an interview with Red Hat co-founder Bob Young (now founder and CEO of Lulu) in two parts (1, 2). In it, he discusses collaboration and how it is actually a natural part of our society; the same is true in many industries. From part 1: "The reason you can walk down the street and not be mugged is because we default to thinking if we were robbed, the police would put the robber in jail. But the reason we can walk down the street and not worry for our safety is not that there's a policeman on every corner--there isn't. It's because it's in my interest to enable you to walk down the street safely, and it's in your interest to make sure I can walk down the street safely. So if somebody tries to mug you, the real problem is the other people on the street. You have terrible stories where people stood around watching, but those are the exceptions."

Comments (12 posted)

New Books

Advanced Software Testing, Vol. 3--New from Rocky Nook

Rocky Nook has released "Advanced Software Testing -- Vol. 3" by Rex Black and Jamie L Mitchell.

Full Story (comments: none)

Education and Certification

LPI Joins ITCC "TechCertRegistry"

The Linux Professional Institute (LPI) has announced that they are joining the Information Technology Certification Council's "TechCertRegistry". "This service is a first-of-its-kind Web-based credential registry for the IT industry and allows IT candidates, certification program managers and employers, to aggregate, validate and verify IT credentials from participating members."

Full Story (comments: none)

Upcoming Events

PyCon DE 2011 - Registration is Open

PyCon DE takes place October 4-9, 2011 in Leipzig, Germany. Registration is now open.

Full Story (comments: none)

PyCon 2012

The PyCon staff and Python Software Foundation have announced the launch of PyCon 2012. The conference will take place in Santa Clara, California, from March 7-15, 2012. There will be two days of tutorials (March 7-8), followed by three days of talks (March 9-11) and will wrap up with 4 days of sprints (March 12-15).

Full Story (comments: none)

Events: August 11, 2011 to October 10, 2011

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
August 6 August 12	Desktop Summit	Berlin, Germany
August 10 August 12	USENIX Security ’11: 20th USENIX Security Symposium	San Francisco, CA, USA
August 10 August 14	Chaos Communication Camp 2011	Finowfurt, Germany
August 13 August 14	OggCamp 11	Farnham, UK
August 15 August 16	KVM Forum 2011	Vancouver, BC, Canada
August 15 August 17	YAPC::Europe 2011 “Modern Perl”	Riga, Latvia
August 17 August 19	LinuxCon North America 2011	Vancouver, Canada
August 20 August 21	PyCon Australia	Sydney, Australia
August 20 August 21	Conference for Open Source Coders, Users and Promoters	Tapei, Taiwan
August 22 August 26	8th Netfilter Workshop	Freiburg, Germany
August 23	Government Open Source Conference	Washington, DC, USA
August 25 August 28	EuroSciPy	Paris, France
August 25 August 28	GNU Hackers Meeting	Paris, France
August 26	Dynamic Language Conference 2011	Edinburgh, United-Kingdom
August 27	PyCon Japan 2011	Tokyo, Japan
August 27	SC2011 - Software Developers Haven	Ottawa, ON, Canada
August 27 August 28	Kiwi PyCon 2011	Wellington, New Zealand
August 30 September 1	Military Open Source Software (MIL-OSS) WG3 Conference	Atlanta, GA, USA
September 6 September 8	Conference on Domain-Specific Languages	Bordeaux, France
September 7 September 9	Linux Plumbers' Conference	Santa Rosa, CA, USA
September 8	Linux Security Summit 2011	Santa Rosa, CA, USA
September 8 September 9	Italian Perl Workshop 2011	Turin, Italy
September 8 September 9	Lua Workshop 2011	Frick, Switzerland
September 9 September 11	State of the Map 2011	Denver, Colorado, USA
September 9 September 11	Ohio LinuxFest 2011	Columbus, OH, USA
September 10 September 11	PyTexas 2011	College Station, Texas, USA
September 10 September 11	SugarCamp Paris 2011 - "Fix Sugar Documentation!"	Paris, France
September 11 September 14	openSUSE Conference	Nuremberg, Germany
September 12 September 14	X.Org Developers' Conference	Chicago, Illinois, USA
September 14 September 16	Postgres Open	Chicago, IL, USA
September 14 September 16	GNU Radio Conference 2011	Philadelphia, PA, USA
September 15	Open Hardware Summit	New York, NY, USA
September 16	LLVM European User Group Meeting	London, United Kingdom
September 16 September 18	Creative Commons Global Summit 2011	Warsaw, Poland
September 16 September 18	Pycon India 2011	Pune, India
September 18 September 20	Strange Loop	St. Louis, MO, USA
September 19 September 22	BruCON 2011	Brussels, Belgium
September 22 September 25	Pycon Poland 2011	Kielce, Poland
September 23 September 24	Open Source Developers Conference France 2011	Paris, France
September 23 September 24	PyCon Argentina 2011	Buenos Aires, Argentina
September 24 September 25	PyCon UK 2011	Coventry, UK
September 27 September 29	Nagios World Conference North America 2011	Saint Paul, MN, USA
September 27 September 30	PostgreSQL Conference West	San Jose, CA, USA
September 29 October 1	Python Brasil [7]	São Paulo, Brazil
September 30 October 3	Fedora Users and Developers Conference: Milan 2011	Milan, Italy
October 1 October 2	WineConf 2011	Minneapolis, MN, USA
October 1 October 2	Big Android BBQ	Austin, TX, USA
October 3 October 5	OpenStack "Essex" Design Summit	Boston, MA, USA
October 4 October 9	PyCon DE	Leipzig, Germany
October 6 October 9	EuroBSDCon 2011	, Netherlands
October 7 October 9	Linux Autumn 2011	Kielce, Poland
October 7 October 10	Open Source Week 2011	Malang, Indonesia
October 8	PHP North West Conference	Manchester, UK
October 8	FLOSSUK / UKUUG's 2011 Unconference	Manchester, UK
October 8 October 9	PyCon Ireland 2011	Dublin, Ireland
October 8 October 9	Pittsburgh Perl Workshop 2011	Pittsburgh, PA, USA
October 8 October 10	GNOME "Boston" Fall Summit 2011	Montreal, QC, Canada
October 9 October 11	Android Open	San Francisco, CA, USA

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol