LWN.net Weekly Edition for December 8, 2011

Channels for community communications

In the dim and distant past, even before the 0.01 Linux kernel was posted, developers communicated with each other via email, often sent over the UUCP-based USENET network. It had its challenges; life was better for those who could arrange a link to a machine owned by a company that paid little attention to its phone bills, and the need for senders to manually determine the routing of their message made communications uncertain at times. Your editor received a full netnews feed over a 2400-baud modem, and everybody complained that there was far too much traffic to keep up with. But a lot of collaboration was done over this medium, and a lot of early free software was developed and distributed that way.

Happily, the days of broadcasting software releases over modem-based connections are long behind us. But the free software community still relies on electronic mail for the bulk of its communications, despite the fact that a great many alternatives have become available. Email has a lot of advantages: we have developed a lot of tools for working with it (complaints about the quality of free email clients notwithstanding), it works well for multi-sided communications, it is distributed and can be dealt with offline, it is easily searched, and it is readily managed and archived on non-proprietary sites. It is not surprising that email has been the tool of choice for so long, especially when one looks at some of the alternatives.

Recently, your editor received a grumpy communication (via email, naturally) from a prominent community member who was unhappy about a recent quotes of the week article that included a quote posted on the Google+ site. Posting to links on Google+ legitimizes it as a valid place for community discussions, your editor was told; it makes it more likely that more conversations - perhaps more useful ones than the one chosen by LWN - will move to that site. And this was seen as a bad thing.

There are certainly reasons to be concerned about engaging in community discussions on a site like Google+. The site is corporate-owned, making it subject to all of the usual types of pathological corporate behavior, in the present or in the future. Its owner will make use of the content of messages and the whole "circles" structure for targeted advertising and any number of other purposes. Governments seem to feel free to help themselves to the information held by these sites at will. There are no tools to help readers deal with messages on the site, no RSS feeds, and no ability to read messages offline.

But the biggest complaint of all seemed to be that when sites like Google+ (or even LWN) host conversations of interest, they spread those conversations out and make it harder to keep up with what is being said. The result is a more fragmented community with groups that no longer communicate with each other and with people who are left in the dark about important discussions and decisions. It would be better, this correspondent said, to improve the mailing list experience if need be and keep the useful discussions there.

Your editor understands these concerns and is all for improving the email experience whenever possible. But he is less clear on whether it has ever been possible to concentrate community communications into a single channel, or whether it is a desirable goal.

Consider, for example, the venerable IRC channel. IRC is clearly an important part of how members of our community work together. But IRC, too, works poorly offline; it also often - and often by design - lacks a permanent record. Extracting information from IRC logs when they are available is a tedious exercise at best. IRC involvement can also be incompatible with actually getting work done, so some people find that they need to avoid it. IRC, in other words, is a channel that is available to parts of the community part of the time, but it has its value anyway.

An even more exclusive resource is the famed conference hallway track, where a lot of real work gets done. Your editor's job would be much facilitated by readily-accessible logs of conference hallway (and pub) conversations; the "quotes of the week" section, in particular, would benefit mightily. But no such logs exist, and that is certainly for the best. There is also a lot of valuable conversation to be found in vast numbers of bug trackers, patch trackers, weblogs, retail site comment areas, and, sadly, forum sites.

In other words, community discussions are already widely fragmented. The key is communicating important decisions back to the wider group and making people aware of useful discussions. Often that can be achieved with an email to a prominent development list; one might also hope that sites like LWN can help in this regard. Indeed, one might argue that pointing to something interesting on Google+, rather than encouraging fragmentation of our community, actually serves to help pull it back together.

That said, the other concerns about a platform like Google+ remain. If anything important was ever posted on Google's "Wave" or "Knol" platforms, it will disappear early next year. The same fate could certainly befall discussions on Google+. That platform also excludes anybody who is unwilling to provide yet more information to Google about his or her activities and connections on the net. Google+ might serve as a sort of stand-in for a conference hotel bar, but it is not suitable as a piece of significant community infrastructure. One could name any of a number of other proprietary platforms that are equally unsuited, if not more so.

It seems that there is a place in our lives for a service with which we can post updates on our development projects, air travel horror stories, silly cat pictures, desktop environment flames, lazyweb requests, cooking experiments, new techno-toys, and musings on how Cher songs relate to the harsh life of NMI watchdog handlers. What we seem to be missing is the ability to create such a platform for ourselves and, crucially, to get a critical mass of people to give it a try. Even back in the USENET days, we demonstrated that we can create distributed communications services when we set our minds to it. Someday we'll solve that problem again for the contemporary net; until then, we're limited to the channels that exist, some of which are not all we would like them to be.

Comments (33 posted)

Google Authenticator for multi-factor authentication

The security-conscious will tell you that a multi-factor authentication scheme involves requiring items from two or more of the categories "things you know," "things you have," and "things you are." Passwords and passphrases both fall under the "things you know" umbrella, and while there are commercially viable options for the latter two categories — security dongles and biometric fingerprint scanners, for example — neither have taken off with the general public. Partly that is a cost issue, to be sure, but the complexity of public-key infrastructure (PKI) smart cards does not help, either.

Late in 2010, however, Google unveiled an open source project called Google Authenticator that allowed the common smartphone to serve as the "thing you have" factor. The search giant subsequently rolled out support for the scheme for its web applications, but its standards-based functionality and Pluggable Authentication Module (PAM) support have brought it success in a variety of third-party systems as well.

Client

On the smartphone side, the Google Authenticator project provides application software for Android devices, Apple iPhones, and Blackberry handsets. Because these applications are software and generate authentication strings for the user to enter at login time, however, there is some question whether or not they truly "count" as a second factor. Traditionally, hardware authentication tokens must be physically connected to the computer to authenticate a user, though some one-time password (OTP) generators are standalone. Unlike the Android app, however, those devices are meant to make it difficult to extract the key without destroying them. Accessing the key from a phone, then running the app elsewhere (e.g. an Android emulator) would circumvent the "things you have" requirement.

The project's official tagline is "Two-step verification," a subtle acknowledgment of its distinction from a strict definition of multi-factor authentication. But to narrow that security risk, the system generates OTPs that are only valid for a short duration (generally no more than one minute). That does not stop an attacker who steals your device from getting a valid OTP from the application, but it makes it more difficult for an attacker to forge a login through eavesdropping. Specifically, it prevents replay attacks (such as wandering eyes behind you in line at the coffee shop jotting down the passcode) and makes interceptions (such as wandering eyes paired with extremely fast fingers) highly unlikely. Using an OTP will help against credential-stealing attacks, such as those used to compromise kernel.org.

Google Authenticator supports both the Hashed Message Authentication Code (HMAC)-based OTP (HOTP) and the Time-based OTP (TOTP) specifications, both of which were developed by the Initiative for Open Authentication (OATH). Both algorithms use a pseudo-random seed value or key that is known only to the server and the client; the seed is concatenated with a "moving factor" — the piece of the puzzle that makes the password one-time-use only — and the result hashed to create the expected passcode.

HOTP, the older of the two, uses an integer counter that increments for every new passcode. TOTP uses a timestamp as the moving factor instead, with provisions for the server to define whatever time window it wishes to accept as valid. That value enables humans to type in the passcode at a convenient speed, and also allows for clock drift between server and client. Although TOTP is regarded as more secure, HOTP is already finalized as RFC 4226, while TOTP was still at the draft stage when the Google Authenticator project was launched (it has since become RFC 6238).

The mobile clients provided by Google support storing multiple HOTP and TOTP credentials, and can be used to generate passwords for any compliant implementation of the algorithms. There are two mechanisms for entering the seed value associated with each saved account: devices with a camera can take a photograph of a key in an appropriately-formatted QR code, or the user can manually enter the key as a Base32 string.

Server

For the server side of the system, the project provides a command-line tool to provision new keys, and a PAM module — although "server" is not quite accurate; the module will work on traditional desktop Linux distributions as well. According to the user comments on the module's wiki page, Linux appears to be the only operating system supported out of the box; there is a patch reported to work on FreeBSD 8.2 and another for OpenSolaris, but no luck yet for Mac OS X or other PAM-supporting OSes.

The key provisioning tool is called google-authenticator. When run, it generates a new key (primed from /dev/urandom), along with a "verification code" and five "emergency scratch codes." The verification code is used by Google to associate an account with a cellular number via SMS message, and the scratch codes are eight-digit numeric strings designed for use in account recovery — neither is part of the core algorithm. The google-authenticator tool writes the key and the emergency scratch codes into the ~/.google_authenticator file. If libqrencode is installed, it will also generate a QR code formatted for consumption by the mobile applications — the format includes metadata for use by the application, including the hostname, user name, and key type ("totp" or "hotp"). It also writes a URL to stdout containing an HTTPS link to Google's online QR code generator, with the same information as the payload (although, obviously, using the online generator does send your secret key and user/host data to a Google web server...).

The PAM module is named pam_google_authenticator.so, and can be enabled by editing the PAM configuration file for the appropriate service (typically located in /etc/pam.d/). Simply adding

    auth required pam_google_authenticator.so

to, for example, /etc/pam.d/gdm will activate the Google Authenticator module for all accounts; it will request the HOTP or TOTP passcode after asking for the username and account password. Because the keys are stored in $HOME by default, workarounds are required to use the module with encrypted home directories. The project README recommends rearranging the PAM configuration to decrypt home directories earlier, or storing the keys in a different location (which must then be passed to the module as an argument in the configuration file line).

There is a patch available that will require the Google Authenticator module only if a .google_authenticator file is present. This approach could be used to partition accounts into "high-security" and "low-security" groups, but it could also come in handy if something goes haywire during setup. TOTP can handle a certain degree of clock shift, but the last thing you want to do is activate it for all of your login accounts before thoroughly testing it. To that end, the project also provides a demo utility for testing TOTP passcodes, and an interactive online debugger to ferret out clock and interval problems.

Authenticating in practice

Of course, the GDM example above is not Google Authenticator's intended use-case. Securing SSH and other remote-access methods makes more sense for those whose machines are safely tucked away in an office or closet; any attacker with physical access to a local login prompt has many other attractive opportunities. Any PAM-aware application can take advantage of the module, however — including Apache, which opens the door for many multi-user, web-based applications.

Indeed, there are already numerous third-party projects to integrate Google Authenticator into WordPress, Drupal, and other Content Management Systems (CMS), as well as instructions to use the PAM module for security-conscious packages like OpenSSH and OpenVPN.

Perhaps the more pressing question is whether or not it is wise to introduce a dependence on an application available only for iPhone, Android, and Blackberry devices. As popular as these platforms are, they still leave some users out in the cold, and there may be times when you need to log into a machine and your phone is unavailable for any number of reasons — from job site security, to accidental loss, to simple battery failure. USB security fobs do not suffer from the same limitations.

To that end, however, Google Authenticator benefits from the existence of other HOTP and TOTP implementations. While there are not many to choose from, there are a few: OATH Toolkit, multiOTP, and LinOTP are all free software. I tested Google Authenticator TOTP codes against OATH Toolkit's command-line oathtool. You can perform a passcode-generation hash by running:

    oathtool --totp --now="the_current_time" your_secret_key

The passcodes matched, once I figured out how to correctly convert the Base32 encoding produced by Google Authenticator into the hexadecimal required by oathtool — namely, that the Base32 encoding scheme defined by RFC 4648 is not the same as base-32 mathematical notation (because the encoding avoids easy-to-confuse characters like I and O).

Thus, there are options for users who cannot or will not carry a device with an official Google Authenticator client application. The availability of compatible command-line options is not a completely secure solution for the lost-phone emergency scenario, though, because it requires you to store the secret key in a third (or fourth, or fifth, etc.) location.

Nevertheless, as the CMS extensions and wrappers for various languages demonstrate, the Google Authenticator project has taken off to a degree that other OTP utilities have not (and there are several using other algorithms, including OTPW and Mobile OTP, both of which have PAM support). That makes the project a net gain for system security, perfect multi-factor authentication or not. The code is packaged for Debian and Ubuntu, and there are contributed builds available for OpenSUSE and Fedora, although there does not appear to be an official package for either of the latter distributions. If the distributions come on-board in force, perhaps OTP adoption will one day be commonplace.

Comments (53 posted)

2011 Linux and free software timeline - Q2

Here is LWN's fourteenth annual timeline of significant events in the Linux and free software world for the year. We will be breaking the timeline up into quarters, and this is our report on April-June 2011. Over the next few weeks, we will be putting out timelines of the other quarters of the year.

• January-March 2011

---

This is version 0.8 of the 2011 timeline. There are almost certainly some errors or omissions; if you find any, please send them to timeline@lwn.net.

LWN subscribers have paid for the development of this timeline, along with previous timelines and the weekly editions. If you like what you see here, or elsewhere on the site, please consider subscribing to LWN.

For those with a nostalgic bent, our timeline index page has links to the previous thirteen timelines and some other retrospective articles going all the way back to 1998.

April

Texas Linux Fest is held April 2 in Austin (LWN articles: HeliOS and The mobile ecosystem).

Mozilla reabsorbs Mozilla Messaging, which makes the Thunderbird email client, after spinning it off in 2007 (announcement).

Camp KDE is held in San Francisco April 4-5 (LWN coverage: Qt open governance, Using Slackware to investigate KDE 4, and Geolocation).

The Linux Filesystem, Storage, and Memory Management Summit is held April 4-5 in San Francisco (LWN coverage: Future storage technologies and Linux plenary session, the rest of Day 1, and Day 2).

The Yocto Project releases version 1.0 of the embedded distribution builder (announcement, release notes).

GNOME 3.0 is released (press release, release announcement, Grumpy Editor review).

LLVM 2.9 is released as the last in the 2.x series from the compiler and toolchain project. (announcement, release notes).

The Linux Foundation Collaboration Summit is held April 6-8 in San Francisco (LWN coverage: Linux penetration, Yocto, and hardware success stories, Kernel panel, Project Harmony, and Building the kernel with Clang).

Pamela Jones announces the end of Groklaw, which is supposed to happen May 16, though, instead of shutting down, Mark Webbink takes over (LWN blurb).

CyanogenMod 7.0 is released bringing Android 2.3.3 ("Gingerbread") to the alternative firmware for mobile devices (announcement, LWN Grumpy Editor review).

The Embedded Linux Conference is held April 10-12 in the second week of multiple conferences held at the Hotel Kabuki in San Francisco (LWN coverage: Linaro power management work and A PREEMPT_RT roadmap).

Stefano Zacchiroli is re-elected as Debian project leader in an uncontested election (announcement).

The MySQL Conference & Expo is held April 11-14 in Santa Clara, California, and covers much more than just MySQL (LWN article).

The first Android Builders Summit is held April 13-14 in San Francisco (LWN coverage: Android beyond the phone and The guts of Android

FVWM releases version 2.6 after nearly 10 years of development on the window manager (announcement, LWN review).

Oracle announces that OpenOffice.org will become a "community-based project", which is the start of the move to Apache (press release).

The US Department of Justice announces a change to Novell's patent sale to CPTN Holdings (owned by Microsoft, Oracle, Apple, and EMC), which is aimed at allowing free software implementations of things covered by those patents (press release).

Google falls prey to the patent troll Bedrock Technologies for the Linux routing table implementation based on a 1997 patent on what is essentially open hashing (LWN article).

LWN offers a "maniacal supporter" subscription level for those who would like to pay $50/month—a surprising number of folks signed up at that level, but we can always use more ... (announcement).

WebM announces a cross-license initiative that is meant to operate as a kind of patent pool for users and developers of the video format (announcement).

Slackware 13.37 is released (announcement, LWN review).

Ubuntu 11.04 ("Natty Narwhal") is released with Unity as its default shell (announcement, press release).

LinuxFest Northwest is held in Bellingham, Washington on April 30 and May 1 (LWN coverage: Getting HTTPS everywhere).

May

OpenBSD 4.9 is released (announcement, list of changes).

Mozilla refuses a request from the US Department of Homeland Security to remove an add-on, which routes around domain seizures made by the agency, because there is no court order to do so (blog posting).

The Ubuntu Developer Summit is held in Budapest, Hungary May 9-13 along with the Linaro Development Summit (LWN coverage: UDS keynotes, Mark Shuttleworth interview, ARM platform consolidation (from LDS), UDS security discussions, LTTng in the Ubuntu kernel, and Linaro keynotes).

The Netherlands Unix Users Group (NLUUG) Spring conference is held May 12 in Ede (LWN article: Open telephony).

The Libre Graphics meeting is held May 10-13 in Montreal, Canada (LWN articles: Krita talks and Adaptable GIMP).

The LyX document processor releases version 2.0 (announcement, LWN pre-review).

Perl 5.14.0 is released (announcement, perldelta details).

Linux 2.6.39 is released (announcement, KernelNewbies summary, development statistics).

Miguel de Icaza announces the formation of Xamarin to create Mono-based products for Android and iOS after the Mono team is laid off by Attachmate as part of the Novell acquisition (announcement).

Former Red Hat general counsel Mark Webbink officially takes over Groklaw (announcement, parting interview with Pamela Jones at The H).

PGCon, the PostgreSQL conference, is held in Ottawa, Canada May 19-20 (LWN article).

Linus Torvalds starts considering a kernel version number change, which eventually results in 3.0 instead of 2.6.40 (lkml posting).

Fedora 15 is released (announcement, feature list).

MeeGo 1.2 is released in what turns out to be the last major MeeGo release (announcement).

The MeeGo conference is held in San Francisco May 23-25, presumably the last of those as well (LWN coverage: MeeGo's openness and transparency and MeeGo 1.2 on the N900).

Linux Mint 11 is released (announcement, LWN review).

openSUSE renames the openSUSE Build Service to the Open Build Service, changing the focus a bit while preserving the OBS acronym (announcement).

June

LinuxCon Japan is held in Yokohama June 1-3 (LWN coverage: A conversation with Linus and Android, forking, and control).

LibreOffice 3.4.0 is released (announcement, release notes, Michael Meeks's look at the cleanups that went into 3.4.0).

Mageia 1, the community-driven fork of Mandriva, is released (announcement, release notes).

Ubuntu's first long-term support release, 6.06 ("Dapper Drake") reaches its end of life (announcement).

Oracle starts the process of donating OpenOffice.org to the Apache Software Foundation (LWN blurb and article).

The venerable Kermit serial communications program is finally released as free software under a BSD license (LWN article).

Karen Sandler is named as the new GNOME Foundation executive director, succeeding Stormy Peters (announcement, LWN interview with Sandler).

Mozilla releases Firefox 5 (announcement).

AVM vs. Cybits case heard in Berlin, Germany, which is an important test of the GPL for embedded devices running Linux. A decision will have to wait until November (Free Software Foundation Europe press release, Harald Welte's blog post).

Nokia releases the MeeGo-ish N9 handset (LWN blurb and article).

GNU Awk (Gawk) 4.0 is released (announcement, LWN review).

Comments (none posted)

Page editor: Jonathan Corbet

Security

Loading signed kernel modules

Inserting loadable modules into a running kernel is clearly a convenient feature. It allows distributions to have relatively small kernels while still supporting a wide variety of hardware and use-cases, but it can also allow unwanted modules to be loaded in a way that may not be all that easy to detect. Those modules could simply be binary-only drivers that the distribution or service provides doesn't want to support—or they could be some kind of malware. A recently posted patch set will help to avoid those problems by giving the option of building a kernel that will only allow modules that have been cryptographically signed to be loaded, or to simply detect the presence of unsigned modules.

David Howells posted the patches, which are based on code that has been running in Fedora and RHEL kernels for years. Therefore, it should have been "thoroughly tested" Howells said in the final patch in the series (which also contains the module-signing.txt kernel Documentation file). The patch allows for signing modules using the RSA signature algorithm with any of the SHA family of hash algorithms.

The basic idea is that public keys can be built into the kernel and, if CONFIG_MODULE_SIG is enabled, used to check the validity of modules before they are loaded. If CONFIG_MODULE_SIG_FORCE is enabled at compile time (or enforcemodulesig=1 is passed on the kernel command line), only those modules that can be verified with one of the public keys built into the kernel will be loaded. If the "force" option is not used, unsigned modules will still be loaded. In either case, modules with corrupt or incorrect signatures, or those that are signed with a key that is not on the keyring, will be rejected.

In order to make that work, there needs to be a way to build signed modules. That is done by creating public and private keys in the top-level kernel directory (in kernel.pub and kernel.sec by default). The public key will be processed with the kernel's bin2c utility and written as crypto/signature/key.h. The public and secret key will then be used with GNU Privacy Guard (GPG) to sign the modules automatically as they are built. There are several options that can be passed on the make command line to govern the location of the key files as well as options to pass to gpg.

In addition, the modules can be stripped for inclusion into initial ramdisk images and debuginfo can be included in a separate ELF section that is not included in the signature calculation. That means that all of the variants of a particular module can share a single signature that is stored in the module itself (in the .module_sig ELF section). In addition, the output of /proc/modules has been changed to add a "U" to unsigned modules so that they can be detected.

The patch also modifies the kernel's crypto subsystem to allow for the new key type and to add an RSA signature verification algorithm. That requires using the multi-precision integer (MPI) library from GPG, which was reworked from the Red Hat version for kernel inclusion by Dmitry Kasatkin. That code is already in the security tree in order to support the Extended Verification Module (EVM) digital signature extension. It also requires a minimal parser for the OpenPGP format (which is the kind of keys and signatures that GPG generates).

There is also something of a chicken-and-egg problem. Many distributions have crypto and hash algorithms built as modules, but the RSA algorithm and whichever hash is being used to generate the signatures needs to be present or an enforcing kernel won't be able to load any modules at all. For that reason, the patches ensure that if module signatures are selected at compile time, the RSA and chosen hash algorithms are not built as modules themselves. Modules are then loaded in the usual way, with insmod, but the signature will be checked by suitably configured kernels.

There has been relatively little discussion or complaints about the patches in the three revisions that have been posted since Howells began the process in late November. H. Peter Anvin is concerned about adding a OpenPGP parser to the kernel, and Howells was quick to point out that the parser is just the minimal amount needed to pull keys and signatures out of OpenPGP-formatted data. Both Anvin and James Morris were unconvinced about the need for supporting the (now deprecated) DSA signature algorithm and Howells has pulled that code out in the most recent revision.

This is not the first time Howells has proposed these (or similar) changes as those efforts stretch back to (at least) 2004. He has now requested that the code be included into the Morris's security tree. If that happens, and no major complaints arise, we could potentially see signed module support in Linux 3.3.

While it is a useful feature, particularly for those trying to support Linux kernels without random drivers from who-knows-where, it only places another set of hurdles in front of malware authors. Since root privileges are required to load modules in the first place, a malware author will only need to find a way to insert code into the running kernel without using the module loading facility. Once upon a time, /dev/kmem could be used for that, but many distribution kernels don't support it any more. Prior to the advent of CONFIG_STRICT_DEVMEM, /dev/mem would have provided another way, but distributions are generally enabling that option as well. Exploiting some kind of kernel bug is the most probable route for these root-privileged attackers, but is certainly a more fragile approach than simply inserting a module.

Another potential use (or abuse depending on perspective) of the feature is for device makers or distributions to lock down their kernels. That would leave users who wish to add functionality (or remove anti-features) in the same place as the malware author: looking for a kernel bug to exploit. Of course, some users may just find a way to replace the kernel entirely in that scenario.

Comments (7 posted)

Brief items

Security quotes of the week

-- Moxie Marlinspike

-- Bruce Schneier

-- Dan Walsh debugs some problems that were causing RSA to recommend turning off SELinux

Comments (none posted)

Paper: Capability leaks in Android phones

Michael Grace, Yajin Zhou, Zhi Wang, and Xuxian Jiang have published a paper [PDF] describing research they have done into the Android security model as implemented on actual handsets. "In this paper, we analyze eight popular Android smartphones and discover that the stock phone images do not properly enforce the permission model. Several privileged permissions are unsafely exposed to other applications which do not need to request them for the actual use. To identify these leaked permissions or capabilities, we have developed a tool called Woodpecker. Our results with eight phone images show that among 13 privileged permissions examined so far, 11 were leaked, with individual phones leaking up to eight permissions. By exploiting them, an untrusted application can manage to wipe out the user data, send out SMS messages, or record user conversation on the affected phones - all without asking for any permission." The Google "Nexus" phones were the happy exception, with almost no leaks. (Seen on The H).

Comments (3 posted)

C|Net Download.Com accused of bundling Nmap with malware

Nmap hacker Fyodor has discovered that the download.com site is offering a version of Nmap (for Windows) with an installer that makes a number of unwelcome changes. "The way it works is that C|Net's download page offers what they claim to be Nmap's Windows installer. They even provide the correct file size for our official installer. But users actually get a Cnet-created trojan installer. That program does the dirty work before downloading and executing Nmap's real installer." One assumes they do similar things with other free programs. The number of LWN readers obtaining software from download.com is likely to be quite small, but it is still good to be aware of what's going on. This is why some projects adopt draconian trademark policies, unfortunately. (Thanks to Mattias Mattsson).

Full Story (comments: 38)

New vulnerabilities

clearsilver: arbitrary code execution

Package(s):	clearsilver	CVE #(s):	CVE-2011-4357
Created:	December 1, 2011	Updated:	December 23, 2011
Description:	From the Debian advisory: Leo Iannacone and Colin Watson discovered a format string vulnerability in the Python bindings for the Clearsilver HTML template system, which may lead to denial of service or the execution of arbitrary code.
Alerts:	Fedora FEDORA-2011-17040 2011-12-12 Fedora FEDORA-2011-17042 2011-12-12 Debian DSA-2355-1 2011-11-30

Comments (none posted)

colord: SQL injection

Package(s):	colord	CVE #(s):	CVE-2011-4349
Created:	December 5, 2011	Updated:	December 8, 2011
Description:	The colord daemon suffers from SQL injection vulnerabilities that could allow a local attacker to corrupt its databases or, possibly, databases belonging to other applications; see the Red Hat bugzilla entry for more information.
Alerts:	Ubuntu USN-1289-1 2011-12-07 Fedora FEDORA-2011-16451 2011-11-26 Fedora FEDORA-2011-16453 2011-11-26

Comments (none posted)

glibc: code execution

Package(s):	glibc	CVE #(s):	CVE-2009-5064
Created:	December 7, 2011	Updated:	December 7, 2011
Description:	From the Red Hat advisory: A flaw was found in the way the ldd utility identified dynamically linked libraries. If an attacker could trick a user into running ldd on a malicious binary, it could result in arbitrary code execution with the privileges of the user running ldd.
Alerts:	Scientific Linux SL-glib-20111206 2011-12-06 Red Hat RHSA-2011:1526-03 2011-12-06 Red Hat RHSA-2012:0126-01 2012-02-13 Red Hat RHSA-2012:0125-01 2012-02-13 CentOS CESA-2012:0125 2012-02-14 CentOS CESA-2012:0126 2012-02-14 Oracle ELSA-2012-0125 2012-02-14 Oracle ELSA-2012-0126 2012-02-14 Scientific Linux SL-glib-20120214 2012-02-14 Scientific Linux SL-glib-20120214 2012-02-14

Comments (none posted)

ipa: cross-site request forgery

Package(s):	ipa	CVE #(s):	CVE-2011-3636
Created:	December 7, 2011	Updated:	January 11, 2012
Description:	A CSRF vulnerability in ipa can allow an attacker to perform Red Hat identity management configuration changes with the privileges of a logged-in user.
Alerts:	Red Hat RHSA-2011:1533-04 2011-12-06 Scientific Linux SL-ipa-20111206 2011-12-06

Comments (none posted)

kernel: privilege escalation

Package(s):	linux kernel	CVE #(s):	CVE-2011-4330
Created:	December 5, 2011	Updated:	December 7, 2011
Description:	A bounds-checking error in the HFS filesystem can be exploited by a local user to crash the system or gain privileges.
Alerts:	Ubuntu USN-1312-1 2011-12-19 Ubuntu USN-1311-1 2011-12-19 Oracle ELSA-2011-2037 2011-12-15 SUSE SUSE-SU-2011:1319-2 2011-12-14 SUSE SUSE-SU-2011:1319-1 2011-12-13 Ubuntu USN-1304-1 2011-12-13 Ubuntu USN-1303-1 2011-12-13 Ubuntu USN-1302-1 2011-12-13 Ubuntu USN-1301-1 2011-12-13 Ubuntu USN-1300-1 2011-12-13 Ubuntu USN-1299-1 2011-12-13 SUSE SUSE-SA:2011:046 2011-12-13 Ubuntu USN-1293-1 2011-12-08 Ubuntu USN-1292-1 2011-12-08 Ubuntu USN-1291-1 2011-12-08 Ubuntu USN-1286-1 2011-12-03 Ubuntu USN-1322-1 2012-01-09 Red Hat RHSA-2012:0007-01 2012-01-10 CentOS CESA-2012:0007 2012-01-11 Scientific Linux SL-kern-20120112 2012-01-12 Oracle ELSA-2012-0007 2012-01-12 Ubuntu USN-1330-1 2012-01-13 Ubuntu USN-1340-1 2012-01-23 Red Hat RHSA-2012:0358-01 2012-03-06 Oracle ELSA-2012-0150 2012-03-07 SUSE SUSE-SU-2012:0736-1 2012-06-14 openSUSE openSUSE-SU-2012:0799-1 2012-06-28 openSUSE openSUSE-SU-2012:1439-1 2012-11-05

Comments (none posted)

kexec-tools: information disclosure

Package(s):	kexec-tools	CVE #(s):	CVE-2011-3588 CVE-2011-3589 CVE-2011-3590
Created:	December 7, 2011	Updated:	March 8, 2012
Description:	The kexec-tools package contains a number of information disclosure vulnerabilities exploitable by a local user.
Alerts:	Scientific Linux SL-kexe-20111206 2011-12-06 Red Hat RHSA-2011:1532-03 2011-12-06 Red Hat RHSA-2012:0152-03 2012-02-21 Scientific Linux SL-kexe-20120306 2012-03-06 Oracle ELSA-2012-0152 2012-03-07

Comments (none posted)

krb5: denial of service

Package(s):	krb5	CVE #(s):	CVE-2011-1530
Created:	December 7, 2011	Updated:	February 1, 2012
Description:	The kerberos key distribution center can be made to dereference a null pointer by an authenticated attacker, leading to a server crash.
Alerts:	CentOS CESA-2011:1790 2011-12-22 Oracle ELSA-2011-1790 2011-12-17 Fedora FEDORA-2011-16296 2011-11-23 Mandriva MDVSA-2011:184 2011-12-12 Scientific Linux SL-krb5-20111206 2011-12-06 Ubuntu USN-1290-1 2011-12-08 Red Hat RHSA-2011:1790-01 2011-12-06 Gentoo 201201-13 2012-01-23 Fedora FEDORA-2011-16284 2012-01-31

Comments (none posted)

libarchive: arbitrary code execution

Package(s):	libarchive	CVE #(s):	CVE-2011-1777 CVE-2011-1778
Created:	December 1, 2011	Updated:	February 21, 2012
Description:	From the Red Hat advisory: Two heap-based buffer overflow flaws were discovered in libarchive. If a user were tricked into expanding a specially-crafted ISO 9660 CD-ROM image or tar archive with an application using libarchive, it could cause the application to crash or, potentially, execute arbitrary code with the privileges of the user running the application. (CVE-2011-1777, CVE-2011-1778)
Alerts:	Ubuntu USN-1310-1 2011-12-19 Mandriva MDVSA-2011:191 2011-12-18 Mandriva MDVSA-2011:190 2011-12-18 Scientific Linux SL-liba-20111201 2011-12-01 Oracle ELSA-2011-1507 2011-12-01 Red Hat RHSA-2011:1507-01 2011-12-01 Debian DSA-2413-1 2012-02-20

Comments (none posted)

libxml2: code execution

Package(s):	libxml2	CVE #(s):	CVE-2011-0216
Created:	December 7, 2011	Updated:	September 27, 2012
Description:	The libxml2 library contains an off-by-one error leading to a buffer overflow vulnerability exploitable via a specially-crafted XML file.
Alerts:	Mandriva MDVSA-2011:188 2011-12-15 Scientific Linux SL-libx-20111206 2011-12-06 Red Hat RHSA-2011:1749-03 2011-12-06 Red Hat RHSA-2012:0016-01 2012-01-11 Red Hat RHSA-2012:0017-01 2012-01-11 CentOS CESA-2012:0016 2012-01-11 CentOS CESA-2012:0017 2012-01-11 Oracle ELSA-2012-0016 2012-01-12 Scientific Linux SL-libx-20120111 2012-01-11 Scientific Linux SL-libx-20120112 2012-01-12 Oracle ELSA-2012-0017 2012-01-12 Ubuntu USN-1334-1 2012-01-19 Debian DSA-2394-1 2012-01-26 Oracle ELSA-2012-0324 2012-03-09 Fedora FEDORA-2012-13820 2012-09-26 Fedora FEDORA-2012-13824 2012-09-27 Red Hat RHSA-2013:0217-01 2013-01-31 CentOS CESA-2013:0217 2013-02-01 Oracle ELSA-2013-0217 2013-02-01 Scientific Linux SL-ming-20130201 2013-02-01

Comments (none posted)

mojarra: code injection

Package(s):	mojarra	CVE #(s):	CVE-2011-4358
Created:	December 7, 2011	Updated:	December 7, 2011
Description:	Mojarra (a JavaServer Faces implementation) can be made to execute untrusted values as EL expressions in some configurations.
Alerts:	Debian DSA-2359-1 2011-12-06

Comments (none posted)

nginx: remote code execution

Package(s):	nginx-1.0	CVE #(s):	CVE-2011-4315
Created:	December 5, 2011	Updated:	February 9, 2012
Description:	The DNS resolver built into nginx suffers from a buffer overflow that could enable remote code execution attacks.
Alerts:	Fedora FEDORA-2011-16110 2011-11-19 Fedora FEDORA-2011-16075 2011-11-19 SUSE SUSE-SU-2011:1300-1 2011-12-05 openSUSE openSUSE-SU-2012:0237-1 2012-02-09 Gentoo 201203-22 2012-03-28

Comments (none posted)

psi: input validation failure

Package(s):	psi	CVE #(s):	CVE-2011-3365 CVE-2011-3366
Created:	December 6, 2011	Updated:	December 7, 2011
Description:	From the Red Hat bugzilla: An input validation failure was discovered in KSSL (CVE-2011-3365) and Rekonq (CVE-2011-3366) in KDE SC 4.6.0 up to and including KDE SC 4.7.1, however upstream indicates that ealier versions of KDE SC may also be affected.
Alerts:	Fedora FEDORA-2011-16476 2011-11-27 Fedora FEDORA-2011-16488 2011-11-27

Comments (none posted)

qemu-kvm: privilege escalation

Package(s):	qemu-kvm	CVE #(s):	CVE-2011-4111
Created:	December 7, 2011	Updated:	December 22, 2011
Description:	From the Red Hat advisory: A flaw was found in the way qemu-kvm handled VSC_ATR messages when a guest was configured for a CCID (Chip/Smart Card Interface Devices) USB smart card reader in passthrough mode. An attacker able to connect to the port on the host being used for such a device could use this flaw to crash the qemu-kvm process on the host or, possibly, escalate their privileges on the host.
Alerts:	CentOS CESA-2011:1777 2011-12-22 CentOS CESA-2011:1801 2011-12-22 Oracle ELSA-2011-1777 2011-12-17 Red Hat RHSA-2011:1801-01 2011-12-08 Red Hat RHSA-2011:1777-01 2011-12-06 Scientific Linux SL-qemu-20111206 2011-12-06

Comments (none posted)

ruby: predictable random numbers

Package(s):	ruby	CVE #(s):	CVE-2011-3009
Created:	December 7, 2011	Updated:	January 31, 2012
Description:	The Ruby interpreter does not reinitialize the random number generator after creating a child process, leading to a situation where two processes may get the same number.
Alerts:	Scientific Linux SL-ruby-20111206 2011-12-06 Red Hat RHSA-2011:1581-03 2011-12-06 Red Hat RHSA-2012:0070-01 2012-01-30 CentOS CESA-2012:0070 2012-01-30 CentOS CESA-2012:0070 2012-01-30 Oracle ELSA-2012-0070 2012-01-31 Oracle ELSA-2012-0070 2012-01-31 Scientific Linux SL-ruby-20120130 2012-01-30 openSUSE openSUSE-SU-2012:0228-1 2012-02-09

Comments (none posted)

ruby-on-rails: multiple vulnerabilities

Package(s):	rubygem-*	CVE #(s):	CVE-2010-3933 CVE-2011-0448 CVE-2011-0449
Created:	December 7, 2011	Updated:	December 7, 2011
Description:	The Ruby on Rails package suffers from vulnerabilities enabling arbitrary modification of records via crafted form parameters (CVE-2010-3933), SQL injection (CVE-2011-0448), and access restriction bypass (CVE-2011-0449).
Alerts:	openSUSE openSUSE-SU-2011:1305-1 2011-12-07

Comments (none posted)

sos: key disclosure

Package(s):	sos	CVE #(s):	CVE-2011-4083
Created:	December 7, 2011	Updated:	January 17, 2013
Description:	From the Red Hat advisory: The sosreport utility incorrectly included Certificate-based Red Hat Network private entitlement keys in the resulting archive of debugging information. An attacker able to access the archive could use the keys to access Red Hat Network content available to the host.
Alerts:	Scientific Linux SL-sos-20111206 2011-12-06 Red Hat RHSA-2011:1536-03 2011-12-06 Red Hat RHSA-2012:0153-03 2012-02-21 Oracle ELSA-2012-0153 2012-03-07 Scientific Linux SL-sos-20120321 2012-03-21 CentOS CESA-2012:0153 2013-01-09

Comments (none posted)

torque: user impersonation

Package(s):	torque	CVE #(s):
Created:	December 5, 2011	Updated:	December 7, 2011
Description:	A user connecting to the torque "pbs_server" is able to impersonate another user in the torque batch system.
Alerts:	Fedora FEDORA-2011-16128 2011-12-04

Comments (none posted)

util-linux-ng: denial of service

Package(s):	util-linux-ng	CVE #(s):	CVE-2011-1675 CVE-2011-1677
Created:	December 7, 2011	Updated:	May 29, 2012
Description:	Vulnerabilities in the util-linux-ng package allow a local user with the ability to mount an unmount filesystems to corrupt the mtab file and leave a stale lock file around, interfering with others' ability to mount filesystems.
Alerts:	Scientific Linux SL-util-20111206 2011-12-06 Red Hat RHSA-2011:1691-03 2011-12-06 Red Hat RHSA-2012:0307-03 2012-02-21 Oracle ELSA-2012-0307 2012-03-07 Scientific Linux SL-util-20120321 2012-03-21 Mandriva MDVSA-2012:083 2012-05-29

Comments (none posted)

virt-v2v: privilege escalation

Package(s):	virt-v2v	CVE #(s):	CVE-2011-1773
Created:	December 7, 2011	Updated:	December 16, 2011
Description:	From the Red Hat advisory: Using virt-v2v to convert a guest that has a password-protected VNC console to a KVM guest removed that password protection from the converted guest: after conversion, a password was not required to access the converted guest's VNC console.
Alerts:	Scientific Linux SL-virt-20111206 2011-12-06 Red Hat RHSA-2011:1615-03 2011-12-06

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 3.2-rc4, released on December 1. Linus says that things are quieting down, but he's worried: "I'm waiting for the other shoe to drop. Maybe Davem and GregKH are holding back - they've been suspiciously quiet, and I think I can hear some evil chuckling going on there. But maybe it's just time for my meds."

Stable updates: no stable updates have been released in the last week. The 2.6.32.50 (27 patches) 3.0.13 (80 patches) and 3.1.5 (104 patches) updates are in the review process as of this writing; they can be expected on or after December 9.

Comments (none posted)

Quotes of the week

-- Alan Cox

-- Russell King

-- Linus Torvalds

Comments (none posted)

Bufferbloat: Dark Buffers in the Internet (ACM Queue)

Jim Gettys and Kathleen Nichols have published a comprehensive, updated description of the bufferbloat problem in ACM Queue. "For TCP congestion avoidance to be useful to people using that link, a TCP connection causing congestion must react quickly to changes in demand at the bottleneck link, but TCP's reaction time is quadratic to the amount of overbuffering. A link that is 10 times overbuffered not only imposes 10 times the latency, but also takes 100 times as long to react to the congestion. Your short, interactive TCP connection loses completely to any long-lived flow that has saturated your link."

Comments (58 posted)

Kernel development news

Per-cgroup TCP buffer limits

Control groups are used for a number of purposes, but the most important role for many is the application of resource limits to groups of processes. The memory usage controller has seen a lot of development effort in recent years, but it still has a number of shortcomings, one of which being that it only applies to user-space memory. Some workloads can cause the use of large amounts of memory in the kernel, leading to overall system memory pressure, but the memory controller is unable to place limits on that memory usage. A first step toward fixing that problem may yet find its way into the 3.3 kernel, but it shows how hard the job can be.

Glauber Costa's per-cgroup TCP memory pressure controls patch starts by adding a bit of infrastructure to the memory controller for the tracking of kernel-space memory use. A pair of new knobs is added: memory.kmem.limit_in_bytes controls how much kernel memory a control group is allowed to use, while memory.independent_kmem_limit is a boolean controlling whether the kernel limits are to be managed separately from the user-space limits. If that boolean is false, kernel memory usage is simply added to user-space memory usage and the kernel memory limit is ignored. There is also a new memory.kmem.usage_in_bytes value that can be read to see what the current memory usage is.

This infrastructure is nice, but there is one other little difficulty: there are thousands of places in the kernel where memory is allocated and none of them are instrumented in a way that allows those allocations to be charged to a specific control group. An implementation of that accounting on a kernel-wide basis may never happen; it certainly will not happen at the outset. So developers seeking to add this functionality need to focus on specific data structures. Some past work has been done to apply limits to the dentry cache, but Glauber chose a different limit: buffers used in the implementation of the TCP network protocol.

Those buffers hold packet data as it passes through a socket; in some situations they can become quite large, so they are a logical place to try to apply limits. What is even better is that the networking stack already has logic to place limits on buffer sizes when memory is tight. In current kernels, if the system as a whole is seen to be under memory pressure, the networking code will do its best to help. Its responses can include decisions not to increase the size of TCP windows, to drop packets, or to refuse to expand buffers for outgoing data.

Given that this infrastructure was in place, all that Glauber needed to do was to enhance the stack's idea of what constitutes "memory pressure." That means instrumenting the places that allocate and free buffer memory to charge those allocations to the owning control group. Then, "memory pressure" becomes a combination of the previous global value and the current control group's status with regard to its kernel memory limit. If that limit has been exceeded, the stack will behave (with regard to that control group's sockets) as if the system as a whole were under memory pressure, even if the global state is fine.

The first patches did exactly that and seemed to be on their way toward inclusion in the 3.2 merge window. It ran into a bit of a snag when the core networking developers took a look at it, though. Networking maintainer David Miller rejected the patch out of hand, complaining about the overhead it adds to the networking fast paths even in cases where the feature is not in use. He added:

There are a lot of important memory-allocation sites in the kernel that can be found in relatively hot execution paths; the networking developers may obsess about adding overhead to those paths, but they are certainly not alone in that concern. So a solution had to be found that did not impose that overhead on systems where the feature is not in use.

The words "in use" are important here as well. If kernel memory usage limits are useful, distributors will want to enable them in the kernels they ship. But it may still be that most users will not actually make use of that feature. So it is not sufficient to remove its overhead only in cases where it has been configured out of the kernel. A feature like this really needs to avoid imposing costs when it is not in use, even when it is configured into the running kernel.

Glauber had to make some significant changes to the patch set to meet these requirements. TCP buffer limits are now managed separately from kernel limits as a whole; there is a new knob (kmem.tcp.limit_in_bytes) to control that limit. All of the fast-path code is now contained within a static branch, meaning that, when the code is not enabled, it can be skipped over with almost no overhead at all. That static branch is only turned on when a TCP buffer limit is established for a non-root control group. So, as required, it should not have significant effects in kernels where the feature is configured in but not being used.

As of this writing, no decision to merge these patches for 3.3 has been announced, but the number of criticisms has been steadily dropping. So it may well get into the mainline in the next merge window. But Glauber's experience shows how hard it will be to add more kernel memory accounting; the requirement that its impact be imperceptible will lead to more work and trickier code. For that reason alone, accounting for all kernel memory use seems unlikely to ever happen. Developers will focus on the few cases where the application of limits can bring about significant changes in behavior and not worry about the rest.

Comments (none posted)

Irked by NO_IRQ

The kernel has always used small integer numbers to represent interrupt (IRQ) lines internally; those numbers usually correspond to the numbers of the interrupt lines feeding into the CPU, though it need not be that way. The kernel has also traditionally used the value zero to indicate that there is no IRQ assignment at all - except in the places where it hasn't. That inconsistency has, after many years, popped up to bite the ARM architecture, with the result that there may have to be a big scramble to fix things for the 3.3 merge window.

While the x86 architecture and core interrupt code have used zero for "no IRQ," various architectures have made other decisions, often using a value of -1 instead. The reason for making such a choice is straightforward: many architectures have a valid IRQ line numbered zero. In that situation, a few options present themselves. That inconveniently-numbered IRQ can be hidden away where nobody actually sees it; that is what x86 does on systems where the timer interrupt shows up as zero. Another option is to remap that interrupt line to a different number in software so that IRQ 0 never appears outside of the core architecture code. Or the architecture can treat zero as a valid IRQ number and define a symbol like NO_IRQ to a different value; a number of architectures have done that over time, though some have switched to zero in recent years. But the ARM architecture still uses -1 as its NO_IRQ value.

It is worth noting that NO_IRQ has been a topic of discussion at various times over the years. The earliest appears to have been in response to this patch from Matthew Wilcox, posted in 2005. At that time, Linus made it clear that he saw zero as the only valid NO_IRQ value. His reasoning was that code reading:

    if (!dev->irq)
	/* handle the no-IRQ case */

is easier to read and more efficient to execute than comparisons against some other value. Beyond that, IRQ numbers were (and are) represented by unsigned variables in many parts of the kernel; values like -1 are clearly an awkward fit in such situations. Given that, he said:

In the intervening six years, most architectures have followed that suggestion, but ARM has not. That has mattered little thus far, but, as ARM's popularity grows and the amount of code shared with the rest of the kernel grows with it, that inconsistency is leading to problems. Some recent changes to the device tree code threaten to break ARM entirely (for some subplatforms, anyway) if things are not changed.

How hard will things be to fix? Alan Cox offered this suggestion:

As it happens, though, it's not quite that easy. The core ARM code will need to remap IRQ 0 in places where it is a valid number. Arguably the most unpleasant problem, though, was pointed out by Dave Martin:

So it will be necessary to find all of the places where the code assumes that a negative IRQ number means "no interrupt," places where -1 has been hard coded, and so on. One can argue (as ARM maintainer Russell King has) that, as long as request_irq(0) fails and physical IRQ 0 is handled properly, most of that code will do the right thing without being changed. But there is still a fair amount of stuff to clean up.

Putting it into perspective, though: the work required to fix NO_IRQ in the ARM tree is likely to be tiny relative to the general cleanup that is happening there. A bit of churn, some drivers to fix, and ARM should come into agreement with the rest of the kernel on the way to indicate a missing interrupt. That, in turn, will increase robustness and allow the sharing of more code with the rest of the kernel - not something to get too terribly irked about.

Comments (4 posted)

Validating Memory Barriers and Atomic Instructions

Some of the most obscure corners of the Linux kernel are those employing the dark arts of atomic instructions and memory barriers. Although things are better than they were (say) ten years ago, atomic instructions and memory barriers are still a not-infrequent source of confusion.

One of the difficulties with atomic instructions and especially with memory barriers is that the documentation for the underlying machine instructions can be a bit difficult to understand. It is not hard to work around obtuse documentation for arithmetic instructions: Simply test them. However, simple testing does not work for memory barriers and atomic instructions because these instructions must handle non-deterministic situations. Furthermore, unlike arithmetic instructions, a given piece of hardware is free to be more strict that absolutely necessary in its interpretation of ordering requirements. Therefore, your test results might not apply to the next generation of that same hardware, which in turn might result in your code breaking horribly.

What we need is a tool that evaluates a code sequence containing atomic operations and memory barriers, telling the developer whether or not a given assertion is guaranteed never to trigger. Given such a guarantee, the developer could then be reasonably confident that this code sequence would operate correctly on all current and future versions of the hardware in question.

Previous LWN articles (here and here), as well as Mathieu Desnoyers's doctoral dissertation [PDF], describe how to use Promela and spin to test assertions on code fragments that execute concurrently. Because Promela and spin search the code fragments' full state space, the developer can be reasonably confident that the code fragments will operate correctly on real hardware. Unfortunately, most code fragments using atomic instructions and memory barriers require detailed knowledge of the semantics of a given CPU's memory barriers and atomic instructions. Promela and spin do not have such knowledge, so the developer must explicitly write Promela/spin code that captures this detailed knowledge. This is not helpful if the developer in question does not possess this detailed knowledge in the first place.

It would clearly be much better to have a tool that already understands this detailed knowledge. I am happy to report that such tools are now becoming available, for example, from Peter Sewell and Susmit Sarkar at the University of Cambridge, Luc Maranget, Francesco Zappa Nardelli, and Pankaj Pawan at INRIA, and Jade Alglave at Oxford University. This group has done some excellent work over the past few years formalizing memory models for real computer systems, including that of ARM, Power, some additional 64-bit members of the Power family, x86 (and yes, academics consider x86 to be a weak-memory system), and C/C++11's new memory model.

A prototype interactive version of this tool is available for ARM and Power.

This article describes this tool as follows:

1. Anatomy of a Litmus Test.
2. What Does This Litmus Test Mean?
3. Running a Litmus Test.
4. Full State-Space Search.
5. Conclusions.

Following this is the obligatory Answers to Quick Quizzes.

Anatomy of a litmus test

The input to this tool is a “litmus test” that consists of assembly language, register initializations, and an assertion, for example, as follows for the Power interface to the model:

  1 PPC SB+lwsync-RMW-lwsync+isync-simple
  2 ""
  3 {
  4 0:r2=x; 0:r3=2; 0:r4=y;   0:r10=0 ; 0:r11=0; 0:r12=z ;
  5 1:r2=y; 1:r4=x;
  6 }
  7  P0                 | P1           ;
  8  li r1,1            | li r1,1      ;
  9  stw r1,0(r2)       | stw r1,0(r2) ;
 10  lwsync             | sync         ;
 11                     | lwz r3,0(r4) ;
 12  lwarx  r11,r10,r12 | ;
 13  stwcx. r11,r10,r12 | ;
 14  bne Fail1          | ;
 15  isync              | ;
 16  lwz r3,0(r4)       | ;
 17  Fail1:             | ;
 18 
 19 exists
 20 (0:r3=0 /\ 1:r3=0)

The ARM interface works exactly the same way, but with ARM instructions substituted for the Power instructions and with the initial “PPC” replaced by “ARM”. You can select the ARM interface by clicking on “Change to ARM Model” at the web page called out above.

In the example, line 1 identifies the type of system (“ARM” or “PPC”) and contains the title for the model. Line 2 provides a place for an alternative name for the test, which you will usually want to leave blank as shown in the above example. Comments can be inserted between lines 2 and 3 using the OCaml (or Pascal) syntax of “(* *)”.

Lines 3-6 give initial values for all registers; each is of the form “P:R=V”, where “P” is the process identifier, “R” is the register identifier, and “V” is the value. For example, process 0's register r3 initially contains the value 2. If the value is a variable (“x”, “y”, or “z” in the example) then the register is initialized to the address of the variable. It is also possible to initialize the contents of variables, for example, “x=1” initializes the value of “x” to 1. Uninitialized variables default to the value zero, so that in the example, “x”, “y”, and “z” are all initially zero.

Line 7 provides identifiers for the two processes, so that the “0:r3=2” on line 4 could instead have been written “P0:r3=2”. Line 7 is required, and the identifiers must be of the form “Pn”, where “n” is the column number, starting from zero for the left-most column. This may seem unnecessarily strict, but it does prevent considerable confusion in actual use.

Lines 8-17 are the lines of code for each process. A given process can have empty lines, as is the case for P0's line 11 and P1's lines 12-17. Labels and branches are permitted, as demonstrated by the branch on line 14 to the label on line 17. That said, too-free use of branches will expand the state space. Use of loops is a particularly good way to explode your state space.

Lines 19-20 show the assertion, which in this case indicates that we are interested in whether P0's and P1's r3 registers can both contain zero after both threads complete execution. This assertion is important because there are a number of use cases that would fail miserably if both P0 and P1 saw zero in their respective r3 registers.

This should give you enough information to construct simple litmus tests. Some additional documentation is available, though much of this additional documentation is intended for a different research tool that runs tests on actual hardware. Perhaps more importantly, a large number of pre-existing litmus tests are available with the online tool (available via the “Select ARM Test” and “Select POWER Test” buttons). It is quite likely that one of these pre-existing litmus tests will answer your Power or ARM memory-ordering question.

What does this litmus test mean?

P0's lines 8 and 9 are equivalent to the C statement x=1 because line 4 defines P0's register r2 to be the address of x. P0's lines 12 and 13 are the mnemonics for load-linked (“load register exclusive” in ARM parlance and “load reserve” in Power parlance) and store-conditional (“store register exclusive” in ARM parlance), respectively. When these are used together, they form an atomic instruction sequence, roughly similar to the compare-and-swap sequences exemplified by the x86 lock;cmpxchg instruction. Moving to a higher level of abstraction, the sequence from lines 10-15 is equivalent to the Linux kernel's atomic_add_return(&z, 0). Finally, line 16 is roughly equivalent to the C statement r3=y.

P1's lines 8 and 9 are equivalent to the C statement y=1, line 10 is a memory barrier, equivalent to the Linux kernel statement smp_mb(), and line 11 is equivalent to the C statement r3=x.

Putting all this together, the C-language equivalent to the entire litmus test is as follows:

  1 void P0(void)
  2 {
  3   x = 1; /* Lines 8 and 9 */
  4   atomic_add_return(&z, 0); /* Lines 10-15 */
  5   r3 = y; /* Line 16 */
  6 }
  7 
  8 void P1(void)
  9 {
 10   y = 1; /* Lines 8-9 */
 11   smp_mb(); /* Line 10 */
 12   r3 = x; /* Line 11 */
 13 }

Now that we have a meaningful litmus test, it is time to run it.

Running a litmus test

To run the litmus test, paste it into the textbox at http://www.cl.cam.ac.uk/~pes20/ppcmem/, but use a version without line numbers. Press the "Interactive" button at the lower left-hand side of the page, and you should then see something like this:

(Click on the images for full-size versions)

The upper portion of the screen beginning with “Storage subsystem state” tracks an abstraction of the state of the processor's write buffers, caches, and cache-coherence messages. The next two portions, beginning with “Thread 0 state” and “Thread 1 state”, track the instruction and memory-reference progress of each of the two threads. For more details on the information presented by this tool, refer to the PLDI 2011 paper entitled “Understanding Power Multiprocessors”, which may be found here.

Because this model is intended for a super-scalar weak-memory system, a given thread can have several different operations that might happen at a given time. For example, for thread 0 there are two operations, indicated by the links (“Commit” and “Read reserve k:W z 0”), while thread 1 has only a single “Commit” link. Clicking on a given link allows you to manually advance the state of the system.

In this particular case, it is most instructive to start by clicking the links for thread 0, ignoring thread 1 completely and also ignoring any links that appear in the “Storage subsystem state” section preceding thread 0. There will come a time when the “stwcx” instruction gives you a choice between “Commit (fail)” and “Commit (succ)”. Choosing “Commit (succ)” results in the following state:

At this point, it is time to click the links for thread 1, resulting in the following state:

Now, click the last link, which is in the “Storage subsystem state” section and is labeled “(0:) Write propagate to thread: f:W y 1 to Thread 0”, and then click the “(0:) Barrier propagate to thread: g:Sync to Thread 0” that replaces it, and then the second link (labeled “Write reaching coherence point: f:W y 1”), and finally the “Acknowledge sync: Sync g:Sync”. This results in the following state:

This will result in thread 1 gaining a “Read i:W x 0” link. Click this and the “Commit” link that replaces it. Then click all the remaining links in any order, resulting in the following final state:

This shows that the values loaded into both threads' r3 registers to be zero, in violation of the requirements set forth in Documentation/atomic_ops.txt. This bug is easily fixed by replacing the isync instruction with sync, and a patch exists to carry out this fix.

The fact that this tool is capable of finding this sort of bug is a testament to its usefulness.

Full state-space search

The interactive web-based tool is fun to use and can be quite enlightening, but it is very difficult to make sure that you have checked for every possible error condition. For that reason, there is an executable tool that conducts a full state-space search, whose source code may be downloaded here, covered by the BSD license, with some components licensed under LGPL with an exception for linking. The tool may be built by following the instructions in the README file, which requires a recent version of OCaml (3.12.0 works for me, but 3.11.2 doesn't cut it). Please note that this tool is a research prototype that is completely unsupported. That said, I have found it to be quite useful.

The tool is run from its build directory. For litmus tests containing only normal memory accesses and memory barriers, the following command suffices:

    ./ppcmem filename.litmus

where “filename.litmus” is the path to the file containing your litmus test.

The tail end of the output of this tool when presented the litmus test discussed earlier is as follows:

States 6
0:r3=0; 1:r3=0;
0:r3=0; 1:r3=1;
0:r3=1; 1:r3=0;
0:r3=1; 1:r3=1;
0:r3=2; 1:r3=0;
0:r3=2; 1:r3=1;
Ok
Condition exists (0:r3=0 /\ 1:r3=0)
Hash=e2240ce2072a2610c034ccd4fc964e77
Observation SB+lwsync-RMW-lwsync+isync-simple Sometimes 1 

The list of states includes “0:r3=0; 1:r3=0;”, indicating once again that the old powerpc implementation of atomic_add_return() does not act as a full barrier. The “Sometimes” on the last line confirms this: the assertion triggers for some executions, but not all of the time. As before, the fix is to replace P0's isync with sync, which results in the following at the end of the tool's output:

States 5
0:r3=0; 1:r3=1;
0:r3=1; 1:r3=0;
0:r3=1; 1:r3=1;
0:r3=2; 1:r3=0;
0:r3=2; 1:r3=1;
No (allowed not found)
Condition exists (0:r3=0 /\ 1:r3=0)
Hash=77dd723cda9981248ea4459fcdf6097d
Observation SB+lwsync-RMW-lwsync+sync-simple Never 0 5

This output confirms the fix: “0:r3=0; 1:r3=0;” does not appear in the list of states, and the last line calls out “Never”. Therefore, the model predicts that the offending execution sequence cannot happen.

Conclusions

These tools promise to be of great help to people working on low-level parallel primitives that run on ARM and on Power. These tools do have some intrinsic limitations:

1. These tools do not constitute official statements by IBM or ARM on their respective CPU architectures. For example, both corporations reserve the right to report a bug at any time against any version of any of these tools. These tools are therefore not a substitute for careful stress testing on real hardware. Moreover, both the tools and the model that they are based on are under active development and might change at any time. On the other hand, this model was developed in consultation with the relevant hardware experts, so there is good reason to be confident that it is a robust representation of the architectures.

2. These tools currently handle a subset of the instruction set. This subset has been sufficient for my purposes, but your mileage may vary. In particular, the tool handles only word-sized accesses (32 bits), and the words accessed must be properly aligned. In addition, the tool does not handle some of the weaker variants of the ARM memory-barrier instructions.

3. The tools are restricted to small loop-free code fragments running on small numbers of threads. Larger examples result in state-space explosion, just as with similar tools such as Promela and spin.

4. The full state-space search does not give any indication of how each offending state was reached. That said, once you realize that the state is in fact reachable, it is usually not too hard to find that state using the interactive tool.

5. The tools will detect only those problems for which you code an assertion. This weakness is common to all formal methods, and is yet another reason why testing remains important. In the immortal words of Donald Knuth, “Beware of bugs in the above code; I have only proved it correct, not tried it.”

That said, one strength of these tools is that they are designed to model the full range of behaviors allowed by the architectures, including behaviors that are legal, but which current hardware implementations do not yet inflict on unwary software developers. Therefore, an algorithm that is vetted by these tools likely has some additional safety margin when running on real hardware. Furthermore, testing on real hardware can only find bugs; such testing is inherently incapable of proving a given usage correct. To appreciate this, consider the table in Section 8 on page 10 of Understanding POWER Multiprocessors [PDF]: the researchers routinely ran in excess of 100 billion test runs on real hardware to validate their model. In one case, behavior that is allowed by the architecture did not occur, despite 176 billion runs. In contrast, the full-state-space search allows the tool to prove code fragments correct.

It is worth repeating that formal methods and tools are no substitute for testing. The fact is that producing large reliable concurrent software artifacts, the Linux kernel for example, is quite difficult. Developers must therefore be prepared to apply every tool at their disposal towards this goal. The tools presented in this paper are able to locate bugs that are quite difficult to produce (let alone track down) via testing. On the other hand, testing can be applied to far larger bodies of software than the tools presented in this paper are ever likely to handle. As always, use the right tools for the job!

Of course, it is always best to avoid the need to work at this level by designing your parallel code to be easily partitioned and then using higher-level primitives (such as locks, sequence counters, atomic operations, and RCU) to get your job done more straightforwardly. And even if you absolutely must use low-level memory barriers and read-modify-write instructions to get your job done, the more conservative your use of these sharp instruments, the easier your life is likely to be.

Acknowledgments

I am grateful to Peter Sewell and Susmit Sarkar at the University of Cambridge, Luc Maranget, Francesco Zappa Nardelli, and Pankaj Pawan at INRIA, Jade Alglave at Oxford University, and a number of their colleagues for their efforts on this research topic. We all are indebted to Ben Herrenschmidt for fixing the bug noted in this article. I am thankful to great number of members of the C and C++ standards committees for many stimulating discussions on memory models and concurrency, including Hans Boehm, Lawrence Crowl, Peter Dimov, Clark Nelson, Raul Silvera, Mark Batty, N.M. Maclaren, Anthony Williams, Blaine Garst, Scott Owens, Tjark Weber, Michael Wong, Benjamin Kosnik, and Bjarne Stroustrup. I owe thanks to Derek Williams and Richard Grisenthwaite for their patient explanations of the Power and ARM memory models, respectively, and to Jim Wasko of IBM and Dave Rusling of Linaro for their support of this effort.

Answers to quick quizzes

Quick Quiz 1: Yeah, right!!! Since when has anything ever made the Linux kernel's memory barriers and atomic instructions easier???

Answer: There really have been some improvements over the years, including updates to Documentation/memory-barriers.txt and Documentation/atomic_ops.txt. Perhaps more important, the scripts/checkpatch.pl script complains if memory barriers are not accompanied by a comment, which has made the code using memory barriers a bit less obscure.

But yes, more help is needed, hence this article.

Back to Quick Quiz 1.

---

Quick Quiz 2: But what about x86?

Answer: The ppcmem tool could in fact be extended to handle x86 as well as ARM and Power, but there are no definite plans to carry out this work at the moment.

Back to Quick Quiz 2.

---

Quick Quiz 3: Why does line 8 initialize the registers? Why not instead initialize them on lines 4 and 5?

Answer: Either way works. However, in general, it is better to use initialization than explicit instructions. The explicit instructions are used in this example to demonstrate their use. In addition, many of the litmus tests available on the tool's web site were automatically generated, which generates explicit initialization instructions.

Back to Quick Quiz 3.

---

Quick Quiz 4: But whatever happened to line 17, the one that is the Fail: label?

Answer: The implementation of powerpc version of atomic_add_return() loops when the stwcx instruction fails, which it communicates by setting non-zero status in the condition-code register, which in turn is tested by the bne instruction. Because actually modeling the loop would result in state-space explosion, we instead branch to the Fail: label, terminating the model with the initial value of 2 in thread 1's r3 register, which will not trigger the exists assertion.

There is some debate about whether this trick is universally applicable, but I have not seen an example where it fails. (Famous last words!)

Back to Quick Quiz 4.

---

Quick Quiz 5: Can both threads' r3 registers be equal to zero after both threads complete?

Answer: Not if atomic_add_return() acts as a full barrier, as is set out as required behavior in Documentation/atomic_ops.txt.

Back to Quick Quiz 5.

---

Quick Quiz 6: But suppose that I didn't think to try this particular execution sequence. How could I have found this bug using this tool?

Answer: You can easily miss problematic execution sequences when using the interactive tool. Which is why the next section covers the full state-space-search tool.

Back to Quick Quiz 6.

---

Quick Quiz 7: Does the ARM Linux kernel have a similar bug?

Answer: ARM does not have this particular bug, given that it places smp_mb() before and after the atomic_add_return() function's assembly-language implementation. Finding any other bugs that ARM might have is left as an exercise for the reader.

Back to Quick Quiz 7.

---

Quick Quiz 8: But when I built the tool, I didn't find an armmem program. How am I supposed to test ARM code fragments???

Answer: Just replace the litmus file's initial “PPC” with “ARM” and put ARM assembly language into your litmus file and pass it to the ppcmem program. Yes, ppcmem is multilingual. Perhaps this is because the researchers who created it have a variety of native languages. Or maybe because none of them live or work in the USA. ;–)

Back to Quick Quiz 8.

---

Quick Quiz 9: Where can I find out more about the formal model underlying this tool?

Answer: Currently the best source of information is the PLDI 2011 paper entitled “Understanding Power Multiprocessors”, which may be found here. This paper covers memory barriers, but not atomic instruction sequences. Additional papers that include atomic-instruction-sequence extensions to this model are likely to appear Real Soon Now.

Back to Quick Quiz 9.

---

Comments (9 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.2-rc4 . (December 2, 2011)

• Thomas Gleixner: 3.2-rc4-rt5 . (December 5, 2011)

• Thomas Gleixner: 3.2-rc4-rt6 . (December 6, 2011)

• Steven Rostedt: 3.0.12-rt29 . (December 1, 2011)

Core kernel code

• Suresh Siddha: nohz idle load balancing patches . (December 2, 2011)

• Lucian Adrian Grijincu: v6: faster tree-based sysctl implementation . (December 5, 2011)

Development tools

• Frederic Weisbecker: selftests: New very basic kernel selftests directory . (December 5, 2011)

• Jiri Olsa: ftrace, perf: Adding support to use function trace . (December 6, 2011)

• jim.cromie@gmail.com: dynamic debug enhancements: multi-queries during mod-init . (December 7, 2011)

• Andrew Vagin: Profiling sleep times (v4) . (December 7, 2011)

Device drivers

• Ming Lei: media&omap4: introduce face detection(FD) driver . (December 2, 2011)

• Lin Ming: ata port runtime power management support . (December 5, 2011)

• NeilBrown: Add driver for TCA6507 LED controller. . (December 6, 2011)

• Linus Walleij: leds: driver for ab5500 high voltage leds . (December 6, 2011)

• Sergio Aguirre: v4l2: OMAP4 ISS driver + Sensor + Board support . (December 6, 2011)

• David Gibson: RFC: Device isolation infrastructure . (December 7, 2011)

• Sjur Brændeland: XSHM: Shared Memory Driver for ST-E Thor M7400 LTE modem . (December 7, 2011)

Filesystems and block I/O

• Al Viro: [RFC] __d_path() API change (was Re: [PATCH] Remove use of mnt_ns->root and fix a couple of bugs in d_namespace_path) . (December 5, 2011)

Memory management

• Mel Gorman: Reduce compaction-related stalls and improve asynchronous migration of dirty pages v5 . (December 2, 2011)

• Sumit Semwal: Introduce DMA buffer sharing mechanism . (December 2, 2011)

• Glauber Costa: per-cgroup tcp memory pressure controls . (December 6, 2011)

• Ying Han: memcg softlimit reclaim rework . (December 7, 2011)

Networking

• Jesse Gross: Open vSwitch . (December 6, 2011)

Architecture-specific

• Joerg Roedel: Initial AMD IOMMUv2 support . (December 5, 2011)

Security-related

• Kees Cook: security: Yama LSM . (December 1, 2011)

• David Howells: Crypto keys and module signing [ver #3] . (December 5, 2011)

• Kees Cook: fs: symlink restrictions on sticky directories . (December 7, 2011)

Virtualization and containers

• Amit Shah: virtio: s4 support . (December 7, 2011)

Miscellaneous

• Darrick J. Wong: crc32c: Add faster algorithm and self-test code . (December 2, 2011)

• Ben Hutchings: ethtool 3.1 released . (December 6, 2011)

Page editor: Jonathan Corbet

Distributions

Distribution "popularity"

There has been quite a bit of press, and some hand-wringing, over reports that Linux Mint has overtaken Ubuntu as the "most popular" Linux distribution. The reports are based on the DistroWatch rankings, which some—though notably not the DistroWatch folks—seem to think indicates the popularity of various distributions. While it's a bit hard to imagine that untold legions of Ubuntu users have switched to Linux Mint en masse, it does have a non-zero probability of being true. But there aren't, and really can't be, any numbers to back that up. Is popularity even really the best measure of a distribution?

The "rankings" that have spawned the uproar are simple page-hit counts. Each unique IP address that lands on DistroWatch's page for a given distribution increments the count for the day. It is, at best, a count of the amount of "buzz" a particular distribution has over the past one, three, six, and twelve months. It can also be fairly easily manipulated by someone who has unfettered access to a large number of IP addresses or a botnet—as well as by over-exuberant distribution fans—though there is no evidence to suggest that's what's happening here. As DistroWatch says, those numbers are:

But, for whatever reason, Mint shows up at the top of the list for average number of hits per day (HPD) for each of the four periods. In fact, Ubuntu has "slipped" to fourth place over the last month with Fedora and openSUSE taking second and third place respectively. Mint shows nearly three times the number of HPD that any of the rest of the top four do. That's interesting, perhaps, but not meaningful. It is a self-selected "poll" that could be fairly easily manipulated—likely unintentionally.

The ranking is also heavily skewed toward desktop distributions, as can be seen by the numbers for server-oriented distributions like Red Hat (which ranks below things like GhostBSD, Zorin, and Tiny Core) or SUSE (which ranks a bit lower). Both of those distributions should have accurate sales numbers that may show a tad more popularity than reading things into the DistroWatch numbers will show. In short, even a brief look at the rankings page should be enough to deter anyone from deriving conclusions that result in headlines like "Ubuntu sees massive slide in popularity, Mint sprints ahead ... but why?".

Part of the problem here is that it is somewhere between difficult and impossible to get accurate figures for distribution usage. In fact, it goes well beyond just distributions; accurately counting users of any free or proprietary software is well-nigh impossible. Vendors who sell their software have some advantage, but even they don't know how many users there are. Microsoft can undoubtedly report how many copies of Windows it sold in the last month (quarter, year, ...), but that most certainly doesn't count the number of Windows users. That number is likely to be much higher due to unlicensed users, which probably dwarfs the not completely insubstantial number of pre-installed systems that get wiped to run other operating systems.

The usual methods to try to track users, like phoning home with some kind of unique ID, are intrusive. For free software, those mechanisms are unlikely to be tolerated by some, but even users of proprietary software may find ways to avoid being counted. Companies selling software count their users in terms of dollars (euros, ...) so, other than being able to report inflated piracy numbers as "lost sales", there is no real need for additional counting. Free software projects and distributions are different.

Those who work on free projects would certainly like to feel that their work is being used and appreciated. That's not unreasonable at all, but is popularity really the right measure of that? Even if it can be reliably measured, popularity just measures ... well ... what's popular—not what works best, solves the most problems, or anything else. Does it really matter if Ubuntu has X million users and Linux Mint has X/4 million—or the reverse? In both cases, the distributions are serving a substantial number of people and, presumably, solving lots of their problems.

There are some "active counting" efforts by various distributions but, as would be expected for free software projects, they are "opt-in" services. Fedora and openSUSE both use smolt to gather semi-anonymized installation data. Debian and Ubuntu use popcon to generate information on the popularity of various packages. While users are asked to enable these counting mechanisms at install time, it's not clear how many actually do so.

Since directly measuring users is difficult, distributions often use indirect (and fairly inaccurate) methods to try to get a handle on their number of users. Both Fedora and openSUSE count unique IP-address connections to their update servers and have fairly detailed pages that outline what they are counting (openSUSE, Fedora). Ubuntu has been notoriously lax in providing any real information on its methodology—without being shy about producing numbers like 20 million Ubuntu users—but one would guess it is doing something similar.

That kind of data collection isn't really accurate to generate a "number of users" figure, though it may be fine as an estimate. Assuming the methodology remains the same, it may also serve as a reasonable indicator of trends in the number of users. If Fedora 16 has 50% more unique IPs getting updates, that's a pretty good indicator that F16 has been adopted more widely. Comparing F16's raw numbers to those of openSUSE 11.3, for example, is much less useful.

But obsessing over estimated numbers—or illusory trends based on web page hits—seems counterproductive. While it is harder to generate numbers, the measure of a community distribution really should be how vibrant its community is. Are new people showing up, filing bugs, participating in development or design discussions, packaging new software, translating existing software, taking on new tasks, running for elected positions, and so on? Those are certainly measures of growth, though numerically hard to quantify.

Focusing on a "zero sum" game for Linux distributions is equally counterproductive. While the GNOME 3 and Unity decisions made by various distributions have generated a lot of noise (and likely some distribution and desktop environment switches), it's pretty hard to justify a "Ubuntu users are running to Mint because of Unity" stance on anything other than anecdotal evidence. If the suggested trend is even real, it could be that Mint is attracting many of the first-time Linux users that Ubuntu once did, or that it is attracting more than Ubuntu currently is. That could be due to the "buzz" factor for Mint these days, for example. Not all (or even most) growth of Linux distributions needs to come at the expense of other distributions.

Unlike the choice between Windows and OS X (or Linux and either of those), the choice between Linux distributions is far less susceptible to concerns about lock-in. Part of what free software enables is relatively easy migration between distributions, with full data and application portability, which undoubtedly leads to some "distro hopping". But it's also true that providing that freedom can attract new users. We've seen it over the past 20 years, even if the growth on the desktop is not up to what most had hoped for. Focusing on serving existing users, while attracting new ones, rather than worrying about pumping up popularity numbers, is a much more likely road to success.

Comments (12 posted)

Brief items

Distribution quote of the week

-- Goswin von Brederlow

-- Stephen Smoogen

Comments (24 posted)

Some Cerowrt updates

For those who are looking forward to the 1.0 release of the Cerowrt bufferbloat-fighting router distribution, Dave Täht has posted a status update. "In summary, it IS possible to build a device far superior to anything shipped today, one that can set an example for the rest of the industry, save everyone time, and hassle, and do it for very low cost, AND consisting almost purely of software that can be applied to any hardware with the same capabilities, thus introducing competition to drive the costs down while holding the desirable feature set to a whole new standard." There is discussion of reducing the feature set to make the distribution more manageable, though; possible "puppies to shoot" include bind9 and ISC-ntp, mesh networking, and extensive IPv6 support.

Comments (28 posted)

Red Hat Enterprise Linux 6.2 released

Red Hat has announced the release of RHEL 6.2. "Red Hat Enterprise Linux 6.2 adds enhancements to storage and file system features including full support of iSCSI extension for RDMA. Now, benefits of low latency and high throughput through a standard SAN implementation based on 10Gb Ethernet are available to even the most demanding storage environments. This allows customers to opt out of expensive Infiniband hardware or other dedicated interconnect fabrics. Other enhancements around file system include delayed meta data logging, asynchronous and parallel file system writes, as well as support for multiple active instances of Samba in a cluster which improves overall throughput and increases availability for large Samba clustered deployments."

Comments (52 posted)

Ubuntu's Precise Pangolin Alpha 1 Released

The first alpha for Ubuntu 12.04 is available for testing. Images are currently available for Ubuntu Desktop, Server, ARM, Server Cloud and EC2, as well as Xubuntu, Edubuntu and Lubuntu.

Full Story (comments: 9)

Distribution News

Debian GNU/Linux

Two month advance notification for upcoming end-of-life for Debian oldstable

Debian Lenny (5.0) will reach its end of supported life February 6, 2012. "The Debian project released Debian GNU/Linux 6.0 alias "squeeze" on the 6th of February 2011. Users and distributors have been given a one-year timeframe to upgrade their old installations to the current stable release. Hence, the security support for the old release of 5.0 is going to end on the 6th of February 2012 as previously announced."

Full Story (comments: none)

Fedora

Election Results: Fedora Board, FAmSCo, and FESCo

The latest round of Fedora elections is over. Jaroslav Reznik and Christoph Wickert were elected to the Fedora Board. Christoph Wickert, Neville A. Cross, Igor Soares, Clint Savage, Zoltan Hoppar, Gerard Braad, and Caius Chance have been elected to the Fedora Ambassadors Steering Committee (FamSCo). Marcela Mašláňová, Miloslav Trmač, Matthew Garrett, and Jon Ciesla has been elected to the Fedora Engineering Steering Committe (FESCo).

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• DistroWatch Weekly, Issue 434 (December 5)
• openSUSE Weekly News, Issue 204 (December 4)
• Ubuntu Weekly Newsletter, Issue 244 (December 4)

Comments (none posted)

End of life approaching for RHEL, CentOS and Scientific Linux 4 (The H)

The H notes the End of life security support for Red Hat 4, along with its clones; CentOS and Scientific Linux.

Comments (none posted)

Sabily Linux review (TechWorld)

TechWorld has a review of Sabily Linux, an Ubuntu derivative aimed at the Muslim community. ""I launched the Sabily project when I discovered Ubuntu (version 7.04), and found the Ubuntu Christian edition," Sabily administrator Mehdi Magnon told Techworld Australia. "I thought it would be a great idea to provide Muslims with a Linux distribution with many Islamic applications installed, so they don't have to install them manually, as well as a more Islamic look and feel.""

Comments (none posted)

Page editor: Rebecca Sobol

Development

A proposal for a LibreOffice UI overhaul

The Document Foundation split from the OpenOffice.org project just over one year ago, and has merged new features into LibreOffice, as well as cleaned up the code base and project infrastructure. The next milestone release, version 3.5.0, is in beta testing, and things appear to be running smoothly. To some contributors, then, now is the right time to revisit the office suite's user interface — an interface that can be complicated, given the scope of the components the suite includes.

Volunteer Miroslav Mazel has attracted significant attention with his series of UI mock-ups, which he calls "Citrus." Mazel was active during OpenOffice.org's Project Renaissance brainstorming effort in 2009 and 2010, submitting a detailed proposal, and is now part of the LibreOffice Design Team. Nevertheless, the Citrus designs are his personal work, not an official product of the Design Team. Consequently, they face a long road ahead before they would reach the actual LibreOffice repositories.

The Citrus work builds on many of the same ideas in Mazel's earlier OpenOffice.org proposal, most notably the removal of dialog boxes wherever possible, the use of contextual menus, an innovative way of color-coding buttons (and other UI elements) by their intended use, and moving key functionality to a sidebar that sits next to the open document. But while that proposal was primarily a text-driven description of concepts, Mazel has been creating mock-up images for Citrus, and blogging about them frequently.

Peeling back the layers

The Citrus idea has morphed over the months that Mazel has been working on it. At present, he breaks the ideas into five main categories: reorganizing the commands, adding "inline" controls, integrating the various navigational tools, adding support for open font repositories, and simplifying the Draw vector graphics application. In addition, all of the mock-ups sport a minimalist, light-gray widget theme that represents a distinct departure from the current LibreOffice look-and-feel.

The command reorganization is the most substantial set of proposed changes. It involves breaking all of the commands into groups based on what part of the document editing process they affect — e.g., application-wide functions, document-centric functions, and separate categories for altering document display, inserting content, or operating on a selected object. The menus then change content depending on the state of application: if the user selects an image, an "Image" menu appears; if they select several items, a "Group" menu appears. At no point are inactive menu commands visible. Mazel also suggests regrouping the toolbar buttons to match the menu layout, and making sure that all functions are accessible from a menu, a toolbar button, and a keyboard shortcut.

The inline controls changes center around a "floatbar" — a small, floating toolbar that appears just above a selected object. Like the menus and toolbar, it is context-dependent, so when text is selected, only font and style functions appear in the floatbar. He also proposes adding a "new page" button to the bottom of every document (where it would be closer to the cursor when the user is editing the last existing page), and using "handles" to select all objects — including full pages, which must be selected in order to change page-specific settings.

The "navigator" is a sidebar that integrates the Find/Replace search box, the built-in Help search tool, and an in-document browser that allows the user to jump between pages, slides, or document sections. This feature is akin to the slide thumbnail browser already used in the LibreOffice Impress presentation tool, but is also similar to document sections in Writer and analogous navigation aids in the other applications. It is also intended to hold some of the selection tools, so that there is one place to go to perform a "select all" on a particular content type (image, table, etc.).

The font integration proposes two changes: a reworked Font drop-down menu, and under-the-hood support for loading and using remote fonts from open font repositories like the Open Font Library or League of Movable Type. Mazel proposes that LibreOffice support transparent fetching of fonts from these remote sites, creating the notion of "supported" fonts that are accessible to all LibreOffice users even if they are not locally installed. He recommends showing only supported fonts in the font selector by default, as well as sorting the fonts into categories (serif, sans-serif, monospaced, handwriting, and "display" fonts).

Finally, the drawing changes include adding an "insertbar" (a vertical toolbar docked on the left-hand side of the window) that holds buttons for inserting various objects into the document, a layer-management tool for the sidebar, and a pop-up color picker.

Sprinkled throughout the mock-up images are an assortment of notes that point to other features, such as the monochrome, indicator-style button icons, and the color-coding of objects by their function. The image used to illustrate the layers tool, for instance, also describes the color-coding scheme. Red icons are used for animated and video content, orange for images, yellow for vector shapes, green for tables and data cells, blue for text, and purple for audio. Various shades of each color make subtle distinctions between related element types, such as selected words and selected paragraphs.

Response

Mazel's mock-ups have generated a lot of praise on Linux news-discussion blogs in recent weeks, although the positive feedback focuses largely on the look and feel of the widget set used, rather than the more functional proposals. Within the LibreOffice project, most of the debate around Citrus has come on the libreoffice-design list, where the Design Team and interested community members regularly discuss and refine user experience (UX) issues.

On the list, the attention from various blogs sparked several discussions in mid-to-late November. The mock-ups have attracted both ardent supporters and critics with detailed concerns. Stefan Knorr pointed out several issues related to the command reorganization. The strict-ordering of commands in the toolbar is problematic because LibreOffice users have long been able to customize their toolbars, he said. It also diverges from the decades-old menu structure other office applications use, in ways that may confuse users. Knorr went on to say that the menu reorganization "is throwing (useful) conventions out of the window: should we really move Cut, Copy and Paste into completely different menus?"

Knorr did not care for the insertbar concept, but Kévin Peignot liked it, as well as the floatbar. However, he too disagreed with the menu reorganization, saying that "the current menus, even if not perfect are great." He also observed that color-coding UI elements can have unintended consequences, such as the fact that red is generally used for alarms and other "attention needed" urgent messages. Knorr also pointed out that color-coding icons could put color-blind users at a disadvantage.

Commenting on a similar UX proposal from another user, Design Team lead Christoph Noack observed that context hierarchies are very tricky to get right for all use cases, and that LibreOffice must cope with additional challenges like cross-module functionality and extensions. Opinion on the design list is divided on many issues, particularly the menu restructuring and context-dependent functions, but on a lot of minor changes, too. Some individual elements fared particularly well, such as the "add page" button, but ultimately, no consensus was reached.

Peignot floated the idea of a user survey to gauge broader support for many of the individual changes, but Björn Balazs (a psychologist with experience in user surveys) cautioned against using such a survey to measure acceptance for a new UX:

Balazs also warned that the discussion around Citrus needed to be grounded in the fact that the mock-ups came from outside the official project, and "is not on the agenda of any LO-developer." That stance is borne out by comments on the developers' list, where Michael Meeks said:

He also expressed concern that mandating a Citrus-like interface risks alienating enterprise and power users.

From mock-up to code

Balazs' comments about the future of the proposed changes deflated some of the more enthusiastic supporters, but he added support for the continued development of the ideas, saying: "We will get to the point where this is constructive - but at the moment it is just the opposite of it. If anyone wants to do something really [helpful]: try to find and document the small improvements that are possible to make LO rock even more." One such supporter, Andrew Pullins, then asked how to proceed getting the proposals into the LibreOffice roadmap.

The Design Team's Charles H. Schulz suggested that, for starters, the mock-ups and blog posts be developed into more detailed, feature-by-feature specifications, noting that "the developers cannot implement a complete redesign all at once. Therefore you need to push changes one by one. Little by little." Michel Renon added that the proposed changes should use the Design Team's whiteboard section, which already hosts a long list of UI and UX proposals, and recommended that the mock-up images be re-done using wireframe graphics, so that the aesthetics of the theme do not distract from the UX discussion.

Renon emphasized that while he has nothing against the Citrus mock-ups or UX proposals, it is important for the project to make it clear to the public that Mazel's work does not represent an official or approved direction for the LibreOffice user interface. "I would propose to use a neutral form: "future UI". So let's make a lot of one-step proposals to reach one day the "future UI"!"

On December 5, Mazel began to draft more detailed specifications for his proposed changes in the whiteboard section of the wiki. They indeed focus on small changes, initially covering only the "add page" button, insertbar, a full-page-select button, and an "overflow" button to capture toolbar buttons that do not fit on screen. Incremental changes to the code may not be as exciting to look at as full-blown UI renovations, and the path to ultimate adoption by the project is still long, but the Citrus mock-ups are finally beginning to make their way into the project's workflow. As the public discussions over Citrus show, many users and project contributors seem to agree that LibreOffice is in need of a UI and UX refresh. What it will end up looking like, however, won't be known for some time to come.

Comments (2 posted)

Brief items

Quotes of the week

-- Colin Walters

-- Bastien Nocera

Comments (3 posted)

Arduino 1.0 released

Version 1.0 of the Arduino development environment has been announced. "The language changes include modifications to the Serial class, addition of DHCP and DNS support to the Ethernet library, a new SoftwareSerial library, multi-file support in the SD library, modifications to the Wire library and UDP class, etc." Details can be found in this blog post.

Comments (none posted)

Buildroot 2011.11 released

Version 2011.11 of the Buildroot embedded system creation tool is out. Enhancements include support for local packages, improved documentation, Xenomai and RTAI support, and a number of new packages.

Full Story (comments: none)

extensions.gnome.org launches

The "public alpha" version of extensions.gnome.org is now available for those who wish to fiddle with their GNOME shell experience. "If you have GNOME Shell 3.2 on your system, you should be able to install extensions from the website via your browser. This uses the 'GNOME Shell Integration' browser plugin which is likely already installed on your system if you have GNOME 3.2." The number of extensions on the site is relatively small, but is expected to grow in the near future.

Full Story (comments: 52)

GNUnet 0.9.0

GNUnet is "a framework for secure peer-to-peer networking. GNUnet's primary design goals are to protect the privacy of its users and to guard itself against attacks or abuse." Enhancements in the 0.9.0 release include a new virtual private network application, ARM support, mesh routing, HTTPS and WLAN transports, and more.

Full Story (comments: none)

LLVM 3.0 released

Version 3.0 of the LLVM compiler suite is out. "Some of the bigger leaps include a new register allocator (which can provide substantial performance improvements in generated code), full support for atomic operations and the new C++ memory model, major improvement in the MIPS backend, and support for gprof/gcov style of profiling information." See the release notes and the Clang release notes for more information.

Full Story (comments: 32)

PostgreSQL 2011-12-05 Update Release

The PostgreSQL project has released versions 9.1.2, 9.0.6, 8.4.10, 8.3.17 and 8.2.23 with fixes for a number of important bugs, many of which are applicable to users of the binary replication feature. This is the final update for version 8.2.x, which is now at end-of-life.

Full Story (comments: none)

QEMU 1.0 released

After eight years of development, version 1.0 of the QEMU system emulator is out. There's lots of important changes; see the changelog in the announcement for details. "QEMU now uses a separate thread for VCPU execution. This merges the biggest difference between the qemu-kvm tree and upstream QEMU."

Full Story (comments: 27)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (December 6)
• GCC 4.7.0 status report (December 6)
• LibreOffice development summary (December 5)
• Perl Weekly (December 5)

Comments (none posted)

Evolution of shells in Linux (developerWorks)

This developerWorks article looks at the history of Unix shells over time. "Ken Thompson (of Bell Labs) developed the first shell for UNIX called the V6 shell in 1971. Similar to its predecessor in Multics, this shell (/bin/sh) was an independent user program that executed outside of the kernel. Concepts like globbing (pattern matching for parameter expansion, such as *.txt) were implemented in a separate utility called glob, as was the if command to evaluate conditional expressions. This separation kept the shell small, at under 900 lines of C source."

Comments (195 posted)

Page editor: Jonathan Corbet

Announcements

Brief items

SFLC, EFF seek DMCA exemptions

It's that time in the three-year cycle of the US Digital Millennium Copyright Act where interested parties can petition for exemptions to its anti-circumvention provisions. The Software Freedom Law Center has put out a press release on its request for an exemption allowing owners of personal computing devices to install any software they choose on those devices. The Electronic Frontier Foundation, instead, is asking for an extended right to jailbreak devices, once again, so that different software can be installed.

Comments (14 posted)

New Books

Two New Web Design Books from SitePoint

SitePoint has released two new books on web site design; "The WordPress Anthology" by Mick Olinik and Raena Jackson Armitage, and "PHP Master: Write Cutting Edge Code" by Davey Shafik, Lorna Mitchell and Matthew Turland.

Full Story (comments: none)

Upcoming Events

Final keynote for linux.conf.au 2012 to be given by Jacob Appelbaum

The final keynote speaker at the 2012 linux.conf.au (LCA) will be Jacob Appelbaum, an "independent internet security professional, accomplished photographer, software hacker and world traveller." LCA 2012 will take place January 16-20, 2012 in Ballarat, Australia.

Full Story (comments: none)

SCALE: The Next Generation Invitation

The Southern California Linux Expo has announced a conference for the next generation of free and open source (FOSS) community enthusiasts. SCALE: The Next Generation will be held January 21, 2012, during the SCALE conference in Los Angles, California.

Full Story (comments: none)

Events: December 8, 2011 to February 6, 2012

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
December 4 December 9	LISA ’11: 25th Large Installation System Administration Conference	Boston, MA, USA
December 27 December 30	28th Chaos Communication Congress	Berlin, Germany
January 12 January 13	Open Source World Conference 2012	Granada, Spain
January 13 January 15	Fedora User and Developer Conference, North America	Blacksburg, VA, USA
January 16 January 20	linux.conf.au 2012	Ballarat, Australia
January 20 January 22	Wikipedia & MediaWiki hackathon & workshops	San Francisco, CA, USA
January 20 January 22	SCALE 10x - Southern California Linux Expo	Los Angeles, CA, USA
January 27 January 29	DebianMed Meeting Southport2012	Southport, UK
January 31 February 2	Ubuntu Developer Week	#ubuntu-classroom, irc.freenode.net
February 4 February 5	Free and Open Source Developers Meeting	Brussels, Belgium

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol