LWN.net Weekly Edition for June 18, 2009

FreedomHEC Taipei 2009

FreedomHEC (Freedom Hardware Engineer's Conference) Taipei was held June 10 and 11 in, unsurprisingly, Taipei, Taiwan. The event, sponsored by the governmental Institute for Information Industry, followed the huge Computex conference in the hope of attracting hardware developers who are interested in supporting Linux. Your editor, who had the honor of being invited to speak at FreedomHEC Taipei, would assert that the goal was achieved; over 200 developers showed up for two days of technical talk about the Linux kernel and the best ways to contribute to it.

Your editor arrived prepared with the talks which have become his stock in trade in recent years. Of the two talks, the most fun was certainly had on the second day, when the topic was how to work with the kernel development process. Anybody who has given talks in eastern Asia knows that getting audience members to participate and ask questions is not an easy task. But the FreedomHEC audience was (after just a bit of encouragement) full of questions and interested in having a discussion which could have extended far beyond the allotted time. There was, beyond doubt, a great deal of interest in working more closely with the wider development community. Your editor offers his apologies to Greg Kroah-Hartman, who had to wait rather longer than was proper for the lectern to become free.

Greg's talks were quite well received; his 2.5-hour tutorial on writing a device driver had the undivided attention of the full audience at the end of a long day. Peter Stuge gave a talk on CoreBoot (once known as LinuxBIOS). Harald Welte's talk on GPL compliance was also well received; the audience seemed much more interested in how to work with the GPL than how to get around it.

FreedomHEC was not dominated by outside speakers, though. Developers Jim Huang and Matt Hsu from 0xlab, who were clearly having a great time, discussed their work with the Qi bootloader. Fred Chien covered a number of hacks ("dirty" and otherwise) to make Linux boot more quickly. Joseph Chan of VIA discussed the various challenges he encountered while working to merge a driver into the mainline. They all appeared to be interesting talks; unfortunately for LWN readers, they were also all in Chinese, so your editor's reporting is necessarily spotty.

The Chinese-language talk which looked like the most fun was a high-energy session led by Chen Ing Hau, perhaps best known as the (since repentant) author of the Chernobyl worm. CIH (as he was referred to there) went deeply into the process of reverse engineering the appropriate register settings to drive a wireless network adapter. The talk proceeded with all the ups and downs of a good detective story, with the culprit, naturally, being caught at the end.

There is a certain missionary aspect to attending this kind of event. Harald Welte described it this way:

The group of industries often referred to as "Taiwan, Inc." has often shown a lack of understanding of how our community operates and, seemingly, a lack of interest in being a part of it. So FreedomHEC looks like an attempt to bring the Good Word to that part of the world. Certainly some people need to hear it. But your editor is not sure that those people were at this conference. What went down at FreedomHEC was hackers talking to other hackers, and they were having quite a good time in the process. It looked an awful lot like a Linux-oriented development conference.

Your editor has spoken in parts of the world where people seem to view software development as just another job, and not a particularly inspiring one at that. But Taiwan was not one of them. There are a lot of people working on interesting projects, and they show that excited, creative spark that suggests good things are on the way. They look like they want to be a bigger part of our community.

To that end, events which present information on how to work with the global community are helpful. But one thing is worth pondering on. When asked if they felt that their managers understood the importance of and value in working with the free software community, very few members of the audience raised their hands. Taiwanese developers, it seems, don't need much help in this area, but their bosses do. This is natural; why should Taiwanese managers be more enlightened than their counterparts elsewhere in the world? To solve this problem, we are going to have to engage in education at higher corporate levels. Management can be won over with sufficient persistence and energy - most of the time.

Events like FreedomHEC are helpful in this regard; they can give the local development community encouragement and the tools needed to carry the message to their companies. And there is always value in gatherings where developers can meet and talk about things over beer. FreedomHEC Taipei 2009 was an exemplary event of its kind; its organizers (led by Chao Lung Huang) deserve strong congratulations. Your jet-lagged editor is happy to have gone.

Comments (6 posted)

Python and ipaddr.py

On June 4th, the Python developers removed a recently-added module, ipaddr.py, from the upcoming Python 3.1 release. At the time 3.1 was still in release candidate status — rc1 had been released on May 30th — and the usual policy for Python's release candidates is that only bug fixes are permitted. Adding or removing an entire feature is normally forbidden. Why did the Python developers change their minds? The story behind the decision shows a possible communication problem as projects grow and the number of available channels increases.

The story begins in September 2008 with Issue 3959 being filed in Python's bug tracker by Guido van Rossum, recording a suggestion that ipaddr.py be added to the standard library. After the issue was created a few messages were exchanged discussing the authors' willingness to move the master copy into Python's SVN repository, and then the issue fell silent. (The Python developers were probably distracted by preparations for the final release of Python 2.6 that occurred a week later.)

ipaddr.py parses IPv4 numeric addresses such as 172.16.40.35 and address ranges such as 192.168.0.0/16, as well as the corresponding IPv6 formats. Users can create IPv4 and IPv6 address ranges, remove a subset of addresses from a range, check whether a given address is included in a range, and compare addresses and networks for equivalence. The module had been used internally at Google for some time and bears a copyright statement dated 2007; the first open-source release was in September 2008 as the ipaddr-py project on Google Code.

After the initial messages, it was some months before discussion revived. Some commenters noted similar libraries such as netaddr and IPy. After some further discussion, in February 2009 Martin von Löwis wrote "Yes, I think it can be integrated now.". There was another delay of a few months before the code was actually committed, but on May 1st Gregory P. Smith committed ipaddr.py to the Python 2.7 branch in SVN revision 72173, also merging the change to the Python 3.1 branch.

But, in June Clay McClure argued against the inclusion of ipaddr.py on the bug tracker:

McClure's objection stems from ipaddr.py using the same class to represent both individual IP addresses and network address ranges. Network ranges are usually represented as a starting address and the number of bits that are the fixed portion of addresses within the range. 192.168.0.0/16 is an IPv4 range, for example; the first 16 bits of the address (the 192.168. portion) are fixed and the remaining bits can vary. ipaddr.py represents a IPv4 host's address, which is a specific 32-bit value, as the address plus a /32 value. Treating addresses in this way lets a single class be used to represent either a single address or a range. (IPv6 uses 128-bit addresses and hex notation, but the principles are the same.) The debate is about whether this representation is reasonable.

    >>> from ipaddr import *
    >>> a1 = IPv4('127.0.0.1')
    >>> a1  
    IPv4('127.0.0.1/32')
    >>> a2 = IPv4('127.0.0.1/32')
    >>> a1 == a2  
    True

Core contributor Raymond Hettinger brought the discussion to the attention of the python-dev mailing list:

van Rossum initially thought the reported flaws were not so significant:

The resulting discussion thread received dozens of messages, and a consensus formed that the module's interface needed some reworking. Eventually van Rossum agreed it should just be removed from 3.1:

To a first approximation, this is just business as usual for any project: a feature was proposed, the code got committed, but later the developers collectively changed their minds and backed out the change from the 3.1 branch. The impact is minimal, because the code was only in one release tarball, a short-lived 3.1rc1 release candidate that was replaced two weeks later by 3.1rc2. The module remains in the 2.7 branch for now, but 2.7's release is months away, leaving plenty of time to remove the module or to rework the module's API and include it in Python 3.2. A Python Enhancement Proposal may be written defining an IP-address module.

However, the incident brought some process issues to attention, and the discussion divided into three major strands.

Mailing list vs. bug tracker

Part of the reason for the late reversal was that objections were being made through commenting on a particular issue in the bug tracker. A January thread on python-dev didn't cause much discussion, and the disagreement on the issue wasn't noticed by 3.1 release manager Benjamin Peterson.

Glyph Lefkowitz of the Twisted project detailed the procedures for the Twisted project, where all discussions are in the bug tracker:

Bug trackers encourage focused conversations between people with an interest in a bug or patch, preventing an unmanageably busy mailing list. But there's a cost; this increased focus makes it harder to obtain a general view, or to notice that one issue is particularly serious or important. One suggested improvement was adding the python-dev mailing list to the Roundup bug tracker as a user. Roundup creates a 'nosy list' for each issue; users on the nosy list receive e-mail notifications of new comments or modifications to the issue. If python-dev existed as a Roundup user, adding python-dev to the nosy list for an issue would send a clear signal to other core developers.

Unclear decision-making

The developers of Apache have a formal voting process that's specified in careful detail. Python follows some of Apache's conventions, such as using +1 and -1 to indicate support or opposition, but these aren't votes, merely polls showing general opinions. For example, under Apache's rules a -1 vote is a veto and must be accompanied by a technical justification, but many changes to Python have received -1 votes and still been accepted.

In Lefkowitz's posting cited earlier, he also noted that there is a vocabulary for describing the next action to take on a patch:

Selecting additions to the standard library

A number of different modules for representing IP addresses have been written, including ipaddr.py. David P. D. Moss, author of the netaddr module, drew up a list of IP-address modules that contains 12 projects, the oldest dating back to 2000. ipaddr.py is comparatively new, having been publicly released in September 2008; so why was it chosen?

To some degree it's just a question of luck and timing: the suggestion to add ipaddr.py was just the first one to be recorded in the bug tracker, but a policy of "whoever asks first" does not ensure that the best module is chosen for addition.

What selection process would work better? Barry Warsaw thought usage information from the package index could be helpful, suggesting "It would be really nice if say the Cheeseshop had a voting feature". The Cheeseshop is another name for Python's package index. Neil Schemenauer further noted Debian's popularity-contest script, which reports the packages installed on a system to a central server.

van Rossum didn't like the idea at all, saying "Whoa. Are you all suddenly trying to turn Python into a democracy? I'm outta here if that ever happens (even if I were voted BDFL by a majority :-).".

But van Rossum had previously noted that he wasn't a likely user of the module and could not really assess the suitability of the module's interface. In the absence of a strong opinion based on technical grounds, how should such decisions be made? Going by popularity seems to be as reasonable a way to choose as any other. Of course, the sheer number of modules may imply that this is an area where people are finicky about the approach taken, and perhaps the standard library needn't take a position on it.

Conclusion

In order to make progress, open-source projects, like people, must continuously decide which action to take next. Much has been written about personal time management, including books such as Stephen R. Covey's 7 Habits of Effective People and David Allen's Getting Things Done. Often the key to improvement is taking a careful inventory of tasks and organizing this inventory to minimize effort spent figuring out what to do and maximize effort spent actually implementing or fixing.

As free software projects spread out across multiple communication channels — a set of mailing lists, the bug tracker, SourceForge forums, personal weblogs, IRC conversations, Twitter, and soon enough new technologies such as Google Wave — their developers will face many of the same issues at a collective level. Taking inventory is relatively easy because there are many tools for doing so: version-control systems, bug trackers, and mailing list archives. But how do you ensure that the right people are paying attention?

Comments (10 posted)

openSUSE readies for community contributions

In the past, only Novell employees could contribute code to Factory, the development version of openSUSE. However, this policy has started to change in the last few weeks, ever since openSUSE board member Henne Vogelsang announced sweeping changes to how Factory operates. These changes include a reorganization of Factory into specialized teams, and permitting community members to contribute code directly — a move that has the distribution rethinking its security measures as well.

The opening of the distribution to community contributions is only one of the changes to the distribution in recent years to make it more open (and, perhaps, to silence accusations that openSUSE is controlled entirely by Novell). These changes have included the appointment of a community manager, the transition to an elected board, and increasing access to the project through such features as openFATE, the feature-tracking service, and the openSUSE Build Service.

Joe "Zonker" Brockmeier, openSUSE community manager, explained that the changes were made "to continue enabling the contributor community outside of Novell and make it easier for people who are doing work on openSUSE to participate directly. We want to enable and encourage contributions from outside of Novell, and continue to make openSUSE a more independent project."

Similarly, Vogelsang said that most of the requirements for open contribution have been gradually coming into place over the last year or so, but "What we did not have was the possibility to take responsibility. That's why we changed our policies now so people can not only contribute but also take responsibility."

Vogelsang is referring to the division of Factory into working teams for patches and packaging. To a limited extent, this division already exists. Informally, teams for major areas such as GNOME, KDE, and Education already have their own web pages and ways of working, and exist semi-autonomously within openSUSE.

The difference now, as Vogelsang said in his initial announcement is that "What we want now is to use this model for every aspect of the distribution, not only the big parts where this comes naturally."

To this end, Vogelsang and the current package maintainers have developed a preliminary list of over 85 possible teams. These teams are listed as having responsibility for maintaining packages and reviewing contributions for as few as three or four packages, or for dozens, depending on the area that they cover.

In the announcement, Vogelsang wrote that, "As our goal is to make it possible for everybody to work on the distribution, we will simply allow the topic groups to organize themselves. We think that there is not the one and only way such a group of maintainers can work together, but it depends on the involved people and also on the topic . . . . So to transition we will throw together the people that are currently maintainers of packages and help them work out how to best collaborate with each other and [with] new people."

What is not immediately clear (at least, not without a lot of web searches) is which of the proposed teams are active and which are still largely hypothetical. But, for now, the distribution will be working with existing maintainers, and adding Novell employees to fill in where volunteers are lacking.

Policy vs. procedure

A particular area of concern is security, as Marcus Meissner pointed out. As Meissner explained, "Every single package can disturb / destroy the integrity of the whole openSUSE distribution (As in 'install trojans, rootkits, etc.')." Even if a package is not installed by default, Meissner warned, it could still be installed by the automatic resolution of dependencies.

Under these circumstances, Meissner stresses that the role of packagers needs to change along with openSUSE's policies. In particular, packagers will need to review all submissions with extreme caution, checking for malicious code and maintaining a workable set of project and package submission. They will need to decide how much they trust each contributor, and, at the same time, watch for email addresses that spoof those of trusted contributors, or cracked openSUSE accounts.

"Always keep such a thing in the back of your head and watch for behavioural changes," Meissner advised. "Trust your gut feeling and check twice if in doubt!"

Meissner's warning sparked a brief discussion on how the submission process could be made more secure. Karsten König suggested that the use of MD5 of SHA-1 hashes become mandatory in all submitted tarballs and within package scripts, so that reviewing the integrity of submissions would become easier. König also proposed that all hashes for packages be available on a web interface. Such changes in procedure would probably be resisted by some maintainers, but Vincent Untz wrote that they might be accepted more easily if they were made voluntary for one release cycle, then mandatory for the next.

Allowing community changes is a reversal of existing policy that can only add to the credibility of openSUSE as a free software project. However, as the security issue shows, changes in policy can require rethinking of procedures as well. While no final decision has been announced about how to handle the security issues created by the policy change, the need for some guidelines is inevitable. Nor are policies for acceptable structures or standards likely to be far behind, as other community distributions have discovered in the past. The devil, openSUSE is finding out, is not in the policies so much as in thrashing out the details.

Comments (none posted)

Page editor: Jake Edge

Security

Dealing with weakness in SHA-1

The SHA-1 hash function serves an auxiliary role in a number of cryptography utilities, notably OpenPGP, where it is used to sign documents and generate key fingerprints. Researchers recently published an attack on SHA-1 that can find collisions in drastically shorter time than previously thought, accelerating the move to replace SHA-1. A contest is underway to select a replacement, that will be designated SHA-3, but it will not be standardized until 2012. Between now and then, there are several steps interested individuals can take to harden themselves against attack — starting with understanding just what a hash collision can and cannot compromise.

SHA-1 was created by the National Security Agency (NSA) in 1995. It computes a 160-bit hash or "digest" of any message less than 2^64 bits long. Like any cryptographic hash function, its value as a message authentication tool depends on it being mathematically hard to find a collision: two messages that generate the same hash value. A brute-force search would, on average, take 2^80 evaluations of the function to find a collision (80 being half of the 160 bit digest length). Such a search would find two arbitrary messages that result in the same hash, not allow an attacker to find a collision with any specific message, but the 2^80 steps of a brute-force attack serves as a metric for the comparative efficiency of other attacks.

In recent years, the most efficient known attack on SHA-1 required 2^63 evaluations — around 1/100,000th the number of steps for a brute force search, but still safely outside the reach of a real-world attacker. That changed in April of 2009, when Cameron McDonald, Josef Pieprzyk, and Phil Hawkes presented findings [PDF] at the Eurocrypt 2009 conference that lowered the bar to 2^52 — a 2,000-fold speedup over 2^63. The existence of such an attack is far from a crisis-level weakness, but the upshot is that it is better to start migrating away from SHA-1 while it is still relatively safe.

SHA-1 in free software cryptography

SHA-1 is used in public-key cryptographic systems, including the OpenPGP specification (RFC 4880) implemented on most Linux desktop and server distributions by GnuPG. Since SHA-1 is a hash function, and not a cipher, it does not play a direct role in encryption, but it is used for digital signatures. In addition, OpenPGP key fingerprints are created with SHA-1, and key fingerprints are in turn used in key revocation and modification detection codes (MDC).

An OpenPGP digital signature involves computing a hash of the original message, then encrypting the hash with the signer's private encryption key. To verify the signed message's integrity, the recipient also needs to be able to compute the same hash on the received text. That requires support from the software and the keys used — although OpenPGP supports multiple hash algorithms in addition to SHA-1, old DSA keys can only use 160-bit signatures. Historically, that meant SHA-1, although RIPEMD-160 is compatible as well. Consequently, selecting a stronger algorithm when signing messages is possible with an application like GnuPG, but in the worst case scenario a user wishing to avoid SHA-1 would need to create a new DSA2 or RSA signing key.

Key fingerprints are digests of public keys, useful for key management because they are considerably shorter than the key from which they are hashed — thus making them human-readable so they can be compactly referenced more easily. Applications often list keys in a user's keyring by their fingerprint, so a SHA-1 collision that results in two keys having the same fingerprint could cause user confusion or unpredictable application behavior. Perhaps more importantly, key revocation certificates reference keys using fingerprints, again opening the door to unpredictable behavior if the application finds two keys with identical fingerprints. OpenPGP specifies SHA-1 as the only hash algorithm for version 4 keys (the latest revision), so there is no current workaround for fingerprint collisions.

MDC is an OpenPGP system to provide message integrity-checking with less overhead and less stringent requirements than full digital signatures. RFC 4880 describes it as "analogous to a checksum." MDC also specifies SHA-1 as its sole hash algorithm, but because its modest goals cover message integrity but not authentication, the existence of collisions does not adversely affect it. The checksum-like usage of the hash algorithm in this context simply verifies that the message content was not altered or corrupted in transit.

Although 2^52-evaluation collisions represent a significant weakening of SHA-1, it is important to note that hash collisions are not as easy to exploit as broken ciphers. On the GnuPG users' mailing list, maintainer David Shaw evaluated some of the possible scenarios, such as attempting to forge a signature. Even with the easier-to-exploit MD5 collision problem, thus far no one has been able to create a phony signature to match the signature of an existing key; the closest anyone has come is to generate two keys that can be used to create the same signature — an attack with little practical value. The prevailing opinion on the IETF's OpenPGP Working Group list was much the same. A more likely problem is the unexpected behavior of applications when confronted with fingerprint collisions.

Practical migration and looking forward

Nevertheless, users are encouraged to transition away from SHA-1 usage to stronger hash algorithms. The US government has mandated deprecation of SHA-1 for its use by the end of 2010. There are several alternative hash functions available today, including the family known as SHA-2. SHA-2 includes several functions that are related but use different digest lengths: SHA-224, SHA-256, SHA-384, and SHA-512. The SHA-2 functions are algorithmically similar to SHA-1, and so would be vulnerable to the same type of attacks, but because of their larger digest size they remain significantly more secure.

The National Institute of Standards and Technology (NIST) is currently holding a competition to select a next-generation hash algorithm to be designated SHA-3. Submissions were due in October of 2008, and the final winner is expected to be announced in 2012.

Debian's Daniel Kahn Gillmor posted a step-by-step guide to migrating away from SHA-1 in GnuPG. Included are instructions for setting up signing algorithm preferences in gpg.conf, attaching digest preferences to a public key so that other users will select a stronger algorithm when sending a message, and generating a replacement for an old DSA key. It is an important read particularly for key replacement, because setting strong digest preferences must be done before generating a new key — otherwise GnuPG will default to using SHA-1.

GnuPG for its part is planning to change its defaults in future releases, so that newly-created keys will default to RSA instead of DSA, and be able to use newer hash algorithms. Gillmor has also proposed a tool to scan OpenPGP keys and offer suggestions to the user for strengthening them — including using the current format, key type and size, appropriate sub-keys, and several other parameters.

The trickier problem is the OpenPGP specification's inclusion of SHA-1 as the "hardwired" choice for fingerprinting, revocation, and MDC. A thread on the OpenPGP Working Group's mailing list exposes several points of view. Some think that the group should wait for SHA-3, some think a change is due now, and others think that hash collisions even on fingerprints are not a significant enough security risk to warrant changing the specification.

As Gillmor's migration guide indicates, Debian is trying migrate its developers, maintainers, and teams away from SHA-1 digests and DSA keys and towards RSA keys with SHA-512 digests. Likewise, the Fedora project has undertaken a concerted migration to SHA-2 hashes. Ubuntu's security team administrator Kees Cook says that that distribution will update its keys over time, but that there is no rush. OpenSUSE's Marcus Meissner echos that sentiment, observing that the distribution is phasing out SHA-1 and MD5 for signing, but that collisions do not constitute a security threat for simple download integrity checking. All four distributions already use RSA master keys to sign packages.

Shaw emphasized that the recent attacks on SHA-1 still require a significant amount of work, and at best would allow an attacker to produce two original documents that hash to the same value, which does not directly impact most people's usage of OpenPGP. "This is not an attack where someone could take an existing OpenPGP-signed document and make a new document that matches the signature or the like." He advised individuals and maintainers who know that their intended recipients can accept larger hashes to use larger hashes, particularly when signing documents created by someone else (such as at a key-signing event), but not to worry unduly about using SHA-1 when that is the only option. In other words, he said, walk, but don't run, for the exits.

Comments (11 posted)

Brief items

Researchers Build Anonymous, Browser-Based 'Darknet' (DarkReading)

DarkReading reports on an upcoming Black Hat presentation which creates a "darknet" from within browsers. A darknet is a network like Tor or Freenet that allows for secure, anonymous use. "Veiled is basically a 'zero footprint' network, in which groups can rapidly form and disappear without a trace. It connects the user's HTML 5-based browser to a single PHP file, which downloads some JavaScript code into the browser. Pieces of the file are spread among the members of the Veiled darknet. It's not peer-to-peer, but rather a chain of 'repeaters' of the PHP file, the researchers say."

Comments (none posted)

New vulnerabilities

acroread: multiple vulnerabilities

Package(s):	acroread	CVE #(s):	CVE-2009-0198 CVE-2009-0509 CVE-2009-0510 CVE-2009-0511 CVE-2009-0512 CVE-2009-0888 CVE-2009-0889 CVE-2009-1855 CVE-2009-1856 CVE-2009-1857 CVE-2009-1858 CVE-2009-1859 CVE-2009-1861 CVE-2009-2028
Created:	June 17, 2009	Updated:	September 1, 2009
Description:	acroread has a number of vulnerabilities. From the Red Hat alert: Multiple security flaws were discovered in Adobe Reader. A specially crafted PDF file could cause Adobe Reader to crash or, potentially, execute arbitrary code as the user running Adobe Reader when opened. (CVE-2009-0198, CVE-2009-0509, CVE-2009-0510, CVE-2009-0511, CVE-2009-0512, CVE-2009-0888, CVE-2009-0889, CVE-2009-1855, CVE-2009-1856, CVE-2009-1857, CVE-2009-1858, CVE-2009-1859, CVE-2009-1861, CVE-2009-2028)
Alerts:	SuSE SUSE-SR:2009:014 2009-09-01 Gentoo 200907-06 2009-07-12 SuSE SUSE-SR:2009:012 2009-07-03 Red Hat RHSA-2009:1109-01 2009-06-17

Comments (none posted)

coccinelle: symlink attack

Package(s):	coccinelle	CVE #(s):	CVE-2009-1753
Created:	June 16, 2009	Updated:	June 17, 2009
Description:	From the CVE entry: Coccinelle 0.1.7 allows local users to overwrite arbitrary files via a symlink attack on an unspecified "result file."
Alerts:	Fedora FEDORA-2009-5368 2009-05-25

Comments (none posted)

drupal-views: cross site scripting

Package(s):	drupal-views	CVE #(s):
Created:	June 16, 2009	Updated:	June 17, 2009
Description:	From the Fedora advisory: The Views module provides a flexible method for Drupal site designers to control how lists of content are presented. In the Views UI administrative interface when configuring exposed filters, user input presented as possible exposed filters is not correctly filtered, potentially allowing malicious users to insert arbitrary HTML and script code into these pages. In addition, content entered by users with 'administer views' permission into the View name when defining custom views is subsequently displayed without being filtered. Such cross site scripting (XSS) attacks may lead to a malicious user gaining full administrative access. An access bypass may exist where unpublished content owned by the anonymous user (e.g. content created by a user whose account was later deleted) is visible to any anonymous user there is a view already configured to show it incorrectly. An additional access bypass may occur because Views may generate queries which disrespect node access control. Users may be able to access private content if they have permission to see the resulting View.
Alerts:	Fedora FEDORA-2009-6171 2009-06-15 Fedora FEDORA-2009-6389 2009-06-15 Fedora FEDORA-2009-6049 2009-06-15

Comments (none posted)

firefox: multiple vulnerabilities

Package(s):	firefox	CVE #(s):	CVE-2009-2043 CVE-2009-2044 CVE-2009-2061 CVE-2009-2065
Created:	June 17, 2009	Updated:	August 3, 2009
Description:	Firefox has a multiple vulnerabilities. From the Mandriva alert: CVE-2009-2043: firefox - remote TinyMCE denial of service CVE-2009-2044: firefox - remote GIF denial of service CVE-2009-2061: firefox - man-in-the-middle exploit CVE-2009-2065: firefox - man-in-the-middle exploit
Alerts:	Mandriva MDVSA-2009:185 2009-07-30 Mandriva MDVSA-2009:134 2009-06-17 Gentoo 201301-01 2013-01-07

Comments (none posted)

firefox: multiple vulnerabilities

Package(s):	firefox	CVE #(s):	CVE-2009-1392 CVE-2009-1832 CVE-2009-1833 CVE-2009-1834 CVE-2009-1835 CVE-2009-1836 CVE-2009-1837 CVE-2009-1838 CVE-2009-1839 CVE-2009-1840 CVE-2009-1841
Created:	June 12, 2009	Updated:	August 3, 2009
Description:	From the Red Hat advisory: Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code as the user running Firefox. (CVE-2009-1392, CVE-2009-1832, CVE-2009-1833, CVE-2009-1837, CVE-2009-1838, CVE-2009-1841) Multiple flaws were found in the processing of malformed, local file content. If a user loaded malicious, local content via the file:// URL, it was possible for that content to access other local data. (CVE-2009-1835, CVE-2009-1839) A script, privilege elevation flaw was found in the way Firefox loaded XML User Interface Language (XUL) scripts. Firefox and certain add-ons could load malicious content when certain policy checks did not happen. (CVE-2009-1840) A flaw was found in the way Firefox displayed certain Unicode characters in International Domain Names (IDN). If an IDN contained invalid characters, they may have been displayed as spaces, making it appear to the user that they were visiting a trusted site. (CVE-2009-1834) A flaw was found in the way Firefox handled error responses returned from proxy servers. If an attacker is able to conduct a man-in-the-middle attack against a Firefox instance that is using a proxy server, they may be able to steal sensitive information from the site the user is visiting. (CVE-2009-1836)
Alerts:	Mandriva MDVSA-2009:185 2009-07-30 Fedora FEDORA-2009-7614 2009-07-15 Fedora FEDORA-2009-7567 2009-07-15 Debian DSA-1830-1 2009-07-12 Slackware SSA:2009-178-01 2009-06-29 Mandriva MDVSA-2009:141 2009-06-17 Ubuntu USN-782-1 2009-06-25 CentOS CESA-2009:1126 2009-06-26 Red Hat RHSA-2009:1126-01 2009-06-25 Red Hat RHSA-2009:1125-01 2009-06-25 CentOS CESA-2009:1095 2009-06-19 Debian DSA-1820-1 2009-06-18 Slackware SSA:2009-167-01 2009-06-17 Mandriva MDVSA-2009:134 2009-06-17 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 Fedora FEDORA-2009-6411 2009-06-15 Fedora FEDORA-2009-6366 2009-06-15 SuSE SUSE-SA:2009:034 2009-06-16 Ubuntu USN-779-1 2009-06-12 CentOS CESA-2009:1096 2009-06-12 Red Hat RHSA-2009:1096-01 2009-06-11 Red Hat RHSA-2009:1095-01 2009-06-11 Gentoo 201301-01 2013-01-07

Comments (none posted)

gupnp: crash on zero length content

Package(s):	gupnp	CVE #(s):
Created:	June 16, 2009	Updated:	June 17, 2009
Description:	From the Fedora advisory: Version 0.12.8 fixes a bug where the gupnp stack crashes when passed empty content.
Alerts:	Fedora FEDORA-2009-5861 2009-06-04 Fedora FEDORA-2009-5865 2009-06-04

Comments (none posted)

icu: cross-site scripting

Package(s):	icu	CVE #(s):	CVE-2009-0153
Created:	June 16, 2009	Updated:	October 8, 2009
Description:	From the CVE entry: International Components for Unicode (ICU) in Apple Mac OS X 10.5 before 10.5.7 does not properly handle invalid byte sequences during Unicode conversion, which might allow remote attackers to conduct cross-site scripting (XSS) attacks.
Alerts:	Debian DSA-1889-1 2009-09-16 SuSE SUSE-SR:2009:014 2009-09-01 CentOS CESA-2009:1122 2009-06-26 Red Hat RHSA-2009:1122-01 2009-06-25 Fedora FEDORA-2009-6121 2009-06-15 Fedora FEDORA-2009-6273 2009-06-15 Ubuntu USN-846-1 2009-10-08

Comments (none posted)

irssi: off-by-one error

Package(s):	irssi	CVE #(s):	CVE-2009-1959
Created:	June 16, 2009	Updated:	December 8, 2009
Description:	From the Mandriva advisory: Off-by-one error in the event_wallops function in fe-common/irc/fe-events.c in irssi 0.8.13 allows remote IRC servers to cause a denial of service (crash) via an empty command, which triggers a one-byte buffer under-read and a one-byte buffer underflow.
Alerts:	Mandriva MDVSA-2009:133-1 2009-12-08 Gentoo 200909-13 2009-09-12 Fedora FEDORA-2009-7004 2009-06-27 Fedora FEDORA-2009-7012 2009-06-27 Ubuntu USN-800-1 2009-07-13 SuSE SUSE-SR:2009:012 2009-07-03 Mandriva MDVSA-2009:133 2009-06-16

Comments (none posted)

libtorrent-rasterbar: directory traversal

Package(s):	libtorrent-rasterbar	CVE #(s):	CVE-2009-1760
Created:	June 15, 2009	Updated:	July 17, 2009
Description:	From the Debian advisory: It was discovered that the Rasterbar Bittorrent library performed insufficient validation of path names specified in torrent files, which could lead to denial of service by overwriting files.
Alerts:	Gentoo 200907-14 2009-07-17 Fedora FEDORA-2009-6760 2009-06-19 Fedora FEDORA-2009-6682 2009-06-19 Fedora FEDORA-2009-6502 2009-06-18 Fedora FEDORA-2009-6547 2009-06-18 Fedora FEDORA-2009-6619 2009-06-18 Mandriva MDVSA-2009:139 2009-06-24 Debian DSA-1815-1 2009-06-14

Comments (none posted)

mingw32-libpng: information disclosure

Package(s):	mingw32-libpng	CVE #(s):	CVE-2009-2042
Created:	June 16, 2009	Updated:	August 17, 2010
Description:	From the CVE entry: libpng before 1.2.37 does not properly parse 1-bit interlaced images with width values that are not divisible by 8, which causes libpng to include uninitialized bits in certain rows of a PNG file and might allow remote attackers to read portions of sensitive memory via "out-of-bounds pixels" in the file.
Alerts:	CentOS CESA-2010:0534 2010-08-16 CentOS CESA-2010:0534 2010-07-21 CentOS CESA-2010:0534 2010-07-21 CentOS CESA-2010:0534 2010-07-14 Red Hat RHSA-2010:0534-01 2010-07-14 Debian DSA-2032-1 2010-04-11 Mandriva MDVSA-2010:063 2010-03-22 Ubuntu USN-913-1 2010-03-16 Gentoo 200906-01 2009-06-27 Slackware SSA:2009-170-01 2009-06-22 Fedora FEDORA-2009-6506 2009-06-18 Fedora FEDORA-2009-6531 2009-06-18 Fedora FEDORA-2009-6603 2009-06-18 Fedora FEDORA-2009-6400 2009-06-15 Fedora FEDORA-2009-5977 2009-06-15 Oracle ELSA-2012-0317 2012-02-21

Comments (1 posted)

mutt: SSL certificate vulnerability

Package(s):	mutt	CVE #(s):	CVE-2009-1390
Created:	June 16, 2009	Updated:	June 17, 2009
Description:	From the Fedora advisory: Version 1.5.19 fixes a problem with SSL certificate chain verification.
Alerts:	Fedora FEDORA-2009-6465 2009-06-15

Comments (none posted)

perl: buffer overflow

Package(s):	perl	CVE #(s):	CVE-2009-1391
Created:	June 16, 2009	Updated:	December 4, 2009
Description:	From the Red Hat bugzilla: Compress::Raw::Zlib versions before 2.017 contain a buffer overflow in inflate(). A badly formed zlib-stream can trigger this buffer overflow and cause the perl process at least to hang or to crash.
Alerts:	Mandriva MDVSA-2009:157-1 2009-12-03 Mandriva MDVSA-2009:207 2009-08-19 Gentoo 200908-07 2009-08-18 Mandriva MDVSA-2009:174 2009-07-29 Mandriva MDVSA-2009:157 2009-07-19 Fedora FEDORA-2009-7680 2009-07-15 SuSE SUSE-SR:2009:012 2009-07-03 Ubuntu USN-794-1 2009-07-02 Fedora FEDORA-2009-6033 2009-06-15

Comments (none posted)

php-ZendFramework: local file inclusion

Package(s):	php-ZendFramework	CVE #(s):
Created:	June 16, 2009	Updated:	June 17, 2009
Description:	From the Red Hat bugzilla: Upstream Zend Framework 1.7.5 contains a security fix for a potential Local File Inclusion (LFI) vulnerability in the Zend_View::render() method. This fixed is tagged upstream as "controversial", as it breaks backwards compatibility and existing uses of method.
Alerts:	Fedora FEDORA-2009-2823 2009-03-18

Comments (none posted)

tomcat: multiple vulnerabilities

Package(s):	tomcat6	CVE #(s):	CVE-2008-5515 CVE-2009-0033 CVE-2009-0580 CVE-2009-0781 CVE-2009-0783
Created:	June 15, 2009	Updated:	November 2, 2010
Description:	From the Ubuntu advisory: Iida Minehiko discovered that Tomcat did not properly normalise paths. A remote attacker could send specially crafted requests to the server and bypass security restrictions, gaining access to sensitive content. (CVE-2008-5515) Yoshihito Fukuyama discovered that Tomcat did not properly handle errors when the Java AJP connector and mod_jk load balancing are used. A remote attacker could send specially crafted requests containing invalid headers to the server and cause a temporary denial of service. (CVE-2009-0033) D. Matscheko and T. Hackner discovered that Tomcat did not properly handle malformed URL encoding of passwords when FORM authentication is used. A remote attacker could exploit this in order to enumerate valid usernames. (CVE-2009-0580) Deniz Cevik discovered that Tomcat did not properly escape certain parameters in the example calendar application which could result in browsers becoming vulnerable to cross-site scripting attacks when processing the output. With cross-site scripting vulnerabilities, if a user were tricked into viewing server output during a crafted server request, a remote attacker could exploit this to modify the contents, or steal confidential data (such as passwords), within the same domain. (CVE-2009-0781) Philippe Prados discovered that Tomcat allowed web applications to replace the XML parser used by other web applications. Local users could exploit this to bypass security restrictions and gain access to certain sensitive files. (CVE-2009-0783)
Alerts:	Fedora FEDORA-2010-16248 2010-10-14 Mandriva MDVSA-2010:176 2010-09-12 CentOS CESA-2010:0580 2010-08-03 Red Hat RHSA-2010:0580-01 2010-08-02 SuSE SUSE-SR:2010:008 2010-04-07 SuSE SUSE-SR:2009:013 2009-08-11 CentOS CESA-2009:1164 2009-07-29 Mandriva MDVSA-2009:163 2009-07-28 Red Hat RHSA-2009:1164-01 2009-07-21 Red Hat RHSA-2009:1563-01 2009-11-09 Red Hat RHSA-2009:1562-01 2009-11-09 SuSE SUSE-SR:2009:012 2009-07-03 Mandriva MDVSA-2009:138 2009-06-22 Mandriva MDVSA-2009:136 2009-06-22 Fedora FEDORA-2009-11352 2009-11-11 Fedora FEDORA-2009-11374 2009-11-11 Fedora FEDORA-2009-11356 2009-11-11 Ubuntu USN-788-1 2009-06-15 Gentoo 201206-24 2012-06-24

Comments (1 posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current stable 2.6 kernel remains 2.6.30. The 2.6.31 merge window is open (see below).

The current stable 2.6 kernel update is 2.6.29.5, released on June 15. 2.6.27.25 was released on June 11. Both have a long list of fixes throughout the kernel.

Comments (none posted)

Kernel development news

Quotes of the week

-- H. Peter Anvin; vikings never were good at forward compatibility

-- Alan Cox

-- Linus Torvalds draws a line in the sand

Comments (1 posted)

2.6.31 merge window, week 1

The process of merging patches for 2.6.31 has begun. As of this writing, 6220 non-merge changesets have been added to the mainline. Some of the more interesting, user-visible change include:

• Reads from /dev/zero can now be interrupted by signals. In theory, no application should be adversely affected by this change, but is still a true user-space ABI change.

• The x86 architecture can now handle 46-bit physical addresses, allowing the use of up to 64TB of physical memory.

• A number of Xen features meant to support Dom0 functionality (the /dev/xen/evtchn event channel code ant /sys/hypervisor, for example) have been merged. The Dom0 code itself remains outside, though.

• Support for the rt_tgsigqueueinfo() system call, which sends a signal (with ancillary information) to a thread group, has been added.

• A number of ftrace features have been added. These include a function profiler, a number of event tracer improvements, some new tracepoints, and a new docbook document describing static tracepoints in the kernel.

• The USB TTY port driver has been reworked in ways which bring it into closer alignment with POSIX behavior and that of other types of serial ports. Alan Cox fears that some applications which depended on the old behavior might break - though others, which had problems with USB serial ports, should now begin to work.

• There is a new sysctl knob (/proc/sys/kernel/modules_disabled); writing "1" to that file will cause module loading to be forevermore disallowed.

• The SMACK security module now has a comprehensive logging facility.

• The splice() system call now works when both the input and output files are pipes.

• The storage topology patches (covered briefly in April) have been merged. This code allows the kernel to export much more information about how storage devices are structured, enabling support for upcoming hardware.

• The performance counters patch set (last covered in December) have been merged. This code provides a new API for the use of hardware performance counters; it edges out perfmon and a number of other implementations in this area.

• The character devices in user space (CUSE) patch set has been merged.

• arch-imx has been removed from the ARM architecture as it has been superseded by the MXC architecture.

• The s390 architecture has added support for dynamic ftrace, as well as the system call and function graph tracers.

• The packet_mmap changes for the transmission side of packet sockets was merged, which allows for more efficient, nearly zero-copy, packet sends from user space.

• The controller area network (CAN) subsystem has added a network device driver interface and a netlink interface. New CAN device drivers have also been merged using the driver interface (see below).

• Support for IEEE 802.15.4 sockets has been added to the network subsystem. This is for low-cost, low-speed "personal area networks".

• Passive OS fingerprinting has been added to the netfilter code.

• The FAT filesystem has added an "errors" mount option which governs its behavior in the presence of critical errors.

• The s390 architecture has added support for hibernation.

• Support has been added for USB 3.0/xHCI host controllers, though none are yet available.

• Kernel modesetting for the radeon driver, supporting R1XX, R2XX, R3XX, R4XX, R5XX, and radeon up to X1950 hardware, has been merged.

• There is the usual pile of new drivers:
  • Architectures/processors/systems: SuperH SH7724 processors, Hitachi SH7724 (SolutionEngine7724) boards, memory error detection and correction (EDAC) support for AMD K8, F10h, and F11h processors, ARM PCM043 boards, ARM Atmark Armadillo 500 boards, OMAP3 Zoom2 boards, OMAP4430 SDP boards (including SMP support), ARM MX35PDK boards, Marvell 88F6281 GTW GE boards,Samsung S3C6400 SoCs, ARMv6/v7 big-endian, Stargate 2 boards, Freescale STMP platforms, ST-Ericsson U300 series platforms, PowerPC MPC8569MDS boards, PowerPC P2020DS boards.

  • Miscellaneous: Timberdale FPGA UARTs, 64-bit VIA hardware random number generators, Mediama RMTx add-on board for ATNGW100, Wacom Graphire Bluetooth tablet, CB710/720 memory card readers, Maxim 1586 voltage regulators, TI TMP401 temperature sensor, Intel Langwell USB device controllers, Intel Langwell USB On-the-go controllers, TI VLYNQ bus, ST Microelectronics DDC I2C interface.

  • Networking: Broadcom NetXtremeII gigabit Ethernet cards (offload features in particular), Intellon PLC (Powerline communications) USB adapters, Marvell SD8688 wireless chips, Ralink rt2800 wireless USB chipsets, TI wl1251/wl1271 wireless chipsets, TI DaVinci Ethernet MACs, Phillips SJA1000 CAN controllers, Kvaser PCIcanx and Kvaser PCIcan PCI CAN Cards, Intel wireless Multicomm 3200 devices, Micrel KS8842 ethernet switches.

  • Sound: Wolfson Micro WM8988, WM8940, WM9081, and WM8960 codecs, Digigram LX6464ES boards, ESI Maya44 boards, several Creative Sound Blaster X-FI devices based on the 20K1 and 20K2 chipsets, a USB Audio gadget driver.

  • Graphics: AMD r600 chipset, Intel IGDNG chipset.

  • Video: DVB-S/S2/DSS Multistandard Professional/Broadcast demodulators, STV6110/(A) based tuners, ISL6423 SEC controllers, TI THS7303 video amplifiers, Analog Devices I2C bus based ADV7343 encoders.

Changes visible to kernel developers include:

• There is a new atomic function:

      int atomic_dec_and_mutex_lock(atomic_t *cnt, struct mutex *lock);
  

  This function will decrement cnt and, if cnt reaches zero, it will acquire the given lock.

• A number of block layer request queue API changes have been merged; all drivers must now dequeue requests before executing them. Beyond that, the merging of the storage topology patches (in preparation for 4K-sector disks) mean that block drivers must now distinguish between the physical block size on the disk and the logical block size used by the kernel.

• The 32-bit x86 architecture now supports the atomic64_t type.

• The kernel memory leak detector has been merged at last.

• The fsnotify backend has been merged. This code provides a new, common implementation for dnotify and inotify; it also will serve as the base for the "fanotify" code (formerly TALPA), which has not been merged as of this writing.

• Btrfs has seen a number of enhancements, including one which involves an on-disk format change. Existing btrfs filesystems will work with the new code, but, once mounted on a 2.6.31 system, those filesystems will no longer work with older kernels.

• Tree read-copy update (RCU) is now the default, though Classic RCU is still available.

• Changes to the include/asm-generic header files were merged. These changes are meant to serve as a model for or be used directly by new architectures rather than copying from an existing architecture. The S+core (score) architecture depends on these changes and the MicroBlaze architecture will be using them to clean up its ABI.

• Some rather large cleanups for XFS were merged, including switching to the generic POSIX ACL code and removal of the xfs_qmops quota ops function vector.

• All network drivers have converted to the new net_device_ops API and the old API available with COMPAT_NET_DEV_OPS has been removed.

• The rfkill core has been rewritten for devices that implement a way to stop all radio transmission from the device (in response to a laptop key for turning off wireless, for example). Various drivers have also been updated to use the new rfkill API.

• Debugfs has had all of its references throughout the tree turned into /sys/kernel/debug/ in both documentation and code. In addition, LWN's updated guide to debugfs was added to the Documentation directory.

• Unicode handling in the kernel has been updated, with functions like utf_mbstowcs() being renamed to utf8s_to_utf16s() for better readability.

• The kmemcheck kernel memory checker to detect the use of uninitialized memory has been merged.

• The TTM GPU memory manager (covered a bit over a year ago) has been merged.

Linus started merging patches on June 10, suggesting that the merge window will remain open until sometime around June 24. That puts us roughly half way through the merge window, in terms of time. The merge rate will likely slow down some for the remainder of the merge window, but there are, undoubtedly, more interesting changes to come. Stay tuned.

Comments (24 posted)

What ever happened to chunkfs?

"What ever happened to chunkfs?" This is a question I hear every few months, and I always answer the same way, "Chunkfs works, the overhead is reasonable, and it is only practical if it is part of the file system design from the beginning, not tacked on after the fact. I just need to write up the paper summarizing all the data." Thanks to your benevolent LWN editor, you are now reading that paper.

Background

Before we describe chunkfs, we should first talk about the problems that led to its creation. Chunkfs was motivated by the growing difficulty of reading and processing all the data necessary to check an entire file system. As the capacity of storage grows, the capacity of the rest of the system to read and check that data is not keeping pace with that growth. As a result, the time to check a "normal" file system grows, rather than staying constant as it would if the memory, bandwidth, seek time, etc. of the system grew in proportion to its storage capacity. These differential rates of growth in hardware - like the differences between RAM capacity, bandwidth, and latency - are part of what keeps systems research interesting.

To understand the change in time to check and repair a file system, a useful analogy is to compare the growth of my library (yes, some people still own books) with the growth of my ability to read, organize, and find the books in the library. As a child, my books fit all on one shelf. I could find the book I wanted in a matter of seconds and read every single one of my books in a week. As an adult, my books take up several bookshelves (in addition to collecting on any flat surface). I can read many times faster than I could when I was a kid, and organize my books better, but finding a book can take several minutes, and reading them all would take me a year or more. If I was trying to find a twenty dollar bill I left in a book, it would take me several hours to riffle through all the books to find it. Even though I am a better reader now, my library grew faster than my ability to read or search my library. Similarly, computers are faster than ever, but storage capacity grew even faster.

The consequence of this phenomenon that first caught my attention was the enormous disparity between seek time and capacity of disks. I calculated a projected change in fsck time given these predictions [PDF] from a storage industry giant in 2006:

Improvement in disk performance, 2006 - 2013
Capacity:	16.0x
Bandwidth:	5.0x
Seek time:	1.2x
=> at least 10x increase in fsck time!

Since then, commodity flash-based storage finally became a practical reality. Solid-state disk (SSDs) have much faster "seeks", somewhat improved bandwidth, and drastically reduced capacity compared to disks, so the performance of fsck ought to be extremely good. (I am unable to find any measurements of fsck time on an SSD, but I would estimate it to be on the order of seconds. Readers?) However, SSDs don't solve all our problems. First, SSDs are an element in the cache hierarchy of a system, layered between system RAM and disk, not a complete replacement for disks. The sweet spot for SSDs will continue to expand, but it will be many years or decades before disks are completely eliminated - look at the history of tape. It's easy for a laptop user to wave their hands and say, "Disks are obsolete," but try telling that to Google, your mail server sysadmin, or even your MythTV box.

Second, remember that the whole reason we care about fsck in the first place is because file systems get corrupted. One important source of file system corruption is failures in the storage hardware itself. Ask any SSD manufacturer about the frequency of corruption on its hardware and they will inundate you with equations, lab measurements, and simulations showing that the flash in their SSDs will never wear out in "normal" use - but they also won't reveal the details of their wear-leveling algorithms or the workloads they used to make their predictions. As a result, we get surprises in real-world use, both in performance and in reliability. When it comes to failure rates, we don't have good statistics yet, so I have to fall back on my experience and that of my kernel hacker friends. What we see in practice is that SSDs corrupt more often and more quickly than disks, and prefer to devour the tastiest, most crucial parts of the file system. You can trust the hardware manufacturers when they wave their hands and tell you not to worry your pretty little head, but I put more weight on the money I've made consulting for people with corrupted SSDs.

So, if corruption is a concern, an important benefit of the chunkfs approach is that file system corruption is usually limited to a small part of the file system, regardless of the source of the corruption. Several other repair-driven file system principles - such as duplicating and checksumming important data - make recovery of data after corruption much more likely. So if you don't care about fsck time because you'll be using an SSD for the rest of your life, you might still read on because you care about getting your data back from your SSD.

The conclusion I came to is that file systems should be designed with fast, reliable check and repair as a goal, from the ground up. I outline some useful methods for achieving this goal in a short paper: Repair-driven File System Design [PDF].

Chunkfs design

Chunkfs is the unmusical name for a file system architecture designed under the assumption that the file system will be corrupted at some point. Therefore the on-disk format is optimized not only for run-time performance but also for fast, reliable file system check and repair. The basic concept behind chunkfs is that a single logical file system is constructed out of multiple individual file systems (chunks), each of which can be checked and repaired (fscked) individually.

This is great and all, but now we have a hundred little file systems, with the namespace and disk space fragmented amongst them - hardly an improvement. For example, if we have a 100GB file system divided into 100 1GB chunks, and we want to create a 2GB file, it won't fit in a single chunk - it has to somehow be shared across multiple chunks. So how do we glue all the chunks back together again so we can share the namespace and disk space while preserving the ability to check and repair each chunk individually? What we want is a way to connect a file or directory in one chunk and another file or directory in another chunk in such a way that the connection can be quickly and easily checked and repaired without a full fsck of the rest of the chunk. The solution is something we named a "continuation inode". A continuation inode "continues" a file or directory into another chunk.

You'll notice that there are two arrows in the picture to the left, one pointing from the original inode to the continuation inode, and another pointing back from the continuation inode to the original inode. When checking the second chunk, you can check the validity of the continuation inode quickly, by using the back pointer to look up the original inode in its chunk. This is a check of a "cross-chunk reference" - any file system metadata that makes a connection between data in two different chunks. Cross-chunk references must always have forward and back pointers so that they can be verified starting from either chunk.

Cross-chunk references must satisfy one other requirement: You must be able to quickly find all cross-references from chunk A to arbitrary chunk B, without searching or checking the entire chunk A. To see why this must be, we'll look at an example. Chunk A has an inode X which is continued into chunk B. What if chunk A was corrupted in such a manner that inode X was lost entirely? We would finish checking and repairing chunk A, and it would be internally self-consistent, but chunk B would still have a continuation inode for X that is now impossible to look up or delete - an orphan continuation inode. So as the second step in checking chunk A, we must then find all references to chunk A from chunk B (and all the other chunks) and check that they are in agreement. If we couldn't, we would have to search every other chunk in the file system to check a single chunk - and that's not quick.

A chunkfs-style file system can be checked and repaired incrementally and online, whereas most file systems must be checked all at once while the file system is offline. Some exceptions to this general rule do exist, such as the BSD FFS snapshot-based online check, and an online self-healing mechanism for NTFS, but in general these facilities are hacked on after the fact and are severely limited in scope. For example, if the BSD online check actually finds a problem, the file system must be unmounted and repaired offline, all at once, in the usual way, including a rerun of the checking stage of fsck.

The chunkfs design requires solutions to several knotty problems we don't have room to cover in this article, but you can read about in our 2006 Hot Topics in Dependability paper: Chunkfs: Using divide-and-conquer to improve file system reliability and repair [PDF].

Measurements

Repair-driven file system design, chunkfs in particular, sounds like a neat idea, but as a file system developer, I've had a lot of neat ideas that turned out to be impossible to implement, or had too much overhead to be practical. The questions I needed to answer about chunkfs were: Can it be done? If it can be done, will the overhead of continuation inodes outweigh the benefit? In particular, we need to balance the time it takes to check an individual chunk with the time it takes to check its connections to other chunks (making sure that the forward and back pointers of the continuation inodes agree with their partners in the other chunk). First, let's take a closer look at file system check time on chunkfs.

The time to check and repair a file system with one damaged chunk is the sum of two different components, the time to check one chunk internally and the time to check the cross references to and from this chunk.

T_fs = T_chunk + T_cross

The time to check one chunk is a function of the size of the chunk, and the size of the chunk is the total size of the file system divided by the number of chunks.

T_chunk = f(size_chunk)

size_chunk = size_fs / n_chunks

The time to check the cross-chunk references to and from this chunk depends on the number of those cross references. The exact number of cross-chunk references will vary, but in general larger chunks will have fewer cross references and smaller chunks will have more - that is, cross chunk check time will grow as the number of chunks grow.

T_cross = f(n_chunks)

So, per-chunk check time gets smaller as we divide the file system into more chunks, and at the same time the cross-chunk check time grows larger. Additionally, the extra disk space taken up by continuation inodes grows as the number of chunks grows, as does the overhead of looking up and following continuation inodes during normal operation. We want to find a sweet spot where the sum of the time to check a chunk and its cross-references is minimized, while keeping the runtime overhead of the continuation inodes small.

Does such a sweet spot exist? The answer depends on the layout of files and directories on disk in real life. If cross-chunk references are extremely common, then the overhead of continuation inodes will outweigh the benefits. We came up with an easy way to estimate the number of cross-chunk references in a chunkfs file system: Take a "real" in-use ext3 file system and for each file, measure the number of block groups containing data from that file. Then, for each directory, measure the number of block groups containing files from that directory. If we add up the number of block groups less one for all the files and directories, we'll get the number of cross-chunk references in a similar chunkfs file system with chunks the size of the ext3 file system's block groups (details here).

Karuna Sagar wrote a tool to measure these cross-references, called cref, and I added some code to do a worst-case estimate of the time to check the cross-references (assuming one seek for every cross-reference). The results were encouraging; assuming disk hardware progresses as predicted, the average cross-chunk cross-reference checking time would be about 5 seconds in 2013, and the worst case would be about 160 seconds (about 2.5 minutes). This is with a 1GB chunk size, so the time to check the chunk itself would be a few seconds. This estimate is worst-case in another way: the ext3 allocator is in no way optimized to reduce cross-chunk references. A chunkfs-style file system would have a more suitable allocator.

Implementations

Chunkfs was prototyped three times. The first prototype, written by Amit Gud for his master's thesis [PDF] in 2006-2007, implemented chunkfs as modifications to the ext2 driver in FUSE. Note that the design of chunkfs described in the thesis is out of date in some respects, see the chunkfs paper [PDF] for the most recent version. He also ported this implementation to the kernel in mid-2007, just for kicks. The results were encouraging. Our main concern was that continuation inodes would proliferate wildly, overwhelming any benefits. Instead, files with continuation inodes were uncommon - 2.5% of all files in the file systems - and no individual file had more than 10 continuation inodes. The test workload included some simulation of aging by periodically deleting files while filling the test file system. (It would be interesting to see the results from using Impressions [PDF], a very sophisticated tool for generating realistic file system images.)

These implementations were good first steps, but they were based on an earlier version of the chunkfs design, before we had solved some important problems. In these implementations, any chunk that had been written to since the file system was mounted had to be fscked after a crash. Given that most of the file system is idle in common usage, this reduced check time by about 2/3 in the test cases, but we are looking for a factor of 100, not a factor of 3. It also lacked a quick way to locate all of the references into a particular damaged chunk, so it only checked the references leading out of the chunk. It used an old version of the solution for hard links which would allow hard links to fail if the target's chunk ran out of space, instead of growing a continuation for the target into a chunk that did have space. It was a good first step, but lacked a key feature: drastically reduced file system repair time.

In mid-2007, I decided to write a prototype of chunkfs as a layered file system, similar to unionfs or ecryptfs, that was completely independent of the client file system used in each chunk. Continuation inodes are implemented as a regular files, using extended attributes to store the continuation-related metadata (the forward and back pointers and the offset and length of the file stored in this continuation inode). When the file data exceeds an arbitrary size (40K in my prototype), a continuation inode is allocated in another chunk and the data after that point in the file is stored in that inode's data. All of the continuation inodes emanating from a particular chunk are kept in one directory so that they can be quickly scanned during the cross-chunk pass of fsck.

To test that the prototype could recover from file system corruption in one chunk without checking the entire file system, I implemented fsck as a simple shell script. First, it fscks the damaged chunk by running the ext2 fsck on it. This may end up deleting or moving arbitrary files in the chunk, which could make it out of sync with the other chunks. Second, it mounts the now-repaired file systems and reads their /chunks directories to find all connections to the damaged chunk and consistency check them. If it finds an orphaned continuation - a continuation whose origin inode in the damaged chunk was destroyed or lost - then it deletes that continuation inode.

The fsck script is deficient in one particular aspect: it checks all the chunks because I didn't write the code to mark a chunk as damaged. This is a difficult problem in general; sometimes we know that a chunk has been damaged because the disk gave us an IO error, or we found an inconsistency while using the file system, and then we mark the file system as corrupted and needing fsck. But plenty of corruption is silent - how can we figure out which chunk was silently corrupted? We can't, but we can quietly "scrub" chunks by fscking them in the background. Currently, ext2/3/4 triggers a paranoia fsck every N mounts or M days since the last check. Unfortunately, this introduces an unexpected delay of minutes or hours at boot time; if you're lucky, you can go have a cup of coffee while it finishes, if you're unlucky, you'll be figuring out how to disable it while everyone who has come to your talk about improvements in Linux usability watches you. (N.B.: Reboot and add "fastboot" to the kernel command line.) With chunkfs, we could run fsck on a few chunks at every boot, adding a few seconds to every boot but avoiding occasional long delays. We could also fsck inactive chunks in the background while the file system in use.

Results

I gave a talk at LCA in January 2008 about chunkfs which included a live demo of the final chunkfs prototype in action: creating files, continuing them into the next chunk, deliberately damaging a chunk, and repairing the file system. Because I carefully prepared a typescript-based fallback in case things went sideways, the demo went perfectly. A video of the talk [Ogg Theora] is available.

Note that I have never been able to bring myself to watch this talk, so I don't know if it contains any embarrassing mistakes. If you want, you can follow along during the demo section with the annotated output from the demo.

The collection-of-file-systems approach ended up being more complex than I had hoped. The prototype used ext2 as the per-chunk file system - a reasonable choice for some throw-away code, but not workable in production. However, once you switch to any reliable file system, you end up with one journal per chunk, which is quite a lot of overhead. In addition, it seemed likely we'd need another journal to recover from crashes during cross-chunk operations. Another source of complexity was the use of a layered file system approach - basically, the chunkfs layer pretends to be a client file system when it's talking to the VFS, and pretends to be the VFS when it's talking to the per-chunk file system. Since none of this is officially supported, we end up with a long list of hacks and workarounds, and it's easy to run into surprise reference counting or locking bugs. Overall, the approach worked for a prototype, but it didn't seem worth the investment it would to make it production quality, especially with the advent of btrfs.

Future work

When I began working on chunkfs, the future of Linux file systems development looked bleak. The chunkfs in-kernel prototype looked like it might be the only practical way to get repair-driven design into a Linux file system. All that has changed; file system development is a popular, well-funded activity and Linux has a promising next-generation file system, btrfs, which implements many principles of repair-driven file system design, including checksums, magic numbers, and redundant metadata. Chunkfs has served its purpose as a demonstration of the power of the repair-driven design approach and I have no further development plans for it.

Conclusions

The three chunkfs prototypes and our estimates of cross-chunk references using real-world file systems showed that the chunkfs architecture works as advertised. The prototypes also convinced us that it would be difficult to retrofit existing journaling file systems to the chunkfs architecture. Features that make file system check and repair are best when designed into the architecture of the file system from the beginning. Btrfs is an example of a file system designed from the ground up with the goal of fast, reliable check and repair.

Credits

Many people and organizations generously contributed to the design and prototyping of chunkfs. Arjan van de Ven was the co-inventor of the original chunkfs architecture. Theodore Ts'o and Zach Brown gave invaluable advice and criticism while we were thrashing out the details. The participants of the Hot Topics in Dependability Workshop gave us valuable feedback and encouragement. Karuna Sagar wrote the cross-reference measuring tool that gave us the confidence to go forward with an implementation. Amit Gud wrote two prototypes while a graduate student at Kansas State University. The development of the third prototype was funded at various points by Intel, EMC, and VAH Consulting. Finally, too many people to list made valuable contributions during discussions about chunkfs on mailing lists, and I greatly appreciate their help.

Comments (37 posted)

Avoiding a read-only filesystem on errors

Errors in the storage subsystem can happen due to a variety of known or unknown reasons. A filesystem turns read-only when it encounters errors in the storage subsystem, or a code path which the filesystem code base should not have taken (i.e. a BUG() path). Making the filesystem read-only is a safeguard feature that filesystems implement to avoid further damage because of the errors encountered. Filesystems that change to read-only end up stalling services relying on data writes and, in some cases, may lead to an unresponsive system. In embedded devices, dying applications due to a filesystem turning read-only may render the device useless, leaving the user confused.

Filesystem errors can either be recoverable or non-recoverable. Errors from bad block links in the inode data structures or block pointers can be recovered by filesystem checks. On the other hand, errors due to memory corruption, or a programming error, might not be recoverable because one cannot be sure which data is correct.

Denis Karpov proposed a patchset that would notify user space, so that a user-space policy can be defined to take the appropriate action to avoid turning the filesystem read-only. The patchset is currently limited to FAT filesystems. User-space notifications would allow a filesystem to continue to be used after encountering "correctable errors" if proactive measures are taken to correct them. Such corrective actions can obviate the need for lengthy filesystem checks when the device is mounted next.

The patchset adds a /sys/fs/fat/<volume>/fs_fault file which is initialized to 0 when the filesystem is mounted. The value of fs_fault changes to 1 on error. A user-space program can poll() on this file to check if the value of the file changed which is an indication that an error has occurred. Besides the file polling, a KOBJ_CHANGE uevent is generated, with the uevents environment variable FS_UNCLEAN set to 0 or 1. A udev rule can then trigger the correct program to take care of the damage done by the error.

Kay Sievers is not convinced with the idea of using uevents for user-space notifications, as uevents are designed for device notifications, so they do not fit the design goals of reporting filesystem errors. Filesystem errors are quite often a repeated series of events. For example, a read failure may result in printing multiple read errors in dmesg for each block it is not able to read. An event generated for each block may be too much for udev to handle. Some of the events may get lost, or worse still, may cause udev to ignore uevents from other devices which occurred during the burst of errors.

Keeping <volume>/fs_fault in sysfs is also not the best solution, because sysfs is unaware of filesystem namespaces. The primary responsibility of sysfs is to expose core kernel objects. Filesystem namespaces are a set of filesystem mounts that are only visible to a particular process and may be invisible to the rest of the processes.

A process with a private namespace contains a copy of the namespace instead of a shared namespace. When the system starts, it contains one global namespace which is shared among all processes. Mounts and unmounts in a shared namespace are seen by all the processes in the namespace. A process creates a private namespace if it was created by the clone() system call using the CLONE_NEWNS flag (clone New NameSpace). A process sharing a public namespace can also create a private namespace by calling unshare() with CLONE_NEWNS flag. Mounts and unmounts within a private namespace are only seen by the process that created the namespace. A child process created by fork() shares its parent's namespace.

Because of this, processes might receive errors on a filesystem in a different namespace, so they would not know which device to act on. The problem is also noticeable with processes accessing bind mounts created in a different namespace (bind mounts are a feature in which a sub-tree of a filesystem can be mounted on another directory). Moreover, filesystems spanning multiple devices, such as btrfs, would not be able to report the device name under the current naming structure.

Kay recommends /proc/self/mountinfo as a better alternative, because it contains the list of mount points in the namespace of the process with the specified PID (self). Currently, /proc/self/mountinfo changes when the mount tree changes. This can be extended to propagate errors to user space in the correct namespace using poll(), regardless of the device name. /proc/self/mountinfo can accommodate optional fields which hold values in the form of "tag[:value]" that can be used to communicate the nature of the problem. Instead of using the existing udev infrastructure, this would require a dedicated application to monitor /proc/self/mountinfo, identify the error by parsing the argument, and act accordingly.

Jan Kara further suggests using /proc/self/mountinfo as a link to identify the filesystem device generating the errors:

Despite these objections, everyone agrees that error reporting to user space must not be limited to dmesg messages. User space should be notified of the errors reported by the filesystem, so that it can proactively handle errors and try to correct them. The namespace-unaware /sys filesystem or notifications through uevent may not be the best solution, but, for a lack of a better alternative interface, the developers used sysfs and uevents. The design is still under discussion, and will take some time to evolve, though it is likely that some kind of solution to this problem will make its way into the kernel.

Comments (3 posted)

Linux kernel design patterns - part 2

Last week we discussed the value of enunciating kernel design patterns and looked at the design patterns surrounding reference counts. This week we will look at a very different aspect of coding and see why the kernel has special needs, and how those needs have been addressed by successful approaches. The topic under the microscope today is complex data structures.

By "complex data structures" we mean objects that are composed of a variable number of simpler objects, possibly a homogeneous collection, possibly a heterogeneous collection. In general it is a structure to which objects can be added, from which objects can be removed, and in which objects can be found. The preferred way to work with such data structures when we study them in an introductory programming course is to use Abstract Data Types.

Abstract Data Types

The idea behind Abstract Data Types is to encapsulate the entire implementation of a data structure, and provide just a well defined interface for manipulating it. The benefit of this approach is that it provides a clean separation. The data type can be implemented with no knowledge of the application which might end up using it, and the application can be implemented with no knowledge of the implementation details of the data type. Both sides simply write code based on the interface which works like a contract to explicitly state what is required and what can be expected.

On the other hand, one of the costs of this approach is tightly connected with the abstract nature of the interface. The point of abstracting an interface is to remove unnecessary details. This is good for introductory computing students, but is bad for kernel programmers. In kernel programming. performance is very important, coming as a close third after correctness and maintainability, and sometimes taking precedence over maintainability. Not all code paths in the kernel are performance-critical, but many are, and the development process benefits from the same data structures being using in both performance critical and less critical paths. So it is essential that data types are not overly abstracted, but that all details of the implementation are visible so that the programmer can make optimal choices when using them.

So the first principle of data structures in the kernel is not to hide detail. To see how this applies, and to discover further principles from which to extract patterns, we will explore a few of the more successful data types used in the Linux kernel.

Linked Lists

Starting simply, the first data type we will explore are doubly linked lists. These are implemented by a single include file, <linux/list.h>. There is no separate ".c" file with any library of support routines. All of the code for handling linked lists is simple enough to be implemented using inline functions. Thus it is very easy to use this implementation in any other (GPLv2-licensed) project.

There are two aspects of the "list.h" lists which are worth noting as they point to possible patterns. The first is struct list_head, which serves not only as the head of a list, but also as the anchor in items that are on a list. Your author has seen other linked list implementations which required that the first and second element in any data structures meant to be stored in lists be the "next" and "previous" pointers, so that common list-walking code could be used on a variety of different data structures. Linux kernel lists do not suffer from this restriction. The list_head structure can be embedded anywhere in a data structure, and the list_heads from a number of instances of that structure can be linked together. The containing structure can be found from a ->next or ->prev pointer using the list_entry() macro.

There are at least two benefits of this approach. One is that the programmer still has full control of placement of fields in the structure in case they need to put important fields close together to improve cache utilization. The other is that a structure can easily be on two or more lists quite independently, simply by having multiple struct list_head fields.

This practice of embedding one structure inside another and using container_of() (which is the general form of list_entry()) to get the parent from the child structure is quite common in the Linux kernel and is somewhat reminiscent of object oriented programming. The container is like a subtype of the embedded structure.

The other noteworthy aspect of list.h is the proliferation of "for_each" macros - the macros that make it easy to walk along a list looking at each item in turn. There are 20 of them (and that isn't counting the four more in rculist.h which I'll choose to ignore in the hope of brevity).

There are a few different reasons for this. The simplest are that

• We sometimes want to walk the list in the "reverse" direction (following the "prev" link). There are five macros that go backward, and 15 that go forward.

• We sometimes want to start in the middle of a list and "continue" on from there, so we have four "continue" macros and three "from" macros which interpret that starting point slightly differently.

• We sometimes want to work with the struct list_head embedded in the target structure, but often we really want to use the list_entry() macro to get the enclosing structure; we find it easiest if the "for_each" macro does that for us. This provides the "entry" versions of the "for_each" macro, of which there are 13 (more than half).

Getting to the more subtle reasons, we sometimes want to be able to delete the "current" item without upsetting the walk through the list. This requires that a copy of the "next" pointer be taken before providing "this" entry to be acted upon, thus yielding the eight "safe" macros. An "ADT" style implementation of linked lists would quite likely only provide "safe" versions so as to hide these details. However kernel programmers don't want to waste the storage or effort for that extra step in the common case were it isn't needed.

Then there is the fact that we actually have two subtly different types of linked lists. Regular linked lists use struct list_head as the head of the list. This structure contains a pointer to the start and to the end. In some use cases, finding the end of the list is not needed, and being able to halve the size of the head of the list is very valuable. One typical use case of that kind is a hash table where all these heads need to go in an array. To meet this need, we have the hlist, which is very similar to the regular list, except that only one pointer is needed in struct hlist_head. This accounts for six of the different "for_each" macros.

If we had every possibly combination of forward or reverse, continue or not, entry or not, safe or not, and regular or hlist, we would have 32 different macros. In fact, only 19 of these appear to have been needed and, thus, coded. We certainly could code the remaining eleven, but as having code that is never used tends to be frowned upon, it hasn't happened.

The observant reader will have noticed a small discrepancy in some of the above numbers. Of the 20 macros, there is one that doesn't fit the above patterns, and it drives home the point that was made earlier about kernel programmers valuing performance. This final "for_each" macro is __list_for_each(). All of the other macros use the "prefetch" function to suggest that the CPU starts fetching the ->next pointer at the start of each iteration so that it will already be available in cache when the next iteration starts (though the "safe" macros actually fetch it rather than prefetch it). While this will normally improve performance, there are cases when it will slow things down. When the walk of the list will almost always abort very early - usually only considering the first item - the prefetch will often be wasted effort. In these cases (currently all in the networking code) the __list_for_each() macro is available. It does not prefetch anything. Thus people having very strict performance goals can have a better chance of getting the performance they want.

So from this simple data structure we can see two valuable patterns that are worth following.

• Embedded Anchor: A good way to include generic objects in a data structure is to embed an anchor in them and build the data structure around the anchors. The object can be found from the anchor using container_of().

• Broad Interfaces: Don't fall for the trap of thinking that "one size fits all". While having 20 or more macros that all do much the same thing is uncommon, it can be a very appropriate way of dealing with the complexity of finding the optimal solution. Trying to squeeze all possibilities into one narrow interface can be inefficient and choosing not to provide for all possibilities is counter-productive. Having all the permutations available encourages developers to use the right tool for the job and not to compromise. In 2.6.30-rc4, there are nearly 3000 uses of list_for_each_entry(), about 1000 of list_for_each_entry_safe(), nearly 500 of list_for_each(), and less than 1000 of all the rest put together. The fact that some are used rarely in no way reduces their importance.

RB-trees

Our next data structure is the RB-Tree or red-black tree. This is a semi-balanced, binary search tree that generally provides order "log(n)" search, insert, and delete operations. It is implemented in <linux/rbtree.h> and lib/rbtree.c. It has strong similarities to the list.h lists in that it embeds an anchor (struct rb_node) in each data structure and builds the tree from those.

The interesting thing to note about rbtree is that there is no search function. Searching an rbtree is really a very simple operation and can be implemented in just a few lines as shown by the examples at the top of rbtree.h. A search function certainly could be written, but the developer chose not to. The main reason, which should come as no surprise, is performance. To write a search function, you need to pass the "compare" function into that search function. To do that in C, you would need to pass a function pointer. As compare functions are often very simple, the cost of following the function pointer and making the function call would often swamp the cost of doing the comparison itself. It turns out that having the whole search operation compiled as one function makes for more efficient code. The same performance could possibly be achieved using inline functions or macros, but given that the search function itself is so short, it hardly seems worthwhile.

Note also that rbtree doesn't exactly provide an insert function either. Rather, the developer needs to code a search; if the search fails, the new node must be inserted at the point where it was found not to exist and the tree must be rebalanced. There are functions for this final insertion and rebalancing as they are certainly complex enough to deserve separate functions.

By giving the developer the responsibility for search and for some of insert, the rbtree library actually is giving a lot of valuable freedom. The pattern of "search for an entry but if it isn't there, insert one" is fairly common. However the details of what happens between the "search" and "add" phases is not always the same and so not something that can easily be encoded in a library. By providing the basic tools and leaving the details up to the specific situation, users of rbtree find themselves liberated, rather than finding themselves fighting with a library that almost-but-not-quite does what they want.

So this example of rbtrees re-enforces the "embedded anchors" pattern and suggests a pattern that providing tools is sometimes much more useful than providing complete solutions. In this case, the base data structures and the tools required for insert, remove, and rebalance are provided, but the complete solution can still be tailored to suit each case.

This pattern also describes the kernel's approach to hash tables. These are a very common data structure, but there is nothing that looks even vaguely like a definitive implementation. Rather the basic building blocks of the hlist and the array are available along with some generic functions for calculating a hash (<linux/hash.h>). Connecting these to fit a given purpose is up to the developer.

So we have another pattern:

• Tool Box: Sometimes it is best not to provide a complete solution for a generic service, but rather to provide a suite of tools that can be used to build custom solutions.

Radix tree

Our last data structure is the Radix tree. The Linux kernel actually has two radix tree implementations. One is in <linux/idr.h> and lib/idr.c, the other in <linux/radix-tree.h> and lib/radix-tree.c. Both provide a mapping from a number (unsigned long) to an arbitrary pointer (void *), though radix-tree also allows up to two "tags" to be stored with each entry. For the purposes of this article we will only be looking at one of the implementations (the one your author is most familiar with) - the radix-tree implementation.

Radix-tree follows the pattern we saw in list.h of having multiple interfaces rather than trying to pack lots of different needs into the one interface. list.h has 20 "for_each" macros; radix-tree has six "lookup" functions, depending on whether we want just one item or a range (gang lookups), or whether we want to use the tags to restrict the search (tag lookups) and whether we want to find the place where the pointer is stored, rather than the pointer that is stored there (this is needed for the subtle locking strategies of the page cache).

However radix-tree does not follow the embedded anchor pattern of the earlier data structures, and that is why it is interesting. For lists and rbtree, the storage needed for managing the data structure is exactly proportional to the number of items in the data structure on a one-to-one basis, so keeping this storage in those item works perfectly. For a radix-tree, the storage needed is a number of little arrays, each of which refers to a number of items. So embedding these arrays, one each, in the items cannot work. This means that, unlike list.h and rbtree, radix-tree will sometimes need to allocate some memory in order to be able to add items to the data structure. This has some interesting consequences.

In the previous data structures (lists and rbtrees), we made no mention of locking. If locking is needed, then the user of the data structure is likely to know the specific needs so all locking details are left up to the caller (we call that "caller locking" as opposed to "callee locking". Caller locking is more common and generally preferred). This is fine for lists and rbtrees as nothing that they do internally is affected particularly by locking.

This is not the case if memory allocation is needed, though. If a process needs to allocate memory, it is possible that it will need to sleep while the memory management subsystem writes data out to storage to make memory available. There are various locks (such as spinlocks) that may not be held while a process sleeps. So there is the possibility for significant interaction between the need to allocate memory internally to the radix-tree code, and the need to hold locks outside the radix-tree code.

The obvious solution to this problem (once the problem is understood) is to preallocate the maximum amount of memory needed before taking the lock. This is implemented within radix-tree with the radix_tree_preload() function. It manages a per-CPU pool of available radix_tree nodes and makes sure the pool is full and will not be used by any other radix-tree operation. Thus, bracketing radix_tree_insert() with calls to radix_tree_preload() and radix_tree_preload_end() ensures that the radix_tree_insert() call will not fail due to lack of memory (though the radix_tree_preload() might) and that there will be no unpleasant interactions with locking.

Summary

So we can now summarize our list of design patterns that we have found that work well with data structures (and elsewhere) in the kernel. Those that have already been detailed are briefly included in this list too for completeness.

• Embedded Anchor. This is very useful for lists, and can be generalized as can be seen if you explore kobjects (an exercise left to the reader).

• Broad Interfaces. This reminds us that trying to squeeze lots of use-cases in to one function call is not necessary - just provide lots of function calls (with helpful and (hopefully) consistent names).

• Tool Box. Sometimes it is best not to provide a complete solution for a generic service, but rather to provide a suite of tools that can be used to build custom solutions.

• Caller Locks. When there is any doubt, choose to have the caller take locks rather than the callee. This puts more control in that hands of the client of a function.

• Preallocate Outside Locks. This is in some ways fairly obvious. But it is very widely used within the kernel, so stating it explicitly is a good idea.

Next week we will complete our investigation of kernel design patterns by taking a higher level view and look at some patterns relating to the design of whole subsystems.

Exercises

For those who would like to explore these ideas further, here are some starting points.

1. Make a list of all data structures that are embedded, by exploring all uses of the "container_of" macro. Of these, make a list of pairs that are both embedded in the same structure (modeling multiple inheritance). Comment on how this reflects on the general usefulness of multiple inheritance.

2. Write a implementation of skiplists that would be suitable for in-kernel use. Consider the applicability of each of the patterns discussed in this article. Extra credit for leveraging list.h lists.

3. Linux contains a mempool library which radix-tree chooses not to use, preferring it's own simple pool (in radix_tree_preload). Examine the consequences of changing radix-tree to use mempool, and of changing mempool to be usable by radix-tree. Provide patches to revert this design choice in radix-tree, or a pattern to explain this design choice.

4. Compare the radix-tree and idr implementations to see if one could be implemented using the other without sacrificing correctness, maintainability or performance. Provide either an explanation of why they should stay separate, or a patch to replace one with the other.

Comments (18 posted)

Patches and updates

Kernel trees

• Greg KH: Linux 2.6.27.25 . (June 16, 2009)

• Greg KH: Linux 2.6.29.5 . (June 16, 2009)

• Thomas Gleixner: 2.6.29.4-rt17 . (June 13, 2009)

• Thomas Gleixner: 2.6.29.4-rt18 . (June 16, 2009)

• Thomas Gleixner: 2.6.29.4-rt19 . (June 16, 2009)

Core kernel code

• Paul Mackerras: lib: Provide generic atomic64_t implementation . (June 16, 2009)

• Lennart Poettering: [PATCH] scheduler: introduce SCHED_RESET_ON_FORK scheduling policy flag, fourth try . (June 16, 2009)

• Fabio Checconi: Use EDF to throttle RT task groups . (June 16, 2009)

Development tools

• Steven Rostedt: [GIT PULL][for 2.6.32] lockless ring buffer . (June 13, 2009)

• Steven Rostedt: [GIT PULL][for 2.6.32] new event print tag language . (June 13, 2009)

• Jon Masters: Hardware Latency Detector (formerly SMI detector) . (June 13, 2009)

• Mathieu Desnoyers: LTTng 0.140 for Linux 2.6.30 . (June 13, 2009)

• Ingo Molnar: Performance Counters for Linux . (June 16, 2009)

• Josh Stone: SystemTap release 0.9.8 . (June 16, 2009)

• Jason Baron: add syscall tracepoints . (June 16, 2009)

• Masami Hiramatsu: kprobes: Kprobes jump optimization support . (June 16, 2009)

Device drivers

• Alex Chiang: Dynamic ACPI-PCI binding . (June 13, 2009)

• Jerome Glisse: Radeon kernel modesetting patch (as staging) please review . (June 13, 2009)

• Thomas Hellstrom: drm: Add the TTM GPU memory manager subsystem. . (June 13, 2009)

• Magnus Damm: Driver Core: Add platform device arch data V3 . (June 16, 2009)

• Jerome Glisse: Radeon kernel modesetting patch (as staging) please review . (June 16, 2009)

• Dave Airlie: drm v2.6.31 merge (part 1) . (June 16, 2009)

• Rafael J. Wysocki: PM: Introduce core framework for run-time PM of I/O devices . (June 16, 2009)

• Alek Du: [PATCH] gpiolib: Add gpio_detect, gpio_debounce and gpio_alt_func features to GPIOLIB . (June 16, 2009)

• Scott James Remnant: proc connector: add event for process becoming session leader . (June 17, 2009)

Filesystems and block I/O

• Kiyoshi Ueda: [PATCH block#for-2.6.31] block: add request clone interface (v2) . (June 16, 2009)

• Denis Karpov: FS: userspace notification of errors . (June 16, 2009)

• Eric Paris: fsnotify notification infrastructure . (June 16, 2009)

• Chris Mason: Btrfs updates for 2.6.31-rc . (June 16, 2009)

• Jens Axboe: Per-bdi writeback flusher threads v10 . (June 16, 2009)

• Marco: Pramfs: Persistent and protected ram filesystem . (June 16, 2009)

• Ryo Tsuruta: [PATCH 1/2] dm-ioband: I/O bandwidth controller v1.12.0: main part . (June 17, 2009)

• David Howells: [RFC] AFS: Implement OpenAFS pioctls(version)s . (June 17, 2009)

Memory management

• Pekka J Enberg: Early boot SLAB for 2.6.31 . (June 13, 2009)

• Pekka J Enberg: SLUB: Out-of-memory diagnostics . (June 16, 2009)

• Jan Kara: Fix page_mkwrite() for blocksize < pagesize (version 3) . (June 16, 2009)

Networking

• Fischer, Anna: net/bridge: add basic VEPA support . (June 16, 2009)

Architecture-specific

• Arnd Bergmann: new files form asm-generic . (June 16, 2009)

• Wu Fengguang: HWPOISON: Intro (v5) . (June 16, 2009)

• Tejun Heo: percpu: convert most archs to dynamic percpu, take#3 . (June 17, 2009)

• Tejun Heo: x86,percpu: fix pageattr handling with remap allocator, take#4 . (June 17, 2009)

• Tejun Heo: x86,percpu: generalize 4k and lpage allocator . (June 17, 2009)

• Tejun Heo: percpu: teach lpage allocator about NUMA . (June 17, 2009)

Virtualization and containers

• Gregory Haskins: iosignalfd . (June 17, 2009)

Miscellaneous

• Andrew Morton: 2.6.31 -mm merge plans . (June 13, 2009)

• Karel Zak: util-linux-ng v2.15.1 . (June 13, 2009)

Page editor: Jake Edge

Distributions

News and Editorials

NixOS: purely functional system configuration management

System configuration management is a notoriously difficult task. Upgrading packages, editing configuration files, and so on; there will always come a time that it goes wrong. To mitigate this problem, Eelco Dolstra of Delft University of Technology invented another approach. He implemented a purely functional package manager called Nix, which means "nothing" in Dutch. Dolstra began his work on the Nix package manager as a part of his PhD research [PDF] at Utrecht University:

This operating system NixOS was the work of Armijn Hemel, who wrote a prototype for his Master's thesis. After this success, other developers joined. There are no official statistics of the number of users, but according to Dolstra the latest release of the Nix Packages collection had 19 contributors.

Imperative versus functional systems

To be able to grasp the basics of NixOS, we first have to distinguish between imperative systems and functional systems. Traditionally, software packages and configuration data (/bin and /etc, respectively) are imperative data structures. System administrators update them in-place with various administration commands, e.g. a RPM or APT package manager or a configuration tool such as Cfengine. This is analogous to how imperative programming languages such as C work. Each configuration action is stateful: it depends on the current state of the system and transforms this state. This has some fundamental consequences, including:

• No traceability: a specific configuration can generally not be recreated from scratch on a pristine system. That is, there may not be a record of the sequence of configuration actions over time. So it's not easy to reproduce a configuration.
• No predictability: if a configuration action acts upon an ill-defined state, the end result may be equally ill-defined and thus unpredictable.
• No rollbacks: if the user upgrades his system configuration (e.g. by upgrading a set of packages), this is a destructive process and undoing this is hard. Possible solutions are reverting to a backup or installing previous versions of the packages, but both solutions are error-prone.

In contrast, NixOS uses a functional approach, analogous to functional programming languages like Haskell. As Dolstra and Hemel state in their paper Purely Functional System Configuration Management [PDF]:

The functional approach has several advantages:

• Traceability: a configuration can be realised deterministically from a formal description and reproduced easily on another machine.
• Predictability: realisations of a configuration are not stateful, and hence upgrading a configuration is as safe as installing from scratch.
• Rollbacks: configuration changes are not destructive. As a result, the user is always able to roll back to a previous configuration.

How does NixOS work?

All this sounds exciting, but is it more than an academic exercise? How does NixOS work in practice? To put this to the test, your author downloaded the latest ISO of the installation CD. This CD contains a basic NixOS installation and doesn't do any installation preparation. So the user has to partition and format the drives himself and mount it on the target file system. The installation procedure is explained in an online manual.

The installation itself is uncommon, already showing signs of the functional nature of the operating system. The user has to write a description of the configuration to a file on the target file system. This file contains a 'Nix expression' that defines the root file system, kernel modules and services. Fortunately, the user can generate an initial configuration with the command nixos-hardware-scan. The nixos-install command reads this file and installs the system.

The result is a bare bones Linux distribution with the Nix package management system and the Upstart init system. The Nix package collection contains about 2200 software packages. That makes it a rather small distribution, but it is usable, as it contains server software such as Apache and SSH, and desktop software such as X.org 7.4, KDE 4.2, parts of Gnome, Firefox and more. However, the curious user will soon find other signs of the strangeness of this distribution. In the filesystem layout, for example: there's no /sbin, /usr/ or /lib in the filesystem. There's only one symlink in /bin, /bin/sh, because Glibc's system() function hard codes the location of the shell, as many other programs do. Even most files in /etc are symlinks.

All this is by design: to be able to work in a purely functional way, all static parts of the NixOS operating system are stored as immutable files in directories under /nix/store. Each package has a 'Nix expression', which is a function that builds and installs this package from source. The build scripts store the built packages in the Nix store. Each package is stored in a directory with a name that begins with a 160-bit cryptographic hash of all inputs involved in building the package, for example 22bharrqlcisnwa11a5qr0xazgvv64hk-firefox-3.5b4. This means that any change to an input value causes the package to be rebuilt in a different path, which has as a side effect that previous versions of the package are left untouched. Input values include the sources of the package, the build script, any arguments or environment variables passed to the build script, and build time dependencies. Each package directory contains bin, lib, man, and other sub-directories for the package and is read-only.

The same scheme works for system configuration files and control scripts. So the system has Nix expressions for sshd_config, to build the Linux kernel, to build initrd, for boot scripts, etc. There's also a top-level Nix expression, system.nix, that builds the entire system configuration by calling all expressions. The output is an activation script that can be executed to make this configuration the current configuration of the system. For example, it modifies the GRUB boot menu to boot the system with the new configuration. Previous configurations are retained in the boot menu to roll back. The nixos-rebuild switch command builds system.nix, makes it the default configuration in the GRUB boot menu and calls the activation script.

By not storing components such as libraries, header files or programs in global locations, all packages are forced at build time to use a specific version of their dependencies, located in the Nix store. To make this happen, the developers have patched Glibc, GCC and the dynamic linker ld to not search files in any default locations. So if a dynamic library is not explicitly declared with its full path in an executable, the dynamic linker will not find it. This also means that if the developer fails to specify a dependency explicitly in the Nix expression language, the package will fail deterministically, even if the dependency already happens to be available in the Nix store.

Advantages and disadvantages

Fortunately the user doesn't have to know about the Nix store. The user doesn't have to type /nix/store/22bharrqlcisnwa11a5qr0xazgvv64hk-firefox-3.5b4/bin/firefox to start their favorite browser. Nix creates directory trees of symlinks to all activated components and calls them user environments, which also reside in the Nix store. After each Nix package action, such as an install or a rollback, a new generation is made, which is a symlink outside the store that points to a user environment in the store. All generations of a user are grouped together in a profile. The user's current profile is pointed to by the symlink ~/.nix-profile. Putting the directory ~/.nix-profile/bin in the user's PATH environment variable completes the picture. As a consequence, non-privileged users can also securely install software. If a user installs a package that another user has already installed previously, the package won't be built or downloaded a second time.

Apart from the obvious advantages of predictability and rollbacks, it's even possible to copy a package from one machine to another in NixOS. The command nix-copy-closure copies a Nix store path along with all its dependencies to or from another machine via the ssh protocol. It does this efficiently, because it doesn't copy store paths that are already present on the target machine. This makes Linux packages literally "portable".

One major drawback to the functional model is that it requires significant disk space. This is understandable as each time the user makes a change to his configuration a new package will be added without overwriting the older one. In the worst case disk space doubles, for example when the C library or compiler is changed, propagating to all other packages. Even if the user removes a component, Nix doesn't actually remove the component from the Nix store, because it might still be in use by another user's environment or be a dependency of another component. Moreover, if the component were removed, it would no longer be possible to perform a rollback. However, the user can always remove any old generations of his profile by nix-env --remove-generations old and then he can execute a garbage collection of all unneeded components with nix-store --gc. If your hardware is not supported out-of-the box, it can be a challenge to write the correct Nix expression to load the firmware. There's also no distinction between a stable and an unstable branch, so things tend to break now and then. Fortunately, a broken system can be rolled back easily.

One final apparent drawback is that the NixOS directory structure doesn't comply with the Linux Standard Base. However, according to Dolstra this is not such a big problem as it seems:

The future: declarative specifications of networks

Of course the work on NixOS is ongoing. The developers are currently working on a branch named modular-nixos to make it easier to extend NixOS with hardware support or system services. There's also a research project about distributed deployment. Dolstra explains this:

Conclusion

The purely functional model of Nix and the cryptographic hashing scheme of the Nix store give the user some important features that are lacking in most Linux distributions. It makes one wonder why enterprise Linux distributions haven't picked up this approach (or a more LSB-compliant version of it). A drawback is the amount of disk space and bandwidth used when upgrading a fundamental dependency. Perhaps the most pressing issue is that it requires a radically different mindset from the user.

Comments (15 posted)

New Releases

Karmic Alpha 2 released

The second alpha of Ubuntu Karmic Koala (9.10) is out. Kubuntu and Xubuntu variants are also available. "Alpha 2 includes a number of software updates that are ready for large-scale testing. Please refer to http://www.ubuntu.com/testing/karmic/alpha2 for information on changes in Ubuntu."

Full Story (comments: 1)

Fedora Electronic Lab 11 Leonidas Released

The Fedora Electronic Lab (FEL) is an official Fedora spin. The fourth consecutive release of FEL (Fedora 11 version) is now available. "Fedora Electronic Lab 11 Leonidas provides a vibrant environment for designing and simulating ASIC design and embedded design. The opensource EDA solutions are composed to satify high-end mixed-signal hardware design flows from design specification to final project handoff. This release comprises Perl modules to facilitate both design, HDL code generation and brings additional support for Engineering Change Order (ECO). After post chip fabrication, evaluation boards of those chips can also be designed."

Full Story (comments: none)

Announcing LXDE Fedora Remix 11

A community remix of Fedora 11 featuring the LDXE desktop is available. "It is a Live CD with LXDE as the default desktop environment. LXDE is a new light weight desktop environment. Refer to http://lxde.org/ for more details. This Live CD is only available for Intel compatible 32-bit systems currently."

Full Story (comments: none)

Distribution News

Fedora

Fedora 9 End Of Life (EOL)

Now that Fedora 11 is out the end-of-life for Fedora 9 is in sight. Specifically support for F9 ends July 10, 2009. "With the release of Fedora 11 now past us, it's come time to remind folks that per the release policy, maintenance for the N-2 Fedora release ends one month after the Fedora N comes out."

Full Story (comments: 3)

Fedora Board Recap 2009-06-11

Click below for a brief recap of the June 11, 2009 meeting of the Fedora Advisory Board. Topics include What is Fedora, Status of sponsorship work, Fedora 11 Release Follow-up, Security notification plan, PPC as a secondary arch, Toxicity--follow-ups, and Security notification plan.

Full Story (comments: none)

Slackware Linux

Slackware X.org overhaul

Slackware current has gotten a major overhaul of X.Org, according to the June 10 changelog entry. "This batch of updates includes a major overhaul of X.Org -- thanks to Robby Workman for doing a huge amount of work updating the build scripts and testing everything. A large number of packages were recompiled or upgraded to drop references to the now-obsolete libxcb-xlib and libXaw8 libraries. We have workarounds in place for old binaries so it wasn't strictly required, but recompiling anyway gives a cleaner system. Enjoy!"

Full Story (comments: none)

SUSE Linux and openSUSE

Stephen Shaw has been appointed as openSUSE Board Member

The openSUSE Board has announced that Stephen Shaw is its newest member. "Federico Mena-Quintero has ceded his seat on the openSUSE Board. We are all sad to see him go, but know we are grateful for his continued and ongoing contributions to the openSUSE Project."

Comments (none posted)

Ubuntu family

Ubuntu archive reorganisation: summary of pending work

Ubuntu is planning on a major reorganization of its repositories. "We divided the archive into components before there was even such a thing as Kubuntu, let alone the rich landscape of Ubuntu derivatives that exists today. The modern Ubuntu community has teams of developers working on everything from mobile devices through video editing to cloud computing. It is no longer clear to users which packages are recommended for and supported on which flavours of Ubuntu, and the simple division of the development team into ubuntu-core-dev and MOTU [Masters of the Universe] is looking increasingly artificial and does not do a good job of modelling how our development community really works. To fix these problems, we need to evolve our original design into a more fine-grained system."

Full Story (comments: none)

New Distributions

TITAN LEV

TITAN LEV is a Linux desktop OS for users with Windows experience, from the Israeli company Affordy. It comes bundled with 150 applications and supports English, Russian, Polish, Spanish, Portuguese, Hebrew and Arabic. (Found on blogs.zdnet)

Comments (none posted)

Distribution Newsletters

CentOS Pulse #0902

The CentOS Pulse for June 16, 2009 is out. "This issue of CentOS Pulse looks at the Artwork SIG, desktop multimedia capabilities and interesting threads in the CentOS community. An overview of the important security updates and upcoming events are included as well."

Comments (none posted)

DistroWatch Weekly, Issue 307

The DistroWatch Weekly for June 15, 2009 is out. "The delayed Fedora 11 was finally released last week. Does the new version of the popular distribution live up its standards? Did the delay help to squash all the bugs? And how does it fare in comparison with other desktop Linux products? Read our first-look review to find out. This week also sees the release of a new project to create more up-to-date installation media for FreeBSD. Currently shipping a 32-bit Xfce desktop, the project hopes to expand to many other areas, as needed. Meanwhile Fedora's Leonidas release is in full swing, but some users are encountering an issue when installing via the live CD as the system cannot yet boot from the default ext4 file system. Read on to discover the simple fix! Also, Debian derivative distribution sidux has copped some heat over its decision to remove non-free firmware from its 2.6.30 kernel, while Novell gets its users to help advertise their products with an online "Custom Geeko" creation tool. Finally, don't miss the freshly posted development roadmaps for Mandriva Linux 2010 and Fedora 12. Happy reading!"

Comments (none posted)

Fedora Weekly News 180

The Fedora Weekly News for June 15, 2009 is out. "In this week's issue, we open with useful links announcing the 'hot-off-the-bitpress' Fedora 11 (Leonidas) release, and also reminders about voting for the code name for Fedora 12 and other Fedora elections. There are many FUDCons, FADs and other Fedora events, helpfully listed as well. From Planet Fedora, two interesting samples: news from Fedora blogs and contributors including an interview with Eric Sandeen about ext4, linux filesystems and Fedora 11, and rave reviews on Presto, Fedora 11's enhanced DeltaRPM service that can be configured. In the Quality Assurance beat, review of the Bugzappers weekly meeting and changes this will have to triage work, as well as availability of a set of Fedora 11 delta ISO images." ...and much more.

Full Story (comments: none)

The Mint Newsletter - issue 86

This issue of the Mint Newsletter covers the release of Linux Mint 7 x64 RC1, a call for promotional materials, and more.

Comments (none posted)

OpenSUSE Weekly News/76

This issue of the OpenSUSE Weekly News covers OpenSUSE at the Southeast Linuxfest, Status Reports from Google Summer of Code, susegeek : openSUSE 11.2 Milestone 2 step by step install procedure, OpenSUSE-EDU, and more.

Comments (none posted)

Ubuntu Weekly News #146

The Ubuntu Weekly News for June 14, 2009 is out. "In this issue we cover SanDisk collaborates to improve Ubuntu netbook SSD performance, MOTU Council Results, Ubuntu Stats, Calling all LoCo Teams!, In the Press & Blogosphere, Upcoming Meetings & Events, Updates & Security, and much, much more!"

Full Story (comments: none)

Newsletters and articles of interest

Ubuntu aims for ten-second boot time with 10.04 (ars Technica)

ars Technica reports on efforts being made to reduce the boot time of Ubuntu. "In a presentation at the Ubuntu Developer Summit in Barcelona, developer Scott James Remnant noted that boot time decreased from 65 seconds in version 8.10 to only 25 seconds in 9.04. This is already a substantial improvement, but he believes that there is still room for more aggressive optimization. Canonical, the company behind Ubuntu, will continue pushing the limits of boot performance during the upcoming development cycle for Ubuntu 9.10, which is codenamed Karmic Koala. According to Remnant, the company aims to achieve a ten-second boot time next year for Ubuntu 10.04, the release that will follow after Karmic."

Comments (68 posted)

Distribution reviews

What's new in Fedora 11 (The H Open)

Thorsten Leemhuis reviews Fedora 11. "It's not just the new design and updated software that brings a sparkle to the eleventh version of Fedora (Leonidas), there are also a whole raft of technical enhancements. Fedora once again finds itself in the vanguard – expect to see many of these changes coming to other Linux distributions in the near future."

Comments (none posted)

Hands-on: new Fedora release goes up to 11 but doesn't rock (Ars Technica)

Ryan Paul reviews Fedora 11. "Fedora tends to ride the cutting edge of desktop Linux technology and is often the first distribution to include new upstream features. This makes Fedora a compelling choice for developers and Linux enthusiasts who like to stay ahead of the curve. The downside of dogfooding the latest stuff is that users occasionally have to contend with a release that is slightly undercooked. Fedora 11 isn't raw but it falls short of well done."

Comments (none posted)

Page editor: Rebecca Sobol

Development

Rob Savoye on the Cygnal rich media server

On June 11, 2009 Rob Savoye spoke to the Boulder Linux Users Group about the Cygnal rich media server project.

Cygnal is part of the Gnash SWF movie player project and development is being funded by Open Media Now (OMN). Open Media Now projects include Gnash, Cygnal and Ming, an SWF output library and PHP module. OMN is particularly focused on performing legal, clean-room reverse engineering of proprietary protocols. Supporters of the project include the Electronic Frontier Foundation, the Software Freedom Law Center, the Free Software Foundation, and notable individuals such as John Gilmore, Bob Young, and Mark Shuttleworth, along with a community of developers.

Rob discussed the underlying concept of rich media, also known as interactive media. Unlike basic streaming media, rich media is interactive, involving the bidirectional transport of complex objects. Typical uses include video conferencing, electronic whiteboards, and other types of groupware.

Cygnal is loosely aimed at being a replacement for Adobe's commercial Flash Media Server, with enhancements. The motivation behind Cygnal was to create a 100% free video conferencing server. Cygnal development was started because there were no really good free software rich media server projects available. Compared to the commercial offering, the project aims to be more secure, lack a central point of control, and have a greater respect for user privacy.

Cygnal supports a variety of patent-free and proprietary CODECs and protocols including HTTP, Adobe's proprietary Real Time Messaging Protocol (RTMP), RTMPT, and ActionScript protocols as well as the free Ogg Vorbis audio and Ogg Theora video protocols from the Xiph.org project. Cygnal also aims to support PHP and Python inside of a server sandbox environment.

Some of the Cygnal features that Rob highlighted include built-in clustering and load balancing and support for Oggz chopping so that one could, for example, edit video on a distant server by remote control from a cell phone. For a complete feature list, consult the Cygnal web site. Rob pointed out inefficiencies with streaming video servers, such as YouTube, that a rich media server can greatly improve upon. If you want to move around a video randomly, or play just the last few seconds, with YouTube it is necessary to load the entire video first. The bidirectional capability of a rich media server would allow a finer level of control over which data is sent over the network.

Cygnal supports statistics gathering for tuning and optimizing its behavior. It is possible to connect to the server's system console via tcp, and the plan is to be able to remotely dump statistics while the server is operating. Rob emphasized that there was a fine line between gathering statistics and invading privacy, so the project would aim to protect privacy first.

A large amount of work has been done on the legal clean-room reverse engineering of the RTMP protocol and the ActionScript class library. RTMP has a lot of advanced capabilities including the transfer of ActionScript objects, the ability to drop frames when network congestion is detected, and the ability to seek within a video stream. All of this functionality makes decoding the protocol a fairly difficult job. RTMP is not publicly documented, so reverse engineering is the only way to gain access to its inner workings.

A number of special techniques and limitations were used in the reverse engineering process in order to keep everything legal. It is critical that people doing the reverse engineering don't install any Adobe software so as to avoid agreeing to the EULA. The EULA forces the user to agree not to redistribute any software relating to decoding Adobe's proprietary protocols. It is legal to have a third party install the commercial software and agree to the EULA, then have a developer sniff network packets and sift through the voluminous hex dumps. Adobe has already shown their legal teeth, they recently sent a DMCA letter to SourceForge concerning the rtmpdump utility (documented on FlashComGuru). Rob described the reverse engineering process as sneaky, brutal, and fun.

At the heart of Cygnal is the network engine. It must support multiple threads for incoming connections. As soon as the maximum number of threads has been reached, the engine multiplexes the i/o operations through the existing threads. This is currently an area of experimentation and flux within the project; time and experience will eventually reveal the most efficient approach to achieving good operation. Rob emphasized the network engine's use of zero-copy techniques; messages are put into a queue and pointers are passed around, resulting in very good performance. Interestingly, Rob mentioned that the PowerTop utility was one of the more useful tools that he used for optimizing the code.

Rob pointed out a few weaknesses in the Linux kernel and libraries that required new implementations of existing software. He pointed out that the Linux Asynchronous I/O library uses three threads per connection and was a source of much inefficiency. Also, using jemalloc [pdf] (A Scalable Concurrent malloc Implementation for FreeBSD) instead of malloc can improve the server's speed by up to 15% on multi-core machines.

The Cygnal project is in need of a wide variety of help including performing language translations, testing and feedback, bug reporting, documentation, build farm maintenance, and donations of cash (and beer). Volunteers are encouraged to help out, or, at least, to send Rob some free beer.

Comments (1 posted)

System Applications

Database Software

MySQL Community Server 5.0.83 released

Version 5.0.83 of MySQL Community Server has been announced. "MySQL Community Server 5.0.83, a new version of the popular Open Source Database Management System, has been released. This and future releases in the MySQL Community Server 5.0 series share version numbers with their MySQL Enterprise Server counterparts."

Full Story (comments: none)

PostgreSQL 8.4 Release Candidate 1 now available

Release Candidate 1 of the PostgreSQL 8.4 DBMS has been announced. "Final release of PostgreSQL 8.4 comes much closer today with our first Release Candidate. Now, we count on all of the PostgreSQL users and developers worldwide to test this Release Candidate to make sure that it is stable, reliable, secure and high-performance enough to be called an official PostgreSQL release. Please download 8.4 RC1 today and test it with your applications."

Comments (none posted)

PostgreSQL Weekly News

The June 14, 2009 edition of the PostgreSQL Weekly News is online with the latest PostgreSQL DBMS articles and resources.

Full Story (comments: none)

py-postgresql 0.9 released

Version 0.9 of py-postgresql has been announced. "I'm pleased to announce the release of py-postgresql 0.9.0 and 0.8.2. py-postgresql is a Python programmer's client (driver) for PostgreSQL and general toolkit for working with PostgreSQL in Python."

Full Story (comments: none)

SQLite release 3.6.15 announced

Release 3.6.15 of the SQLite DBMS has been announced, it includes a number of new features and some bug fixes.

Comments (none posted)

Device Drivers

CWiid project status update

Some progress has been made on the CWiid project, a collection of Linux tools written in C for interfacing to the Nintendo Wiimote. "As promised, the Motion Plus is out, and I'm actively working on CWiid again. Many outstanding tickets have been fixed and/or replied to, and I hope to continue resolving outstanding tickets as I add Motion Plus support."

Comments (1 posted)

Interoperability

Samba versions 3.3.5 and 3.2.12 are available

Stable version 3.3.5 and Maintenance Release version 3.2.12 of Samba have been announced. For V3.2.12: "This is the latest bug fix release for Samba 3.2 and is the version recommended for all production Samba servers running this release series."

Comments (none posted)

Printing

CUPS 1.4rc1 announced

Version 1.4rc1 of CUPS, the Common Unix Printing System, has been announced. "As per the CUPS Configuration Management Plan, we now start our two week "soak" of each release candidate. Once we are happy with the quality, we'll do the first stable release, 1.4.0. If you experience problems with the release candidate, please post your issues to the cups.general forum or mailing list. Confirmed bug reports should be posted to the Bugs & Features page. CUPS 1.4 adds over 67 new features and changes to CUPS 1.3.x, including greatly improved Bonjour/DNS-SD and Kerberos support, SNMP-based page accounting and monitoring, security and logging improvements to the scheduler, integration of the CUPS driver development kit components, and IPP/2.1 support."

Comments (none posted)

Telecom

Introducing the Android Scripting Environment

Google has announced a new Android Scripting Environment. "The Android Scripting Environment (ASE) brings scripting languages to Android by allowing you to edit and execute scripts and interactive interpreters directly on the Android device. These scripts have access to many of the APIs available to full-fledged Android applications, but with a greatly simplified interface that makes it easy to: * Handle intents * Start activities * Make phone calls * Send text messages * Scan bar codes * Poll location and sensor data * Use text-to-speech (TTS) * And more". (Thanks to Francisco Borges).

Comments (13 posted)

Web Site Development

Announcing bobo

The Bobo project has been launched. "Bobo is a light-weight framework for creating WSGI web applications. It's goal is to be easy to use and remember. You don't have to be a genius. It addresses 2 problems: - Mapping URLs to objects - Calling objects to generate HTTP responses ".

Full Story (comments: none)

web2py 1.64.0 released

Version 1.64.0 of web2py has been announced. "new features: - full Jython support, including xzJDBC for sqlite and postgresql. install jython and run: jython web2py.py -h - models are 2.5x faster - better LDAP support - custom forms".

Full Story (comments: none)

Desktop Applications

Audio Applications

lv2fil version 2.0 released

Version 2.0 of lv2fil has been announced. "Four-band parametric equaliser LV2 plugin. DSP code by Fons Adriaensen."

Full Story (comments: none)

Data Visualization

RRDtool 1.4rc2 announced

Version 1.4rc2 of RRDtool, a data visualization application for time-series data, has been announced. "RRDtool 1.4rc[2] opens the last phase of the road to the 1.4 release."

Comments (none posted)

Desktop Environments

GNOME Software Announcements

The following new GNOME software has been announced this week:

• Anjuta and gdl 2.27.3 (new features, bug fixes and code cleanup)
• Brasero 2.27.3 (bug fixes and translation work)
• Cheese 2.27.3 (new features, bug fixes and translation work)
• Deskbar-Applet 2.27.3 (translation work)
• Empathy 2.27.3 (bug fixes and translation work)
• Ethos 0.2.0 (initial release)
• Evince 2.27.3 (new features, bug fixes and translation work)
• Eye of GNOME 2.27.3 (bug fixes and translation work)
• File Roller 2.27.1 (new features, bug fixes and translation work)
• gbrainy 1.11 (new features, bug fixes and translation work)
• GCalctool 5.27.3 (bug fixes and translation work)
• Glade 3.6.5 (bug fixes and build improvements)
• GLib 2.21.2 (new features, bug fixes and translation work)
• gnome-applets 2.27.3 (bug fixes, documentation and translation work)
• gnome-bluetooth 2.27.6 (new features and bug fixes)
• Gnome Games 2.27.3 (new features, bug fixes and translation work)
• GNOME Media 2.27.3 (bug fixes and translation work)
• gnome-settings-daemon 2.27.3 (bug fixes and translation work)
• GNOME Utilities 2.27.2 (new feature and libgnome dependency removal)
• GOK 2.27.3 (bug fixes and translation work)
• GTK+ 2.17.2 (new features, bug fixes and translation work)
• Java ATK Wrapper 0.27.2 (bug fixes)
• krb5-auth-dialog 0.11 (new features, bug fixes, documentation and translation work)
• libchamplain 0.3.3 (new features and bug fixes)
• libvtemm 0.20.3 (API improvements)
• mousetweaks 2.27.3 (bug fixes and translation work)
• Orca 2.27.3 (new features, bug fixes and translation work)

You can find more new GNOME software releases at gnomefiles.org.

Comments (none posted)

KDE Software Announcements

The following new KDE software has been announced this week:

• Amarok 2.1.1 (new features and bug fixes)
• Cirkuit 0.1 (initial release)
• digiKam 1.0.0-beta1 (beta for KDE4)
• Firewall Builder 3.0.5 (bug fixes)
• Free Music Charts 1.3.1 (new features)
• Frescobaldi 0.7.11 (stability improvements)
• KAlarm 2.2.1/2.2.2 (bug fixes and translation work)
• KAlarm 2.2.1/2.2.3 (bug fixes)
• kikkanji 1.0 (initial release)
• KPackageKit 0.4.1.1 (bug fix)
• KpGen 1.1 Beta (new features, bug fixes and translation work)
• Maestro Metronome 1.2 (unspecified)
• MameExecutor 1.0 (new features, bug fixes and translation work)
• MameExecutor 1.0.1 (bug fix)
• Package KIO 0.2 (unspecified)
• PeaZip 2.6.2 (code cleanup)
• PlasmaNotify Firefox addon 0.3.0 (unspecified)
• 'Q' DVD-Author 1.9.0 (new features and bug fixes)
• VariCAD 2009 1.05 (new features)

You can find more new KDE software releases at kde-apps.org.

Comments (none posted)

samurai-x 0.2 released

Version 0.2 of samurai-x has been announced. "We are happy to release version 0.2 of samurai-x. samurai-x is a window manager written in pure python using ctypes, xcb and cairo. A lot has happened since version 0.1 including: * a new plugin system - the core samurai-x is now very small with all other functionality added via plugins * a new xcb binding - ooxcb - for more information see http://docs.samurai-x.org/ooxcb/. * lots of plugins! we now have plugins for most common features found in other window managers"

Full Story (comments: none)

Xorg Software Announcements

The following new Xorg software has been announced this week:

• dri2proto 2.1 (new feature)
• pixman 0.15.12 (ARM updates and clipping rule improvements)

More information can be found on the X.Org Foundation wiki.

Comments (none posted)

Desktop Publishing

pdfrecycle 0.05 released

Version 0.05 of pdfrecycle has been announced, it includes some new functionality. "pdfrecycle creates a PDF file by composing pages from other PDF files. It can add PDF bookmarks and metadata, scale, rotate and crop pages and put multiple logical pages onto each physical sheet."

Full Story (comments: none)

Graphics

cairo release 1.8.8 now available

release 1.8.8 of the cairo graphics library has been announced. "The cairo community is pleased to announce the 1.8.8 release of the cairo graphics library. This is the fourth update to cairo's stable 1.8 series and contains a small number of bug fixes (in particular a few corrections to the documentation and a few fixes in the FreeType font backend). This is being released just over six months after cairo 1.8.6."

Full Story (comments: none)

Inkscape prerelease 0.47 is out

Prerelease 0.47 of the Inkscape scalable vector graphics editor has been announced. "Later than expected we uploaded a prerelease of upcoming Inkscape 0.47. We are working on rescheduling the final release." Also, the winner of the Inscape 0.47 About Screen Contest has been announced.

Comments (none posted)

GUI Packages

PyQt 4.5.1 released

Version 4.5.1 of PyQt, a set of Python bindings for Qt, has been announced. "The highlights of this release include: - support for Python v3 - support for Qt v4.5 - a new Pythonic API for connecting signals and slots that doesn't require any knowledge of C++ data types - support for the GTK+ theme engine."

Full Story (comments: none)

Math Applications

openopt 0.24 released

Version 0.24 of openopt, a free numerical optimization framework, has been announced. "OpenOpt 0.24, a free Python-written numerical optimization framework with some own solvers and connections to tens of 3rd party ones, has been released. BSD license allows to use it in both free opensource and commercial closed-code software."

Full Story (comments: none)

Multimedia

Moovida Media Center 1.0.3 released

Version 1.0.3 of Moovida Media Center, a cross-platform and open-source Media Center written in Python, has been announced. "Important features of this release include a much faster media scanning and thumbnailing for videos and pictures, a better handling of plural forms in translations and updated French translations."

Full Story (comments: none)

Music Applications

a2jmidid version 5 released

Version 5 of a2jmidid has been announced. "a2jmidid is a project that aims to ease usage of legacy ALSA sequencer applications, in a JACK MIDI enabled system. a2jmidid implementation is based on jack-alsamidi-0.5 that is [almost] same as jackd ALSA "seq" MIDI backend, both created by Dmitry Baikov."

Full Story (comments: none)

zynjacku 5 released

Version 5 of zynjacku, a JACK-based, GTK2 host for LV2 synths, has been announced. "In this release: * slv2 is no longer required * cache list of suitable plugins * speedup plugin list window * new tool, zynspect, that can be used to list and inspect available lv2 plguins. * Fix assert when restoring rack presets * By default, sort plugins by name * Experimental support for dynmanifest extension. Combined with NASPRO allows loading ladspa plugins in lv2rack. * Set plugin GUI window's role to "plugin_ui" (for WM kludges etc) * single plugin mode for lv2rack * Hide external UIs when zynjacku/lv2rack quits".

Full Story (comments: none)

Web Browsers

Firefox 3.0.11 available for download

Firefox 3.0.11 has been released with several security fixes. See these known issues for more information or see the LWN vulnerability report for vendor responses.

Full Story (comments: none)

CPython in the web browser under Native Client

Mark Seaborn has announced an effort to put CPython into Google's Native Client. "I have been doing some work to extend Google's Native Client to support dynamic linking. For those who haven't heard of it, Native Client is a sandboxing system for running a subset of x86 code. It is proposed as a way of running native code inside web apps. One of my aims has been to get CPython working in the web browser under Native Client without having to modify CPython. I recently got to the point where modules from the Python standard library are importable under Native Client, including (as a demonstration) the Sqlite extension module."

Full Story (comments: none)

Languages and Tools

Caml

Caml Weekly News

The June 16, 2009 edition of the Caml Weekly News is out with new articles about the Caml language.

Full Story (comments: none)

Perl

Parrot 1.3.0 released

Version 1.3.0 of Parrot, a virtual machine aimed at running all dynamic languages, has been announced. This release includes numerous bug fixes and optimizations.

Full Story (comments: none)

PHP

PHP 5.2.10RC2 and PHP 5.3.0RC3 released

Versions 5.2.10RC2 and 5.3.0RC3 of PHP have been announced. "The PHP development team is proud to announce the second release candidate of PHP 5.2.10 (PHP 5.2.10RC2) and the third release candidate of PHP 5.3.0 (PHP 5.3.0RC3). These RCs focuses on bug fixes and stability improvements, and we hope only minimal changes are required for the next candidate or final stable releases."

Comments (none posted)

Python

Jython 2.5.0 Final is out

Version 2.5.0 Final of Jython, an implementation of Python in Java, has been announced. "Jython 2.5.0 brings us up to language level compatibility with the 2.5 version of CPython. This release has had a strong focus on CPython compatibility, and so this release of Jython can run more pure Python apps then any previous release."

Full Story (comments: none)

Python 3.1 Release Candidate 2

Version 3.1 Release Candidate 2 of Python has been announced. "Python 3.1 focuses on the stabilization and optimization of the features and changes that Python 3.0 introduced. For example, the new I/O system has been rewritten in C for speed. File system APIs that use unicode strings now handle paths with undecodable bytes in them. Other features include an ordered dictionary implementation, a condensed syntax for nested with statements, and support for ttk Tile in Tkinter."

Full Story (comments: none)

Announcing Python svn builds

The Python svn builds project has been announced. "The project aim is to provide developers (and non developers) tested python binary releases. The main feature is they install under /opt/opt-python-svnXXXXX2.7a0 directory. Doing so they could be installed side by side allowing module developers to test their modules against different builds."

Full Story (comments: none)

SIP 4.8.1 released

Version 4.8.1 of SIP has been announced. "SIP is a tool for generating Python modules that wrap C or C++ libraries. It is similar to SWIG. It is used to generate PyQt and PyKDE. SIP is licensed under the Python License and runs on Windows, UNIX, Linux and MacOS/X. SIP requires Python v2.3 or later. The main focus of this release is support for Python v3."

Full Story (comments: none)

Sphinx 0.6.2 released

Version 0.6.2 of Sphinx has been announced. "I'm proud to announce the release of Sphinx 0.6.2, which is a bugfix-only release in the 0.6 series. Sphinx is a tool that makes it easy to create intelligent and beautiful documentation for Python projects (or other documents consisting of multiple reStructuredText source files)."

Full Story (comments: none)

Python-URL! - weekly Python news and links

The June 10, 2009 edition of the Python-URL! is online with a new collection of Python article links.

Full Story (comments: 1)

Tcl/Tk

Tcl-URL! - weekly Tcl news and links

The June 14, 2009 edition of the Tcl-URL! is online with new Tcl/Tk articles and resources.

Full Story (comments: none)

Version Control

Torvalds: Happiness is a warm SCM

Linus Torvalds is happy about the 2.6.31 merge window, mostly because people have figured out how to use git most effectively. He also provides a bit more detail on the tradeoffs involved in using git (or any distributed SCM really) in a comment response. "Part of the problem is that 'git' is such a flexible tool that you can use it in various modes, and mix things up freely. The whole distributed nature means that there's no gatekeeper, you can do whatever you want. And the flexibility and power is good, but it does mean that it's also easy to make a mess of it - the old UNIX philosophy of giving people rope, and letting them hang themselves with it if they want to. [...] So it takes time for people (me included) to learn the rules that work. And it seems people are learning. And that feels really good."

Comments (4 posted)

Page editor: Forrest Cook

Linux in the news

Recommended Reading

Linux first to offer USB 3.0 driver (Linux Devices)

Linux Devices reports on the addition of USB 3.0 support in the upcoming 2.6.31 kernel. "A kernel hacker from Intel has posted a Linux driver for USB 3.0, making Linux the first operating system (OS) to support the new connectivity standard. Sarah Sharp announced that her xHCI-compliant driver for USB 3.0 will be supported in Linux 2.6.31 in September. The xHCI (Extensible Host Controller Interface) host controller driver is available now, and will be formally supported by the Linux kernel in September in version 2.6.31, reports Sharp, also known as the "Geekess." This should make Linux the first OS to offer a USB 3.0 driver, according to Sharp, who says she has worked a year and a half on the project."

Comments (48 posted)

Trade Shows and Conferences

KOffice 2009 Sprint In Berlin (KDEDot)

KDEDot reports on the KOffice Sprint. "Last weekend - it seems like yesterday and like a year ago at the same time - the KOffice team came to Berlin for the first post 2.0 sprint. Graciously hosted by KDAB and smoothly organized by Alexandra Leisse, this sprint was one of the most productive sprints ever for KOffice. Not only because there were many developers attending, among them three out of four of our KOffice Summer of Code students, but also because everyone was filled to the brim with joy and relief about having release 2.0 and eager to forge forwards to 2.1."

Comments (none posted)

Network Manager Sprint In Oslo (KDEDot)

KDE.News covers a recent Network Manager Sprint. "A small but intense code sprint took place in Oslo last weekend. Peder Osevoll Midthjell, Sveinung Dalatun and Anders Sandven, who work on mobile broadband connections for Linux as their thesis project, met with Darío Freddi, Will Stephenson and Frederik Gladhorn of KDE. Knut Yrvin spent his weekend with us to make us feel comfortable at Qt Software."

Comments (none posted)

The SCO Problem

SCO rises from the dead (Computerworld)

Steven J. Vaughan-Nichols reports that bankrupt SCO may have a buyer. "I've never been a fan of horror-movie series where no matter what happens to the baddie, such as Jason Voorhees from the Friday the 13th movies, he's up and ready to kill again in the next sequel. So, you can imagine just how pleased I am to see that SCO, just when it looked like it was dead as a doornail, came up with a buyer at the 11th hour and 59th minute. According to reports on Groklaw, Gulf Capital Partners LLC, a group formed by Stephen Norris of Stephen Norris & Co. Capital Partners, a private-equity firm, has offered to buy SCO, just as the company faced the end of the bankruptcy road. If the deal is real and goes through, SCO's nearly dead Unix business will continue, and, oh the pain of it all, so will its zombie-like lawsuits against IBM, Novell, and other Linux companies."

Comments (7 posted)

Linux Adoption

Is Android the key to the GNU/Linux desktop? Really? (Free Software Magazine)

Tony Mobily wonders if Android is the key to the Linux desktop on Free Software Magazine. "I don't know. First it was "network computing". Then it was Java. Now Android. As much as Android can be considered mature in terms of a phone-like environment. If used on a laptop, will it recognise your scanner or your printer? Will it recognise an external webcam? Will OpenOffice ever work on it? What about a secondary monitor? Will we ever be able to create proper Android applications in Python, or Ruby?"

Comments (17 posted)

Interviews

Interview with Intel's Pankaj Kedia at Computex 2009 (c|net)

c|net Asia features an interview with Intel's Pankaj Kedia, director of ecosystems for Intel's ultra-mobility group. "Why is Intel focusing on the Moblin OS rather than MS' Windows OS for its Atom and Moorestown platforms? Kedai: I think association of the Intel chipset and MS Windows OS is the best thing that differentiates with ARM-based Netbooks and MIDs which getting the spotlight during Computex 2009, such as Qualcomm's Snapdragon and Nvidia's Tegra-based Netbooks and MIDs. All those devices are based on the Android or Linux operation systems because of cost-efficiency issues but also because of lack of support from Microsoft. (This was also confirmed by Steve Guggenheimer, Microsoft's corporate VP for original equipment manufacturers during his interview at Computex)."

Comments (none posted)

Resources

More Pre Hacks and Inner Details Revealed (Palm Infocenter)

The Palm Infocenter has information about hacking the Palm Pre. "Intrepid hackers have accomplished everything from custom Universal Search engines to unlocking the three-page limitation in the Launcher, but the most truly impressive achievement thus far is the ability to directly install the 1700+ Optware Linux command-line packages - all without interfering with Palm's own built-in webOS applications."

Comments (none posted)

Reviews

Upbeat about Updates (Linux Journal)

Dave Phillips looks at updates to some audio software, including MusE, SuperCollider, Jackbeat, Rationale, SLV2, and midish. "MusE has had an uneven development history. The project first took shape as an audio/MIDI sequencer with notation capabilities until Werner Schweer (MusE's original designer) extracted the notation parts and turned them into the excellent MuseScore (MScore). Alas, directed work on MusE slowed for a while, but now we have a revived and revivified MusE 1.0 rc2. As the rc2 indicates, this version is the second release candidate, so if no further egregious bugs or annoyances are found, this version will serve as the 1.0 public release. The MusE developers encourage users to stress test this candidate as hard as possible."

Comments (none posted)

Open source Carrier Grade Linux middleware rev'd (LinuxDevices)

LinuxDevices reviews OpenSAF 3.0. "The OpenSAF project announced a new version of its open source High Availability (HA) middleware platform for Carrier Grade Linux (CGL) networking systems. Release 3.0 of the Open Service Availability Framework (OpenSAF) adds numerous platform management features, usability improvements, and support for Java APIs, says the project."

Comments (none posted)

Palm's Linux phone "exceptional," says reviewer (LinuxDevices)

LinuxDevices takes a look at the Palm Pre with screenshots and links to reviews. "Several reviews came to the same conclusion: The Pre may not be an Apple iPhone killer, especially with the arrival this week of the iPhone 3GS, but with its multitasking and synchronization features, it may give RIM's BlackBerry a run for its money in the corporate market. Others note that in order to do that, however, the device will first need to spread beyond the Sprint network, as planned for early next year, to Verizon Wireless and possibly AT&T. Palm will also need to quickly boost the number of available WebOS apps, say reviewers."

Comments (25 posted)

Miscellaneous

Linux Foundation Takes Training Online (Linux Journal)

Linux Journal takes a look at the Linux Foundation's online training. "The original sessions of the Linux Foundation Training Program, held at the annual Collaboration Summit, took the form of in-person, hands-on training, a method that provides students an unmatched learning opportunity, but limits the number of students the program can reach. The addition of online courses takes the program to the students, complementing the continued on-site courses offered at Linux Foundation events, local training sessions in select U.S. cities, and by request, specialized training for corporate developers."

Comments (none posted)

Page editor: Forrest Cook

Announcements

Non-Commercial announcements

EFF Challenges Government's 'Back Door Wiretap'

The Electronic Frontier Foundation is challenging the US government's Back Door Wiretap policy. "The Electronic Frontier Foundation (EFF) and other civil liberties groups filed an amicus brief in Warshak v. United States urging the 6th U.S. Circuit Court of Appeals Wednesday to hold that the government's seizure of email without a warrant violated the Fourth Amendment and federal privacy statutes, as well as the Justice Department's own surveillance manual."

Full Story (comments: none)

EFF busts bogus Internet subdomain patent

The Electronic Frontier Foundation has announced the winning of a case against a bogus patent. "The U.S. Patent and Trademark Office has announced that it will revoke an illegitimate patent on Internet subdomains as a result of the Electronic Frontier Foundation's (EFF) Patent Busting Project campaign. U.S. Patent No. 6,687,746, now held by Hoshiko, LLC, claimed to cover the method of automatically assigning Internet subdomains, like "action.eff.org" for the parent domain "eff.org." Previous patent owner Ideaflood used this bogus patent to demand payment from website hosting companies offering personalized domains, such as LiveJournal, a social networking site where each of its three million users may have their own subdomain."

Full Story (comments: 4)

EFF and Public Knowledge reluctantly drop lawsuit for information about ACTA

The EFF and Public Knowledge have dropped a lawsuit seeking information about the Anti-Counterfeiting Trade Agreement. "The Obama Administration's decision to support Bush-era concealment policies has forced the Electronic Frontier Foundation (EFF) and Public Knowledge (PK) to drop their lawsuit about the proposed Anti-Counterfeiting Trade Agreement (ACTA). EFF and PK had been seeking important documents about the secret intellectual property enforcement treaty that has broad implications for global privacy and innovation."

Full Story (comments: none)

Gnash V9 Summer Bash (Open Media Now)

Open Media Now has announced the Gnash V9 Summer Bash which takes place from mid ­May through mid­ August, 2009. "The Bash is a summer project designed to achieve a critical leap in version 9 free flash functionality by enlisting the help of student interns to hammer our way through a big stack of ActionScript3 (AS3) Class Libraries. The 2009 Gnash Summer Bash will result in Gnash v9 (and v10) compatibility with a number of high-demand websites -- including educational, major media, and other popular sites." See blog posting #1 for the current project status.

Comments (none posted)

SFLC Files Amicus Brief in Jacobsen v. Katzner

The Software Freedom Law Center has announced the filing of an Amicus Brief. "Today, SFLC filed brief amicus curiae in the case, Jacobsen v. Katzer before the Court of Appeals for the Federal Circuit (CAFC). In the brief, SFLC argues that a Free, Libre, and Open Source Software (FLOSS) developer whose license has been violated should be able to call upon the courts to prevent further infringing distributions."

Full Story (comments: none)

Commercial announcements

Kineo Open Source supports Moodle

Kineo Open Source has announced its monetary supports for Moodle. "Kineo Open Source has donated $3000AUS to Moodle, the open source learning management system. This will be put into the development fund run by the Moodle Trust, which contracts programmers to work on Moodle, to help fund the development of the Site-Wide Groups feature in Moodle 2.0 which is due at the end of 2009."

Full Story (comments: none)

New Books

Head First Networking--New from O'Reilly

O'Reilly has published the book Head First Networking by Al Anderson and Ryan Benedetti.

Full Story (comments: none)

Resources

Linux Foundation Newsletter

The June, 2009 edition of the Linux Foundation Newsletter has been published. "In this month's Linux Foundation newsletter: * Linux Foundation Expands Individual Membership Program * Provider of Interactive Teaching Solutions Joins Linux Foundation * Open Voices Interview Series Continues with IBM's Bob Sutor * Online Training Courses Launched This Month * Linux Foundation in the News * From the Director"

Full Story (comments: none)

Contests and Awards

Public voting Awards Persistent Model Patterns.

Nominations are open for the ODBMS.ORG Awards. "Dear Colleagues I`d like to encourage you and your colleagues to take part to the public voting for the ODBMS.ORG Awards - for the most valuable Persistent Model Patterns."

Full Story (comments: none)

Book created with Scribus wins Belgian design award

A book created with the Scribus drawing application has won a Belgian award. "Our dear friends at OSP in Brussels have won the 2009 Plantin Moretus Award for the best-designed Belgian book in the category non-fiction: Liesbeth Huybrechts, CrossOver: Kunst, media en technologie in Vlaanderen, Gent 2009. The book was produced with Open Source tools, and the final layout has been created in Scribus. Moreover, OSP used the Free Liberation font family exclusively."

Comments (none posted)

Surveys

Survey predicts continued strong growth of Linux use on mainframes

A survey commissioned by CA reports that Linux use on the mainframe will continue to grow. "The study surveyed 100 IT executives and managers at companies with at least $2 billion in annual revenue about their use of the Linux operating system on IBM mainframes. 93% of respondents projected that their use of IBM's IFL (Integrated Facility for Linux) specialty mainframe processor would increase or at least remain steady over the course of the next two years. 42% projected that their use of the IFL would grow between 21% and 40%, and 10% projected that it would grow more than 76%. The two main reasons cited by respondents for this increased use of Linux on the mainframe were 1) the desire to take advantage of computing capacity available on their mainframe's central processors and/or IFLs, and 2) their assessment that using Linux on the mainframe would be more cost-effective than other platforms."

Comments (none posted)

June 2009 Web Server Survey (Netcraft)

The Netcraft June 2009 Web Server survey is out. "In the June 2009 survey we received responses from 238,027,855 sites, an increase of 2,137,329 on last month. A reduction in activity at Microsoft Live Spaces was responsible for the large drop in the number of Microsoft-IIS sites detected. Apache retains the dominant market share of 47.12%, approximately 112.2 million sites in total, and saw a modest increase in market share of 0.63 percentage points this month. Meanwhile, Google has increased its market share by 1.3 percentage points to nearly 12 million sites, due mainly to increased activity on Blogger. nginx continues to grow strongly, increasing its market share by 1 percentage point."

Comments (none posted)

Education and Certification

Announcing the XtreemOS Summer School

The XtreemOS Summer School will be held at Wadham College, University of Oxford Oxford, UK, on September 7-11, 2009. "XtreemOS is a Linux-based operating systems that includes Grid functionalities. It is characterised by properties such as transparency, hiding the complexity of in the underlying distributed infrastructure; scalability, supporting hundreds of thousands of nodes and millions of users; and dependability, providing reliability, highly availability and security. The XtreemOS Summer School will include lectures on modern distributed paradigms such as Grid computing, Cloud computing, and network-centric operating systems."

Comments (none posted)

Event Reports

Code Sprint challenges Ingres open source community

An event report from the recent Ingres Code Sprint has been published. "Whoever said the finish line is the end of the race clearly has never participated in an Ingres Code Sprint. Ingres, the leading provider of open source database management software and pioneer of the New Economics of IT, hosted the Ingres Code Sprint, a two-day event held last week in advance of the UK Ingres Users Association Annual Conference in London. Ingres Code Sprint brought together customers, partners, and Ingres engineers to design, code, and create new features. As a result of the two-day code-fest, compelling new features will now be added to the Ingres product line."

Comments (none posted)

Calls for Presentations

LCA 2010: Call for Miniconfs now open

A Call for Miniconfs has been posted for LCA 2010. "We are pleased to invite proposals for Miniconfs on any Free Software related subject; from Linux and the BSDs to OpenOffice.org, from networking to audio-visual magic, from deep hacks to Creative Commons. The LCA2010 Miniconf Review Committee will be accepting Miniconf Proposals up until the close of business on Friday 17 July 2009. Successful Miniconf Proposals will be notified in early September 2009." The event takes place on January 18-23, 2010 in Wellington, New Zealand.

Full Story (comments: none)

2009 Linux Plumbers Conference Deadline Extension

The Linux Plumbers Conference has extended its deadline for proposal abstracts until Monday June 22. There are numerous tracks covering many different parts of the Linux plumbing. "We have an exciting program shaping up! Tutorials include an advanced git tutorial from Linus Torvalds. Keynotes include Vivek Kundra, Federal CIO, live via video with real time Q&A (invitee, to be confirmed), and Keith Packard, Intel, X Window guru." Click below for the full announcement.

Full Story (comments: none)

NLUUG autumn conference call for papers

A call for papers has gone out for the NLUUG autumn conference, which will take place in Ede, The Netherlands. "Important dates: Deadline abstracts: June 29, 2009 Acceptance notification: July 5, 2009 Conference: October 29, 2009".

Full Story (comments: none)

Request for proposals: PostgreSQL Conference 2009 Japan

A request for proposals has gone out for the PostgreSQL Conference 2009 Japan. "Submission deadline is June 30th, 2009 17:00 JST. Please send your proposal as soon as possible. Please note that if you are selected as a speaker at the conference, the admission fee will be exempted (it will be around 10,000 Japanese yen, which is about US $102.04 according to today's rate)." The event will be held on November 20 and 21, 2009 at AP Hamamatsucho, Tokyo.

Full Story (comments: none)

Upcoming Events

Call for keys for DebConf9 keysigning

The DebConf9 keysigning call for keys been announced. "As part of the 10th Debian Conference in Cáceres, Extremadura, Spain, will be OpenPGP (pgp/gpg) keysignings. If you intend to participate in the DebConf9 keysignings, please send your ascii armored public key as explained at not later than 23:59 UTC on Sunday 12th of July, 2009."

Full Story (comments: none)

Flash Memory Summit announced

The 2009 Flash Memory Summit will take place in Santa Clara, CA on August 11-13, 2009. "The Flash Memory Summit is the only place where you will hear the people making these products happen! Network with companies and people that will create the next generation of hardware and software at the Flash Memory Summit. This power-packed 3-day event will cover the latest topics in flash memory. Hear about flash in embedded systems, laptop design, flash in enterprise storage systems, green flash, solid state drives, flash performance, flash-based design, flash in computers, mobile applications, software, new non-volatile memory technologies, reliability, security, and much more!"

Comments (none posted)

The Gran Canaria Desktop Summit Keynotes (KDEDot)

KDE.News has announced the keynotes from the Gran Canaria Desktop Summit which will take place on July 3-11 in Las Palmas, Gran Canaria. "The GNOME Foundation and KDE e.V. are excited to announce the keynotes for the first ever co-located Akademy and GUADEC, over 100 talks as well as BOFs, keynote sessions, lightning talks and many opportunities to meet other developers and begin collaborating between projects. Current confirmed keynotes are: * Richard Stallman, Free Software Foundation * Walter Bender, Executive Director, Sugar Labs * Robert Lefkowitz, Distinguished Engineer of the ACM * Jakub Pavelek, Nokia".

Comments (none posted)

OSCON 2009 early bird registration extended

Early bird registration for OSCON 2009 has been extended to June 23. "Registration is now open for the O'Reilly Open Source Convention (OSCON). OSCON 2009 will be July 20-24 in San Jose, California."

Full Story (comments: none)

OSCON: booth volunteers wanted

A call for booth volunteers has gone out for OSCON. "I'm organizing a Python booth for the OSCON Exhibit Hall Weds 7/22 and Thurs 7/23. If you'd like to help staff the booth, please join the OSCON mailing list".

Full Story (comments: none)

Events: June 25, 2009 to August 24, 2009

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
June 20 June 26	Beginning iPhone for Commuters	New York, USA
June 24 June 27	LinuxTag 2009	Berlin, Germany
June 24 June 27	10th International Free Software Forum	Porto Alegre, Brazil
June 26 June 28	Fedora Users and Developers Conference - Berlin	Berlin, Germany
June 26 June 30	Hacker Space Festival 2009	Seine, France
June 28 July 4	EuroPython 2009	Birmingham, UK
June 29 June 30	Open Source China World 2009	Beijing, China
July 1 July 3	OSPERT 2009	Dublin, Ireland
July 1 July 3	ICOODB 2009	Zurich, Switzerland
July 2 July 5	ToorCamp 2009	Moses Lake, WA, USA
July 3	PHP'n Rio 09	Rio de Janeiro, Brazil
July 3 July 11	Gran Canaria Desktop Summit (GUADEC/Akademy)	Gran Canaria, Spain
July 4	Open Tech 2009	London, UK
July 6 July 10	Python African Tour : Sénégal	Dakar, Sénégal
July 7 July 11	Libre Software Meeting	Nantes, France
July 13 July 17	(Montreal) Linux Symposium	Montreal, Canada
July 15 July 16	NIT Agartala FOSS and GNU/Linux fest	Agartala, India
July 15 July 17	Kernel Conference Australia 2009	Brisbane, Queensland, Australia
July 18 July 19	Community Leadership Summit	San Jose, CA, USA
July 19	pgDay San Jose	San Jose, CA, USA
July 19 July 20	Open Video Conference	New York City, USA
July 20 July 24	2009 O'Reilly Open Source Convention	San Jose, CA, USA
July 24 July 30	DebConf 2009	Cáceres, Extremadura, Spain
July 25 July 26	EuroSciPy 2009	Leipzig, Germany
July 25 July 26	PyOhio 2009	Columbus, OH, USA
July 25 July 30	Black Hat Briefings and Training	Las Vegas, NV, USA
July 26 July 27	InsideMobile	San Jose, CA, USA
July 31 August 2	FOSS in Healthcare unconference	Houston, TX, USA
August 3 August 5	YAPC::EU::2009	Lisbon, Portugal
August 7	August Penguin 2009	Weizmann Institute, Israel
August 7 August 9	UKUUG Summer 2009 Conference	Birmingham, UK
August 10 August 14	USENIX Security Symposium	Montreal, Quebec, Canada
August 11	FOSS Dev Camp - Open Source World	San Francisco, CA, USA
August 11 August 13	Flash Memory Summit	Santa Clara, CA, USA
August 12 August 13	OpenSource World Conference and Expo	San Francisco, CA, USA
August 12 August 13	Military Open Source Software	Atlanta, Georgia, USA
August 13 August 16	Hacking At Random 2009	Vierhouten, The Netherlands
August 18 August 23	2009 Python in Science Conference	Pasadena, CA, USA
August 22 August 23	Free and Open Source Conference (FrOSCon)	St. Augustin, Germany
August 22 August 23	OpenSQL Camp	St. Augustin, Germany

If your event does not appear here, please tell us about it.

Audio and Video programs

Open Voices Interview with Bob Sutor, IBM (Linux.com)

Jim Zemlin interviews IBM's Bob Sutor in a fairly lengthy Linux.com podcast. "In this episode of Open Voices, Linux Foundation Executive Director Jim Zemlin talks with newly appointed VP of Linux and Open Source at IBM Bob Sutor. They cover IBM’s current support of Linux, the origin of that support, and the hotspots Bob sees in the Linux and open source market today. Highlights include conversation about cloud computing, Linux on the desktop, ODF, and the growth of the Linux community. Bob will cover these topics in more detail during his keynote at the upcoming LinuxCon conference in September in Portland, OR."

Comments (none posted)

Page editor: Forrest Cook