LWN.net Weekly Edition for January 14, 2010

When does a bug turn into a feature?

Sometimes bugs are in the eye of the beholder as a recent PHP bug report illustrates. That report also illustrates how quickly discussions in bug reports can spiral out of control, turning to anger and insults. There are some comical aspects to the thread, but the underlying issue, maintaining compatibility with existing bugs, is one that many projects struggle with.

A PHP user ("endosquid") reported that the number_format() function had changed behavior in PHP 5.3; that is, when number_format("",0) is called, it no longer returns "0", instead it returns an empty string. Given that the first argument to the function is supposed to be a number, in particular a floating point number that is to be formatted based on the rest of the arguments, an empty string might seem like the right thing to return. On the other hand, all earlier versions of the function returned a string containing "0".

It turns out that part of the work that went into version 5.3 was to clean up the parameter parsing code in PHP, and to use one routine, zend_parse_parameters(), internally. As PHP creator Rasmus Lerdorf related in the thread: "Most of PHP was using this already, but there were still some stragglers like number_format()." Lerdorf also suggested casting the first argument to a float (i.e. number_format((float)"",0)) as a solution to the problem.

As one would guess, endosquid's application wasn't calling number_format() directly with an empty string, but was instead passing a variable that may or may not have been initialized. In general that is a bad programming practice, but it is quite common in PHP code where the language has often tried to "do the right thing" with uninitialized variables. But if the "right thing" changes, lots of code that relied on it can break.

The argument that endosquid makes about what number_format() should return is not entirely without merit. The function is supposed to return a formatted number, and the empty string is hardly that, so endosquid believes that it should return "0". But, as Lerdorf points out, what would one expect number_format("a",0) to return? The unfortunate answer is that pre-5.3 versions did return "0" in that case. So, in tightening up the PHP parameter parsing code, a substantial difference in the behavior of number_format() was introduced.

The documentation for number_format() is not terribly helpful as it doesn't address error conditions at all. It does specify that the first parameter is a float, but PHP will happily take strings like "9" or "3.14159" for that parameter, converting as needed. Given all that, programmers have to rely on what the language actually does, and since at least PHP 3, number_format() has always returned "0" when handed random strings.

It doesn't take long for the bug report thread to descend into flames. Evidently endosquid works in a tightly controlled environment that requires a raft of paperwork to accompany code changes, but that still doesn't justify a claim of "MONTHS [of] fixing code for no real benefit". It seems clear that endosquid didn't quite understand who it was responding to the bug report when asking Lerdorf to "escalate this to someone who can answer the question as to why this was changed". Lerdorf responds: "Escalate? Oh how I wish I had someone to escalate to."

Lerdorf also explained that the change was first made public as part of the first 5.3 release candidate in March 2009. He said that interested folks had until July to make a case that any particular change shouldn't go into the release. While endosquid complained that 5.3 had only recently become available on the platform he was using, Lerdorf pointed out that users have some responsibility to keep up with their tools:

There is certainly some truth to Lerdorf's admonishment, but it didn't sit well with endosquid, who plans to change the C code back to the old behavior. Patching the language source—rather than making a fairly simple textual substitution to the number_format() call sites—seems a bit extreme, but is evidently easier in that environment. Unlike some proprietary alternatives, though, free software allows just that kind of change.

But free software developers should not have to deal with insulting comments from bug reporters. There are multiple alternatives for endosquid, including staying with the 5.1.x version of PHP, patching the 5.3.x source, or fixing the actual calls, so getting angry and lashing out in the bug report is not likely to help anyone. It is, as Lerdorf points out, "a classic case of how not to treat unpaid volunteers who provide critical pieces of your money-making infrastructure".

There is always the question, though, of when a "bug" has lived long enough that it becomes something that needs to be carried forward. Once applications start depending on buggy behavior, there will always be annoyed users when the bug gets fixed. The Linux kernel has run into this problem numerous times, generally opting to maintain the "insanity" (in the words of Al Viro) for compatibility's sake.

It is a difficult balance to strike. PHP developers cannot possibly know all of the different corner-cases and quirks that PHP applications depend on. When fixing what they see as a bug, they have to rely on users testing betas and release candidates to find places where the "bug" label may not be appropriate—or at least requires some discussion. But users are often busy with other things, so we are likely to see this kind of situation play out for various projects in the future.

Comments (27 posted)

The Grumpy Editor's Tomato review

Your editor has just completed an important transition: moving his Internet connectivity from one evil branch of the local telecom duopoly to the other, equally evil branch. This change required the acquisition of a new router; that, in turn, provided the opportunity to play with Linux-based router software, and Tomato in particular. Read on for your editor's impressions of this impressive bit of (mostly) free software.

Tomato has its roots in the original Linksys WRT54G firmware. This firmware was first distributed as if it were proprietary software, but Linksys, under heavy GPL-enforcement pressure, eventually made the source available under the GPL. The existence of this source, along with the ease by which the Linksys routers could have new firmware installed, led to the creation of a number of firmware distributions, all of which added new features and otherwise improved on the original Linksys offering. Over time, Linksys (Cisco) has incorporated some of these improvements; the company also continues to offer a special version of its basic household router (the WRT54GL) which is explicitly designed to allow firmware replacement.

If a company is going to make a competitively-priced, Linux-based, user-hackable router, your editor feels an obligation to buy it. That choice is easy, but the choice of which replacement firmware to use is harder. There's a wide variety of offerings, including OpenWrt, DD-WRT, FreeWRT, and Tomato. There appears to no easy way to pick one in particular; your editor started with Tomato because the screen shots looked nice and the installation instructions were straightforward. On the other hand, OpenWRT's installation instructions are simply missing (though some information is available on the OpenWRT wiki), and those for DD-WRT are lengthy and intimidating, making the process look similar to installing Gentoo.

The funny thing, of course, is that installing replacement firmware on a WRT54GL router is a trivial task: download firmware, go to the router's "upgrade firmware" screen, and upload the new blob. Two minutes later the job is done.

Your editor's first impression of Tomato is that it is great stuff - though reflection yields some concerns which will be discussed below. Tomato brings a whole range of new functionality to a cheap consumer device, yielding a degree of visibility into and control over the network which your editor has never had before. The web-based interface is slick - if JavaScript heavy - and mostly easy to use. It would have been nice to bring this device into the house some time ago, even if Evil Telecom #1's network did not require its presence.

One nice feature is simple bandwidth monitoring and display; there are a number of plots which can be brought up and watched in real time. The router is also able to store network statistics for a long period of time and produce plots on daily, weekly, or monthly scales. The only problem there is that the hardware lacks the storage for this amount of data; Tomato can work around that little limitation by using a built-in CIFS client to use storage found elsewhere on the net.

The Linux kernel has the facilities to exercise a great deal of control over the processing of network traffic. There is simple firewalling, of course, with the ability to decide which traffic is worthy of passage and which should be denied. But there is also an extensive traffic control subsystem allowing the user to prioritize the use of the available bandwidth. That feature is arguably underused because it takes a while to figure out how to configure it with the available command-line clients. Tomato provides a relatively straightforward mechanism for the creation of both access control and quality-of-service rules.

On the access control side, Tomato has a screen which allows the creation of rules for specific addresses and port numbers. Rules can be global, or they can apply only to traffic from specific machines on the local network. Rules can have a schedule attached so that, say, distracting web sites can be blocked during the day - encouraging accomplishment - while serious sites can be blocked at night - encouraging relaxation. Specific systems can be blocked from the net entirely on a schedule, a potentially useful feature for parents who have long since given up on trying to keep wireless-enabled devices out of the kids' rooms late at night.

Interestingly, Tomato does not stop with port-based restrictions; it also incorporates the L7-filter and IPP2P classifiers. Both modules are essentially deep packet inspection implementations, allowing the classification (and, thus, control) of traffic based on a look at the actual bits passing through. With L7-filter, for example, an administrator can block specific role-playing games, regardless of whether the official servers or ports are being used. There's a vast set of canned rules, enabling control of various instant messaging protocols, file formats, and more. It is now possible to block the downloading of Perl scripts - something which, while tempting, is probably unwise to actually do. IPP2P, instead, is more directly focused on the detection of peer-to-peer protocols. Together, they are a control freak's dream; network neutrality stops at the local router.

Even if a network administrator does not wish to ban, say, role-playing games outright, there is value in saying that such uses of the network should not interfere with real work like reading XKCD. That's where the quality of service (QOS) screens come in. QOS is a two-step process: dividing the available bandwidth among various classes of traffic, and assigning specific types of traffic to those classes. Tomato provides ten different classifications, each of which has a priority and a guaranteed bandwidth portion - all of which can be changed, of course. By default, only outbound (to the wide-area network) traffic is subject to control; it is possible to control inbound traffic, but, since that traffic has already passed over the WAN link by the time the router can work with it, there's usually little point. Classification rules look a lot like access control rules, allowing the use of addresses, port numbers, or classification by IPP2P or L7-filter.

With all this, the administrator can decree that, say, a certain proprietary role-playing game favored by the children is a very low priority stream - but it still gets a few percent of the available bandwidth so the kids do not suffer permanent trauma as a result of lag-induced fragging. Tomato can also generate pie charts showing (by classification) how bandwidth is being used currently; clicking on a classification yields a list of current connections. All told, it's a capable and easy-to-use way of ensuring that the network functions well even under heavy use.

Other features abound. There is a DHCP server, of course, along with a nice screen for doing static DHCP assignments without ever having to type a MAC address. The router can report its globally-visible address to a wide variety of dynamic DNS services. Incoming connections can be forwarded to internal machines in a flexible way. There is a "triggering" mechanism which automatically opens specific incoming ports in response to specific outgoing connections. Old-timers will see triggering as a way to support the full FTP protocol; everybody else will use it to enable incoming BitTorrent connections. And so on. It is, to say the least, a highly capable system.

The biggest operational problem your editor has experienced is the occasional dropping of long-lived SSH connections. A bit of research led to the tweaking of a few of the rather intimidating array of connection tracking parameters, and things would appear to have improved.

There are a couple of more general concerns, though. Like many of its peers, Tomato appears to be well past its active development phase; there were a few releases in 2009, but they did not make a great many changes. Meanwhile, its 2.4.20 kernel is rather far back from the leading edge, and both L7-Filter and IPP2P are explicitly unmaintained. Given the steady stream of security updates for protocol dissectors in WireShark, your editor has a hard time believing that these other classifiers can be completely free of security issues. But there is nobody maintaining them, and Tomato has no apparent means for the monitoring of security problems or the distribution of updates. Given that these routers are directly exposed to the net and are the first line of defense for many networks, the combination of ancient software and no security support is worrying.

Tomato is also not 100% free software. The core Linux system is, of course, free, but the user interface code carries a "for use with Tomato only" copyright notice. There is also the issue of the proprietary Broadcom network driver, but that's a problem any 2.4-based firmware for this router will have.

These concerns are strong enough that, despite Tomato's many qualities, your editor is not yet sure that he has found the final distribution for his router. In particular, OpenWRT - which offers a 2.6 kernel, a seemingly larger and more active development team, release notes with CVE numbers included, and a packaging system allowing others to add features to the router - seems worth a detailed look. The good news is that this choice exists and is easy to execute. That, in turn, is the result of the GPL and the developers who made an effort to enforce it.

Comments (52 posted)

RawTherapee: the newest open source raw photo editor

Gábor Horváth has been developing the raw photo converter RawTherapee single-handedly, on Linux and Windows, since 2006. The application has been freeware the entire time, with Horváth accepting Paypal donations through the project's web site. Consequently, although there are significant changes in the 3.0 alpha release announced on January 4th, it was arguably bigger news that the project was switching to the GPLv3.

RawTherapee is a raw image conversion and editing utility that (like most raw converters) supports the native file formats of virtually all digital cameras courtesy of the dcraw project. It offers exposure control, highlight and shadow recovery, color and tint balancing and adjustments, sharpening and noise reduction, and basic crop/rotation tools. On the workflow side, it supports color management, Exif and IPTC tagging, quality ratings, batch processing, saved snapshots, and sending images to an external editor for detailed work.

Getting started

Builds for 3.0 alpha 1 are available for Linux and Windows, and for the first time, source tarballs as well. The Linux builds are provided as 32-bit and 64-bit standalone binaries; simply extract the package and run ./rtstart from a shell prompt to get started. There is no dependency checking, but RawTherapee is compiled against standard GTK+ and GNOME libraries. A more complete list of dependencies is found in a forum thread about compiling the source on Linux; the only special-purpose libraries are libtiff and libiptcdata, which should already be pulled in by other modern image editing packages.

In use, RawTherapee behaves like most comparable raw converters, sporting a three-pane window with a file browser in the left-hand column, an image viewer in the center, and a tabbed image-adjustment toolbox on the right. The vast majority of raw converters take this approach, exposing the image adjustment controls as a vertical stack of sliders and checkboxes. Novices may need to familiarize themselves with the terminology before feeling comfortable tweaking the myriad of settings, but on the positive side, RawTherapee is non-destructive — it saves adjustments not by changing the original image, but by storing an auxiliary "sidecar" file in the same directory.

As raw converters go, RawTherapee offers a full palette of controls, with multiple user-selectable sharpening algorithms, separate luminance- and color-noise reduction sliders, an RGB channel mixer, and multiple demosaicing algorithms. Nevertheless, the tool layout is organized, providing a sensible division of the potentially overwhelming controls into four main tabs (Exposure, Detail, Color, and Transform), and sub-dividing each tab into groups. Batch operations are easy to queue, offering the choice of a specified output folder or a user-defined template, with which you can rename and store output files based on their original name and directory.

RawTherapee does diverge from other converters in a few areas, such as its use of tabbed windows. Starting with 3.0, opening an image to edit opens it in a separate tab. This allows the user to keep multiple editing sessions open at once without exporting, and is definitely a nice feature. There is also no "filmstrip" window pane displaying other image thumbnails in the current directory; the only way to open an new image for editing is through the file browser — a difference that some users might find less convenient. It also provides floating "magnify" windows to zoom in on particular parts of the current image without zooming the entire image view, something not every editor supports.

Linux users will find several oddities in the user interface, though, such as the lack of any menus (standard or otherwise) — the closest thing are the "Preferences" and "Exit" text-buttons on the bottom right-hand corner. And those users with a scroll mouse must take care when scrolling the vertical toolbox; it is easy to accidentally throw off an adjustment slider if the cursor happens to land hovering over one of the controls. This release also lacks tooltips for many of the settings, which would be a boon to new users.

For real-world work, it is also critical to take the "alpha" status of this release seriously. 3.0 alpha 1 is crash-prone, and the adjustment sidecar files it creates automatically are not compatible with the 2.x-series. Those who use the current, stable release of RawTherapee (2.4.1) must be sure to back up their work before testing 3.0.

Open source and further development

Horváth cited three factors behind his decision to change the licensing of RawTherapee: personal lack of time, the difficulty of reproducing and fixing reported bugs, and interest in focusing his own time on the core image-processing features of the program rather than the GUI and other components. He set up a RawTherapee project on Google Code, including Subversion access to the source, build instructions, and an issue tracker. He has also opened developer discussion forums on the main RawTherapee site.

The RawTherapee code breaks into three parts: the image processing library, an Exif support library, and the GUI application itself. Bug reports and enhancement requests have already begun to appear at the Google Code site; Horváth has stated that his top priority for the moment is working out the kinks in the CMake build system.

Moving forward, Horváth's intent to focus on the image processing core is a key component of the 3.x roadmap. Part of the rewrite that led up to 3.0 alpha 1 — although not yet visible to end users — is a separation of the editor component to make it easier to add more algorithms, such as additional demosaicing and noise-reduction choices and new tools to correct fringing and perspective distortion.

Looking at the state of RawTherapee and its user base, the decision to move the code to an open source license is undoubtedly a good one. The application already has an active community, including many Linux users and language translators. But as Horváth discovered maintaining the project in closed source state, supporting that user community's bug reports and support requests became more and more time consuming as the project grew in popularity — a fact many solo software developers may not consider when starting a new project.

Furthermore, Horváth wants to focus on the part of the code he finds most interesting, the image adjustment algorithms. By adopting a free software license, RawTherapee might be able to slim down by swapping out some other components for existing open libraries (such as libexiv, rather than its own separate Exif library).

There is clearly room for what Horváth wants to do with RawTherapee in the open source graphics space. Arguably the most similar raw converter, Rawstudio, takes a different approach, aiming to make raw image editing accessible for the average non-technical photographer. RawTherapee's decision to make multiple user-selectable algorithms available for so many controls will make it appealing to a different crowd, those that like to experiment or who have very specific opinions about their image editing. There are other raw-capable editors and applications, such as Digikam, that emphasize more image collection management, raster editing, or other functions.

All in all, RawTherapee has been a consistently good performer on Linux and Windows for years. As one of the few free choices in a space dominated by high-priced applications, it was a standout. Considering that most of the underpinnings of raw image editing — dcraw, Exif and IPTC, and the various mathematical algorithms — are not proprietary, it only makes sense that good, open source solutions would emerge. With the upcoming 3.0 release, it is excellent to see that RawTherapee will be among them.

Comments (37 posted)

Page editor: Jonathan Corbet

Security

SSH: passwords or keys?

A recent discussion on the OpenSSH developers mailing list (openssh-unix-dev) debated the relative merits of passwords versus keys as ssh authentication methods. While password authentication has fallen out of favor over the last few years, there are still situations where it makes more sense than key-based authentication. As with many security decisions, the right choice is largely dependent on the threat model one is defending against.

Through no fault of its own, ssh is probably one of the most used (or abused) mechanisms for system compromise. Repeated brute force password-guessing attacks are a common "script kiddie" activity, which is why many administrators have turned off password authentication entirely. That means that users must have keys installed on hosts they need to access, leaving open another avenue of attack: the corresponding private keys.

When sshd is configured to disallow password authentication (via the PasswordAuthentication no directive in sshd_config), the server will no longer allow the traditional username/password pair to be used to authenticate a user. Instead, the user must generate a public/private key pair on each host that is to be used to access the sshd host (or one pair that gets shared among various client hosts—generally a bad practice). The public key gets installed in the user's authorized_keys file on the server and authentication is handled directly between the ssh client and server.

But what protects the private key? Depending on the user, and their level of security consciousness, protection could range from directory and file permissions on the private key file to a password that encrypts the private key. For password-protected keys, that means that the user must enter the password to decrypt the private key before the ssh client can use it to authenticate with the server. Since many users like key-based authentication because it doesn't require passwords, this extra level of key security can be annoying—and often gets omitted. That leaves private keys potentially accessible on the client system.

Davi Diaz wanted to know how to detect password-less keys on the server side so that authentications from those clients could be rejected. But, as Aris Adamantiadis and others pointed out, there is no way for the ssh server to know:

While password authentication has its downsides, there are some advantages to it as Mark Janssen noted: "while keys are better [than] passwords, it's impossible to enforce passphrase quality on keys, while it is possible to enforce some quality on passwords." Passwords can also be aged, so that they must be changed with some frequency. Because account passwords are under the control of the server administrator, unlike most private key passwords, an administrator can enforce strict requirements on them.

But, unlike passwords, private keys generally aren't used in multiple places, nor are they transmitted anywhere. It is a common, if insecure, practice for humans to use the same password on their LWN account that they do to log in with ssh to some other system. So, once a password is cracked or captured, it can often be used to gain access elsewhere.

There are also ways that compromising a single private key can lead to the compromise of multiple systems, however. If a password-less private key can be accessed—via the compromise of a client system or the theft of a laptop for example—an attacker can access any systems that have authorized that key. A single compromised private key will often allow an attacker access to multiple systems, either directly using the compromised key or by hopping to new systems that have their own password-less private keys. One particularly ugly scenario is for root to have authorized keys that allow a regular user on one system to automatically authenticate as root on the other. That is one good reason to disallow all root logins via ssh (PermitRootLogin no in sshd_config).

So, it would seem that disallowing password authentication for ssh and requiring users to password protect their private keys would go a long way towards eliminating compromises via ssh. There aren't any technological means to force passwords on private keys, but an administrator must either trust their users or disable their access. As Daniel Kahn Gillmor said:

On the other hand, requiring both keys and passwords would be even more secure. If the key was also password-protected—with a different password of course—that would make it stronger still. But the usual security/convenience tradeoff applies. That much protection will certainly annoy users, so it may only be necessary for the most sensitive systems.

There are many things that must be considered when making security decisions: the sensitivity of the data, the trustworthiness of the users, the threats being defended against, and so on. One of the things that makes security so difficult is that there is no "one size fits all" solution, each situation is different. The various authentication choices for ssh, and their relative strengths and weaknesses, just bear that out.

Comments (60 posted)

Brief items

BerliOS compromised

The BerliOS repository site has been compromised; indeed, it appears it has been compromised since 2005. What little information is available can be found from this (German) Heise article (Google translation) and a screen shot from the defaced site. According to the BerliOS system admin (a certain Jörg Schilling), no data has been tampered with, but those who have worked with or gotten code from BerliOS might want to be careful regardless.

Update: the Heise article is now available in English.

Comments (19 posted)

Google: a new approach to China

It may be a little off the LWN topic, but Google's a new approach to China is worth a read for anybody who hasn't yet seen it. It's a reminder of how important security practices are and what the risks of storing important data in "the cloud" can be. "Third, as part of this investigation but independent of the attack on Google, we have discovered that the accounts of dozens of U.S.-, China- and Europe-based Gmail users who are advocates of human rights in China appear to have been routinely accessed by third parties."

Comments (38 posted)

New vulnerabilities

DevIL: buffer overflow

Package(s):	DevIL	CVE #(s):	CVE-2009-3994
Created:	January 13, 2010	Updated:	January 13, 2010
Description:	The DevIL image processing library suffers from a buffer overflow vulnerability exploitable via a specially-crafted image file.
Alerts:	Fedora FEDORA-2009-13255 2009-12-16 Fedora FEDORA-2009-13219 2009-12-16

Comments (none posted)

firefox: multiple vulnerabilities

Package(s):	firefox-3.5	CVE #(s):	CVE-2009-3980 CVE-2009-3982 CVE-2009-3388 CVE-2009-3389
Created:	January 8, 2010	Updated:	June 14, 2010
Description:	From the Ubuntu advisory: Jesse Ruderman, Josh Soref, Martijn Wargers, Jose Angel, Olli Pettay, and David James discovered several flaws in the browser and JavaScript engines of Firefox. If a user were tricked into viewing a malicious website, a remote attacker could cause a denial of service or possibly execute arbitrary code with the privileges of the user invoking the program. (CVE-2009-3980, CVE-2009-3982) David Keeler, Bob Clary, and Dan Kaminsky discovered several flaws in third party media libraries. If a user were tricked into opening a crafted media file, a remote attacker could cause a denial of service or possibly execute arbitrary code with the privileges of the user invoking the program. (CVE-2009-3388, CVE-2009-3389)
Alerts:	Debian DSA-2045-1 2010-05-11 SuSE SUSE-SR:2010:013 2010-06-14 SuSE SUSE-SR:2010:008 2010-04-07 Mandriva MDVSA-2010:043 2010-02-19 Ubuntu USN-878-1 2010-01-08

Comments (none posted)

firefox: denial of service

Package(s):	firefox	CVE #(s):	CVE-2010-0220
Created:	January 11, 2010	Updated:	January 13, 2010
Description:	From the Mandriva advisory: The nsObserverList::FillObserverArray function in xpcom/ds/nsObserverList.cpp in Mozilla Firefox before 3.5.7 allows remote attackers to cause a denial of service (application crash) via a crafted web site that triggers memory consumption and an accompanying Low Memory alert dialog, and also triggers attempted removal of an observer from an empty observers array (CVE-2010-0220).
Alerts:	Mandriva MDVSA-2010:000 2010-01-10

Comments (none posted)

gif2png: buffer overflows

Package(s):	gif2png	CVE #(s):
Created:	January 13, 2010	Updated:	November 22, 2010
Description:	The gif2png utility suffers from buffer overflow vulnerabilities exploitable from the command line.
Alerts:	Fedora FEDORA-2010-0358 2010-01-08 Fedora FEDORA-2010-0330 2010-01-08

Comments (none posted)

horde3: cross-site scripting vulnerability

Package(s):	horde3	CVE #(s):	CVE-2009-3701
Created:	January 7, 2010	Updated:	April 1, 2010
Description:	From the Debian alert: It has been discovered that the horde3 administration interface is prone to cross-site scripting attacks due to the use of the PHP_SELF variable. This issue can only be exploited by authenticated administrators.
Alerts:	Fedora FEDORA-2010-5520 2010-04-01 Fedora FEDORA-2010-5483 2010-04-01 SuSE SUSE-SR:2010:004 2010-02-16 Debian DSA-1966-1 2010-01-07

Comments (none posted)

horde3: cross-site scripting vulnerability

Package(s):	horde3	CVE #(s):	CVE-2009-4363
Created:	January 7, 2010	Updated:	April 1, 2010
Description:	From the Debian alert: It has been discovered that horde3 is prone to several cross-site scripting attacks via crafted data:text/html values in HTML messages.
Alerts:	Fedora FEDORA-2010-5520 2010-04-01 Fedora FEDORA-2010-5483 2010-04-01 SuSE SUSE-SR:2010:004 2010-02-16 Debian DSA-1966-1 2010-01-07

Comments (none posted)

Kerberos: possible remote exploit

Package(s):	krb5	CVE #(s):	CVE-2009-4212
Created:	January 13, 2010	Updated:	January 19, 2010
Description:	The Kerberos daemon does not properly handle invalid AES blocks; this vulnerability can be used to crash the service and, possibly, execute arbitrary code as root.
Alerts:	SuSE SUSE-SA:2010:006 2010-01-19 CentOS CESA-2010:0029 2010-01-15 CentOS CESA-2010:0029 2010-01-15 Mandriva MDVSA-2010:006 2010-01-14 Fedora FEDORA-2010-0503 2010-01-14 Fedora FEDORA-2010-0515 2010-01-14 Debian DSA-1969-1 2010-01-12 CentOS CESA-2010:0029 2010-01-13 Red Hat RHSA-2010:0029-01 2010-01-12 Ubuntu USN-881-1 2010-01-12 Gentoo 201201-13 2012-01-23

Comments (none posted)

kernel: denial of service

Package(s):	kernel	CVE #(s):	CVE-2009-4138
Created:	January 7, 2010	Updated:	August 17, 2010
Description:	From the SuSE alert: drivers/firewire/ohci.c in the Linux kernel when packet-per-buffer mode is used, allows local users to cause a denial of service (NULL pointer dereference and system crash) or possibly have unknown other impact via an unspecified ioctl associated with receiving an ISO packet that contains zero in the payload-length field.
Alerts:	Red Hat RHSA-2010:0631-01 2010-08-17 Ubuntu USN-894-1 2010-02-05 Debian DSA-2004-1 2010-02-27 CentOS CESA-2010:0046 2010-01-20 Red Hat RHSA-2010:0046-01 2010-01-19 SuSE SUSE-SA:2010:005 2010-01-15 SuSE SUSE-SA:2010:012 2010-02-15 SuSE SUSE-SA:2010:001 2010-01-07

Comments (none posted)

kernel: denial of service

Package(s):	kernel	CVE #(s):	CVE-2009-4306
Created:	January 7, 2010	Updated:	January 13, 2010
Description:	From the SuSE alert: Unspecified vulnerability in the EXT4_IOC_MOVE_EXT (aka move extents) ioctl implementation in the ext4 filesystem in the Linux kernel allows local users to cause a denial of service (filesystem corruption) via unknown vectors, a different vulnerability than CVE-2009-4131.
Alerts:	SuSE SUSE-SA:2010:001 2010-01-07

Comments (none posted)

kernel: denial of service

Package(s):	kernel	CVE #(s):	CVE-2009-4307
Created:	January 7, 2010	Updated:	February 18, 2011
Description:	From the SuSE alert: The ext4_fill_flex_info function in fs/ext4/super.c in the Linux kernel allows user-assisted remote attackers to cause a denial of service (divide-by-zero error and panic) via a malformed ext4 filesystem containing a super block with a large FLEX_BG group size (aka s_log_groups_per_flex value).
Alerts:	Mandriva MDVSA-2011:029 2011-02-17 Red Hat RHSA-2010:0380-01 2010-04-27 Red Hat RHSA-2010:0178-02 2010-03-30 SuSE SUSE-SA:2010:005 2010-01-15 SuSE SUSE-SA:2010:001 2010-01-07 Debian DSA-2443-1 2012-03-26

Comments (none posted)

kernel: denial of service

Package(s):	kernel	CVE #(s):	CVE-2009-4308
Created:	January 7, 2010	Updated:	October 8, 2010
Description:	From the SuSE alert: The ext4_decode_error function in fs/ext4/super.c in the ext4 filesystem in the Linux kernel allows user-assisted remote attackers to cause a denial of service (NULL pointer dereference), and possibly have unspecified other impact, via a crafted read-only filesystem that lacks a journal.
Alerts:	Mandriva MDVSA-2010:188 2010-09-23 Mandriva MDVSA-2010:198 2010-10-07 CentOS CESA-2010:0147 2010-03-18 Red Hat RHSA-2010:0147-01 2010-03-16 Ubuntu USN-894-1 2010-02-05 Debian DSA-2004-1 2010-02-27 SuSE SUSE-SA:2010:005 2010-01-15 SuSE SUSE-SA:2010:012 2010-02-15 SuSE SUSE-SA:2010:001 2010-01-07

Comments (none posted)

kernel: multiple vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2007-4567 CVE-2009-4536 CVE-2009-4537 CVE-2009-4538
Created:	January 8, 2010	Updated:	July 5, 2011
Description:	From the Red Hat advisory: a flaw was found in the IPv6 Extension Header (EH) handling implementation in the Linux kernel. The skb->dst data structure was not properly validated in the ipv6_hop_jumbo() function. This could possibly lead to a remote denial of service. (CVE-2007-4567) a flaw was found in each of the following Intel PRO/1000 Linux drivers in the Linux kernel: e1000 and e1000e. A remote attacker using packets larger than the MTU could bypass the existing fragment check, resulting in partial, invalid frames being passed to the network stack. These flaws could also possibly be used to trigger a remote denial of service. (CVE-2009-4536, CVE-2009-4538) a flaw was found in the Realtek r8169 Ethernet driver in the Linux kernel. Receiving overly-long frames with network cards supported by this driver could possibly result in a remote denial of service. (CVE-2009-4537)
Alerts:	SUSE SUSE-SU-2011:0737-1 2011-07-05 SUSE SUSE-SU-2011:0711-1 2011-06-29 Red Hat RHSA-2010:0882-01 2010-11-12 openSUSE openSUSE-SU-2010:0664-1 2010-09-23 SUSE SUSE-SA:2010:036 2010-09-01 MeeGo MeeGo-SA-10:01 2010-07-07 Debian DSA-2053-1 2010-05-25 SuSE SUSE-SA:2010:023 2010-05-06 Red Hat RHSA-2010:0342-01 2010-04-06 openSUSE openSUSE-SU-2010:0397-1 2010-07-19 Mandriva MDVSA-2010:066 2010-03-24 CentOS CESA-2010:0146 2010-03-17 Red Hat RHSA-2010:0149-01 2010-03-16 Red Hat RHSA-2010:0148-01 2010-03-16 Red Hat RHSA-2010:0146-01 2010-03-16 SUSE SUSE-SA:2010:031 2010-07-20 Ubuntu USN-947-2 2010-06-04 Ubuntu USN-947-1 2010-06-03 SuSE SUSE-SA:2010:014 2010-03-03 SuSE SUSE-SA:2010:009 2010-02-05 Pardus 2010-25 2010-02-02 Red Hat RHSA-2010:0076-01 2010-02-02 SuSE SUSE-SA:2010:007 2010-01-22 Red Hat RHSA-2010:0041-01 2010-01-21 Debian DSA-2004-1 2010-02-27 Pardus 2010-35 2010-02-25 SuSE SUSE-SA:2010:013 2010-02-18 Fedora FEDORA-2010-1787 2010-02-12 SuSE SUSE-SA:2010:012 2010-02-15 Debian DSA-1996-1 2010-02-12 Pardus 2010-28 2010-02-02 Red Hat RHSA-2010:0053-01 2010-01-19 CentOS CESA-2010:0020 2010-01-15 SuSE SUSE-SA:2010:005 2010-01-15 SuSE SUSE-SA:2010:010 2010-02-08 Red Hat RHSA-2010:0079-01 2010-02-02 CentOS CESA-2010:0019 2010-01-08 Red Hat RHSA-2010:0020-01 2010-01-07 Red Hat RHSA-2010:0019-01 2010-01-07 Debian DSA-2003-1 2010-02-22 Red Hat RHSA-2010:0111-01 2010-02-16 Fedora FEDORA-2010-1500 2010-02-05 Ubuntu USN-894-1 2010-02-05 CentOS CESA-2010:0076 2010-02-04

Comments (none posted)

openttd: denial of services

Package(s):	openttd	CVE #(s):	CVE-2009-4007
Created:	January 13, 2010	Updated:	January 13, 2010
Description:	The OpenTTD server can be caused to crash by a remote attacker; version 0.7.5 fixes the bug.
Alerts:	Fedora FEDORA-2010-0144 2010-01-04 Fedora FEDORA-2010-0135 2010-01-04

Comments (none posted)

pdns-recursor: multiple vulnerabilities

Package(s):	pdns-recursor	CVE #(s):	CVE-2009-4009 CVE-2009-4010
Created:	January 7, 2010	Updated:	February 16, 2010
Description:	From the Red Hat bug report: This Wednesday the release of the PowerDNS Recursor 3.1.7.2 will be made public, which fixes two important security issues, one of which is remotely exploitable. Given the critical nature of these vulnerabilities, we are trying to keep details confidential for a few more days.
Alerts:	Debian DSA-1968-2 2010-01-28 SuSE SUSE-SR:2010:004 2010-02-16 Debian DSA-1968-1 2010-01-08 Fedora FEDORA-2010-0228 2010-01-07 Fedora FEDORA-2010-0209 2010-01-07

Comments (none posted)

phpldapadmin: remote file inclusion

Package(s):	phpldapadmin	CVE #(s):	CVE-2009-4427
Created:	January 7, 2010	Updated:	January 21, 2010
Description:	From the Debian alert: It was discovered that phpLDAPadmin, a web based interface for administering LDAP servers, doesn't sanitize an internal variable, which allows remote attackers to include and execute arbitrary local files.
Alerts:	Mandriva MDVSA-2010:023 2010-01-21 Debian DSA-1965 2010-01-07

Comments (none posted)

pidgin: directory traversal

Package(s):	pidgin	CVE #(s):	CVE-2010-0013
Created:	January 12, 2010	Updated:	April 29, 2010
Description:	From the Mandriva advisory: Directory traversal vulnerability in slp.c in the MSN protocol plugin in libpurple in Pidgin 2.6.4 and Adium 1.3.8 allows remote attackers to read arbitrary files via a .. (dot dot) in an application/x-msnmsgrp2p MSN emoticon (aka custom smiley) request, a related issue to CVE-2004-0122. NOTE: it could be argued that this is resultant from a vulnerability in which an emoticon download request is processed even without a preceding text/x-mms-emoticon message that announced availability of the emoticon.
Alerts:	Mandriva MDVSA-2010:085 2010-04-28 SuSE SUSE-SR:2010:006 2010-03-15 Slackware SSA:2010-024-03 2010-01-25 Ubuntu USN-886-1 2010-01-18 CentOS CESA-2010:0044 2010-01-15 CentOS CESA-2010:0044 2010-01-14 Red Hat RHSA-2010:0044-01 2010-01-14 Fedora FEDORA-2010-1383 2010-02-19 Fedora FEDORA-2010-0429 2010-01-12 Fedora FEDORA-2010-0368 2010-01-12 Mandriva MDVSA-2010:002 2010-01-11 Mandriva MDVSA-2010:001 2010-01-11

Comments (none posted)

sendmail: several vulnerabilities

Package(s):	sendmail	CVE #(s):	CVE-2009-4565
Created:	January 12, 2010	Updated:	February 16, 2011
Description:	From the Mandriva advisory: sendmail before 8.14.4 does not properly handle a '\0' (NUL) character in a Common Name (CN) field of an X.509 certificate, which (1) allows man-in-the-middle attackers to spoof arbitrary SSL-based SMTP servers via a crafted server certificate issued by a legitimate Certification Authority, and (2) allows remote attackers to bypass intended access restrictions via a crafted client certificate issued by a legitimate Certification Authority, a related issue to CVE-2009-2408
Alerts:	Red Hat RHSA-2011:0262-01 2011-02-16 Fedora FEDORA-2010-5399 2010-03-30 Red Hat RHSA-2010:0237-05 2010-03-30 rPath rPSA-2010-0022-1 2010-03-15 Fedora FEDORA-2010-5470 2010-03-30 SuSE SUSE-SR:2010:006 2010-03-15 Mandriva MDVSA-2010:003 2010-01-11 Debian DSA-1985-1 2010-01-31

Comments (none posted)

sssd: authentication bypass

Package(s):	sssd	CVE #(s):	CVE-2010-0014
Created:	January 13, 2010	Updated:	January 13, 2010
Description:	In some situations, sssd will accept any password as valid when Kerberos is unreachable.
Alerts:	Fedora FEDORA-2010-0451 2010-01-12 Fedora FEDORA-2010-0413 2010-01-12

Comments (none posted)

trac: multiple vulnerabilities

Package(s):	trac	CVE #(s):	CVE-2009-4405
Created:	January 13, 2010	Updated:	January 13, 2010
Description:	Versions of trac prior to 0.11.6 suffer from "multiple unspecified vulnerabilities" with "unknown impact and attack vectors."
Alerts:	Fedora FEDORA-2010-0007 2010-01-02

Comments (none posted)

transmission: directory traversal

Package(s):	transmission	CVE #(s):	CVE-2010-0012
Created:	January 8, 2010	Updated:	January 18, 2010
Description:	From the Debian advisory: Dan Rosenberg discovered that Transmission, a lightweight client for the Bittorrent filesharing protocol performs insufficient sanitizing of file names specified in .torrent files. This could lead to the overwrite of local files with the privileges of the user running Transmission if the user is tricked into opening a malicious torrent file.
Alerts:	Mandriva MDVSA-2010:014 2010-01-18 Mandriva MDVSA-2010:013 2010-01-18 Fedora FEDORA-2010-0388 2010-01-12 Ubuntu USN-885-1 2010-01-14 Fedora FEDORA-2010-0197 2010-01-05 Debian DSA-1967-q 2010-01-08

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 2.6.33-rc4, released on January 12. "Hmm. Odd release. Something like 40% of the patches are in DRM (mostly nouveau and radeon, both staging, so it's a bit less scary than it sounds. But there's a noticeable i915 component too). That's all pretty unusual." There's also a couple of new low-level drivers, support for LZO-compressed kernels, and the new generic list_sort() function. Full details can be found in the long-format changelog.

Stable updates: the only stable update in the last week is 2.6.31.11, released on January 7 to fix a build error introduced with 2.6.31.10.

Comments (none posted)

Quotes of the week

-- Casey Schaufler

-- Greg Wettstein redefines mission-critical

Comments (none posted)

Timer slack

One of the best ways to reduce a system's power usage is to avoid waking up the CPU whenever possible. Minimizing wakeups, in turn, is facilitated by ensuring that timers expire at the same time when it makes sense to do so. Waking the processor once to handle two timers is much more efficient than handling them in two separate wakeups. But doing so typically requires adjusting expiration times. For standard (not high resolution) kernel timers, the only way to make that adjustment is with the round_jiffies() function, which makes timeout periods coarser in the hopes that they will coincide more often. This method works to an extent, but it requires code changes wherever timers are used.

Arjan van de Ven has proposed an enhancement to the timer API - called timer slack - which should make it easier to coalesce timer events. In essence, it adds a certain amount of fuzziness to timer expiration times, giving the kernel some flexibility in how the timers are scheduled. That fuzziness is set with:

    void set_timer_slack(struct timer_list *timer, int slack_hz);

In essence, this call says that any timeout scheduled with the given timer can be delayed by up to slack_hz jiffies. By default, the slack is set to 0.4% of the total timeout period - a very conservative value. When the timer is queued, the actual expiration time is determined by means of a simple algorithm to choose a well-defined time within the slack period.

The value of this approach is that it makes it easy to coalesce timer events from multiple sources without needing to change every call site. Additional flexibility can then be had by increasing the slack for specific, frequently-used timers, but, even without that, slack timers should improve power efficiency on many systems.

Comments (4 posted)

The end of user-space mode setting?

It has now been a year since kernel mode setting (KMS) went into the mainline. KMS moves control of low-level graphics processor modes into the kernel and away from user-space drivers, with a number of associated advantages. Initially only the Intel driver supported KMS, but it has found its way into the Radeon and Nouveau drivers. Now developers are beginning to talk about eliminating user-space mode setting support entirely.

On the Nouveau front, Ben Skeggs posted a patch to remove non-KMS support, saying:

The main objection to the removal of this code is that BSD-based systems do not support KMS, but the current driver does not work on those systems anyway. So, while this patch has not found its way to the mainline, it would not be surprising if that happened before the 2.6.34 release.

At about the same time, some Intel driver developers started to ask whether non-KMS support could be dropped. There, too, it seems that the user-space mode setting code is unloved and proving hard to maintain. This code looks like it will remain an unwelcome guest for a while, though; Linus is in no hurry to remove it, and Dave Airlie is even more reluctant:

So the removal of non-KMS support from the Intel driver is being held up by concerns about the stability of the KMS code. But there is a bigger issue as well: Intel support has been in the kernel for years, so there are plenty of systems which are dependent on user-space mode setting. That means that the support needs to be maintained for long enough to be sure of not breaking those systems. Nouveau, instead, has the advantage of not having been in the mainline until now, so the same regression concerns do not apply. There are advantages, sometimes, to being the latecomer.

Comments (19 posted)

Kernel development news

sys_membarrier()

Mathieu Desnoyers is the longtime developer of the LTTng tracing toolkit. A current project of his is to provide for fast tracing of multithreaded user-space applications; that, in turn, requires a fast, multithreaded tracing utility. Tracing is controlled through a shared memory area; to make that control as fast as possible, Mathieu would like to use the read-copy-update (RCU) algorithm. That, in turn, means that he has been working on porting RCU - a kernel-only technology - to user space. In the process, he has run into some interesting challenges.

As with the kernel version, user-space RCU works by deferring the cleanup of in-memory objects until it is known that no more references to those objects can exist. The implementation must be done differently, though, since user-space code is unable to run in the same atomic mode used by RCU in the kernel. So, in user space, a call to rcu_read_lock() sets a variable in shared memory indicating that the thread is in an RCU critical section. Within that critical section, it's safe for the thread to access RCU-protected variables.

...at least, it's safe as long as nobody reorders operations in a way that causes an access to happen to an RCU-protected variable before the effects of rcu_read_lock() are visible to other CPUs. That kind of reordering can indeed happen, at both the compiler and CPU levels, so it's a problem which must be addressed. Compile-time reordering is relatively easy to avoid, but runtime reordering in the CPU requires the issuing of a memory barrier instruction. And, indeed, user-space RCU can be made to work by putting memory barriers into the rcu_read_lock() call.

The problem with that solution is that memory barriers slow things down significantly. Even worse, they slow down the fast path for a case - a change to an RCU-protected variable - which happens rarely. So Mathieu would like to get rid of that barrier. To that end, he coded up a solution which sends a signal to every thread when an RCU-protected variable is about to be changed, forcing each thread to execute a memory barrier at that time. This solution does speed things up, believe it or not, but signals are almost never the optimal solution to any problem. Mathieu would like to do something better.

His "something better" turned out to be a simple system call:

    void membarrier();

The initial implementation would simply send an inter-processor interrupt to every CPU in the system; the receiving CPUs would respond by executing an explicit memory barrier instruction. The solution worked, but it ran into a couple of objections in review:

• By allowing a user-space program to force interrupts to all processors on the system, membarrier() presented an easy way to create denial-of-service attacks on the system.

• The system call interrupted every processor on the system. Interrupting processors running different applications is a small but useless waste. The problem gets a little worse if some of those CPUs are running realtime tasks, which, presumably, would not welcome the forced addition of a bit of latency into their world. It would even interrupt processors which were currently sleeping - a useless exercise which would also increase energy use.

What followed was a long discussion on how to optimize the patch, whether an explicit memory barrier is needed even after the CPU has taken an inter-processor interrupt (the safe answer appears to be "yes"), and so on. All told, an impressive amount of effort has gone into the optimization of a small patch which is, at its core, implementing the slow path which should be rarely executed.

Current status, as of this writing, is that Mathieu has posted a new version of the patch with a number of changes. The first of those is the addition of an integer "expedited" parameter. If this value is zero, the system call simply calls synchronize_sched() and returns; this is the cheapest way of getting the needed functionality, but it comes at the cost of a latency of some milliseconds for the caller. It seems clear that Mathieu expects the "expedited" mode to be used most of the time.

For an expedited barrier, the system call will look at every CPU in the system, building a mask of those which are running in the same address space as the caller; those CPUs will then receive the inter-processor interrupt asking them to execute a memory barrier instruction. It's a rather more complicated implementation, but, since it only interrupts processors which are running the calling application, the denial of service, performance, and energy use concerns are no longer relevant. One assumes that the patch is getting close to its final form, but it's hard to say for sure: sometimes it's the smallest and simplest patches which are scrutinized the most.

Comments (2 posted)

FBAC-LSM

Process sandboxes for security are getting lots of attention these days. There are standalone utilities like isolate and Rainbow, sandboxes integrated into applications like the Chromium sandbox, as well as tools that use existing LSMs such as the SELinux sandbox. Furthermore, there are various proposals floating around to add Linux kernel features in support of application sandboxes, such as the seccomp additions and network restrictions. An LSM specifically designed for application sandboxing, which uses a new model called Functionality-Based Application Confinement (FBAC), was introduced on linux-kernel back in December.

FBAC-LSM came out of Z. Cliffe Schreuders's PhD research, and is a prototype implementation of the FBAC model. It uses an earlier version of the LSM interface, with the AppArmor pathname-based hooks and still needs "quite a bit of work to be done before it is ready for production systems or formal code review." Schreuders is looking for collaborators to work on completing the project, presumably with an eye towards getting it into the mainline.

The basic idea behind FBAC is to make security policy more accessible and understandable to users, so that application restrictions are more widely adopted. A major component of the FBAC system is a GUI-based policy manager that can guide users through setting policies for particular applications. Users specify the high-level needs of an application based on its type (such as web browser or file editor) and the policy manager will help create the policies that will govern its behavior.

In developing the policy manager, Schreuders analyzed over a hundred different applications to extract common behaviors that could be encapsulated in the FBAC policies. This allows the policy manager to automate certain aspects of developing policies for new applications, including things like configuration files, network ports, and other resources that the application requires. The policy manager also has a "learning mode" where it can observe the application and suggest additional privileges that might be granted.

FBAC has the underlying concept of "functionalities", which are essentially a set of permissions for file and network operations that are allowed. These are fine-grained permissions for things like "file_read", "file_getattr", "file_execute", "dir_mkdir", "network_incoming", etc. The permissions which are granted to a particular functionality are listed in its definition.

Functionalities are hierarchical, so that they can incorporate other, lower-level permissions into one that governs an entire application or class of applications. In addition, they are parameterized so that a functionality can be applied to multiple different applications, with the parameters specifying the particular files, directories, and network destinations that the permissions are granted for.

Both mandatory access control (MAC) and discretionary access control (DAC) are supported by FBAC. Application policy can be permanently set by an administrator, so that an ordinary user cannot make changes, or FBAC can be configured to allow users to further restrict applications beyond the policies set by the administrator. The confinement of an application then depends on the intersection of these mandatory and discretionary policies.

Allowing users to specify reduced privileges for arbitrary applications risks running afoul of the problems with setuid() programs that other sandboxing mechanisms (the network restrictions feature mentioned above for example) have encountered. Some means of stopping unprivileged users from changing the environment expected by setuid() programs will need to be provided.

The interface to FBAC-LSM is via a filesystem which is mounted at /sys/kernel/security/fbac-lsm. Various files in the directory allow querying the existing installed policies as well as adding new ones. There are several steps to sending the policy information, with each piece being written to a separate file in the directory. That is followed by "commit" being written to /sys/kernel/security/fbac-lsm/commit, which actually causes the policy to be processed. That is rather race-prone, but is required by the sysfs "one value per file" rule. It seems likely that FBAC-LSM will eventually change its interface to a private filesystem like those used by Smack and SELinux.

FBAC is a different approach from that taken by other security solutions, but it has enough similarities that Schreuders has plans to make the policy manager read and write AppArmor and SEEdit policies. But FBAC definitely lives up to its prototype billing. The code is rather disorganized and littered with commented-out sections that make it somewhat hard to follow.

The current incarnation of FBAC-LSM certainly has the feel of code that was put together somewhat hurriedly for a PhD dissertation, rather than as a "real" LSM. But it does embody some interesting ideas that merit further attention. One of the biggest hurdles faced by various security solutions (for which SELinux is the poster child) is the complexity of developing and—more importantly—understanding the policies that are being used. That complexity is something that Schreuders set out to reduce with FBAC. It remains to be seen if he has succeeded with that, but any such attempt is worthy of a look.

Comments (5 posted)

Speculating on page faults

Improving the performance of the kernel is generally a good thing to do; that is why many of our best developers have put considerable amounts of time into optimization work. One area which has recently seen some attention is in the handling of soft page faults. As the course of this work shows, though, performance problems are not always where one thinks they might be; sometimes it's necessary to take a step back and reevaluate the situation, possibly dumping a lot of code in the process.

Page faults can be quite expensive, especially those which must be resolved by reading data from disk. On a typical system, though, there are a lot of page faults which do not require I/O. A page fault happens because a specific process does not have a valid page table entry for the needed page, but that page might already be in the page cache, in which case handling the fault is just a matter of fixing the page table entry and increasing the page's reference count; this happens often with shared pages or those brought in via the readahead mechanism. Faults for new anonymous pages (application data and stack space, mostly), instead, can be handled through the allocation of a zero-filled page. In either case, the fault is quickly taken care of with no recourse to backing store required.

In many workloads, this kind of "soft" fault happens much more often than hard faults requiring actual I/O. So it's important that they be executed quickly. Various developers had concluded that the kernel was, in fact, not handling this kind of fault quickly enough, and they identified the use of the mmap_sem reader/writer semaphore as the core of the problem. Contention wasn't the issue in this case - only a reader lock is required for page fault handling - but the cache line bouncing caused by continual acquisition of the lock was killing performance. As the number of cores in systems increases, this kind of problem can only get worse.

In response, Hiroyuki Kamezawa posted the first speculative page fault patch back in November. The core idea behind the patch was to try to handle page faults without taking mmap_sem at all. Doing so invites race conditions; in particular, the vm_area_struct (VMA) structure which controls the memory mapping can change while the work is in progress. So the speculative fault code must handle the fault, then check for concurrent changes and, if necessary, redo the work the older, slower way. That's the "speculative" part: doing the work in a lockless mode in the hope that the world will not change in the meantime and force that work to be done again.

The speculative page fault code must also ensure that no changes which could create real trouble (such as freeing the VMA outright) can happen while the fault is being handled. To this end, various versions of the patch have tried techniques like adding reference counts to the VMA structure or using read-copy-update with the red-black tree code (which is used to locate the VMA covering a specific address within an address space) to defer changes while the speculative code is doing its thing.

This work yielded some real results: the number of page faults per unit time that the system could handle, when running a fault-heavy benchmark, approximately doubled. The idea clearly had merit, but Peter Zijlstra felt that Kamezawa-san's patches "weren't quite crazy enough". He set out to rectify this problem with a speculative page fault patch of his own, which saw a number of revisions. Peter's approach included the addition of speculative page table locks and the use of RCU to manage VMA structures. The result was a patch which would "sometimes boot" and which seemed to improve performance.

This is about when Linus got involved, pointing out some problems with Peter's patch, concluding:

Peter agreed with this conclusion, noting that he'd never thought it was ready yet.

It was in further discussion that Linus, looking at a profile of page fault handling activity, noticed something funny: the real overhead seemed to be in spinlock operations, which shouldn't be involved in the x86-optimized rwsem implementation at all. It turns out that said optimization was only applied to 32-bit systems; on 64-bit builds, reader/writer semaphores were using the generic, semaphore-based implementation. That meant that they were rather slower than they needed to be.

So Linus tossed out a new rwsem implementation based on the x86 exchange-and-add (xadd) instruction with a typical warning:

Kamezawa-san bravely tried the code anyway, and got an interesting result. His pets and his beer both came through without trauma - and the page fault performance was better than with his speculative fault patch. Peter, too, ran some tests against his own speculative code; those results showed that the rwsem change tripled page fault performance. His speculative fault patch improved performance by just a tiny bit more than that, and the two together a little more yet. But the rwsem patch is a small and clear fix, while the speculative page fault patch is large, widespread, scary, and with known problems. So nobody really disputed Peter's conclusion:

As of this writing, nobody has posted a final version of the rwsem patch. Linus has noted that there are things which can be improved with it, but it would be fairly typical for him to leave that work to others. But, one assumes, some version of this patch will be waiting in the wings when the 2.6.34 merge window opens. It will be a clear demonstration that low-hanging performance fruit exists even in our highly-optimized kernel; one need only think to look in the right place.

Comments (7 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 2.6.33-rc4 . (January 13, 2010)

• Greg KH: Linux 2.6.31.11 . (January 7, 2010)

Core kernel code

• Mathieu Desnoyers: introduce sys_membarrier(): process-wide memory barrier . (January 7, 2010)

• Mathieu Desnoyers: [RFC PATCH] introduce sys_membarrier(): process-wide memory barrier (v5) . (January 13, 2010)

• Arjan van de Ven: timers: Introduce the concept of timer slack for legacy timers . (January 11, 2010)

• Dave Chinner: lib: introduce generic list_sort() . (January 12, 2010)

• Arnaldo Carvalho de Melo: rbtree: Introduce rb_for_each_entry . (January 13, 2010)

• Davide Libenzi: eventfd - allow atomic read and waitqueue remove . (January 13, 2010)

Development tools

• Dan Carpenter: smatch 1.54 . (January 11, 2010)

• Srikar Dronamraju: [PATCH 0/7] UBP, XOL and Uprobes . (January 11, 2010)

• Jason Baron: jump label v4 . (January 12, 2010)

Device drivers

• srinidhi kasagar: i2c: Add support for Ux500/Nomadik I2C controller . (January 7, 2010)

• Rafael J. Wysocki: PCI run-time PM support (rev. 3) . (January 11, 2010)

• Eric Miao: gpio: introduce gpio_request_one() and friends . (January 11, 2010)

• Maxim Levitsky: RFC: [PATCH V2 0/9 Integration of SmartMedia/xD into mtd subsystem . (January 11, 2010)

• Rick L. Vinyard Jr.: hid: Logitech G13 driver 0.0.3 . (January 11, 2010)

• Jeff Kirsher: virtual function ixgbe driver . (January 11, 2010)

• David Daney: Add I2C support for Octeon SOCs. . (January 11, 2010)

• srinidhi kasagar: I2C driver support for Nomadik/Ux500 . (January 11, 2010)

• Cory Maccarrone: [MFD] i2c-htcpld: Add the HTCPLD driver . (January 11, 2010)

Filesystems and block I/O

• Christoph Hellwig: XFS status update for December 2009 . (January 12, 2010)

Kernel building

• Michal Marek: nconfig . (January 7, 2010)

Memory management

• Gleb Natapov: add MAP_UNLOCKED mmap flag . (January 12, 2010)

Benchmarks and bugs

• Rafael J. Wysocki: 2.6.33-rc3-git3: Reported regressions from 2.6.32 . (January 11, 2010)

• Rafael J. Wysocki: 2.6.33-rc3-git3: Reported regressions 2.6.31 -> 2.6.32 . (January 11, 2010)

Miscellaneous

• Karel Zak: util-linux-ng v2.17 (stable) . (January 11, 2010)

• Kay Sievers: udev 150 release . (January 11, 2010)

Page editor: Jonathan Corbet

Distributions

News and Editorials

Ubuntu Women Project

The Ubuntu-Women project "is a team functioning under Ubuntu to provide a platform and encouragement for women to contribute to Ubuntu-Linux" Women are generally under-represented in Free/Open Source software and this project seeks to get more women involved in free software in general and in Ubuntu in particular.

The project was founded in 2006, according to the project wiki and it is currently quite active. There is a mailing list, an IRC channel (#ubuntu-women at irc.freenode.net), a forum and even its own planet.

The project has gotten so large lately that they feel the need for more leadership, and have asked the Ubuntu Community Council to appoint an interim leader. Elizabeth Krumbach wrote: "This team leader will hold this position for a minimum of 6 months, at which point the position will be re-evaluated. She will guide the project through formalizing a "voting team" for election of the next leader(s) and helping us work through our RoadMap for the Lucid Cycle." There are three candidates for interim leader: Amber Graner, Penelope Stowe and Melissa Draper.

In addition to finding a leader, the project seeks to clarify the purpose of the IRC channel. The channel is currently a place to hold project meetings and discuss project business, a place for idle (off-topic) chit-chat, and everything in between. For example, some women see the channel as a safe haven to to go when they are being harassed elsewhere. The channel logs are not archived and that is a point of contention. Some think that project business discussions should be archived, but not the idle chit-chat. Logging complaints could help to document the situation. But the logs might also be used against the complainant, for example during a job interview. Several options have been proposed.

One option is to split the current channel into two channels, leaving #ubuntu-women as a social channel, which is not archived, and create a separate channel called #ubuntu-women-project which would be a logged channel for project business. The second option is to ban off-topic chatter in #ubuntu-women and to create a channel specifically for complaints. The third option is to log all chatter on the #ubuntu-women channel, and also create a separate channel where the project leader and her team can discuss any issues that arise. That second channel would not be publicly logged, but the logs would be available on request by the Community Council. There are a few that feel that creating a second channel would fracture the project and would eventually lead to its demise, but overall there is some consensus that a second channel is needed.

Melissa Draper posted her concerns on her blog.

Melissa is also in favor of separating the business side of the channel from the social side. "I genuinely believe it will be more effective to split out the project stuff and have #ubuntu-women-project. I believe it is harder to move social/emotional discussion as doing so breaks the mood or potentially hits nerves."

The project is also working on revamping its wiki page, and continues to increase the participation of women in the Ubuntu project. It's a place where people can go for mentoring and encouragement. Men are welcome to join the project, participate in the mailing list and IRC, to help and be helped. If you have considered getting involved in Ubuntu but are not sure where to start, check out the Ubuntu Community website. If you need more mentoring though, Ubuntu Women might be able to help.

Comments (none posted)

New Releases

Mandriva Linux 2010 Spring alpha1 is out

The first alpha for Mandriva Linux 2010.1 (Spring edition) is now available. "Many improvements and new functionalities are planned for this new version: your desktop will be smart and connected! Smart desktop is still one of the focus of main version, you can have a look on the coming roadmap. But you will find also easy home encryption so that your personal data are secured even where ever you are. Also planned a big work on our tools to manage software installation and update to give more useful information and help user in choosing the best of open source software."

Comments (none posted)

Distribution News

Debian GNU/Linux

Changes for Squeeze in Debian Installer

Frans Pop takes a look at some changes to the Debian installer for the upcoming squeeze release. Some of the changes include: recommended packages are installed by default, changes in the selection of language/country/locale, and changes in the partitioner. "So here's an overview of the more important changes in D-I since Lenny that are available in current daily and weekly built images. Note that for different reasons there are issues with daily/weekly images for various architectures. The images for i386, amd64, armel and sparc are fairly reliable. Images for other architectures may at times be either outdated, unavailable or broken."

Full Story (comments: none)

Fedora

Final Fedora Board appointment

Fedora Project Leader Paul Frields has announced that Colin Walters is the final appointee on the Fedora board. "To fill the final open seat on the Board for the next two releases, I am appointing Colin Walters. Colin has spent several years developing technology and community in the GNOME Project and around the varied landscape of Fedora's desktop. He brings to the Board a constructive, positive spirit to solving problems in Fedora and upstream. His recent work on advancing ideas and code for a unique but highly usable personality for the free desktop is also very exciting."

Full Story (comments: none)

Colin Walters: On the Fedora Board

Colin Walters has been appointed to the Fedora Advisory Board and posts on his plans. "Now that I'm on the Fedora Project Board, you may be wondering what my plans are. The first answer is - ideally - not much! Ideally, no one posts semi-nude material on the planet, we all cooperate nicely on the mailing lists, and in general the construction of a Free Software operating system and applications basically runs itself, and I can spend most of my time working on code too. However, we aren't quite in an ideal state, so let me give you a sense of my thoughts and goals."

Comments (none posted)

John Poelstra: The Fedora Board So Far

Fedora board member John Poelstra reflects on his term on the board. "I believe it is the job of the Fedora board to provide vision and leadership. Right now a big part of this vision needs to be who the Fedora distribution is for. This isn't to say these ideas and leadership can't come from others in Fedora. It is great when they do. Ultimately though, the Fedora Board is accountable for providing a vision for the future, conveying that that vision in a compelling way to Fedora, making changes to that vision based on feedback from other project members, and making sure the right things are in place for success."

Comments (none posted)

Fedora Board Recap 2010-01-07

Click below for a recap of the January 7, 2010 meeting of the Fedora Board. Topics include fedoraturkiye.org, Appropriate material for Planet, and more.

Full Story (comments: none)

Fedora mailing list migration complete!

Jon Stanley reports that Fedora's mailing list migration to Fedora infrastructure has been completed. "Many thanks to all involved in this effort, specifically Marek Mahut from Red Hat IT, and Dennis Gilmore from Fedora Infrastructure. Without both of them, this would never have been possible."

Full Story (comments: none)

Ubuntu family

Ubuntu Developer Week

The next Ubuntu Developer Week takes place January 25 - 29, 2010. "Ubuntu Developer Week is a series of online workshops where you can: * learn about different packaging techniques * find out more about different development teams * check out the efforts of the world-wide Development Community * participate in open Q&A sessions with Ubuntu developers * much more... "

Comments (none posted)

Distribution Newsletters

Arch Linux Magazine, January 2010

The Arch Linux Magazine for January 2010 is available with the latest Arch Linux news. Inside you'll find a report from Devland, a featured interview with Ionut Mircea Biru (Wonder), and much more.

Comments (none posted)

DistroWatch Weekly, Issue 336

The DistroWatch Weekly for January 11, 2010 is out. "Linux distributions come in many flavours; some include thousands of packages on a half a dozen of DVDs, while others fit on a 30 MB media. SliTaz GNU/Linux falls into the latter category. But despite its small size, it is a highly versatile and modern distribution, featuring the latest Linux kernel and many extra applications in its online repositories. Read our first-look review to find out more. In the news section, Debian project leader hints at a possible release date of the project's next version, Slackware removes the last vestiges of the old IDE/ATA system from its current kernels, BSD Magazine transforms itself into an free online publication, and Foresight Linux promises to re-activate the development of its GNOME-centric distribution. Other topics covered in this issue include release roadmap for Mandriva Linux 2010.1, a comparative review of several netbook-oriented distributions, and a quick tip on restoring the GRUB bootloader in case of trouble. Happy reading!"

Comments (none posted)

Fedora Weekly News 208

The Fedora Weekly News for January 10, 2010 is out. "This issue kicks off with announcements, including a note on the final open seat on the Fedora Board being appointed with Colin Walters, upcoming deadline details for Fedora 13 new features and spins, and a Bugzilla upgrade and outage last week. In news from the Fedora Planet, new Chromium packages and SELinux tips, Fedora 13 marketing plans, and details on a class on Inkscape recently taught at a Boston middle school. In news from the Fedora Ambassadors, details on last week's Fedora Ambassador IRC class. In Quality Assurance news, many updates on the first weekly QA team meeting of 2010, details on a new test case for preupgrade, and details on an initial set of desktop validation test cases for Fedora. In Translation news, the very latest on Fedora 13 documentation and translation schedule, discussion with the Fedora QA Team for help with the Fedora Localization Project's testing events, and an announcement of new team members for French, Arabic and Russian translation teams. In news from the Design team, find out about the start of a new Fedora Design Spin, and graphic concepts for Fedora 13. This issue wraps with with security advisories for Fedora 11 and 12. Enjoy FWN 208 and welcome to 2010!"

Full Story (comments: none)

openSUSE Weekly News/105

This issue of the openSUSE Weekly News covers openSUSE Spotlight: The next openSUSE Survey, * Katarina Machalkova: YaST is falling, make a wish, * Joe Brockmeier: Bash 101: Working at the CLI, * openSUSE Forums: Kaffeine in KDE4, * h-online/Thorsten Leemhuis: Kernel Log - Coming in 2.6.33 (Part 1) - Networking, and several other topics.

Comments (none posted)

Ubuntu Weekly Newsletter #175

The Ubuntu Weekly Newsletter for January 9, 2010 is out. "In this issue we cover: Edubuntu bug day on Tuesday, January 12th, 2nd call for votes: Ubuntu Developer Membership Board Election, Simplified Main Inclusion Request process, New MOTU members, Ubuntu Manual Project, 2010: Your Year for Ubuntu Membership, Ubuntu Florida Team and the "Youth Build Day", Lanuchpad - Jonathan Lange: The Road Ahead, Community and Ubuntu Live Videocast, Ubuntu Women project growing in Strength, and much, much more!"

Full Story (comments: none)

Interviews

Interview: Arch Linux Team (OSnews)

OSnews has an interview with several members of the Arch Linux Team. "Tobias Kieslich: If it's not your first time [installing Arch Linux] and you have a decent connection, a running desktop is doable in 30 minutes. However, that requires reading and understanding of the documentation. To put it in other words, there are a lot of people out there driving cars. The majority of them are intimidated by looking under the hood. Arch Linux is targetting people who are not."

Comments (none posted)

Distribution reviews

Fedora 12 -- A 'Must Upgrade' and 'Strongly Consider' Distro (LinuxPlanet)

LinuxPlanet has a review of Fedora 12. "Fedora 12 is a great Linux distribution with an impeccable pedigree. While it might not be the best distribution to throw at a total newbie, it definitely provides one of the more technically solid and stable platforms around."

Comments (none posted)

Ubuntu Netbook Remix vs Moblin (TuxRadar)

TuxRadar looks at several netbook distributions particularly Ubuntu Netbook Remix and Moblin. "We're going to document the current state-of-the-art in mobile Linux, and uncover the innovation and the technology that has enabled recent developments to happen. And we're going to start with netbooks, as these desirable items are becoming increasingly important. Ideally, a netbook OS needs to take into consideration three things: the limited amount of screen space that these devices typically have, the need for applications to be quick and responsive, and fact that these devices have to last as long as possible without being connected to a power source. And this is exactly what both Moblin and Canonical's UNR have been designed to accomplish."

Comments (none posted)

Page editor: Rebecca Sobol

Development

Log message classification with syslog-ng

Operating systems, applications, and network devices generate text messages of the events that happen to them: a user logs in, a file is created, a network connection is opened to a remote host, etc. These messages, called log messages, can be used to detect security incidents, operational problems, policy violations, and are useful in auditing and forensics situations. Traditionally, classifying log messages has been done external to the syslog system, with various log file analysis utilities, but a new feature in syslog-ng seeks to do that processing within the syslog daemon itself. By using a simpler syntax for describing log messages, along with a fast mechanism for recognizing them, message classification in syslog-ng can decrease the need for log file post-analysis, which will help ease the burden for system administrators.

Log messages do not have a predefined content, they can be straightforward or obscure, depending on the attitude of the developer who wrote them. Either way, most of the time they are written with human readers in mind. This ignores the fact that these days more and more companies and organizations collect the log messages of their computers on a central log server and try to process them automatically to detect break-in attempts, network errors, and other issues.

Classifying messages with syslog-ng attempts to remedy this situation by making it possible to add metadata (e.g., event type like user login, hardware error) to the log messages. It can also extract the relevant data (like the username) from the messages and determine what to do or where to store the log message based on this information. For example, if you need to create reports about specific events, you can collect the messages of the relevant events into a separate log file, which can be used as the basis of the reports.

A brief introduction to syslog and syslog-ng

Applications usually send their log messages to the system logging daemon of the operating system, which delivers the messages to the place where the log messages are stored: to log files on the local machine (found typically under /var/log/), or to a remote server. Most UNIX and Linux operating systems use the syslogd application as the system logging daemon. The syslog daemon adds some meta-information (called the syslog header) to the received log messages, like the date and time the message was received, or the name or address of the host where it was created.

The nine-year-old syslog-ng project is a popular, alternative syslog daemon — licensed under GPLv2 — that has established its name with reliable message transfer and flexible message filtering and sorting capabilities. In that time it has gained many new features including the direct logging to SQL databases, TLS-encrypted message transport, and the ability to parse and modify the content of log messages. The SUSE and openSUSE distributions use syslog-ng as their default syslog daemon.

In syslog-ng 3.0 a new message-parsing and classifying feature (dubbed pattern database or patterndb) was introduced. With recent improvements in 3.1 and the increasing demand for processing and analyzing log messages, a look at the syslog-ng capabilities is warranted.

The main task of a central syslog-ng log server is to collect the messages sent by the clients and route the messages to their appropriate destinations depending on the information received in the header of the syslog message or within the log message itself. Using various filters, it is possible to build even complex, tree-like log routes. For example:

It is equally simple to modify the messages by using rewrite rules instead of filters if needed. Rewrite rules can do simple search-and-replace, but can also set a field of the message to a specific value: this comes handy when client does not properly format its log messages to comply with the syslog RFCs. (This is surprisingly common with routers and switches.) Version 3.1 of makes it possible to rewrite the structured data elements in messages that use the latest syslog message format (RFC5424).

Artificial ignorance

Classifying and identifying log messages has many uses. It can be useful for reporting and compliance, but can be also important from the security and system maintenance point of view. The syslog-ng pattern database is also advantageous if you are using the "artificial ignorance" log processing method, which was described by Marcus J. Ranum (MJR):

Artificial ignorance is a method to detect the anomalies in a working system. In log analysis, this means recognizing and ignoring the regular, common log messages that result from the normal operation of the system, and therefore are not too interesting. However, new messages that have not appeared in the logs before can signify important events, and should therefore be investigated.

The syslog-ng pattern database

The syslog-ng application can compare the contents of the received log messages to a set of predefined message patterns. That way, syslog-ng is able to identify the exact log message and assign a class to the message that describes the event that has triggered the log message. By default, syslog-ng uses the unknown, system, security, and violation classes, but this can be customized, and further tags can be also assigned to the identified messages.

The traditional approach to identify log messages is to use regular expressions (as the logcheck project does for example). The syslog-ng pattern database uses radix trees for this task, and that has the following important advantages:

• Classifying messages is fast, much faster than with methods based on regular expressions. The speed of processing a message is practically independent from the total number of patterns. What matters is the length of the message and the number of "similar" messages, as this affects the number of junctions in the radix tree.

• Regular-expression based methods become increasingly slower as the number of patterns increases. Radix trees scale very well, because only a relatively small number of simple comparisons must be performed to parse the messages.

• The syslog-ng message patterns are easy to write, understand, and maintain.

For example, compare the following:

A log message from an OpenSSH server:

    Accepted password for joe from 10.50.0.247 port 42156 ssh2

A regular expression that describes this log message and its variants:

    Accepted \ 
        (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \
        for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))? 

An equivalent pattern for the syslog-ng pattern database:

    Accepted @QSTRING:auth_method: @ for @QSTRING:username: @ from \ 
        @QSTRING:client_addr: @ port @NUMBER:port:@ @QSTRING:protocol_version: @

Obviously, log messages describing the same event can be different: they can contain data that varies from message to message, like usernames, IP addresses, timestamps, and so on. This is what makes parsing log messages with regular expressions so difficult. In syslog-ng, these parts of the messages can be covered with special fields called parsers, which are the constructs between '@' in the example. Such parsers process a specific type of data like a string (@STRING@), a number (@NUMBER@ or @FLOAT@), or IP address (@IPV4@, @IPV6@, or @IPVANY@). Also, parsers can be given a name and referenced in filters or as a macro in the names of log files or database tables.

It is also possible to parse the message until a specific ending character or string using the @ESTRING@ parser, or the text between two custom characters with the @QSTRING@ parser.

A syslog-ng pattern database is an XML file that stores patterns and various metadata about the patterns. The message patterns are sample messages that are used to identify the incoming messages; while metadata can include descriptions, custom tags, a message class — which is just a special type of tag — and name-value pairs (which are yet another type of tags).

The syslog-ng application has built-in macros for using the results of the classification: the .classifier.class macro contains the class assigned to the message (e.g., violation, security, or unknown) and the .classifier.rule_id macro contains the identifier of the message pattern that matched the message. It is also possible to filter on the tags assigned to a message. As with syslog, these routing rules are specified in the syslog-ng.conf file.

Using syslog-ng

In order to use these features, get syslog-ng 3.1 - older versions use an earlier and less complete database format. As most distributions still package version 2.x, you will probably have to download it from the syslog-ng download page.

The syntax of the pattern database file might seem a bit intimidating at first, but most of the elements are optional. Check The syslog-ng 3.1 Administrator Guide [PDF] and the sample database files to start with, and write to the mailing list if you run into problems.

A small utility called pdbtool is available in syslog-ng 3.1 to help the testing and management of pattern databases. It allows you to quickly check if a particular log message is recognized by the database, and also to merge the XML files into a single XML for syslog-ng. See pdbtool --help for details.

Closing remarks

The syslog-ng pattern database provides a powerful framework for classifying messages, but it is powerless without the message patterns that make it work. IT systems consist of several components running many applications, which means a lot of message patterns to create. This clearly calls for community effort to create a critical mass of patterns where all this becomes usable.

To start with, BalaBit - the developer of syslog-ng - has made a number of experimental pattern databases available. Currently, these files contain over 8000 patterns for over 200 applications and devices, including Apache, Postfix, Snort, and various common firewall appliances. The syslog-ng pattern databases are freely available for use under the terms of the Creative Commons Attribution-Noncommercial-Share Alike 3.0 (CC by-NC-SA) license.

A community site for sharing pattern databases is reportedly also under construction, but until this becomes a reality, pattern database related discussions and inquiries should go to the general syslog-ng mailing list.

Comments (4 posted)

System Applications

Database Software

PostgreSQL Weekly News

The January 10, 2010 edition of the PostgreSQL Weekly News is online with the latest PostgreSQL DBMS articles and resources.

Full Story (comments: none)

SQLObject 0.11.3 released

Version 0.11.3 of SQLObject has been announced, it is a minor bugfix release. "SQLObject is an object-relational mapper. Your database tables are described as classes, and rows are instances of those classes."

Full Story (comments: none)

SQLObject 0.12.1 released

Version 0.12.1 of SQLObject has been announced, it is a bugfix release. "SQLObject supports a number of backends: MySQL, PostgreSQL, SQLite, Firebird, Sybase, MSSQL and MaxDB (also known as SAPDB)."

Full Story (comments: none)

Embedded Systems

Arduino Ethernet 1.0b2 released

Version 1.0b2 of Arduino Ethernet has been announced. "Arduino Ethernet is a collection of libraries I have written to use within the Arduino programming environment in conjunction with the Ethernet shield." The Arduino is an open-hardware microprocessor platform.

Comments (none posted)

Interoperability

Samba 3.4.4 and 3.5.0rc1 released

Two new releases of Samba are available. Samba 3.4.4: "This is the latest stable release of the Samba 3.4 series." Samba 3.5.0rc1: "This is the first release candidate of Samba 3.5.0. This is *not* intended for production environments and is designed for testing purposes only."

Comments (none posted)

Package Management

RPM 4.8.0 released

Version 4.8.0 of RPM has been announced, it includes general bugfixes and enhancements.

Full Story (comments: none)

Virtualization Software

Anatomy of the libvirt virtualization library (developerWorks)

Over at developerWorks, M. Tim Jones examines the libvirt virtualization control API. "From just the small amount of capabilities that I've demonstrated in this article, you can see the power that libvirt provides. And as you can expect, there are a number of applications that are being successfully built on libvirt. One of the interesting applications is virsh (demonstrated here), which is a virtualization shell. There's also virt-install, which can be used to provision new domains from operating system distributions. The utility virt-clone can be used to clone a VM from another VM (covering both operating system and disk replication). Some of the higher-level applications include virt-manager, which is a general-purpose desktop-management tool, and virt-viewer, which is a lightweight tool for securely attaching to the graphical console of VMs."

Comments (7 posted)

Desktop Applications

Audio Applications

Amarok 2.2.2 released

Version 2.2.2 of the Amarok music player has been announced. "the Amarok team released version 2.2.2 of their music player today. This release includes the return of moodbar, custom labels and more."

Full Story (comments: none)

Klactoveedsedstene - a new MPlayer frontend

Viggo Simonsen has announced the Klactoveedsedstene project. "*Klactoveedsedstene* is an Audio Player frontend to the popular Mplayer engine, written in Java. It is very fast, light, simple - and with an advanced support for Album Art. It recognizes embedded Album Art, and is also mostly able to find the correct Album Art from the Internet, based on the "Artist" and "Album" ID3-tags".

Full Story (comments: none)

Desktop Environments

GNOME Software Announcements

The following new GNOME software has been announced this week:

• at-spi 1.29.5 (bug fixes and translation work)
• AT-SPI2 0.1.5 (new features and bug fixes)
• Cheese 2.29.5 (new features, bug fixes, code cleanup and translation work)
• cluttermm 0.9.5, clutter-gtkmm 0.9.5, clutter-box2dmm 0.9.1 (new features, bug fixes and documentation work)
• Emerillon 0.1.1 (bug fixes and translation work)
• EOG Plugins 2.29.5 (new features and translation work)
• Evince 2.29.5 (new features, bug fixes and translation work)
• Eye of GNOME 2.29.5 (new features, bug fixes and translation work)
• gbrainy 1.30 (new features, bug fixes and translation work)
• GNOME Development Monitor 0.1 (initial release)
• GNOME games 2.29.5 (new features, bug fixes and translation work)
• gnome-keyring 2.29.5 (new features and bug fixes)
• GNOME Media 2.28.5 (bug fixes and translation work)
• gnome-settings-daemon 2.29.5 (new features, bug fixes and translation work)
• GNOME System Tools 2.29.2 (new features, bug fixes and translation work)
• GLib 2.22.4 (bug fixes)
• GTK+ 2.18.6 (new features, bug fixes and translation work)
• GTK+ 2.19.3 (new features, bug fixes and translation work)
• gtk-engines 2.19.0 (new features, bug fixes and translation work)
• gtkmm 2.19.2 (new features and bug fixes)
• gtksourceviewmm 2.9.0 (new features, documentation and translation work)
• libchamplain 0.4.2 (new features and bug fixes)
• libgnomekbd 2.29.5 (new features and bug fixes)
• libgweather-2.29.5 (translation work)
• liboobs 2.29.2 (bug fixes)
• librep 0.90.5 (new features, bug fixes and code cleanup)
• libxklavier 5.0 (new features and API change)
• mousetweaks 2.29.5 (new feature and translation work)
• Orca 2.29.5 (bug fixes and translation work)
• osm-gps-map 0.4 (new features, bug fixes and documentation work)
• rep-gtk 0.90.2 (build improvements and and API changes)
• Sawfish 1.6.1 (new features, bug fixes and documentation work)
• SeaMonkey 2.0.2 (unspecified)
• tracker 0.7.15 (new features, bug fixes and documentation work)
• Zeitgeist 0.3.1 (new features, bug fixes and code cleanup)

You can find more new GNOME software releases at gnomefiles.org.

Comments (none posted)

KDE Software Compilation 4.4 Release Candidate 1 is available

Version 4.4 Release Candidate 1 of KDE has been announced. "Release Candidate 1 provides a testing base for identifying bugs in the upcoming KDE Software Compilation 4.4, with its components the KDE Plasma Workspaces, the Applications powered by KDE, and the KDE Development Platform. The list of changes between 4.3 and 4.4 is especially long. Important changes can be observed all over the place".

Full Story (comments: none)

KDE Software Announcements

The following new KDE software has been announced this week:

• Baires 1.1 R3 (unspecified)
• Krecipes 2.0-alpha5 (bug fixes)
• VisualNotification for Pidgin 0.1 (initial release)
• vlc append 0.1 (initial release)

You can find more new KDE software releases at kde-apps.org.

Comments (none posted)

Xorg Software Announcements

The following new Xorg software has been announced this week:

• dri2proto 2.2 (new features, bug fixes and documentation work)
• glproto 1.4.11 (new features, bug fixes and documentation work)
• xf86-video-qxl 0.0.6 (new features and bug fixes)
• xorg-server 1.7.4 (bug fixes)

More information can be found on the X.Org Foundation wiki.

Comments (none posted)

Encryption Software

GPGME 1.3.0 released

Version 1.3.0 of GPGME has been announced, it includes a number of enhancements. "We are pleased to announce version 1.3.0 of GnuPG Made Easy, a library designed to make access to GnuPG easier for applications."

Full Story (comments: none)

Geographical Software

Location-aware search with Apache Lucene and Solr (developerWorks)

developerWorks has posted a lengthy and detailed article on the implementation of spatial searches with the Lucene and Solr libraries. "I'll start with a brief review of some key Lucene concepts, leaving the deeper details to the reader to research. Next, I'll cover some of the basic concepts of geospatial search. GIS is a large field that could easily consume this entire article and many more, so I will instead focus on some basic concepts that should be fairly intuitive given the need to find services, people, and other items of interest on a daily basis. I'll round out the article with some discussion of the approaches available for indexing and searching spatial information using Lucene and Solr. I'll ground these concepts in a real, albeit simple, example using data from the OpenStreetMap (OSM) project."

Comments (1 posted)

Interoperability

Wine 1.1.36 announced

Version 1.1.36 of Wine has been announced. Changes include: "- Completion of the 16-bit separation. - Improved Shader Model 4 support. - A ton of memory leak fixes. - Improved debugging support for MinGW. - A number of MSHTML fixes. - Various bug fixes."

Comments (none posted)

Mail Clients

Claws Mail 3.7.4 unleashed

Version 3.7.4 of Claws Mail has been announced, it includes new capabilities, bug fixes and translation work. "Claws Mail is a GTK+ based, user-friendly, lightweight, and fast email client."

Full Story (comments: none)

Claws Mail Extra Plugins 3.7.4 unleashed

Version 3.7.4 of Claws Mail Extra Plugins has been announced. "The claws-mail-extra-plugins-3.7.4 package contains 20 plugins, including 1 new plugin: GeoLocation!"

Full Story (comments: none)

Music Applications

guitarix 0.05.8-1 bugfix release

Version 0.05.8-1 of guitarix, an electric guitar amplifier simulator, has been announced. "I know, there are many guitarix release last month, but this release fix a memory leak witch we have oversee in a long run. I strongly recommend guitarix users to update to this version."

Full Story (comments: none)

Office Applications

RawTherapee 3.0 alpha 1 and license changes

Version 3.0 alpha 1 of RawTherapee, a RAW editor/workflow manager, has been announced, the software has just been released under the GPL. "The first alpha version of RawTherapee 3.0 is available for download. Note that this is not a feature complete version. In V3.0 both major GUI/workflow and algorithm changes are planned. This alpha version demonstrates the new, much more efficient GUI, but it does not contain any algorithmic changes yet." (Thanks to Spider).

Comments (none posted)

Office Suites

OpenOffice.org Newsletter

The December, 2009 edition of the OpenOffice.org Newsletter is out with the latest OO.o office suite articles and events.

Full Story (comments: none)

Languages and Tools

Caml

Caml Weekly News

The January 12, 2010 edition of the Caml Weekly News is out with new articles about the Caml language.

Full Story (comments: none)

Java

Jato 0.0.2 released

Version 0.0.2 of Jato, a JIT-only virtual machine for Java, is out. "Jato is a JIT-only virtual machine for Java that can run some Java applications under GNU/Linux on modern 32-bit x86 CPUs that support the SSE2 instruction set. A port to the x86-64 machine architecture is currently being developed. Jato depends on GNU Classpath to provide core Java runtime classes. The VM is licensed under the GPLv2 with GNU Classpath linking exception."

Full Story (comments: 1)

Python

Python 2.7 alpha 2 released

Version 2.7 alpha 2 of Python has been announced. "Python 2.7 is scheduled to be the last major version in the 2.x series. It includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, and support for ttk Tile in Tkinter."

Full Story (comments: none)

IMDbPY 4.4 released

Version 4.4 of IMDbPY has been announced. "IMDbPY is a Python package useful to retrieve and manage the data of the IMDb movie database about movies, people, characters and companies. In this release, a huge number of bugs were fixed and many parsers were made more robust."

Full Story (comments: none)

Sphinx 0.6.4 released

Version 0.6.4 of Sphinx has been announced, it includes over 20 bug fixes. "Sphinx is a tool that makes it easy to create intelligent and beautiful documentation for Python projects (or other documents consisting of multiple reStructuredText source files)."

Full Story (comments: none)

Python-URL! - weekly Python news and links

The January 13, 2010 edition of the Python-URL! is online with a new collection of Python article links.

Full Story (comments: none)

Tcl/Tk

Tcl-URL! - weekly Tcl news and links

The January 8, 2010 edition of the Tcl-URL! is online with new Tcl/Tk articles and resources.

Full Story (comments: none)

Page editor: Forrest Cook

Announcements

Commercial announcements

CadSoft releases Eagle 5.7

CadSoft has released version 5.7 of their Eagle printed circuit CAD application. This release adds improvements to a number of different commands. See the What's new document for details.

Comments (none posted)

Articles of interest

Lenovo reveal non-Wintel Skylight Smartbook (GeekWithLaptop)

GeekWithLaptop reviews Lenovo's Skylight Smartbook. "Lenovo, the world’s fourth largest computer maker has just revealed details of their first Smartbook, and it doesn’t come with an Intel chip or a Microsoft Windows operating system either. The Skylight as it is called, runs on Qualcomm Inc’s Snapdragon Chip, a chip that is also used in SmartPhones and the device is based on Linux. It has a 10 inch screen with a 1,280 x 720 resolution; it has 20GB flash storage, 8GB miniSD card, and 2GB storage space in the cloud. Lenovo are aiming the Skylight at people who just want to get online so it has built in Wi-Fi and 3G and onboard 1.3 megapixel webcam and apparently the Skylight will give you more than 10 hours battery life so that’s cool."

Comments (none posted)

New Books

O'Reilly Media announces five new books

O'Reilly Media has announced the publishing of the following new books: CSS Cookbook, Third Edition By Christopher Schmitt, HTML & XHTML Pocket Reference, Fourth Edition By Jennifer Niederst Robbins, Make: Technology on Your Time Volume 21 By Mark Frauenfelder, R in a Nutshell By Joseph Adler and Bioinformatics Programming Using Python By Mitchell L. Model.

Full Story (comments: none)

Resources

The Gifting Season: Linux Audio For The Holidays (Linux Journal)

Over at Linux Journal, Dave Phillips takes a peek at progress made in various Linux audio tools over the last month or two. For example: "Werner Schweer's MusE audio/MIDI sequencer has attained release level 1.0. This milestone release has been a long time coming, and it seems to have been worth the wait. Stability has greatly improved in MusE's last few releases, and this version continues that trend. New features include support for VST/VSTi plugins over the dssi-vst bridge, scripting with the Python language, extended support for JACK, and many other additions."

Comments (1 posted)

Blog Postings

CodePlex Foundation Reports on 100 Day Goals (The Standards Blog)

Linux Foundation lawyer Andy Updegrove reports on the CodePlex Foundation's 100 day mark. "As I had ventured to guess in my earlier post, getting the technical program up and running has ranked as a higher priority than some of the other goals that had been announced for the 100 day burn-in period. Stated another way by Sam, "We bit off more than we could chew" with the original schedule. Not surprisingly today's announcement opens with reference to the technical achievements to date, as follows: 'A software code contribution agreement, new approaches to open source licensing, a project acceptance process, a first technology gallery and donated projects - these are among the early accomplishments of The CodePlex Foundation,...'" See this article for background information.

Comments (4 posted)

Stallman: On selling exceptions to the GPL

Richard Stallman, responding to Oracle's attempted purchase of Sun, has posted an essay on selling GPL exceptions. "In other words, selling exceptions permits some embedding in proprietary software, and the X11 license permits even more embedding. If this doesn't make the X11 license unacceptable, it doesn't make selling exceptions unacceptable."

Comments (68 posted)

Here We Go Again: Video Standards War 2010 (The Standards Blog)

Andrew Updegrove discusses the latest video format standards war, fueled by DRM. "One thing that is different this time around is that it's the content owners, and not the consumer electronics firms that are pushing hard for a solution, so it's not patent royalties and the ability to sell a new generation of electronic devices that is driving the action. Instead, what's involved are two different approaches intended to help content vendors somehow survive in the face of plummeting revenues and a continuing plague of piracy while allowing legal content owners to watch videos on whatever device they want, wherever and whenever they want, without having to carry around the original media on which they purchased, for example, a movie."

Comments (1 posted)

Wheeler: California: Open Source Software is Okay!

David A. Wheeler looks at California's recent declaration that the use of open source software is an "acceptable practice" for the state government. In his blog posting, Wheeler considers the impact that the declaration will have: "I think this is a big deal. Officially saying 'it's okay to use free/libre/open source software (FLOSS)' is really important before FLOSS can get widespread use in governments. Most technologists already understand the potential advantages of FLOSS, but they encounter a lot of resistance when they try to use or develop FLOSS in large organizations like governments. Far too many middle managers are instinctively afraid of change from 'the way we've always done it'. For example, they may be afraid of unseen problems, or afraid their bosses will rake them over the coals later."

Comments (none posted)

Interviews

Red Hat CEO On Recession, Virtualization, Ballmer (InformationWeek)

InformationWeek interviews Red Hat CEO Jim Whitehurst. "Now the MySQL database appears about to go inside Oracle, as it awaits final approvals on its bid to acquire Sun. It's possible once it does, Michael "Monty" Widenius, one of MySQL's authors will start promoting his version, MariaDB, as the only legitimate open source MySQL. If that happens, "the code will fork, which is one of the worst things that can happen to open source code," he said. A development project with outside reviewers, testers and contributors is stronger when the code remains intact, instead of "fracturing the community" that surrounds it, he said. If that occurs with MySQL, it would not necessarily be Oracle's exclusive fault, he added."

Comments (19 posted)

Contests and Awards

Voting open for the 2009 LinuxQuestions.org members choice awards

Voting is open for the 2009 LinuxQuestions.org members choice awards. "The Members Choice Awards allow the Linux community to select their favorite products in a variety of categories. Awards will be given out in 27 categories this year, including Server Distribution of the Year, Desktop Distribution of the Year, Browser of the Year, Office Suite of the Year, Desktop Environment of the Year and Database of the Year. The polls will close on February 9th."

Full Story (comments: none)

Event Reports

Moblin Linux on x86 smartphone: Intel's small step forward (ars technica)

Ryan Paul reports on Moblin devices unveiled at CES. "The GW990 will be Moblin's first real test running on a smartphone form factor, but the platform is becoming an increasingly desirable choice on netbooks. Samsung had a large number of netbooks on display at CES, but its N127—running Novell's SUSE-based variant of Moblin—really stood out. Moblin's snappy and visually refined user interface is impressive and demonstrates the value that Linux can bring to the netbook market."

Comments (16 posted)

Tux takes a bow: Linux makes presence known at CES (ars technica)

ars technica reports on the Linux presence at the 2010 Consumer Electronics Show. "The open source Linux operating system is arguably a major force in the mobile and embedded space and can be found on a growing number of popular devices ranging from the TiVo to Amazon's Kindle. It's not surprising that the proverbial penguin has a strong presence at CES this week, where gadget makers from around the world are unveiling their latest and greatest toys. Touchscreen devices are the new hotness this year and are arriving with Linux in a number of different form factors and configurations."

Comments (none posted)

Calls for Presentations

LAC2010 call for music

A call for music has gone out for LAC2010, submissions are due by February 14. "Linux Audio Conference 2010 The conference about Open Source Software for music and audio May 1-4 2010 Hogeschool voor de Kunsten Utrecht (HKU) Utrecht, The Netherlands".

Full Story (comments: none)

Linux-Kongress 2010 call for papers

For those who like lots of advance warning: the call for papers for Linux-Kongress 2010 is out. The event is happening September 21 to 24 in Nuremberg; submissions are due by May 24. "You are invited to participate in this traditional Linux event by giving a presentation about your current development work, the future plans and experiences with Linux on a high level. The Linux-Kongress is the best opportunity for Open Source developers to meet in Europe and exchange ideas about current and upcoming projects that play and will play an important role for Linux."

Comments (none posted)

Netbook Summit Call for Presentations

A call for presentations has gone out for the Netbook Summit 2010. The event takes place in San Francisco, CA on May 24-25, submissions are due by February 12. "The Netbook Summit will provide attendees with practical information on the current state of netbooks, the new low-cost mobile computers."

Full Story (comments: none)

PostgreSQL Conference East, Change of Venue and Dates

PostgreSQL Conference East has announced a change of venue and dates. "The aggressive marketing campaign has caused the conference to be four days, March 25th - 28th. We have also moved from Drexel University to the Radison Plaza, Warwick Hotel. This is to better allow for business professionals outside of our normal community to attend the conference. It is also to allow for the most exposure to potential exhibitors."

Full Story (comments: none)

Call for Papers: sambaXP 2010

a call for papers has gone out for sambaXP 2010. "From May 3rd to 7th 2010 developers and users will meet again in Goettingen, Germany at the ninth international Samba conference, the "samba eXPerience 2010". The sambaXP is the leading event with focus on the most important free alternative to proprietary SMB/CIFS servers. The call for papers and early bird registration are open until January 31st 2010."

Full Story (comments: none)

Upcoming Events

More FOSDEM speaker interviews

Another round of four interviews with FOSDEM speakers have been published. This time around, the speakers are Sergey Petrunya (MariaDB), Brooks Davis (Promoting open source methods at a large company), David Recordon (Scaling Facebook), and Dylan Schiemann (Dojo Toolkit). From David Recordon's interview: "We're planning to talk about how Facebook has been able to scale to over 350 million monthly active users via open source software. Some of this infrastructure was developed outside of Facebook, but we've also released about a half-dozen core pieces of infrastructure we've developed. If it wasn't for the LAMP stack, Mark Zuckerberg never could have built Facebook from his Harvard dorm room. This is a common story for many sites that we all use every day."

Comments (none posted)

OpenClinica Global Conference announced (LinuxMedNews)

LinuxMedNews has announced the first OpenClinica Global Conference. "The worldwide community around OpenClinica, the rapidly growing open source clinical trial software, will gather on March 22nd, 2010 in Bethesda, Maryland (USA) for the first ever OpenClinica Global Conference. The event will bring together users and developers from diverse backgrounds to share experiences and expertise in using the increasingly popular open source software for clinical trial electronic data capture and clinical data management."

Comments (none posted)

PGCon 2010 announced

PGCon 2010 has been announced. "PGCon 2010 will be held 20-21 May 2010, in Ottawa at the University of Ottawa. It will be preceded by two days of tutorials on 18-19 May 2010. We are now accepting proposals for talks."

Full Story (comments: none)

Keynotes picked for SCALE 8x

The SCALE 8x keynotes have been announced. "Tarus Balog and Karsten Wade have been chosen to give the keynotes at SCALE 8x, held at the Westin Los Angeles Airport hotel from Feb. 19-22."

Full Story (comments: none)

Events: January 21, 2010 to March 22, 2010

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
January 15 January 22	Camp KDE 2010	San Diego, CA, USA
January 18 January 23	linux.conf.au	Wellington, New Zealand
January 23	Workshop on GCC Research Opportunities	Pisa, Italy
January 23 January 24	DrupalSouth Wellington 2010	Wellington, New Zealand
February 2	Prague PostgreSQL Developers' Day 2010	Prague, Czech Republic
February 5 February 7	Frozen Perl 2010	Minneapolis, MN, USA
February 6	Super Happy Dev Castle #0	Belfast, N. Ireland, United Kingdom
February 6 February 7	Free and Open Source Developers' European Meeting	Brussels, Belgium
February 10	Red Hat Cloud Computing Forum	Online, Online
February 11 February 13	Bay Area Haskell Hackathon	Mountain View, USA
February 15 February 18	ARES 2010 Conference	Krakow, Poland
February 17 February 25	PyCon 2010	Atlanta, GA, USA
February 19 February 20	GNUnify	Pune, India
February 19 February 21	SCALE 8x - 2010 Southern California Linux Expo	Los Angeles, USA
February 20 February 21	FOSSTER '10	Amritapuri, India
February 22 February 24	O'Reilly Tools of Change for Publishing	New York, NY, USA
February 27 February 28	The Debian/GNOME bug weekend	Online, Internet
March 1 March 5	Global Ignite week	Online, Online
March 2 March 4	djangoski	Whistler, Canada
March 2 March 5	FOSSGIS 2010	Osnabrück, Germany
March 2 March 6	CeBIT Open Source	Hannover, Germany
March 5 March 6	Open Source Days 2010	Copenhagen, Denmark
March 7 March 10	Bossa Conference 2010	Recife, Brazil
March 13 March 19	DebCamp in Thailand	Khon Kaen, Thailand
March 15 March 18	Cloud Connect 2010	Santa Clara, CA, USA
March 16 March 18	Salon Linux 2010	Paris, France
March 17 March 18	Commons, Users, Service Providers	Hannover, Germany
March 19 March 20	Flourish 2010 Open Source Conference	Chicago, IL, USA
March 19 March 21	Panama MiniDebConf 2010	Panama City, Panama
March 19 March 21	Libre Planet 2010	Cambridge, MA, USA

If your event does not appear here, please tell us about it.

Web sites

OpenMediaVault web site launched

The OpenMediaVault open network attached storage solution project, a spin-off of FreeNAS, has launched its web site. See lwn article for the project's history. (Thanks to Koen Vervloesem).

Comments (none posted)

Page editor: Forrest Cook