LWN.net Weekly Edition for June 16, 2011

GNOME Shell, extensions, and control

It has now been three months since your editor posted his experience with GNOME 3 and GNOME Shell. Much of the intervening time has been spent gaining more experience with the system; it even included getting a better video card to enable the full GNOME Shell experience. The time has now come to start playing with the GNOME Shell extension mechanism. Some interesting things can be done with extensions, but it has also come out that parts of the project are hostile to the very idea of GNOME Shell extensions. In the end, one may well wonder: who controls our desktop software?

For the most part, your editor must conclude that GNOME 3 works well enough - though with some glitches. Moving the pointer too close to the upper-left-corner minefield and getting an unwanted overview mode experience still happens several times per day. It's still annoying to have to go through extra steps to get a second terminal or browser window. Workspaces must be set up with care to ensure that windows end up where they are expected to be. The top panel (the black bar at the top of the screen) still represents wasted space that could be put to better use. But, as a whole, things work well enough for a dot-zero release; one assumes that things will get better. GNOME 3 is not that bad.

One thing that is happening is that the number of extensions for GNOME Shell is growing. The Rawhide repository now includes a number of them. One extension puts a music player controller into the black bar; a nice idea, but it seemingly does not work with audacious. Another claims to remove the accessibility menu, but that one does not work at all. There are others your editor has not tried, adding a "places" menu, restoring the "shutdown" option, adding a dock-style task switcher, and more.

The "GNOME Shell frippery" extensions are not currently packaged by Rawhide, but your editor decided to give them a try anyway. With the full set installed, the down side of extensions quickly became clear; they simply crashed the shell altogether. The GNOME system was very helpful: the message read "Something went wrong, please log out and try again." One assumes that these extensions have fallen behind the current state of GNOME Shell. As it happens, there is no real mechanism (yet) for ensuring that the shell and extensions match; it seems likely that a number of extension users will run into this particular trap in the coming years.

Happily, one of the extensions in that package works just fine: the one which adds the "favorite" applications to the top panel. Clicking on the terminal icon there simply yields a new terminal, just as $DEITY intended. That one little change has helped to make your editor's GNOME Shell experience quite a bit more productive.

The extension mechanism built into GNOME Shell is quite flexible, so one could imagine that quite a few useful extensions will show up in the coming months and years. In a normal world, one would expect that the GNOME developers would welcome this development - others are making use of the capabilities provided to make the platform better. In the world we inhabit, the community has been somewhat less welcoming. Consider the discussion that ensued after Jasper St. Pierre outlined his plans for a web site where users could find (and post) their extensions. Allan Day, who does marketing work for the GNOME Foundation under contract, a former GNOME marketing contractor, responded:

He did admit that extensions can be "valuable as a crutch for our traditional users" and, perhaps, for experimenting with features. But, in general, it seems that, in his view, GNOME Shell users should accept what they are given and not seek to change the experience. Otherwise, all that design effort will have gone to waste, and, horrifyingly, the project's marketing might be impaired:

This is the point where one needs to ask what the purpose of the project is: to enable its users to be maximally productive and happy in front of their computers, or to increase the project's brand presence? One might actually think that those two goals should not conflict with each other, but, should that come to pass, one would hope that the users would win. The fact that they might not is cause for concern.

Allan is not alone in his views, but neither are those views unanimously held; others have understood that extensions are going to happen regardless:

Owen Taylor has also agreed that there will be extensions out there regardless of the project's wishes; he seems to feel that a central web site might help to bring some order and quality control to the situation:

A certain amount of order and control will be wanted; among other things, extensions are unconstrained code that can do anything the user is allowed to do on the system. A way to collect useful extensions, review them, and provide access to those known not to be malicious could make life a lot easier for everybody involved. This model has worked fairly well for the Firefox browser's extension mechanism; it could probably be made to work for GNOME Shell too.

In the end, this is not an Apple-style walled garden; we're the free software community, so chances are that the hackers will win out. We're not accustomed to letting others tell us how the software on our systems should work. That is doubly true when the system in question has an extension mechanism built into it. If the GNOME project does not create a mechanism by which extensions can be gathered, vetted, and distributed to users, somebody else will. One way or another, GNOME Shell will evolve in the directions its users want it to go.

Comments (230 posted)

GNU Telephony releases SIP Witch 1.0 and announces Free Call

In development since 2008, SIP Witch, the call server developed by the GNU Telephony project, made its stable 1.0 release in May. In conjunction with that milestone, GNU Telephony has also unveiled its next major project, GNU Free Call — a free, peer-to-peer routed voice calling network.

Thanks to the differences between SIP (Session Initiation Protocol) and other communication protocols, it can be confusing to explain exactly what SIP Witch does. SIP Witch is designed to be a low-resource connection router, maintaining the actual network location of active clients, and negotiating routes between clients whenever a call is made. Because SIP addresses are URIs, a router like SIP Witch is needed to map the URI to the actual endpoint where the user can receive a call. Establishing the connection also involves tasks such as NAT traversal. However, once the connection between the endpoints is set up, the client applications communicate directly with one another. In that respect, SIP is similar to XMPP (aka Jabber), where the "reachability" of the user is distinct from the call contents.

In contrast, full-blown call servers such as Asterisk or Bayonne handle the audio and video media streams between endpoints (including, potentially, transcoding between different formats), and run applications such as voicemail and various menu-driven interactive systems. SIP Witch can route calls to an Asterisk server, but also to standalone soft-phone applications, or to hardware devices like IP telephones or analog telephone adapters (ATAs). It can be used to manage SIP accounts provided by third-party service providers, to provide extensions within a LAN, or any combination of the two.

Ding-dong, the witch is calling

The 1.0 release does not add many features over the previous few milestones. According to the changelogs, most of the work has been on improving the build process, ensuring cross-platform compatibility, and stabilizing IPC hooks for integration with desktop environments. Since SIP Witch is a GNU project, source code releases are available directly from the GNU Project servers, but lead developer David Sugar has also made a special effort to ensure that SIP Witch is well-integrated with both the Ubuntu and Fedora distributions.

On a desktop system, SIP Witch can act as a proxy for local soft-phone clients, alleviating the need to individually manage network address translation (NAT), firewall traversal, and other settings. Although at the moment running multiple SIP clients is probably not commonplace, it is easy to imagine a few scenarios in which it would be useful. For example, since it is up to the individual client to determine which media payload to use, a user might have a preferred open source client, but also keep a back-up client installed in order to handle incoming calls using older or proprietary codecs. SIP Witch does not care what codecs are used between the call endpoints, since the media connection is made directly between the clients.

A useful side effect of that feature is that SIP Witch is automatically compatible with Phil Zimmermann's ZRTP end-to-end voice over IP encryption scheme because the key exchange and encrypted data streams exist outside of the SIP call set-up process. On the other hand, SIP Witch is also compatible with applications that use SIP for instant messaging, but because it does not handle the media payload, it does not queue or store any SIP IMs that arrive when the application is not running.

Where SIP Witch gets more interesting is in its routing functionality. First, more than one endpoint "device" can register with SIP Witch under the same ID. Thus, SIP Witch can ring multiple hardware or software phones for any incoming connection request. Second, SIP Witch supports peer-to-peer routing. Multiple SIP Witch instances can be set up to redirect calls to each other, a technique that can be used to provide fail-over, or simply to connect two or more locations into a single SIP "realm" as if they were a single site.

GNU Free Call

This sense of "peer-to-peer" routing is rather limited, but the project has loftier goals in mind. Namely, Sugar plans to add peer-to-peer SIP URI discovery to the code in the future, so that SIP Witch servers can automatically collect and cache the addresses of reachable SIP call endpoints. At the moment, any SIP user can call any other SIP user (including those on remote networks), but only if he or she already has the other party's SIP URI. There is no built-in way in SIP to discover another person's (or business's) URI.

Adding that peer-to-peer discovery functionality to SIP Witch would be straightforward from a technical perspective, but it very quickly escalates into a human interface issue. Once SIP Witch can discover the URIs of users on nearby nodes, the user must be pulled in to select the right John Doe out of a potentially long list of possible matches. That, in turn, means SIP Witch will need a GUI, and if SIP Witch has a GUI, new users are likely to expect it to provide them with an account (just as the proprietary alternative Skype does).

Consequently, GNU Telephony is not only planning to extend SIP Witch in the post-1.0 development cycle, but it is also launching a free calling network called GNU Free Call. In its official announcement, the project describes its goal as making GNU Free Call as ubiquitous and usable as Skype, including support on "all platforms" for use by the general public. In addition, it highlights goals of providing secure calls for both known and anonymous callers, but without requiring a central service provider, secret binary protocols, or network "control points" that could be exploited.

There is a basic roadmap sketched out in the announcement, starting with adding peer-to-peer discovery to SIP Witch as described earlier. The initial plan is to start by adding the caching of SIP URIs to the server, then to build a mechanism for publishing routes to connected peers — which is similar to the way peer-to-peer file-sharing services have operated. Sketches for the GUI front-end (based on the OLPC Sugar interface) are on the GNU Telephony wiki, with mock-ups that show local URI discovery, existing contacts, and tag-based searching.

Beyond the decentralized address-location problem, the other big challenge presented in GNU Free Call is providing secure communications. Although ZRTP is supported by a growing number of soft-phone clients (and indeed there is a GNU Telephony implementation of it), it cannot be added after the fact to most hardware devices, which leaves those users without a secure option. The project plans on supporting these users by adding a GnuPG public-key exchange step to SIP itself. That will allow security to be handled during the call set-up phase (as handled by SIP Witch), by establishing a secure SRTP channel between the endpoints, and using the GPG keys to mutually verify signed hashes of the session keys created.

Generally speaking, the project has frowned on security systems that rely on certificate authorities or public key infrastructure. This scheme has yet to be documented in detail, however it could be implemented with locally-generated key pairs. The traditional wisdom is that public key cryptography is too complicated for lay people to use, but perhaps the SIP use case will prove simpler to understand than has email encryption.

The future

The bigger question is whether or not GNU Free Call can produce a system that is easily as usable as Skype. Several other free software projects have set the same grand goal over the years, but for the most part they relied on duplicating Skype's business model, but with a different network (based on SIP). Ekiga, WengoPhone/QuteCom, and the others all produced standalone SIP clients that were designed to connect to centralized SIP networks run by the provider (even though they could still call other SIP users).

At the very least, GNU Free Call is taking a markedly different approach. SIP Witch allows the user to choose any client application, and the wiki and archived conference presentations show strong ties to mobile devices (including the possibility of an oFono-based cellular back-end), as well as a desire to integrate with existing desktop services such as address books, D-Bus signaling, and notification frameworks. Automatic node-discovery and peer-to-peer routing may be associated with difficult-to-use file-sharing services in many users' minds, but those networks were centered around broad searches and multiple active connections. SIP Witch's interest in point-to-point communication may not share much in common with them at all. None of those factors guarantees its success, but the GNU Free Call team is certainly in it for the long term.

Furthermore, the small-and-light SIP Witch code base is much easier to manage than the heavier GUI soft-phone clients, and free from the headaches of media codec and transcoding support (the 1.0 release weighs in at just 480K, compared to 10MB for Ekiga). In addition to the GUI interface needed for searching or filtering discoverable URIs, SIP Witch will also need to build a more user-friendly configuration system. Right now, it uses XML configuration files which, although they are well documented, might be intimidating to inexperienced users. SIP comes with its own vocabulary, some of which is not immediately intuitive to those who do not deal in real-time network communications.

GNU Free Call is not scheduled to roll out its actual service until "later this [northern hemisphere] summer." It will be an interesting debut to watch. It is sure to garner support from security-conscious free software types, but is aiming for a much wider audience.

Comments (2 posted)

WebOS: the other Linux-based mobile platform

Android handsets may get a lot of attention, but they are far from the only Linux-based devices on the market. HP's webOS is also very much a Linux-based system; it can be found on a handful of handsets and an upcoming tablet product. The folks at HP were kind enough to send your editor a Pre 2 handset to play with; what follows are some impressions from that experience. In summary: the Pre 2 is an open device and webOS is an interesting alternative, but webOS devices will have a hard time pushing Android handsets aside.

The Pre 2 is a smallish device with a somewhat rounded shape. Your editor's wife described it as "girlish," pointing out that it even has a mirror on the back side of the slider. The phone slides vertically when it is held in the portrait orientation, exposing a small QWERTY keyboard. Actually, it must be said that the keyboard is very small; it is difficult for your fat-fingered editor to use. Use of the keyboard is obligatory, though; there is no on-screen keyboard provided with the device. Charging is via a standard micro-USB port. One nice feature is a separate switch for putting the phone into a silent mode; the switch, unlike the profiles on many other handsets, is difficult to change accidentally.

The handset has an ARM OMAP processor with a mighty 300 bogomips of performance; it has 512MB of main memory and 16GB of flash storage. The screen is 320x480 - on the small side for contemporary devices, but the Pre 2 is a small device. The unlocked GSM version of the device has, in addition to the four standard GSM bands, support for 3G at both 850MHz and 1900MHz. 850MHz support, of course, is nice for those of us using a certain US-based carrier which, despite its size and appetite for competing carriers, tends to be overlooked by unlocked Android devices.

The webOS interface takes some getting used to, but it has its good points. The "home" screen is called the "card view"; it contains five application icons at the bottom. Whenever a new application is run, it is placed on the card view much like playing cards on a table. Each application's "card" tracks the current state of its screen, so it is possible to scroll through the card view and see what every running application is up to.

WebOS makes heavy use of gestures for control, to the point that the device wants first-time users to go through a tutorial (with tests at the end) before doing much of anything else. There is no "back" button; that functionality comes with a right-to-left swipe along the bottom of the screen. An upward swipe from the bottom goes to the card view; a second swipe pulls up the application menu. Running applications are stopped by flinging their cards off the top of the display. It takes a bit of practice to get used to the interface, but, before long, it feels quick and natural.

Other aspects of the interface are a little harder to get used to. A depth-first traversal of the Android settings menu will expose the user to almost every option the device has; webOS, instead, tends to hide things. A user wishing to turn off mobile data use, for example, will search for vain in the system settings menus; one must, instead, pull up the dialer application and hit the button in the upper left which contains the current mobile carrier's name. The "location services" screen controls access to a number of location-based options, but not whether GPS is enabled - that option is under a separate "preferences" menu. There may be some consistent philosophy behind the placement of these options but, to your editor, it seems somewhat arbitrary and confusing.

In some ways, the interoperability of the device is quite nice; contacts and calendar data, for example, can be obtained from a variety of sources and integrated on the phone. An Android user wanting to make the switch to webOS will find that much of the information stored in the Google mothership is equally accessible. In other places it falls down; it was not possible to test the email application because LWN's self-signed IMAP server certificate was not "trusted." Your editor, who created that certificate, trusts it just fine; it should be possible to impress that trust upon a phone. There is an involved series of steps said to force that trust; it would be much easier if the application simply asked.

Naturally enough there is a market for add-on applications; there is a fair amount of stuff there, but nowhere near as many applications as Android offers. Android users may be surprised at the lack of permissions questions; webOS seems to lack the set of privileges that Android provides. There is exactly one exception: users will be warned about applications which will request location data. It is also possible to configure the phone to ask before providing location information to any application, yielding an experience similar to that of configuring a web browser to ask about every cookie. But any other sort of access - network access, for example - seems to be entirely uncontrolled.

Various other little things seem to be missing. The camera hardware seems reasonable, but the associated application is minimal. The music player cannot play Ogg files. There's no indication when GPS or mobile data are in use. The Pre 2 cannot simultaneously function as a phone and as a USB storage device. The web browser lacks the semi-magic "format the page for the screen" mode that works so well on Android. There is no word completion or typo correction for keyboard input. And so on. Fans of webOS say that the system is more tightly integrated and consistent than Android, and that it handles multitasking in a more natural way. All of these may be true. Had your editor come to webOS directly from less-smart phones, he would certainly have been much impressed. As it is, it is hard not to conclude that this platform needs some work to become truly competitive in today's market.

Hackability and community

If one opens the phone while it is in the card view and types "upupdowndownleftrightleftrightbastart" (a spelled out version of the famous Konami Code), the result is a new icon offering to put the phone into developer mode. That turns out to be the key to what is a nicely open and hackable device. There's little that will get in the way of anybody wanting to make their Pre 2 do a number of interesting things. The first step is to get and build novacom, a tool which is the webOS analog to Android's adb. Plugging the phone to a computer's USB port, putting it into developer mode, and running novacom will yield a root shell on the device. From this point of view, webOS looks like a straightforward Debian-derived distribution (correction: it's actually based on OpenEmbedded); it is much easier to move around in and inspect than Android's minimal shell. It actually feels like Linux.

After that, there's a lot to be found on webos-internals.org, starting with Preware, a sort of alternative application repository. Through Preware, one can install application tweaks, useful features, and more. Your editor started with the "developer mode" button which took away the pain of typing the Konami Code on the Pre 2 keyboard. There is an on-screen keyboard (though, alas, it was not yet working with webOS 2.1.0) and a WiFi tethering application which, unlike the official one, works regardless of whether one's mobile carrier has blessed it. Some of what's in Preware looks more useful than the rest, but, as a whole, it's a repository that serious webOS users are likely to want to enable.

The device comes with a 2.6.24 kernel. HP has put up a nice page where source for all of its open-source components can be found; they provide original tarballs and the patches they have applied as separate files. The webOS 2.1.0 kernel patch is a daunting 10MB monster which touches almost 1200 files and adds nearly 350,000 lines of code. Much of that code is clearly post-2.6.24 backports, some of it is board support for the handset, and some is drivers. There is also a custom cpufreq governor. There are instructions for building and installing kernels on the device, but it's not possible to build the whole thing: webOS includes a small number of binary-only modules for wireless networking, graphics, and more. As a result, there are a lot of tweaked 2.6.24 kernels out there for webOS devices, but no current kernels.

The binary-only modules are unfortunate, even if the Pre 2 is far from unique in requiring them. The same could be said of the mostly proprietary application space. WebOS also has a development process which is even less open than Android's; source for shipped code is thrown over the wall, but there is no repository, no associated open source project, and no way for others to even try to add changes. That makes the generation of a high level of enthusiasm in the development community hard; that may explain why there does not appear to be an equivalent to CyanogenMod in the WebOS world.

That is a bit of a shame; the Pre 2 otherwise is a remarkably open device; one can get into it and hack away without having to go through any cumbersome rooting rituals. WebOS is a Linux-based handset (and, soon, tablet) platform that actually looks like Linux; unlike MeeGo, it is actually available on current handsets. It is an interesting platform which, one hopes, will continue to survive. Given the strength of the competition - Android has a much larger user base, more features, many more handsets, and many more applications - one might think that webOS could use all the help it can get. WebOS seems to have a crowd of dedicated users and developers who would likely provide that help if HP would just let them.

Comments (64 posted)

Page editor: Jonathan Corbet

Security

UEFI and "secure boot"

Hardware-based secure boot mechanisms are clearly useful for some users. By determining that firmware, bootloaders, and operating systems are not compromised, these mechanisms can protect systems against rootkits and other low-level attacks. Typically, the way that is done is by cryptographically signing the binaries in question such that they can be verified before being run. But disallowing unsigned binaries from running has a potentially problematic side effect: booting free operating systems becomes difficult or, in the worst case, impossible. It all depends on who holds the signing keys.

The Linux kernel has the integrity measurement architecture (IMA) and the proposed extended verification module (EVM) which could be combined with system hardware—such as the Trusted Platform Module (TPM)—to provide a secure boot environment. There have been concerns about these mechanisms as they can be used for both good and ill: either preventing rootkits and other tampering or preventing users from running code of their choice on their hardware. The unified extensible firmware interface (UEFI) specification has recently added some features that could be used similarly, leading to many of the same concerns. But there is also an additional wrinkle for systems that use the GRUB 2 bootloader.

UEFI is a relatively new standard for firmware that provides the boot environment for computer systems. For x86-64 systems (there are no plans to put UEFI in 32-bit x86 systems), it is aimed at replacing the 20+ year-old BIOS interface that is used to boot systems today. While BIOS is x86-specific, UEFI aims to be cross-architecture, and various vendors of ARM-based systems—Apple is rumored to be among them—have started adopting it. The UEFI 2.3.1 specification [agreement required] has a number of new features, one of which is the optional "secure boot" protocol. So far, UEFI 2.3.1 is not being shipped by any vendor and is only available on evaluation boards, but that will change in time.

The basic idea behind secure boot is to sign executables using a public-key cryptography scheme (RSA with 2048-bit keys with SHA-1 or SHA-256 as the hash). The public part of a "platform key" (PK) can be stored in the firmware for use as a root key. Additional "key exchange keys" (KEKs) can also have their public portion stored in the firmware in what is called the "signature database". That database contains public keys that can be used to verify different components that might be used by UEFI (e.g. drivers) as well as bootloaders, and operating systems that get loaded from external sources (disks, USB devices, network, and so on). The signature database will also contain "forbidden" signatures which correspond to a revocation list of previously valid keys. The signature database is meant to contain the current list of authorized and forbidden keys as determined by the UEFI organization.

Before a PK is loaded into the firmware, UEFI is considered to be in "setup" mode, which allows anyone to write a PK to the firmware. Writing the PK switches the firmware into "user" mode. Once in user mode, PKs and KEKs can only be written if they are signed using the private portion of the PK, though KEKs can be freely written during setup mode. Essentially, the PK is meant to authenticate the platform "owner", while the KEKs are used to authenticate other components, like operating systems.

All of this presents some problems for free software. If device makers create PKs and KEKs for the devices before shipping them to users, and do not provide the private portion of the keys, only entities listed in the signature database will be able to sign bootloaders and OSes. That is a fairly secure option for the device maker, but makes it difficult for those who might want to choose what code gets run on their hardware.

Secure boot is optional, but there is likely to be a fair amount of pressure applied by proprietary OS makers to enable it. One could imagine that those vendors might also provide a way to turn off secure boot (from a BIOS-like menu for example), but that is something that might be exploited by rootkits and other malware, so there may well be resistance to allowing that kind of option. Protecting users from rootkits and the like is certainly useful, but there is a competitive advantage as well. Hardware vendors can ensure that only the code they approve can run on the hardware, and proprietary OS vendors will be largely unaffected because their keys will be in the signature database. One would hope that the protection against malware is the primary motivation, but the ability to lock out free OSes is likely seen as a plus.

It is Linux and other free systems that could suffer most from secure boot implementations. While it would be possible for various distributions to get their keys added, that wouldn't help anyone who wanted to run a tweaked version of the "approved" bootloader or kernel. Distributors would not be able to release their private keys to allow folks to sign their own binaries either. Each key is just as valid as any other, so malware authors would just pick up those keys to sign their wares. Exposed keys would also find their way onto the forbidden list rather quickly one suspects.

But there is another potential pitfall here. The GRUB 2 bootloader, which is finding its way into some distributions, is licensed under GPLv3. One of the attributes of GPLv3 (the so-called "anti-Tivo-ization" language) requires that any keys needed to "install and execute modified versions of a covered work" must be disclosed just like the source code. So, any distribution that wanted to get a key and keep it private so that their systems will boot on locked-down hardware will not be able to do so if it uses GRUB 2.

Platform vendors are likely to use a key from UEFI as the PK, and distribute updated signature databases from the organization signed by that key. While any KEK that gets compromised will be added to the forbidden list, updating the firmware will presumably be optional so that existing hardware will still boot from code signed by compromised keys; newer hardware would get the updated forbidden list. But it would certainly be possible for an OS to "phone home" for the most recent signature database and refuse to run until the database in the firmware is current. That could be seen as reasonable protection (against malware signed by the compromised key) but would also be a fairly effective anti-jailbreaking feature.

It will be important for free software OSes (Linux distributions, the BSDs, etc.) and users to ensure that hardware vendors are aware of their needs here. Microsoft and Apple have no interest in enabling anything other than their own code to boot and run on tomorrow's hardware, and it wouldn't bother them at all to see free software get left out in the cold. In the consumer device space, it is certain that some vendors will take the opportunity to lock down their devices using UEFI, but in the server space, Linux has such a presence that one would guess some kind of solution (perhaps just an "off" switch) will be found. Desktop computers, on the other hand, are dominated by Microsoft (and to a lesser extent Apple), so our leverage there may be insufficient. If we don't keep an eye on it, your next desktop may simply refuse to boot your OS of choice.

[ I would like to thank Manoj Iyer for giving me a heads-up about this issue. ]

Comments (19 posted)

Brief items

Security quotes of the week

-- The New York Times with an interesting definition of "ingenious"

-- Dark Reading

-- "Security Curmudgeon" in "A concise history of recent Sony hacks"

-- Robert X. Cringely

Comments (12 posted)

2011 Linux Security Summit schedule published

James Morris has announced that the schedule for the Linux Security Summit is now available. The summit will be held on September 8 as part of the Linux Plumbers Conference in Santa Rosa, CA. Talks on Smack, MeeGo security, and various topics around the Linux integrity subsystem are planned. In addition, two roundtable discussions will be held in the afternoon, one on kernel hardening and the other on the Linux Security Module (LSM) architecture including ideas for ways to stack LSMs.

Comments (none posted)

Jacob: Cross-domain WebGL textures disabled in Firefox 5

Over at hacks.mozilla.org, Benoit Jacob explains why cross-domain textures have been disabled for WebGL in Firefox 5. "When a cross-domain image was used as a WebGL texture, the WebGL canvas was "tainted" so that reading from it was no longer possible. Theoretically, that eliminated the concern. But a while ago, a researcher wrote to the public WebGL list with a possible attack that could still allow reading pixels from WebGL textures. The idea was to paint the texture one pixel at a time with a WebGL fragment shader crafted to take an amount of time proportional to its brightness, and then time how long it takes: that would conceivably allow to get an approximation of the original image. Initially this attack seemed difficult to execute practically, but since then further research, including a proof-of-concept has been published which shows the attack to be more realistic than initially expected." LWN looked at WebGL vulnerabilities a few weeks back.

Comments (15 posted)

New vulnerabilities

ConsoleKit: privilege escalation

Package(s):	ConsoleKit	CVE #(s):	CVE-2010-4664
Created:	June 15, 2011	Updated:	June 15, 2011
Description:	ConsoleKit treats users logged in via ssh as if they were local, possibly giving them extra privileges.
Alerts:	openSUSE openSUSE-SU-2011:0639-1 2011-06-15

Comments (none posted)

fex: authentication bypass

Package(s):	fex	CVE #(s):	CVE-2011-1409
Created:	June 13, 2011	Updated:	June 18, 2011
Description:	From the Debian advisory: It was discovered that fex, a web service for transferring very large, files, is not properly validating authentication IDs. While the service properly validates existing authentication IDs, an attacker who is not specifying any authentication ID at all, can bypass the authentication procedure.
Alerts:	Debian DSA-2259-1 2011-06-12

Comments (1 posted)

httpcomponents-client: credentials disclosure

Package(s):	httpcomponents-client	CVE #(s):	CVE-2011-1498
Created:	June 15, 2011	Updated:	June 16, 2011
Description:	According to the 4.1.1 release notes, HttpClient suffers from a vulnerability whereby it can send proxy authorization headers to sites other than the proxy.
Alerts:	Fedora FEDORA-2011-7747 2011-06-02

Comments (2 posted)

jabberd: denial of service

Package(s):	jabberd	CVE #(s):	CVE-2011-1755
Created:	June 10, 2011	Updated:	June 15, 2011
Description:	From the Red Hat advisory: jabberd2, when expat is used, does not properly detect recursion during entity expansion, which allows context-dependent attackers to cause a denial of service (memory and CPU consumption) via a crafted XML document containing a large number of nested entity references, aka the "billion laughs attack."
Alerts:	Fedora FEDORA-2011-7805 2011-06-03 Fedora FEDORA-2011-7818 2011-06-03 Fedora FEDORA-2011-7801 2011-06-03

Comments (none posted)

java: multiple vulnerabilities

Package(s):	java	CVE #(s):	CVE-2011-0786 CVE-2011-0788 CVE-2011-0815 CVE-2011-0817 CVE-2011-0866 CVE-2011-0872
Created:	June 14, 2011	Updated:	August 30, 2011
Description:	Multiple vulnerabilities have been fixed in Oracle Java update 26. See the Oracle advisory for details.
Alerts:	Gentoo 201111-02 2011-11-05 SUSE SUSE-SU-2011:0966-1 2011-08-30 SUSE SUSE-SA:2011:036 2011-08-29 SUSE SUSE-SU-2011:0863-2 2011-08-05 SUSE SUSE-SU-2011:0863-1 2011-08-02 SUSE SUSE-SU-2011:0807-1 2011-07-19 SUSE SUSE-SA:2011:030 2011-07-18 Pardus 2011-95 2011-07-11 openSUSE openSUSE-SU-2011:0706-1 2011-06-28 Ubuntu USN-1154-1 2011-06-17 Fedora FEDORA-2011-8028 2011-06-08 SUSE SUSE-SA:2011:032 2011-08-04 Fedora FEDORA-2011-8020 2011-06-08 SUSE SUSE-SU-2011:0632-1 2011-06-14 SUSE openSUSE-SU-2011:0633-1 2011-06-14

Comments (none posted)

java-1.6.0-openjdk: mysterious vulnerabilities

Package(s):	java-1.6.0-openjdk	CVE #(s):	CVE-2011-0822 CVE-2011-0870
Created:	June 15, 2011	Updated:	June 28, 2011
Description:	The java-1.6.0-openjdk packages suffers from vulnerabilities described as "integer overflows in 2D code" (CVE-2011-0822) and "vulnerability in SAAJ" (CVE-2011-0870).
Alerts:	openSUSE openSUSE-SU-2011:0706-1 2011-06-28 Ubuntu USN-1154-1 2011-06-17 Fedora FEDORA-2011-8028 2011-06-08 Fedora FEDORA-2011-8020 2011-06-08

Comments (2 posted)

mutt: man-in-the-middle attack

Package(s):	mutt	CVE #(s):	CVE-2011-1429
Created:	June 13, 2011	Updated:	April 2, 2012
Description:	From the CVE entry: Mutt does not verify that the smtps server hostname matches the domain name of the subject of an X.509 certificate, which allows man-in-the-middle attackers to spoof an SSL SMTP server via an arbitrary certificate, a different vulnerability than CVE-2009-3766.
Alerts:	Ubuntu USN-1221-1 2011-09-29 Red Hat RHSA-2011:0959-01 2011-07-19 Scientific Linux SL-mutt-20110719 2011-07-19 Fedora FEDORA-2011-7756 2011-06-02 Fedora FEDORA-2011-7751 2011-06-02 Fedora FEDORA-2011-7739 2011-06-02 Mandriva MDVSA-2012:048 2012-04-02

Comments (none posted)

networkmanager: password information disclosure

Package(s):	NetworkManager	CVE #(s):	CVE-2011-1943
Created:	June 10, 2011	Updated:	June 15, 2011
Description:	From the Fedora advisory: NetworkManager-0.8.999-3.git20110526 inadvertently included a piece of debugging code that may have logged some VPN passwords to syslog. That version was available as an update for five (5) days before the fixed version was available. Users are advised to inspect log files in /var/log (and any backups) for VPN passwords and remove any that are found. The string "destroy_one_secret" may be used to identify files to be cleaned, such as by using the following command: 'grep -riI "destroy_one_secret" /var/log'.
Alerts:	Fedora FEDORA-2011-7919 2011-06-07

Comments (none posted)

open-vm-tools: multiple vulnerabilities

Package(s):	open-vm-tools	CVE #(s):	CVE-2011-1681 CVE-2011-1787 CVE-2011-2145 CVE-2011-2146
Created:	June 9, 2011	Updated:	June 15, 2011
Description:	From the Novell bugzilla [1, 2]: CVE-2011-2146: The mount.vmhgfs utility makes a call to stat() to check for the existence and type (file, directory, etc.) of the user-supplied mountpoint, and provides an error message if the provided argument does not exist or is not a directory. Because mount.vmhgfs is setuid-root, a local attacker can leverage this behavior to identify if a given path exists in the guest operating system and whether it is a file or directory, potentially violating directory permissions. CVE-2011-1787: The mount.vmhgfs utility checks that the user-provided mountpoint is owned by the user attempting to mount an HGFS share prior to performing the mount. However, a race condition exists between the time this checking is performed and when the mount is performed. Successful exploitation allows a local attacker to mount HGFS shares over arbitrary, potentially root-owned directories, subsequently allowing privilege escalation within the guest. CVE-2011-2145: The vmware-user-suid-wrapper utility attempts to create a directory at /tmp/VMwareDnD. Next, it makes calls to chown() and chmod() to make this directory root-owned and world-writable. By placing a symbolic link at the location of this directory, vmware-user-suid-wrapper will cause the symbolic link target to become world-writable, allowing local attackers to escalate privileges within the guest. Only FreeBSD and Solaris versions of VMware Tools are affected. CVE-2011-1681: vmware-hgfsmounter in VMware Open Virtual Machine Tools (aka open-vm-tools) 8.4.2-261024 and earlier attempts to append to the /etc/mtab file without first checking whether resource limits would interfere, which allows local users to trigger corruption of this file via a process with a small RLIMIT_FSIZE value, a related issue to CVE-2011-1089.
Alerts:	openSUSE openSUSE-SU-2011:0617-1 2011-06-09

Comments (none posted)

redmine: multiple vulnerabilities

Package(s):	redmine	CVE #(s):
Created:	June 15, 2011	Updated:	June 15, 2011
Description:	According to the Debian update, the redmine package contains vulnerabilities which can enable logged-in users to access private data, facilitate cross-site scripting attacks, and remotely execute code via the Bazaar repository adapter.
Alerts:	Debian DSA-2261-1 2011-06-15

Comments (none posted)

sudo: multiple vulnerabilities

Package(s):	sudo	CVE #(s):
Created:	June 10, 2011	Updated:	June 15, 2011
Description:	From the sudo changelog: A bug has been fixed that would allow a command to be run without the user entering a password when sudo's -g flag is used without the -u flag. If user has no supplementary groups, sudo will now fall back on checking the group file explicitly, which restores historic sudo behavior. A crash has been fixed when sudo's -g flag is used without the -u flag and the sudoers file contains an entry with no runas user or group listed. A crash has been fixed when the Solaris project support is enabled and sudo's -g flag is used without the -u flag. Sudo no longer exits with an error when support for auditing is compiled in but auditing is not enabled. Fixed a bug introduced in sudo 1.7.3 where the ticket file was not being honored when the "targetpw" sudoers Defaults option was enabled. The LOG_INPUT and LOG_OUTPUT tags in sudoers are now parsed correctly. A crash has been fixed in "sudo -l" when sudo is built with auditing support and the user is not allowed to run any commands on the host.
Alerts:	Fedora FEDORA-2011-7867 2011-06-04

Comments (none posted)

vlc: arbitrary code execution

Package(s):	vlc	CVE #(s):	CVE-2011-2194
Created:	June 10, 2011	Updated:	July 14, 2011
Description:	From the Debian advisory: Rocco Calvi discovered that the XSPF playlist parser of vlc, a multimedia player and streamer, is prone to an integer overflow resulting in a heap-based buffer overflow. This might allow an attacker to execute arbitrary code by tricking a victim into opening a specially crafted file.
Alerts:	Pardus 2011-99 2011-07-14 Debian DSA-2257-1 2011-06-10

Comments (none posted)

webmin: cross-site scripting

Package(s):	webmin	CVE #(s):	CVE-2011-1937
Created:	June 13, 2011	Updated:	June 15, 2011
Description:	From the Mandriva advisory: Cross-site scripting (XSS) vulnerability in Webmin 1.540 and earlier allows local users to inject arbitrary web script or HTML via a chfn command that changes the real (aka Full Name) field, related to useradmin/index.cgi and useradmin/user-lib.pl
Alerts:	Mandriva MDVSA-2011:109 2011-06-13

Comments (none posted)

wireshark: multiple vulnerabilities

Package(s):	wireshark	CVE #(s):	CVE-2011-2175 CVE-2011-2174 CVE-2011-1959 CVE-2011-1957 CVE-2011-1958
Created:	June 9, 2011	Updated:	January 14, 2013
Description:	From the Fedora advisory: Bug #710109 - CVE-2011-2175 wireshark: Heap-based buffer over-read in Visual Networks dissector Bug #710097 - CVE-2011-2174 wireshark: Double-free flaw by uncompressing of a zlib compressed packet Bug #710039 - CVE-2011-1959 wireshark: Stack-based buffer over-read from tvbuff buffer when reading snoop capture files Bug #710021 - CVE-2011-1957 wireshark: Infinite loop in the DICOM dissector Bug #710184 - CVE-2011-1958 wireshark (64bit): NULL pointer dereference by processing of a corrupted Diameter dictionary file
Alerts:	openSUSE openSUSE-SU-2011:1142-1 2011-10-18 Gentoo 201110-02 2011-10-09 Debian DSA-2274-1 2011-07-07 Fedora FEDORA-2011-7846 2011-06-03 Fedora FEDORA-2011-7858 2011-06-03 Fedora FEDORA-2011-7821 2011-06-03 Red Hat RHSA-2012:0509-01 2012-04-23 Scientific Linux SL-wire-20120423 2012-04-23 Oracle ELSA-2012-0509 2012-04-23 CentOS CESA-2012:0509 2012-04-24 Oracle ELSA-2013-0125 2013-01-12 Scientific Linux SL-wire-20130116 2013-01-16

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 3.0-rc3, released on June 13. Linus says: "There's a lot of small one-liners, but a few bigger chunks too: Radeon DRI updates, some btrfs updates, and fixing Sparc LEON support (and supporting PCI). Smaller updates to nilfs2 and ceph, and s390 and arm. Other than that, it's mostly random driver updates all over." The short-form changelog is in the announcement, or see the full changelog for all the details.

No stable updates have been released in the last week. The 2.6.32.42, 2.6.33.15, and 2.6.39.2 updates are in the review process as of this writing; these updates can be expected on or after June 18.

Comments (2 posted)

Quotes of the week

-- "pageexec"

-- Linus Torvalds prepares for 3.0.0

-- Greg Kroah-Hartman delays the pain into the eastern hemisphere

Comments (7 posted)

Native Linux KVM tool v2

The second iteration of the Native Linux KVM tool (a QEMU replacement for KVM) has been posted. The tool now has graphics support, SMP support, networking, file I/O said to be faster than QEMU, and more. The developers are now "officially aiming" to get the tool merged in the 3.1 kernel development cycle.

Full Story (comments: 27)

Kernel development news

A reworked contiguous memory allocator

The problem of allocating large chunks of physically-contiguous memory has often been discussed in these pages. Virtual memory, by its nature, tends to scatter pages throughout the system; the kernel does not have to be running for long before free pages which happen to be next to each other become scarce. For many years, the way kernel developers have dealt with this problem has been to avoid dependencies on large contiguous allocations whenever possible. Kernel code which tries to allocate more than two physically-contiguous pages is rare.

Recently the need for large contiguous allocations has been growing. One source of demand is huge pages, and the transparent huge pages feature in particular. Another is an old story with a new twist: hardware which is unable to perform scatter/gather DMA. Any device which can only do DMA to a physically contiguous area requires (in the absence of an I/O memory management unit) a physically contiguous buffer to work with. This requirement is often a sign of relatively low-end (stupid) hardware; one could hope that such hardware would become scarce over time. What we are seeing, though, are devices which have managed to gain capabilities while retaining the contiguous DMA requirement. For example, there are video capture engines which can grab full high-definition data, perform a number of transformations on it, but still need a contiguous buffer for the result. The advent of high definition video has aggravated the problem - those physically-contiguous buffers are now quite a bit bigger and harder to allocate than they were before.

Almost one year ago, LWN looked at the contiguous memory allocator (CMA) patches which were meant to be an answer to this problem. This patch set followed the venerable tradition of reserving a chunk of memory at boot time for the sole purpose of satisfying large allocation requests. Over the years, this technique has been used by the "bigphysarea" patch, or simply by booting the kernel with a mem= parameter that left a range of physical memory unused. The out-of-tree Android pmem driver also allocates memory chunks from a reserved range. This approach certainly works; nearly 20 years of experience verifies that. The down side is that the reserved memory is not available for any other use; if the device is not using the memory, it simply sits idle. That kind of waste tends to be unpopular with kernel developers - and users.

For that reason and more, the CMA patches were never merged. The problem has not gone away, though, and neither have the developers who are working on it. The latest version of the CMA patch set looks quite a bit different; while some issues still need to be resolved, this patch set looks like it may have a much better chance of getting into the mainline.

The CMA allocator can still work with a reserved region of memory, but that is clearly not the intended mode of operation. Instead, the new CMA tries to maintain regions of memory where contiguous chunks can be created when the need arises. To that end, CMA relies on the "migration type" mechanism built deeply into the memory management code.

Within each zone, blocks of pages are marked as being for use by pages which are (or are not) movable or reclaimable. Movable pages are, primarily, page cache or anonymous memory pages; they are accessed via page tables and the page cache radix tree. The contents of such pages can be moved somewhere else as long as the tables and tree are updated accordingly. Reclaimable pages, instead, might possibly be given back to the kernel on demand; they hold data structures like the inode cache. Unmovable pages are usually those for which the kernel has direct pointers; memory obtained from kmalloc() cannot normally be moved without breaking things, for example.

The memory management subsystem tries to keep movable pages together. If the goal is to free a larger chunk by moving pages out of the way, it only takes a single nailed-down page to ruin the whole effort. By grouping movable pages, the kernel hopes to be able to free larger ranges on demand without running into such snags. The memory compaction code relies on these ranges of movable pages to be able to do its job.

CMA extends this mechanism by adding a new "CMA" migration type; it works much like the "movable" type, but with a couple of differences. The "CMA" type is sticky; pages which are marked as being for CMA should never have their migration type changed by the kernel. The memory allocator will never allocate unmovable pages from a CMA area, and, for any other use, it only allocates CMA pages when alternatives are not available. So, with luck, the areas of memory which are marked for use by CMA should contain only movable pages, and it should have a relatively high number of free pages.

In other words, memory which is marked for use by CMA remains available to the rest of the system with the one restriction that it can only contain movable pages. When a driver comes along with a need for a contiguous range of memory, the CMA allocator can go to one of its special ranges and try to shove enough pages out of the way to create a contiguous buffer of the appropriate size. If the pages contained in that area are truly movable (the real world can get in the way sometimes), it should be possible to give that driver the buffer it needs. When that buffer is not needed, though, the memory can be used for other purposes.

One might wonder why this mechanism is needed, given that memory compaction can already create large physically-contiguous chunks for transparent hugepages. That code works: your editor's system, as of this writing, has about 25% of its memory allocated as huge pages. The answer is that DMA buffers present some different requirements than huge pages. They may be larger, for example; transparent huge pages are 2MB on most architectures, while DMA buffers can be 10MB or more. There may be a need to place DMA buffers in specific ranges of memory if the underlying hardware is weird enough - and CMA developer Marek Szyprowski seems to have some weird hardware indeed. Finally, a 2MB huge page must also have 2MB alignment, while the alignment requirements for DMA buffers are normally much more relaxed. The CMA allocator can grab just the required amount of memory (without rounding the size up to the next power of two as is done in the buddy allocator) without worrying about overly stringent alignment demands.

The CMA patch set provides a set of functions for setting up regions of memory and creating "contexts" for specific ranges. Then there are simple cm_alloc() and cm_free() functions for obtaining and releasing buffers. It is expected, though, that device drivers will never invoke CMA directly; instead, awareness of CMA will be built into the DMA support functions. When a driver calls a function like dma_alloc_coherent(), CMA will be used automatically to satisfy the request. In most situations, it should "just work."

One of the remaining concerns about CMA has to do with how the special regions are set up in the first place. The current scheme expects that some special calls will be made in the system's board file; it is a very ARM-like approach. The intent is to get rid of board files, so something else will have to be found. Moving this information to the device tree is not really an option either, as Arnd Bergmann pointed out; it is really a policy decision. Arnd is pushing for some sort of reasonable default setup that works on most systems; quirks for systems with special challenges can be added later.

The end result is that there's likely to be at least one more iteration of this patch set before it gets into the mainline. But CMA addresses a real need which has been met, thus far, with out-of-tree hacks of varying kludginess. This code has the potential to make physically-contiguous allocations far more reliable while minimizing the impact on the rest of the system. It seems worth having.

Comments (none posted)

Debating overlayfs

Union filesystems allow multiple filesystems to be combined and presented to the user as a single tree. In typical use, a writable filesystem is overlaid on top of a read-only base, creating the illusion that all files on the filesystem can be changed. This mode of operation is useful for live CD distributions, embedded systems where a quick "factory reset" capability is desired, virtualized systems built on a common base filesystem, and more. Despite the value of this feature, Linux has never had an in-kernel union filesystem option, despite several attempts to create one. A recent attempt to change that situation may or may not succeed.

LWN looked at the overlayfs filesystem last year. Overlayfs, written by Miklos Szeredi, is distinguished by its relative simplicity. Recently, Miklos asked if overlayfs could be merged for the 3.1 development cycle. He may get his wish, but some worries will have to be addressed first.

Andrew Morton has raised a couple of concerns; one of which is that the problem might be better solved in user space. He dismissed the simplicity of overlayfs, saying "Not merging it would be even smaller and simpler," and suggested that performance problems should be addressed by making the user-space implementation faster. Linus has pretty much ended that aspect of the debate by saying "People who think that userspace filesystems are realistic for anything but toys are just misguided." So the way seems to be clear for a union filesystem implementation in the kernel.

Andrew's other concern is that overlayfs may not be a sufficiently complete solution:

That objection is harder to answer. It has been pointed out that OpenWRT is happily using overlayfs and Ubuntu is considering it. About the only viable alternative project is union mounts, which has not seen much developer attention recently. On the feature front, it doesn't seem like anything else will come along and outshine overlayfs in the near future.

On the technical side, union filesystems have always presented some unique challenges. Valerie Aurora, who has done a fair amount of work in this area, looked at overlayfs in March and seemed to be positive about it:

She has changed her tune a bit in the current discussion, suggesting that there are some difficulties which need to be addressed:

She raised some locking concerns as well, which Miklos addressed in detail; the concern about changing the underlying filesystem has not been answered, though. So it's possible that technical correctness issues may yet delay the merging of overlayfs into the kernel. That said, it seems clear that there is demand for this feature, and that overlayfs appears to satisfy that demand nicely. There will likely come a time when keeping it out of the kernel becomes too hard to justify.

Comments (10 posted)

The videobuf2 API

Video4Linux2 drivers are charged with the task of acquiring video data from a sensor (via some sort of DMA controller, usually) and transferring those video frames to user space. The amount of data being moved makes performance a consideration; to that end, V4L2 has defined a somewhat complex API to handle streaming data. Implementing this API adds a certain amount of complexity to V4L2 drivers, but much of that complexity is the same from one driver to the next. To make life easier for driver writers (and their users), the "videobuf" subsystem was created to handle many of the details of streaming I/O buffer management. LWN documented videobuf toward the end of 2009; a version of that article also found its way into the kernel's documentation directory.

This is the Linux kernel we're talking about, though, so 2009 is ancient history. The videobuf interface has since been superseded by videobuf2, which, while clearly inspired by the original, has a different way of doing things. So this article will be an attempt to get caught up with the current state of the art - an effort which will certainly inspire the creation of videobuf3 in the near future.

Why videobuf2? The original videobuf worked well, but it turned out to be inflexible in a number of ways. The API varied considerably depending on which type of buffer was in use, and there was no real way for drivers to add their own specific memory management needs. Videobuf2 has created a more consistent API which allows for more customization in the drivers themselves.

Buffers

Like the original videobuf, videobuf2 implements three basic types of buffers. Vmalloc buffers are allocated with vmalloc(), and are thus virtually contiguous in kernel space; drivers working with these buffers almost invariably end up copying the data once as it passes through the system. Contiguous DMA buffers are physically contiguous in memory, usually because the hardware cannot perform DMA to any other type of buffer. S/G DMA buffers are scattered through memory; they are the way to go if the hardware can do scatter/gather DMA.

Depending on the type of buffer being used, the driver will need to include one of the following header files:

    #include <media/videobuf2-vmalloc.h>
    #include <media/videobuf2-dma-contig.h>
    #include <media/videobuf2-dma-sg.h>

One nice difference with videobuf2 is that a driver can be written to support more than one mode if need be. The above include files do not conflict with each other, and the videobuf2 interface is nearly the same for all three modes.

A buffer for a video frame is represented by struct vb2_buffer, defined in <media/videobuf2-core.h>. Usually drivers will want to track per-buffer information of their own, so, in the usual style, the driver will define its own buffer type that includes a vb2_buffer instance. However, the videobuf2 authors didn't read Neil Brown's Object-oriented design patterns in the kernel, so they don't have the driver allocate the resulting structures (in all fairness, said developers may offer lame excuses to the effect that said article had not been written at the time). That means that (1) the driver has to tell videobuf2 what the real size of the structure is, and (2) the vb2_buffer instance must be the first field in the driver's structure. Your editor may just post a patch to fix that in the near future.

A videobuf2 driver must create a set of callbacks to give to the videobuf2 subsystem, five of which are specific to the management of buffers; they function in a similar (but not identical) manner to the videobuf versions. These callbacks are:

    int (*buf_init)(struct vb2_buffer *vb);
    int (*buf_prepare)(struct vb2_buffer *vb);
    void (*buf_queue)(struct vb2_buffer *vb);
    int (*buf_finish)(struct vb2_buffer *vb);
    void (*buf_cleanup)(struct vb2_buffer *vb);

Videobuf2 will call buf_init() for each new buffer after it has been allocated; the driver can then perform any additional initialization that needs to be done. Returning a failure code will abort the setup of the buffer queue.

The buf_prepare() callback is invoked when user space queues the buffer (i.e. in response to a VIDIOC_QBUF operation); it should do any preparation which might be required before the buffer is used for I/O. buf_queue(), instead, is called to pass actual ownership of the buffer to the driver, indicating that it's time to actually start I/O on it.

buf_finish() will be called just before the buffer is passed back to user space. One might question the need for this callback; the driver already knows when a buffer has a new frame for user space and, indeed, must tell videobuf2 about it. One possible answer is that the completion of I/O to the buffer is often handled in interrupt context, while buf_finish() will be called in process context.

Finally, buf_cleanup() is called just before a buffer is freed so that the driver can do any additional cleanup work required.

Other videobuf2 callbacks

In the original videobuf, the only callbacks provided by the driver had to do with buffer management. With videobuf2, there are a few others, starting with:

    int (*queue_setup)(struct vb2_queue *q, unsigned int *num_buffers,
		       unsigned int *num_planes, unsigned long sizes[],
		       void *alloc_ctxs[]);

This function, called in response to a VIDIOC_REQBUFS ioctl() operation, allows the driver to influence how the buffer queue is set up. The vb2_queue structure describes the queue as a whole; we'll see more about it shortly. The num_buffers argument is the number of buffers requested by the application; the driver can modify it if needed. num_planes contains the number of distinct video planes needed to hold a frame; for packed formats, it will be one, but it will be larger if planar formats are in use (see this article for more information on formats). The sizes array should contain the size (in bytes) of each plane.

The alloc_ctxs field contains the "allocation context" for each plane; it is currently only used by the contiguous DMA mode. Drivers which use contiguous DMA should call:

    void *vb2_dma_contig_init_ctx(struct device *dev)

to get that context; it needs to be remembered and passed to vb2_dma_contig_cleanup_ctx() when the driver shuts down.

There are two callbacks which tell the driver when to start and stop acquiring video data:

    int (*start_streaming)(struct vb2_queue *q);
    int (*stop_streaming)(struct vb2_queue *q);

Videobuf2 will call start_streaming() whenever user space wants to start grabbing data. That may happen in response to a VIDIOC_STREAMON ioctl(), but the videobuf2 implementation of the read() system call can also use it. A call to stop_streaming() will be made when user space no longer wants data; this callback should not return until DMA has been stopped. It's worth noting that, after the stop_streaming() call, videobuf2 will grab back all buffers passed to the driver; the driver should forget any references it may have to those buffers.

Locking

The final two callbacks deserve a section of their own. The locking model used for videobuf2 is not documented all that well; from what your editor has been able to gather, the rules are mostly like the following. The driver needs a lock (probably a mutex) which governs access to the device as a whole. Then:

• Calls that the driver makes directly into videobuf2 should be made with the device lock held.

• Callbacks to the driver from videobuf2 should assume that the lock has already been taken. For example, a start_streaming() call will result from a user-space request to acquire data which will have necessarily passed through the driver before videobuf2 gets involved, so, by the time start_streaming() is called, the device lock will be held.

With that context, one needs to consider one little problem: a user-space invocation of VIDIOC_DQBUF, meant to get a buffer full of data from the kernel, may block waiting for a buffer to become available. That, in turn, may not happen until user space (perhaps in a different thread) hands a buffer back to the kernel with VIDIOC_QBUF. If the first call blocks with the lock held, the application will end up waiting for a very long time. For this situation, videobuf2 provides two more callbacks:

    void (*wait_prepare)(struct vb2_queue *q);
    void (*wait_finish)(struct vb2_queue *q);

Before a VIDIOC_DQBUF operation blocks to wait for a buffer, it will call wait_prepare() to release the device lock; once it stops waiting, a call to wait_finish() will reacquire the lock. It might have been better to call them release_lock() and reacquire_lock(), but so it goes.

Queue setup

With all of the above in place, the driver can introduce itself to videobuf2. The first step is to fill in a vb2_ops structure with all of the callbacks described above:

    static const struct vb2_ops my_special_callbacks = {
		.queue_setup = my_special_queue_setup,
		/* ... */
    };

Then, to set up a videobuf2 queue (normally done either at device registration or device open time), the driver should allocate a vb2_queue structure:

    struct vb2_queue {
	enum v4l2_buf_type		type;
	unsigned int			io_modes;
	unsigned int			io_flags;
	const struct vb2_ops		*ops;
	const struct vb2_mem_ops	*mem_ops;
	void				*drv_priv;
	unsigned int			buf_struct_size;
	/* Lots of private stuff omitted */
    };

The structure should be zeroed, and the above fields filled in. type is the buffer type, usually V4L2_BUF_TYPE_VIDEO_CAPTURE. io_modes is a bitmask describing what types of buffers can be handled:

• VB2_MMAP: buffers allocated within the kernel and accessed via mmap(); vmalloc and contiguous DMA buffers will usually be of this type.

• VB2_USERPTR: buffers allocated in user space. Normally, only devices which can do scatter/gather I/O can deal with user-space buffers. Interestingly, videobuf2 supports contiguous buffers allocated by user space; the only way to get those, though, is to use some sort of special mechanism like the out-of-tree Android "pmem" driver. Contiguous I/O to huge pages is not supported.

• VB2_READ, VB2_WRITE: user-space buffers provided via the read() and write() system calls.

The mem_ops field is where the driver tells videobuf2 what kind of buffers it is actually using; it should be set to one of vb2_vmalloc_memops, vb2_dma_contig_memops, or vb2_dma_sg_memops. If a situation arises where none of the existing modes works for a specific device, the driver author can create a custom set of vb2_mem_ops to meet that need; as of this writing, there are no drivers in the mainline kernel that have supplied their own memory operations.

Finally, drv_priv is a place where the driver can stash a pointer of its own (usually to its device structure), and buf_struct_size is where the driver tells videobuf2 how big its buffer structure is. Once the structure has been filled in, it can be passed to:

    int vb2_queue_init(struct vb2_queue *q);

A call to vb2_queue_release() should be made when the device is shut down.

Device operations

Now most of the infrastructure for videobuf2 is in place; what's left is (1) making the connection between user space operations and their implementation in videobuf2, and (2) actually performing I/O. For the first step, the driver needs to create V4L2 callbacks for the various I/O-oriented ioctl() calls in the usual way. Most of these callbacks, though, can simply acquire the device lock and call directly into videobuf2. There is a whole set of functions to be used in this role:

    int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b);
    int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req);
    int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b);
    int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, 
		  bool nonblocking);
    int vb2_streamon(struct vb2_queue *q, enum v4l2_buf_type type);
    int vb2_streamoff(struct vb2_queue *q, enum v4l2_buf_type type);

A similar thing needs to be done with a number of entries in the driver's file_operations structure. To that end, videobuf2 provides:

    int vb2_mmap(struct vb2_queue *q, struct vm_area_struct *vma);
    unsigned int vb2_poll(struct vb2_queue *q, struct file *file, 
			  poll_table *wait);
    size_t vb2_read(struct vb2_queue *q, char __user *data, size_t count,
		    loff_t *ppos, int nonblock);
    size_t vb2_write(struct vb2_queue *q, char __user *data, size_t count,
		     loff_t *ppos, int nonblock);

This, of course, is where the payoff happens: all the grungy details of buffer management, implementing mmap(), and more are handled in videobuf2 with no further mess. So the driver code is significantly shorter, the core code is known to be well debugged, and devices behave more consistently toward user space.

There's only one little bit of work left to do: actually getting the data into the buffers. For vmalloc buffers, that task is usually accomplished with something like memcpy(); one useful helper function in this case is:

    void *vb2_plane_vaddr(struct vb2_buffer *vb, unsigned int plane_no);

which returns the kernel-space virtual address for the given plane in the buffer.

Contiguous DMA drivers will need to get the bus address to hand to the device for I/O; that address can be had with:

    dma_addr_t vb2_dma_contig_plane_paddr(struct vb2_buffer *vb, 
                                          unsigned int plane_no);

For scatter/gather drivers, the interface is just a bit more complex:

    struct vb2_dma_sg_desc {
	unsigned long		size;
	unsigned int		num_pages;
	struct scatterlist	*sglist;
    };

    struct vb2_dma_sg_desc *vb2_dma_sg_plane_desc(struct vb2_buffer *vb, 
						  unsigned int plane_no);

In this case, the driver can obtain the scatterlist from videobuf2 which can then be used to program the device for I/O.

For all three cases, the buffer will eventually be filled with frame data which needs to be passed back to user space. The vb2_buffer structure contains v4l2_buffer structure (called v4l2_buf) which should be filled in with the usual information: image size, sequence number, time stamp, etc. Then the buffer should be passed to:

    void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state);

The state parameter should be passed as VB2_BUF_STATE_DONE for a normal completion, or VB2_BUF_STATE_ERROR if something went wrong. Happily videobuf2, unlike its predecessor, does not get upset if buffers are completed in an arbitrary order.

And that is a fairly complete summary of the state of the art with regard to the videobuf2 API. Those wanting to see the complete interface can find it in the include files mentioned above. As always, the "virtual video" driver (in drivers/media/video/vivi.c) serves as a sort of showcase for almost everything Video4Linux2 can do; it uses videobuf with vmalloc-style buffers. As of this writing, there is no videobuf3 in sight, so hopefully the information found here will remain useful for a while.

Comments (8 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.0-rc3 . (June 14, 2011)

Core kernel code

• Tejun Heo: ptrace: implement PTRACE_SEIZE/INTERRUPT and group stop notification, take#5 . (June 14, 2011)

Device drivers

• Carlos Chinea: HSI framework and drivers . (June 13, 2011)

• Lars-Peter Clausen: ASoC: Add ADAU1701 codec driver . (June 13, 2011)

• Rafael J. Wysocki: [PATCH 0/8] PM / Domains: Support for generic I/O PM domains (v5) . (June 13, 2011)

• Eric Miao: input: add support for mma8450 accelerometer . (June 13, 2011)

• Marius B. Kotsbak: net/usb: Add Samsung Kalmia driver for Samsung GT-B3730 . (June 14, 2011)

• Jonathan Corbet: Refactor cafe_ccic and add Armada 610 driver [V2] . (June 14, 2011)

• Linus Walleij: drivers: create a pinmux subsystem v3 . (June 14, 2011)

• Kamil Debski: [PATCH 0/4 v9] Multi Format Codec 5.1 driver for s5pv210 and exynos4 SoC . (June 14, 2011)

• Nao Nishijima: [RFC] Persistent device name using preferred name . (June 15, 2011)

Documentation

• Grant Likely: (early draft) dt: Linux dt usage model documentation . (June 13, 2011)

• Andy Lutomirski: Document the vDSO and add a reference parser . (June 13, 2011)

• Randy Dunlap: Documentation: update printk-formats.txt . (June 15, 2011)

Filesystems and block I/O

• Hugh Dickins: tmpfs: simplify by splice instead of readpage . (June 13, 2011)

• Hugh Dickins: tmpfs: convert from old swap vector to radix tree . (June 14, 2011)

• Wu Fengguang: bdi write bandwidth estimation . (June 13, 2011)

• Arne Jansen: btrfs: generic readeahead interface . (June 13, 2011)

Memory management

• Mel Gorman: Swap-over-NBD without deadlocking v4r3 . (June 9, 2011)

• KAMEZAWA Hiroyuki: [RFC][PATCH] A scan node selection logic for memcg rather than round-robin. . (June 9, 2011)

• Marek Szyprowski: Contiguous Memory Allocator . (June 13, 2011)

Architecture-specific

• Daniel Drake: OLPC Power Management . (June 13, 2011)

• Luck, Tony: reworked machine check recovery patches . (June 13, 2011)

• Colin Cross: CPU PM notifiers . (June 13, 2011)

• green@linuxhacker.ru: Initial B&N Nook Color support, take six . (June 14, 2011)

Virtualization and containers

• Timur Tabi: [v4] drivers/virt: introduce Freescale hypervisor management driver . (June 9, 2011)

• Glauber Costa: KVM steal time implementation . (June 14, 2011)

Benchmarks and bugs

• Rafael J. Wysocki: 3.0-rc2: Reported regressions from 2.6.39 . (June 13, 2011)

• Rafael J. Wysocki: 3.0-rc2: Reported regressions 2.6.38 -> 2.6.39 . (June 13, 2011)

Miscellaneous

• Mathieu Desnoyers: Userspace RCU 0.6.0 . (June 13, 2011)

Page editor: Jonathan Corbet

Distributions

Fedora, systemd, and changes

Significant changes in distributions brought by new releases often make some segment of the users unhappy—sometimes very vocally so. One need only look at the GNOME 3 arguments that have raged here and elsewhere to see one example. But another change that came along with Fedora 15 is getting a similar reaction: systemd. This replacement for the init process has many technical advantages, along with bugs to still be worked out of course, but its inclusion into Fedora 15 has been met with some rather loud complaints. There is merit to some of them certainly, but many seem to fall into the category of "how dare you touch my init".

init is the first user-space process that gets started on a Linux system and is the only user-space process that is started by the kernel—the others are descendants of init. It is responsible for setting up the system for use by other processes: mounting filesystems, starting services, and so on. Traditionally, much of the work has been done using shell scripts, either with the venerable sysvinit or the more recent Upstart, but systemd is on a path to change much of that. Instead of invoking the shell multiple times as part of the boot process, systemd allows much of that work to be done from within the systemd executable itself, which saves a lot of overhead in terms of starting the shell and executing the same basic shell script code multiple times.

But systemd can still use the existing sysv/Upstart-style init scripts, and Fedora 15 makes use of plenty of them. Over time, the plan is to phase those out in favor of configuring those services directly using systemd's configuration files. While systemd seems to be working well for many users, it isn't surprising that a low-level component might not suit everyone's needs. Several recent threads in the fedora-devel mailing list highlight some of those problems, but more than that, they also highlight just how unhappy some people can get with changes of this sort.

Denys Vlasenko made a reasonable point—amongst some flames—when he asked about the memory usage of systemd: "Granted, systemd does a bit more that "typical" init, but I think using *eleven plus megabytes* of malloced space is a bit much." He went on to list multiple things that systemd does which, in his mind, should not be part of the init process.

Eleven megabytes of malloced space did seem somewhat excessive, so Adam Jackson dug in to try to learn more. His numbers showed 21M of malloced space, 11M of which was all in 2064-byte allocations. Since malloc uses 16 bytes for its own bookkeeping, it meant that something was doing more than 5000 allocations of 2048-byte chunks. As Michal Schmidt was quick to point out, those come from a known problem with SELinux. Jackson also noted that Vlasenko wasn't really helping his case by not investigating the problem further:

Contrary to Vlasenko's prediction that his memory complaints would be ignored by systemd developer Lennart Poettering, it would seem that work started more-or-less immediately on finding ways to improve libselinux. Dan Walsh released an updated version that sped up the loading of the file context information by a factor of four. Discussion also started on ways to reduce the memory usage of libselinux.

But there is an overarching issue here: it is long past time to complain about systemd's architecture or its inclusion into Fedora 15. As Simo Sorce put it:

New features for Fedora are proposed and discussed before a decision is made to include them. Fedora nearly shipped systemd in Fedora 14 before deciding against it in the final stages of the release process. That process was done in the open and the decision was made by the Fedora engineering steering committee (FESCo) in accordance with how Fedora is governed. Perhaps it was the wrong choice for the distribution and its users, but it's a little late to make that argument now.

Fedora users who are upset by the change have multiple choices, of course. Some other distributions are also adopting systemd, but Ubuntu seems unlikely to drop Upstart anytime soon. In addition, Debian will still maintain Upstart and shell init scripts, even as it starts supporting systemd, because it wants to support non-Linux environments and systemd is very Linux-specific. Other distributions may also stick with Upstart (or still be using sysvinit). Another option would be to maintain Upstart and the init scripts as an option in Fedora going forward.

While I have heard multiple complaints (on two continents) about systemd over the last few months, I haven't heard much in the way of strong technical arguments against it. The same is true of GNOME 3, actually. Those arguments may well exist—and even be compelling—but most of what I have heard has been an overall resistance to change. There is certainly nothing wrong with that, and change for change's sake is generally a bad thing, but the systemd and GNOME developers believe that they are making truly useful changes. There are enough other options in the free software universe that it probably makes sense for those who are unhappy to either engage with the development communities to move them in a different direction or to switch to one of the myriad other choices that exist.

Comments (73 posted)

Brief items

Distribution quotes of the week

-- Mathieu Bouffard

-- Seth Vidal

-- Stefano Zacchiroli (via an interview on Muktware)

Comments (none posted)

dyne:bolic 3.0 beta1

A beta version of dyne:III, the third series of dyne:bolic development, is available for testing. "this version of our OS is based on the last stable version of pure:dyne "carrot and coriander" ( see http://puredyne.org ) closely reviewed following the 100% free guidelines set by the linux-libre project." Pure:dyne is a community effort maintained by media artists for media artists, using free software.

Full Story (comments: none)

Fedora 16 to use Btrfs as its default filesystem

At the June 8 meeting of the Fedora engineering steering committee (FESCo), the group decided that Fedora 16 will ship with Btrfs as its default filesystem. Btrfs is a relatively new copy-on-write filesystem with many interesting features such as read-only and writeable snapshots, multiple device support for RAID, online filesystem defragmentation, and more, though it is still marked as experimental in the kernel. "AGREED: Feature is approved. Will add some base critera to the page to be met by feature freeze. This is just a swap of ext4 to btrfs for default, not change of lvm or other parts of default."

Full Story (comments: 106)

openSUSE Medical release and new leadership

The openSUSE Medical project is aimed at packaging and publishing software for medical purposes. "openSUSE Medical is now part of the Linux Medical Taskforce and has exciting plans for the future included importing diabetes care and diabetics applications, some educational applications for students and of course the usual updates, bugfixes and translations." The first beta of openSUSE Medical 11.4 is available for testing. Feedback from medical professionals, medical students, physicians and patients is welcome.

Full Story (comments: none)

Tin Hat 20110613 released.

Tin Hat is a desktop system based on Hardened Gentoo which runs purely in RAM. "[Tin Hat 20110613] is a major release with upgrades to every aspect of the system. The toolchain was updated to Gentoo's hardened gcc-4.4.5, glibc 2.12.2 and binutils-2.20.1. The kernel was bumped to vanilla 2.6.38 plus GRSEC/PaX patches. System init was updated to openrc. In all, about 400 packages were upgraded, including Gnome 2.32.1 and Namoroka (aka Firefox) 3.6.17"

Full Story (comments: none)

Distribution News

Fedora

Election Results for FESCo and Fedora Board seats

Kevin Fenzi (nirik), Bill Nottingham (notting), Tomáš Mráz (t8m), Peter Jones (pjones), and Stephen Gallagher (sgallagh) have been elected to serve on the Fedora Engineering Steering Committee (FESCo). Rex Dieter (rdieter), Jon Stanley (jds2001), and Peter Robinson (pbrobinson) have been elected to serve on the Fedora Board.

Full Story (comments: none)

Fedora 13 end of life on 2011-06-24

Fedora 13 will reach end of life on June 24, 2011. There will be no updates after that time. "Please see http://fedoraproject.org/wiki/DistributionUpgrades for more information on upgrading from Fedora 13 to a newer release."

Full Story (comments: 1)

openSUSE

The road to systemd for openSUSE 12.1

The openSUSE project aims to have systemd as the default in its next release. "As you might guess, switching boot manager is not a trivial task and issues will be found. So, we want to have as much feedback and testing as possible, to try to tackle as much (if not all) issues in time for 12.1."

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• DistroWatch Weekly, Issue 409 (June 13)
• Fedora Weekly News Issue 278 (June 8)
• openSUSE Weekly News, Issue 179 (June 11)
• Ubuntu Weekly Newsletter, Issue 220 (June 13)

Comments (none posted)

Zimmerman: DEX finishes first batch of derivative patches for Debian

On his blog, Matt Zimmerman posts a report on DEX, which is a project to get Debian and its derivatives working more closely together. Debian project leader Stefano Zacchiroli had a hand in both the project and the report. It looks like the first project, "ancient-patches", was a success: "This has left only two patches out of the original list of 277. Both of them are filed in the BTS and have been discussed with the relevant maintainer team. One of them is expected to be obsoleted when a new upstream version is packaged, which implements similar functionality. The other is being discussed with the upstream developers, but there is no conclusion yet about whether it can be merged upstream or in Debian."

Comments (3 posted)

Ubuntu 11.04 (Natty Narwhal), Reviewed In Depth (Tom's Hardware Guide)

Tom's Hardware Guide has a lengthy review of Ubuntu 11.04 and the Unity interface. "Unity incorporates some great design concepts, but they are counterbalanced by other concepts that aren't so great. The new Panel is streamlined, and an explanation of the indicator colors is a welcome addition. However, the global menu is of questionable merit and the lack of GNOME applets is disappointing. The concept of Dash is much-needed on desktops, notebooks, and tablets. But the non-customizable layout and reliance on typed searches can slow down even the most basic tasks. The design of the new Launcher is definitely a high point, while its rigid auto-hide behavior is unquestionably a bummer."

Comments (none posted)

Page editor: Rebecca Sobol

Development

KDE moves forward on Frameworks

The KDE Project recently got together for a sprint called Platform 11 in Randa, Switzerland. One of the major decisions coming out of that gathering is to start work on the next KDE Frameworks — a set of high-level components for applications to use — immediately following KDE 4.7. What does this mean for the community? This time around, KDE won't be dragging users or application developers through the "big change" phase of underlying technologies for KDE.

The KDE Software Collection (KSC), what was once known simply as the K Desktop Environment, comprises everything from the KDE Developer Platform and the KDE Workspaces (such as the desktop and netbook workspaces) to KDE's applications. In the past, this entire collection has moved forward together — or tried to, at any rate. KDE developer Aaron Seigo describes major release cycles as "that 'big change' phase" that included applications, desktop, and pretty much everything else.

That worked well enough when KDE was a smaller and less complex beast and "the overlap between 'people who work on kdelibs' and 'people who work on applications' was very high." As has been noted by many, however, that worked less well when work began on KDE 4:

At Platform 11, KDE folks started thinking that having a "welded together" platform was not the best course of action. Putting application development on hold while doing library development seems a bad idea. It also happens that some libraries offered by KDE may not in fact be of use only to KDE applications. In a discussion over Skype, Seigo noted that the KDE Project wants to allow developers to choose KDE components — such as the Solid Hardware Library — without having to pull in all of the KDE libraries.

The trigger for this decision? As Sebastian Kügler wrote during Platform 11, the impetus to begin working on KDE's Frameworks starts with Qt 5. As Kügler notes, the fact that Qt 5 has a change in its binary interface, "is a good point in time to introduce some changes to our frameworks that are only possible if we change the ABI -- which Qt 5 will do for us anyway."

The decision has been made to start working on the next generation of KDE Frameworks following KDE 4.7 but not applications or workspaces. What are Frameworks? That's what is currently referred to as kdelibs, kde-runtime, kdepimlibs, kdepim-runtime, and kdesupport — essentially the core libraries for KDE user interface elements that are not provided by Qt. These are the libraries used by major KDE applications like the KDE Addressbook, KMail, and Dolphin.

Kügler also says that this will not only make it easier for developers to use KDE Frameworks, but also lower the barrier for contributions to KDE, as well as making Frameworks more suitable for mobile devices. Seigo also emphasized the benefits for using KDE Frameworks on mobile devices such as phones, tablets, and set-top devices: "We want to make it possible that an application developer can use [just one component like Solid] a la carte with a clear set of dependencies."

What Frameworks will be

According to Kevin Ottens, the libraries will be sorted in two categories: the Framework type and into Tiers based on the level of dependencies. The Framework types will be Solutions, OS integration, and functional Qt Addons. Solutions would include a full technology like the Akonadi personal information manager (PIM) or KIO, which provides access to files stored locally or remotely. Solutions would also have libraries and mandatory runtime dependencies. OS integration Addons would either use a built-in feature for an OS or implement it directly. The example given by Ottens is libkdatetime which watches ktimezoned for time zone changes on Linux, but might use the Windows APIs directly. Finally, the functional Addons have no runtime dependencies at all. These would include new libraries like libkcore, which Ottens says "will contain the job classes, handling of command line arguments, text handling classes, file locking, and not much more."

The Frameworks will also be split into Tiers, with the first Tier depending only on Qt itself, Tier 2 depending on Tier 1 frameworks, and Tier 3 having dependencies on Tier 1, 2 and other Tier 3 Frameworks. So, for example, Ottens says that window management would be a Tier 2, OS integration Framework: "The window management features of kdeui will be split out into their own library. It will depend on libkgui and Qt, granting it the Tier 2 label. As it provides integration with the OS which in particular might require a runtime payload (although not in that particular case), it is granted the OS Integration label." Ottens also provides a graph (small version at right, click for a larger version) that visualizes the organization of existing KDE components.

The Frameworks plan also comes with new rules, or at least more explicit rules, that codify (as Ottens says) "changes to our habits which already happened a while ago during the Qt 3 to Qt 4 port." This includes following the license policy, following a consistent coding standard (preferably the kdelibs standard — but that's not required), meeting the criteria of the Tier and Framework type, and maintaining binary compatibility until the next major release of KDE Frameworks.

Next steps

The next steps for the Frameworks plan is to keep working on libraries until KDE 4.7 is branched, as well as trying to contribute to Qt 5. Seigo says that contributions to Qt 5 are going well so far, and that the Qt open governance process is working out satisfactorily. Though there is still work to be done and "lots of details left unresolved" he says that there have already been examples of pushing development upstream to Qt rather than trying to implement or fix things in KDE. Seigo gave the example of a problem with keyboard shortcuts due to a limitation of Qt, and says that he pushed to fix it in in Qt rather than KDE: "in two weeks it was done and in Qt."

After the 4.7 branch has been created, David Faure says the plan is to create a new branch in kdelibs for the work on splitting kdelibs. This will still continue based on Qt 4, says Faure, because "we don't need Qt 5 to reorganize our own code." At the same time, application developers will continue working in master with the kdelibs frozen.

How long will this go on? Seigo said that it will likely be at least two KDE releases before moving to Qt 5 and the new KDE Frameworks:

At the same time, KDE will continue bugfixes and features in kdelibs, but there will be no major changes.

When the time comes for applications to be ported to the new Frameworks, Seigo says that the application developers will find "a cleaner API and a much better dependencies story" and users will see performance gains — but there won't be a lot of changes due to Frameworks themselves. Seigo also says that it will mean dramatic improvements for "those using free software on tablets, set-top boxes" and a next-generation UI for mobile devices on top of Qt 5.

Naturally, mobile is another driver for the project re-thinking how it handles Frameworks. Seigo says that mobile and cross-platform development is very important for the KDE project and there will be a great deal of focus on Android and MeeGo for KDE.

Given the pain that users and application developers went through with the early releases of 4.x, this plan does seem to make a lot of sense for the project. Though it will take some time for it to come to fruition, there should be few downsides for application developers or users while other KDE developers focus on KDE Frameworks.

Comments (5 posted)

Brief items

Quotes of the week

-- Dan Williams on why NetworkManager got slower

-- Greg Stein

-- Tim Starling (thanks to Sumana Harihareswara)

Comments (none posted)

Apache traffic server v3.0.0 released

The Apache Software Foundation has announced the release of Apache traffic server v3.0.0. "Apache Traffic Server is a Cloud Computing 'edge' service, able to handle requests in and out of the Cloud, both by serving static content (images, JavaScript, CSS, and HTML files), and routing requests for dynamic content to a Web server (such as the Apache HTTP Server)." New features include better 64-bit support, better IPv6 support, web cache communication protocol (WCCP) support, a better plug-in API, and a number of performance improvements.

Comments (none posted)

Apache votes to accept OpenOffice.org for incubation

Apache held a vote over the weekend to decide whether to accept OpenOffice.org as an Apache incubator project. The result was overwhelmingly in favor of the proposal, so the project will move ahead under the Apache Software Foundation. It will presumably be several months (or perhaps quite a bit more) before the project can get on its feet and potentially be considered for top-level project status.

Full Story (comments: 41)

The Document Foundation's advisory board

The Document Foundation has announced the corporate members of its advisory board: Google, SUSE, Red Hat, Freies Office Deutschland e.V., Software in the Public Interest, and the Free Software Foundation. "The body represents The Document Foundation's sponsors, with each sponsor having the right to one representative. They will provide the future Board of Directors with advice, guidance and proposals, and will consult regularly on the further development of the Foundation and its associated projects."

Full Story (comments: none)

Results from the GNOME Foundation board election

The preliminary results of the 2011 GNOME Foundation board election have been posted. The upcoming board will consist of Shaun McCance, Emmanuele Bassi, Stormy Peters, Bastien Nocera, Brian Cameron, Germán Póo-Caamaño, and Ryan Lortie.

Full Story (comments: none)

Kontact Suite 4.6 released

The KDE project has announced the release of version 4.6 of the Kontact Suite. "Among the most noticable improvements are faster email notifications, vastly improved performance for IMAP email accounts, and improved interoperability with other applications that consume contacts, calendars and other groupware-related information."

Full Story (comments: none)

PathScale EKOPath 4 compiler suite open-sourced

PathScale has announced that its EKOPath 4 compiler suite will be turned into an open-source project. "This release includes documentation and the complete development stack, including compiler, debugger, assembler, runtimes and standard libraries. EKOPath is the product of years of ongoing development, representing one of the industries highest performance Intel 64 and AMD C, C++ and Fortran compilers." All that's missing is an actual link to the source; it appears to be showing up on github; the license seems to be GPLv3.

Comments (25 posted)

SeaMonkey 2.1 released

Version 2.1 of the SeaMonkey browser (and more) is out. "Building on the same Mozilla platform as Firefox 4, it delivers the latest developments in web technologies such as HTML5, hardware acceleration and improved JavaScript speed." Also included are cross-device synchronization, "personas," a new unified personal data manager, better plugin handling, and more; see the release notes for details.

Full Story (comments: none)

Upstart 1.3 released

Upstart 1.3 is out; it includes a number of enhancements developed for the Ubuntu "natty" release and more. This is the first release under the maintainership of James Hunt, who has implemented a new review policy aimed at, among other things, ensuring that copyrights for contributions have been appropriately assigned to Canonical.

Full Story (comments: 4)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (June 14)
• PostgreSQL Weekly News (June 12)
• Python-URL! (June 14)
• Tcl-URL! (June 13)

Comments (none posted)

Taylor: Benchmarking compositor performance

On his blog, Owen Taylor investigates compositor performance. "Now, this immediately gets us to a sticky problem: there are no mechanisms to throttle application frame rate to the compositor frame rate on the X desktop. Any app that is doing animations or video, or anything else, is just throwing frames out there and hoping for the best. Really, doing compositor benchmarks before we fix that problem is just pointless. Luckily, there's a workaround that we can use to get some numbers out in the short term — the same damage extension that compositors use to find out when a window has been redrawn and has to be recomposited to the screen can also be used to monitor the changes that the compositor is making to the screen. (Screen-scraping VNC servers like Vino use this technique to find out what they need to send out over the wire.) So, our benchmark application can draw a frame, and then look for damage events on the root window to see when the drawing they've done has taken effect."

Comments (4 posted)

Page editor: Jonathan Corbet

Announcements

Brief items

Announcing AndroidQuestions.org

The Questions Network has announced that AndroidQuestions.org is officially out of beta and open for business. "AndroidQuestions.org is for the discussion of all Android-related topics; from phones, tablets and other hardware devices to Android applications and development."

Full Story (comments: none)

Articles of interest

Adobe: Android is the Linux desktop

Adobe has announced that it will no longer be producing versions of Adobe AIR - or security updates - for desktop Linux; instead, they see Android as the future. Some more information can be found in this blog entry: "So, with Desktop Linux, we see a basically flat growth curve hovering around 1%. And since the release of AIR, we've seen only a 0.5% download share for desktop Linux. For Android and IOS we see substantial growth in share, and see predictions that indicate that in the Mobile OS market, the Android share could be 46%, with iOS at 16% (IDC March 2011)."

Comments (41 posted)

SCOTUS makes patent holders happy (ars technica)

Ars technica reports on the US Supreme Court's ruling in the i4i case. "In a New York Times op-ed supporting Microsoft, UCLA law professor Doug Lichtman had argued that changing the standard of proof would 'give relief to the countless businesses that today find themselves vulnerable to patents that shouldn't have been issued in the first place.' A wide variety of companies and public interest groups, including Google, Red Hat, Walmart, the Electronic Frontier Foundation, and the Apache Software Foundation, filed briefs echoing that point. But the Supreme Court decided that whatever the merits of these policy arguments, they couldn't overrule the text of the patent law and the courts' long history of employing the higher standard."

Comments (12 posted)

linux.conf.au finally controls domain name (iTWire)

ITwire reports that the well-known Linux conference linux.conf.au finally owns the domain name. ""So I'm very pleased to announce that, from this point onwards, the linux.conf.au domain will be available all year round, rather than the somewhat traditional six monthly on/off cycle that many conference regulars have become familiar with," wrote [Linux Australia president John] Ferlito."

Comments (6 posted)

Proprietary software keeps users helpless (TechRadar)

TechRadar covers a Linux Format Magazine interview with Richard Stallman. "In fact, when the open source philosophy spreads a lot - which it has - it tends to close people's minds to the ideas of free software. It even tends to cover up our existence. Most of the articles that talk about the GNU system, they don't call it the GNU system and they don't call it free software. They describe it as open source, and they give the impression that we - its developers - agree with the open source ideas that the readers have heard of already, and would never guess at what we're really standing for."

Comments (63 posted)

Calls for Presentations

PyTexas 2011 Call for Proposals

The fourth annual Python programming conference for Texas and the surrounding region (PyTexas 2011) will take place September 10-11, 2011 in College Station, Texas. The call for proposals is open until July 16.

Full Story (comments: none)

Upcoming Events

Explore Java 7 at OSCON Java

OSCON Java will take place July 25-27, 2011; co-located with O'Reilly's Open Source Convention (OSCON) in Portland, Oregon.

Full Story (comments: none)

USB mini-summit at LinuxCon Vancouver

There will be a USB mini-summit on August 16, 2011 at LinuxCon Vancouver. "Proposed topics include USB virtualization, and improved bandwidth APIs between the USB core and drivers (especially webcam drivers). See the detailed topic list below. Anyone is also welcome to propose or show up with a USB related topic. MUSB? USB 3.0 gadget drivers? USB-IP?"

Full Story (comments: none)

X.Org Developers Conference 2011

The X.Org Developers' Conference (XDC2011) will be held September 12-14, 2011 in Chicago, Illinois.

Full Story (comments: none)

Events: June 23, 2011 to August 22, 2011

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
June 20 June 26	EuroPython 2011	Florence, Italy
June 21 June 24	Open Source Bridge	Portland, OR, USA
June 27 June 29	YAPC::NA	Asheville, NC, USA
June 29	Scilab conference 2011	Palaiseau, France
June 29 July 2	12º Fórum Internacional Software Livre	Porto Alegre, Brazil
July 9 July 14	Libre Software Meeting / Rencontres mondiales du logiciel libre	Strasbourg, France
July 11 July 12	PostgreSQL Clustering, High Availability and Replication	Cambridge, UK
July 11 July 15	Ubuntu Developer Week	online event,
July 11 July 16	SciPy 2011	Austin, TX, USA
July 15 July 17	State of the Map Europe 2011	Wien, Austria
July 17 July 23	DebCamp	Banja Luka, Bosnia
July 19	Getting Started with C++ Unit Testing in Linux	,
July 24 July 30	DebConf11	Banja Luka, Bosnia
July 25 July 29	OSCON 2011	Portland, OR, USA
July 30 July 31	PyOhio 2011	Columbus, OH, USA
July 30 August 6	Linux Beer Hike (LinuxBierWanderung)	Lanersbach, Tux, Austria
August 4 August 7	Wikimania 2011	Haifa, Israel
August 6 August 12	Desktop Summit	Berlin, Germany
August 10 August 12	USENIX Security ’11: 20th USENIX Security Symposium	San Francisco, CA, USA
August 10 August 14	Chaos Communication Camp 2011	Finowfurt, Germany
August 13 August 14	OggCamp 11	Farnham, UK
August 15 August 16	KVM Forum 2011	Vancouver, BC, Canada
August 15 August 17	YAPC::Europe 2011 “Modern Perl”	Riga, Latvia
August 17 August 19	LinuxCon North America 2011	Vancouver, Canada
August 20 August 21	PyCon Australia	Sydney, Australia
August 20 August 21	Conference for Open Source Coders, Users and Promoters	Tapei, Taiwan

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol