Leading items

The unstoppable Perl release train?

There are a number of release management styles to be observed in the free software community, but many of them fall within the two broad categories of "time-based" or "feature-based". A time-based project fixes a date for its next release, then adjusts the set of features included in that release to make the intended date at least somewhat plausible. Feature-based projects, instead, pick a set of desired features and hold the actual release "until it's ready." Arguably, the trend over the last decade has been in the direction of time-based releases. Perl 5 is a relatively recent convert to the time-based model; the project is now faced with a decision between making a release with known problems - including possible security issues - or delaying its 5.16 release.

Contemporary Perl's release schedule is approximately one major release per year. The 5.14 release came out on May 14, 2011, so it is not too soon to be thinking about 5.16. That release is indeed stabilizing with a number of new features. A key aspect of this release is a familiar story: as with most other languages, Perl developers are trying to improve their support for Unicode in all situations. Many developers have participated in this work, but Tom Christiansen has arguably been the most visible. His recent work includes the preparation of an extensive Perl Unicode cookbook demonstrating Perl's Unicode-related features which, he has suggested, are now second to none.

As this cookbook was being discussed, Karl Williamson pointed out that some of the examples where Perl is put into full-time UTF8 mode (a very useful mode for contemporary programs) are unsafe because the language does not properly handle malformed strings. At best, such problems could lead to the corrupted strings seen in so many settings where character encodings are not properly handled. But, as Christian Hansen suggested, things can be worse than that:

What followed was a classic missive from Tom trying to get to the bottom of the problem. He listed nine different ways to tell Perl to operate with UTF8 and asked how many of them were truly vulnerable to undetected encoding problems. If Perl's UTF8 handling is unsafe by default, he said, it needs to be fixed:

And, he said, this situation really needs to be treated as a blocker for the 5.16 release; to do otherwise would be to delay the fix excessively and cause the world to be filled with bad workaround code.

In many projects, the prospect of open security-related problems would at least cause people to think about delaying a release. On the Perl list, though, Tom found little support. Aristotle Pagaltzis responded:

Ricardo Signes added:

And, with those words, the public email discussion faded away. At this point, there is little clarity on which Unicode features are safe to use, what might be required to fix the rest, how many of those fixes might be ready in time for the 5.16 release, or whether programs using UTF8 in Perl 5.16 (and earlier releases) suffer from known-exploitable security problems. Tom, who probably understands these issues better than just about anybody else, said:

The response he got claimed another train would be coming along in a year and the fixes could catch a ride on that one.

Releasing software with known bugs is a common practice; even a project like Debian cannot make the claim that there are no known problems with its releases. To do otherwise would make software releases into rare events indeed. It is also true that time-based releases have value; users know that they will get useful new code in a bounded period of time. Anybody who has watched release dates slip indefinitely knows how frustrating that can be for both users and developers; the 2.4.0 or 2.6.0 kernel releases are a classic example - as is Perl 5.10. So the Perl developers who say that the show must go on are doing so in accordance with many years of free software development experience.

That said, releasing software with known, security-related issues in something as fundamental as UTF8 support risks tarnishing the image of the project for a long time. There is not enough information publicly available to say whether Perl's UTF8 problems are severe enough to incur this risk. But it is probably safe to assume that, as a result of this conversation, crackers are looking at Perl's UTF8 handling rather more closely than they were before. Perl may have had some of these problems for years without massive ill effect, but they may not remain undiscovered and undisclosed for much longer. If the problem is real and exploitable, people are going to figure it out and take advantage of it.

One would assume that, behind the public positioning, the relevant developers understand what's at stake and are taking the time to understand the scope of the problem. They may not want to stop the release train, but seeing it derailed by a known security problem after release would not be much fun either. A release delay now may prove less painful than a security-update fire drill later.

Comments (24 posted)

Mozilla announces HTML5-based phone

Mozilla announced a deal with Latin American mobile carrier Telefónica Digital on February 27 to start shipping HTML5-driven smartphones by the end of 2012. Although the press release dubs the new platform "Open Web Device," many Mozilla watchers know it better as Boot To Gecko (B2G), a lightweight Linux-based system that uses Gecko as its primary (if not sole) application framework. B2G has been in development since July 2011, but it has advanced significantly since we last examined it in mid-August.

B2G architecture

Initially, the project debated several options for the underlying operating system on which to perch the Gecko runtime — including Android (which served as the basis for the earliest experimental builds), MeeGo, and webOS. The team has since settled on an architecture of its own, which takes some components from Android, but constructs its own stack, called Gonk.

Gonk is a stripped-down Linux distribution, using the kernel and some standard userspace libraries for hardware access (e.g., libusb and Bluez). It also borrows hardware abstraction layer code from Android — the architecture document on the Mozilla wiki lists camera and GPS devices in particular. The architecture document also makes references to both the Android kernel and to modified kernels supplied by hardware vendors; evidently B2G is designed to tolerate minor variations.

Gonk also includes an Android-derived init program, D-Bus, wpa_supplicant, and Network Manager. The media playback functionality also comes from Android: there is a mediaserver player for the audio and video formats that Gecko can decode natively, and libstagefright to access binary-only codecs and hardware decoding chips.

Mozilla's B2G team could not be reached for comment, but several of the Android components in the Gonk stack appear to have been chosen for the practical task of bootstrapping the OS on available smartphone hardware (such as the Samsung Galaxy S II, which is the testbed device at Mozilla) rather than on technical grounds. For instance, the setup currently uses Android's Bionic library, but there has been some discussion of replacing it with a more general libc implementation.

Apart from those components and the cellular modem layer, however, Gecko is the only runtime process started. In B2G, there is a master "Gecko server" process named b2g that spawns lower-privileged child processes for each web application run. Only the b2g process can directly access hardware devices; it mediates access requested by the child processes. b2g and the application processes can pass messages to each other using what Mozilla calls "IPC (Interprocess communication) Protocol Definition Language" or IPDL, which originates from the Electrolysis project. Electrolysis is Mozilla's effort to refactor Firefox as a multi-process application; it is still in development, but it provides both synchronous and asynchronous communication, and isolates child processes from each other.

Ultimately, Gonk is different enough from both Android and typical desktop Linux systems to warrant its own build target for Gecko. Among other distinctions, it implements a lightweight input event system, and draws directly to OpenGL ES rather than the X Window system. But in order to provide the full smartphone application framework, the Gecko engine must also have direct access to low-level functionality that is normally handled by other processes — such as the telephony stack.

New APIs

B2G's telephony stack is called the Radio Interface Layer (RIL), and is derived from the corresponding code in Android. RIL caters to device vendors by providing a rilproxy mechanism to manage connections between the b2g process and the cellular modem driver. The driver, called rild, can be proprietary (in fact, the RIL design assumes it is proprietary) and thus link to binary-only firmware blobs without triggering the license inheritance conditions from other parts of the stack.

RIL implements voice and 3G data calls, network status, text and MMS messaging, and SIM card commands (such as storing and retrieving contacts, or entering and validating unlock codes). These commands are exposed to HTML5 applications through the WebTelephony and WebSMS JavaScript APIs, which are still in development.

In fact, much of the B2G-specific functionality requires the definition of new APIs that "traditional" Gecko applications do not expect. Mozilla has already invested in the drafting of OS-independent web APIs for other reasons, including its Web App Store project. Mozilla is also participating in the standardization effort being managed by the W3C, primarily the Device APIs Working Group. At the moment, not every API created by Mozilla has a corresponding W3C specification, although Mozilla says standardizing all of them is the ultimate goal. Mozilla's list includes a mixture of high-level interfaces (such as the Contacts or Camera APIs) and low-level interfaces (such as the Bluetooth and near field communication (NFC) stack APIs).

The most far-reaching APIs in the library are Open Web Apps, which specifies a manifest file format and packaging guidelines for "installable" web applications, and WebRTC, which provides a real-time communication framework used for streaming media as well as network-centric features like peer-to-peer networking. Some of the less familiar specifications under development are APIs for detecting screen orientation, processing notifications when the device is idle, and locking hardware resources (such as preventing screen-dimming).

Applications and interfaces

The plan is for every application in a B2G device to be written in HTML5, CSS, and JavaScript, and naturally the list begins with the generic phone applications: a "home" screen, phone dialer, camera, contact list, calendar, and so on. While phone vendors may prefer to write their own code in-house, Mozilla is at least in the planning and mock-up stage of creating some demo B2G applications.

The demo application suite is code-named Gaia. The wiki page has placeholders for a dozen or so basic applications, including more detailed mock-ups for the web browser, image gallery, and camera application. There are also mock-up images on the Demo page for a home screen, lock screen, and dialer.

More interesting than static mock-ups are the in-development demos found at B2G developer Andreas Gal's GitHub site (screen shots of some are shown here). A recent version of Firefox will open the HTML files from the repository; the homescreen.html file shows the home screen and lock screen, and several other applications are accessible through home screen launchers (the Mozilla wiki's Demo page also links to a "live" demo, but that appears to be out-of-order at the moment). The UI responds to mouse clicks and "slide" gestures, although the applications vary widely in completion, of course — some are just static images. Still, the home screen and generic applications are in place along with basic navigation and UI elements.

The road ahead

The B2G roadmap is ambitious; it predicts a product-ready milestone by the end of the second quarter of 2012. Considering that not all of the fourth quarter 2011 goals have been met yet, that may not be attainable without an infusion of developer time from a hardware partner, but the project did bring a B2G demonstration to Mobile World Congress alongside the February 27 announcement.

Perhaps notably, the Bluetooth, NFC, and USB tracking bugs appear to be behind schedule at the moment, while most of the higher-level work for sensor support and user applications appear to be on track. Arguably the biggest piece of the puzzle yet to get underway is WebRTC support. WebRTC itself is still undergoing rapid development, of course, but without it, the goal of supporting real-time audio and video on a web-runtime smartphone is difficult to imagine.

Resources for potential Open Web Device application developers are also in short supply at the moment. Although Mozilla curates a collection of HTML5 resources at its Mozilla Developer Network (MDN) site, so far only the Open WebApps APIs are covered. Mozilla formally opened its Open Web App "Marketplace" on February 28, in an announcement that referenced the in-progress Web API effort in addition to the traditional desktop platform. That is probably a sign that the Web APIs will make their way to MDN in short order as well.

In its own press release, Telefónica Digital says only that its first Open Web Device-powered phone (which has not been detailed in hardware specification or price) will ship in 2012. That leaves considerable time to complete the project on schedule, although from Mozilla's perspective getting the new JavaScript APIs through the not-known-for-its-rapidity W3C standardization process may be a tight fit.

Comments (82 posted)

Various notes on /usr unification

"/usr unification" (or simply "usrmove") is the idea of moving the contents of /bin, /lib, and related root-level directories into their equivalents under /usr. That this scheme is controversial can be seen from that fact that, when LWN pointed to a document describing the motivations behind the move, a thread of well over 250 comments resulted. One would think, from the volume of comments on LWN and elsewhere, that this change is a threat to the very nature of Linux. Either that, or it has inspired one of the biggest bikeshedding exercises in some time.

One of the "advantages" of running a Rawhide system is that one gets to try out the new toys ahead of the crowd. Said toys also have a certain tendency to arrive at random and inopportune times. In the case of the /usr move, most Rawhide users got little or no warning before updates simply failed to work. The Rawhide repository, it seems, had suddenly been populated with packages requiring a system where the /usr move had already been done. Actually causing this move to happen was left as an exercise for the system administrator.

As it happens, this is one of the more controversial aspects of the /usr move in the Fedora community. Fedora policy has always said that using the yum tool to update an installation to a newer release of the distribution is not supported; one is supposed to use a CD or the preupgrade tool for that purpose. But, in reality, just using yum tends to work; the prospect of it not working to get to Fedora 17 has caused some grumpiness in the Fedora user community. If this limitation remains in the Fedora 17 release, expect to hear some complaints; users don't like to have their ponies taken away, even if nobody had ever said they could have a pony in the first place.

For Rawhide users, it is possible to move past the usrmove flag day by following this set of instructions. The process involves editing the grub configuration files, installing a special initramfs image, then rebooting with a special option that causes the actual thrashing of the system to be done from the initramfs before the real root is mounted. Your editor, setting out along this path with a well backed-up system, was reminded of doing the manual transition from the a.out to the ELF executable format on a Slackware system many years ago. In both cases, one had the sense of toying with fundamental forces with no guarantee of a non-catastrophic outcome.

In both cases, as it happens, things worked out just fine. Directories like /bin, /lib, /lib64, and /sbin are now symbolic links into /usr, and the system works just like it always did. No programs or scripts needed to be changed, custom kernels work as they always did, and so on. That will almost certainly be most users' experience of this transition - they will not even notice that it has happened. This is not a particularly surprising result; the practice of moving files and leaving a symbolic link in their place has a long history. Of course, somebody has probably patented it anyway.

Of course, an equally successful result for all systems depends on the Fedora developers creating a bombproof automatic mechanism for carrying out this transition prior to the Fedora 17 release. Hopefully something will have been learned from the GRUB 2 mess, where the Fedora 16 upgrade left a lot of bricked systems in its wake. If Fedora 17 creates similar messes on systems that, perhaps, are not set up quite as the Fedora developers expect, a lot of unhappiness will ensue.

As an aside, it is interesting to compare how Fedora is managing this transition with Debian's multiarch transition. Fedora has forced a flag day requiring the entire thing to happen at once; Debian, instead, has carefully avoided flag days in favor of a carefully-planned step-by-step change. One could argue that the Debian approach is better: it lets the transition "just happen" through normal package updates without the need for any special actions on the part of administrators or users. On the other hand, the multiarch transition has been a multi-year process and is still not complete; Fedora, instead, has pushed this change through in a small number of months.

Now that Fedora has made this jump and seems to be past the turning-back point, will other distributions follow suit? Some investigation suggests that Fedora will not be alone in this change:

• There have been no official statements from Red Hat, naturally, but it seems likely that, given how much effort has been put into this change by Red Hat employees, RHEL7 will feature the unified directory organization. The RHEL clones, including Oracle's distribution and CentOS, will have little choice but to incorporate this change.

• OpenSUSE has been discussing the idea for a while and seems to be receptive to it. There does not appear to be an official statement that openSUSE will make the /usr move, but there is a web page set up to track progress in that direction.

• Debian has had a number of long discussion threads about usrmove; it will not surprise longtime Debian watchers that there is a variety of opinions on the issue. There is some concern about possibly breaking systems with strange setups, though the most often cited case - systems with /usr on a separate partition - has been thought about and is relatively easy to support. In the end, it may come down to making the change to keep life easier; as Marco d'Itri put it: "I am not really looking forward to keep reverting these changes in my package, and since Red Hat controls most Linux infrastructure now other packages will face the same problem."

  Of course, Debian has some time to think about the problem. Nobody has suggested that usrmove could be added to the list of goals for the upcoming "wheezy" release, so any transition, even in the unstable tree, is likely to be at least a year away.

• Ubuntu has been relatively quiet about the idea; what has been posted by Ubuntu developers suggests a lack of enthusiasm for the idea. But if Debian makes the change, it could be hard for Ubuntu to retain the current filesystem layout.

In short, it looks like, over time, the move toward keeping all binaries and libraries in /usr will spread through most Linux distributions.

Given that most users will likely not even notice the change, it is natural to wonder why the change is so controversial. Undoubtedly, some people dislike significant changes to how traditional Linux systems have been laid out. Possibly, some creatively-organized systems will not handle the transition well, leading to problems that must be fixed. Conceivably, the change strikes some people as useless churn with no real benefits - at least for them. And, almost certainly, it is a bikeshed-type issue that is easy to have an opinion on and complain about.

The truth of the matter is that it appears to be a reasonably well-thought-out change driven by some plausible rationales. Filesystem hierarchies are not set in stone; there are times when it makes sense to clean things up. The usrmove operation yields a filesystem that is easier to understand, easier to manage, and more consistent in its organization. Having been through the transition, your editor thinks it is probably worth the trouble.

Comments (131 posted)

Page editor: Jonathan Corbet
Next page: Security>>