LWN Weekly Edition
Leading itemsSecurity
Kernel development
Distributions
Development
Announcements
->Multipage format
This page
Previous weekFollowing week
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
LWN.net Weekly Edition for January 20, 2011Reporting bugs upstreamBug reports from users are an important way for projects to find and eventually fix problems in their code. For most Linux users, though, the code from those projects makes its way to them via distributions, so that's where they report any bugs they find. What happens next is something of an open question, as a thread on the debian-devel mailing list highlights. Should it be up to the bug reporter to push the bug upstream if it is found to not be specific to the distribution? Or should it be up to the package maintainer? Brian M. Carlson started the thread by noting that he has been frequently asked recently to forward bugs reported to the Debian bug tracking system (BTS) to upstream projects. For a number of reasons, he doesn't think that's the right way to deal with these bugs:
I understand that maintainers' time is limited and that forwarding bugs
is not an enjoyable task. But I also understand that having a BTS
account for the upstream BTS of each of the 2405 packages I have
installed on my laptop (not to mention my other machines) is simply not
practical. I also don't have the benefit of the rapport that a
maintainer has with upstream and knowledge of upstream practices.
Essentially, Carlson is arguing that package maintainers already have the requisite relationships and accounts to properly move the bug upstream, whereas a regular user, even one who is fairly technically sophisticated, will not. But, of course, it is the user who has the intimate knowledge of the bug and can hopefully reproduce it or at least give the upstream developers a better idea of how to. Inserting the package maintainer into the middle of the bug triaging/fixing process may just slow things down. Unfortunately, most bug tracking systems don't provide a way for users without accounts to be copied on bug report traffic, which leads back to the problem that Carlson identified: users aren't interested in creating an account for each project's bug tracker, instead they rely on their distribution to upstream bugs. In addition, it may well be that the user doesn't really have the knowledge necessary to interact with upstream, especially for distributions that target less-technical users. Many of the bug reports that come into any bug tracker (Debian's, other distributions', or the projects') are inadequate for various reasons, so forwarding them or asking the user to report them upstream is pointless—potentially counterproductive because it annoys the upstream developers. It comes down to a question of what the responsibilities of a package maintainer are. Many seem to take a proactive approach by trying to reproduce the problem and, if they can, reporting it upstream. Others, though, are completely swamped by the sheer number of bugs that get reported, finding it difficult to do more than try to determine if the bug is Debian-specific and asking for it to be reported upstream if it isn't. There is a large disparity in the size of the packages that are maintained by Debian (or any other distribution), of course. Some smaller or less-popular packages may allow their maintainers to keep up with the bugs and shepherd those that deserve it upstream. While other, larger packages, like the kernel, X, GNOME, KDE, LibreOffice, and so on, just don't have enough people or time to do that. Thus it is very unlikely that there can be a "one size fits all" solution; each package and maintainer may necessarily have their own bug management style. John Goerzen explained how the process works for the Debian package of the Bacula backup program, noting that he sits between the bug reporter and the upstream project, but that it is often wasted effort:
I'm adding zero value here. Zero. It is a huge and frustrating waste of
my time. It is also frustrating for upstream, who would rather just talk
with the user directly and involve me if they think there's a
Debian-specific question. I don't understand why some users want it to go
this way, but many clearly do despite the fact that they get worse service.
Being a "copy and paste monkey between emails and web forms" is not what he wants to be doing. He points out that various unnamed major packages tend to just ignore many of their bug reports for years at a time. Overall, he doesn't think that being a maintainer should necessitate being a bug intermediary as well:
But really, I think it is
nonsensical for an end user to expect me to do this because the user
doesn't want to spend 30 seconds creating an account on an upstream
BTS. That's not what Free Software is all about. And it has Debian
maintainers wasting time.
I think that promising that Debian maintainers will always shepherd bugs upstream is promising something we don't actually deliver on very well, and probably never have. Perhaps we should stop promising it. Ben Finney (and others) see it as, at least partially, a technical problem in the bug tracking software. While requiring bug reporters and those copied on the bug comments to be registered in the system may make sense—for reducing spam problems if nothing else—it definitely puts up a barrier for users:
Quite the opposite, from my position. A great feature of the Debian BTS
is that any user can interact with it through a standard interface
without maintaining multiple separate identities; heck, without having
to create a single new identity at all.
Having to create and maintain (or fail to maintain) identities with balkanised upstream BTSen is bad enough as a package maintainer. As a mere user of all the other packages on my computers, I consider it too high a barrier. Finney looks at the problem as an opportunity to try to find better technical solutions. There may be ways to enhance the Debian BTS to file bugs upstream and/or CC the Debian bug reporter on upstream questions, as Don Armstrong suggests. Ubuntu's Launchpad has made some efforts to link downstream and upstream bugs to address some parts of the problem, but it is more than just a technical problem as various posters pointed out. Many upstream projects are largely uninterested in bugs reported against older versions of their code. Unless the bug can be reproduced in the latest release—or sometimes the latest commit to the project's source code repository—it may not be investigated at all, even if the bug is reported upstream. It is not just package maintainers that have far more work, and bugs, than they can possibly deal with. But distributions have something of a contract with their users to support the package versions that make up the distribution release. It can be very difficult to have a user test with updated versions of the code in question to try to reproduce the problem. Bugs that can be easily reproduced are obviously much easier to handle, but they are also, seemingly, much less common. Fedora has struggled with similar issues, including discussions last July on the fedora-devel and test mailing lists; undoubtedly other distributions have as well. Also, it isn't just the distribution and user side that suffers, as there have been cases where bugs and fixes known to downstream distributions haven't made their way upstream. It would seem that there is an opportunity for projects and distributions to work more closely together, but that won't be easy given the workloads on both ends of the stream. Bug reports are a difficult problem in general, as good reports are sometimes hard to find. There may also be a few layers between the reporter and someone who might be able to actually fix the problem, which can cause all manner of frustrations for anyone involved. On the bright side, though, the situation is far better than the proprietary software world, where reporting a bug may require paying for a "trouble ticket" and waiting for a release that actually addresses it. At least free software allows an interested hacker to poke around in the code and fix it for themselves (or their customers).
The Cr-48 and Chrome OS: Google's vision of the netThe Cr-48 is, according to Google, the "first of its kind - a notebook built and optimized for the web." It is the next step in the promotion of Chrome OS, Google's other Linux-based distribution. As a way of showing off what it has accomplished and building interest in the system, Google has distributed Cr-48 machines widely. Your editor was a lucky, if late, recipient of one of these devices; what follows are his![[Cr-48]](/images/2011/cr48.png)
impressions after some time playing with it. The Cr-48 and Chrome OS are
an interesting vision of where computing should go, even if that vision is not
for everybody.
The hardware itself is quite nice at a first glance. This machine is not a netbook; it is a small notebook device which clearly has taken some inspiration from Apple's hardware. Except, of course, that Apple's machines are not jet black, with no logos or markings of any type. It exudes a sort of Clarke-ian "2001 monolith" feel. There's an Intel Atom dual-core processor, 2GB of memory, and a 16GB solid-state drive. The silence of the device is quite pleasing; also pleasing is the built-in 3G modem with 100MB/month of free traffic by way of Verizon (which, unsurprisingly, is more than prepared to sell you more bandwidth once that runs out). Other connectivity includes WiFi and Bluetooth (though there appears to be no way to use the latter); there is no wired Ethernet port. There's a single USB port, an audio port, a monitor port, and what appears to be an SD card reader. Battery life is said to be about eight hours. Despite the small disk, it's a slick piece of hardware.
Using Chrome OSThe operating system and the hardware work nicely together. A cold boot takes a little over ten seconds; suspend and resume are almost instantaneous. In normal use, one simply lifts the lid and the system is ready to go; by default, the system does not even request a password at resume time if somebody is logged in - a setting that security-conscious users may want to change. There is a large trackpad with some simple multitouch capability. Interestingly, there is no "caps lock" key; Google, in its wisdom, replaced it with a "search" key. Happily, Google was also wise enough to allow the key to be remapped by the user; it can be restored to caps lock or, instead, as $DEITY intended, set to be a control key. Where one would expect to find the function keys are more web-centric buttons: Google has dedicated keys to operations like "back," "forward," and "reload." Of course, they're really just function keys underneath as far as the X server is concerned.
The system software is Linux-based, of course, but there's no way for a
casual user to notice that. The core idea behind Chrome OS is that
anything of interest can be had by way of a web browser, so that's all you
get. Like an Android phone, the system starts by asking for the user's
That is why the Cr-48 comes with such a small SSD; very little is stored there beyond the operating system image itself, and that image is small. Most of the space, in fact, is set aside for a local cache, but it's entirely disposable; everything of interest lives in the Google "cloud." So if, as the startup tutorial says, the device succumbs to an "unexpected steamroller attack," nothing is lost except the hardware. The user can sign onto a new device and everything will be there. The appeal of this arrangement is clear: no backups, no lost data, no hassles upgrading to a new machine. Just browse the web and let Google worry about all the details. Of course, there are some costs; the Cr-48 can do almost nothing which cannot be done via the web. There is no way to get a shell (though see below) and no way to install Linux applications. Even updates are out of the user's hands: they happen when the Chrome OS Gods determine that the time is right. There is a "web store" where browser-based applications can be had. At this time there is a surprising variety of them, almost all of which are free of charge. The application selection still falls far short of what is available with a standard Linux distribution or on Android, though. It's also not at all clear how many (if any) of these applications are actually free software. The "no local installations" philosophy means that Chrome browser plugins (which hook into the browser at a lower level than "applications" do) cannot be installed; that, in turn, means that any application which requires a plugin, while usable on regular Linux or Windows, is not installable on Chrome OS. It turns out that quite a few web store applications need plugins; annoyingly, the only way to find out if any given application can be installed is to try. Your editor wanted to take a screenshot or two of the system in operation. The store offers a few screenshot applications, one provided by Google itself. The Google tool, though, needs a plugin and thus refused to install. An alternative application did install, but the "save" button, needing a plugin, was not able to save the result anywhere. The application could, though, "share" the screenshot through any of a number of web services - though the image itself (to your editor's surprise) is stored on the web site of the company providing the screenshot application. Something as simple as taking a screenshot should not be so hard - and it should not broadcast screenshots to the world by default.
Under the hoodThe Cr-48 is a locked-down system. Its firmware will only load Google-signed images, so it's not possible for the user to make any changes. The root filesystem is mounted read-only. The whole verified boot mechanism is designed to ensure that the device's software has not been compromised and that the user can trust it. That said, the design goals are also expressed this way:
It is important to note that restraining the boot path to only
Chromium-project-supplied code is not a goal. The focus is to
ensure that when code is run that is not provided for or maintained
by upstream, that the user will have the option to immediately
reset the device to a known-good state. Along these lines, there is
no dependence on remote attestation or other external
authorization. Users will always own their computers.
The way this works on the Cr-48 is through a "developer switch," which is cleverly hidden behind a piece of tape inside the battery compartment. The instructions describe a lengthy series of events that will happen when that switch is flipped, including a special warning screen and a five-minute delay while the system cleans up any personal data which may be cached locally. What actually happened was a warning that the system is corrupted; hitting control-D at that screen did manage to boot the system into the developer mode, though.
Developer mode looks much like the regular operating mode with one
exception: the other virtual consoles are now enabled, allowing the user to
get to a shell and explore the system a bit. The system, it turns out, is
based on a 2.6.32.23 kernel; it's said to be based on The system uses the ext3 filesystem for local data storage. There are two sets of root filesystem partitions; one is in use while updates are loaded into the other. It also uses eCryptfs to store user-specific data; in theory that means that such data is safe from prying eyes when the user is not actually logged into the system. Given access to developer mode, one can go as far as installing an entirely new operating system on the device. The instructions for doing so are intimidating at best, though; Google has not gone out of its way to make displacing Chrome OS easy. Your editor will probably give it a try at some point, but the job did not look like something which could be done within any sort of deadline. It sure would have been nice if the system could just boot from an external device.
What it's good forThe appeal of a system like this is easy enough to understand. Here is a computer which can access all kinds of web-based services, never needs to be backed up, is highly malware-resistant, and which can be easily replaced. It could be handed to one's children with minimal fear of the consequences, and it is easily operated by people who are intimidated by any sort of system management task. A Chrome OS device is the contemporary equivalent of an X terminal; it is little more than a window into services which are managed elsewhere.
Your editor, who is not afraid to That said, such machines are not without their applications. Thousands of people, it seems, have had their laptops searched at the US border; your editor, who crosses that border frequently, has not, yet, had that experience. Should it ever come to pass, it might be nice to have a laptop which contains no local data at all. A throwaway Google account could be used for plausible deniability, and, in the unlikely case of a border agent who knows about the developer switch, any user-specific data on the system (which is encrypted anyway) should be gone by the time it becomes accessible. "Data in the cloud" systems have security concerns of their own (it would be nice if a Chrome OS system could be backed up by providers other than Google, for example), but there are times when having all of one's data be elsewhere can be comforting. The locked-down nature of Chrome OS is thus not without its value, but locked-down is only good as long as the owner wants things that way. The Chrome OS documentation suggests that Google wants all devices to include a developer switch. In the real world, it would be unsurprising if some vendors somehow never quite got around to adding that switch. Without full access, one of these laptops becomes something more like a television: useful for displaying content, but something short of a real computer. Chrome OS is clearly not meant to be a "real computer" of the sort that LWN readers are likely to want. The target user base is different, to say the least. As such, it is an interesting exercise in what can be done to package Linux for other classes of users. At the beginning of the year, your editor predicted that Chrome OS would struggle; who wants such a limited system when a real computer can be so easily had? Based on this experience, your editor is not quite ready to change his mind, but he is willing to admit that Chrome OS may be the experience some people are looking for.
Review: The Linux Programming InterfaceMichael Kerrisk's (relatively) new book, The Linux Programming Interface (TLPI), is targeted at Linux system programmers, but it is not just those folks who will find it useful. While it is a hefty tome ("thick enough to stun an ox" as Laurie Anderson might say), it is eminently readable, both by browsing through it or by biting the bullet and reading it straight through. The coverage of the Linux system call interface is encyclopedic, but the writing style is very approachable. It is, in short, an excellent reference that will likely find its way onto the bookshelves of user-space developers and kernel hackers—including some who aren't necessarily primarily focused on Linux. ![[TLPI cover]](/images/2011/TLPI-front-cover.png)
Kerrisk has been the maintainer of the Linux man pages since 2004, which gives him a good perspective on the Linux API. As he says in the preface, it is quite likely that you have already read some of his work in sections 2, 3, 4, 5, and 7 of those pages. But the book is not a collection of man pages though it covers much of the same ground. The style and organization is much less dry, and more explanatory, than a typical man entry. The book is some 1500 pages in length, which makes it a rather daunting prospect to review. Once I started reading it, though, it was quite approachable. Kerrisk's clear descriptions of various system calls and other parts of the Linux API made it easy to keep reading. I set out to pick and choose certain chapters to read, and just skim the others, but found myself reading quite a bit more than that—which might partially explain the lateness of this review. The book is organized into 64 chapters of around 20 pages each, which makes for nice bite-sized chunks that allow for reading the book around other tasks. While the focus is on Linux, Kerrisk doesn't neglect other Unix varieties and notes where they differ from Linux. He also pays careful attention to the various standards that specify Unix behavior—like POSIX and the Single Unix Specification (SUS)—pointing out where Linux does and does not follow those standards. TLPI was written for kernel version 2.6.35 and glibc 2.12. In the text, though, Kerrisk is careful to indicate which kernel version introduced a new feature, so that those working with older kernels will know which they can use. While it is primarily looking at the 2.6 series, 2.4 is not neglected, and the text notes features that were introduced at various points in the 2.4 kernel history. The book starts with a bit of history, going all the way back to Ken Thompson and Dennis Ritchie and then moving forward to the present, looking at the various branches of the Unix tree. It then moves into a description of what an operating system is, the role that the kernel plays, and some of the overarching concepts that make up Unix (and Linux). While this information may be unnecessary for most Linux hackers, it will come in handy for those coming to Linux from other operating systems. The ideas that "everything is a file" and that files are just streams of bytes are described in ways that will quickly get a system programmer up to speed on the "Unix way". After that introductory material, Kerrisk launches into the chapters that cover aspects of the system call interface. This makes up the vast majority of the book and each of these chapters is fairly self-contained. They build on the earlier chapters, but the text is replete with references to other sections. In the preface, Kerrisk says that he attempted to minimize forward references, but that clearly was a difficult task as there are often as many forward as backward references in a chapter. Navigating within the book is easy to do because there are frequent numbered section and subsection headings, along with the chapter number on each page. Other technical books could benefit from that style. There is also an almost too detailed index that runs to more than 50 pages. Each chapter comes with sample code that is easy to read and understand. Importantly, the examples also do a good job of demonstrating the topic at hand and some of them could be adapted into useful utilities. The code is available from the TLPI web site and is free software released under the Affero GPLv3. Each chapter also has a handful of exercises for the reader, some of which have answers in one of the appendices. So, what does the book cover? It would be easy to say "all of it", but that would be something of a cop-out, and a bit inaccurate as well. There are multiple chapters on files, file I/O, filesystems, and file attributes, extended attributes, and access control lists (ACLs). There is a chapter covering directories and links, as well as one that looks at the inotify file event notification call. There are multiple chapters on processes, threads, signals, as well as chapters covering process groups and sessions, and process priorities and scheduling. Of particular interest to me were a chapter on writing secure privileged programs and one on Linux capabilities. There are two chapters on shared libraries, the first of which is more about the ideas underlying libraries and shared libraries along with how to build them, rather than the dlopen() system call (and friends), which is covered in the second. There are, perhaps, too many chapters covering interprocess communication (IPC), with separate chapters devoted to each System V IPC mechanism (shared memory, message queues, and semaphores). There is also a chapter for each of the POSIX variants of those three IPC types. Both POSIX and System V IPC get their own introductory chapter in addition to the chapters focusing on the details of each type. Sandwiched between the System V and POSIX IPC mechanisms are two chapters on memory mapping and virtual memory operations that might have been better placed elsewhere in the book. There is also a chapter devoted to an introduction to IPC and one that looks at the more traditional Unix pipes and FIFOs. In all, there are twelve chapters on IPC before we even get to the sockets API. After IPC, comes a chapter on file locking followed by six chapters covering sockets. Those chapters look at Unix and internet domain sockets, along with server design and advanced sockets topics. The book wraps up with a chapter on each of terminals and pseudoterminals, with something of an oddly placed "Alternative I/O Models" chapter in between them. It's an interesting chapter, covering select(), poll(), epoll(), signal-driven I/O, and a few other topics, but it seems weird where it is. There is more, of course, and looking at the detailed table of contents will fill out the list. One thing that stands out from the book is the vast size of the Linux/Unix API. It also points out some of the warts and historical cruft that is carried along in that API. Kerrisk is not shy about noting things like that where appropriate in the text: "In summary, System V message queues are often best avoided." There were two specific topics that I looked forward to reading about but were only marginally covered by the book. The first is containers and namespaces, which are very briefly mentioned in a discussion of the flags to the clone() system call. A more puzzling omission is that there is almost no mention of the ptrace() system call. In the few places it does come up, readers are referred to the ptrace(2) man page. There are certainly other parts of the Linux API that could have been covered, beyond the system call interface—sysfs, splice(), and perf come to mind—but Kerrisk undoubtedly needed to draw the line somewhere. Overall, he did an excellent job of that. Technical books, especially those covering Linux, have a tendency to get stale rather quickly, but TLPI shouldn't suffer from that as much as a kernel internals book would, for example. There should really only be additions down the road as the user-space API is maintained by the kernel developers "forever", but updates will presumably need to be made eventually. There are a handful of additional complaints I could make about the book, but they are all quite minor, as were those mentioned above. The biggest nit is that the "asides" in the text, which are numerous, are really often much more than just asides. Each is set off from the rest of text, indented and rendered in a slightly smaller font (which is typographically a bit annoying to me), and are meant to contain additional information that is not necessarily critical to understanding the topic. In my experience, though, many of them might best have been worked into the main text. See what I mean about minor complaints? This is a book that will be useful to application and system-level developers, primarily, but there is much of interest for others as well. Kernel hackers will find it useful to ensure their new feature (or fix) doesn't break the existing API. Programmers who are primarily targeting other Unix systems may also find it useful for making their code more portable. I found it to be extremely useful and expect to return to it frequently. Anyone who has an interest in programming for Linux will likely feel the same way.
Security Tarsnap advisory provides a few lessonsAn interesting and brutally honest security advisory for the Tarsnap "secure online backup service" was released on January 18. It certainly shows a refreshing amount of candor that other projects and companies would do well to emulate. But there are some other lessons to be learned from the vulnerability including the value of source code availability and the perils of refactoring. Tarsnap is a company founded by Colin Percival that provides encrypted online storage for backups. The client code is available, but it is not free software. The code can only be used, unmodified, to talk to the Tarsnap service. The server code is evidently completely unavailable, but Percival is interested in hearing from folks with ideas for improvement to the client—or those who have found a security hole. Percival was contacted by Taylor R. Campbell on January 14 with just such a bug. It turns out that a refactoring of the code for the 1.0.22 release, which was made in June 2009, introduced a bug that potentially would allow anyone with access to the data to decrypt it. The data is stored in the Amazon S3 "cloud", which limits the access to a small group, but that doesn't really fit well with the security model espoused by Tarsnap. In the advisory, Percival makes that clear:
I will not attempt to decrypt and read your data. Amazon claims that it
does not inspect Amazon Web Services users' data. And the US government is
theoretically bound by a constitution which prohibits unreasonable
searches. This is all, however, entirely irrelevant: The entire point of
Tarsnap's security is to remove the need for such guarantees. You shouldn't
need to trust me; you shouldn't need to trust Amazon; and you most
certainly shouldn't need to trust the US government.
In doing the refactoring, Percival removed an auto-increment of a nonce value used in the Advanced Encryption Standard (AES) Counter (CTR) mode for encrypting blocks of data. The impact of that is that someone can decrypt the data without having the key. There are two ways that the decryption could be done when the nonce value is reused, either by comparing two ciphertexts or by using known plaintext. The former attack is considered by Percival to be unusable on the Tarsnap data because of the compression done to the data blocks before they are encrypted. On the other hand, known plaintext attacks are quite plausible if there is some known data in the blocks. As Percival points out, full backups are likely to have any number of files with known contents, namely the files that are installed by the operating system—binaries, configuration files, and so on. The bug was found by Campbell by "reading the Tarsnap source code purely out of curiosity", which certainly shows the advantage of making that source available. One wonders if the server code might also benefit from curious hackers. Percival is creating a bug bounty program (and seemingly retroactively paying one out to Campbell) to hopefully ferret out any other problems in the client sooner. Refactoring is meant to be strictly a clean-up operation that does not change the semantics of the code in question. When doing refactoring, it is helpful if there are a set of regression tests that can detect when refactoring has gone awry. In the comments on the advisory, Percival said that Tarsnap does not have a test suite of that sort, and pointed out that it is difficult to create one for cryptographic software, but "I should probably find some way of automatically testing and/or assert()ing for nonce-reuse bugs though". The lack of regression tests is unfortunate, but Tarsnap is hardly alone in that. There are countless projects that refactor their code without such a test suite. This particular incident should serve as something of a reminder to projects, especially those that are implementing security features, that refactoring can and does introduce bugs. A test suite is great, but even just some regression testing of the areas that have been refactored may find bugs like this one. Percival is to be congratulated for quickly turning around a fix for the problem, as well as for being so forthright with the gory details of the bug and its impact. It is far too often that we see companies trying to sweep the details of their security holes under the rug—free software projects sometimes do as well. Bugs happen, security or otherwise, and there is value in seeing what they are and how they came about. We can learn from incidents like this.
Brief items A critical security bug in tarsnapThe author of tarsnap ("online backups for the truly paranoid") has sent out an advisory describing a "critical" security bug in versions 1.0.22 through 1.0.27. "It may be possible for me, Amazon, or US government agencies with access to Amazon's datacenters to decrypt data stored with those versions of Tarsnap. This is an absolutely unacceptable compromise of Tarsnap's security principles, and I sincerely apologize to everyone affected." The posting describes how to respond to the problem and is an interesting discussion of how easily things can go wrong in security-related code.
New vulnerabilities ccid: arbitrary code execution
chromium: mysterious vulnerabilities
gif2png: denial of service
hplip: arbitrary code execution
java-1_6_0-openjdk: security manager bypass
kernel: denial of service
kernel: multiple vulnerabilities
libtiff: denial of service
mydms: directory traversal
pcsc-lite: arbitrary code execution
perl-CGI: HTTP response splitting attacks
pimd: insecure temporary files
prewikka: password leak
sssd: denial of service
subversion: denial of service
sudo: group-related vulnerabilities
tor: multiple vulnerabilities
wireshark: arbitrary code execution
xfig: multiple vulnerabilities
Page editor: Jake Edge Kernel development Brief items Kernel release statusThe current development kernel is 2.6.38-rc1, released on January 18. "It's been two weeks, and the merge window for 2.6.38 is thus closed. And an interesting merge window it has been." All told, a little over 7,600 changes were merged in this merge window. Some of them affect core code in fairly invasive ways, and there have been some significant regressions reported already. People testing 2.6.38-rc1 might want to be just a little more careful (and better backed up) than usual. See the full changelog for all the details.Stable updates: there have been no stable updates released in the last week.
Quotes of the week
Free software's awfully like sausages - wonderfully tasty, but
sometimes you suddenly discover that you've been eating sheep
nostrils for the past 15 years of your life.
-- Matthew Garrett
It is often the case that certain developers don't have a full
understanding of the cost of the code they're writing, even when
they're using C. Not everyone is forced to spend part of their life
looking at the compiler output from their code and seeing what it
actually does, although they ought to be.
-- David Woodhouse
Bypassing linux-nextIt has been almost three years since the creation of the linux-next tree; during that time, it has become an indispensable part of the kernel development process. By the time code is merged into the mainline during the merge window, it has already seen a fair amount of integration and compilation testing in linux-next - and even some actual run testing. That has helped to make the merge window run more smoothly. So it's not surprising that developers are getting increasingly grumpy when code is seen to be circumventing linux-next and creating problems in the mainline.We've had a couple of examples of that grumpiness in the 2.6.38 cycle. When Al Viro posted his first VFS pull request, linux-next maintainer Stephen Rothwell complained that this was his first sighting of that code, despite the fact that it had apparently been around for a few months. Al is known for pulling together mainline submissions at the last minute, so this sort of thing is not entirely surprising; it remains to be seen whether he can be pushed into changing his ways. The other complaint came after the merging of the transparent huge pages patch set, which went in by way of Andrew Morton's -mm tree. Tony Luck, having discovered that the ia64 architecture no longer built in the mainline, asked:
Didn't Andrew make some rash promise at kernel summit about
stopping eating if "-mm" wasn't included in linux-next by the end
of November? Must be getting pretty hungry by now.
Andrew responded that "It's taking a while - Stephen and I are discussing a plan." Integrating -mm was always going to be a bit of a challenge; linux-next is supposed to contain code which is ready for merging into the mainline, while -mm can carry under-development code for years. Until that gets worked out, though, memory management developers are going to be in a bit of a difficult position; there is no maintainer tree they can get into which feeds into linux-next. Those developers will need to either get their own trees into linux-next (an easy thing to do) or take the complaints when code which lived in -mm is seen by testers for the first time when it hits the mainline.
Kernel development news 2.6.38 merge window part 2As of the 2.6.38-rc1 release, some 7616 non-merge changesets had been pulled into the mainline kernel. A number of significant changes have been merged since last week's summary; the most interesting changes visible to users are:
Changes visible to kernel developers include:
With the 2.6.38 feature set complete, the process of stabilizing all of this new code can continue; expect a final 2.6.38 release sometime in late March.
Transparent huge pages in 2.6.38The memory management unit in almost any contemporary processor can handle multiple page sizes, but the Linux kernel almost always restricts itself to just the smallest of those sizes - 4096 bytes on most architectures. Pages which are larger than that minimum - collectively called "huge pages" - can offer better performance for some workloads, but that performance benefit has gone mostly unexploited on Linux. That may change in 2.6.38, though, with the merging of the transparent huge page feature.Huge pages can improve performance through reduced page faults (a single fault brings in a large chunk of memory at once) and by reducing the cost of virtual to physical address translation (fewer levels of page tables must be traversed to get to the physical address). But the real advantage comes from avoiding translations altogether. If the processor must translate a virtual address, it must go through as many as four levels of page tables, each of which has a good chance of being cache-cold, and, thus, slow. For this reason, processors maintain a "translation lookaside buffer" (TLB) to cache the results of translations. The TLB is often quite small; running cpuid on your editor's aging desktop machine yields:
cache and TLB information (2):
0xb1: instruction TLB: 2M/4M, 4-way, 4/8 entries
0xb0: instruction TLB: 4K, 4-way, 128 entries
0x05: data TLB: 4M pages, 4-way, 32 entries
So there is room for 128 instruction translations, and 32 data translations. Such a small cache is easily overrun, forcing the CPU to perform large numbers of address translations. A single 2MB huge page requires a single TLB entry; the same memory, in 4KB pages, would need 512 TLB entries. Given that, it's not surprising that the use of huge pages can make programs run faster. The main kernel address space is mapped with huge pages, reducing TLB pressure from kernel code. The only way for user-space to take advantage of huge pages in current kernels, though, is through the hugetlbfs, which was extensively documented here in early 2010. Using hugetlbfs requires significant work from both application developers and system administrators; huge pages must be set aside at boot time, and applications must map them explicitly. The process is fiddly enough that use of hugetlbfs is restricted to those who really care and who have the time to mess with it. Hugetlbfs is often seen as a feature for large, proprietary database management systems and little else. There would be real value in a mechanism which would make the use of huge pages easy, preferably requiring no development or administrative attention at all. That is the goal of the transparent huge pages (THP) patch, which was written by Andrea Arcangeli and merged for 2.6.38. In short, THP tries to make huge pages "just happen" in situations where they would be useful. Current Linux kernels assume that all pages found within a given virtual memory area (VMA) will be the same size. To make THP work, Andrea had to start by getting rid of that assumption; thus, much of the initial part of the patch series is dedicated to enabling mixed page sizes within a VMA. Then the patch modifies the page fault handler in a simple way: when a fault happens, the kernel will attempt to allocate a huge page to satisfy it. Should the allocation succeed, the huge page will be filled, any existing small pages in the new page's address range will be released, and the huge page will be inserted into the VMA. If no huge pages are available, the kernel falls back to small pages and the application never knows the difference. This scheme will increase the use of huge pages transparently, but it does not yet solve the whole problem. Huge pages must be swappable, lest the system run out of memory in a hurry. Rather than complicate the swapping code with an understanding of huge pages, Andrea simply splits a huge page back into its component small pages if that page needs to be reclaimed. Many other operations (mprotect(), mlock(), ...) will also result in the splitting of a page. The allocation of huge pages depends on the availability of large, physically-contiguous chunks of memory - something which Linux kernel programmers can never count on. It is to be expected that those pages will become available at inconvenient times - just after a process has faulted in a number of small pages, for example. The THP patch tries to improve this situation through the addition of a "khugepaged" kernel thread. That thread will occasionally attempt to allocate a huge page; if it succeeds, it will scan through memory looking for a place where that huge page can be substituted for a bunch of smaller pages. Thus, available huge pages should be quickly placed into service, maximizing the use of huge pages in the system as a whole. The current patch only works with anonymous pages; the work to integrate huge pages with the page cache has not yet been done. It also only handles one huge page size (2MB). Even so, some useful performance improvements can be seen. Mel Gorman ran some benchmarks showing improvements of up to 10% or so in some situations. In general, the results were not as good as could be obtained with hugetlbfs, but THP is much more likely to actually be used. No application changes need to be made to take advantage of THP, but interested application developers can try to optimize their use of it. A call to madvise() with the MADV_HUGEPAGE flag will mark a memory range as being especially suited to huge pages, while MADV_NOHUGEPAGE will suggest that huge pages are better used elsewhere. For applications that want to use huge pages, use of posix_memalign() can help to ensure that large allocations are aligned to huge page (2MB) boundaries. System administrators have a number of knobs that they can tweak, all found under /sys/kernel/mm/transparent_hugepage. The enabled value can be set to "always" (to always use THP), "madvise" (to use huge pages only in VMAs marked with MADV_HUGEPAGE), or "never" (to disable the feature). Another knob, defrag, takes the same values; it controls whether the kernel should make aggressive use of memory compaction to make more huge pages available. There's also a whole set of parameters controlling the operation of the khugepaged thread; see Documentation/vm/transhuge.txt for all the details. The THP patch has had a bit of a rough ride since being merged into the mainline. This code never appeared in linux-next, so it surprised some architecture maintainers when it caused build failures in the mainline. Some bugs have also been found - unsurprising for a patch which is this large and which affects so much core code. Those problems are being ironed out, so, while 2.6.38-rc1 testers might want to be careful, THP should be in a usable state by the time the final 2.6.38 kernel is released.
Reworking disk events handlingA look inside any contemporary desktop-oriented system is likely to reveal a process which is steadfastly polling removable drives on the off chance that somebody might have removed or inserted a disk. Indeed, as your editor can attest, it can be hard to turn that polling off; there's little room in the world for strange people who have their own ideas of what they want to happen when they put a disk into a drive. Be that as it may, if the system is going to poll drives, it would be nice to do so in the best way possible. That is not currently the case, but, thanks to a patch by Tejun Heo, drive polling should be better in 2.6.38.There are a few problems with how polling is done on Linux; these were nicely outlined by Tejun in the patch changelog. Polling on Linux requires opening the device; this is a somewhat heavyweight operation which does not naturally line up with other operations which might wake the processor. Opening the device in this way might interfere with other users; optical disk burning, in particular, is susceptible to this kind of problem. And polling the disk in this way generates a different set of commands than Windows uses; as Linux driver developers have discovered many times, behavior that differs from Windows is not well tested by vendors and tends to have unpleasant bugs. All that notwithstanding, user-space polling works well enough most of the time, but it would still be nice to make it better. Tejun's patch works by moving the polling into the kernel. That makes the polling more efficient by removing the need to open the device and by allowing the kernel to delay polling wakeups until something else is going on as well. There is a new function added to struct block_device_operations which should be implemented by drivers:
unsigned int (*check_events) (struct gendisk *disk, unsigned int clearing);
This function should check the device for new events and return a mask of any which were found. Two events are currently defined: DISK_EVENT_MEDIA_CHANGE and DISK_EVENT_EJECT_REQUEST, the latter of which is new. The clearing parameter is a mask of events which should be cleared until they happen again. The old media_changed() block device operation still exists, but its use has been deprecated; drivers should be updated to use check_events() instead. Drivers should also, before adding a block device, initialize two new struct gendisk fields:
unsigned int events;
unsigned int async_events;
A mask of all events which can be reported by the device should be stored in events, while async_events should list the events which can be reported without needing to poll the device. A new sysctl knob (block.events_dfl_poll_msecs) tells the kernel how often it should (by default) poll block devices. A value of zero (the default, currently) disables polling entirely. Polling intervals for specific devices can be set in their sysfs directories. If a device says that it can report all events asynchronously, and no polling interval has been explicitly set for it, that device will not be polled at all. Since user space is no longer polling the device with this scheme, it needs a new way to find out when a disk event has happened. These events are now signaled via a uevent, meaning they can be handled via udev or some other utility which is watching those events. Note that any driver which handles asynchronous event reporting must call kobject_uevent_env() itself to send the event to user space. No driver in 2.6.38-rc1 does that; the first developer to add such a call may want to add a helper function to the core block code for that purpose. Since polling is disabled by default, the kernel will behave the way it always has and existing user space applications will work. Once the user space environments have been changed to take advantage of this feature, they can turn on kernel polling and stop opening the devices themselves. That should lead to better power consumption and more reliable operation, which can only be a good thing.
Patches and updates Kernel trees
Core kernel code
Development tools
Device drivers
Filesystems and block I/O
Memory management
Networking
Architecture-specific
Virtualization and containers
Miscellaneous
Page editor: Jonathan Corbet Distributions Debian "Squeeze" release in sightDebian 6.0 ("Squeeze") has now gone into "hard freeze," and the release candidate for the installer was announced on January 12. It would seem that a final release of Debian 6 is finally within sight, carrying a new installer, a completely free kernel, and an experimental Debian flavor based on FreeBSD for x86 and AMD64 systems. ![[Squeeze desktop]](/images/2011/squeeze-desktop-sm.png)
The previous stable Debian release, Debian 5.0 or "Lenny," was announced on February 14, 2009. If two years between releases seems long, it's worth remembering that Debian 3.1 ("Sarge") was just under three years in the making. Debian 4.0 ("Etch") and 5.0 were each in development for about 22 months. Debian releases used to come at a much faster pace, but also didn't have to carry the same number of packages or ports. For instance, Debian 1.1 through 2.2 came out at fairly regular intervals, with roughly a year between releases — Debian had two major releases in 1996, and one every year from 1997 through 2000. Hamm (2.0 in 1998) was the first release to carry two ports, Slink (2.1) brought four ports, and Potato (2.2) came with six. Lenny had official support for twelve processor architectures, but this has been winnowed down a bit in Squeeze. Specifically, it has removed support for Alpha, HP PA-RISC (hppa), and the original ARM port. The ARM EABI (armel) port supersedes ARM in Squeeze, and is described as "the more efficient successor" to ARM. It has quite a bit of overlap in terms of hardware support with the original ARM port — so it's really more of a transition than dropping a port altogether. That leaves support for x86, AMD64, Itanium, MIPS, PowerPC, and SPARC. 6.0 adds a new port that merges the Debian userland packages with the FreeBSD kernel called Debian GNU/kFreeBSD. It is available in 32- and 64-bit version and, though it's considered a "technology preview", it is usable. A glance through the "What's new in Debian GNU/Linux 6.0" guide (which is far more comprehensive than one usually sees for distribution releases) will look very familiar in spots. That is because much of what's new in Debian 6.0 is actually a bit old for other Linux distributions. For instance, the default desktop for 6.0 is GNOME 2.30 (with some 2.32 modules). GNOME 2.30 was released in March of 2010. The release also includes KDE 4.4, which is nearly two releases behind, and the default kernel is 2.6.32.5. Those who'd like slightly less crusty packages will be able turn to the Debian Backports service, which was dubbed an official Debian service in September of 2010. Since Squeeze is not yet released, only Lenny and Etch are currently supported — but packages should start rolling into the backports service shortly after the official Squeeze release. Another addition to 6.0 is the stable-updates repository, which will make updates for non-security updates available before the point updates for Squeeze. Major changes in Squeeze![[Squeeze installer]](/images/2011/squeeze-installer-sm.png)
Aside from the usual package updates, Squeeze does come with a number of changes over the previous releases. Though many users simply ride Artwork is not the only improvement to the installer, of course. The new installer now automatically installs packages which are marked as recommended as well as those marked required. The installer now has support for Ext4 and Btrfs, but the default remains Ext3, most likely due to performance problems with dpkg on Ext4 that were still being worked out late in the development cycle. ZFS is an option for installs of Debian kFreeBSD. The actual practical experience of using the installer is little changed — depending on one's point of view, the installer either offers much more flexibility or much more complexity. It is worth noting that the Debian crew could at least shave a few steps off the installation by allowing users to enter related details during a single step. It seems unnecessary, for example, to have a single screen to enter a user's full name, then a screen for username, then for the user's password. Another major change in Squeeze is the removal of non-free firmware from the kernel. For free software purists, this is a win. For those who are slightly less stringent about maintaining a 100% free system, it's a bit more of a mixed bag. I tested Debian Squeeze on three machines, two desktops and a laptop, and it had no problem with the desktop systems' hardware. However, it didn't like a USB Wi-Fi adapter that works perfectly with Ubuntu and openSUSE. It also had some problems with Wi-Fi on the test laptop — which is not terribly surprising. Users with hardware that requires non-free firmware are not out of luck. The Debian project is providing tarballs of non-free firmware and CD images of the net install CD that include non-free firmware. Note that Debian provides several choices of install ISOs — a business card CD that will require Net access for the base installation packages, a net install image that includes base packages and network access for everything else, and full CD/DVD sets that include full sets of recommended packages. Note that only the first CD image for an architecture is necessary — the remainder carry packages that one might wish to add after an install. If you're planning an eclectic install away from network access, a full 56 ISO images are available for the AMD64 architecture alone. Compare this to the first official Debian release, which fit on one CD-ROM and contained a whopping 474 packages. The project also has a few new "Pure Blends" being introduced with Squeeze. Formerly known as Custom Debian Distributions, the Pure Blends are collections of the more than 22,000 Debian packages for specific audiences. Squeeze adds two new Pure Blends to the mix: Debian GIS for Geographical Information Systems (GIS) applications and users, and Debichem for users of chemical applications in Debian. On January 18 Neil McGovern announced that a second release candidate is expected by the weekend of January 22nd, with a target date for the final release the weekend of February 5th. Any non-critical fixes that don't make it by this deadline will be held for Debian 6.0.1. While I tend to change distributions frequently, I've grown used to distributions like Linux Mint, openSUSE, and Ubuntu which attempt to provide a more user friendly experience than Debian. The time I've spent recently running Debian has been by turns refreshing and frustrating. Debian gives the user much more control over configuring a system, and is much closer to upstream projects like GNOME than, say, Ubuntu. It's also a bit frustrating in the sense that projects like Mint do take the time to do things like tweak font settings so they're not so glaringly ugly. There's also the matter of the default application set — which need a bit more tweaking for this user in Debian than in Mint, Fedora, or Ubuntu. Applications like Dropbox may not have native packages for Squeeze (the Debian packages for Dropbox do not work with Squeeze) so odds are a few packages will require building from source. Others, like Gwibber, are simply hopelessly outdated. Aside from minor frustrations, though, Debian 6.0 looks to be a solid release. It should be — all of its components have had ample time to settle. For users who want a very stable and basic distribution, this is as good as it gets. Of course, many of Debian's users ride the testing or unstable releases and pay little attention to stable releases. This user, for instance, intends to switch to unstable almost immediately. But Debian's stable releases do have their place — namely, anywhere one would like to set up and then forget (modulo running occasional security updates) a desktop or server.
Brief items Distribution quotes of the week
You're right. No Debian developer is involved in large institutions or
corporations where hundreds of such servers are in use. All Debian
developers are kids playing on their parents' computer to build a
distro, during hacking nights, instead of doing their home work and
learn at school.
-- Christian Perrier
I'm really happy that you finally found the fundamental problem of this project.
Let me repeat that: In the nearly two year development cycle for Debian 6.0 "Squeeze" the Debian Project handled nearly one hundred and fifty thousand bugs!
-- Alexander
Reichle-Schmehl
Having currently a bit more than 600 000 bugs in total, that means one fourth of all bug reports where dealt with in Debian "Squeeze"!
Red Hat Enterprise Linux 5.6 Now AvailableThe release of the sixth update for Red Hat Enterprise Linux 5 has been announced. Highlights of this release include support for new hardware, virtualization improvements, and updated DNS packages. See the release notes for details.
Distribution News Fedora Fedora board meeting, January 17A summary of the January 17 meeting of the Fedora board has been posted. Fedora strategic goals were discussed with FAmSCo.
openSUSE A note from the openSUSE boardThe openSUSE board has sent out a long note explaining and justifying its decision to evict an individual from the openSUSE community. "First of all, the Board never had to deal with a situation like this one before, where many attempts of mediation and dialogue didn't result in the person who violated the Guiding Principles understanding the concerns and going back into behavior that is socially acceptable (because that's, in essence, what the Guiding Principles are about). Hence we, as a Community, didn't have any precedence to look for, nor any mistakes we did in the past to improve the process upon." The board is trying to maintain the privacy of the person involved - an approach which is not going over entirely well in the subsequent decision. Your editor would request, though, that the board's wish be respected in any discussion on this site.
open-slx end-user platform opensopen-slx has announced the launch of a new community platform for the German-speaking openSUSE community. "Starting in June 2010, a team of German openSUSE community members, open-slx employees and from ubuntuusers.de has worked on the introduction of the new community portal, which consists of three parts: Wiki, Forum and Blog aggregation. In the wiki, more than 2000 articles from ubuntuusers.de's knowledge base have been imported, almost 500 of which serve as the base for the new open-slx platform, the team's goal is to finalize all 2000 articles within this year. The webforum can be used to discuss these articles or get in contact with other openSUSE users. The blog aggregation ("Planet") has a news feed which gives insight into newest developments and articles covering more specific topics."
openSUSE 11.1 has reached end of Novell support - 11.1 Evergreen goes onopenSUSE 11.1 has reached its end of official support. Evergreen is a community maintainance project aimed at extending support for openSUSE releases.
Ubuntu family Ubuntu Server Survey 2011 -- How do You Ubuntu?The Ubuntu Server community and Canonical would like to know how Ubuntu users are using Ubuntu Server Edition, and in what kinds of organizations. "The anonymous survey takes 10 to 20 minutes to complete and is open to anyone deploying Linux servers today, whether or not they use Ubuntu. The Ubuntu Server Community Team will present the results publicly at the beginning of February."
Newsletters and articles of interest Distribution newsletters
The Arch Way (Linux Journal)Linux Journal has several good reasons to use Arch Linux. "Speaking of documentation, the Arch Wiki is the most comprehensive repository of Linux information out there. Back when I used Ubuntu, I often found the answer to a tough configuration issue on The Arch Wiki. Yes, the Arch Wiki is tailored to Arch, but it can often help you solve problems with other distributions as well."
Linux Mint Debian Edition 10: Rolling Release Nirvana (Linux Magazine)Joe "Zonker" Brockmeier likes LMDE. "One of the questions I usually ask when considering a new Linux distribution is what, if anything, it adds to the community. There's a lot of redundancy out there, and while anybody can make their own Linux distribution it doesn't mean they should. I think LMDE does add something to the mix, especially as Ubuntu evolves further and further away from its Debian heritage."
Lumsden: 2011.Q1 - ForeverwareAlasdair Lumsden has a proposal for an OpenIndiana server release in February. "It's also worth keeping in mind that despite warning users that oi_147 and oi_148 were development releases, people are already using it in production environments, myself included, due to a lack of alternatives. The great news is that it has proven to be exceedingly reliable, and I have no hesitation in recommending it for busy workloads. All we need to do is add security updates and critical bug fixes on top and we'll be in a great position. No small feat I grant you, but we can start off small and work our way up. Now is also an opportune time to do this - our next release will be based on Illumos, which has seen rapid development and will involve some integration pain. Some have called for a stable branch after Illumos is integrated, but it could be many months until we have an Illumos dev build suitable for respinning as a stable branch. That's months of lost opportunity."
Lawyers Can Leave Windows for Linux OS Ubuntu (SEOLawFirm.com)SEOLawFirm.com provides a perspective on switching to Linux that is a little different than those we often see. After a largely accurate, if a bit simplified, "crash course" on open source, the article shows law firms how they can save money by switching from Windows to Ubuntu—and how to use web services to replace some tools like Quicken and case management applications. "You work with the law on a daily basis. There is a good chance you did not write the law, but by the labors and efforts of others in the past, you now have the law to work with. What if you had the ability to change the law depending on the case you were working on? Does your client need additional rights? Give it to them. Does your client need a second chance? Write a process doing just that. Then contribute those changes to the legal community and other lawyers can then build upon those laws and do the same. [...] It would probably be chaotic if laws were open source, but when it comes to software, open source contributions allow technology to rapidly evolve."
Page editor: Rebecca Sobol Development On the maintainability of RubyLucas Nussbaum, longtime maintainer of Debian's Ruby packages, announced that he was stepping back from the role on January 2, making the future of Ruby packaging on the distribution uncertain. He cited a list of reasons leading to the decision on his blog — an assortment of project management, release process, and packaging concerns with Ruby itself that compounded his frustration to the point where he felt quitting was the best option. The public reaction to the announcement has been sympathetic to the hardships of Debian package maintainer-ship, but it has also sparked an interesting debate about the competing needs of developers, end users, and Linux distributors. ManagementNussbaum had several criticisms of the overall management of the Ruby project. The most obvious is the confusing array of development branches. He observes that there are currently six branches of the project in active development: 1.8, 1.8.6, 1.8.7, 1.9.1, 1.9.2, and trunk. While the present status of two of them (1.8.6 and 1.9.1) is clearly marked as important bug-fixes only, the rest are unclear. If 1.9.x is the current development branch, he asks, why are there any active branches in 1.8.x? He also asks about the relationship between 1.9.2 and trunk, and about whether or not there is any API or ABI compatibility between the two. He also observed that there appears to be no "common understanding" of the release expectations of any of the branches. That phrase could be understood in several different ways, but he adds that it "would be fantastic to have something similar to Python Enhancement Proposals" for Ruby, which suggests the lack of a clear roadmap and/or sunset period for older releases is the underlying issue. Ruby's direction is set primarily through mailing list discussions, which constitutes another issue. Nussbaum laments that the primary discussion list, ruby-dev, is written in Japanese, while a second list for English-speakers, ruby-core, misses out on the discussion and gets out-of-step. Ruby was invented in Japan, of course, and most of its core developers are Japanese, but maintaining two lists divides the community, keeps out most of the world, and even risks the possibility of a project fork somewhere down the line. ReleasesReleases themselves constitute another problem for Nussbaum, who observes that 1.8.7 and 1.9.2 releases were both made on December 25, 2010, apparently without any beta testing period or Release Candidates preceding them. On a note similar to the language barrier between the lists, he observes that the Ruby project is in the habit of regularly making its major releases on December 25th, which is the traditional Western date for Christmas, and is a day when a large percentage of offices are closed. The result is that users in Europe, North and South America, and Australia are unavailable to test and provide feedback for several days or weeks. But most importantly, Nussbaum argues that the Ruby platform consists of more than just the standard implementation Ruby interpreter, known as Matz's Ruby Interpreter (MRI). There are an increasing number of alternative interpreters popular with developers, including the Java Virtual Machine-based JRuby, Rubinius, and MacRuby, but there is no clear process for integrating them with the rest of the Ruby ecosystem. Furthermore, the ecosystem includes large-scale projects and libraries such as Ruby on Rails, on which a great deal of real-world Ruby code depends. The development community, Nussbaum says, should create a process to ensure that new releases of MRI work with those "big players" before making a release. Ruby's release process has been criticized by others in the past. In 2008, security holes based on integer overflows were uncovered in the released code, which might have been caught by a public beta testing period, and the fixes pushed out by the project broke the Rails framework. Users, developers, and systems administrators, oh my....The most contentious issue cited by Nussbaum in his blog post is how Ruby and Ruby libraries are packaged for distribution. The official, preferred method for installing Ruby is Ruby Version Manager (RVM), a command-line tool that allows individual users to install multiple self-contained Ruby environments, including interpreters, libraries, and applications. Ruby libraries are generally distributed through RubyGems, an online distribution network that functions much like Perl's CPAN. There is a command-line client, which fetches and installs individual packages in the Ruby-specific gem format. This system works quite well for developers, Nussbaum says, but is
unnecessarily hard on both system administrators and end users. End users,
he says, expect to be able to install a Ruby-based application using the
normal system tools (such as
The Ruby community should work with their target platforms to improve how Ruby is distributed instead of reinventing the wheel. That includes Debian, but also RedHat-based distros, for example. It is likely that it won't be possible to reach a one-size-fits-all situation. But that's real life.
Nussbaum also cites several examples of negative, offensive behavior in public — some of it directed at him (accusing him of intentionally creating "crippled" Ruby packages for Debian), but more directed at others. As for the future, he points out that there are two other Ruby packagers for Debian, who may together decide to continue maintaining the package without him, and he wishes them luck. What is less clear is the future of the Ruby-extras package, which includes most of the Ruby libraries. There appears to be no successor for the maintainer's role for that package. On the plus side, he does say that he is open to returning as a Ruby packager in the future, if the landscape improves. ReactionNussbaum's post spawned quite a bit of discussion, both in comments to his blog and on YCombinator. A few commenters on both sites accused him of "imperialism" because of the list language issue, before it was pointed out that Nussbaum himself is a native French speaker. But by and large, the debate swirled around the clash between the Ruby project's goals and the Debian project's goals. The essential argument seems to be that Ruby, as a language project, should and does seek to make life as easy as possible for Ruby developers. Debian, on the other hand, seeks to package upstream software in a way to make life as easy as possible for end users. Developers will almost always want to run bleeding-edge code, to make use of the latest and greatest language and interpreter features. Consequently, they want to compile and install the Ruby environment from source, and want to have access to specific versions of Ruby libraries — sometimes multiple versions, as concurrent projects may demand — and they want to manage those dependencies in gem packages, not Debian packages. Users, in stark contrast, want installation of Ruby applications to be simple, which demands integration with the main distribution package management system, not use of a specialized Ruby tool. Stuck right in the middle are systems administrators, who get asked to support scores of locally-compiled and installed Ruby environments, and distributions, who get asked to package scores of minor releases for Ruby interpreters and libraries. Commenter Meneer R pointed out that this a common tension when dealing with interpreted languages, because the execution environment and the development environment are the same. Compiled languages always have the relatively easy out of statically linking-in an old library. Meneer R and several others argue that the "Debian philosophy" needs to change; namely that the distribution should accept the Ruby community's packaging and otherwise get out-of-the-way. Developers, he said, target specific versions of the Ruby platform, not specific releases of Debian, and trying to shoehorn the two into one release system only complicates things for Ruby developers using Debian. Several others pointed out that Ruby and Ruby libraries have to work across all operating systems, and that Debian's packaging techniques simply don't apply to Windows, OS X, or other platforms. Others from the Debian community take the opposite stance. Commenter Np237 argued that allowing multiple versions of one library to co-exist in the system is itself the problem, because:
You cannot turn a system into a production without stable versions for each
of its components, and you cannot expect long-term maintenance of each of
those versions. As such, you can prototype your applications with a
language like Ruby, but you cannot put them into production in decent
conditions.
Debian developer Gunnar Wolf posted a response on his own blog, generally showing support for Nussbaum. Regarding the conflict between Ruby packages and Debian packages, he observed that maintaining multiple versions of the same library on a system in order to support multiple incompatible applications has inherent risks, such as bug fixes not getting back-ported, and security and stability problems developing over time. The need to keep multiple versions of an interpreter or library installed generally means there are big changes between different releases, he said, and that means that the library is immature and not ready for long-term API commitment — which in turn makes it a bad bet for the developer. Wolf also weighed in on one of the repeated claims in support of Ruby's packaging system: that because the language is primarily used by web developers, "end users" never need to install it anyway. Web frameworks like Rails unduly influence the discussion, he said, because the core packaging issues exist for non-web developers, too. In addition, saying that web applications live by different rules ignores the needs of systems administrators, he said. in Debian (and in other distributions, surely) we try to make sysadmin's lives simpler - I have (again, talking out of personal experience) installed several webapps (and system tools, and whatnot) for which I never followed any instructions besides aptitude install foo - Using different languages, frameworks, and so on. Can I troubleshoot their installs? Probably, as there is a common logic for how the distribution I have chosen and specialized in works. Can I find causes for bugs in them? Possibly, although there are some languages and frameworks I dare not touch. Can I get help on getting them out of a tight spot? Surely, as there is a central bug tracking system for my distribution - And one of the maintainer's tasks is to separate the problems related to the distribution (packaging, installing, simple user questions and misconceptions) to those derived from real bugs upstream.
Nussbaum is not the only packager to grapple with the difficulty of fitting Ruby and RubyGems into a distribution's package management system. Fedora has a Ruby special interest group (Ruby SIG) which tackled many of the same issues one year ago. Fedora manages multiple concurrent Ruby releases (1.85, 1.8.6, 1.9.1, and 1.9.2) using the alternatives system, and has packaging guidelines that attempt to wrap gems inside RPM files. Nussbaum commented on his blog that alternatives was not the solution for Debian, because by definition it is designed for tools that can be used interchangeably, which different releases of Ruby interpreters cannot. Despite all of the arguments on both sides of the packaging issue, the discussion did remain completely civil. Nussbaum updated his post to point to two positive steps taken by the Ruby community: a JRuby maintainer outlining his project's vision and goals as relates to the larger Ruby development ecosystem, and a message from a core Ruby developer inviting more discussion on the English-speaking mailing list, and clarifying that there is no intention of keeping the two subscriber bases separate. Paul Brannan, one of the original authors of RubyGems, even joined in the discussion, explaining that gem packages were designed with the goal of reducing the strain on administrators and users (including the ability to package a gem inside of an RPM or Debian package). He also offered mea culpas for those design decisions that, with hindsight, did not work out so well. Language maturityWolf made an interesting aside in blog post, noting along the way that Ruby is no longer a "new language." That sentiment is clearly present in Nussbaum's retirement announcement, even if he did not explicitly say it. With a young language, people (developers and systems administrators) expect things to change rapidly and sometimes without warning. But Ruby is fifteen years old now. That may be young compared to C and FORTRAN, but it is certainly not a flash-in-the-plan. Issues like those Nussbaum raised are difficult for projects to deal with. It is awkward to change the existing culture and tell developers that old language features are deprecated. It is particularly awkward to suggest that a project has grown so much that the main mailing list needs to shift to the lingua franca of distributed development, rather than the original authors' language. But like it or not, even if the two mailing lists do not separate developers into first- and second-class community members, they do split the community into two circles that don't work together. Yet not all of the Nussbaum's issues are that difficult; there are things that Ruby can do to make life easier downstream. There is still no formal language specification, for example. Standardizing that would help unify the competing interpreters. A more formal process for language extension would ease a lot of minds, and a stricter release schedule with targeted betas would make the community feel more involved. Those are goals any open source project would be proud of, all the more so for the language designated as "A programmer's best friend".
Brief items Quotes of the week
Rebase is not (yet) a first class command in Mercurial, so we get
fewer misguided users of it than Git, but even in my own work I
occasionally have to stop and think: is everything I'm about to
rebase local only or am I going to end up with a bunch of duplicate
csets on the server?
-- Matt Mackall
But wht did you expect? The original authors of the code are long
gone and maintenance is done by newcomers who are patching the code
bit by bit. What you get from such a development model is pretty
predictable: ~1 billion years old spaghetti DNA that no-one truly
understands.
-- Ingo Molnar takes the long view
Amarok 2.4 releasedVersion 2.4 of the Amarok media player is available. "This release brings significant performance, usability, and stability improvements: Most of the bugs involving the context pane have been squished, a completely rewritten collection scanner is presented (hint: it's the thing that reads your music files' tags) -- Amarok now has the ability to write your statistics and covers directly to the audio files to keep the information across different devices and should detect compilations a lot better. At long last, we also have an option to hide the menu bar again." Also in this release are new device support, transcoding of tracks, and more.
A mailing list for Android audio developersA new mailing list has been created for developers interested in audio issues on the Android platform. "This list is meant to be a place to discuss about audio development, in the context of music and sound applications, but also games and other apps which use audio on Android."
FFmpeg turmoilA group of FFmpeg developers has announced that the project has a new set of maintainers - news which came as a surprise to existing maintainer Michael Niedermayer. Developer Diego Biurrun has posted an explanation of how the coup came to be, but it's clear that not everybody is satisfied. "The discontent reached the point where a fork was being contemplated and then planned, but it turned out that the momentum had soared way past critical mass and turned into a tidal wave of revolution. The focus moved from forking to avoiding a fork if possible. Since git was being set up on videolan.org, setting up an alternative git tree on mphq was the natural choice. With development moving to videolan.org and such a large group of developers already part of the revolution keeping the infrastructure was the logical consequence." (Thanks to Mattias Mattsson).
MDP release 3.0Version 3.0 of the Modular Toolkit for Data Processing is out. "MDP is a Python library of widely used data processing algorithms that can be combined according to a pipeline analogy to build more complex data processing software." Changes include Python 3 support, some new extensions, new algorithms, an improved tutorial, and a switch to the BSD license.
Parrot 3.0.0 ReleasedVersion 3.0.0 of the Parrot virtual machine is available. "On behalf of the Parrot team and an enthusiastic but undiscriminating dachshund that followed me home last week, I'm proud to announce Parrot 3.0.0, also known as "Beef Stew", or at the insistence of a shadowy government organization, "Snowflake". Parrot (http://parrot.org) is a virtual machine that dreams about running all dynamic languages everywhere, even the one you're think about right now. Parrot has big plans, even if needs a haircut and sometimes goes outside with its shoes untied." This version includes improvements in the core code and the documentation, new language support, and better test coverage.
Xfce 4.8 releasedVersion 4.8 of the Xfce desktop has been announced. "With Xfce 4.8 our users will be able to browse remote shares using a variety of protocols (SFTP, SMB, FTP and many more). The window clutter has been reduced by merging all file progress dialogs into a single one. Our panel application has been rewritten, thereby improving positioning, transparency, item and launcher management. It also introduces a new menu plugin to view directories. Its plugin framework remains compatible with 4.6 plugins." LWN previewed this release back in December.
Newsletters and articles Development newsletters from the last week
More about the Chrome HTML Video Codec Change (The Chromium Blog)Google's Chromium blog has an update on the decision to remove H.264 support from the Chrome/Chromium browser. "We acknowledge that H.264 has broader support in the publisher, developer, and hardware community today (though support across the ecosystem for WebM is growing rapidly). However, as stated above, there will not be agreement to make it the baseline in the HTML video standard due to its licensing requirements. To use and distribute H.264, browser and OS vendors, hardware manufacturers, and publishers who charge for content must pay significant royalties—with no guarantee the fees won't increase in the future. To companies like Google, the license fees may not be material, but to the next great video startup and those in emerging markets these fees stifle innovation."
Page editor: Jonathan Corbet Announcements Brief items The Document Foundation joins OpenDoc SocietyThe Document Foundation has become an organizational member of the OpenDoc Society. "OpenDoc Society brings together individuals and organizations with a stake or interest in the openness and future of documents, to learn from each other and share knowledge and best practices about core technologies, available tools, policy issues, transition strategies, legal aspects and of course the latest innovations."
New Report about Open Data for an Open SocietyMarco Fioretti has announced the availability of the Open Data, Open Society report: "The report discusses the current and potential role, in a truly open society, of raw Public Sector Information (PSI) that is really open, that is fully accessible and reusable by everybody. The general characteristics of PSI and the conclusions are based on previous studies and on the analysis of current examples both from the European Union and the rest of the world. [...] The "Open Data, Open Society" project will continue with an online survey that will attempt to assess how many types of raw PSI are already released, in which formats and under which licenses, by the city and regional administration of the EU-15 countries. A final report will summarize and comment the results of the survey."
FSF: supporting Google's push for WebMThe Free Software Foundation has sent out a release stating its support for the WebM project. "We want the world to know that we also support WebM: with its developer-friendly patent license and free software reference implementation, it's a good choice to help ensure the Web fulfills its promise of providing a free way for the world to communicate."
Articles of interest Google's dropping H.264 from Chrome a step backward for openness (ars technica)Ars technica has a lengthy criticism of the removal of H.264 support from the Chrome browser. "Google is now building a community around WebM (similar to that around Theora), but it hasn't taken any steps to submit WebM to ISO, ITU, or SMPTE for formal open standardization. The company is preferring to keep it under its own sole control. For Google to claim that it is moving to 'open codecs' is quite absurd: H.264 is very much an open codec. WebM is not."
PS3 jailbreak prompts restraining order from Sony (CNet)CNet covers Sony's DMCA suit against George Hotz, who figured out how to jailbreak the PS3 game console. "Sony is seeking the impoundment within the next 10 days of all circumvention technology that Hotz and his team employ. It also wants all mention of the circumvention removed from the Web, and has asked the court to force Hotz to ignore calls for help from others attempting to install packages on the console."
CES 2011: A Tale of an Android Onslaught (LinuxPlanet)LinuxPlanet covers the array of Android devices seen at this year's Consumer Electronics Show. "While it seems every Consumer Electronics Show (CES) in recent memory has some theme that never really panned out (see 3-D TV), you might want to rethink the trend when it comes to Android. Google's Android attack was in full force at this year's edition with a vengeance. From smart phones to tablet devices of all sizes to Google TV, you couldn't travel the exhibit floor very far without bumping into something Android. For some companies, like Motorola, Android has fueled impressive comebacks. For others, like Sony, it's new territory."
Firefox 4 Beta 9 Gives Short Shrift to Linux Users (PCWorld)PCWorld looks at the lack of hardware acceleration in the Linux version of the latest Firefox beta. ""We tried enabling OpenGL on Linux, and discovered that most Linux drivers are so disastrously buggy (think 'crash the X server at the drop of a hat, and paint incorrectly the rest of the time' buggy) that we had to disable it for now," wrote Mozilla developer Boris Zbarsky Friday in a comment on the Mozilla Hacks developer blog. "Heck, we're even disabling WebGL for most Linux drivers, last I checked.""
Phipps: Apple and Google and ODFSimon Phipps notes that in Latvia (and other European countries) governments must accept documents in ODF format, and bemoans the lack of ODF support from Apple and Google. "So I remain surprised that neither Apple nor Google are taking ODF support seriously. Apple still don't support ODF in their applications (despite it being available in their TextEdit gadget on Mac OS X) or the iPhone or iPad, and the ODF support in Google Docs is so weak that documents I try to upload from LibreOffice are routinely rejected in ODF and yet accepted if I save the identical document in .doc format. It's ironic that the best proprietary ODF support right now is from Microsoft." (Thanks to Davide Del Vento)
Upstreaming your code - a primerArmijn Hemel and Ralf Baechle have written an introduction to upstreaming code to the kernel. It's aimed at chip vendors, but should be applicable to other aspiring kernel hackers. "It is likely that the first submission to a Linux kernel developer of your code will not be accepted. This could be for various reasons: the code does not follow the coding style, code quality might not be as high as is expected, and so on. Do not see this as a rejection of your code, but as a first step for successful inclusion into the mainline kernel. Sometimes developers might react very terse, or seemingly irritated. Once again, do not see this as a rejection, but as a sign that you need to put in a little bit more effort."
New Books Eloquent JavaScript--New from No Starch PressNo Starch Press has released "Eloquent JavaScript", by Marijn Haverbeke.
Gimp 2.6 for Photographers--New from Rocky NookRocky Nook has released "GIMP 2.6 for Photographers", by Klaus Goelker.
Programming Python, 4th Edition--New from O'ReillyO'Reilly Media has released "Programming Python, Fourth Edition", by Mark Lutz.
Software Testing Foundations, Third Edition--New from Rocky NookRocky Nook has released "Software Testing Foundations, Third Edition", by Andreas Spillner, Tilo Linz and Hans Schaefer.
Resources Linux Foundation Monthly Newsletter: January 2011The January edition of the Linux Foundation newsletter covers the 2011 Event Schedule, a new fellow, five new members, individual membership drive winners announced, Linux Powering Your Cell Phone Network: A Case Study, and more.
Contests and Awards Voting for the 2010 LinuxQuestions.org Members Choice Awards is Now OpenLinuxQuestions.org has announced that voting for the 2010 LinuxQuestions.org Members Choice Awards is now open. "Awards will be given out in 31 categories this year, including Server Distribution of the Year, Desktop Distribution of the Year, Mobile Distribution of the Year, Browser of the Year, NoSQL Database of the Year and Open Source Web Framework of the Year. The polls will close on February 17th."
Calls for Presentations Linux Plumbers' Conference 2011: Call for TracksThe Linux Plumbers' Conference will take place in Santa Rosa, CA, September 7-9, 2011 and the program committee is looking for track proposals. "The ideal track proposal is focused enough to allow progress and attract the right participants, but also spans several components of the Linux technology stack and a set of problems that can specifically benefit from face to face discussion between the various teams involved."
NLUUG Spring Conference 2011NLUUG will be holding their spring conference May 12, 2011. "We are looking for presentations to complete the program for our Spring Conference of 2011. We would like to receive abstracts on technical subjects (40 minutes time slots) and practical experiences (20 minutes) related to the theme "Open is Efficient". With these 20 minutes shorter presentations we hope to achieve that more people contribute to the conference." The proposal deadline is February 12.
Texas Linux Fest 2011 Call for Papers now openTexas Linux Fest 2011 takes place April 2, in Austin, Texas. The call for papers is open until February 14.
Upcoming Events The first set of FOSDEM interviewsFOSDEM (Brussels, February 5-6) has posted the first set of its traditional speaker interviews; featured this time are a certain LWN editor, Eben Moglen, James Turnbull, Boudewijn Rempt, and Gratien D'haese. Quoting Eben: "Currently, governments are very eager to control the Net, which is a disaster for freedom. Industry is responding to the mobile computing revolution by trying to lock down the devices people use in their daily lives. And powerful industrial players who make money by surveilling people and data-mining the resulting information are encouraging more people to give up their basic privacy for small, if not meaningless, benefits. All are large threats to freedom, and if our strategies worked against only one or two such threats, we could still fail."
LCA 2011 will be happeningThe linux.conf.au 2011 team has announced that the event will happen as scheduled, despite the flooding in Brisbane which has left them with some problems to work out still. "At this stage the venue location is still to be confirmed as are a number of social events, though there are some backup plans in place."
LCA moves to a new venueThe venue for linux.conf.au, starting January 24 in Brisbane, has been moved to the Kelvin Grove campus of the Queensland University of Technology - a shift of about 4km. The current recommendation seems to be that attendees should keep their current hotel reservations and use the bus service to get to the venue. "We do ask you to be patient as we work through all the logistics - but the conference will go ahead and no miniconfs or conference streams will be canceled." One only hopes that the organizers have planned a big party after the event; they will certainly have earned it.
Events: January 27, 2011 to March 28, 2011The following event listing is taken from the LWN.net Calendar.
If your event does not appear here, please tell us about it.
Page editor: Rebecca Sobol
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![[Chrome home screen]](/images/2011/chromehome-sm.png)