A LibreOffice update

The LibreOffice office suite is a large program with an active, growing development community; it can be hard to keep up with what is happening with it. Fortunately, LibreOffice developer Michael Meeks showed up at LinuxCon Europe to update the audience on what LibreOffice has been up to and what can be expected in the near future. The picture that emerged showed a busy community rapidly adding features and fixing bugs while trying to ease the way for new developers to join the project.

Easier development

Much of the work in LibreOffice, Michael said, continues to be oriented toward making the code base easier to work with. The community wants to grow, and making the code easy to work with seems like a good way to make progress toward that goal. Recently we have seen the completion of an effort to replace the build system which, he said, was "traditionally appalling." All it took to make things better was "three years of hard work."

Another attempt to make things friendly for new developers is the easy hacks page. Experienced developers routinely deny themselves the pleasure of making an easy improvement, choosing instead to add a description of the problem to this page. That allows new developers to come in and find a straightforward project that lets them immediately make things better for LibreOffice users.

A unique development tool used in the LibreOffice community is the "BiBisection" tool. A lot of changes can go in during a six-month development cycle; by the time somebody finds a bug, it can be difficult to determine which change caused it. Such problems call for bisection, but asking users to go through the process of building LibreOffice a dozen or more times to isolate a buggy commit is a hard sell. So, instead, the BiBisection system automatically creates a binary build every ten commits or so, then checks the resulting binary in as a git commit. Git, it seems, is good at compressing the result, to the point the repository holding binaries for the full LibreOffice development history fits in a mere 3GB of space. Now users can quickly bisect a problem by checking out the binaries and running them directly.

Releases, past and future

The LibreOffice 4.0 release came out in January 2013, with a lot of new stuff. Some of the headline features Michael mentioned include range comments (allowing a comment to be attached to a range of text), import of files in the RTF format, "ink annotation" support (drawing over a document, essentially), support for the CMIS protocol (allowing interoperability with applications like SharePoint, Alfresco, Nuexo, and more), Microsoft Publisher import, Visio import, support for Firefox themes, and an implementation of the Logo language.

LibreOffice 4.1 followed in May. A significant new feature in this release is font embedding, where the fonts used in a document can be embedded within the document itself. That improves interoperability between office suites, and, even, between systems; one need no longer fear the mess that results from the replacement of one's fonts when opening a document on a new machine. There is also support for the easy creation of slide decks from photo albums, 90° rotation of images in Writer (Draw has long since had arbitrary rotation), GStreamer 1.0 support, an experimental side bar taken from Apache OpenOffice, and about 3,000 bug fixes, 400 of which came from OpenOffice. With those fixes, there are "only" 5,000 open bug reports for LibreOffice — many of those, he said, are likely to be duplicates.

The 4.2 release can be expected in January 2014. It should feature an improved "start center," the screen that appears when the application first starts up. There will be a new math panel for improved entry of mathematical expressions, a much improved Android remote for running presentations, an iOS remote control app, and integration with Google's GDrive service.

Another big change in this release is a "data structure drug bust" in the Calc spreadsheet application. Calc, it seems, suffers from "rampant object orientation," with the concept of a cell used as the base class for everything. That results in cells being used for everything, including undo, change tracking, and so on; everything has to know about cells. The result is extreme memory use, slow computation, and difficult code. The new code gets rid of the cell base class and, whenever possible, stores data contiguously in arrays. That lets things run faster with a lot less memory; it also opens up possibilities for using the graphics processor (GPU) for computations in the future.

4.2 will have a much faster XML parser. It is fashionable to be against XML, Michael said; XML is seen as being slow to parse, but that doesn't have to be. The LibreOffice developers have sped things up; they have also split XML parsing into a separate thread that can execute in parallel with ordinary processing. There will be better layout integration, with many of the dialogs created with Glade, even though LibreOffice is not based on GTK. The result should be more consistent dialogs and, someday, the ability to use native toolkit dialogs.

This release will also have a fast Firebird embedded database. It replaces HSQLDB, which, he said, is a living proof that Java code can be difficult to integrate with a C++ application. The configuration code will be cleaned up. Even people who like to tweak their applications heavily might think that the 25,000 knobs provided by LibreOffice are a bit much. Configuration options are not being removed — Michael was clear that the project doesn't believe in that — but many of them will be hidden in an "expert" menu. StarBasic will support code completion. And there will be better support for the display of character borders.

Further in the future, we should see a much-improved version of LibreOffice running on Android. There is also a "CloudOffice" prototype HTML5 application, but there is not much additional work being done in that area yet. Collaborative editing is seeing rather more work, with a new mechanism (based on undo tracking) to help keep documents synchronized. Work is progressing on "hybrid PDF files" where the OpenDocument file is embedded within a PDF generated from it. That is a path toward the creation of editable PDF files. Finally, "flattened" files run the normal file format through a pretty printer, allowing them to be inspected (or changed) in an ordinary text editor.

In summary, Michael said, the LibreOffice community is thriving and growing. There are not only a lot of contributors; there are also a number of companies participating; he suggested that projects supported primarily by one company are somewhat risky to be a part of. But, he said, more contributors are always welcome; the audience was invited to jump in and start sending patches.

[Your editor would like to thank the Linux Foundation for travel support to attend LinuxCon.]

Comments (8 posted)

A kernel panel convenes in Edinburgh

One of the most popular sessions at any LinuxCon is the kernel panel, Jim Zemlin, executive director of the Linux Foundation, said in his introduction. Another lively panel made its appearance on October 22 in Edinburgh, Scotland as part of LinuxCon Europe. The panel was ostensibly focused on "core and embedded", but predictably ranged further than that.

The panel was moderated by LWN executive editor Jonathan Corbet and had four panelists who are developers from all over the kernel. Each panelist introduced himself, with Greg Kroah-Hartman starting things off. He, of course, works for the Linux Foundation on a number of different kernel areas including USB and the driver core as well as both the stable and staging trees, which he called "the two extremes in the kernel". Will Deacon works for ARM on very low-level architecture-specific code like memory barriers and TLB handling—"stuff that most people hate". He also co-maintains the 64-bit ARM (aarch64) support in the kernel.

Sebastian Hesselbarth was invited as a "hobbyist" attendee to the Kernel Summit. He started working on the mainline kernel about a year ago. Peter Zijlstra from Red Hat rounded things out. He rattled off a number of things that he co-maintains with Ingo Molnar including the scheduler, perf, and lockdep. He also works on the memory management subsystem, IRQs, timers, and various low-level architectures "when I have to". "I get around a bit", he said.

Corbet noted that there is a huge amount of code coming in now from the mobile and embedded world mostly in the ARM tree, but that when you look at just the contributions to the core kernel (scheduler, kernel and mm directories), it is companies like Red Hat and IBM. He asked "are we able to work together and make one kernel that works for everybody?" Zijlstra said that it was a "fun exercise" to make one kernel that scaled from tiny little machines to the largest ones out there.

Power

That led Corbet to direct a question to Zijlstra about some frustrations he had heard from the ARM community about getting scheduler patches reviewed. Zijlstra said that he wants to avoid having a different scheduler for every architecture. Power-awareness is important to ARM, but it will be (or already is) important for others as well. There are several subsystems that currently work together to handle power management (CPU idle, CPU frequency, and the scheduler), but he wants to see a "coherent framework" proposed that will help solve all the problems and not just "random tweaks here and there". There is a mini-summit for power-aware scheduling later in the week, he said, where he hoped that some of the issues could be resolved.

But, as Deacon pointed out, the problem is not just for ARM and mobile, the server community is interested in better power performance as well. Zijlstra agreed but noted that it is "difficult to get a straight answer" from that community about its needs. The hardware vendors really don't want to talk to each other, he said. Deacon pointed out that it might just be that they aren't used to talking to each other and that they see it as much easier to get things done when they don't have to. Kroah-Hartman said that the reason hardware vendors start out with drivers is because they are self-contained and don't require coordination with other vendors. It is harder to change the core kernel, he said, as it should be, but companies eventually get there. Neither Intel nor IBM worked on the core a while back, but now that's a large part of what they do. Moving into the core is simply a migration that many companies eventually make.

The ARM big.LITTLE architecture is different than anything we have seen before, moving us from symmetric multiprocessing (SMP) to asymmetric, heterogeneous multiprocessing, Corbet said. He asked how that would impact the scheduler. Zijlstra was upbeat about supporting that model, saying that the scheduler already deals with SMP systems that have asymmetric loads due to realtime or interrupt processing. Big.LITTLE is different than that, certainly, but there are enough similarities that some of the code which tries to balance "fairness" by tracking the realtime and interrupt usage on the CPUs, which could map to more and less powerful processors

Kroah-Hartman isn't sure that the idea behind big.LITTLE will really pan out. There are other system components (memory, buses, etc.) that take more power and still need to stay powered up even when only a "little" CPU is running, so it's not clear that just turning off the more power-expensive processors is enough of a win. It may be an "experiment that is failing", he said. Deacon had not heard that, he said. Companies are putting big.LITTLE into products, so they believe it will bring power savings. We should see proof one way or another before too long, he said.

Large, small, and tiny

There have long been complaints that the kernel is enterprise-focused, though they have died out some over the years; does that make it hard for the embedded side, Corbet asked. Deacon said that there is a history of that in the kernel, and that many of the maintainers came about during that era, but that he, at least, didn't see a real problem with that. As the kernel evolves, so do the maintainers and the focus; it all balances out. Kroah-Hartman was quick to note that the enterprise systems of five or ten years ago were now in many people's pockets, so that those differences flatten out over time. In addition, the choices rarely come down to making one better completely at the expense of the other, Deacon said, so people air their concerns and a compromise is found.

Going the other direction, Corbet noted that Tim Bird has been concerned that Linux is leaving tiny systems behind. Most embedded devices these days are fairly capable systems with fewer memory and other constraints from years past, but ignoring the really small systems may leave Linux out of the "internet of things". He asked if the panelists were concerned about that.

Zijlstra didn't think it was a real problem as there are patches to shrink the kernel that are still accepted. In addition, a recent suggestion to remove uniprocessor (UP) support—essentially turning it into an SMP system with one processor, though with all of the extra SMP data structure overhead—was rejected as there are still plenty of single-core processors out there and Intel recently introduced more with the Quark UP line. Deacon noted that support for a tiny ARM core ("M class") was recently merged as well. As long as people are still running tiny systems and are willing to put in the work, support for the super low-end should be fairly easy to maintain, but once the use of UP systems goes away, for example, "that code will bitrot really fast", Zijlstra said. "I can't wait for that day, but I'm afraid I will have to wait a long time."

Community

Somewhat controversially, this year's Kernel Summit committee set aside summit slots for hobbyists, Corbet said, so he wanted to explore that a bit. He asked the audience how many had contributed a patch to the kernel and somewhere around one-third raised their hands; perhaps half of those indicated they had done it on their own, not as part of their job. On stage, Hesselbarth served as a representative from the hobbyist community; Corbet started by asking him why he contributed to the kernel on his own when so many around him were being paid to do so.

Hesselbarth said that he had a personal interest in tiny devices, but that he probably was not a prototypical Linux hobbyist. He is a hardware engineer with an interest in building systems-on-chip (SoCs) for devices. He writes Linux drivers as part of his job, which led him to work on the ARM SoC code on the side.

Corbet asked: "Is our community friendly?" He wondered if Hesselbarth was able to get the answers he needed on mailing lists, for example. Hesselbarth said that it depends on how you approach the mailing lists. If you start out "kindly", most on the list will accept you, will look at your code, and will correct you if you are wrong, even if they have done that many times for others in the past. Zijlstra had some advice for anyone looking for assistance: ask specific questions, and always on the mailing list. Specific questions are easier to answer and emailing him privately puts a message on "a very short path to /dev/null".

Corbet brought up the steady decline in the percentage of hobbyist developer patches over the years. Zijlstra and Deacon both thought that perhaps the absolute numbers weren't really declining that much, it was that the number of paid contributors has risen so much. Kroah-Hartman said that people who show any ability to get patches into the kernel immediately get job offers. He knows of at least five people who were doing style patches to staging drivers in their spare time, got offers, and now do kernel development full time. It is quite difficult to find kernel developers, so companies watch the lists and ask other kernel developers for leads on new employees, he said.

When everyone gets hired, though, that causes a problem as sometimes they can no longer maintain the code they were working on. Or, as Zijlstra pointed out, they get reassigned to a different project and maintenance falters. Corbet mentioned that he sat in on a Matthew Garrett talk earlier in the conference where he talked about some buggy code he had merged before getting reassigned and just kind of left it all behind.

Relatedly, Kroah-Hartman recently backed out a change when the email to the developer bounced. It is a problem, Kroah-Hartman said, because a maintainer needs to trust that someone contributing will be around to fix the code, "or I have to". He noted a "big, hairy" network change that went in a few years ago, where, literally, email to the developer started bouncing the day after it was merged. It took many months to unwind those changes, which is part of what makes it difficult to get large changes into the network subsystem today.

Device tree woes

Non-discoverable hardware and the ARM device tree solution for that was next up. Once upon a time, the devices attached to a system could be determined at run time, but that is often not the case anymore, so some kind of external description of those devices is required. For ARM, device trees are being used to do that, but there are some problems with the consistency in how those device trees are specified (i.e. the bindings). A bigger issue is whether device tree bindings constitute an unbreakable kernel ABI, which means that even more care is required before merging any device tree support. Corbet asked: "Is device tree the right answer, what have we done wrong, and how can we fix that?"

Deacon said that a full day of the ARM mini-summit (concurrent with the conference) is devoted to trying to work out those problems. Device tree has been an improvement over the older mechanisms, and it has allowed a lot more code and drivers to go into the kernel. The ABI question is a "religious" one and there is a fundamental disagreement between those who think it is an unstable interface that shouldn't be used in any products and those who think it is an unbreakable ABI that is fixed in stone. He was hopeful that some of that could be ironed out in the mini-summit.

Hesselbarth found that device tree makes things harder for hobbyists because they have to consider all of the possible ways that someone might hook up a particular IP block when designing the device tree for it. There is something of a trend to cut the static configuration data right out of the drivers and essentially paste it into a device tree, Deacon said, which is not the right way to approach creating a device tree entry. In addition, Kroah-Hartman said that it is hard for driver maintainers to decide whether to merge a driver because they are unsure if the device tree support is "correct"—or even what "correct" looks like.

Kernel ARM maintainer Russell King recently put out a detailed critique of device tree that Corbet said he hadn't had a chance to digest yet, but clearly concerned the ability of device tree to describe the complexities of today's devices. They are no longer just a single device, often, but a collection of devices connected by buses. "Is there a fundamental flaw there?", he asked.

Kroah-Hartman likened the problem to that which the Video4Linux (V4L) developers have been grappling with for years. There needs to be a way to describe devices and how they interconnect, which is a complex problem, but it has to be described somewhere. "What's wrong with making it all discoverable?", Zijlstra asked. That's what kernel developers want, but it's difficult to convince the hardware makers, Kroah-Hartman said. Zijlstra's suggestion to "take them out back" and make them "do what we want" was greeted with laughter, but lots of nodding heads as well.

Corbet noted that there was a trend toward getting the knowledge from user space, both in the V4L world with the media controller interface and in the ION memory allocator; are we getting to the point where we just can't solve those problems in the kernel, he asked. Deacon said that he doesn't think it is unsolvable, but it isn't fun to think about how to solve and it is easier to think it is someone else's problem. He said that not only is hardware configuration being pushed into user space, it is also being pushed into the firmware.

The hardware today is "plug and play" modules, Kroah-Hartman said, where hardware makers buy IP blocks from multiple vendors and hook them all up in different ways. Linux has drivers for each of the pieces, but not all the different ways they might be hooked up and communicate with each other. The hardware folks have solved the problem, but it needs to be handled on the kernel side. He returned to the discoverable hardware idea, noting that simple, static tables that could be read by the kernel would help solve the problem. Zijlstra suggested that perhaps the tools used by the hardware designers could be changed to make it easier to provide the kernel what it needed. Hesselbarth seemed a bit skeptical that Synopsys and other tool vendors would be all that interested in helping out.

Security

The conversation turned to security and whether the kernel developers were doing enough to deliver a secure kernel. Kroah-Hartman said that the community can "always do better", but that the bugs reported to the kernel security email address are fixed "as soon as we possibly can". He noted that various static code analysis tools are being run on the kernel and are finding lots of bugs that get fixed right away. Some of the creators of those tools, Julia Lawall (Coccinelle) and Dan Carpenter (Smatch) for example, have fixed more security bugs than anyone, he said. It is an area that is ripe for more research and the community is always open to better ways.

Some 4000 patches were made to the 3.0 kernel in the two years it has been maintained as a stable kernel (it reached end of life with 3.0.101, which was released on the day of the session). Corbet asked if that was reasonable. Deacon said that you have to trust that kernel developers are not adding bugs on purpose, so when they are found, they need to be fixed; "what's the alternative?", he asked. Kroah-Hartman noted that many of those patches were for things like new device IDs, but that the kernel developers are learning over time. Mistakes were made—and fixed.

Corbet asked if we could expect fewer fixes for 3.10 (which will also be supported for two years), but Kroah-Hartman said that there would likely be more. He now maintains the stable kernel as part of his job, so he has more time to find patches that need to go into the tree. Beyond that, the code base has grown. But the number of fixes going into the kernel starts to tail off significantly after a year, he said. For the stable updates released on that day, 3.0 had eight patches, while 3.10 had nearly 100. The world changes; processors speed up (which leads to new timing issues), for example. "If we stop our rate of change, then we are dead", he said, we have to keep up with the changes going on in the rest of the world.

Referring to Deacon's assertion that bugs are not being introduced on purpose, Corbet asked the panel how it is we know that. The whole maintenance system relies on trust, Deacon said, if that's missing, the whole thing breaks down. Kroah-Hartman said that if you look at the known exploits, they are all attacking some "stupid" mistake that someone (including him) has made, not some kind of introduced backdoor. People who research flaws per line of code find Linux to have lower rates than anything else, he said, so we are doing something right. Corbet pointed out that the kernel developers introduce enough bugs on their own, so there is no real need for anyone else to do so—to some chuckles across the stage.

2038

While OpenBSD has fixed its "2038 problem" (when 32-bit timestamps will wrap), Linux still has not, Corbet said, and wondered if Linux would be ready for that event. He also asked: Will there be no 32-bit processors to be affected in 2038? Deacon noted that billions of 32-bit ARM processors are shipping today, so it is hard to believe they will all be gone in 25 years. Kroah-Hartman said that Intel came out with a 486-based processor recently as well. He suggested that they could all come out of retirement to fix the problems.

Corbet said that OpenBSD broke its ABI to handle the change—something it can do because it ships its user space with the kernel—but that is not something that Linux can do. Something clever will be required to fix the problem, which suggests we should be thinking about it now. Deacon indicated that he thought an ABI break will eventually have to happen at some point. The real problem will be for devices that are being deployed now that will still be running in 2038.

That led to the last question, which is how to handle things like control groups that were added to the kernel, but were "wrong" both internally and externally. The internal problem can be fixed relatively easily, but how can we continue on without carrying tons of baggage from early mistakes at the ABI level, he asked. Zijlstra said that even "simple things" need to be written at least three times before you get them "sort of right". Other operating systems have a way to deprecate things, and some of those have been tried in Linux without success, Kroah-Hartman said. The Linux method is to write something new and wait for the old one to die, then try to sneak the code out of the kernel and wait to see if anyone screams, Zijlstra said.

On the other hand, Deacon said, if you wait for a perfect solution, you'll never get anywhere. Kroah-Hartman said that there is no model for Linux, everything it is doing is new; it is at the forefront and we are learning how to deal with these kinds of problems. "We do it all in public", unlike companies that struggle with the same things, he said. There are lots of hard, and fun, problems to be solved going forward, he said. It is a high-quality problem to have, Corbet said to general agreement.

With that, the time was up and Zemlin retook the stage. Mostly, he wanted to clear up some misinformation that he heard during the session: Kroah-Hartman would not be retired by 2038, Zemlin said—to laughter from the assembled attendees.

[Thanks to the Linux Foundation for travel assistance to Edinburgh for LinuxCon Europe.]

Comments (15 posted)

Building APIs on top of GStreamer

The GStreamer multimedia framework is around twelve years old, but it just made its official 1.0 release last year. The 1.0 milestone was, among other things, a statement of the project's commitment to keeping the API and ABI stable throughout the duration of the 1.x cycle. As several talks at the 2013 GStreamer Conference demonstrate, however, such stability does not mean that there is nothing left to be done in order to make GStreamer an appealing framework for developers. Instead, several ancillary projects have taken on the task and are building higher-level APIs meant to attract further development.

Getting nonlinear

One of the higher-level API projects is GStreamer Editing Services (GES), which is a framework designed to support nonlinear video editing. There have been several GStreamer-based nonlinear editors (NLEs) in the past—most notably PiTiVi—but even fans of those applications would have to admit that they have not experienced the same level of (relative) popularity as seen by GStreamer-based media players.

Within the GStreamer community, the reason for this disparity is generally accepted as the fact that GStreamer itself is optimized for tasks like playback, capture, and transcoding. NLEs, while they obviously need to make use of such functionality, have their own set of primitives—for example, video effects and transitions between two (or more) tracks.

GES is a framework that implements NLE functions. Mathieu Duponchelle and Thibault Saunier spoke about it on the first day of the conference. While GES is now used as the NLE framework for PiTiVi, the two said, it is intended to serve as a general-purpose framework that developers can use to add video editing functionality to their own applications or to build other editors.

In fact, they explained, GES itself is a wrapper around another intermediary framework, GNonLin. The goal is for GES to offer just the higher-level NLE APIs, while GNonLin serves as the glue between GES and GStreamer itself. As such, GNonLin implements things like the GnlObject base class, which adds properties not found in base GStreamer elements like duration, start and stop positions, and an adjustable output rate (for speeding up or slowing down playback without altering the underlying file). Similarly, the GnlOperation class encapsulates a set of transformations on GnlObjects, as is needed to define filters and effects.

GES, in turn, defines the objects used to build an editing application. The most important, they said, is the idea of the GESTimeline. A timeline is the basic video editing unit; it contains a set of layers (GESLayers in this case) stacked in a particular priority. GESLayers contain audio or video clips, and the timeline can be used to re-arrange them, to change the layer stacking order, and to composite layers together. But ultimately a GESTimeline is just a GStreamer element, the speakers said, so it can be used like any other element: its output can be plugged into a pipeline, which makes it easy for any NLE to output video in a supported GStreamer format or to send it to another application.

GES also defines several features of interest to NLE users, they said. First, it has a high-level Effects API, which is a wrapper around GStreamer filter elements. The Effects API exposes features necessary for using video effects in an editor, such as keyframes. Keyframes are control points in a media track, where the user can set a property of interest (for example, audio volume). GES will automatically interpolate the property's value between keyframes, allowing smooth changes. But GES also implements some of the most common transition effects, like cross-fading and screen swipes, making those effects trivial to use. The previous version of GES was not nearly as nice, they said; it required the user to manually create and delete even simple transition effects.

GES's other editing APIs include an advanced timeline editing API, which implements trimming a clip, "rippling" a timeline (which shifts all of the clips further down the timeline whenever a change is made to a clip earlier in the timeline), and "rolls" (switching instantly between two clips on different tracks). GES attempts to implement the most-often-used features by default, so for instance it automatically rescales imported clips to be the same size, but this behavior can be manually switched off when not needed. There is also a titling API, which overlays a text layer on top of a video track.

GES is currently at version 1.1.90, and should reach 1.2.0 shortly—which will be compatible with GStreamer 1.2 (which was released September 25). It represents nearly two years of work, they said, and although they are doing a lot of testing and debugging, GES naturally needs real-world testing on real-world media clips in order to really uncover all of its bugs. They have an integrated test suite that tests a lot of media formats (input and output) and the various effects, but real-life scenarios are often quite different.

PiTiVi is meant to be a general-purpose NLE, they said, but there are several different editing scenarios they hope GES will be useful for, such as an online video editor (perhaps a collaborative one) and a live-editing NLE for use with broadcasting. GES should also be useful for any GStreamer application that needs to mix video tracks; even if you just have two tracks, they said, mixing them in GES will be easier than doing it in lower-level GStreamer pipelines.

The work on GES is not finished, they said. Things still on the to-do list include playback speed control (the first implementation of which is being worked on by a Google Summer of Code student), automatic hardware acceleration (which is scheduled for GStreamer itself), nesting timelines (for combining multiple scenes that are edited separately into a longer finished product), and proxy editing (where a low-resolution version of a video is used in the editor but the high-resolution version is used for the final output). The latter two features are important for high-end video work.

Playback made simple

In contrast to GES, which has been developed in the open for several years, the other new GStreamer API layer discussed at the conference was Fluendo's new media player API, FluMediaPlayer, which is not open source ... yet. As Julien Moutte explained it in his session, the goal of the player is to fill in a missing piece that keeps GStreamer from being used in more third-party projects.

Ultimately, Fluendo wants world domination for GStreamer, Moutte said. So when the company sees a recurring problem, it wants to do something to fix it. Consequently, Fluendo has been spending time at developer events for Android, Windows, and OS X, the platforms where GStreamer is available but not dominating. One of the most common problems that seems to be encountered by people who incorporate GStreamer into their products is that they want to use GStreamer's powerful playback functionality to put a video on screen, but they do not want to take a course or spend a lot of time learning GStreamer internals to do so. In other words, GStreamer needs to improve its "ease of integration" offerings with simple, high-level APIs.

Of course, there is already an abstraction intended to do drop-it-in-and-run playback: playbin. But using playbin still requires developers to learn about GStreamer events, scheduling, and language-specific bindings, he said. There are some other good options, such as the Bacon Video Widget, but it is very GTK+-specific and is GPL-licensed, which means many third-party developers will not use it.

Fluendo's solution is FluMediaPlayer, a new player library that implements the same feature set as playbin2 (the current incarnation of playbin), and is built on top of the GStreamer SDK. The SDK is a bit of a controversial topic on its own; it was created by Fluendo and Collabora specifically to target third-party developers, many of whom are on proprietary operating systems. It is also not up-to-date with the latest GStreamer release (relying instead on GStreamer 0.10), but Moutte said the company intends to handle the transition from 0.10 to 1.x transparently with FluMediaPlayer. The player also adds some new features, such as platform-specific bindings and the ability to connect playback to a DRM module.

FluMediaPlayer uses a simple C API; there are no dependencies to worry about, Moutte said, "just use the header." The player object takes a URI as input, creates the output media stream, and listens for a simple set of events (_play, _stop, _close, etc.). Media streams themselves are created by the player on demand (alleviating the need for the developer to set up parameters by hand), newer streaming protocols like DASH are supported, and multiple players can be run simultaneously, each with its own controls and its own error events.

Moutte then said that his aim for the talk was to get feedback from the GStreamer community: was this a good approach, for example, and if Fluendo were to open the source code up, would others be interested in participating? The company was already moving forward with the product, Moutte said, but he hoped to take the temperature of the GStreamer community and make a case to management for releasing the source. By a show of hands, it seemed like most people liked the approach, but opinion was more divided about participating. One audience member observed that several of the features Moutte had described should be landing in upcoming GStreamer releases, which makes the player seem less appealing. Another commented that the audience present might not be the best group to ask—after all, it is a self-selected group of people who are quite comfortable digging into GStreamer itself. Ideally, such a player would draw in new developers not already working with GStreamer.

Of course, the FluMediaPlayer product can certainly coexist with other GStreamer initiatives. Furthermore, if GStreamer itself does implement several of the higher-level features built into FluMediaPlayer, that will not reduce the appeal of the product to outside developers, but will simplify Fluendo's maintenance. There does seem to be a general agreement that GStreamer itself is technically sound at this point in its history, and that the next big hurdle to overcome is building a layer of services on top of it—services that up until now many GStreamer users have had to re-implement on their own.

[The author would like to thank the Linux Foundation for travel assistance to Edinburgh for GStreamer Conference 2013.]

Comments (none posted)

Verified U-Boot

More systems these days are using some type of verified boot mechanism to ensure that only the expected code can be used to boot the system. Chrome OS uses verified boot to ensure that only approved images can be used on the Chromebook (in normal, non-developer mode). That feature is an integral part of the overall security architecture. The widespread use of UEFI on PCs in recent years has resulted in a lot of attention to its Secure Boot feature. It provides a way to ensure that only software signed by a vendor can be used on that PC.

For embedded systems that don't use Chrome OS and where UEFI is not in widespread use, some form of verified boot is useful for similar reasons. It helps to prevent malware from inserting itself into the boot process, provides user confidence that software updates are safe, and can help manufacturers control the software running on devices in the field.

The downside (at least from a technical user's point of view) may be this manufacturer control. This can be addressed by allowing the user to install keys as UEFI does, or having an option to turn off the verified boot. As an example of the latter, some Chrome OS devices have a mechanical screw which controls the firmware write-protection, so it is possible to disable verified boot.

U-Boot verified boot

U-Boot 2013.07 introduces a feature allowing for the verification of a kernel and other images. This can be used to implement a form of secure boot which we will call "verified boot", to avoid confusion with the UEFI implementation. U-Boot's new verified boot feature provides a mechanism for verifying images while still allowing them to be field-upgraded. It fits in seamlessly with the existing image loading infrastructure in U-Boot.

U-Boot verified boot relies on two familiar technologies: cryptographic hashing (e.g. SHA-1) and public key cryptography (e.g. RSA). Using these technologies it is possible to distribute images and have them verified on a device. Specifically we can create a key, hash an image, sign that hash, and publish the public key. On the device we can obtain an image and verify it was signed by the private key.

Images can be chained one after the other and signed in reverse order either using the same keys or sub-keys (keys derived from other keys). For example, U-Boot may load an image containing a new U-Boot, then boot that. That U-Boot in turn may load an image containing a kernel. Doing that would allow U-Boot itself to be updated with the firmware without risking having an unbootable device due to a bad update.

In principle this chain can be any length, but there must be an initial trusted image ("root of trust") that can start the process. This can be stored in read-only media during manufacture or perhaps protected by on-chip crypto using its own signing scheme. The "root of trust" U-Boot must include the initial public key, held in U-Boot's device tree (often called the flattened device tree or FDT). A more sophisticated scheme would allow the public keys to be provided by the user, perhaps by inserting an SD card containing the key. This could be implemented using a U-Boot script or with a more sophisticated user interface.

Overview of the verified boot flow

How does an image get signed initially, and then later verified?

First, on a host machine, we create a key pair. We make sure that the public key is distributed on each device in a tamper-proof place such as read-only memory or a verified key store. The private key is hidden away (i.e. not on the device!) since an attacker could use it to sign malicious images which all devices would accept as valid.

Second, an image is created. This is the image we want the device to load. The image is hashed and the hash is signed with the private key. This signature is sent to the device along with the image. The image typically contains one or more kernels and perhaps (on ARM at least) some device tree files for devices supported by that kernel.

Third, the device receives the image. It also hashes the image, then verifies that the hash agrees with the signature provided with the image. If it matches then the image is known to be signed as in step two, and it is safe to be used.

Signing and verifying

The implementation in U-Boot follows the above process closely. We will illustrate this by describing the process and also following along with an example. You can try the example yourself, or you can use U-Boot's vboot_test.sh which is a ready-made example using U-Boot's sandbox environment. The creation of a key pair is completed using openssl (you may need to install libssl-dev or similar package on your machine). These commands create a key pair and certificate (containing the public key) using 2048-bit RSA.

    $ openssl genrsa -F4 -out keys/dev.key 2048
    $ openssl req -batch -new -x509 -key keys/dev.key -out keys/dev.crt

Having created these, we want to sign an image. U-Boot's primary image format is FIT (Flat Image Tree), a very flexible, structured container format which supports multiple kernel images, device trees, ram disks, etc. It is easy to add signatures and other metadata to a FIT either during or after creation. The signing can also be performed at the same time as image creation, or later.

So first, to create the FIT:

    $ mkimage -f image.its image.fit

Then to sign it (with U-Boot 2013.07 or later):

    $ mkimage -F image.fit -k keys -K u-boot.dtb -c "sjg@kiwi_signer 5-Sep-2013" -r

Here, -k provides the directory containing keys, -K is the FDT where the public key information will be stored (this file needs to be available on the device). The -c option provides a comment and -r means that any keys used for signing in this pass will be marked as "required", meaning that U-Boot will fail the image if they do not all verify.

The signing command can be run multiple times with different key directories. Each time it is run, it tries to find the keys mentioned in the FIT and sign images using them. The -K argument is itself a device tree file. Verified boot uses this to hold the public key information as mentioned above, and mkimage stores the public key there ready for use.

When creating a FIT with mkimage, a source file is provided. This tells mkimage what images should be included in the FIT. The keys used to sign the image are also specified in this file. An example source file is available in the U-Boot source tree.

The above scheme has a weakness. An attacker can create a new configuration to mix and match images and device tree files and thereby perhaps cause an incorrect combination to be booted. To avoid this, it is possible to sign configurations. This is more secure since it allows U-Boot to check not only that images are correct, but also that the correct images are being used together (example).

The setup is very similar, except now the signature is attached to the configuration(s) and not the images. Each image instead has a simple hash to protect it. The signing process ensures that the configuration and the hash for each associated image is signed. When booting, the configuration is checked first before any images can be loaded. As before, it is possible to sign multiple configurations at different times with different keys, without affecting the validity of previous signatures.

In the general case, there is nothing special to be done in U-Boot for images to be verified when loaded. When the "bootm" command runs, it will run through the FIT as normal — including checking hashes as previously supported. When U-Boot sees a signature node it will verify it (using public keys in the U-Boot control FDT) and display a plus for success and minus for failure. In this case failures are informational and do not cause the image to be rejected. But if any key is marked "required" then U-Boot will fail the image if it does not verify. This marking is done using the -r parameter to mkimage, as described earlier. A bootm run will then include the line:

       Verifying Hash Integrity ... sha1,rsa2048:dev+ OK

in its output, which indicates that RSA signature checking was performed on the image.

Other options and features

Apart from the core function of signing and verifying images, there are a number of optional features and expansions that can be used. For example is possible to use SoC-accelerated crypto, automatically update U-Boot in the field, provide a recovery mode from external media or the network when image verification fails, or to allow user-installed keys.

A Trusted Platform Module (TPM) can be used to hold rollback counters, to protect against rolling back to an older, compromised firmware. U-Boot also provides TPM support for trusted boot and remote attestation.

Performance

Verified boot does not necessarily mean slow boot. Chrome OS starts up U-Boot, loads and verifies a second U-Boot, loads and verifies a kernel and jumps to it all in under 700ms on a modern SoC. Similarly, U-Boot's verified boot implementation is optimized for code size and performance. Keys are pre-processed by mkimage so that U-Boot has to do only minimal processing to use them. The crypto code is simple and does not deal with multiple indirections and data conversions. In fact, it typically takes longer to hash a kernel than it does to verify it using RSA. On a BeagleBone Black (OMAP4), it takes about 6ms to verify a configuration and only adds 6.2KB to U-Boot's code size.

Wrapping up

Verified boot came from Google's Chrome OS, which uses a similar scheme with its own key format and verified boot flow. Initial patches were posted in January 2013 and the upstreaming work was completed for the 2013.07 U-Boot release in July.

The verified boot feature provides an easy way to enable a more secure boot on a device with minimal code size and boot-time impact. It uses the existing U-Boot FIT format and mkimage tool and is compatible with existing boot scripts.

More detailed design information is provided in the U-Boot source tree in verified-boot.txt and signature.txt.

Comments (3 posted)

Security quotes of the week

— Ted Ts'o

— Russ Nelson

Comments (2 posted)

chromium: multiple vulnerabilities

Package(s):	chromium	CVE #(s):	CVE-2013-2906 CVE-2013-2907 CVE-2013-2908 CVE-2013-2909 CVE-2013-2910 CVE-2013-2911 CVE-2013-2912 CVE-2013-2913 CVE-2013-2914 CVE-2013-2915 CVE-2013-2916 CVE-2013-2917 CVE-2013-2918 CVE-2013-2919 CVE-2013-2920 CVE-2013-2921 CVE-2013-2922 CVE-2013-2923
Created:	October 17, 2013	Updated:	December 13, 2013
Description:	From the openSUSE advisory: CVE-2013-2906: Races in Web Audio CVE-2013-2907: Out of bounds read in Window.prototype object CVE-2013-2908: Address bar spoofing related to the “204 No Content” status code CVE-2013-2909: Use after free in inline-block rendering CVE-2013-2910: Use-after-free in Web Audio CVE-2013-2911: Use-after-free in XSLT CVE-2013-2912: Use-after-free in PPAPI CVE-2013-2913: Use-after-free in XML document parsing CVE-2013-2914: Use after free in the Windows color chooser dialog CVE-2013-2915: Address bar spoofing via a malformed scheme CVE-2013-2916: Address bar spoofing related to the “204 No Content” status code CVE-2013-2917: Out of bounds read in Web Audio CVE-2013-2918: Use-after-free in DOM CVE-2013-2919: Memory corruption in V8 CVE-2013-2920: Out of bounds read in URL parsing CVE-2013-2921: Use-after-free in resource loader CVE-2013-2922: Use-after-free in template element CVE-2013-2923: Various fixes from internal audits, fuzzing and other initiatives
Alerts:	Gentoo 201403-01 chromium 2014-03-05 openSUSE openSUSE-SU-2014:0065-1 chromium 2014-01-15 openSUSE openSUSE-SU-2013:1861-1 chromium 2013-12-12 Debian DSA-2785-1 chromium-browser 2013-10-26 Mageia MGASA-2013-0306 chromium-browser-stable 2013-10-17 openSUSE openSUSE-SU-2013:1556-1 chromium 2013-10-16

Comments (none posted)

dropbear: denial of service

Package(s):	dropbear	CVE #(s):	CVE-2013-4421
Created:	October 21, 2013	Updated:	November 18, 2013
Description:	From the Red Hat bugzilla: It was reported that versions of dropbear prior to 2013.59 suffered from a flaw where large decompressed payloads could cause memory exhaustion, resulting in a denial of service.
Alerts:	openSUSE openSUSE-SU-2013:1696-1 dropbear 2013-11-15 openSUSE openSUSE-SU-2013:1616-1 dropbear 2013-10-31 Mandriva MDVSA-2013:261 dropbear 2013-10-28 Mageia MGASA-2013-0318 dropbear 2013-10-25 Fedora FEDORA-2013-18593 dropbear 2013-10-18 Fedora FEDORA-2013-18606 dropbear 2013-10-18

Comments (none posted)

fedmsg: parsing error

Package(s):	fedmsg	CVE #(s):
Created:	October 21, 2013	Updated:	October 23, 2013
Description:	From the Fedora advisory: Bugfix to broken CRL serial parsing code.
Alerts:	Fedora FEDORA-2013-17904 fedmsg 2013-10-18 Fedora FEDORA-2013-17925 fedmsg 2013-10-18

Comments (none posted)

jakarta-commons-fileupload: overwrites content

Package(s):	jakarta-commons-fileupload	CVE #(s):	CVE-2013-2186
Created:	October 23, 2013	Updated:	December 24, 2013
Description:	From the openSUSE advisory: A remote attacker could supply a serialized instance of the DiskFileItem class, which would be deserialized on a server and write arbitrary content to any location on the server that is permitted by the user running the application server process.
Alerts:	Red Hat RHSA-2016:0070-01 RHOSE 2016-01-26 Debian DSA-2827-1 libcommons-fileupload-java 2013-12-24 Ubuntu USN-2029-1 libcommons-fileupload-java 2013-11-13 SUSE SUSE-SU-2013:1660-1 jakarta-commons-fileupload 2013-11-12 openSUSE openSUSE-SU-2013:1596-1 jakarta-commons-fileupload 2013-10-28 openSUSE openSUSE-SU-2013:1571-1 jakarta-commons-fileupload 2013-10-23

Comments (none posted)

java-1.7.0-oracle: multiple vulnerabilities

Package(s):	java-1.7.0-oracle	CVE #(s):	CVE-2013-3829 CVE-2013-4002 CVE-2013-5772 CVE-2013-5774 CVE-2013-5775 CVE-2013-5776 CVE-2013-5777 CVE-2013-5778 CVE-2013-5780 CVE-2013-5782 CVE-2013-5783 CVE-2013-5784 CVE-2013-5787 CVE-2013-5788 CVE-2013-5789 CVE-2013-5790 CVE-2013-5797 CVE-2013-5800 CVE-2013-5801 CVE-2013-5802 CVE-2013-5803 CVE-2013-5804 CVE-2013-5809 CVE-2013-5810 CVE-2013-5812 CVE-2013-5814 CVE-2013-5817 CVE-2013-5818 CVE-2013-5819 CVE-2013-5820 CVE-2013-5823 CVE-2013-5824 CVE-2013-5825 CVE-2013-5829 CVE-2013-5830 CVE-2013-5831 CVE-2013-5832 CVE-2013-5838 CVE-2013-5840 CVE-2013-5842 CVE-2013-5843 CVE-2013-5844 CVE-2013-5846 CVE-2013-5848 CVE-2013-5849 CVE-2013-5850 CVE-2013-5851 CVE-2013-5852 CVE-2013-5854
Created:	October 18, 2013	Updated:	March 30, 2016
Description:	From the Red Hat advisory: CVE-2013-5803 OpenJDK: insufficient checks of KDC replies CVE-2013-5772 OpenJDK: insufficient html escaping in jhat CVE-2013-5797 OpenJDK: insufficient escaping of window title string CVE-2013-5784 OpenJDK: insufficient InterfaceImplementor security checks CVE-2013-5790 OpenJDK: insufficient security checks CVE-2013-5849 OpenJDK: insufficient DataFlavor security checks CVE-2013-5800 OpenJDK: default keytab path information leak CVE-2013-5780 OpenJDK: key data leak via toString() methods CVE-2013-5840 OpenJDK: getDeclaringClass() information leak CVE-2013-5820 OpenJDK: insufficient security checks CVE-2013-5851 OpenJDK: XML stream factory finder information leak CVE-2013-5778 OpenJDK: image conversion out of bounds read CVE-2013-5782 OpenJDK: Incorrect awt_getPixelByte/awt_getPixelShort/awt_setPixelByte/awt_setPixelShort image raster checks CVE-2013-5830 OpenJDK: checkPackageAccess missing security check CVE-2013-5809 OpenJDK: JPEGImageReader and JPEGImageWriter missing band size checks CVE-2013-5829 OpenJDK: Java2d Disposer security bypass CVE-2013-5814 OpenJDK: RMIConnection stub missing permission check CVE-2013-5817 OpenJDK: VersionHelper12 does not honor modifyThreadGroup restriction CVE-2013-5842 OpenJDK: ObjectInputStream/ObjectOutputStream missing checks CVE-2013-5850 OpenJDK: Missing CORBA security checks CVE-2013-5802 OpenJDK: javax.xml.transform.TransformerFactory does not properly honor XMLConstants.FEATURE_SECURE_PROCESSING CVE-2013-5804 OpenJDK: javac does not ignore certain ignorable characters CVE-2013-3829 OpenJDK: java.util.TimeZone does not restrict setting of default time zone CVE-2013-5783 OpenJDK: JTable not properly performing certain access checks CVE-2013-5825 OpenJDK: XML parsing Denial of Service CVE-2013-5823 OpenJDK: com.sun.org.apache.xml.internal.security.utils.UnsyncByteArrayOutputStream Denial of Service CVE-2013-5774 OpenJDK: Inet6Address class IPv6 address processing errors CVE-2013-4002 OpenJDK: XML parsing Denial of Service CVE-2013-5838 OpenJDK: Vulnerability in Libraries component CVE-2013-5824 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5788 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5787 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5789 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5843 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5832 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5852 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5812 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5801 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5776 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5818 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5819 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5831 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5848 Oracle JDK: unspecified vulnerability fixed in 7u45 CVE-2013-5846, CVE-2013-5810, CVE-2013-5844, CVE-2013-5777, CVE-2013-5775, CVE-2013-5854 Oracle JDK: multiple unspecified vulnerabilities fixed in 7u45
Alerts:	Arch Linux ASA-201604-3 jre7-openjdk-headless 2016-04-01 Arch Linux ASA-201604-2 jre7-openjdk 2016-04-01 Arch Linux ASA-201604-1 jdk7-openjdk 2016-04-01 Fedora FEDORA-2014-10649 xerces-j2 2014-09-25 Gentoo 201406-32 icedtea-bin 2014-06-29 Red Hat RHSA-2014:0675-01 java-1.7.0-openjdk 2014-06-10 Red Hat RHSA-2014:0685-01 java-1.6.0-openjdk 2014-06-10 Mandriva MDVSA-2014:100 java-1.7.0-openjdk 2014-05-16 Mageia MGASA-2014-0189 java-1.7.0-openjdk 2014-04-23 Scientific Linux SLSA-2014:0407-1 java-1.7.0-openjdk 2014-04-16 Scientific Linux SLSA-2014:0408-1 java-1.6.0-openjdk 2014-04-16 Oracle ELSA-2014-0406 java-1.7.0-openjdk 2014-04-16 Oracle ELSA-2014-0407 java-1.7.0-openjdk 2014-04-17 Oracle ELSA-2014-0408 java-1.6.0-openjdk 2014-04-16 Oracle ELSA-2014-0408 java-1.6.0-openjdk 2014-04-16 Red Hat RHSA-2014:0414-01 java-1.6.0-sun 2014-04-17 CentOS CESA-2014:0407 java-1.7.0-openjdk 2014-04-16 CentOS CESA-2014:0406 java-1.7.0-openjdk 2014-04-16 CentOS CESA-2014:0408 java-1.6.0-openjdk 2014-04-16 CentOS CESA-2014:0408 java-1.6.0-openjdk 2014-04-16 Scientific Linux SLSA-2014:0406-1 java-1.7.0-openjdk 2014-04-16 Red Hat RHSA-2014:0407-01 java-1.7.0-openjdk 2014-04-16 Red Hat RHSA-2014:0406-01 java-1.7.0-openjdk 2014-04-16 Red Hat RHSA-2014:0408-01 java-1.6.0-openjdk 2014-04-16 Gentoo 201401-30 oracle-jdk-bin 2014-01-26 Ubuntu USN-2089-1 openjdk-7 2014-01-23 openSUSE openSUSE-SU-2013:1968-1 openjdk 2013-12-27 Ubuntu USN-2033-1 openjdk-6 2013-11-21 SUSE SUSE-SU-2013:1677-1 IBM Java 5 2013-11-15 SUSE SUSE-SU-2013:1666-1 OpenJDK 7 2013-11-13 openSUSE openSUSE-SU-2013:1663-1 java-1_7_0-openjdk 2013-11-13 Mandriva MDVSA-2013:267 java-1.7.0-openjdk 2013-11-19 SUSE SUSE-SU-2013:1677-2 Java 6 2013-11-19 Red Hat RHSA-2013:1507-01 java-1.7.0-ibm 2013-11-07 Red Hat RHSA-2013:1508-01 java-1.6.0-ibm 2013-11-07 Red Hat RHSA-2013:1509-01 java-1.5.0-ibm 2013-11-07 Scientific Linux SLSA-2013:1505-1 java-1.6.0-openjdk 2013-11-05 Oracle ELSA-2013-1505 java-1.6.0-openjdk 2013-11-05 Oracle ELSA-2013-1505 java-1.6.0-openjdk 2013-11-05 CentOS CESA-2013:1505 java-1.6.0-openjdk 2013-11-05 Red Hat RHSA-2013:1505-01 java-1.6.0-openjdk 2013-11-05 SUSE SUSE-SU-2013:1677-3 IBM Java 7 2013-11-22 Mandriva MDVSA-2013:266 java-1.6.0-openjdk 2013-11-19 Scientific Linux SLSA-2013:1451-1 java-1.7.0-openjdk 2013-10-23 Oracle ELSA-2013-1451 java-1.7.0-openjdk 2013-10-22 CentOS CESA-2013:1451 java-1.7.0-openjdk 2013-10-23 CentOS CESA-2013:1447 java-1.7.0-openjdk 2013-10-22 Red Hat RHSA-2013:1451-01 java-1.7.0-openjdk 2013-10-22 Oracle ELSA-2013-1447 java-1.7.0-openjdk 2013-10-21 Scientific Linux SLSA-2013:1447-1 java-1.7.0-openjdk 2013-10-21 Red Hat RHSA-2013:1447-01 java-1.7.0-openjdk 2013-10-21 Red Hat RHSA-2013:1440-01 java-1.7.0-oracle 2013-10-17 SUSE SUSE-SU-2013:1669-1 IBM Java 5 2013-11-14 Mageia MGASA-2013-0322 java-1.7.0-openjdk 2013-11-13 Mageia MGASA-2013-0323 java-1.6.0-openjdk 2013-11-13

Comments (none posted)

kernel: information leak

Package(s):	kernel	CVE #(s):	CVE-2013-4299
Created:	October 17, 2013	Updated:	December 20, 2013
Description:	From the Red Hat advisory: An information leak flaw was found in the way Linux kernel's device mapper subsystem, under certain conditions, interpreted data written to snapshot block devices. An attacker could use this flaw to read data from disk blocks in free space, which are normally inaccessible.
Alerts:	SUSE SUSE-SU-2015:0812-1 kernel 2015-04-30 SUSE SUSE-SU-2015:0652-1 Linux kernel 2015-04-02 SUSE SUSE-SU-2014:0908-1 Linux kernel 2014-07-17 SUSE SUSE-SU-2014:0909-1 Linux kernel 2014-07-17 SUSE SUSE-SU-2014:0910-1 Linux kernel 2014-07-17 SUSE SUSE-SU-2014:0911-1 Linux kernel 2014-07-17 SUSE SUSE-SU-2014:0912-1 Linux kernel 2014-07-17 Debian DSA-2906-1 linux-2.6 2014-04-24 Ubuntu USN-2067-1 linux-ti-omap4 2014-01-03 Ubuntu USN-2066-1 kernel 2014-01-03 Oracle ELSA-2014-3002 kernel 2014-02-12 Red Hat RHSA-2013:1860-01 kernel 2013-12-19 Mageia MGASA-2013-0375 kernel-vserver 2013-12-18 Mageia MGASA-2013-0373 kernel-tmb 2013-12-18 Mageia MGASA-2013-0374 kernel-rt 2013-12-18 Mageia MGASA-2013-0372 kernel-linus 2013-12-18 Mageia MGASA-2013-0371 kernel 2013-12-17 Ubuntu USN-2050-1 linux-ti-omap4 2013-12-07 Ubuntu USN-2049-1 kernel 2013-12-07 Oracle ELSA-2013-1790 kernel 2013-12-06 Red Hat RHSA-2013:1783-01 kernel 2013-12-05 Ubuntu USN-2046-1 linux-ti-omap4 2013-12-03 Ubuntu USN-2044-1 linux-ti-omap4 2013-12-03 Ubuntu USN-2042-1 linux-lts-saucy 2013-12-03 Ubuntu USN-2041-1 linux-lts-raring 2013-12-03 Ubuntu USN-2040-1 linux-lts-quantal 2013-12-03 Ubuntu USN-2045-1 kernel 2013-12-03 Ubuntu USN-2043-1 kernel 2013-12-03 Red Hat RHSA-2013:1520-01 kernel 2013-11-14 Ubuntu USN-2015-1 kernel 2013-11-08 Ubuntu USN-2016-1 EC2 kernel 2013-11-08 CentOS CESA-2013:X012 Xen4CentOS kernel 2013-11-06 Red Hat RHSA-2013:1490-01 kernel-rt 2013-10-31 Oracle ELSA-2013-1645 kernel 2013-11-26 Red Hat RHSA-2013:1519-01 kernel 2013-11-13 Scientific Linux SLSA-2013:1449-1 kernel 2013-10-23 Oracle ELSA-2013-1449 kernel 2013-10-23 CentOS CESA-2013:1449 kernel 2013-10-23 Red Hat RHSA-2013:1450-01 kernel 2013-10-22 Red Hat RHSA-2013:1449-01 kernel 2013-10-22 CentOS CESA-2013:1436 kernel 2013-10-17 Scientific Linux SLSA-2013:1436-1 kernel 2013-10-17 Oracle ELSA-2013-1436 kernel 2013-10-16 Red Hat RHSA-2013:1436-01 kernel 2013-10-16

Comments (none posted)

kernel: multiple vulnerabilities

Package(s):	kernel	CVE #(s):	CVE-2013-2898 CVE-2013-4300
Created:	October 22, 2013	Updated:	October 23, 2013
Description:	From the CVE entries: drivers/hid/hid-sensor-hub.c in the Human Interface Device (HID) subsystem in the Linux kernel through 3.11, when CONFIG_HID_SENSOR_HUB is enabled, allows physically proximate attackers to obtain sensitive information from kernel memory via a crafted device. (CVE-2013-2898) The scm_check_creds function in net/core/scm.c in the Linux kernel before 3.11 performs a capability check in an incorrect namespace, which allows local users to gain privileges via PID spoofing. (CVE-2013-4300)
Alerts:	openSUSE openSUSE-SU-2014:1669-1 kernel 2014-12-19 openSUSE openSUSE-SU-2014:1677-1 kernel 2014-12-21 Oracle ELSA-2014-3002 kernel 2014-02-12 Mageia MGASA-2013-0375 kernel-vserver 2013-12-18 Mageia MGASA-2013-0373 kernel-tmb 2013-12-18 Mageia MGASA-2013-0374 kernel-rt 2013-12-18 Mageia MGASA-2013-0372 kernel-linus 2013-12-18 Mageia MGASA-2013-0371 kernel 2013-12-17 Ubuntu USN-1995-1 linux-lts-raring 2013-10-21 Ubuntu USN-1998-1 kernel 2013-10-21

Comments (none posted)

kernel: information leak

Package(s):	kernel	CVE #(s):	CVE-2013-4368
Created:	October 23, 2013	Updated:	October 23, 2013
Description:	From the Red Hat advisory: An information leak flaw was found in the way Xen hypervisor emulated the OUTS instruction for 64-bit paravirtualized guests. A privileged guest user could use this flaw to leak hypervisor stack memory to the guest.
Alerts:	Debian DSA-3006-1 xen 2014-08-18 Gentoo 201407-03 xen 2014-07-16 SUSE SUSE-SU-2014:0470-1 Xen 2014-04-01 SUSE SUSE-SU-2014:0446-1 Xen 2014-03-25 SUSE SUSE-SU-2014:0411-1 Xen 2014-03-20 openSUSE openSUSE-SU-2013:1953-1 xen 2013-12-25 Oracle ELSA-2013-1790 kernel 2013-12-06 CentOS CESA-2013:X013 xen 2013-11-25 openSUSE openSUSE-SU-2013:1636-1 xen 2013-11-07 Scientific Linux SLSA-2013:1449-1 kernel 2013-10-23 Oracle ELSA-2013-1449 kernel 2013-10-23 Fedora FEDORA-2013-19048 xen 2013-10-23 Fedora FEDORA-2013-19053 xen 2013-10-23 CentOS CESA-2013:1449 kernel 2013-10-23 Red Hat RHSA-2013:1449-01 kernel 2013-10-22

Comments (none posted)

mysql: multiple unspecified vulnerabilities

Package(s):	mysql	CVE #(s):	CVE-2012-2750 CVE-2013-3839
Created:	October 17, 2013	Updated:	November 4, 2013
Description:	From the Mandriva advisory: Unspecified vulnerability in MySQL 5.5.x before 5.5.23 has unknown impact and attack vectors related to a Security Fix, aka Bug #59533. NOTE: this might be a duplicate of CVE-2012-1689, but as of 20120816, Oracle has not commented on this possibility (CVE-2012-2750). Unspecified vulnerability in the MySQL Server component in Oracle MySQL 5.1.70 and earlier, 5.5.32 and earlier, and 5.6.12 and earlier allows remote authenticated users to affect availability via unknown vectors related to Optimizer (CVE-2013-3839).
Alerts:	Gentoo 201409-04 mysql 2014-09-04 CentOS CESA-2014:0189 mariadb55-mariadb 2014-02-26 CentOS CESA-2014:0173 mysql55-mysql 2014-02-19 Red Hat RHSA-2014:0189-01 mariadb55-mariadb 2014-02-19 Scientific Linux SLSA-2014:0186-1 mysql55-mysql 2014-02-18 Oracle ELSA-2014-0186 mysql55-mysql 2014-02-18 CentOS CESA-2014:0186 mysql55-mysql 2014-02-19 Red Hat RHSA-2014:0186-01 mysql55-mysql 2014-02-18 Red Hat RHSA-2014:0173-01 mysql55-mysql 2014-02-13 Debian DSA-2818-1 mysql-5.5 2013-12-16 Fedora FEDORA-2013-19648 mysql 2013-11-02 Fedora FEDORA-2013-19654 community-mysql 2013-11-02 Ubuntu USN-2006-1 mysql-5.5, mysql-dfsg-5.1 2013-10-24 Debian DSA-2780-1 mysql-5.1 2013-10-18 Mandriva MDVSA-2013:250 mysql 2013-10-17

Comments (none posted)

Parallel-ForkManager: symlink attack

Package(s):	Parallel-ForkManager	CVE #(s):	CVE-2011-4115
Created:	October 17, 2013	Updated:	October 23, 2013
Description:	From the Gentoo advisory: The Perl Parallel-ForkManager module does not handle temporary files securely. A local attacker could perform symlink attacks to overwrite arbitrary files with the privileges of the user running the application.
Alerts:	Gentoo 201310-11 Parallel-ForkManager 2013-10-17

Comments (none posted)

polarssl: multiple vulnerabilities

Package(s):	polarssl	CVE #(s):	CVE-2011-1923 CVE-2012-2130
Created:	October 17, 2013	Updated:	October 23, 2013
Description:	From the Gentoo advisory: Multiple vulnerabilities have been discovered in PolarSSL. A remote attacker might be able to cause Denial of Service, conduct a man-in-the middle attack, compromise an encrypted communication channel, or obtain sensitive information.
Alerts:	Gentoo 201310-10 polarssl 2013-10-17

Comments (none posted)

polarssl: code execution

Package(s):	polarssl	CVE #(s):	CVE-2013-5914
Created:	October 21, 2013	Updated:	October 23, 2013
Description:	From the Debian advisory: Paul Brodeur and TrustInSoft discovered a buffer overflow in the ssl_read_record() function, allowing the potential execution of arbitrary code.
Alerts:	Debian DSA-2782-1 polarssl 2013-10-20

Comments (none posted)

python-crypto: information leak

Package(s):	python-crypto	CVE #(s):	CVE-2013-1445
Created:	October 21, 2013	Updated:	October 29, 2013
Description:	From the Debian advisory: A cryptographic vulnerability was discovered in the pseudo random number generator in python-crypto. In some situations, a race condition could prevent the reseeding of the generator when multiple processes are forked from the same parent. This would lead it to generate identical output on all processes, which might leak sensitive values like cryptographic keys.
Alerts:	Mandriva MDVSA-2013:262 python-pycrypto 2013-10-28 Mageia MGASA-2013-0319 python-pycrypto 2013-10-25 Fedora FEDORA-2013-19472 python-crypto 2013-10-27 Fedora FEDORA-2013-19441 python-crypto 2013-10-27 Debian DSA-2781-1 python-crypto 2013-10-18

Comments (none posted)

quassel: SQL injection

Package(s):	quassel	CVE #(s):	CVE-2013-4422
Created:	October 18, 2013	Updated:	February 26, 2014
Description:	From the Mageia advisory: Quassel IRC before 0.9.1 is vulnerable to SQL injection if used with Qt 4.8.5, due to a change in Qt's postgres driver, allowing other IRC users to trick the Quassel core into executing SQL queries.
Alerts:	Fedora FEDORA-2014-1734 quassel 2014-02-26 Fedora FEDORA-2014-1742 quassel 2014-02-26 Gentoo 201311-03 quassel 2013-11-06 Mageia MGASA-2013-0311 quassel 2013-10-17

Comments (none posted)

xen: multiple vulnerabilities

Package(s):	xen	CVE #(s):	CVE-2013-4369 CVE-2013-4370 CVE-2013-4371 CVE-2013-4375
Created:	October 23, 2013	Updated:	October 23, 2013
Description:	From the Red Hat bugzilla: CVE-2013-4369: The libxlu library function xlu_vif_parse_rate does not properly handle inputs which consist solely of the '@' character, leading to a NULL pointer dereference. A toolstack which allows untrusted users to specify an arbitrary configuration for the VIF rate can be subjected to a DOS. CVE-2013-4370: The ocaml binding for the xc_vcpu_getaffinity function incorrectly frees a pointer before using it and subsequently freeing it again afterwards. The code therefore contains a use-after-free and double-free flaws. An attacker may be able to cause a multithreaded toolstack written in ocaml and using this function to race against itself leading to heap corruption and a potential DoS. Depending on the malloc implementation code execution cannot be ruled out. CVE-2013-4371: If realloc(3) fails then libxl_list_cpupool will incorrectly return the now-free original pointer. An attacker may be able to cause a multithreaded toolstack using this function to race against itself leading to heap corruption and a potential DoS. Depending on the malloc implementation code execution cannot be ruled out. CVE-2013-4375: The qdisk PV disk backend in the qemu-xen flavour of qemu ("upstream qemu") can be influenced by a malicious frontend to leak mapped grant references. A malicious HVM guest can cause the backend domain to run out of grant references, leading to a DoS for any other domain which shares that driver domain.
Alerts:	Gentoo 201407-03 xen 2014-07-16 Ubuntu USN-2092-1 qemu, qemu-kvm 2014-01-30 openSUSE openSUSE-SU-2013:1953-1 xen 2013-12-25 CentOS CESA-2013:X013 xen 2013-11-25 Fedora FEDORA-2013-19048 xen 2013-10-23 Fedora FEDORA-2013-19053 xen 2013-10-23

Comments (none posted)

xorg-server: denial of service

Package(s):	xorg-server, xorg-server-lts-quantal, xorg-server-lts-raring	CVE #(s):	CVE-2013-1056
Created:	October 18, 2013	Updated:	October 23, 2013
Description:	From the Ubuntu advisory: It was discovered that non-root X.Org X servers such as Xephyr incorrectly used cached xkb files. A local attacker could use this flaw to cause a xkb cache file to be loaded by another user, resulting in a denial of service.
Alerts:	Gentoo 201405-07 xorg-server 2014-05-15 Ubuntu USN-1990-1 xorg-server, xorg-server-lts-quantal, xorg-server-lts-raring 2013-10-17

Comments (none posted)

Kernel release status

The current development kernel is 3.12-rc6, released on October 19. Linus said: "Nothing major happened last week., and the upcoming week is likely to be quiet too, since a lot of core maintainers will be in Edinburgh for the KS."

Stable updates: 3.11.6 and 3.10.17 were released on October 18, followed by 3.4.67 and 3.0.101 on October 22. This is the end of the line for 3.0: "I will NOT be doing any more 3.0.x kernel releases. If you rely on the 3.0.x kernel series, you should now move to the 3.10.x kernel series, or, at the worse case, 3.4.x."

Comments (none posted)

Quotes of the week

— Matt Porter

— Thierry Reding (Thanks to Neil Brown)

— Linus Torvalds

Comments (none posted)

Nftables pulled for 3.13

For those who have been following nftables, the replacement firewall subsystem for the kernel: this code has just been pulled into the net-next tree. That means that, barring some sort of trouble, it will be merged in the 3.13 development cycle. This code is not ready to replace iptables yet, but the pace of the work should increase once this subsystem is in the mainline. Anybody wanting to try out nftables can see the quick howto page for instructions.

Comments (24 posted)

The power-aware scheduling mini-summit

The first day of the 2013 Kernel Summit was set aside for mini-summits on various topics. One of those topics was the controversial area of power-aware scheduling. Numerous patches attempting to improve the scheduler have been posted, but none have come near the mainline. This gathering brought together developers from the embedded world and the core scheduler developers to try to make some forward progress in this area; whether they succeeded remains to be seen, but some tentative forward steps were identified.

Morten Rasmussen, the author of the big LITTLE MP patch set, was one of the organizers of this session. He started with a set of agenda items and supporting slides; the topics to be discussed were:

• Unification of power-management policies, which are currently split among multiple, uncoordinated subsystems.

• Task packing. Various patch sets have been posted, but none have really gotten anywhere.

• Power drivers and, in particular, the creation of proper abstractions for hardware-level power management.

He attempted to get started with a discussion of the cpufreq and cpuidle subsystems, but didn't get very far before the conversation shifted.

The need for metrics

Ingo Molnar came in with a complaint: none of the power-management work starts with measurements of the system's power behavior. Without a coherent approach to measuring the effects of a patch, there is no real way to judge these patches to decide which ones should go in. We cannot, he said, merge scheduler patches on faith, hoping that they somehow make things better.

What followed was a long and wide-ranging discussion of how such metrics might be made. What the scheduler developers would like is reasonably clear: how long did it take to execute a defined amount of work, and how much energy was consumed in the process? There is also some interest in what worst-case latencies were experienced by the workload while it was running. With reproducible numbers for these quantities, it should be possible to determine which patches help with which workloads and make some progress in this area.

Agreement with this approach was not universal, though. Mark Gross of Intel made the claim that this kind of performance metric was "the road to hell." Instead, he said, power benchmarks should focus on low-level behavior like processor sleep ("C") states. For any given sleep state, the processor must remain in that state for a given period of time before going into that state actually saves power. So the cpuidle subsystem must make an estimate of how long the processor must be idle before selecting an appropriate sleep state. The actual idle period is something that can then be measured; over time, one can come up with a picture of how well the kernel's estimates of idle time match up with reality. That, Mark said, is the kind of benchmark the kernel developers should be using.

Others argued that idle-time estimation accuracy is a low-level measurement that may not map well onto what the users actually want: their work done quickly, without unacceptable latencies, and with a minimal use of power. But actual power-usage measurements have been hard to come by; the processor vendors appear to be reluctant to expose that information to the kernel. So the group never did come up with a good set of metrics to use for the evaluation of scheduler patches. In the end, Ingo said, the first patch that adds a reasonable power-oriented benchmark to the tools directory will go in; it can be refined from there.

What to do

From there, the discussion shifted toward how the scheduler's power behavior might be improved. It was agreed that there is a need for a better mechanism by which an application can indicate its latency requirements to the kernel. These latency requirements then need to be handled carefully: it will not do to keep all CPUs in the system awake because an application running on one of them needs bounded latencies.

There was some talk of adding some sort of energy use model to a simulator — either linsched or the perf sched utility — to allow for standardized testing of patches with specific workloads. Linsched was under development by an intern at Google, but the work was not completed, so it's still not ready for upstreaming. Ingo noted that the resources to do this work are available; there are, after all, developers interested in getting power-aware code into the scheduler.

Rafael Wysocki asked the scheduler developers: what information do you need from the hardware to make better scheduling decisions? Paul Turner responded: the cost of running the system in a given configuration; Peter Zijlstra added that he would like to know the cost of starting up an additional CPU. The response from Mark is that it all depends on which generation of hardware is in use. In general, Intel seems to be reluctant to make that information available, an approach which caused some visible frustration among the scheduler developers.

Over time the discussion did somewhat converge on what the scheduler community would like to have. Some sort of cost metric should be attached to the scheduling domains infrastructure; it would tell the scheduler what the cost of firing up any given processor would be. That information would have to appear at multiple levels; bringing up the first processor in a different physical package will clearly cost more than adding a processor in an already-active package, for example.

Tied into this is the concept of performance ("P") states, generally thought of as the "CPU frequency." The frequency concept is somewhat outdated, but it persists in the kernel's cpufreq subsystem which, it was mostly agreed, should go away. The scheduler does need to understand the cost (and change in performance) associated with a P-state change, though; that would allow it to choose between increasing a CPU's P state or moving a task to a new CPU. How this information would be exposed by the CPU remains to be seen, but, if it were available, it would be possible to start making smarter scheduling decisions with it.

How those decisions would be made remains vague. There was talk of putting together some kind of set of standard workloads, but that seems premature. Paul suggested starting with a set of "stories" describing specific workloads in a human-comprehensible form. Once a collection of stories has been put together, developers can start trying to find the common themes that can be used to come up with algorithms for better, more power-aware scheduling.

There was some brief discussion of Morten's recent scheduler patches. It was agreed that they provide a reasonable start for the movement of CPU frequency and idle awareness into the scheduler itself. A focus on moving cpuidle into the scheduler first was suggested; most developers would rather see cpufreq just go away at this point.

And that was where the group was reminded that lunch had started nearly half an hour before. The guidance produced for the power-aware scheduling developers remains vague at best, but there are still some worthwhile conclusions, with the need for a set of plausible metrics being at the top of the list. That should be enough to enable this work to take a few baby steps forward.

(For more details, see the extensive notes from the session taken by Paul McKenney and Paul Walmsley).

[Your editor would like to thank the Linux Foundation for supporting his travel to Edinburgh.]

Comments (15 posted)

A look at the 3.12 development cycle

As of this writing, the 3.12-rc6 prepatch has been released, Linus seems happy with the state of the kernel, and, in general, there are few reports of problems on the mailing lists. If things continue to stabilize, the 3.12 cycle might be a short one, even by recent standards. So, clearly, it's time to get the traditional development statistics article out there.

There have been 10,480 non-merge changesets pulled into the mainline repository during this cycle (so far). That means that 3.12 may be the slowest cycle since 3.6 which, almost exactly one year ago, came out with 10,247 changesets. An unscientific look at recent release history suggests that kernels released in the (northern hemisphere) fall tend to include fewer changesets than those released at other times of the year, possibly reflecting lower productivity while developers take time off over the summer.

That said, 10,480 changesets is still quite a bit of work. Those changesets were contributed by 1,259 developers (a typical number for recent kernels), added 601,000 lines of code, and removed 279,000 for a net gain of 322,000 lines. The most active developers in this cycle were:

Most active 3.12 developers

By changesets Sachin Kamat 261 2.5% Jingoo Han 241 2.3% Mark Brown 209 2.0% Greg Kroah-Hartman 197 1.9% H Hartley Sweeten 160 1.5% Alex Deucher 151 1.4% Laurent Pinchart 140 1.3% Daniel Vetter 138 1.3% Fabio Estevam 114 1.1% Chris Metcalf 103 1.0% Dan Carpenter 96 0.9% Dave Chinner 90 0.9% Peter Hurley 83 0.8% Joe Perches 80 0.8% Ben Hutchings 77 0.7% Magnus Damm 76 0.7% Rafael J. Wysocki 73 0.7% Lars-Peter Clausen 73 0.7% Trond Myklebust 67 0.6% Axel Lin 65 0.6%

By changed lines Larry Finger 92908 12.6% Jesse Brandeburg 30520 4.2% Greg Kroah-Hartman 29740 4.0% H Hartley Sweeten 25932 3.5% Alex Deucher 18026 2.5% Ben Hutchings 17660 2.4% Rob Clark 15703 2.1% Bradley Grove 15687 2.1% Dave Chinner 15099 2.1% Scott Kilau 14712 2.0% Lidza Louina 13474 1.8% Laurent Pinchart 11676 1.6% Rajendra Nayak 10866 1.5% Chris Metcalf 8924 1.2% Tomi Valkeinen 8881 1.2% Feng-Hsin Chiang 6813 0.9% Yuan-Hsin Chen 6813 0.9% Ambresh K 5528 0.8% Hans Verkuil 5385 0.7% Atul Deshmukh 4849 0.7%

Sachin Kamat and Jingoo Han both contributed a wide range of cleanup patches all over the driver subsystem. Mark Brown continues to do substantial work in the sound, SPI, and regulator driver subsystems, among others. Greg Kroah-Hartman integrated a number of low-level device model changes, along with Lustre filesystem fixups and more. H. Hartley Sweeten's crusade to clean up the Comedi drivers continues; that work resulted in the removal of over 21,000 lines of code this time around.

On the "lines changed" side, Larry Finger added the Realtek RTL8188EU wireless network driver to the staging tree. Jesse Brandeburg added the Intel i40e network driver. Greg's and Hartley's work was, in both cases, dominated by the removal of large amounts of unneeded driver code, while Alex Deucher continues to add functionality to the Radeon driver.

A total of 212 employers (that we know of) supported work on the 3.12 release. The most active of those were:

Most active 3.12 employers

By changesets Intel 1028 9.8% (None) 964 9.2% Linaro 732 7.0% Red Hat 707 6.7% (Unknown) 609 5.8% Samsung 492 4.7% IBM 390 3.7% Freescale 256 2.4% Renesas Electronics 249 2.4% Texas Instruments 245 2.3% Linux Foundation 225 2.1% SUSE 206 2.0% Oracle 183 1.7% Free Electrons 183 1.7% (Consultant) 182 1.7% AMD 178 1.7% Vision Engraving 175 1.7% Google 161 1.5% Huawei Technologies 124 1.2% Broadcom 120 1.1%

By lines changed (None) 134134 18.3% Intel 61227 8.3% Red Hat 49820 6.8% Linux Foundation 32955 4.5% Vision Engraving 26848 3.7% Linaro 26081 3.5% Texas Instruments 24518 3.3% AMD 24389 3.3% (Unknown) 22927 3.1% Renesas Electronics 20656 2.8% Outreach Program for Women 19649 2.7% Solarflare Comm. 18303 2.5% ATTO Technology 15688 2.1% Digi International 14720 2.0% Faraday Technology 13626 1.9% IBM 13554 1.8% Samsung 13083 1.8% Tilera 12256 1.7% Freescale 12104 1.6% SUSE 11265 1.5%

This is not the first time that Red Hat has been upstaged as the top corporate contributor, but it has never been as low as #4. Linaro, instead, continues to increase its contributions to the kernel, as do a number of mobile and embedded companies. The number of changes from volunteers is down slightly, in keeping with the steady trend over the last few years. Developers brought in through the Outreach Program for Women continued to contribute significantly during this cycle.

Your editor is often asked to summarize the origin of kernel patches geographically — how many are coming from $COUNTRY? That question can be hard to answer. But there is another question that is a bit easier: every commit in the repository has a time stamp, and that time stamp includes a time zone. It is a relatively easy matter to pass over a range of commits and summarize which time zones appear most often.

The result of this work appears in the plot to the right. One should bear in mind that this data is necessarily somewhat noisy; there is nothing that constrains developers' machines to have their time set in the zone where they physically reside. Daylight saving time can also add noise to the picture. In the aggregate, though, there are some interesting things to be seen here.

Starting at the top, +10 includes parts of Russia, parts of Indonesia, and, most importantly for this study, parts of Australia. The +9 zone, instead, is mainly Japan and Korea, while +8 is Western Australia and China. About 15% of the changes to 3.12 came from those three time zones.

There is only one country that lives in +5:30 — India. The number of contributions from India has been growing for a while; it's now 6% of the total. Going west from there, +2 to +4 will be dominated by continental Europe, with the central European time zone accounting for 23% of the changes in 3.12. The UK and Ireland, at +1, put in another 10%.

The western hemisphere (negative) zones will be dominated by North America. The -3 zone, however, only covers Newfoundland and Labrador in the northern hemisphere; the relative scarcity of kernel hackers in that part of the world leads one to conclude that 5% of the patches for 3.12 came from Brazil and Argentina, both of which reside in that zone. Your editor's time zone (-6), alas, was the source of only 1% of the changes going into this release; it must be time to pull together some white-space patches to improve that situation.

To state things more generally: one could say that Asia and Australia contributed 22% of the changes to 3.12, while Europe contributed 43%, North America 30%, and South America 5%. These numbers are clearly approximate, but they probably do not hugely distort the reality. The Linux kernel project truly is global in scope, with developers representing much of the planet participating. All told, it has the appearance of a healthy and thriving community.

Comments (29 posted)

The LPC Android microconference

A number of upstream community developers and Google Android developers got together at the 2013 Linux Plumbers Conference discuss some of the non-graphics related out-of-tree or still-in-staging portions of the Android patch set. This discussion followed the Android graphics microconference held earlier in the day. This article contains a summary of the major issues discussed at this gathering.

Unified IPC with binder and kdbus

Earlier in the year, Greg Kroah-Hartman from the Linux Foundation had expressed his goal of getting the user-space binder libraries to run over the still-in-development kdbus interprocess communication mechanism. The session started with a question to Greg: "how is that going?". Bashfully rubbing his head and getting playfully heckled by Kay Sievers (another of the kdbus developers) from Red Hat, Greg suggested that we maybe could check back next year. Greg mentioned that he still has it as a goal, but Kay's concerns about the viability of the concept have made him somewhat less confident. At this point Kay stepped in to describe some of the semantic differences between the mechanisms, such as how everything in kdbus is asynchronous, making it more like a network protocol, while binder calls block to completion in a way that's more akin to system calls. These differences in behavior make it very difficult to support both modes of operation elegantly and efficiently in one IPC mechanism.

When questioned whether this meant that binder should be merged upstream alongside kdbus, Greg was a bit hesitant given the reported security concerns with using binder in a non-Android environment. Colin Cross, one of the Android developers in attendance, noted that the security issue is quite easy to avoid, and wouldn't really be a problem, but from his perspective, he doesn't see an immediate need to get binder out of staging and officially merged upstream. Greg agreed, and also clarified there is no rush, as he is fine with binder staying in staging for however long is necessary.

The Android developers have often been dragged through the mud by the community for implementing their own solutions, focused on solving their own problems, rather than working with the community to fix the existing infrastructure to satisfy their needs. In that context, I somewhat jokingly prodded Greg to explain why it is acceptable to develop kdbus instead of fixing or expanding binder's features to support D-Bus. He acknowledged the contradiction but repeated that kdbus really seems to support a different model, and there isn't likely to be a clean way to support both in one implementation.

Kay mentioned that while, in his opinion, the interfaces likely can't be shared, he did see some hope for sharing some of the underlying infrastructure. He was particularly keen on the concept of "sealed file descriptors" as being something that Android could make use of. Sealed file descriptors are used when two applications need to share memory by passing a file descriptor; they allow the sender to "seal" the descriptor so that the receiver can be confident that the data won't be later modified by the sender. Kay mentioned the parallels with Android's ashmem, which is the interface Android uses to create shared memory regions that can be shared via file descriptor passing. At this point there was some confusion in the discussion; while the feature does seem useful, it didn't seem to actually mirror how Android currently uses ashmem or shared file descriptors, but it seemed like maybe it would be something that might indeed become more useful once the functionality is upstream.

I then discussed some of my reasons for being hopeful that Greg would be able to achieve his goal. In particular, there is the issue of binder's complexity and that there really is only one key developer on the Android team who understands the in-kernel binder driver. This makes binder somewhat risky since that developer could be hit by a bus or otherwise stop participating in discussions, making it hard to find someone to continue to maintain binder upstream. Additionally, things like ioctl() compatibility support are currently lacking in binder, and, because of its design, it's not easy for 32-bit and 64-bit applications to communicate with each other using binder. Greg noted that everything in kdbus is 64-bit safe, but he also didn't see why binder couldn't be fixed since it's not an official, stable API, which caused some cross talk about how important supporting legacy Android environments is. Colin spoke up to say it really wasn't a big issue, since, when it's needed, the binder library could be made to query the system and do the right thing.

With that the discussion seemed to be exhausted, with Greg saying we should check back in next year to see if his thinking has changed.

Netfilter extensions

The next topic was netfilter extensions. JP Abgrall from Google's Android team started with a brief background presentation (slides) on the "xt_qtaguid" filter he developed, along with some other netfilter modifications Android uses (in particular, xt_quota2 and xt_idletimer) in order to provide per-application network data statistics in its settings interface.

When these changes were submitted to the community, it was suggested that he look at using the already upstream xt_owner and xt_nfacct (both of which were merged after xt_qtaguid was developed) to provide the same functionality. But JP had a few concerns about the differences between those modules, particularly that using xt_owner and xt_nfacct would require manipulating the firewall rules during normal operation and would require excessive numbers of rules, both of which could cause performance problems.

Marcel Holtmann from Intel spoke up to say that he thought the functionality being provided was really nice and that his projects would like to have something similar. But there were some concerns about how it would work with control groups. JP mentioned he had briefly looked at control groups, but they didn't seem useful. This caused some quick debate between Marcel and JP on the particular differences between how Android and desktop environments utilize control groups which I'm not sure clarified much.

Eric Dumazet, also from Google, but not an Android developer, piped in that he was a networking developer, that the functionality JP wanted with xt_qtaguid was already upstream, and that NFQUEUE is what they should be using. Since NFQUEUE pushes the packet decisions to user space, this caused a bit of an uproar in the room, as numerous folks were very skeptical that context switching to user space on every packet would be particularly fast or power-efficient.

Eric reassured everyone that it wasn't a concern, and that the enterprise world uses NFQUEUE for workloads up to some millions of packets per second without trouble. After a bit of contentious back-and-forth with JP, it seemed this issue wouldn't be resolved in the time remaining, and Eric suggested JP come over to his side of the Google campus at a later time to discuss it further.

Eric also asked about the out-of-tree xt_quota2 usage and why it was chosen instead of using the already-upstream xt_quota. JP mentioned that the in-tree quota code didn't seem useful at all, and the xt_quota2 code was already implemented. Eric suggested that if the upstream quota didn't work, it should be fixed instead of using xt_quota2. It was brought up that Jan Englehardt, the developer of xt_quota2 had been contacted, and he said the xt_quota2 code had been rejected by the upstream developers. So being able to better describe the limitations with the upstream quota code will be needed to help persuade upstream maintainers that the functionality in xt_quota2 is useful.

Android gadget driver and ConfigFS gadget driver

Closing out the microconference session was the Android gadget driver and ConfigFS gadget driver discussion. To try to liven things up a bit, Benoit Goby, of the Google Android team, and Andrzej Pietrasiewicz from Samsung, were seated face-off style in the front of the room for a dramatic showdown.

Benoit started off with a bit of background on the Android gadget driver. This driver allows Android devices to provide a number of gadget functions, such as support for the picture transfer protocol (PTP), USB mass storage, Ethernet (for tethering), and more over a single USB connection. Additionally, unlike other multi-function gadget drivers already upstream, these multiplexed functions can be dynamically configured at run time. The Android gadget driver patches also provide additional functions that the upstream kernel doesn't yet support, like the media transfer protocol (MTP), the ADB debugger, Android Accessory and Android Audio gadgets. Additionally, the Android gadget driver supports FunctionFS gadgets, which allow gadget functionality to be implemented in user space. In fact, the adbd server has been modified to support the FunctionFS interface, removing the need for an in-kernel ADB function driver.

Andrzej then similarly described ConfigFS gadget as a dynamically configurable gadget device that allows various gadget functions to be multiplexed. It is different from the Android gadget driver in that it uses ConfigFS for setting up and changing the configuration of the various functions. Andrzej talked a bit about the history of the ConfigFS gadget, noting that he originally had taken the Android gadget driver, removed anything that was Android-specific, renamed it Configurable Composite Gadget (CCG) and got it merged into staging. However, when upstream maintainers pushed for use of the ConfigFS interface, the CCG driver was abandoned and Andrzej, along with Sebastian Andrzej Siewior, focused on the ConfigFS gadget implementation. As of 3.10, the ConfigFS gadget is upstream, however it's still missing some desired functionality, like the FunctionFS interface support.

When asked if he had any issues with the ConfigFS gadget as being a potential upstream replacement for the Android gadget driver, Benoit said he had no objections. Once FunctionFS support and the other Android specific out-of-tree functions, like MTP were merged, it was just a matter of changing Android's user space code to support it.

Discussion then moved to exactly what the best approach would be for upstreaming the Android-specific gadget functions, like ADB, MTP, Android Accessory and Android audio support. Benoit mentioned again that adbd already supports FunctionFS, so once FunctionFS support is added to the ConfigFS gadget then an ADB function isn't necessary. He also said the MTP implementation could probably be done in user space, but it wasn't clear if that was the best way forward. The Android Accessory function would probably still need to be merged, but the Android Audio interface, he thought, could likely be replaced with a different audio function (though specifically which one and if it was upstream was a little unclear).

When asked if this all sounded reasonable, Andrzej simply agreed, making this one of the least contentious topics of the day. While it didn't have the lively fireworks hoped for to keep folks awake at the end of a long week, it was a nice and optimistic way to end the planned discussions.

I'd like to thank everyone for attending and participating in the discussions, as well as Zach Pfeffer and Karim Yaghmour for co-organizing the microconference, and helping with the note taking and reporting.

Comments (4 posted)

Debian moving to a CDN?

Delivering content, such as web pages and software packages, with good performance to a worldwide audience is a tricky problem. But content delivery networks (CDNs)—commercial services that cache content so that it is always "close", in network terms, to its consumers—have largely solved the problem. Debian effectively runs its own CDNs today, but that may be changing as there is discussion of moving that content to commercial providers.

Tollef Fog Heen, one of the members of the Debian System Administration (DSA) team, proposed the idea of moving "some of the static hosting that Debian currently provides from our infrastructure to one or more CDNs". Though it was not explicitly mentioned by Fog Heen, the administration of the home-rolled Debian solution is evidently becoming a burden to the DSA team—and one that some CDN companies are willing to take off their hands for free.

Fog Heen suggested a slow approach, perhaps starting with the web content at www.debian.org to, essentially, test the waters. Before taking any steps that might impact the FTP mirror network, which is a voluntary effort provided by many universities and other organizations worldwide, Debian would want to be sure that using CDNs was the right step for the distribution. In addition, he suggested that multiple CDNs be used at once, so that switching away from one that is failing or otherwise misbehaving would be relatively easy to do.

As one might guess, there was some resistance to the idea. Several questioned the use of (largely or partially) non-free software to distribute Debian. But, as Russ Allbery pointed out, CDNs can really be thought of more as infrastructure than as software. The software needed to run a CDN is likely all available in Debian, but it is the packaging and management of the tools, along with the hardware and peering arrangements, that is what CDNs are really providing.

Fog Heen agreed, noting that many CDNs do use at least some free software, but it is the ability to scale more quickly and easily than DSA can that makes a CDN attractive.

There were some more technical objections as well. Some believed that the mirror network already did a good job of getting content close to users. But Stephen Gran noted that the existing automatic "best mirror" choosing actually only gets you a mirror close to the DNS server, or on the same continent as the DNS server. There are also continents without mirrors for some content.

The speed of CDNs vs. that of a local mirror were also called into question, as well as the support for IPv6 by various CDNs. Beyond that, the Snowden revelations brought some worries to the fore. Ingo Jürgensmann listed several potential problems with CDNs regarding privacy, package integrity, and surveillance. Fog Heen agreed with some of those issues, privacy in particular, but felt that most of the same problems exist with today's system. In addition, CDNs would allow more use of HTTPS than the mirror system allows.

Many of the technical issues were already on the list for DSA to work out with potential CDNs, but the higher-level question of whether the distribution should resist moving in that direction for philosophical reasons is still somewhat up in the air. Former project leader Stefano Zacchiroli is worried about the message Debian sends when it uses a "CDN made of non-free parts". But, as Thijs Kinkhorst pointed out, Debian is already distributed on systems that may run non-free software. There is no requirement that mirrors in the network are running free operating systems. In addition, network traffic is undoubtedly routed over Cisco systems running IOS, which is somewhat akin to the role of a CDN.

Current project leader Lucas Nussbaum provided a lengthy summary of the issue that is fairly evenhanded while also giving his opinion. In particular, he looked at the Debian social contract to see what light it may shed on the philosophical question. In his interpretation, the contract doesn't really cover this issue.

Nussbaum suggested that DSA continue to look into and experiment with CDNs, but that any official web sites stay on the existing system until more is known. He listed a number of unresolved technical questions that need to be answered before any kind of official switch could be done. Though experiments using new "official" debian.org subdomains was explicitly encouraged. That's where things seem to stand. Debian may move to a CDN down the road, but there is a fair amount of technical work—and some soul searching perhaps—that needs to happen before it gets there.

Comments (5 posted)

Distribution quotes of the week

-- Lennart Poettering (in the comments)

-- Jef Spaleta (in the comments)

Comments (1 posted)

CentOS-5.10 released

The CentOS Project has released CentOS 5.10. The release notes contain the details.

Full Story (comments: none)

Debian 6.0.8 released

The Debian project has released the eighth point release of the oldstable Debian 6.0 "squeeze". This release fixes bugs and security issues.

Full Story (comments: none)

Ubuntu 13.10 (Saucy Salamander) released

The Ubuntu 13.10 release is out. "Ubuntu 13.10 introduces the first release of Ubuntu for phones and Ubuntu Core for the new 64-bit ARM systems (the "arm64" architecture, also known as AArch64 or ARMv8), and improved AppArmor confinement. In addition to these flagship features there are also major updates throughout." See the release notes for more information.

Full Story (comments: 7)

Yocto Project 1.5 now available.

Yocto Project 1.5 "Dora", built with poky-10.0.0, has been released. This release features major performance improvements, lots of new recipes, and plenty of bug fixes.

Full Story (comments: none)

Trusty Tahr open for development

The Ubuntu project has opened up development for the next Ubuntu release is known as Trusty Tahr, or 14.04 LTS. There are some toolchain updates and this is the first version to support AArch64 from the beginning of the cycle.

Full Story (comments: 3)

Distribution newsletters

• Last Week in CyanogenMod (October 19)
• DistroWatch Weekly, Issue 530 (October 21, 2013)
• Ubuntu Weekly Newsletter, Issue 339 (October 20)

Comments (none posted)

Google’s iron grip on Android (ars technica)

Here's a lengthy ars technica article describing the mechanisms by which Google maintains control of Android. "For some of these apps, there might still be an AOSP equivalent, but as soon as the proprietary version was launched, all work on the AOSP version was stopped. Less open source code means more work for Google's competitors. While you can't kill an open source app, you can turn it into abandonware by moving all continuing development to a closed source model. Just about any time Google rebrands an app or releases a new piece of Android onto the Play Store, it's a sign that the source has been closed and the AOSP version is dead."

Comments (48 posted)

Shuttleworth: Quantal, raring, saucy…

Mark Shuttleworth goes on the offensive against his critics before disclosing that Ubuntu 14.04 will be named "trusty tahr." "Mir is really important work. When lots of competitors attack a project on purely political grounds, you have to wonder what THEIR agenda is. At least we know now who belongs to the Open Source Tea Party ;) And to put all the hue and cry into context: Mir is relevant for approximately 1% of all developers, just those who think about shell development. Every app developer will consume Mir through their toolkit. By contrast, those same outraged individuals have NIH’d just about every important piece of the stack they can get their hands on… most notably SystemD, which is hugely invasive and hardly justified. What closely to see how competitors to Canonical torture the English language in their efforts to justify how those toolkits should support Windows but not Mir. But we’ll get it done, and it will be amazing."

Comments (122 posted)

Stross: The regular holy war...

Charles Stross reviews Ubuntu 13.10. "But Unity is still a mickey mouse level program launcher aimed at non-techies; it's as if the folks at Canonical have tried to copy the toy-like appearance of Mac OS X without actually providing the underlying shortcuts and power tools that make it acceptable to serious folks who like to get stuff done. Luckily salvation isn't far away: one sudo apt-get install xubuntu-desktop later and I had XFCE 4 instead of Unity. Again, some tweaking is needed; but at least it's a real desktop."

Comments (166 posted)

Ubuntu 13.10 review (ars technica)

Ars technica has a lengthy review of the Ubuntu 13.10 release. "Unity 8 is expected to arrive on the desktop in Ubuntu 14.04, at which time Unity 7 will likely be retired from further development. Due to the imminent transition to Unity 8 and Mir, Ubuntu 13.10 is sort of like the calm before the storm. It ships with the familiar old Unity 7 and Xorg—which are reliable but headed for deprecation. As Canonical’s attention is almost entirely dominated by the major technical changes coming in the near future, there wasn't a whole lot of work that surfaced in 13.10—the assortment of new features is pretty thin."

Comments (15 posted)

Where to next for GStreamer

The 2012 release of GStreamer 1.0 marked a significant achievement for the project: the multimedia framework was now a reliable "plumbing" layer used in the majority of Linux distributions and in a wide variety of multimedia products, and it had an API and ABI that it could confidently declare stable. It had even fixed most of the outstanding issues that users had complained about. But that success raises the question of where GStreamer as a project should go next. The topic came up in a number of talks at this year's GStreamer Conference, and there appears to be rough consensus that the next major set of challenges for the GStreamer team includes making development easier for the framework's downstream users.

Tim-Philipp Müller opened the conference on Tuesday with a look at the current status of the project and some ideas for taking things "to the next level" post-1.0. The 1.0 release, he said, allowed the project to fix and overhaul quite a few things that required breaking APIs or ABIs from the 0.x series. In particular, GStreamer largely fixed its memory management problems: it is more efficient, allowing zero-copy buffer management in many cases, and it is more flexible, abstracting away the low-level details of earlier releases. It also implemented the long-requested feature of dynamic pipeline construction and reconfiguration. This had always been a sore point in the past he said; it was sometimes possible to reconfigure a GStreamer pipeline in the 0.10 release, but one could never be sure if it would break. These changes also lay the groundwork for transparent support of hardware acceleration, which he said should land in GStreamer six to twelve months from now. After the 1.0 release came the 1.2 release, which did not add major features, but did include support for additional protocols and was accompanied by binary releases for Windows, OS X, Android, and iOS.

When asking where GStreamer should go next, he said, it is simple to ask "are we done?" and "will we ever be 'done'?" Those questions may not have answers, but there are less abstract questions that are more meaningful, such as "what is needed to keep GStreamer relevant and interesting to developers?" There are a few technical features that are still missing, he said, but most of the challenges are not of a technical nature. On the technical front, for example, there is still work underway to allow sandboxing of GStreamer pipelines—which is good for security and will help to isolate crashes. There is also interest in allowing a mechanism to connect GStreamer to a DRM system; "we don't like DRM," he said, "but manufacturers need it" and it is perceived by them as a major gap between GStreamer and other frameworks. There are also 3D features still to come (including stereoscopic video and handling depth information like the Microsoft Kinect camera captures), as well as Blu-Ray disc support and improvements to subtitle handling.

That may sound like a long list of technical challenges, but none of them are really unsolved problems or tricky challenges (with the possible exception of transparent access to hardware acceleration, which can be quite tricky to implement across the variety of embedded boards). Most of them will be handled by development work that is already underway. The bigger challenges are working more effectively with the developer community. This includes several things, he said: quality assurance (QA), better development tools, engagement with the GStreamer community, and routine maintenance.

The QA challenge is one of the biggest, Müller said. It is not so much that GStreamer lacks quality, he said, as it is that the project needs to work at preventing regressions. In "the good old days," he pointed out, Nokia did the project a major service by coordinating QA. Since Nokia dropped GStreamer, however, no one has really taken up that slack. Consequently, most bugs are discovered by application developers when their own code breaks, or they are discovered by outside companies making GStreamer-based products—which means, of course, that the bug report sits in a private issue tracker and may not be reported upstream for quite some time. Some bugs are also found by Linux distribution build-bots, he said, but even those are mostly just automated unit tests. While make check is useful, he said, it does not reveal bugs found with real media files or interactive usage.

GStreamer needs a proper test suite, he said, one that comes with installable tests (including tests that can be run remotely for when GStreamer is cross-compiled for embedded devices). It also needs nightly builds and tests for extreme conditions, such as running a video capture job for 36 hours (at the end, will the audio and video still be synchronized?). These things require a massive effort to be done correctly, but they also require a lot of "social" buy-in from GStreamer companies: QA tools cannot be allowed to bit-rot, and the companies involved must also care when a bug is uncovered by QA testing, or else they will eventually lose interest in the process.

Developer tools needed by GStreamer start with debugging tools, he said. Certainly GStreamer applications can be debugged today, but only "if you know where to look." Developers need tools that will allow them to take snapshots of pipelines, to trace pipelines (such as to discover where it is in a pipeline that a media frame gets dropped), and to hook their debugging tools into embedded devices. The project also needs to work on tools for Windows developers and other on non-Linux OSes, since those developers will probably not be using GDB. There is some progress on this task, he said; a lot of individuals write their own tools and GStreamer has started to collect them in a repository, but more is still needed.

Engaging with the GStreamer-using community takes several forms, he said. The project needs to encourage more people to work upstream. Too many companies (including system-on-chip (SoC) vendors) write their own plugins in order to check the "GStreamer support" box on their packaging, but they do not actively cooperate with the project. As a result, many of them end up "solving" the same problems over and over again. The project also needs far more documentation: there is a shortage of high-level application design documentation, and there is a long-planned "GStreamer book" that has never materialized. Müller also noted that a lot of people seem confused by the GStreamer SDK effort, which is not a direct product of the GStreamer project (although it was put together by people who are project participants).

Finally, he said, the project needs to take on maintenance in a more deliberate way. It cannot be simply an "after hours" task. Currently, according to the Ohloh statistics, 50% of the project's maintenance fixes are done by three individuals. It is good that the right people are handling maintenance, but more people would be better (as well as more robust against vacation interruptions).

Several of these non-technical challenges have partial solutions already, Müller said, so they are more a matter of focus and execution. One recurring issue in all of them is that it is currently difficult for an interested company to contribute back to GStreamer—outside of assigning an employee to work on the code, or sponsoring the annual GStreamer Conference. Perhaps forming a non-profit organization would help, he said, but that is a big question deserving careful consideration. Ultimately, he concluded, GStreamer is much better technically than it was one year ago, its ecosystem is growing, and there are a lot of companies and projects that have a long-term interest in the project's success—the project just needs to make better use of its strengths in those areas.

Developers, developers, developers

The theme of providing better tooling and support for downstream developers was picked up in the second day's keynote by Wim Taymans. Taymans talk addressed common pitfalls and best practices for developing GStreamer-based products, and many of his observations related to how companies using GStreamer either did or did not engage with the development community. For example, his first observation was that companies often develop their own plugins "on an island" rather than working directly with the project. There are valid reasons for this approach, but it causes complications. Even when a company develops its plugin as an open source component, he said, if they do it in their own GitHub repository lots of people clone and fork it before it is ready. After the fact, it becomes very difficult to locate the original, canonical version of the plugin. If GStreamer hosted a repository for downstream plugin developers, Taymans said (which it is considering), that might alleviate the headache.

Other companies get into trouble by starting their projects off writing their own multimedia layer from scratch ("or writing several of them," he added), then trying to shoehorn the existing codebase into GStreamer when they give up on their own framework. Often the mapping between the existing code and GStreamer is imperfect, or the code does not expose everything that GStreamer needs to operate (such as assuming only one particular media format). The GStreamer community can help if it is asked, he said, but it can only help if it knows about the problem. Similarly, other companies make the mistake of selecting a hardware platform first (such as a SoC), then have to do lots of extra work to make GStreamer run on it when they discover that the only GStreamer plugins for the board are untested or limited in features. Here again, the community would be happy to help, but it often does not find out about these difficulties until late in the process.

The community could help overcome some of these pitfalls with better documentation, he said: insufficient knowledge of GStreamer leads a lot of companies to create inefficient pipelines or build ugly hacks because they are unaware that a feature already exists. There are also a lot of cases where companies fork GStreamer in order to implement their own features, but this ultimately leaves them trapped on an old release when they do not know how to migrate to the latest version of GStreamer. Forks are encouraged, he said, but companies should at least open a bug in the GStreamer issue tracker when they encounter a problem; the community might implement a fix. Better testing tools would also relieve some of the pain of migrating between releases.

Taymans also observed that the non-technical challenges have become GStreamer's next big target. Companies frequently say that GStreamer is hard to debug and test. In particular, developers often test a format decoder only with a small handful of test files, when what they really need is a test harness that can automatically test all of the features of a proper decoder. He also mentioned the documentation problem, noting that too much GStreamer behavior is underspecified—knowledge is available only through "common understandings" between experienced GStreamer developers, and too much of the GStreamer API is "documented" only by the example plugins that ship with the releases. He also cited the need for more high-level base classes that provide useful abstractions, and suggested that the project needs to draw in a number of out-of-tree plugins, keeping them closer to GStreamer itself. That makes them easier for other developers to find, and helps the project maintain a better overview of what downstream projects are doing and what they need.

A lot of the challenges discussed during this year's conference indicate normal growing pains for a useful project; GStreamer has evolved into a reliable infrastructure project. That in turn means that fewer resources will be needed to develop wild new features, but more resources will be needed to test, refine, and battle-harden the code for frequent use. It is a good problem to have, but it may not be an exciting one to talk about. After all, the conventional wisdom is that new developers are attracted to writing interesting features—far fewer are attracted to a project for maintenance duties. However, if the talks at this year's conference are anything to judge by, there are a lot of interesting GStreamer-based projects downstream needing consultation—more than enough to attract the time and attention of developers for several years to come.

[The author would like to thank the Linux Foundation for travel assistance to Edinburgh for GStreamer Conference 2013.]

Comments (1 posted)

Quotes of the week

— Twitter user I Am Developer (hat tip to Matt Ray)

— Xiph.org's Greg Maxwell, at GStreamer Conference 2013, discussing the relative interesting-ness of Opus and Daala.

Comments (none posted)

Wireshark switches to Qt

Beginning with Wireshark 1.11.0 the project has switched its user interface library from GTK+ to Qt. "Both libraries make it easy for developers write applications that will run on different platforms without having to rewrite a lot of code. GTK+ has had a huge impact on the way Wireshark looks and feels and on its popularity but it doesn’t cover our supported platforms as effectively as it should and the situation is getting worse as time goes on." (Thanks to Matthias Berndt)

Comments (43 posted)

XBMC fork announced

XMBC media center developer Arne Morten Kvarving has announced a new fork of the project, titled (at least for now) FYMC. The code is currently available on GitHub for FreeBSD and Linux, with the addendum that "support for building on other free platforms will follow, with free being interpretable as platforms where users maintain their freedom to run and modify software after their own likings". As of now, the primary differences are a move to GPLv3, a CMake build system, and "an agile approach" to development.

Full Story (comments: 3)

OpenStack 2013.2 "Havana" available

OpenStack 2013.2 "Havana" has been released. A long list of new features is included on the OpenStack wiki, covering more than 400 feature blueprints.

Full Story (comments: none)

libguestfs 1.24 released

Version 1.24 of libguestfs, the tool suite for modifying filesystems on virtual machine images, has been released. This version includes several new features, such as support for User Mode Linux as a backend, inclusion of a new tool for building VM images, systemd support, and bindings for Go.

Full Story (comments: none)

Development newsletters from the past week

• Caml Weekly News (October 22)
• What's cooking in git.git (October 18)
• GNU Toolchain Update (October 21)
• Haskell Weekly News (October 17)
• OpenStack Community Weekly Newsletter (October 18)
• Perl Weekly (October 21)
• PostgreSQL Weekly News (October 20)
• Ruby Weekly (October 17)
• Tor Weekly News (October "23th")

Comments (none posted)

Eich: The Bridge of Khazad-DRM

At his blog, Mozilla's Brendan Eich weighs in on the W3C's move to approve Encrypted Media Extensions for HTML5. He notes that "CDMs are analogous to ActiveX components from the bad old days: different for each OS and possibly even available only to the OS’s default browser," and hints at future work still to come on watermarking as a superior alternative to DRM schemes.

Comments (none posted)

The Open Virtualization Alliance moves to the Linux Foundation

The Linux Foundation has announced that it will be hosting the Open Virtualization Alliance, the home of the KVM project. "As many folks already know, The Xen Project is also a Linux Foundation Collaborative Project. We love that a number of open source projects can emerge to address similar questions to complex technology problems. Competition gives way to collaboration and cross-pollination, and everyone benefits. The more open source code, the better for everyone. Historically in open source we've witnessed that competition and collaboration among open source projects can advance and accelerate technology."

Comments (none posted)

OSI Names New General Manager

The Open Source Initiative (OSI) has announced that it has appointed Patrick Masson as General Manager. "Masson is the organization's first full-time management appointment in its fifteen year history. His hiring will support and supplement the organization's volunteer community."

Comments (none posted)

Programming Erlang, 2nd Edition--New from Pragmatic Bookshelf

Pragmatic Bookshelf has released "Programming Erlang, 2nd Edition" by Joe Armstrong.

Full Story (comments: none)

CFP: SciPy India 2013

SciPy India will take place December 13-15 in Bombay. The call for proposals is open until November 24.

Full Story (comments: none)

LibrePlanet 2014 seeks session proposals, volunteers

LibrePlanet will be held in March in Cambridge, MA. Exact dates are not yet available. The call for session proposals is open as is a call for volunteers. The deadline is November 15.

Full Story (comments: none)

CFP Deadlines: October 24, 2013 to December 23, 2013

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
November 1	January 6	Sysadmin Miniconf at Linux.conf.au 2014	Perth, Australia
November 4	December 10 December 11	2013 Workshop on Spacecraft Flight Software	Pasadena, USA
November 15	March 18 March 20	FLOSS UK 'DEVOPS'	Brighton, England, UK
November 22	March 22 March 23	LibrePlanet 2014	Cambridge, MA, USA
November 24	December 13 December 15	SciPy India 2013	Bombay, India
December 1	February 7 February 9	devconf.cz	Brno, Czech Republic
December 1	March 6 March 7	Erlang SF Factory Bay Area 2014	San Francisco, CA, USA
December 2	January 17 January 18	QtDay Italy	Florence, Italy
December 3	February 21 February 23	conf.kde.in 2014	Gandhinagar, India
December 15	February 21 February 23	Southern California Linux Expo	Los Angeles, CA, USA

If the CFP deadline for your event does not appear here, please tell us about it.

Events: October 24, 2013 to December 23, 2013

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
October 22 October 24	Hack.lu 2013	Luxembourg, Luxembourg
October 23 October 25	Linux Kernel Summit 2013	Edinburgh, UK
October 23 October 24	Open Source Monitoring Conference	Nuremberg, Germany
October 24 October 25	Embedded LInux Conference Europe	Edinburgh, UK
October 24 October 25	Xen Project Developer Summit	Edinburgh, UK
October 24 October 25	Automotive Linux Summit Fall 2013	Edinburgh, UK
October 25 October 27	Blender Conference 2013	Amsterdam, Netherlands
October 25 October 27	vBSDcon 2013	Herndon, Virginia, USA
October 26 October 27	T-DOSE Conference 2013	Eindhoven, Netherlands
October 26 October 27	PostgreSQL Conference China 2013	Hangzhou, China
October 28 November 1	Linaro Connect USA 2013	Santa Clara, CA, USA
October 28 October 31	15th Real Time Linux Workshop	Lugano, Switzerland
October 29 November 1	PostgreSQL Conference Europe 2013	Dublin, Ireland
November 3 November 8	27th Large Installation System Administration Conference	Washington DC, USA
November 5 November 8	OpenStack Summit	Hong Kong, Hong Kong
November 6 November 7	2013 LLVM Developers' Meeting	San Francisco, CA, USA
November 8	PGConf.DE 2013	Oberhausen, Germany
November 8	CentOS Dojo and Community Day	Madrid, Spain
November 8 November 10	FSCONS 2013	Göteborg, Sweden
November 9 November 11	Mini DebConf Taiwan 2013	Taipei, Taiwan
November 9 November 10	OpenRheinRuhr	Oberhausen, Germany
November 13 November 14	Korea Linux Forum	Seoul, South Korea
November 14 November 17	Mini-DebConf UK	Cambridge, UK
November 15 November 16	Linux Informationstage Oldenburg	Oldenburg, Germany
November 15 November 17	openSUSE Summit 2013	Lake Buena Vista, FL, USA
November 17 November 21	Supercomputing	Denver, CO, USA
November 18 November 21	2013 Linux Symposium	Ottawa, Canada
November 22 November 24	Python Conference Spain 2013	Madrid, Spain
November 25	Firebird Tour: Prague	Prague, Czech Republic
November 28	Puppet Camp	Munich, Germany
November 30 December 1	OpenPhoenux Hardware and Software Workshop	Munich, Germany
December 6	CentOS Dojo	Austin, TX, USA
December 10 December 11	2013 Workshop on Spacecraft Flight Software	Pasadena, USA
December 13 December 15	SciPy India 2013	Bombay, India

If your event does not appear here, please tell us about it.