LWN.net Weekly Edition for April 19, 2012

LFCS 2012: The Linux System Definition

Open Invention Network (OIN) CEO Keith Bergelt came to the Linux Foundation Collaboration Summit to update attendees on OIN's recent expansion of the "Linux System Definition" as well as its plans for the future. The system definition is important because the packages listed there are eligible for patent protection via OIN and its members' patents. The update was significant, but there are other interesting things going on at OIN as well.

History

OIN was formed nearly eight years ago, "in the wake of SCO", Bergelt said. SCO "went away", but that case served as a wakeup call to some of the companies that were making big commitments to Linux. They recognized that patents could end up being a real problem for Linux. Three companies in particular, IBM, Red Hat, and Novell, got together to try to combat that problem; they were later joined by NEC, Phillips, and Sony. Those companies invested "hundreds of millions" of dollars in a standing fund to acquire patents.

In other businesses, there are a fixed number of competitors that one can build up patent cross-licensing agreements with. But Linux has "neither a head or a tail", Bergelt said, so it can't really handle patents in the standard way. The companies realized that Linux is not a model that will deal well with patent attacks. The idea behind OIN was that those companies would share their patents on a "fully paid-up basis" (i.e. without requiring royalties) among themselves, but also with others who joined the organization.

The basic idea, Bergelt said, was to allow freedom of choice for those who might be looking to adopt Linux by removing some of the patent uncertainty. If Linux is not the best choice, then "shame on the developers", but in most cases it is the best choice when it can compete fairly. Open source does not operate in a vacuum, but is faced with a world where litigation is a vehicle for companies that cannot compete in the marketplace, he said.

OIN activities

OIN has hundreds of patents in "dozens of portfolios", he said, some of which are fundamental or essential in "several key areas". It does 45-50 patent applications each year, as well as filing multiple defensive publications. The defensive publications are meant to combat aggressive patenting by companies outside of the Linux ecosystem, which often results in them "patenting things that open source had developed long ago". These efforts are meant to deal with the excessive number of patents that have been granted in the last 20 years.

The patents that read on the packages that make up the Linux System Definition are available to OIN licensees on a royalty-free basis. That includes the patents that OIN owns as well as any that are owned by the licensees. In addition, licensees must "forbear Linux-related litigation" using their patents, he said.

OIN is also involved in defensive activities of various sorts. It gives assistance to companies that are undergoing patent attacks. That includes finding people that can help the company fend off the attack or for OIN to help them directly. Giving companies ideas of where they might be able to acquire useful patents to head off an attack, consulting on how to create a formidable defense, and offering up information about prior art are the kinds of things that OIN will do. Typically the attack is really not against just the one company, but is part of a larger strategy from the patent aggressor, which means that the information gathered and shared may be able to be reused at some point.

There is also the Linux Defenders project which OIN helped to start with the Software Freedom Law Center and the Linux Foundation four years ago. It seeks to work with the community to identify prior art for patent applications to potentially cause them to be rejected. In addition, it tries to invalidate patents that have already been granted by finding prior art. OIN is considering becoming "much more aggressive" in those endeavors, Bergelt said, and may be hiring more people to work on that. Part of that effort would be to work with projects and community members to help "defuse the time bomb" that patents held by foes represent.

There are currently over 440 OIN licensees, he said. The organization holds 400+ US patents and 600+ when foreign patents are added in. It has also created more than 250 defensive publications.

Those defensive publications generally aren't created by project members, Bergelt said, in answer to a question from the audience. OIN does not want community members to have to become experts in defensive publications and, instead, will do the "heavy lifting" to create them. Anything that a developer thinks is "new or novel" has the potential to be written up as a defensive publication. The bar is "not very high", he said, as the patents that are granted will attest. The bar is even lower for defensive publications.

Community members do not have to codify their idea, if they can just verbalize it, OIN can turn that into a defensive publication. Those publications can then work as an "anti-toxin" of sorts against bad patents and applications. OIN has been trying to get graduate students involved, as well, to work on the publications. It is "inventing in a way that prevents others from getting bad patents". One question that has come up in conversations with Bradley Kuhn, he said, is if we eliminate all of the bad patents, are we left with really good patents? The answer is that what's left after getting rid of the bad patents is "very limited".

Expanding the Linux System Definition

In its effort to "demilitarize the patent landscape", OIN expanded the Linux System Definition in March. The number of packages covered rose from 1100 to 1800 and will grow further over time, he said. OIN is looking at doing yearly updates to the definition, but it may do a mobile Linux update this (northern hemisphere) summer. There are already some packages that have been nominated for inclusion and he encouraged everyone to look at the definition and lists of packages in order to nominate those they thought should be included.

From the audience, Kuhn asked about whether the decisions on adding or not adding packages could be made transparent, so that the community could get feedback on its nominees. Bergelt said that OIN will explain its decisions, and will be putting up a web page for nominations sometime soon. He said that Simon Phipps had already recommended LibreOffice, for example, since Linux distributions have moved to that office suite.

One of the main efforts that OIN is now making is to reach out to the community. He said that while he wears a suit, many of the people that work for him do not and are members of the community. The organization is attending more events, and trying to involve the community in its efforts. In addition, it is doing more workshops, rather than talks, to try to get the community up to speed on the defensive publication anti-toxin. Attending various events has "made me realize what Linux is about", he said, which is "people sharing ideas". Those ideas can get "codified as either swords or shields".

OIN has expanded its staff recently as well. Deb Nicholson was named as the community outreach director for OIN. In addition, Linux Defenders has added Andrea Casillas as its director and Armijn Hemel as its European coordinator.

Unique economic benefits

OIN is also leading an effort to highlight the "unique economic benefits from open source" to the International Trade Commission (ITC). Patent trolls and others are increasingly bringing complaints of patent infringement to the ITC, but the only remedy it has available is to order an injunction against the import of the product. Those injunctions can "put you out of business" because it can take $8-12 million to defend against an ITC injunction. That process takes a year, he said, which sounds good in comparison to patent suits which often stretch much longer, but it means that a company needs "lots of fast money" to defend themselves, so most are willing to settle for a much higher license fee than the market would normally bear.

Those higher license fees are artificially raising the total cost of ownership (TCO) for the Linux platform. It is a "purposeful process" to take away choice by raising the cost of running the Linux platform. OIN is attempting to have the US Congress recognize this and instruct the ITC to allow a parallel District Court process hear the case, as it can make decisions other than just injunctions. Paying some license fee is "not a death knell", Bergelt said, while unfairly high costs are. We want a "yield sign if not a stop sign" to the ITC-ordered injunctions.

Another initiative that OIN is working on with Google and IBM is something that Bergelt is (temporarily) calling "Code Search". It will be a way to search pre-existing code and "unstructured project data" for prior art. It will make existing code searchable and could be used by the patent examiners as well as the community and OIN. There will be an announcement "soon" about that project.

Context

Bergelt then turned to the current patent wars, and what they mean for Linux, especially in the mobile arena. Microsoft has been using 25 patents in its litigation over mobile devices. It is not just selling licenses to smartphone vendors, but is going after the contract manufacturers as well. Both are paying license fees in many cases, so Microsoft is now turning to going after the mobile carriers as well.

There is a strategic agenda at play, he said. In difficult times, companies that appear to be strange bedfellows will come together to attack a rival. The attacks are tightly coordinated and multi-tiered to artificially raise the TCO. Established companies are comfortable with reduced innovation because that means there are fewer threats to their positions.

There are lots of speculators out there who have spent "billions to acquire patents" and expect a return on that investment. But there are also operating companies that are threatened by Linux and open source. He is wondering what will happen with HP's Palm patent portfolio, for example. Patent aggregators are being approached to sell their portfolios to operating companies; those companies may want them for defensive or offensive purposes. In addition, we have seen things like Microsoft's investment that ensured Nokia's relationship with MeeGo ended. Mobile Linux is being attacked from many different directions at this point.

But, efforts like Tizen, webOS, and MeeGo (which is still being used by KDE and others) add resilience to the mobile Linux patent situation. It is much easier for foes of Linux to fight a one front war against Android, rather than have to deal with Tizen, webOS, and MeeGo, he said. He is "very encouraged" to see that there are multiple options for mobile Linux.

There are lots of antagonists lined up against Linux. Some companies are funding patent trolls and providing them with patents to attack other companies. But, so far, we haven't seen funding for direct attacks by patent trolls against Linux, though it could happen. The key is to watch where trolls are using patents from known Linux antagonists, those kinds of attacks could turn to Linux next. There is a lot of litigation activity right now, but he sees things moving in a direction where eventually choice in the market will win out over attempts to artificially raise prices through patent attacks.

Bergelt painted a picture of a complex and active patent attack landscape, particularly against mobile devices. But he also described lots of things that OIN and others are doing to combat those attacks. Reform at the ITC level could effect some major changes to the tactics that are currently being employed, though it is unclear how likely that reform actually is. Until there is some kind of major patent overhaul in the US (and elsewhere, really), the OIN projects and efforts will clearly be needed. Whether they will be enough, eventually, remains to be seen.

Comments (8 posted)

LFCS 2012: The future of GLIBC

The core library that sits between user space and the kernel, the GNU C library (or GLIBC), has undergone some changes recently in its governance, at least partly to make it a more inclusive project. On the last day of the Linux Foundation Collaboration Summit, Carlos O'Donell gave an update on the project, the way it will be governed moving forward, and its plans for the future. GLIBC founder Roland McGrath was on hand to contribute his thoughts as well.

Though he wears several hats, O'Donell introduced himself as an "upstream GLIBC community member", rather than as a maintainer, because the GLIBC developers have recently been trying to change the idea of what it means to be involved in the project. He works for Mentor Graphics—by way of its acquisition of CodeSourcery—on open-source-based C/C++ development tools. Those tools are targeted at Linux developers and Mentor is committed to working upstream on things like GCC and GDB, he said. He personally got involved in the GLIBC project ten years ago to support the PA-RISC (hppa) architecture; he now works on GLIBC both as part of his job and as a volunteer.

Recent changes in the GLIBC community

The changes for GLIBC had been coming for a long time, O'Donell said. The idea is to transition the project from one that has a reputation for being a small community that is hard to work with to one that will work well with the kernel developers as well as other projects in the free software world.

As part of that effort, the moderation of the main GLIBC mailing list (libc-alpha) was removed after four years. The goal of that moderation had been to steer new contributors to the libc-help mailing list so that they could learn about the open source (and GLIBC) development process before they were exposed to the harsher libc-alpha environment. The mentoring process that was done on libc-help has continued; it is a place for "random questions" about GLIBC (both for users and new contributors), while libc-alpha is for more focused discussion and patches once developers have a firm understanding of the process and culture. There has also been a lot of "wiki gardening" to make more internal documentation of GLIBC available, he said.

The most visible recent change was the dissolution of the steering committee in March. The project is moving to a "self-governed community of developers" that is consensus driven, he said. There is a wiki page that describes what the project means by "consensus". Trivial or typo patches can just be checked into the repository, without waiting for approval. The GLIBC community is willing to accept all sorts of patches now, he said, which is a "change from where we were five years ago". All of the changes in the community have come about as gradual process over the last four or five years; there was no "overnight change", he said.

There are around 25-30 committers for GLIBC, O'Donell said in response to a question from the audience, and they are listed on the wiki. Ted Ts'o then asked about getting new features into GLIBC, noting that in the past there was an assumption that trying to do so was not worth the effort. He pointed out that BSD union mounts got help from its libc, but that couldn't be done for Linux in GLIBC, partly because it was not in the POSIX standard. What is the philosophy that is evolving on things like that, he asked.

O'Donell said that it comes down to a question of "relevance"; if there are features that users want, the project may be willing to accept things that are not in POSIX. GLIBC is the layer between programs and the kernel, so if there are things missing in that interface it may make sense to add them. If GLIBC fails to provide pieces that are needed, it will eventually not be relevant for its users. For example, he said, there is a lot of work going on in tracing these days, but GLIBC has not been approached to expose the internals of its mutexes so that users are better able to debug problems in multi-threaded programs; things like that might make good additions. But, "we are conservative", he said.

Ts'o then mentioned problems that had occurred in the past in trying to expose the kernel's thread ID to user space. There has been a huge amount of work done to get that information, which bypassed GLIBC because of the assumption that GLIBC would not accept patches to do so. People are working around GLIBC rather than working with it, he said.

There is no overriding philosophy about what changes would be acceptable, McGrath said. Much like with the kernel, features will be evaluated on a case-by-case basis. There is a need to balance adding something to every process that runs all over the world and adding interfaces that will need to be supported forever against the needs and wishes of users. Things that have "bounced off" GLIBC in the past should be brought up again to "start the conversations afresh". But don't assume that it will be easy to get your pet feature into GLIBC, he said.

With 25-30 committers for the project, how will competing philosophies among those people be handled, Steven Rostedt asked. That problem has not been solved yet, O'Donell said. At this point, they are trying to "bootstrap a community that was a little dysfunctional" and will see how it works out. If problems crop up, they will be resolved then. McGrath said that things will be governed by consensus and that there won't be "I do it, you revert it, over and over" kinds of battles. In addition, O'Donell said, that in a Git-based world reverts won't happen in that way because new features will happen on branches.

Standards

The most static part of GLIBC is the portion that implements standards, O'Donell said, moving on to the next part of his talk. Standards support is important because it allows people and code to move between different architectures and platforms. The "new-ish" standards support that the GLIBC community is working on now is the C11 support, which he guesses will be available in GLIBC 2.16 or 2.17. One of the more interesting features in C11 is the C-level atomic operations, he said. Some of the optional annexes to C11 have not been fully implemented. Ulrich Drepper is also working on conformance testing for POSIX 2008 and any problems that are found with that will need to be addressed, O'Donell said.

There are no plans to add the C11 string bounds-checking interfaces from one of the annexes as there are questions about their usefulness even within the standards groups. That doesn't mean that those interfaces couldn't end up in the libc_ports tree, which provides a place for optional add-ons that aren't enabled by default. That would allow distributions or others to build those functions into their version of GLIBC.

The math library, libm, is considered "almost complete" for C11 support, though there are a "handful" of macros for imaginary numbers that are missing, but Joseph Myers is working on completing those. All of the libm bugs that have been reported have been reviewed by Myers; he and Andreas Jaeger are working on fixing them, O'Donell said. Some functions are not rounding correctly, but sometimes fixing a function to make it right makes it too slow. Every user's requirements are different in terms of accuracy vs. speed, so something needs to be done, but it is not clear what that is. Bugs filed in bugzilla are being worked on, though, so he asked that users file or reopen bugs that need to be addressed.

Short-term challenges

O'Donell then moved on to the short-term issues for the project, which he called "beer soluble" problems because they can be fixed over a weekend or by someone offering a case of beer to get them solved; "the kind of thing you can get done quickly". First up is to grow the community by attracting more developers, reviewers, and testers. The project would also like to get more involvement from distributions and, to that end, has identified a contact person for each distribution.

Part of building a larger community is to document various parts of the development process. So there is information on the wiki about what constitutes a trivial change, what to do when a patch breaks the build, and so on. The idea is that the tree can be built reliably so that regression testing can be run frequently, he said.

The release process has also changed. For a while, the project was not releasing tarballs, but it has gone back to doing so. It is also making release branches early on in the process, he said. GLIBC 2.15 was released on March 21 using the new process. There will be an 2.15.1 update at the end of April and the bugs that are targeted for that release are tagged with "glibc_2.15". In addition, they have been tagging bugs for the 2.16 release and they are shooting for twice-a-year releases that are synchronized with Fedora releases.

Spinning out the transport-independent remote procedure call (TIRPC aka Sun RPC) functions into a separate library is an example of the kinds of coordination and cooperation that the GLIBC project will need to do with others, he said. Cooperation with the distributions and the TIRPC project is needed in order to smooth that transition. There have been some "teething problems" with the TIRPC transition, like some header file overlaps in the installed files. Those problems underscore the need to coordinate better with other projects. It's "just work", he said, but cooperating on who is going to distribute which header and configuration files needs to happen to make these kinds of changes go more smoothly.

Medium-term challenges

The medium-term problems for the project were called "statistically significant" by O'Donell because the only way to solve them is to gather a bunch of the right people together to work on them. A good example is the merger of EGLIBC and GLIBC. The fork of GLIBC targeted at the embedded space has followed all of the FSF copyright policies, so any of that code can be merged into GLIBC. He is "not going to say that all of EGLIBC" will be merged into GLIBC, but there are parts that should be. In particular, the cross-building and cross-testing support are likely to be merged. Another area that might be useful are the POSIX profiles that would allow building the library with only certain subsets of its functionality, which would reduce the size of GLIBC by removing unneeded pieces.

In answer to a question from Jon Masters, O'Donell said that new architecture ports should target GLIBC, rather than EGLIBC. Though if there is a need for some of the EGLIBC patches, that might be the right starting point, he said.

The GLIBC testing methodology needs to enhanced. For one thing, it is difficult to compare the performance of the library over a long period of time. The project gets patches to fix the performance of various parts, but without test cases or benchmarks that could be used down the road to evaluate new patches. Much of the recent work that has gone into GLIBC is to increase performance, so it is important to be able to have some baselines to compare against.

The testing framework also needs work. It is currently just a test skeleton C file, though there have been suggestions to use DejaGNU or QMTest. The test infrastructure in GLIBC is not the "most mature" part of the project. It should be, though, because if the project is claiming that it is conservative, it need tests to ensure that things are not breaking, he said.

More independent testing is needed, perhaps using the Linux Test Project or the Open POSIX test suite. Right now Fedora is used to do full user-space rebuild testing, but it would be good to do that with other distributions as well. Build problems are easy to find that way, but runtime problems are not.

Long-term challenges

In the next section of the talk, O'Donell looked at ideas for things that might be coming up to five years out. No one can really predict what will happen in the kind of time frame, he said, which is why he dubbed it "mad science". One area that is likely to need attention is tracing support. Exposing the internal state of GLIBC for user-space tracing (e.g. SystemTap, LTTng, or other tools) will be needed.

Another idea is to auto-generate the libm math library from a C code description of "how libm should work". There is disappointment in the user community because the libm functions have a wide spectrum between "fast and inaccurate" and "slow and accurate" functions. Auto-generating the code would allow users to specify where on that spectrum their math library would reside.

One last idea that he "threw in for fun" is something that some researchers have been talking to the project about: "exception-less system calls". The idea is to avoid the user to kernel space transition in GLIBC by having it talk to "some kind of user-space service API" that would provide an asynchronous kernel interface, rather than doing a trap into the kernel directly.

To close out his talk, O'Donell stressed that the project is very welcoming to new contributors. He suggested that if you had a GLIBC bug closed or submitted a patch and never heard back, then you should get involved with the project as it will be more open to working with you than it may have been in the past. If you have GLIBC wishlist items, please put them on the wiki; or if you have read a piece of code in GLIBC and "know what it does", please submit a comment patch, he said.

Questions and answers

With that, he moved onto audience questions, many of which revolved around the difference between the glibc_core and glibc_ports. The first was a question about whether it made sense to merge ports into the core. O'Donell said that the two pieces have remained close over the years and essentially live in the same repository, though they are split into two Git trees. There is no real need to merge them, he said, but if it was deemed necessary, it could be done with a purely mechanical merge. Ports is meant as an experimental playground of sorts, that also allows users to pick add-ons that they need.

That "experimental" designation would come back to haunt O'Donell a bit. An audience member noted that the Sparc version of GLIBC lives in the core, while ARM (and others) lives in ports. McGrath said that was really an accident of history. Ports helps ensure that the infrastructure for add-ons doesn't bitrot, he said. "ARM is by no means a second-class citizen" in GLIBC, O'Donell added. The ports mechanism allows vendors to add things on top of GLIBC so keeping it working is worthwhile.

But the audience reminded O'Donell of his statement about ports being experimental, and that it might give the wrong impression about ARM support. "I'm completely at fault", he responded, noting that he shouldn't have used "experimental" for ports. With a bit of a chuckle, McGrath said: "That's the kind of statement GLIBC maintainers now make".

At the time of the core/ports split, all of the architectures that didn't have a maintainer were put into ports, McGrath said. Now it is something of an "artificial distinction" for architectures, O'Donell said. Ts'o suggested that perhaps all of the architectures should be in ports, while the core becomes architecture-independent to combat the perception problem. O'Donell seemed amenable to that approach, as did McGrath, who said that it really depends on people showing up to do the work needed to make things like that happen.

Another question was about the "friction" that led to the creation of EGLIBC; has that all been resolved now? O'Donell said that the issues haven't been resolved exactly, but that there are people stepping up in the GLIBC community to address the problems that led to the split. There may still be some friction as things move forward, but they will be resolved by technical arguments. If a feature makes sense technically, it will get merged into GLIBC, he said.

The last question was about whether there are plans to move to the LGPLv3 for GLIBC. McGrath said that there is a problem doing so because of the complexity of linking with GPLv2-only code. The FSF would like to move the library to LGPLv3, but it is committed to "not breaking the world". There have been some discussions on ways to do so, but most GLIBC developers are "just fine" with things staying the way they are.

The talk clearly showed a project in transition, with high hopes of a larger community via a shift to a more-inclusive project. GLIBC is an extremely important part of the Linux ecosystem, and one that has long suffered from a small, exclusive community. That looks to be changing, and it will be interesting to see how a larger GLIBC community fares—and what new features will emerge from these changes.

Comments (8 posted)

Updates on Flash support for Linux

GPLv3-licensed Flash player Lightspark has released its latest update, version 0.5.6. The new release includes expanded media support, experimental support for the Google Maps "Street View" feature, and a browser plug-in compatible with Firefox 10. Despite the new features, however, the prospect of an open source player for all Flash content does not appear to be getting any closer.

"Flash" as a file format is a complicated beast, of course. Each Flash release incorporates changes to the supported media types, rendering subsystems, application components, and often the JavaScript-like ActionScript programming language itself. Flash 9, released in 2006, introduced an entirely new ActionScript Virtual Machine, AVM2. Lightspark was started in 2009 by Alessandro Pignotti, targeting support for AVM2 and ActionScript 3.

But despite introducing a new virtual machine, Adobe did not remove support for AVM1 code from Flash 9 (or subsequent releases), for fear of breaking existing Flash applications. Lightspark version 0.4.3 overcame this limitation by introducing a fallback mechanism that would call on the stand-alone version of Gnash, the GNU Flash player supporting AVM1, whenever it encountered AVM1 files. Lightspark 0.5.0 introduced a host of new features, but by and large new releases in recent years have incorporated fixes designed to support specific Flash-based web sites or applications.

What's new in Lightspark

Such is the case with the 0.5.6 release, which boasts fixes for YouTube and "experimental" support for Google Maps Street View, plus new features implemented to support Flash-based games like FarmVille. Lightspark includes two front-ends, a stand-alone GTK+ player and an NPAPI web browser plug-in. The new release is compatible with Firefox 10 (which itself landed in January 2012), and is the first release with support for using PNG files. For now, source code is the only downloadable incarnation of the release, but the project maintains a personal package archive (PPA), so Debian packages should arrive soon enough.

Of course, the fact that a Flash game is the motivator for a particular feature implementation in no way lessens its importance. In this case, the new features include support for Flash's NetConnection, which is a two-way client-server RPC channel, and support for custom serialization and deserialization of data, which is likely to prove helpful for other client-server Flash applications relying on JSON or XML, too.

We last looked at Lightspark in 2010; the intervening releases have added much more that is worthy of note than does the 0.5.6 bump on its own. Many more sites are supported, including Vimeo and Flowplayer (which is a web video playback product, not a site itself), and the project now uses the Tamarin test suite to run tests for correctness. While Pignotti took a break, Canonical's Jani Monoses served as release manager and improved ARM support for embedded devices. Of particular interest to embedded users is support for hardware-acceleration through an EGL or OpenGL ES rendering back-end. In addition, the project added support for building the stand-alone player and the browser plug-in on Windows.

Gnash and AVM1

On the other hand, there was also an effort undertaken to add AVM1 support to Lightspark, which would have ultimately enabled the project to play all generations of Flash content. The 0.5.1 release, however, dropped Lightspark's attempt to write its own AVM1 interpreter. That effectively makes Gnash a dependency for any Flash content that uses features from Flash 8 or older.

Similarly, for a while Gnash attempted to add AVM2 support to its own codebase, but it, too, eventually abandoned the effort (at the time of the 0.8.8 release). All open source projects are perpetually starved for developers, but perhaps reverse-engineering two incompatible virtual machine implementations is proving to be beyond even the usual level of overload.

The trouble is that this leaves users without an all-in-one open source Flash implementation, a gap which may feel more acute once Adobe releases Flash Player 11.3, the first version that drops the NPAPI version of the plug-in for Linux.

Lightspark can still be installed as a browser plug-in, and call out to Gnash whenever it encounters AVM1 objects — but Gnash's own future is uncertain at the moment. Lead developer Rob Savoye told the gnash-dev mailing list in December that the 0.8.10 release would likely be the last to incorporate new features, as he has been unable to find funding to support further development. Gnash development had been supported by Lulu.com, who funded four full-time developers, but the company withdrew its support in 2010. Since then, Savoye has maintained the code as best he could, but is limited by financial reality to bug-fixes.

In an email, he said "I'd love to continue working on Gnash full time, but I need an income pretty bad at this point, and can only work for free for so long... So I'm looking for contract work at this point, Gnash related or not. I plan to continue hacking on Gnash, it's been a labor of love for 7 years now." Savoye added that he has brought the idea of funding development to multiple open source companies and foundations — including major Linux distributions — but that none were interested.

Gnash is an FSF high priority project, but that status does not include any financial contribution. For its part, Mozilla is on the record as not being interested in contributing or underwriting code for a Flash plug-in, on the grounds that the format is a vendor-controlled proprietary product, and Mozilla's resources are better used developing open solutions.

Perhaps that is true, but the argument generally goes along the lines of "users are better off without Flash, and everything that Flash does now will soon be replaced by HTML5." Unfortunately, open source advocates have been saying that for years and Flash is still pervasive. It would be betting against history for Linux companies to assume that Flash support will soon be an issue of the past using any reasonable definition of "soon." In the meantime, one can count on Lightspark to fill in the gaps for many modern games and video sites — and hope that Gnash will find new sources of support on its own.

Comments (27 posted)

Page editor: Jonathan Corbet

Security

The perils of desktop tracking

The Zeitgeist project aims to make desktop systems more helpful by keeping close track of what the user has been doing. Its developers describe it this way:

Zeitgeist is ostensibly independent of any specific desktop, but it seems to be driven more from the GNOME side of the house than any other. The fact that it has recently been entirely rewritten in the Vala language and proposed as an official GNOME module tend to reinforce that impression. Canonical has been putting in some of the development effort and Unity makes use of Zeitgeist. Tools like the GNOME Activity Journal also obtain information from Zeitgeist.

The Zeitgeist use cases page makes it clear that the plan is to create a comprehensive mechanism for the tracking, analysis, and sharing of user activity. Some examples include:

Undoubtedly there are a lot of useful things that can be done with this kind of tracking data. But there is also a possible down side; what happens if a detailed log of a user's activities gets into the wrong hands? The Zeitgeist "about" page has a rather unsatisfactory response to this concern: don't run untrusted applications on your system. Certainly that is good advice, but it also misses part of the point.

One can easily imagine an untrusting employer routinely examining the activity logs on all of its computers; it would be a shame, after all, if an employee were to be doing something unproductive on the job. This kind of information would be even more useful to governments that, for good reasons or bad, seek to know what somebody has been up to. The activity log could be a gold mine for inquisitive spouses, concerned parents, or curious roommates. This log could also open up a victim's life to any sort of successful malware attack. The advice to avoid running untrusted applications really only addresses the last of those concerns, and it is a partial response at best.

A somewhat improved response can be seen in this post from Zeitgeist developer Seif Lotfy. He has been working on the Vala port of the "activity log manager" (ALM), a tool for controlling the information tracked by Zeitgeist. Using ALM, it is possible to configure the system to forget events after a specific period of time - or to disable logging entirely. It is also possible to turn off logging involving specific types of files (videos or email messages, say), directories, or applications. After a proper bit of configuration, one's boss can see that flurry of spreadsheet activity but will remain unaware of all the time spent in a web browser.

This kind of configurability is a step in the right direction, but it is still a partial response at best. There will undoubtedly be legions of users who are unaware that this logging is happening at all; they are unlikely to find the utility to fine-tune that logging. Even users who want the functionality provided by this logging may find themselves reconsidering after their activity is exposed to the wrong person.

For a certain class of user, the answer will be to simply turn off features like Zeitgeist altogether - once they become aware of such features. But even the most paranoid among us find ourselves, at times, wishing that our computers were a little smarter in their interaction with us. Many (probably most) of us want the computer to understand how we work and to make that work easier and less repetitive. So, increasingly, those computers will observe what we do and build their own models of who we are and how we work. Progress toward the creation of those models appears to be outpacing the work to protect them; experience suggests that this problem will only be addressed after some people have learned about the issue the hard way.

Comments (20 posted)

Brief items

Security quotes of the week

Comments (none posted)

Critical Flaw Found In Security Pros' Favorite: Backtrack Linux (threatpost)

Comments (none posted)

New Security Sensor Gives Admins Better View of Network Attacks (eWeek)

Comments (8 posted)

New vulnerabilities

apache2: insecure default configuration

Comments (1 posted)

cumin: cross-site scripting

Comments (none posted)

gajim: multiple vulnerabilities

Comments (none posted)

kernel: remote denial of service

Comments (1 posted)

moodle: many vulnerabilities

Comments (none posted)

phppgadmin: cross-site scripting

Comments (none posted)

swftools: code execution

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

Stable updates: the 3.0.28, 3.2.15 and 3.3.2 updates were released on April 13; each contains a long list of important fixes.

Comments (none posted)

Quotes of the week

    #define IS_DEFINED(x) (__stringify(CONFIG_##x)[0]=='1')

Comments (8 posted)

Allocating uninitialized file blocks

For most users, fallocate() works just as it should. In some cases, though, the application in question does a lot of random writes scattered throughout the file. Writing to a small part of an uninitialized extent may force the filesystem to initialize a much larger range of blocks, slowing things down. But if the application knows where it has written in the file, and will thus never read from uninitialized parts of that file, it gains no benefit from this extra work.

How much does this initialization cost? Zheng Liu recently implemented a new fallocate() flag (called FALLOC_FL_NO_HIDE_STALE) that marks new blocks as being initialized, even though the filesystem has not actually written them; these blocks, will thus contain random old data. A random-write benchmark that took 76 seconds on a mainline kernel ran in 18 seconds when this flag was used. Needless to say, that is a significant performance improvement; for that reason, Zheng has proposed that this flag be merged into the mainline.

Such a feature has obvious security implications; Zheng's patch tries to address them by providing a sysctl knob to enable the new feature and defaulting it to "off." Still, Ric Wheeler didn't like the idea, saying "Sounds like we are proposing the introduction a huge security hole instead of addressing the performance issue head on." Ted Ts'o was a little more positive, especially if access to the feature required a capability like CAP_SYS_RAWIO. But everybody seems to agree that a good first step would be to figure out why performance is so bad in this situation and see if a proper fix can be made. If the performance issue can be made to go away without requiring application changes or possibly exposing sensitive data, everybody will be better off in the end.

Comments (21 posted)

Kernel development news

Toward more reliable logging

The same fate seemed to be in store for Lennart Poettering and Kay Sievers when they presented their ideas at the 2011 Kernel Summit; in particular, their concept of attaching 128-bit unique ID numbers to each message was met with almost universal disdain. Lennart and Kay have not given up, though. The latest form of their work on the kernel side of the problem can be seen in the structured printk() patch recently posted by Kay.

The patch does a few independent things - a cause for a bit of complaining on the mailing list. The first of these is to change the kernel's internal log buffer from a stream of characters into a series of records. Each message is stored into the buffer with a header containing its length, sequence number, facility number, and priority. In the end, Kay says, the space consumed by messages does not grow; indeed, it may shrink a bit.

The record-oriented internal format has a number of advantages. If messages are being created faster than they can be consumed by user space, it is necessary to overwrite the oldest ones. Current kernels simply write over the character stream, with the result that truncated messages can find their way into the log. The new implementation drops entire messages at once, so the stream, while no longer complete, is not corrupted. The sequence numbers allow any consumer to know that messages have been dropped - and exactly how many it missed. The record-oriented format also enables improved handling of continuation lines at the cost of making the use of the KERN_CONT "priority" mandatory for such lines.

The second change is to allow the addition of a facility number and a "dictionary" containing additional information that, most likely, will be of interest to automated parsers. The dictionary contains "KEY=value" pairs, separated by spaces; these pairs will contain, for example, device and subsystem names to unambiguously identify the device that generated the message. Kernel code that wants to attach a facility number and/or dictionary to a message will use the new function printk_emit() to do so:

    int printk_emit(int facility, int level, const char *dict, 
			size_t dictlen, const char *fmt, ...);

Regular printk() turns into a call to printk_emit() with a facility of zero and a null dict.

Creation of the dictionary string itself is left as an exercise for the caller; it is not something one is likely to see done in most places where printk() is called. In fact, the only full user of printk_emit() in the patch is dev_printk() which uses it to add a dictionary with SUBSYSTEM and DEVICE fields describing the device of interest. If some form of this patch is merged, one can expect this usage pattern to continue; the creation of dictionaries with ancillary information will mostly be done with subsystem-specific print functions.

Finally, the patch changes the appearance of log messages when they reach user space. After some complaints from Linus, the format has evolved to look something like this:

    7,384,6193747;sr 1:0:0:0: Attached scsi CD-ROM sr0
     SUBSYSTEM=scsi
     DEVICE=+scsi:1:0:0:0

The comma-separated fields at the beginning are the message priority (7, in this case), the sequence number (384), and the system time in microseconds since boot. The rest of the line is the message as passed to printk(). If the message includes a dictionary, it appears in the following lines; following the style set in RFC 821, continuation lines begin with white space. The result, it is hoped, is an output format that is simultaneously easy for humans to read and machines to parse.

The behavior of the /dev/kmsg device changes somewhat as well. In current kernels, this device is only useful for injecting messages into the kernel log stream. Kay's patch turns it into a device supporting read() and poll() as well, with multiple concurrent readers supported. If messages are overwritten before a particular reader is able to consume them, the next read() call will return an EPIPE error; subsequent reads will start from the next available message. This device thus becomes a source of kernel log data that is easy to work with and that reliably delivers log messages or ensures that the recipient knows something was lost.

Modulo some quibbling over the log format, the response to the patch seems to be mostly positive. The biggest exception is arguably Ingo Molnar, whose suggestion that tracepoints and perf should be used instead does not appear to have received a lot of support. Even Linus seems mostly happy; the absence of the 128-bit unique ID perhaps has a lot to do with that. But, beyond that, a more robust log buffer with sequence numbers has some clear advantages; Linus suggested that, if that part were split out, he might even consider merging it for the 3.4 release. That seems unlikely to happen at this point in the cycle, but it wouldn't be surprising to see something come together for the 3.5 development cycle. If that happens, Linux will still lack a true structured logging mechanism, but it would have something with more structure and reliability than it has now.

Comments (72 posted)

The value of release bureaucracy

This patch to the ath9k wireless driver was meant to fix a simple bug; it was merged for 3.4. Since it was a bug fix, it was duly marked for the stable updates and shipped in 3.3.1. It turns out to not have been such a good idea, though; some 3.3.1 users have reported that the "fix" can break the driver and sometimes make the system as a whole unusable. That is not the sort of improvement that stable kernel users are generally hoping for. Naturally, they hoped to receive a fix to the fix as soon as possible.

When the 3.3.2 update went into the review process without a revert for the offending commit, some users asked why. The answer was simple: the rules for the stable tree do not allow the inclusion of any patch that has not already been merged, in some form, into the mainline. Since this particular fix had not yet made it to Linus (it was still in the wireless tree), Greg Kroah-Hartman, the stable kernel maintainer, declined to take it for the 3.3.2 cycle. And that is where the trouble started.

Our lone participant (Felipe Contreras) denounced this decision as a triumph of bureaucratic rules over the need to actually deliver working kernels to users. Beyond that, he said, since reverting the broken patch simply restored the relevant code to its form in the 3.3 release, the code was, in effect, already upstream. Accepting the revert, he said, would have the same effect as dropping the bad patch before 3.3.1 was released. In this view, refusing to accept the fix made little sense.

Several kernel developers tried to convince him otherwise using arguments based on the experience gained from many years of kernel maintenance. They do not appear to have succeeded. But they did clearly express a couple of points that are worth repeating; even if one does not agree with them, they explain why certain things are done the way they are.

The first of those was that experience has proved, all too many times, that fixes applied only to stable kernel releases can easily go astray before getting into the mainline. So problems that get fixed in a stable release may not be fixed in current development kernels - which are the base for future stable kernels. So stable kernel users may see a problem addressed, only to have it reappear when they upgrade to a new stable series. Needless to say, that, too, is not the experience stable kernel users are after. On the other hand, people who like to search for security holes can learn a lot by watching for fixes that don't make it into the mainline.

It is true that dropped patches used to be a far bigger problem than they are now. A patch applied to, say, a 2.2 release had no obvious path into the 2.3 development series; such patches often fell on the floor despite the efforts of developers who were specifically trying to prevent such occurrences. In the current development model, a fix that makes it into a subsystem maintainer's tree will almost certainly get all the way into the mainline. But, even now, it's not all that rare for a patch to get stranded in a forgotten repository branch somewhere. When the system is handling tens of thousands of patches every year, the occasional misrouted patch is just not a surprise.

The simple truth of the matter is that many bugs are found by stable kernel users; there are more of them and they try to use their kernels for real work. As this thread has shown, those users also tend to complain if the specific fixes they need don't get into stable releases; they form an effective monitoring solution that ensures that fixes are applied. The "mainline first" rule takes advantage of this network of users to ensure that fixes are applied for the long term and not just for a specific stable series. At the cost of (occasionally) making users wait a short while for a fix, it ensures that they will not need the same fix again in the future and helps to make the kernel less buggy in general.

Developers also took strong exception to the claim that applying a revert is the same as having never applied the incorrect fix in the first place. That can almost never be strictly true, of course; the rest of the kernel will have changed between the fix and the revert, so the end product differs from the initial state and may misbehave in new and interesting ways. But the real issue is that both the fix and the revert contain information beyond the code changes: they document a bug and why a specific attempt to fix that bug failed. The next developer who tries to fix the bug, or who makes other changes to the same code, will have more information to work with and, hopefully, will be able to do a better job. The "mainline first" rule helps to ensure that this information is complete and that is it preserved in the long term.

In other words, some real thought has gone into the creation of the stable kernel rules. The kernel development community, at this point, has accumulated a great deal of experience that will not be pushed aside lightly. So the stable kernel rules are unlikely to be relaxed anytime soon. The one-sided nature of the discussion suggests that most developers understand all of this. That probably won't be enough to avoid the need to discuss it all again sometime in the near future, though.

Comments (7 posted)

LTTng 2.0: Tracing for power users and developers - part 2

In part 1 of this article, we presented the motivations that led to the creation of LTTng 2.0, its features, along with an overview of the respective strengths of LTTng 2.0, Perf, and Ftrace. We then presented two LTTng 2.0 usage examples.

In this article, we will start with two more advanced usage examples, and then proceed to a presentation of LTTngTop, a low-overhead, top-alike view, based on tracing rather than sampling /proc. This article focuses on some of the "cool features" that are made possible with the LTTng 2.0 design: combined tracing of both kernel and user space, use of performance counters to augment trace data, and combining all these together to generate a higher-level view of the system CPU and I/O activity with LTTngTop. But first, we continue with the examples:

3. Combined user space and kernel tracing

This example shows how to gather a trace from both the kernel and a user-space application. Even though the previous examples focused only on kernel tracing, LTTng 2.0 also offers fast user-space tracing support with the "lttng-ust" (LTTng User-space Tracer) library. For more information on how to instrument your application, see the lttng-ust(3) and lttng-gen-tp(1) man pages.

The hello.c test program is distributed with the lttng-ust source. It has an example tracepoint that associates various types of data with the tracepoint. The tracepoint data, including all of the different types can be seen below in the first instance of hitting the tracepoint.

    # (as root, or tracing group)
    $ lttng create
    $ lttng enable-event --kernel --all
    $ lttng enable-event --userspace --all
    $ lttng start
    $ cd lttng-ust/tests/hello
    $ ./hello		# Very, very high-throughput test application
    $ sleep 10          # [ let system generate some activity ]
    $ lttng stop
    $ lttng view
    $ lttng destroy

Output from lttng view:

    [...]
    [18:47:03.263188612] (+0.000018352) softirq_exit: { cpu_id = 1 }, { vec = 4 }
    [18:47:03.263193518] (+0.000004906) exit_syscall: { cpu_id = 1 }, { ret = 0 }
    [18:47:03.263198346] (+0.000004828) ust_tests_hello:tptest: { cpu_id = 3 }, { \
	intfield = 1676, intfield2 = 0x68C, longfield = 1676, \
	netintfield = 1676, netintfieldhex = 0x68C, arrfield1 = [ [0] = 1, [1] = 2, \
	[2] = 3 ], arrfield2 = "test", _seqfield1_length = 4, \
        seqfield1 = [ [0] = 116, [1] = 101, [2] = 115, [3] = 116 ], \
	_seqfield2_length = 4, seqfield2 = "test", stringfield = "test", \
	floatfield = 2222, doublefield = 2 }
    [18:47:03.263199453] (+0.000001107) sys_write: { cpu_id = 3 }, { fd = 18, \
        buf = 0x7F5C935EAD4D, count = 1 } 
    [18:47:03.263200997] (+0.000001544) sys_poll: { cpu_id = 1 }, { ufds = 0x1C9D8A0, \
        nfds = 6, timeout_msecs = -1 }
    [18:47:03.263201067] (+0.000000070) exit_syscall: { cpu_id = 3 }, { ret = 1 }
    [18:47:03.263204813] (+0.000003746) ust_tests_hello:tptest: { cpu_id = 3 }, { \
        intfield = 1677, [...] }
    [18:47:03.263207406] (+0.000002593) ust_tests_hello:tptest: { cpu_id = 3 }, { \
	intfield = 1678, [...] }
    [...]

In short, the output above shows that CPU 1 is executing the end of a softirq handler, CPU 3 is in user mode within the "hello" test application, writing its high-throughput event to the buffer. This example is taken at the moment the buffer switch occurs within the LTTng-UST tracer, so the application signals the consumer daemon waiting on poll() on CPU 1 that data is ready. The "hello" test application then continues writing into its tracing buffer.

Correlated analysis of events coming from both the kernel and user space, gathered efficiently without round-trips between the kernel and user space, enables debugging systemic problems across execution layers. User-space instrumentation with the LTTng-UST tracepoint event API, and the use of trace log-levels in combination with wildcards, are not covered here for brevity, but you can look at the lttng(1) man page if you are curious.

4. Performance counters and kretprobes

This example shows how to combine kernel instrumentation mechanisms to get information that is otherwise unavailable. In this case, we are interested in the number of LLC (Last Level Cache) misses produced by each invocation of a function in the Linux kernel. We arbitrarily chose the function call_rcu_sched().

First, it is important to measure the overhead produced by kretprobes, reading the performance monitoring unit (PMU), and tracing with LTTng to understand how much of the total count can be attributed to the tracing itself. LTTng has a "calibrate" command to trigger calibration functions which, when instrumented, collect the base cost of the instrumentation.

Here is an example showing the calibration, using an i7 processor with 4 general-purpose PMU registers. The information about PMU registers can be found in the kernel boot messages under "Performance Events", then look for "generic registers". Note that some registers may be reserved by the kernel NMI watchdog.

This sequence of commands will gather a trace executing a kretprobe hooked on an empty function that gathers the LLC-misses information (see lttng add-context --help to get a list of the available PMU counters).

    $ lttng create calibrate-function
    $ lttng enable-event calibrate --kernel --function lttng_calibrate_kretprobe
    $ lttng add-context --kernel -t perf:LLC-load-misses -t perf:LLC-store-misses \
		    -t perf:LLC-prefetch-misses
    $ lttng start
    $ for a in $(seq 1 10); do \
	    lttng calibrate --kernel --function;
    $ done
    $ lttng stop
    $ lttng view
    $ lttng destroy

The output from babeltrace can be analyzed to look at the per-PMU counter delta between consecutive "calibrate_entry" and "calibrate_return" events. Note that these counters are per-CPU, so scheduling events need to be present in the trace to account for migration between CPUs. Therefore, for calibration purposes, only events staying on the same CPU should be considered.

The average result, for the i7, on 10 samples:

				 Average     Std.Dev.
    perf_LLC_load_misses:           5.0       0.577
    perf_LLC_store_misses:          1.6       0.516
    perf_LLC_prefetch_misses:       9.0      14.742

As can be seen, the load and store misses are relatively stable across runs (their standard deviation is relatively low) compared to the prefetch misses. We can conclude from this information that LLC load and store misses can be accounted for quite precisely by removing the calibration base-line, but pre-fetches within a function seem to behave too erratically (not much causality link between the code executed and the CPU pre-fetch activity) to be accounted for.

We can then continue with our test, which was performed on a 2.6.38 Linux kernel, on a dual-core i7 SMP CPU, with hyperthreading (the same system that was calibrated above):

    $ lttng create measure-call-rcu-sched
    $ lttng enable-event call_rcu_sched -k --function call_rcu_sched
    $ lttng add-context --kernel -t perf:LLC-load-misses -t perf:LLC-store-misses \
		    -t perf:LLC-prefetch-misses
    $ lttng start
    $ sleep 10        # [ let system generate some activity ]
    $ lttng stop
    $ lttng view
    $ lttng destroy

Here is some sample output using:

    $ lttng view -e 'babeltrace --clock-raw --no-delta'

    timestamp = 37648.652070250,
    name = call_rcu_sched_entry,
    stream.packet.context = { cpu_id = 1 },
    stream.event.context = {
	    perf_LLC_prefetch_misses = 3814,
	    perf_LLC_store_misses = 9866,
	    perf_LLC_load_misses = 16328
    },
    event.fields = {
	    ip = 0xFFFFFFFF81094A5E,
	    parent_ip = 0xFFFFFFFF810654D3
    }

    timestamp = 37648.652073373,
    name = call_rcu_sched_return,
    stream.packet.context = { cpu_id = 1 },
    stream.event.context = {
	    perf_LLC_prefetch_misses = 3815,
	    perf_LLC_store_misses = 9871,
	    perf_LLC_load_misses = 16337
    },
    event.fields = {
	    ip = 0xFFFFFFFF81094A5E,
	    parent_ip = 0xFFFFFFFF810654D3
    }

An analysis of the 1159 entry/return pairs on CPU 1 that did not migrate between processors yields:

				 Average     Std.Dev.
    perf_LLC_load_misses:          4.727      6.371
    perf_LLC_store_misses:         1.280      1.198
    perf_LLC_prefetch_misses:      1.662      4.832

So the numbers we have here are within the range of the empty function calibration. We can therefore say that call_rcu_sched() is doing a good job at staying within the Last Level Cache. We could repeat the experiment with other kernel functions, targeting L1 misses, branch misses, and various other PMU counters.

LTTngTop

LTTngTop is a ncurses-based tool developed to provide system administrators with a convenient way to browse traces and quickly find a problem, or at least a period of time when a problem occurred. That information considerably reduces the number of events we need to analyze manually. It is designed to suit the system administrators because it behaves like the popular top CPU activity monitoring program. In addition to the usual behavior of top and similar tools, where the display is refreshed at a fixed interval, LTTngTop allows the user to pause the reading of the trace to take time to look at what is happening, and also to go back and forth in time to easily see the evolution between two states.

In order to properly handle the events without the risk of attributing statistics to the wrong process in case of a lost event, we require that the events be self-describing. For use with LTTngTop, it is required that each event include the process identifier (PID), the process name (procname), the thread identifier (tid), and parent process identifier (ppid), all of which can be done using the context information. Although adding this data makes the trace bigger, it ensures that every event is handled appropriately, even if LTTng needs to discard some events (which can happen if the trace sub-buffers are too small).

As of now, LTTngTop only works with recorded traces, but work is in progress to support live tracing. The tool displays statistics such as the CPU usage time, the PMU counters (real data per-process, not sampled), and the I/O bandwidth. By default it reads one second of trace data and refreshes the display every second which gives the feeling of playing back the activity on the system. The intended usage of this tool is to allow non-developers (especially system administrators) to use trace data and to help pinpoint the period of time when a problem occurred on the system.

In the following example, we record a trace suitable for analysis with LTTngTop (with pid, procname, tid, ppid context information associated with each event) and with three PMU counters.

    $ lttng create
    $ lttng enable-event --kernel --all
    $ lttng add-context -k -t pid -t procname -t tid -t ppid \
	    -t perf:cache-misses -t perf:major-faults \
	    -t perf:branch-load-misses
    $ lttng start
    $ sleep 10	# [ let system generate some activity ]
    $ lttng stop
    $ lttng destroy
    $ lttngtop /path/to/the/trace

With this example, you will see exactly the activity that occurred on the system, and can use the left and right arrows on the keyboard to navigate forward and backward in the history. As noted above, work is in progress to support live trace reads. It will be performed through a shared memory map on the local machine, and eventually should support viewing live traces streamed from a remote target. LTTngTop is still in development mode but is usable, it can be found in the Git tree, and the README explains how to get it up and running.

Upstreaming: The road ahead

We really hope that the LTTng features will gradually make their way into the upstream kernel. Our target with LTTng 2.0 has been to ensure that our user base quickly gets access to the features provided by LTTng 2.0 through their Linux distributions, without having to wait for the upstreaming process to come to completion.

It remains to be discussed whether the LTTng-specific focus on integrated, low-overhead, full-system tracing, and its ability to share tools with various industry players, make strong enough arguments to justify merging its ABI into the Linux kernel. Nevertheless, our goal is to share the common pieces of infrastructure with Perf and Ftrace whenever it is beneficial for all of the projects to do so.

Conclusion

The very low overhead and high-scalability of the LTTng tracer makes it an excellent choice to tackle issues on high-availability, low-latency, production servers dealing with high-throughput data. The tracer flexibility allows combining traces gathered from both kernel and user space to be analyzed on a common time-line.

LTTng has been used to debug performance, behavior, and real-time issues at various client sites. Some examples include using the kernel tracer to identify an abnormally long interrupt handler duration and to pinpoint the cause of delays in a soft real-time system due to a firmware bug. At the I/O level, identification of bottlenecks caused by combined fsync() calls and large logs being written by timing-sensitive services was made possible by use of tracing. Another client example was one that experienced slowdowns and timeouts after moving from a local to a distributed filesystem: identifying much longer I/O requests in the distributed setup using the LTTng kernel tracer allowed us to pinpoint a filesystem cache that was too small as the root cause of the problem.

We are currently working on several features for LTTng 2.1: integration of flight recorder "snapshots" into lttng-tools, live trace streaming over the network, system-wide LTTng-UST buffers, and filtering of LTTng-UST event fields at trace collection time. With these features and others down the road on top of the existing LTTng 2.0 base, we hope to succeed in our goal to make developers' and system administrators' lives easier.

[ Mathieu Desnoyers is the CEO of EfficiOS Inc., which also employs Julien Desfossez and David Goulet. LTTng was created under the supervision of Professor Michel R. Dagenais at Ecole Polytechnique de Montréal, where all of the authors have done (or are doing) post-graduate studies. ]

Comments (3 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.4-rc3 . (April 16, 2012)

• Thomas Gleixner: 3.4-rc2-rt4 . (April 13, 2012)

• Greg KH: Linux 3.3.2 . (April 13, 2012)

• Greg KH: Linux 3.2.15 . (April 13, 2012)

• Steven Rostedt: 3.2.15-rt25 . (April 17, 2012)

• Steven Rostedt: 3.2.15-rt26 . (April 17, 2012)

• Greg KH: Linux 3.0.28 . (April 13, 2012)

• Steven Rostedt: 3.0.28-rt47 . (April 17, 2012)

• Steven Rostedt: 3.0.28-rt48 . (April 18, 2012)

Core kernel code

• Oleg Nesterov: task_work_queue() with 2 users . (April 13, 2012)

• Paul E. McKenney: [PATCH RFC 0/7] rcu: v2 Inlinable preemptible rcu_read_lock() and rcu_read_unlock() . (April 15, 2012)

• John Stultz: Volatile Ranges (v7) . (April 16, 2012)

• Kay Sievers: printk: convert byte-buffer to variable-length record buffer . (April 18, 2012)

• Kay Sievers: kmsg: export printk records to the /dev/kmsg interface . (April 18, 2012)

Device drivers

• Laxman Dewangan: staging: iio: add isl29028 driver . (April 12, 2012)

• Bryan Wu: Introduce a led trigger for CPU activity and consolidate LED driver in ARM . (April 13, 2012)

• Tomasz Stanislawski: Integration of videobuf2 with dmabuf . (April 16, 2012)

• Jonas Bonn: leds: add "kickable" LED trigger . (April 16, 2012)

• Roland Stigge: USB: Add driver for NXP ISP1301 USB transceiver . (April 17, 2012)

• Dave Airlie: [PATCH] drm: Initial KMS driver for AST (ASpeed Technologies) 2000 series . (April 17, 2012)

• Sylwester Nawrocki: V4L camera control enhancements . (April 17, 2012)

• Venkatraman S: [FS, MM, block, MMC]: eMMC High Priority Interrupt Feature . (April 18, 2012)

• Daniel Vetter: dma-buf: mmap support . (April 18, 2012)

• Henrik Rydberg: hid: Introduce device groups . (April 18, 2012)

Filesystems and block I/O

• Arne Jansen: btrfs: snapshot deletion via readahead . (April 12, 2012)

• Tejun Heo: block: per-queue policy activation . (April 13, 2012)

• Zheng Liu: add FALLOC_FL_NO_HIDE_STALE flag in fallocate . (April 17, 2012)

• Allison Henderson: ext4: status extents . (April 18, 2012)

Memory management

• Aneesh Kumar K.V: memcg: Add memcg extension to control HugeTLB allocation . (April 16, 2012)

• Mel Gorman: Swap-over-NBD without deadlocking V9 . (April 16, 2012)

• Mel Gorman: Swap-over-NFS without deadlocking V3 . (April 16, 2012)

Networking

• John Fastabend: Managing the forwarding database(FDB) . (April 13, 2012)

Architecture-specific

• Kent Yoder: Platform Facilities Option and crypto accelerators . (April 12, 2012)

• Vivien Didelot: TS-5500 platform support . (April 13, 2012)

• Marek Szyprowski: ARM: replace custom consistent dma region with vmalloc . (April 15, 2012)

• Marek Szyprowski: ARM: DMA-mapping framework redesign . (April 18, 2012)

Security-related

• Kees Cook: Yama: add additional ptrace scopes . (April 17, 2012)

Virtualization and containers

• Xiao Guangrong: KVM: MMU: fast page fault . (April 13, 2012)

Page editor: Jonathan Corbet

Distributions

Whither Mandriva?

The Mandriva distribution has suffered some serious blows over its lifetime. It has been in and out of bankruptcy, and has been, seemingly, perpetually on the brink of financial collapse over the last six years or more. Much of the community has moved on to the Mageia fork, but Mandriva still seems to limp along. How long that will continue at this point is anyone's guess.

The current status of Mandriva is a bit mysterious for a number of different reasons. In a blog post, Mandriva S.A. COO Jean-Manuel Croset notes that the company has been uncommunicative for "the last few months" without much of an explanation why. His post addresses the Mandriva community and is looking for input about where the distribution should be heading:

Croset mentions that there is a shareholders meeting for Mandriva S.A. on April 30, where strategy and priorities for the next year will be decided. Presumably, his outreach to the community is focused on helping the company make those decisions.

In the meantime, though, it would seem that progress on the distribution has stalled. Some developers have moved to ROSA, which is a Russia-based company that is building a desktop distribution (ROSA Marathon 2012) atop Mandriva. For a while, it seems, ROSA was working within the Mandriva community but has more recently moved on. Much of that information comes from a thread on the Mandriva cooker mailing list—Cooker is the name of Mandriva's development distribution, like Fedora's Rawhide.

That thread, which was started by longtime Mandriva developer Per Øyvind Karlsen is unfortunately—perhaps mistakenly—marked as "not to be archived" (i.e. X-No-Archive: yes). The thread started on the closed maintainers mailing list, but Karlsen copied the cooker list because he wanted the topic to be more widely discussed. The flag on the posts means that Gmane, Google, and others will expire them soon, effectively purging them from the internet. So we won't be quoting from or linking to those posts, in keeping with the intent of the flag, but can at least summarize the discussion.

Karlsen is concerned that the Mandriva distribution is dying because the parent company has financial problems and because there has never been a neutral foundation set up to shepherd the distribution. Until recently, ROSA was working closely with Mandriva S.A., but that has ended because of some dispute between the two companies, he said. Meanwhile, Karlsen stopped working for Mandriva S.A. in November and has gone to work for ROSA. It is his belief that the distribution will disappear without a foundation behind it.

Dmitry Komissarov said that ROSA could support a Mandriva foundation, but that the main problems were the need for that foundation to receive the trademark as well as the need for a project leader to be identified. There were suggestions that one of the driving forces behind Mageia, Anne Nicolas, might make a good leader for the project or foundation, but that was questioned by others including Karlsen. The problem of the trademarks may be stickier, as it would seem that Mandriva S.A. is not necessarily interested in handing them over to some kind of independent organization.

Andrey Bondrov sees a schism between the ROSA developers and the Cooker team that is likely to effectively kill off Mandriva. From his perspective, ROSA is headed in a direction very different from the way Mandriva operates (in terms of its desktop focus, in particular). Komissarov disputed much of what Bondrov said, noting that ROSA had talked about making some of the changes, but had not acted on them, at least yet.

Suggestions that folks interested in Cooker move to Mageia were mostly met with disagreement. Various people feel that they have a lot invested in Cooker (and Mandriva) and are not willing to make a switch. The overall impression one gets by reading the thread is that there is a lot of confusion in the Mandriva community about what to do and how to go about doing it.

In many ways, the problem boils down to a distribution that has (or had) an active community, but was still driven by a company (and, importantly, that company's money). Now that the money has largely dried up, the community is somewhat adrift at this point. Because only the trademark holder can actually release a distribution called "Mandriva", the community is to some extent held hostage to the (as yet unknown) plans of Mandriva S.A.

According to a blog post by the founder of Mandrake Linux (Mandriva's predecessor), Gaël Duval, the idea of a foundation has been discussed since 2000 or 2001. He notes that he has not been involved in Mandriva since 2006, but he believes that a distribution-focused foundation would be a boon both for Mandriva itself, and for the public at large. He suggests an "OS in the Public Interest" as a possible future for Mandriva noting that Debian is for "servers & geeks" and Ubuntu is held by Canonical. Because of Mandriva's focus on the desktop, a public-interest OS based on it could serve as a way to break the proprietary stranglehold on the desktop, Duval said.

Any kind of movement in that direction requires assistance from Mandriva S.A., of course. It isn't clear what prompted Croset to ask the community for input recently and the silence from the company over the last few months has certainly worried many in the community. The crux of Croset's message: "For the time being and in short: trust us." may not really alleviate those concerns.

Mageia, on the other hand, seems to be thriving. It has just released the third beta of Mageia 2, and the final release is currently scheduled for May 15. It would seem that some part of the Mandriva community was able to move on. Given that Mandriva is based free software, all that really needed to be done was to pick a new name, set up some infrastructure, and start working on releases. Since Mageia has its own name, it was able to set up a non-profit association to govern the distribution as well.

While it is understandable that some feel strong ties to the Mandriva name, the murky future for the company makes those ties somewhat risky. As Duval notes, at one point Mandrake Linux was available in stores all over the world, and he says that he often hears from people who started their Linux experience using Mandrake. Since that time—and after a trademark dispute caused the name to change—Mandriva has had lots of ups and downs, but its users are often fiercely loyal. If the current financial problems for Mandriva S.A. mean the end of the distribution, that would be a sad outcome.

Comments (30 posted)

Brief items

Distribution quotes of the week

Comments (1 posted)

Fedora 17 beta available

Full Story (comments: none)

FreeBSD 8.3 is available

Full Story (comments: none)

Stefano Zacchiroli re-elected Debian Project Leader

Full Story (comments: 16)

Distribution News

Debian GNU/Linux

(deferred) bits from the DPL: March 2012

Full Story (comments: none)

openSUSE

Heinlein Support Becomes openSUSE Project Sponsor

Comments (none posted)

Ubuntu family

Ubuntu Code of Conduct update

Comments (none posted)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (April 16)
• DistroWatch Weekly, Issue 452 (April 16)
• Maemo Weekly News (April 16)
• Ubuntu Weekly Newsletter, Issue 261 (April 15)

Comments (none posted)

Trisquel moves to GNOME 3 fallback mode (The H)

Comments (none posted)

Page editor: Rebecca Sobol

Development

MythTV turns 0.25

MythTV, the free software DVR project, has released its first stable update in nearly a year and a half. The release brings a list of significant new features, from improved GPU-based video decoding to integration with other home theater components, as well as major changes to the architecture. During the development cycle leading up to this release, the open source DVR landscape has continued to evolve — including a recent fork started by two key MythTV developers. But many of the recent changes to MythTV are designed to put it into a more manageable position for the future.

Never a project to be accused of version-number-inflation, the new release is tagged 0.25, and is available for download directly from the MythTV site as source code tarballs. The core application and the official plugin collection are provided separately. Most Linux users will probably prefer to look for distribution-delivered packages, and there is a list of those on the project's wiki. MythTV is cross-platform, and third-party Mac OS X binaries are available, too. Intriguingly enough, the project notes that it knows of no one packaging the system for Windows, but there are build instructions provided for the sufficiently daring.

For the uninitiated, a MythTV setup uses separate front-end and back-end applications. The back-end handles scheduling, recording, and streaming media, while the front-end presents the UI to the user and handles playback. A single back-end can handle multiple front-ends, and back-ends typically manage an array of auxiliary jobs to index electronic guide data, flag commercials, transcode, and export content. Nevertheless, the back-end and front-ends are tightly coupled; they share access to a single database and must speak the same wire protocol — you cannot update one without the other, a limitation that updaters need to be aware of when migrating.

New features in 0.25

Many of the high-profile changes in MythTV 0.25 are front-end affairs. The most obvious is support for the Freedesktop.org Video Acceleration API (VAAPI), which enables GPU video decoding on most major video chipsets, and allows for smaller and quieter front-end hardware. The playback stack officially drops support for the older XvMC acceleration scheme (VAAPI accelerates a superset of the decoding steps covered by XvMC), deprecates the libmpeg2 library in favor of libav, and supports OpenGL ES 2.0 (which again is good news for those looking to build slim-CPU front-ends). Windows users, unblessed by VAAPI, can use GPU video decoding via Microsoft's DirectX Video Acceleration 2 API. This release also supports 3D video for the first time, at least for side-by-side or top-and-bottom frame orderings.

Audio playback also gets a refresh in this release, adding support for the E-AC3, TrueHD, DTS-HD, and DTS-ES Discrete formats. All are "high definition" audio encodings that are primarily of interest to people passing the audio through to a multi-channel receiver over HDMI. The last format on the list, DTS-ES Discrete, is a peculiar offering that uses a "6.1" channel setup — one that, according to the commit log, users are likely to encounter only in recent Star Wars releases, which certainly gives it geek credibility. There is also a new "audio set-up" page in the front-end settings that allows you to test your speaker output configuration, which is increasingly important as more channels are added.

Interfacing with modern TV sets and home theater receivers is significantly improved with the addition of support for the Consumer Electronics Control (CEC) protocol; this allows the MythTV box to control TV and component settings over an HDMI cable. Front-ends can also stream media to (and play media from) Apple devices using the Apple-only Airtunes/Airplay protocol, and infrared remote controls can now be used to navigate and control Flash-based web videos.

There are other features that offer direct benefits on the front-end, such as a "night mode" screen-dimming option, support for rich-text subtitles (e.g., color and bold/italic markup), and a standby screen that lets a back-end restart without hanging or crashing any connected front-ends. A long list of bugs with the on-screen video editor component (which is used for trimming out commercials) was fixed, and there is an equally long list of fixes to the UI theming engine and the suite of default themes. Extensive details, including links to all of the bug reports closed, are in the release notes wiki page.

Back-ends pick up some new features in 0.25 as well, including out-of-the-box support for CableCARD hardware. Note, though, that even though a CableCARD device may work like a charm, that fact does not prevent a cable company from flagging all of its content as non-recordable. It is unlikely that an open source project like MythTV will ever get the certification needed to access encrypted or access-restricted media streams, so the value of this feature depends greatly on the local cable provider.

Refactoring

User-visible features gather the most attention, but a number of architectural changes in 0.25 probably account for more work. That is a rough guess, of course — VAAPI support is not child's play, after all — but MythTV turns ten years old this month, and refactoring an old and entrenched codebase is tricky. In MythTV's case, it was increasingly obvious that parts of the original design had outlived their usefulness and risked becoming a liability, so clearing them out was a necessity.

That was certainly the case with MythXML, the now-deprecated remote access API. It was limited in functionality, and few plugins or third-party tools took advantage of it. In 0.25, it is replaced with the new Services API, which provides HTTP POST/GET access to a far broader set of back-end and front-end services. The Services API exposes MythTV's DVR, program guide, and content playback functions, and is intended to serve as the basis for HTML-based remote control apps, web front-ends, and better integration into other applications (for example, to provide access to MythTV recordings in XBMC). The API has HTTP and SOAP interfaces, and can return XML or JSON output. It can even stream video over HTTP (as opposed to the MythTV-specific streaming protocol of previous releases).

Although MythTV was originally focused on recording live television, the public's media consumption habits have shifted in the last ten years, and the media center's general audio and video playback plugins have fallen a bit behind. Both components, MythMusic and MythVideo, received rewrites in this release — in fact, MythVideo has now been merged into the main application as "Video Gallery." MythMusic gains support for play counts and ratings stored in ID3 tags, and preliminary support for album art. Video Gallery can look up video metadata from an array of online databases, and supports indexing and organizing videos by season and episode number (a feature also added to the main TV recordings component). Both MythMusic and Video Gallery can function as UPnP clients, browsing and playing media stored on other devices.

Further under the hood, there are important changes like full support of IPv6, a rewritten logging system, and splitting some functions out of the main process into separate threads (the prime example being scanning a storage partition for new media content; for modern, large-scale disks this process could hang the main UI for several minutes). The project also made the decision to build and bundle in its own version of the FFmpeg library. By providing its own mythffmpeg, MythTV can assure that all of its auxiliary tools and scripts will use the exact same version of the library, and not introduce incompatibilities.

Not all of the changes in this release will have an immediate effect. The new APIs will take some time to attract plugin authors and other developers, as will assorted niceties like support for animation in UI themes. But they bring much-needed functionality to the project, which is vital for the project's future in the face of growing competition.

Forking MythTV

MythTV's fastest-growing competitor in recent years has been XBMC, a "media center" application that is also open source and cross-platform. But XBMC has historically focused on local file playback and Internet-delivered media, to the exclusion of live television and DVR functionality. By not worrying about storage, recording, or transcoding media, XBMC has made inroads into low-power set-top boxes, which MythTV never made a significant dent in. The argument from the XBMC camp has been that broadcast television is a dying delivery medium, to be overtaken by Internet entertainment and information services.

Yet that Internet-dominated future still hasn't arrived, and perhaps in recognition of that, XBMC and its derivatives have started adding DVR-like features. One major derivative, Boxee, now sells a live-TV tuner, and there are popular add-ons that turn XBMC (and derivatives) into a MythTV front-end. MythTV responded by putting considerable effort into its recent MythNetvision plugin, which lets users access online content sites from within a MythTV front-end.

But a potential disruptor surfaced in February, with the announcement of Torc, a MythTV fork launched by MythTV developers Mark Kendall and Robert McNamara. Phoronix picked up news of the new project quickly, but there was little public information about the scope or aims of the project. As an AVSForum poster discovered, however, the project did register with SchedulesDirect, the non-profit electronic program guide service catering to open source projects. Projects must apply to be approved users of the service, so presumably the description provided comes from someone on the Torc team:

Subsequently, McNamara released an iOS playback app called "Torc for iOS" which explicitly supports both Torc and MythTV 0.25 back-ends. There is an almost-empty Torc project web site, but the code is hosted at Github, where you can see that the newer project is periodically re-syncing with upstream MythTV. McNamara also continues to commit to MythTV, which lends credence to the viewpoint of some commenters that Torc should be seen not as a fork but as a streamlining branch that will eventually re-merge with MythTV itself.

The SchedulesDirect description certainly emphasizes front-end functionality like hardware acceleration and UI features, which would imply that Torc is more interested in the set-top box experience than in scrapping core MythTV features. MythTV 0.25's Services API explicitly welcomes third-party application integration, so if Torc and various XBMC add-ons find it useful, it may not be a source of conflict.

On the other hand, if Torc intends to diverge from MythTV compatibility on the back-end, then MythTV would face competition on two fronts, which would be far riskier. XBMC's MythTV front-end interfaces have always been incomplete; when coupled with the fact that no other project has mustered a significant back-end DVR service, even those MythTV users who found the project frustrating to set up or troubleshoot — myself included — always came back to it. The API changes and plugin revisions in 0.25 are an excellent sign that even ten years on, MythTV is still capable of changing course and adapting to marketplace demands. For anyone interested in building a home-brewed DVR and media center, that is about all you could ask for.

Comments (1 posted)

Brief items

Quotes of the week

Comments (none posted)

A change of maintainership for bzr

Full Story (comments: none)

Calligra 2.4 released

Comments (1 posted)

LV2 1.0.0 released

Full Story (comments: none)

PacketFence 3.3.0 released

Full Story (comments: none)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (April 17)
• What's cooking in git.git (April 15)
• Perl Weekly (April 16)
• PostgreSQL Weekly News (April 15)
• Ruby Weekly (April 12)
• Tahoe-LAFS Weekly News (April 14)

Comments (none posted)

Langley: False Start's Failure

Comments (25 posted)

Natterer: Goat Invasion in GIMP

Comments (38 posted)

The H Speed Guide to Lua (The H)

Comments (67 posted)

PHP: a fractal of bad design (fuzzy notepad)

Comments (187 posted)

Page editor: Jonathan Corbet

Announcements

Brief items

Linux Foundation Collaboration Summit 2012 | Slides & Video

Comments (6 posted)

New version of MapOSMatic, an OpenStreetMap-based service to render city maps

Full Story (comments: 3)

Articles of interest

Paoli: Microsoft will engage with the open source and standards communities

Comments (107 posted)

FSFE: vendor lock-in in Helsinki

Full Story (comments: none)

New Books

Head First C -- New from O'Reilly Media

Full Story (comments: none)

Calls for Presentations

Linux Security Summit 2012 - Announcement and CFP

Full Story (comments: none)

Registration is open for DebConf12

Full Story (comments: none)

Upcoming Events

OSCON announced and registration open

Full Story (comments: none)

Events: April 19, 2012 to June 18, 2012

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol