LWN.net Weekly Edition for June 28, 2012

LinuxCon Japan: The business of contribution

Bdale Garbee has been involved in free software, particularly Debian, for a long time, but he's also been involved in the "corporate side" of open source at HP for quite some time as well. That gives him a good perspective on the interface between companies and free software projects. In a bit of a reprise of a talk that goes back to 2003 or 2004, he spoke at LinuxCon Japan on "The Business of Contribution"—how companies and free software projects can benefit from each other.

There are multiple ways that someone can become involved in the free and open source software (FOSS) ecosystem. They range from those who take the code and use it in some way all the way up to those that lead and manage a project or subsystem. Garbee said that he hoped to convince attendees to move their way up to higher levels of engagement in FOSS.

In general FOSS projects have certain things in common. They are developed and supported by the community, there is no single company in charge, and there is a wide range of contributions that come from people with a variety of interests, abilities, and motivations. The users of the software have flexibility in how they acquire support; they can become a developer or pay someone to do development for them. This puts them in control of their own destiny, which stands in contrast to license arrangements where users just get binaries.

The key to FOSS licensing is that "people we don't even know are empowered to build things we can't even imagine", Garbee said. Everyone can take advantage of those newly built things, both individuals and companies.

Business and community

Something that is probably immediately obvious, he said, are that the expectations are somewhat different for FOSS developers and companies. It is a bit difficult to characterize what FOSS developers are looking for because each project and developer is different, but there are some common elements.

Developers often want to scratch an itch and solve a problem that they are having. They also are generally not working to a particular schedule, something that is particularly true with Debian for example, he said. FOSS developers are looking for the fun that comes from a collaborative development project. Lastly, FOSS development is a means to develop and maintain a personal reputation, which is something that can be hard for companies to understand when hiring them.

On the other side of the coin, companies need to make money. In fact, in the US, companies are legally required to make decisions based on their revenue generation possibilities. Companies also must grow and expand to provide the financial return that investors expect. That means that companies want to be able to differentiate their products from their competitors' products. Corporate reputation is also important, but there are so many ways to lower a company's reputation, and seemingly few ways to increase it, which makes doing new things risky, he said.

But there is good news in that there is something that the community and companies can agree on: positive user experience. For the community, that will increase the number of people who want to use and benefit from what it has made. For companies, a great user experience will help motivate customers to want to give them money.

Licenses and business models

FOSS licenses have their basis in copyright law, which differs somewhat in various jurisdictions, but there is a lot of commonality. The choice of which license to use for a given project is made by the founder(s) and those that join the project later are explicitly accepting those terms for their contributions. That "very special and unique right" to set the license is an important decision as it can have a "really profound impact" on the nature of the community that forms. In addition, a particular choice of license can sometimes cause new projects to form using different license terms.

There are lots of different open source software licenses available—60 or so—which is a bit unfortunate. But, they break down into two basic types: reciprocal or permissive. Reciprocal licenses require that one pass on the same rights to the code to those you distribute it to, while permissive licenses do not require that. The reality is that each license type has its place, Garbee said. It is important for project founders to carefully consider the license they choose, while it is equally important for contributors and users to understand the license that is being used.

It is vital to recognize that "open source" is not a business model, it is, instead, a process for development. One can use open source to enable the sale of hardware—this is what HP does—by ensuring that Linux runs well on that hardware. That is one possible business model, but there are others.

Selling services, like training or support, is a common example of a business model around FOSS. One could also give the source away, but charge for access to the binaries. That may sound strange, but it is more or less what the enterprise Linux distributions do, he said. Another model is to open up the core of some large software system, perhaps a framework and a few sample modules, and build a business around selling add-ons. That model can work, but it sometimes becomes something of an arms race as FOSS developers create and contribute add-ons that compete with the ones that are sold.

Working with FOSS gives a company the "opportunity to harness the creativity of a very diverse community". There is no company on earth, or even a collection of companies, that can bring to bear the talents that exist in a community like that surrounding the Linux kernel. Finding a way to work with FOSS communities can be an enormous boon for a company.

Deciding to contribute

There are multiple benefits to a company for participating in open source. For one, the software has "zero marginal cost", which means that a second (or third ...) copy costs nothing extra. The web provides a low-cost collaboration and distribution mechanism, which also reduces costs. Most of tools are also available as free software, so building a product is less costly. There is an enormous amount of existing code in the form of libraries and tools that can be used to build a product more cheaply and with a faster time to market.

There are multiple levels of engagement with FOSS, starting with taking the code and running it. Those who do so are valid members of the FOSS community. Going beyond that and monitoring the development or user mailing lists makes you a bit more engaged. The next step might be to find and fix a bug or to answer a question on the list. Each step up gives a person more influence in the direction of the project.

There are more and less efficient ways to use open source in a product, he said. It is common that one or more OSS components will require some change before they can be used, and that is "completely legally OK". If there is a reciprocal license, the changes must be distributed, but that's not a big deal. The problem occurs when it is time to put out a new version of the product. If the changes have not gone upstream, they will need to be ported to any new version of that upstream project.

On the other hand, if the changes have been contributed upstream—and shepherded through the process of getting them merged—the features and bug fixes needed will be there in the next versions. That means that the work that was done once won't have to be repeated each time a new version of the product (which may require a new version of the upstream project) is built. That doesn't mean there is no work that needs to be done, as the new version may have new bugs that need to be addressed, but it is much more efficient to get changes upstream.

For the kernel, things like device drivers will be kept up to date as the internals of the kernel change over time. The kernel is a good example because it is such a fast moving and dynamic body of code, but the situation exists in other projects as well.

Choosing to use FOSS is a "first step to making the best use of resources", Garbee said. That allows a company to focus on differentiating its product. But maximizing community involvement helps to grow the body of technology that's available to be used in future products, without having to reinvest over and over again.

Using FOSS does not mean that a company has to give everything away for free. But it doesn't mean that the company gets everything it wants for free either. There will always be a small piece that the company will need to invest in, but in spite of that, working with FOSS projects is an "incredibly powerful model".

Getting more involved with a project can bring other advantages too. One can help influence the project's direction and learn about enhancements that others are working on. Learning about security vulnerabilities and other problems in the code earlier will also be useful. It will give insight into who is contributing to the project, which may lead to interesting information on what competitors are up to. Because of that, Garbee is surprised by the email he sometimes gets from people working at other companies who didn't expect some announcement that HP has made. Had they been paying attention to the projects that HP was working on, the announcement would probably have been less surprising.

One way to experiment with FOSS is to work with partners on a project without really committing the company to doing the work, but that will not build "FOSS equity". The technical accomplishments and the reputation that the project builds will not accrue to the company (and instead to the partner). A better way is to have company employees who work directly on FOSS projects. The expertise that they gain can be reapplied to other projects, and the company gets to help set the direction for the project. In addition, the company can take credit for the work it is doing.

One important thing to recognize, however, is that reputation is built by individuals. Getting code accepted involves a combination of technical prowess and reputation within the community. In that case, the reputation that matters is that of the individual. Linus Torvalds has said that he doesn't really know the corporate affiliation of most kernel developers. In order for the company to be successful in the community, it must allow individuals to gain reputation in that community. It is important for managers to understand this, he said, and to make it a part of the career development of the employee.

Garbee noted that he once was talking to a peer at a competitor who asked about how he should review a particular employee. That person had been tasked with getting some code upstream, but ran into a competing proposal. The employee then made sure that all of the company's needs were met in the alternative, which eventually was merged. The peer was wondering whether the employee should be rewarded for that, and Garbee's response was "give the guy a raise". He recognized that the alternative proposal was better, dealt with his ego, and ensured that the company's requirements were met. He achieved the desired result, rather than get his own code upstream, which fully met the objective. That is something that many managers have a hard time understanding, he said.

Does this really work?

Garbee then described HP's experiences with FOSS as something of a justification for what he had been saying. In the last ten years or so, HP has accomplished a great deal by trying to engage with FOSS projects "as best we can". That is one of the things that has led HP to be the world's largest IT company by many measures, he said.

One of the more significant recent decisions that HP made with regard to FOSS was for the HP Public Cloud, which is based on OpenStack and Ubuntu LTS. The company has made a "significant commitment" of HP resources to OpenStack. It is now one of the largest technical contributors to the project and hosts its build farm. In addition, HP is helping with the transition of OpenStack to a "foundation model", similar to that of Eclipse or Apache.

HP has recently completed an analysis of all of its products and found that the majority contain at least some FOSS. Any of those that do have FOSS components are increasing that percentage rapidly. It is the most efficient use of HP's software engineering effort, he said. All across HP, there is more and more direct engagement with FOSS projects, which is helping to make the company more successful.

As a consequence of the talk, Garbee said that he hoped the audience would find that becoming more directly engaged with FOSS projects made sense for them and their companies. He ended with a famous quote from Margaret Mead:

That is, he said, a strong statement of how things work in open source. If someday you look around and wonder why it is that something isn't fixed, remember that you are someone.

Garbee's talk was clearly aimed at the mostly Asian audience, and tried to assist those present with arguments to make to their management about the benefits of working more closely with FOSS projects. The arguments are useful for anyone trying to get their company more involved in FOSS projects, of course, but Asian companies have traditionally been less engaged with upstream projects. He also slipped some rocket slides into the talk, which, beyond bringing some of his hobby into view, also gave him an opportunity to show a more personal connection to open source.

One of those slides showed him carrying a rather large rocket to the launching pad (shown at right). That rocket was subsequently lost after reaching Mach 1.3 and attaining 5000m above the ground because of a malfunctioning closed-source altimeter. That led him and Keith Packard into a collaboration to create open hardware and software for rocket telemetry and control, which ultimately helps him "not lose rockets". So, not only is open source good for companies, it's good for rockets—and other hobbies too.

Comments (3 posted)

CloudLinux busted — twice

Loadable kernel modules can only access kernel symbols that have been explicitly exported for that use. Two types of declarations are used to export these symbols. EXPORT_SYMBOL_GPL() is an indication that any module using the symbol is reaching so deeply into the kernel that it is, by necessity, a derived product of the kernel. Instead, EXPORT_SYMBOL() is used for symbols that only might indicate a derived-product status; it is also used for symbols that were already exported before the GPL-only mechanism was added.

The kernel's module loader includes an enforcement mechanism that prevents any module without a GPL-compatible license from accessing explicitly GPL-only symbols. To function, this mechanism clearly must know what the license for a specific module is. That is the weak point of the entire scheme: the kernel uses the honor system. Loadable modules include a special declaration that tell the kernel which license applies; the module loader looks at that declaration and decides whether the module will have access to GPL-only symbols or not. If the module lies about its license, the kernel will normally trust it.

That is where CloudLinux comes in. This company offers a CentOS-like distribution aimed at the needs of cloud computing providers. Essentially, CloudLinux has taken a RHEL clone, added its own container mechanism, and created a business around providing updates and support. The container mechanism is at least partially implemented by the company's "lve" kernel module; lve includes an access control and resource limit mechanism that, seemingly, is implemented independently of the kernel's control group subsystem. It can place limits on CPU usage, I/O bandwidth usage, and memory consumption, among other things; all this is managed through an administrator-friendly XML file.

Matthew Garrett recently became aware that, while the lve module claims to be GPL-licensed, there is not any source available for it; he responded with a patch causing the kernel to recognize and blacklist this module. Since lve uses GPL-only symbols, this change will prevent it from loading, essentially breaking it. This change has yet to be pushed into the mainline, but it has been accepted by the modules maintainer (Rusty Russell) and marked for stable updates as well. Unless something changes, it will soon become impossible to load the lve module into any current kernels.

The immediate effect of this change on CloudLinux is likely to be limited. Chances are that few users download the binary lve module and load it into their own kernels; almost all lve users are probably running the entire CloudLinux distribution. CloudLinux could easily patch its own kernel to remove the check for the lve module and restore the previous functionality. But such a move would almost certainly be noticed, and there is a distinct possibility that one or more kernel developers would react in an unpleasant way. It would not be an advisable course of action.

What CloudLinux did instead was to apologize and plead for time:

This response was not entirely well received; it is not at all clear why CloudLinux is unable to immediately ship the source corresponding to the binary module it is distributing. That said, the community will probably limit itself to grumbling as long as CloudLinux follows through and comes back into compliance with the kernel's licensing. Nobody likes a delay in a source release, but short delays are usually tolerated, grudgingly.

So the licensing part of this episode is, probably, resolved, but it is also interesting to look at why the lve module needs GPL-only symbols in the first place. The specific symbols it needs relate to the creation of binary attributes (essentially unstructured sysfs files) in the sysfs virtual filesystem. Much of the internal sysfs interface is GPL-only, and the functions for the management of binary attributes (which are generally discouraged) are certainly so. It turns out that lve creates binary attributes for a reason that would certainly never get through a proper review:

As a comprehensive security solution, this implementation leaves just a little bit to be desired.

CloudLinux is trying to reduce the amount of damage that could be done should an attacker successfully compromise a daemon running on the system. Kernel developers have been working for years to mitigate just this sort of threat; the results of that work take the form of security modules, namespaces, and more. Had CloudLinux done its work in the open, it would certainly have been directed toward solutions that have support from others and years of experience in actual use. Solutions, in other words, that might actually deliver some additional security. Instead, they got a one-off "not pretty hack" that could never get upstream.

If CloudLinux follows through on its plan to close the module entirely, it will have to cease use of GPL-only symbols and, thus, will not be able to implement this particular mechanism. That may be enough to convince kernel developers to leave it alone. But it should leave others wondering what other surprises may be lurking inside—surprises that cannot be investigated, since the source for the module is not available.

There may well be a viable business in the creation of a well-supported distribution aimed at the needs of cloud providers. With solid security support and some top-tier user-space management tools, a company like CloudLinux could well produce an offering that others are willing to pay for. But the most robust and trustworthy server distribution will come from using (and improving) the work that many developers have done for years to create a useful containers implementation. It is hard to imagine that a proprietary solution put together by a small company can work as well. Compared to that, the risk of licensing troubles seems like a case of adding insult to injury.

Comments (17 posted)

GNOME and input method integration

Those of us who type in Latin characters may easily overlook what it takes to get text into windows or command lines in other writing systems. Entry of characters not found on one's keyboard requires the use of an input method (IM) which turns multiple keystrokes into characters. There are plenty of capable projects, but they often lack deep integration into the desktop environment or widget toolkit. In April, GNOME developer Rui Matos proposed a feature for the upcoming GNOME 3.6 release that would integrate the IBus framework into the core GNOME desktop, tackling this precise challenge. IBus is a framework that allows the user to select — and switch between — multiple IMs. The plan spawned considerable debate, not only on the merits of IBus, but on the wisdom of tightly integrating a single component into the desktop environment. Complicating matters is the divide between the bulk of the GNOME developer community and those users who depend on input methods, primarily from the Chinese-Japanese-Korean (CJK) language communities.

At the heart of the issue is how CJK users input text. In alphabetic or syllabic writing systems (including European, North Indic, and Middle Eastern), there is a known mapping between the pronunciation of a word and its written representation. In the logographic writing systems of CJK, where the strokes of the character do not represent sounds or other subdivisions of the word, users make use of an IM instead. There are phonetic IMs such as Pinyin (in which the user types in the romanization of a word on the keyboard and the IM substitutes the correct character, or logogram), shape-based IMs like Cangjie (which decompose the logogram in a strict order), and many hybrids. In most cases, good dictionaries or tables are required, plus word-prediction, spell-checking, and other add-on features to cope with homophones and other tricky bits. Mobile phone users got a taste of the IM experience through T9 and other numeric-keypad predictive text systems, which have largely been replaced.

For everyday typing, the challenge is greater because no one input method is inherently superior: if you do not know how to pronounce an unfamiliar word, a phonetic method is no use, and switching to shape-based method makes sense. On the other hand, typing a word that you hear but have not seen written down demands a phonetic method. Likewise, regional accents can have very different pronunciations for the same logogram, but there is often more than one way to decompose a logogram by shape — not to mention the problem of writing reforms like Simplified Chinese, which is not simplified uniformly between mainland Chinese hanzi and Japanese kanji. Thus, almost no IM can be relied upon to the exclusion of all others. Throw in open source developers' penchant for reinventing the wheel to scratch their own itches, and the result is multiple IM frameworks, each of which can load individual IMs as the user chooses.

Supporting all of these frameworks is a challenge, one that Matos (who works mostly on gnome-control-center, GNOME Shell, and other desktop components) felt detracted from GNOME's ability to provide a consistent desktop experience and left CJK users a step behind those users whose writing system works out-of-the-box. As he explained on the desktop-devel list, IM framework proliferation reduces the odds that any of the frameworks will get enough testing to be robust, and it greatly complicates bug triage for GNOME. Plus, many of the popular IM frameworks also attempt to be cross-desktop, which makes integrating them seamlessly into GNOME difficult. Their visual appearance does not match, they conflict with core GNOME settings like XKB configuration and key bindings, and they cannot be automatically started (relying either on user interaction or shell scripts to launch). The fix he proposed is to take one IM framework, tie it in more directly to GNOME (including adding or extending GNOME APIs where needed), and make sure that it works for the majority of users.

Specifically, Matos said, GNOME has three requirements to properly integrate an IM framework with the desktop: a GTK+ module, a D-Bus API that can enumerate, activate, and configure installed IMs, and a D-Bus API on GNOME's side that the framework can use to draw predictive-text pop-up windows. Right now, he said, only IBus meets all of the requirements. Owen Taylor added some additional requirements to integrate the framework with GNOME accessibility and configuration standards, and a quality set of existing IMs. The IBus team expressed a willingness to work with GNOME, which was also critical.

Feedback from the CJK community

However, the plan elicited strong reactions from some members of the CJK user community. Marguerite Su replied that few if any CJK users like IBus; with most preferring rival framework Fcitx out of the existing options available on desktop Linux. Others chimed in with criticism of IBus. The complaints included several overlapping issues, including customization features, IM engine quality, and speed. Su elaborated on the feature disparity, saying that some users want to customize shortcuts, rearrange the word-completion suggestions, or fetch new dictionary words from Internet servers. But trading IBus for Fcitx is not the simple solution. IBus works well for Simplified Chinese, but not Traditional Chinese, while Fcitx is focused largely on Chinese and has less robust support for other languages, including Japanese (although it has plans to improve its Japanese support). Chinese users are greater in number, which might argue in favor of Fcitx, but surely the best solution would be to find a framework that serves everyone. Support for writing systems unrelated to the CJK family (such as Tibetan or Thai) is weaker in general across the IM frameworks.

But over the course of the sometimes-heated list discussion, the IBus critics proposed not simply adopting Fcitx outright, but keeping IM frameworks a user-controlled, pluggable option. For example, Liang Suilong, who argued that IBus is laggy, still advocated building an "IM framework framework" for GNOME, thus allowing users to choose the framework they preferred. But that idea was not well-received by GNOME developers. Tomas Frydrych summed up the two sides of the divide:

As others pointed out on the list, the latter point was not accurate; IBus does support CJK, but not all users are happy with its IM implementations or auxiliary features. Complicating matters more is the cultural divide between the groups. Weng Xuetian pointed out that the core GNOME developers are not IM users (most speak European languages), which makes them unqualified to select the "best" IM framework to then be used by a large number GNOME users who were not able to participate in the discussion.

Frameworks, not engines

But the groundswell of IBus criticism was not the final word. Tommy He said that the debate was too focused on CJK input, to the exclusion of other writing systems, and moreover that much of the lobbying in favor of Fcitx focused on the quality of the various IM engines, rather than the frameworks themselves. After all, if Fcitx can add a new and improved Japanese IM, then IBus could add one for Traditional Chinese.

Admittedly, the framework-versus-IM debate is a difficult one to pin down. On the one hand, Fcitx proponents enjoy its more flexible plugin system, which allows user-defined macros and other daily-use features. But that same flexibility could make it more difficult to integrate with GNOME's existing language settings and preferences, when providing a better first-run experience is explicitly one of the goals. As Owen Taylor observed, Fcitx generates its configuration GUI on-the-fly, which is a technique GNOME finds problematic and tries to avoid.

On May 14, Taylor attempted to re-center the discussion to the framework question, arguing that GNOME needs to pick one framework, then make it work "as well as we can possibly make it for all users. Multiple partially working frameworks are not a substitute." When the IBus-versus-Fcitx debate briefly resurfaced, Bastien Nocera advocated starting the IBus integration work as soon as possible, then inviting Fcitx to bring its own code demonstrating that it could fit the role better. "A lot of the code should be reusable, and you can show how much more awesome and complete your favourite IMF is."

When it looked like the core GNOME developers had made their final decision to integrate IBus, a few IBus detractors asked the team to postpone the decision for further discussion, but without success. Nocera reiterated his opinion that it would be better to pick an imperfect IM framework and replace it in later releases than to do nothing:

But Jasper St. Pierre compared the decision to other components, like audio. GNOME does not attempt to support ESD and OSS in addition to PulseAudio due to limited resources, he said, and GNOME's choice of IM frameworks will not force distributions to follow suit. "I doubt we have the resources to support both IBus and FCITX, and provide a good experience for both. Individual distributions may, but that's their call, not GNOME's."

The Fcitx lobby may not be satisfied with that response, but it looks like IBus is here to stay for the 3.6 development cycle at the very least. The GNOME wiki page lists seven tracker bugs to follow the integration with GNOME Shell, GNOME control center, and other components. There is still plenty of integration and debugging work to be done, including improving IBus's support for Hong Kong localization and reducing interference with other applications.

In an email, Matos said that he thought many of the user concerns about losing Fcitx boiled down to UI/UX issues, particularly attachment to "things like skins and themes for the UI popups. In this regard I just can say that as far as gnome-shell is (somewhat) themable people will be able to do themes for gnome-shell's IM popups." On the other hand, he added, there are Fcitx features like spell-checking that should be handled desktop-wide, while features like retrieving word lists from an online service "is orthogonal and there's no reason it can't be implemented [in] IBus." It may not be an easy journey, but clearly pulling CJK users into the fold with first-order input support has potential benefits that far outweigh the costs.

Comments (43 posted)

Page editor: Jonathan Corbet

Security

Ubuntu details its UEFI secure boot plans

UEFI Secure boot is expected to interfere with many users' desire to replace Windows or dual-boot it with Linux, because Microsoft is mandating that secure boot be enabled on Windows 8 machines at the time of sale. On June 5, we reported on Fedora's plans for handling the secure boot mechanism in UEFI. Ubuntu has subsequently announced its own plans, which take a different approach.

To recap, the secure boot feature constrains the hardware only to boot software that has been signed by a known cryptographic key. The point is that booting only signed, trusted binaries prevents attacks through boot-time malware that could be undetectable after the infected system is up and running. Microsoft is requiring hardware vendors to have secure boot enabled if they want to include the official logo for the upcoming Windows 8, although x86 vendors are also required to allow the machine's owner to turn off secure boot entirely or to install new keys. That option is regarded as insufficient for several reasons, notably that there may be users who are required (e.g., by office rules) to keep secure boot switched on, and that entering new keys for every alternative OS is likely to be an arduous process (even more so for the scenario where one needs to boot a temporary OS, such as from a CD or USB key).

Fedora's strategy is to enroll in Microsoft's developer program, which allows the project to purchase an approved $99 key through Verisign, a key which will be recognized by UEFI secure boot. The key will be used to sign the shim bootloader, which is a "trivial UEFI first-stage bootloader" whose only job is to boot GRUB2. Fedora will also sign the GRUB2 bootloader and the kernel, although the latter two binaries can be signed with the Fedora project's own keys.

Ubuntu's plan

Canonical posted a brief announcement about its own secure boot plan on the company blog on June 22, although the details were to be found in Steve Langasek's message to the ubuntu-devel mailing list. Canonical has generated its own signing key which will be pre-loaded on machines that ship with Ubuntu already installed. Ubuntu CDs will ship with a shim bootloader (the same shim bootloader used by Fedora) signed by one of the existing Microsoft-certified keys, much like the Fedora plan.

After that point, however, the distribution is taking a markedly different approach to the trusted bootloader chain. An Ubuntu system will boot into the efilinux bootloader, which will in turn boot an unsigned kernel image. Under Fedora's plan, the shim bootloader verifies the integrity of GRUB2 before loading it, and GRUB2 in turn verifies the integrity of the kernel. Canonical says that their reading of the specification makes it clear that their secure boot responsibilities stop at the bootloader, and do not extend to the kernel:

The decision to use efilinux has its own justification. Because GRUB2 is licensed under the GPLv3, Canonical determined that machines with Ubuntu pre-installed are subject to the "User Product" provisions of GPLv3, which requires that the distributor provide the user with all authorization keys required to install the software. The company consulted with the FSF about that topic, and were warned that the authorization key clause would probably (although not definitely...) apply. Thus, if a hardware vendor shipped an Ubuntu system and did not include a way for users to install keys of their own, Canonical would be compelled to disclose its key. Revealing the signing key would undermine the point of secure boot and "at that point our certificates would of course be revoked and everyone would end up worse off."

Signatures, revocation, and other fine print

Ubuntu's decision to use its own key for pre-installed machines has spawned relatively little debate, but there is a sharp disagreement over the decision not to sign kernel images. Red Hat's Matthew Garrett (who authored the Fedora secure boot plan) argued that signing only the bootloader is insufficient:

Jamie Strandboge replied that "the UEFI specification and the Windows 8 logo requirements is that Secure Boot is designed to protect early boot only," and that signing the kernel and large portions of userspace is unattractive for several reasons, "not least of which is that it reduces the utility of the distribution."

Strandboge also contended that signing the kernel does not offer a significant level of protection over signing the bootloader, because the existence of any exploitable bootloader undermines the trust chain for all OS vendors. The argument goes that if DistroX's signed bootloader is vulnerable, malware authors could use it to create a malicious live CD image that will boot even on a machine that normally runs DistroY's secure bootloader with its signed kernel. Thus, signing the kernel image is useful for creating a trusted environment for user space, but it does not strengthen the protection of secure boot itself.

There is also the open question of how key-revocations and other updates to the secure boot world will work in practice. Both Fedora and Ubuntu plan to make use of a "shim" bootloader so that they can issue updates to the main bootloader without getting the updates signed by Microsoft. But the distributions will also need to issue revocations for vulnerable, signed bootloader and/or kernel images, and the process by which the OS vendor pushes those updates out has yet to be determined.

Although most multi-boot discussions revolve around dual-booting Windows and a single Linux distribution, that is hardly the only scenario. Canonical said that it will not offer its own signing key to sign the bootloaders of other distributions or vendors, which some feared would make it impossible to install, for example, Fedora on a machine that comes with Ubuntu pre-installed. However, the owners of machines pre-loaded with Ubuntu will still be able to install Fedora or other OSes in tandem, because the company will require its OEMs to include the Microsoft key in the secure boot key database alongside the Ubuntu key.

As Windows 8 draws near, the questions about UEFI secure boot and its impact on users continue to swirl. Clearly there are risks in handing the ultimate say in booting one's machine to a third party (particularly a rival OS vendor like Microsoft), and even though two of the largest distributions have crafted a plan for dealing with secure boot's restrictions, how much of an imposition the final product is still hinges on unknowns like the revocation and update process. But the biggest question that remains is whether it is wise to tacitly endorse secure boot by playing its games in first place. On that, the community may never arrive at a single answer.

Comments (26 posted)

Brief items

Security quotes of the week

Comments (1 posted)

Details on Ubuntu's UEFI secure boot plan

Full Story (comments: 112)

Android application reads credit card data over NFC (The H)

Comments (12 posted)

New vulnerabilities

apache: privilege escalation

Comments (2 posted)

asterisk: denial of service

Comments (none posted)

dhcpcd: remote code execution

Comments (none posted)

gdk-pixbuf: integer overflow

Comments (none posted)

ImageMagick: integer overflow

Comments (none posted)

kernel: NX emulation suspected broken

Comments (none posted)

kernel: denial of service and iptables bypass

Comments (none posted)

libpng: multiple vulnerabilities

Comments (none posted)

libwpd: code execution

Comments (none posted)

links: multiple vulnerabilities

Comments (none posted)

logrotate: symlink and hard link attacks

Comments (none posted)

mantis: multiple vulnerabilities

Comments (none posted)

mediawiki: multiple vulnerabilities

Comments (none posted)

mini-httpd: code execution

Comments (none posted)

mono and mono-debugger: multiple vulnerabilities

Comments (none posted)

mosh: denial of service

Comments (none posted)

msmtp: X.509 NULL spoofing

Comments (none posted)

nbd: denial of service

Comments (none posted)

network-manager: insecure WPA AdHoc connections

Comments (none posted)

nvidia-drivers: privilege escalation

Comments (none posted)

openjpeg: code execution

Comments (none posted)

php: information disclosure/arbitrary code execution

Comments (none posted)

python-httplib2: use of incorrect certificates

Comments (none posted)

roundcubemail: cross-site scripting

Comments (none posted)

rpm: multiple vulnerabilities

Comments (none posted)

tomcat: multiple vulnerabilities

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

Stable updates: the 3.0.36 and 3.4.4 stable kernels were released on June 22.

Comments (none posted)

Quotes of the week

Comments (none posted)

Developer profiles: Sarah Sharp and Thomas Gleixner (Linux.com)

Comments (1 posted)

Gettys: The Internet is Broken, and How to Fix It

Comments (8 posted)

Fusion IO picks up another developer

Full Story (comments: none)

Attack of the Kernel Komedians

For example, he found a number of variants on "Acked-by:", including:

    Cautiously-acked-by:
    Delightedly-acked-by:
    Embarrassingly-Acked-by:
    Emphatically-Acked-by:
    Grudgingly-acked-by:
    Hella-acked-by:
    Sort-Of-Acked-By:

And so on. One can only feel bad for the developers who felt the need to add "Repented-by:" or "Fatfingered-by:" and wonder about the story behind tags like "Antagonized-by:" or "Signed-off-and-morning-tea-spilled-by:". Andrew may dislike these tags, but others seem to find them amusing and will give them a "Whatevered-by:" at worst.

Full Story (comments: 8)

Kernel development news

Displaying QR codes for kernel crashes

A proposal from Cong Wang to discuss the various mechanisms to store the kernel's "dying breath" spawned a rather large thread on the ksummit-2012-discuss mailing list. While things like pstore were set up specifically to provide a means to store kernel crash information, that doesn't necessarily make it easy for users to access and report kernel crashes. That led to suggestions and discussion of better ways for users to get the information out of their crashed systems—including using QR codes to facilitate the process.

Most regular users do not have a serial console set up to record crash information on a separate machine. So the kernel backtrace that appears after a crash is just written to the console, which means that much of it will have scrolled off the screen. Even the data that is there is hard to extract, with some folks trying to type the information in, which is tedious, not to mention error-prone. A QR code that encoded the relevant data could certainly help there.

Konrad Rzeszutek Wilk was the first to broach the QR code idea, though he said it did not originate with him. It turns out that H. Peter Anvin and Dirk Hohndel have been "messing with" the idea, but Will Deacon and Marc Zyngier actually showed something along those lines at the recent Linaro Connect in Hong Kong. Deacon was hesitant to call it a prototype, but said that there was some work done on encoding a kernel crash backtrace as a QR code. There were two problems with their approach:

One of the original motivations for kernel modesetting (KMS) was to get readable oops information to the screen. Using KMS to display a fairly simple QR code graphic instead should be workable, rather than creating an ASCII version as Deacon describes. Matthew Garrett noted that it should be fairly straightforward at least for hardware that has KMS support:

There is some disagreement about where the decoding of any QR code should take place. Garrett believes that existing QR apps in phones should be used, while others are not convinced they can be coerced into being flexible enough to deal with the large QR codes that might result from a kernel backtrace. Garrett has also done some work on the problem and described his approach:

Anvin would rather see some kind of web application that accepts a photo of the QR code and decodes it on the server. For one thing, having one (working) decoding code base is desirable: "I can tell you just how bad a lot of the QR decoder software running on smartphones are -- because I have tried them." In addition, though, a web application would also have the photo itself, so even if it didn't decode because of picture quality or other reasons, those photos could be used to improve the quality of the decoder.

But that implies that a user would need to download an app to their phone or use some web application as suggested by John Hawley. Garrett was not in favor of either solution, noting that requiring an app makes its harder for users, while a web application doesn't really make it any better:

Given that many users already use photos to report crashes—taking a picture of the screen with the last part of the backtrace—the QR code mechanism, even if a bit cumbersome, might be able to provide the full backtrace. But, as Dave Jones suggested, just having scrollback available on the console after a crash would make much of the problem disappear: "What would be a thousand times more useful would be having working scrollback when we panic, like we had circa 2.2".

Users could then take a photo, scroll back a ways, take another, and so on. In the thread, there was widespread agreement that console scrollback would be desirable. But it turns out that the advent of USB keyboards caused the loss of that feature. Doing USB handling inside the panic code would be messy, so bringing that feature back is difficult. Other ideas were mentioned, like providing enough of the USB stack to write the crash information to a USB stick as Anvin suggests, or to "auto-scroll" the console output after a crash without requiring keyboard input as proposed by Paul Gortmaker.

Making it easier for users to report crashes with useful information was one branch of the discussion, but the folks who work on the embedded side are looking for more developer-oriented solutions as well. Tony Luck outlined the pstore back-ends that are currently available to store crash and other information in various places (ERST, EFI variables, RAM) that are accessible after a reboot. Wang, Tim Bird, Jason Wessel, and others are interested in discussing that piece of the puzzle.

While QR codes may seem like something of gimmick, they can compress a fair amount of data into a form that can be digested elsewhere. Getting useful information out of an unresponsive, crashed Linux system is fairly difficult at this point, so finding better ways to do so would be good. Should the program committee decide to add this topic, a lively discussion seems likely. If not, though, enough people are looking into the idea that something will emerge sooner or later.

Comments (20 posted)

printk() problems

One of the core design features of the new printk() is a change from byte-streamed output to record-oriented output. Current kernels can easily corrupt messages on their way to the log; for example, when the log buffer overflows, the kernel simply wraps around and partially overwrites older messages. Messages from multiple CPUs can also get confused, especially if one or more CPUs are using multiple printk() calls to output a single line of text. The switch to the record-oriented mechanism eliminates these problems; it also makes it possible to attach useful structured information to messages. As a whole, it looks like a solid improvement to the kernel logging subsystem.

There is just one little problem, though: when the kernel outputs a partial message (by passing a string to printk() that does not end with a newline), the logging system will buffer the text until the rest of the message arrives. The good news is that this buffering causes the full line to be output together once it's complete—if things go well. The situation when things do not go well was best summarized by Andrew Morton:

	printk("testing the frobnozzle ...");
	do_test();
	printk(" OK\n");

Not only is this behavior now broken, but it has also burned at least one developer who ended up spending a lot of time trying to figure out why the kernel was hanging. Kernel developers depend heavily on printk(), so this change has caused a fair amount of concern.

Bugs happen, of course; the important thing is to fix them. A number of possible fixes have been discussed on the list, including:

• Leave printk() as it is, and change specific callers to output only full lines. Kay Sievers, the author of the printk() changes, suggested that approach, saying "We really should not optimize for cosmetics (full lines work reliably, they are not buffered) of self-tests, for the price of the reliability and integrity of all other users."

• Adding a printk_flush() function to be called in places where it is important to see partial lines even if things go wrong before the newline character is printed. The problem with this approach is that, like printing full lines only, it requires changing every place in the code where the problem might hit. Experience says that many of those places can only be found the hard way.

• Add a global knob by which buffering can be turned on or off; this knob might be set by either user space or the kernel. This idea was not particularly popular; it seems unlikely that the knob will be set for unbuffered output when it really matters.

• Simply revert the printk() changes for 3.5 and try again for 3.6 or later. Ingo Molnar posted a patch to this effect, seemingly as a way of pressuring Kay to take the problem more seriously.

As of this writing, most of the discussion centers around this patch from Steven Rostedt which simply removes the buffering from printk(). For the most part, the advantages of the new code remain. But it is now possible that a single line of output created with multiple printk() calls may be split into multiple lines, with messages from other CPUs mixed in between. It seems to many to be a reasonable compromise fix.

Except that Kay still doesn't like the splitting of continuation lines. Andrew Morton is also concerned about where the printk() code is going, saying "The core printk code is starting to make one think of things like 'cleansing with fire'." Steven, meanwhile, is reconsidering the whole thing, saying that, perhaps, printk() is not the right tool for structured logging and other approaches should be considered. And Greg Kroah-Hartman has suggested that it might be better just to fix the call sites rather than further complicating the printk() code.

Linus, however, has argued strongly for the merging of Steven's patch. His view is that buffering at the logging level is fine, but text emitted with printk() has to get to the console immediately. So chances are that some version of Steven's fix will be applied for the 3.5 release. But it has become clear, again, that adding structured logging to the kernel while not making life harder for kernel developers is a difficult problem.

Comments (6 posted)

Tightening security: not for the impatient

Consider the classic symbolic link vulnerability, wherein an attacker fools a privileged program into writing to a file behind an attacker-controlled symbolic link. Such vulnerabilities can be exploited to overwrite files that the attacker would not otherwise have access to. One does not have to dig far into the LWN vulnerability list to see that the identification and patching of symbolic link vulnerabilities is an ongoing process. One might think that, if somebody could come up with a way to eliminate such vulnerabilities altogether, it would be adopted in a hurry.

As it happens, Kees Cook has a way to deal with this class of vulnerabilities. It is based on the observations that symbolic link vulnerabilities almost always involve links placed in /tmp, and that /tmp has the "sticky" bit set in any contemporary distribution. Given that:

In short, this change would make it so that nobody could create symbolic links in /tmp and expect a privileged program to follow them. Lest one think that Kees is taking credit for this concept, he posted a bit of history for this idea, starting with a 1996 Bugtraq message from Zygo Blaxell and a kernel patch by Andrew Tridgell from the same year. This idea, in other words, has been floating around for at least 16 years, but an implementation has never found its way into the mainline kernel. Memory management changes are amazingly fast in comparison.

The reason for the resistance, of course, is that this is a change in filesystem semantics. There are concerns that it would break POSIX compliance, though Kees claims that POSIX is silent on this particular behavior. Also of concern is the possibility of breaking existing applications. Kees responds that any broken applications would be easily noticed (while those suffering from symbolic link vulnerabilities are not), and that no applications relying on existing behavior have ever been found. There have also been disagreements over how the patch should be implemented, but those have seemingly mostly been resolved.

So Kees thinks that his current patch set (a variant of one we have seen before) should be considered for merging, finally. The patches implement the symbolic link restrictions, but also add a new rule for hard links: a hard link to a file can only be created if the user owns the file or has write access to it. Once again, this change eliminates a class of attacks, but at a small cost: older versions of the "at" daemon break unless a small patch is applied. No other problems have been found, Kees says, after 1.5 years of experience with this patch in the Ubuntu kernel.

Whether that is enough evidence to get the changes merged this time around remains to be seen. It has only been 16 years, after all, and one would not want to be too hasty about such a thing.

Meanwhile, Kees has put together a separate security-oriented patch that has run into some concerns of its own. On Linux systems, there is a sysctl knob (suid_dumpable) that controls whether a crashing setuid process generates a core dump or not. Setting it to a non-zero value allows core dumps to happen; setting it to two applies certain restrictions that are intended to make it safe. But, Kees says, that's not the case; it allows a user to create a file called core in almost any directory, containing arbitrary text (environment strings, for example). This capability is not necessarily as harmless as one might think; as the 2006 cron vulnerability shows, some programs will happily pick out the strings they understand in a file full of junk, happily ignoring the rest. Thus, he claims, allowing users to create files in arbitrary locations is asking for trouble.

His response has been through a number of iterations:

• Version 1 disallowed storing core dumps from privileged executables into a file. If the core_pattern knob is set to a pipe, instead, core dumps happen as before. This was seen as an incompatible ABI change, though, and one that would cause surprising results.

• Version 2 added a new setting (3) that would only allow setuid core dumps to a pipe. The previous "safe" setting (2) was deprecated; attempting to set it would fail with an EINVAL error. This version ran into trouble as a result of how it interacted with the sysctl mechanism.

• Version 3 fixed the sysctl difficulties but was opposed by Andrew Morton, who feared that the deprecation of the previous mode would break current systems in surprising ways. He suggested keeping suid_dumpable=2 as a working mode with a warning.

• Version 4 went back to something closer to version 1, but with some loud warnings emitted. But then Eric Biederman asked whether disallowing relative paths would be a sufficient fix.

• Thus, version 5 (the current version, as of this writing), just disallows the writing of setuid core dumps to relative paths. Should core_pattern be set to a relative path ("core", for example), a warning will be logged instead.

Thus far, there has not been much in the way of complaints about the fifth iteration of the patch. So, possibly, it will not be necessary to wait for years until this particular bit of security tightening gets into the mainline kernel. Of course, unlike the system's link behavior, the core dump behavior can be changed now by concerned system administrators—no need to wait at all.

Comments (29 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.5-rc4 . (June 24, 2012)

• Greg KH: Linux 3.4.4 . (June 23, 2012)

• Steven Rostedt: 3.4.4-rt13 . (June 25, 2012)

• Steven Rostedt: 3.4.3-rt12 . (June 25, 2012)

• Steven Rostedt: 3.2.21-rt33 . (June 25, 2012)

• Greg KH: Linux 3.0.36 . (June 23, 2012)

• Steven Rostedt: 3.0.36-rt57 . (June 25, 2012)

Build system

• Steven Rostedt: localmodconfig: Improve the number of modules removed . (June 25, 2012)

Core kernel code

• Paul E. McKenney: [PATCH tip/core/rcu 0/22] v2 Improvements to rcu_barrier() and RT response on big systems . (June 22, 2012)

• Daniel Santos: Generic Red-Black Trees . (June 25, 2012)

• Steven Rostedt: printk: Have printk() never buffer its data . (June 26, 2012)

Development tools

• Akinobu Mita: notifier error injection . (June 25, 2012)

• Luming Yu: [patch update-v1] a simple hardware detector for latency as well as throughput ver. 0.1.0 . (June 25, 2012)

• Anton Vorontsov: Function tracing support for pstore . (June 27, 2012)

Device drivers

• K. Y. Srinivasan: drivers: hv: kvp . (June 22, 2012)

• sjur.brandeland@stericsson.com: STE-modem remoteproc driver . (June 22, 2012)

• David Daney: netdev/phy: 10G PHY support. . (June 25, 2012)

• Mark Brown: mfd: Wolfson Arizona and WM5102 support . (June 25, 2012)

• Mark Brown: Extcon: Arizona: Add driver for Wolfson Arizona class devices . (June 25, 2012)

• Roland Stigge: input: keyboard: Add keys driver for the LPC32xx SoC . (June 25, 2012)

• Lin Ming: SATA ZPODD support . (June 25, 2012)

• Ezequiel Garcia: media: Add stk1160 new driver . (June 25, 2012)

Filesystems and block I/O

• Namjae Jeon: nfs: Support posix_fadvise(POSIX_FADV_RANDOM) on nfs server. . (June 25, 2012)

• Dmitry Monakhov: RFC: introduce extended inode owner identifier v9 . (June 25, 2012)

Memory management

• Rik van Riel: mm: scalable and unified arch_get_unmapped_area . (June 22, 2012)

• Mel Gorman: Swap-over-NFS without deadlocking V7 . (June 22, 2012)

• Mel Gorman: Swap-over-NBD without deadlocking V13 . (June 22, 2012)

• Glauber Costa: kmem controller for memcg: stripped down version . (June 25, 2012)

• Rafael Aquini: make balloon pages movable by compaction . (June 26, 2012)

• Frank Swiderski: Add a page cache-backed balloon device driver. . (June 26, 2012)

• John Stultz: Fallocate Volatile Ranges v5 . (June 27, 2012)

• Yasuaki Ishimatsu: memory-hotplug : hot-remove physical memory . (June 27, 2012)

Networking

• David Miller: ipv4: Early TCP socket demux. . (June 25, 2012)

• Eldad Zack: LLDP implementation for Linux . (June 25, 2012)

Architecture-specific

• Huacai Chen: MIPS: Add Loongson-3 based machines support. . (June 22, 2012)

• Matt Fleming: x86, efi: Handover Protocol . (June 25, 2012)

• Marek Szyprowski: ARM: replace custom consistent dma region with vmalloc . (June 25, 2012)

• Robert Lee: ARM: imx: Add basic imx6q thermal driver . (June 27, 2012)

• Gregory Clement: [PATCH v4] arm: Add basic support for new Marvell Armada 370 and Armada XP SoC . (June 26, 2012)

Security-related

• Kees Cook: fs: introduce pipe-only dump mode suid_dumpable=3 . (June 25, 2012)

• Kees Cook: fs: make dumpable=2 require fully qualified path . (June 27, 2012)

• Kees Cook: fs: add link restrictions . (June 27, 2012)

Virtualization and containers

• Alex Williamson: kvm: level triggered irqfd support . (June 25, 2012)

• Michael S. Tsirkin: kvm: eoi optimization support . (June 25, 2012)

• Jason Wang: Multiqueue virtio-net . (June 25, 2012)

Miscellaneous

• Peiyong Feng: Implement uhook(call kernel func from userspace) driver . (June 25, 2012)

Page editor: Jonathan Corbet

Distributions

DNF, which may or may not replace Yum

Fedora's Engineering Steering Committee (FESCo) recently voted to approve an additional feature for Fedora 18: a new alternative package manager called DNF. The primary author intends for DNF to replace Fedora's current package manager, Yum, but there are questions as to when it will be ready and why starting over is preferable to updating the existing code.

A handful of news sites reported DNF as "the" new package manager for Fedora, which might lead one to conclude that it will be the default option. However, DNF is only approved for inclusion in Fedora 18, as it does not have feature-parity with Yum and is still in heavy development. Yum will remain the system package manager, and may for several more releases.

DNF is the creation of Red Hat's Aleš Kozumplík, and is a fork of Yum 3.4 that uses his own hawkey library to resolve package dependencies. Like Yum, DNF sits on top of the lower-level rpm package installer; its purpose is to simplify package installation, update, and removal by automatically calculating dependencies and handling the management of dependent packages. The moniker "DNF" officially stands for nothing, and was chosen by Kozumplík solely as a placeholder, although the choice prompted humorous responses that it stood for un-complimentary phrases like "Did Not Finish" or the oft-maligned Duke Nukem Forever. Hawkey itself is a wrapper around the libsolv dependency resolver from openSUSE.

Some of the goals described on the wiki page include providing API access to languages in addition to Python (which is currently Yum's only option), a "strict" API, and better performance. Migrating the package manager to libsolv is the primary goal, however, with an eye toward eventually using libsolv as the dependency solver for RPM itself, as well as making it available to other programs that handle package management, such as the installer. For users, a libsolv-powered DNF's principal advantage over Yum is expected to be its speed. At FUDCon in February, for example, Yum was described as being four times slower than untar, which no one seems to regard as acceptable.

Libsolv gains its advantage by using a modern satisfiability-solving algorithm (or "SAT solver"), rather than the ad-hoc dependency checking methods employed by other major Linux package managers. Libsolv's documentation observes that most package managers are optimized to find updates to currently installed packages, with extensions tacked on to handle other situations, resulting in slow and unpredictable code. Libsolv uses a reimplementation of the open source Minisat solver, and has been adopted by openSUSE's Zypper package manager.

Integration with Fedora

DNF uses the more modern libsolv for improved dependency solving, but it does not implement all of Yum's other features. The missing functionality generated some feedback from administrators who were upset that some of the features they depend on had evidently been marked as deprecated. For example, Yum has a "history" function that allows the user to investigate logged installation and update actions, and even roll them back, but the feature was listed on DNF's features considered for dropping page. When fans of the feature complained about its removal, Kozumplík agreed to implement it in a future version.

DNF also lacks PackageKit integration, which PackageKit maintainer Richard Hughes cited as a major problem during the FESCo debate about approving the feature. Others in the discussion were unclear as to the justification and plans for DNF to replace Yum. Bill Nottingham asked why a fork was necessary, and Matthew Garrett asked:

Distinct from the issue of advertising DNF's readiness too soon is the concern that Kozumplík appeared to have launched the project without first attempting to implement his changes within the main Yum trunk or working with the Yum team. Yum developer Tim Lauridsen asked on the desktop-devel mailing list:

Kozumplík replied that a fork was required to clean up the Yum API; deprecating legacy interfaces without worrying about simultaneously preserving backward compatibility. He also said that he is collaborating with the Yum team, which Seth Vidal called a contradiction, considering Lauridsen's question:

The future of solving

Of course, maintaining the delicate balance between paid-company-developers and community developers is a challenge on which most corporate distributions spend a lot of time. At least the two camps appear to be talking now. Doubtless there is interest in the improvements offered by a libsolv-based backend from the Yum project, particularly as other pieces of the Fedora infrastructure (including rpm itself and the installer) move toward it. Libsolv is gaining supporters among other projects as well (Hughes, for instance, is also the author of the alternative package manager zif, and indicated he was planning to migrate it to libsolv in the future). There does not appear to be a plan to migrate apt or apt-rpm to libsolv, but there has been interest in writing a new SAT solver to replace apt's current dependency resolver.

But agreement on the goal does not make the process any less arduous; there are inherent difficulties in replacing a piece of software as close to the core as Yum. Dependency resolving is complicated by the number of options an RPM package can declare, such as specific versions of some libraries or "recommended" packages that a user might select (thus adding still more dependencies to the puzzle). There are also scenarios where a user wants to swap out one package for another, and would prefer to leave the common dependencies in-place in order to cut down on time required. Despite the fact that openSUSE has been using libsolv in its own package manager Zypper, hawkey and DNF will require a lot of testing to get all of the corner cases correct. Until then, Fedora users can count on Yum staying right where it is.

Comments (8 posted)

Brief items

Distribution quotes of the week

Comments (1 posted)

CyanogenMod 9.0-rc1 available

Comments (50 posted)

Red Hat Enterprise Linux 6.3

Comments (17 posted)

UEFI Secure boot using qemu-kvm

Full Story (comments: 3)

Open webOS "Community Edition" released

Comments (11 posted)

Distribution News

Debian GNU/Linux

Bits from the Release Team: Final countdown!

Full Story (comments: none)

Fedora

Fedora Board appointment

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (June 25)
• DistroWatch Weekly, Issue 462 (June 25)
• Maemo Weekly News (June 25)
• Ubuntu Weekly Newsletter, Issue 271 (June 24)

Comments (none posted)

Pettenò: Debunking x32 myths

Comments (112 posted)

Android 4.1 Jelly Bean: faster, smoother, more delightful (ars technica)

Comments (22 posted)

Page editor: Rebecca Sobol

Development

Making some Beautiful Soup

Parsing HTML is sometimes a surprisingly complicated task. Even what seem like fairly trivial constructs seem to have a bunch of "fiddly bits" that one needs to account for. For most of those kinds of tasks, then, one generally turns to a full-blown HTML parser. While Python includes one in its standard library, it can be somewhat painful to use as well. Some recent experiments with Beautiful Soup, in particular version 4 released earlier this year, have shown a parser that is well-designed and easy to use.

Beautiful Soup is available as a tarball that can be installed in the usual Python way (using setup.py). It can also be installed using pip or easy_install from the PyPi repositories. It is also packaged as python-bs4 for Debian and Ubuntu; packages for other distributions will presumably be coming along as well. It supports both Python 2.7 and Python 3, with few external dependencies. Beautiful Soup uses the Python standard library parser, but can also use optional faster parsers (lxml, html5lib) if they are installed.

Getting started with Beautiful Soup is pretty straightforward:

    from bs4 import BeautifulSoup

    soup = BeautifulSoup(string_or_filehandle)

But, there's lots more available than simply cleaner HTML. It can also take HTML fragments, which will be transformed into full HTML documents once parsed. Beautiful Soup breaks the HTML down into objects which correspond to the tags contained in the input. For example:

    print soup.head
    print soup.body

    html_frag = '<p>foo bar</p><p>baz<p>yet another graf</p>'
    soup = BeautifulSoup(html_frag)
    print soup.body.contents
    for c in soup.body.children:
        print c

    # output
    [<p>foo bar</p>, <p>baz</p>, <p>yet another graf</p>]
    <p>foo bar</p>
    <p>baz</p>
    <p>yet another graf</p>

Each tag in the HTML gets turned into a Tag object by Beautiful Soup. Tag objects have attributes like .name, .children, .parent, and so on, which can be used to distinguish various tags and to navigate the tree. In addition, the HTML attributes of a particular tag are available by accessing the tag as a dictionary:

    soup = BeautifulSoup('<b class="boldest">foo</b> bar <b class="bolder">baz</b>')
    print soup.b['class']

    # output
    ['boldest']

But wait, there's more:

    soup.b['class'] = 'waylessbold'
    print soup.body

    # output
    <body><b class="waylessbold">foo</b> bar <b class="bolder">baz</b></body>

For many transformation tasks, though, one might rather not step through the whole HTML tree, and would, instead, want to search for tags of interest. Beautiful Soup has some powerful capabilities in that area too. Using the above example:

    for b in soup.find_all('b'):
        b['class'] = 'justbold'

    for a in soup.find_all('a', href=re.compile(r'^/')):
        a['href'] = 'http://lwn.net%s' % a['href']

One can also create a dictionary with just the attributes needed (or wanted) on a particular tag type and simply assign those tags using the .attrs attribute:

    for img in soup.find_all('img'):
        idict = { 'height' : 22, 'width' : 42, 'src' : img['src'] }
        img.attrs = idict

One can also create and insert entirely new tags into the soup object (and thus the HTML). A tag can be created with new_tag() then inserted before or after any other tag:

    itag = soup.new_tag('i')
    itag.string = 'italicized'
    soup.b.insert_before(itag)
    print soup.body

    # output
    <body><i>italicized</i><b class="justbold">foo</b> bar <b class="justbold">baz</b></body>

There is one seeming oddity with Beautiful Soup in that there is another type of object called a NavigableString. Any string in the input (even those that are the child of a Tag, e.g. the "foo" in <b>foo</b>) will be represented as a NavigableString. These can be encountered while moving around in the HTML tree:

    soup = BeautifulSoup('<i>italicized</i><b class="justbold">foo</b> bar <b class="justbold">baz</b>')

    for t in soup.body.children:
        print t.name

    # output
    i
    b
    Traceback (most recent call last):
    ...
    AttributeError: 'NavigableString' object has no attribute 'name'

    from bs4 import NavigableString

    if isinstance(child, NavigableString):
        ...

Overall, Beautiful Soup is rather impressive. There is quite a bit more to it than this brief overview shows, but the documentation is excellent and provides lots of examples. If you have some need to parse or transform HTML in Python, Beautiful Soup 4 is surely worth a look.

Comments (8 posted)

Brief items

Quotes of the week

Comments (14 posted)

Enlightenment (EFL) gains physics support

Comments (none posted)

KDE Announces 4.9 Release Candidate 1

Full Story (comments: none)

SyncEvolution 1.2.99.1 released

Full Story (comments: none)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (June 26)
• What's cooking in git.git (June 25)
• GNU C Library status update (June 25)
• Perl Weekly (June 25)
• PostgreSQL Weekly News (June 24)
• Ruby Weekly (June 21)
• Tahoe-LAFS Weekly News (June 25)

Comments (none posted)

Firefox for Android may become your favorite mobile browser (ars technica)

Comments (21 posted)

St. Pierre: The Linux Graphics Stack

Comments (128 posted)

Page editor: Nathan Willis

Announcements

Brief items

GNOME Foundation board election results

Comments (none posted)

Articles of interest

Jonnalagadda: outsource vs open source

Comments (none posted)

Why We're Fighting for an Open Cloud (Linux.com)

Comments (30 posted)

LF Announces the Finalists in 2012 T-shirt Design Contest (Linux.com)

Comments (none posted)

FSFE: Fellowship Interview with Bjarni Runar Einarsson

Comments (none posted)

Education and Certification

LPI announces "School-to-Work" Program in Spain

Full Story (comments: none)

LPI Linux Essentials Program Available Now

Full Story (comments: none)

Upcoming Events

Linux Security Summit 2012 - Schedule Published

Full Story (comments: none)

LCA2013 opens call for sponsors

Full Story (comments: none)

Events: June 28, 2012 to August 27, 2012

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol