LWN.net Weekly Edition for February 25, 2010

SCALE 8x: Moving the needle

There are lots of ongoing efforts to increase the number of women participating in free software, but reports on how those efforts have fared are few and far between. Sarah Mei spoke at the Women in Open Source (WIOS) conference, which preceded SCALE 8x, to report on what she and other members of the San Francisco Ruby community have been doing to bring more women into that community. Her talk, Moving the Needle: How the San Francisco Ruby Community got to 18%, looked at the goals that were set, the methods that were used, and the results.

Mei had been involved in various communities over the last 15 years, including Java, PHP, and Linux, and she had never really thought about why there weren't very many other women active in those communities. But, when coming back into the Ruby community after not being a part of it for a few months, she attended the Golden Gate Ruby Conference (GoGaRuCo) in 2009, which was infamous for a presentation that featured soft-core pornography in its slides. That conference, with around 200 attendees, five of which were women—including, in an amusing coincidence, three named "Sarah"—became something of a turning point for Mei.

She started out by posting about it to her blog, but soon realized that the presenter didn't really mean to be demeaning and was, instead, just a "socially awkward computer programmer". She didn't think she could change the person, so she started thinking about changing the community. In particular, if you could "change the audience at these events" such that it was 100 women and 100 men, she believed that inappropriate presentations would naturally fall by the wayside.

So she got together with one of the other Sarahs (Allen) to come up with ideas on how to get more women into the community. What they came up with was workshops to teach Ruby and Rails to women. But they also set a goal of 50% participation by women in two separate community events. The monthly Ruby "meetups", which had about 2% participation by women in January 2009, and the 2010 GoGaRuCo, which will be held in September, were the targets. As of January 2010, they are already up to 18% women at the meetups.

[PULL QUOTE: San Francisco is the "center of the Ruby universe", Mei said, with 1600 people on the meetup mailing list. In contrast, the Silicon Valley list has just 25 people on it. END QUOTE]

San Francisco is the "center of the Ruby universe", Mei said, with 1600 people on the meetup mailing list. In contrast, the Silicon Valley list has just 25 people on it. In addition, Ruby is "trendy", so people are interested, which made them think that free workshops for women covering Ruby would be popular, and "we were right". For other communities, other kinds of events might be better, and anyone targeting those communities needs to figure out what the right kind of event is.

So far, they've had three workshops attended by a total of 250 people. But events aren't all they do. There are three things that need to go together: setting goals, doing events, and cultivating people. Many efforts at community building focus on the events and "fail to set goals and cultivate the people that they get".

Goals should be very specific and should focus on something that you can fix. Mei had not gotten involved before because it seemed like such a huge problem to solve. By focusing on specific, achievable goals, like getting more women to come to each successive monthly meeting, they reduced the problem considerably. Now, that success with the monthly meetings can be used to assist the longer-term GoGaRuCo goal.

For the workshops, they decided to target very specific audiences. Targeting all women is not specific enough, nor is targeting all women developers. Their focus was two groups: women who had been out the workforce for a bit (often due to having a child) and women who work at companies that use Ruby, but aren't programmers. They used the Meetup.com infrastructure to organize the workshops, not because they liked it particularly, but because it tied in well with the existing monthly Ruby meetings.

The workshop logistics were not the hard part, she said. Finding a room, getting enough food, and getting sponsors was fairly straightforward. Sponsors were in fact the easiest part; they told people they wanted to train more women in Ruby and sponsors "threw money at us". One thing she suggested as a way to help people attend was to offer child care. They got a few husbands of attendees to volunteer and "locked [them] and the kids in a room with a Wii". Part of their target was moms, but even if that's not the case, offering child care can help as it may well be that both parents want to attend.

Attendance is not limited to women, as each women can bring a male guest. In addition, men are welcome as volunteers to help teach the workshop material as a TA. It's important to remember that the idea is to integrate women into the wider community, so adding men from the community to the workshop is important, she said. She also suggests having an after-party for all the participants and volunteers. Giving free drink tickets to the volunteers is a good way to get them to stick around for the party, which also helps with community integration.

Cultivating people is the other part of the puzzle. You need to "cultivate people at both ends of the pipeline", first by getting them in the door, and then, once they leave the event, by helping them continue in the community. Sending personal email—not mass email—to participants or potential participants is a good way to connect. They have also been successful in getting participants to volunteer to help with the next workshop, which is another way to keep the connection going.

Mei noted that it is much like sales. You need to get the word out to everyone you meet that might be interested. Printing up business cards with information about the workshops, posting information to a blog, and going to related meetings and conferences to talk about it are all things that can be done to attract more people. It is a "winnowing process", as some small percentage of those you tell will come and a small percentage of those will actually become Ruby developers. Getting five new developers out of the 200 women that have attended the workshops so far would make her happy.

Many women don't like to be visible in the community, but it is essential. When an organizing committee for a conference or event is not all-male, it says something about the organization. Women need to be willing to put their names on events, contribute on mailing lists, and ask questions after talks. She has noticed that it is mostly men who ask questions after a talk—"change that".

One of the interesting outcomes of the workshop effort has been higher attendance by women at the monthly meetings, some of whom hadn't come to one of the workshops. A critical mass effect has been achieved, so that "once the stigma was removed", more women started showing up.

Some unexpected things have happened, which may not be directly attributable to more women being involved, but they are correlated in time. The mailing list has been more active and lively, the talks are more varied and interesting, and more women are volunteering to give talks. She thinks that the influx of women, especially some asking more basic questions, has made the men feel more comfortable on the mailing list because they now "have permission not to know everything". They are more comfortable "not knowing all the answers", she said.

So, why is increasing women's participation so hard? Why haven't things like what has happened in San Francisco happened everywhere? Mei said that it really requires a woman or two to be willing to be visible. Their presentation materials are available if other people want to try the same kind of workshop. The problem is social, not technical, and, while we are "really good at solving technical problems", anything that is "a little more touchy-feely doesn't go so well".

What Mei and others have done in San Francisco looks promising as a model for other communities in other regions. As she pointed out, looking at the community to be served is important, as that will help focus the efforts in a productive direction. She is now evangelizing two things: the Ruby workshops in San Francisco along with using workshops as a tool to bring more women into the community. One can only hope she succeeds with both.

Comments (22 posted)

SCALE 8x: Free software legal issues

The casual view of open source software is that the code always comes first: releases are made when the code is ready, new contributors prove their chops by the quality of their code, and so forth. But in reality the FLOSS ecosystem relies on a complex legal framework in order to run smoothly and to stand up to proprietary software competition: the various software licenses, contribution agreements, copyright and other "intellectual property" law. Every once in a while, a good status check on the legal dimension is healthy for the typical developer, and SCALE 8x offered just that in a series of talks.

Red Hat's licensing and patent attorney Richard Fontana spoke about improving the intra-community open source legal system, Bradley Kuhn of the Software Freedom Conservancy and Software Freedom Law Center (SFLC) spoke about the nuts-and-bolts of bringing GPL violators into compliance, and SFLC counsel Karen Sandler presented a primer on the often misunderstood realm of trademark law.

Brave New World

Fontana's talk "Improving the Open Source Legal System" began by exploring how the real-world practices of the open source software community diverge from the legal realities on which the community depends. He then questioned what the differences reveal about the structure of the community, and suggested steps that major players like Linux distributions and large software projects could take to shore up some of the common misunderstandings and loopholes.

The conventional view of the software licenses that define FLOSS is that they are exotic variants of the licenses that govern the proprietary software market, Fontana said. They impose restrictions, albeit strange ones, and although there are peculiarities, similar peculiarities are found in contracts in the proprietary world, too. Ultimately, as in the proprietary world, participants comply with the licenses to minimize their own risk (in particular, the risk of litigation).

But in actuality, he continued, the FLOSS community acts according to a very different set of rules that are unique to the community. For example, the territoriality of licenses is almost universally ignored: developers act as if there is one, worldwide interpretation of the GPL, which is simply not true. The governing law of different countries can impose different restrictions, such as what constitutes software "distribution" (an example that the Free Software Foundation worked hard to correct for GPLv3 by using different language, such as "convey"). Proprietary companies take full advantage of the differences in local law, but virtually no one in most open source projects knows or cares what the governing law is in their case.

Similarly, there appears to be a set of widely-accepted functional rules for interpreting licenses that has arisen in practice outside of copyright law itself. For example, Fontana said, it is accepted universally that one can add BSD-licensed code to a GPL-licensed project, but in many jurisdictions the law states that a license (in this example, the BSD license) must explicitly address sublicensing or such sublicensing is not allowed.

Rather than strictly conforming to the legal system, Fontana continued, FLOSS functions on its own set of customs. They seem to be rational, but there is no formal description of them (which makes educating newcomers a problem), there are no institutions to handle dispute resolution, and some of the rules may not reflect real consensus. In the long term, this poses a problem for the legitimacy and rationality of FLOSS, he concluded. If we believe in free software ideals, we should strive to make FLOSS law meaningful and rational.

Fontana proposed several steps that vendors, projects, and distributions could take to better rationalize the system. These players should discuss and hopefully come to broad agreement on the boundaries between free and non-free behavior — acts such as nominally free projects shipping non-free code, putting portions of their code under non-free licenses, or applying anti-free interpretations to the licenses. They should also address murky "outbound licensing" issues such as how GPL and non-GPL code can coexist when shipped by the same project, and "inbound licensing" issues such as accepting code contributions without explicit copyrights and licenses attached.

The actual steps that Fontana recommends projects, distributions, and license stewards take come down to documentation and policing. Projects should publicly document their interpretations of licenses and definitions (something that some, like Debian and Fedora, already do). Distributions should document policies for code contributions and carefully police the licenses of the code they include. It is legally acceptable for the FLOSS community to have its own set of governing customs and traditions, but by and large, those customs are not yet documented and assembled — and they should be, for the long-term health of open source.

Lawyers, Code, and Money

Kuhn's talk "Demystifying GPL Enforcement" illuminated one of those traditions: what actually happens when a company is accused of violating the GPL by not the providing source code to a GPL-licensed upstream project (such as the kernel or the BusyBox utility) incorporated in its product. Kuhn works in GPL enforcement both for the SFLC, and as president of the Software Freedom Conservancy, the nonprofit group legally authorized by BusyBox (among other projects) to act on its behalf in enforcement.

Kuhn outlined best practices for doing compliance-friendly development, explained the different compliance options and the pros and cons of each, an outlined what SFLC does when it finds a GPL violation.

For the clueful, he said, avoiding violations in simple — many companies just don't take the steps. Violating companies, for example, never use version control, much less pull in GPL code from upstream as a "vendor branch." They also tend not to tag their releases, document or version their build process, or other common practices in free software projects. The result is that when someone makes a request for source code, it is impossible for the company to comply.

On top of that, he said, the companies he encounters in enforcement actions always make compliance more difficult for themselves by choosing the most arduous source code distribution options. The GPL allows several choices: include the source alongside the binary, make an offer to send the source code to anyone who requests it, and (in version 3), make it available through a peer-to-peer system.

By far the simplest option is including the source alongside the binary, said Kuhn, because the company's obligation ends immediately. In contrast, the offer to send source code upon request must be honored for three years after the last ship date of the project, applies to anyone (not just customers) and is considerably more logistically arduous. But most violators choose the "offer" option, he said, because they want to gamble that no one will actually request the code. They should assume otherwise, he said, since even if no one else ever requests the source code, Kuhn himself will.

That request is how an enforcement action begins; if the company does not comply, the SFLC sends a formal letter directed to the legal counsel or CEO, and attempts to open up active discussions on how to bring the vendor into compliance. Most of the time, the channel of communication is opened. The SFLC makes a series of standard requests, and works with the company to come into compliance on all FLOSS-copyrighted software incorporated in their products. The requests include putting the proper processes into place (including not just the development processes mentioned above, but keeping appropriate records and appointing someone in the company to be in charge of GPL compliance), notifying past recipients of the violating product that source code is now available, and for a financial settlement.

The settlement money is at times controversial, but Kuhn explained that it has several purposes. First, if there was no penalty to GPL violation other than coming into compliance, no one would proactively comply. Second, given that there must be a deterrent, the SFLC feels that GPL violators should bear the cost of defending GPL-licensed software projects — not companies who uphold the GPL, and not individual free software developers. SFLC is a nonprofit, he added, and does not get rich from settlement money. In fact, its status as a nonprofit entity enforces a degree of transparency on the entire enforcement process, with records on file with the IRS.

Only in rare cases does a GPL enforcement action result in a lawsuit, Kuhn said. It has happened in the past, but only after a complete breakdown in communication, and after considerable effort to bring the company into compliance. Kuhn prefers to to think of every GPL violator as a potential new contributor to the FLOSS community, and tells himself that every time he picks up the phone to make a request for source code.

GPL enforcement clarifies that there is one community, with one set of rules — not one set for those who choose to participate, and one for those who choose to remain ignorant. Enforcement itself shows that the community's rules are meaningful, he said, and doing it through a nonprofit group like the SFLC takes the burden off of the individual developers, who don't have the time to pursue violations themselves.

On your mark, get set...

Sandler's talk "What You Need to Know About Trademarks" addressed the legal concept of trademark, the understanding of which (like copyright) is vital to the health of free software projects. A trademark lets a small project protect and defend its identity even against well-funded competitors, but it is a very different animal than a copyright, which forms the foundation of FLOSS software licenses.

Copyright is granted automatically when a work (including software) is created. In contrast, a trademark is created automatically when it is used. The mark, whether a logo or a name, does not need to be registered; instead it is earned and strengthened by its usage. The more one uses a trademark, the stronger it is when challenged in court. Trademarks are also not subject to expiration terms like copyrights; as long as they are continually used, they do not ever expire and enter the public domain.

The legal test for trademark violation is in "the eye of the beholder" — almost literally. The test, Sandler said, is whether or not there is an identity associated with the mark in the public eye. In other words, when a person sees the mark, do they associate it with a particular product. Trademarks are limited by political geography, with different laws in different countries, and are only applicable to the industry or field-of-use in which the trademark is used.

Trademark law does have parallels to more familiar copyright law concepts, though. Where copyright has the doctrine of "fair use" protecting citation, commentary, and parody, trademark has "nominative use" which protects the use of marks to refer factually to the actual trademarked product. In other words, stores can use trademarked names and logos to advertise that the products mentioned are for sale, without seeking permission.

Sandler also addressed two trademark uses of interest to FLOSS software projects: developing a trademark for a project, and responding to "nastygrams" from hostile trademark holders.

Choosing a good trademark involves picking a distinctive name or logo. Commonly-used terms associated with the product cannot be trademarked, and choosing a good mark can be difficult. Sandler recommends doing a trademark search; unlike patents there is no doctrine of "willful infringement" in trademark — trademark infringers are just ordered to stop using the mark. But projects should be careful about their trademarks; registering a trademark is not required, but if it is done, she recommends having the group apply for it collectively, not leaving it up to an individual. An individual holding the mark could leave or fork the project later, thus making it very difficult for the group to regain control. Projects should also create a trademark policy, stating acceptable uses, naming conventions, and merchandising policy — not doing so could create confusion later, ultimately diluting the mark.

Finally, Sandler addressed what to do if a trademark holder accuses a software project of infringing on its mark. The principle question to ask is if the accused usage is genuinely likely to create confusion in consumers. Are the marks similar? Are they in the same field-of-use? Do they give the overall impression of being related products? And, most importantly, does the accuser know of actual cases of consumer confusion? If the answer is no, then there is likely no real infringement. A project should begin by asking those questions, and only needs to worry or seek legal advice (including from SFLC) if the accuser continues.

I learned the law, and we all won

All three talks touched on one common problem: that free software developers are not lawyers, and often prefer not to dwell on potentially thorny legal issues. But the law should not be intimidating to FLOSS software projects; it protects them from abuse by well-heeled enemies, and although it is a different domain, it is certainly well within the grasp of anyone capable of writing device drivers, 3-D animation studios, or any of the other top-notch projects produced by the open source community.

Comments (1 posted)

SCALE 8x: Proprietary software companies and open source

You wouldn't flame a puppy, would you?

Mark Stone, deputy director of the new Microsoft-backed CodePlex Foundation, showed up at the Southern California Linux Expo (SCALE) with a laptop running Puppy Linux, complete with adorable desktop puppy logo. Stone's presentation, shown in the "Puppy HTML Viewer" application, set a new record for graphic simplicity, even by the standards of this year's SCALE, where any slide format other than the OpenOffice.org Impress default was rare.

While the CodePlex Foundation itself is new in 2009, Stone was at the event to make a familiar pitch: companies that do proprietary and in-house software development still need to be persuaded to act in their own best interests, and need help to decide to participate in open source development when they can derive benefit from it. Stone has been making the same point as an editor for O'Reilly and Associates, where he edited the essay collection Open Sources along with other titles, then later as director of the developer relations program for SourceForge. And, he argues, the point still needs to be made.

The CodePlex Foundation, which Stone called a "broker that can mediate," recently saw its first release of a non-Microsoft project, the MVC Contrib model-view-controller framework for the Microsoft ASP.NET platform. More releases, not all .NET related, are on the way, Stone said.

Any big company is likely to be a user of some open source software, he said, "but when you look at what of their own software they release as open source, some are doing better than others," Stone said.

The situation is better than it was in 1995, when almost all free software development happened off the corporate clock. "The trend is for corporate development and open source to overlap more and more." But, he said, the shift to paid development has been more a matter of open source developers getting paid to do it, and less about proprietary or in-house software developers being able to release their work. Open source developers are getting paid to work for companies, but what about taking corporate development organizations and getting them plugged into open source?

Understanding decision makers' motivations is vital. While most software developers view innovation as a good, often the people who make decisions at companies value predictability and "protecting the brand" over improving the product. "Innovation is risky and scary, and something to be avoided at all costs," he said. What goes into the product is a brand management decision.

Some businesses are friendly to customer innovation, and actively look for how people are misusing the product. Skateboarding started with proto-skaters modifying surfboards and scooters, and today, "extreme" sports vendors bring customer's modifications in-house and base products on them. Others are more conservative.

Knowledge above code?

Stone argues that full-bore participation has value that throwing code over the wall doesn't. "The mere act of releasing some code isn't that much. What we care about is not code sharing but knowledge sharing. The source code by itself doesn't actually transfer that much knowledge," he said. "If you want to understand the software you have to understand its caretakers."

Another difference is that companies intend to put more knowledge into formalized systems. In open source, "we're very comfortable with a tribal approach to knowledge," Stone said. Companies, on the other hand, want knowledge better nailed-down and formalized. "They want you as an individual to be replaceable." Differences may be more aspirational than real. Anyone who has tried to build a proprietary or recently-freed codebase for the first time will understand how much "tribal" knowledge is still there. "There are good practices on both sides," Stone said. The "replaceable" individual is impossible in open development, though. "Reputation travels with you as an open source developer," he said.

The process of how to do open source has gotten much easier, with the rise of easy-to-use project hosting sites such as the original SourceForge, Google Code, and GitHub, and what Stone called, "consolidation around a half-dozen or so key approaches to licensing." The hard part, though, is still the decision of whether or not to do open source in the first place. "For business decision-makers, 'why would we release something as open source?' is a hard question."

A common example of a good case for participating is a company that finds itself carrying a substantial "patch load" of local modifications to open source software. For example, Stone worked on a project that modified MediaWiki to add role-based access control support, not part of the upstream project at the time. Do you just carry the patch load, and reapply your modifications when getting a new upstream version, or attempt to participate in the process by offering changes to upstream, or gathering other users and forking the project? Even thinking about the question is outside some users' vision. "That open source decision is a possibility you need to get business decision-makers to think about."

If your worst problem is differences in development practices, he said, "Congratulations, you're 90% of the way there. Good software development looks very much the same," whether it's open or proprietary. "Don't assume there are differences that aren't really there," he said. In addition, corporate decision makers need to learn to disbelieve myths, such as the myth that open source can't do software testing.

What's missing?

Companies expect a legal entity on the other end of a contractual relationship. For example, Microsoft receives automatically generated crash dumps from software running on its Windows platform. But user data is confidential, and Microsoft won't share customer data without an NDA. Someone needs to enter into one in order to see the crash dumps. There are many existing umbrella organizations, but, Stone said, "We exist because none of them is meeting all the needs." Microsoft itself has done some open source releases but the foundation "will make it easier to participate."

The foundation is not tied to Microsoft hosting infrastructure. The new MVC Contrib project has a project profile on codeplex.com but keeps its source code hosted at GitHUb. (Codeplex.com documentation only lists revision control support for Mercurial, Subversion, and Microsoft Team Foundation Server).

For companies to use the CodePlex Foundation is like "not reinventing the wheel" in software, Stone said. "There are legal processes that you want to re-use and leverage as well." With a substantial staff and million-dollar budget, the new foundation is prepared to be flexible helping companies with the legal paperwork. The Apache Software Foundation has one contributor agreement, and one license, but CodePlex can customize these things. "What do you need in terms of contributor agreement and license?" Stone asked. More news will be coming at next month's Open Source Business Conference in San Francisco, Stone said.

Previous commenters have reacted to the prospect of a wholesale dislocation of the software business with something less than panic. Richard Stallman famously pronounced, "Writing non-free software is not an ethically legitimate activity, so if people who do this run into trouble, that's good! All businesses based on non-free software ought to fail, and the sooner the better." Paul Graham later wrote, "When I say business can learn from open source, I don't mean any specific business can. I mean business can learn about new conditions the same way a gene pool does. I'm not claiming companies can get smarter, just that dumb ones will die." Stone and the CodePlex Foundation are offering an alternative that doesn't involve an office chair auction and a massive dump of perfectly good business cards into the recycling bin.

Comments (1 posted)

LWN editor Forrest Cook moves on

I would like to announce my departure from the day-to-day operations at LWN.net. There are a number of factors behind this move. My leaving LWN will reduce the site's expenses in these difficult economic times, this move will allow the company to operate with greater economic flexibility.

After nearly ten years of dealing with weekly deadlines, processing countless software release announcements and performing many other behind-the-scenes tasks, your editor is ready for a change of direction. I plan on dedicating more attention to my Linux-powered CirKits.com mail-order solar power electronics kit company, an early off-shoot of the LWN parent company, Eklektix.

Working for LWN has been a great journey since writing my first feature article. Linux has grown from a small project into a real force in the operating system landscape. Linus's quest for world domination turned out to be more than just joking around. One can derive a lot of satisfaction from knowing that one's contributions, however small, may have helped to push this mighty project forward.

Never one to have any idle time, your author is looking forward to dedicating more effort to his ongoing solar and wind powered off-grid mountain house project with its accompanying alternative energy experiments. He plans to spend more time with electronic circuit tinkering, combining microprocessors with vacuum tubes, and playing around with electric guitars.

Comments (22 posted)

Page editor: Jonathan Corbet

Security

Trustedbird: Additional email security for Thunderbird

A collaboration between the French military, BT, and Mozilla has resulted in a version of Thunderbird that has features more suited to military organizations. Trustedbird includes changes to Thunderbird to support additional encryption and message handling options, and some of that code has made its way into the Thunderbird 3 release. The reasons given for working with free software, rather than a proprietary alternative, make it clear that access to the source and the ability to make changes—hallmarks of free software—were key.

There are a number of message handling features that were added into the Trustedbird core, along with some additional features that were implemented as add-ons that will work with either Trustedbird or Thunderbird. The add-ons are for features that others might find useful outside of organizations that require the level of security that Trustedbird provides. Features like Multi-LDAP directory lookup for addresses, MDN Extended for deletion receipt handling, and Mail XForms that allows adding various headers through forms, are all available as add-ons. There is a list of these add-on on the documentation page.

The Thunderbird changes that make up Trustedbird are all based on various RFCs and may well end up in Thunderbird itself some day. Much of the work was based on RFC 2634 (Enhanced Security Services for S/MIME), which includes "triple wrapping", signed receipts, and security labels. In addition, Trustedbird implements Delivery Status Notification (DSN), based on RFC 3461, and SMTP Priorities based on a draft RFC.

For military organizations, it is important to be able to receive signed and encrypted messages that have not been surreptitiously forwarded. Standard encrypted email only signs the body of an email before encrypting it with the recipients public key. A malicious recipient can re-encrypt the mail with a different recipient's key and forward the mail (presumably with some header forgery). The new recipient may be confused into believing the mail was actually sent to them (as the signature will verify for the original sender).

Triple wrapping allows a recipient to detect that the mail has been forwarded by also signing the encrypted message. That additional signing can be done over some additional headers, along with the encrypted body, but that is not required. A proper message will be signed twice by the sender, while a surreptitiously forwarded one requires that the attacker re-encrypt the body (using the new recipient's public key), which will invalidate the outer signature.

Signed receipts are basically what they sound like. A receipt that a message has been received can be signed by the recipient. When a properly signed receipt is received by the sender, they can be sure that the recipient did receive the message—or at least that their Trustedbird client did.

Security labels are headers that can be added to the signed portion of a triple wrapped message and specify various kinds of information about the security policy that applies to the message. Standard labels like "classified" or "top secret" can be applied, and then be enforced based on the recipient's access level. The labels themselves can be customized in an XML file, but it is unclear from the documentation how exactly the security policies are specified and propagated.

The DSN feature has already been incorporated into Thunderbird 3. It allows clients to ask the Mail Transfer Agent (MTA, e.g. Sendmail or Postfix) for a notification on the delivery status of an email. Three kinds of notifications can be requested: success, failure, or delay in delivering the email.

SMTP Priority allows for five levels of priority (NONE, ROUTINE, PRIORITY, IMMEDIATE, and FLASH) to be sent to an MTA in the envelope part of the SMTP conversation. For additional complexity, different priorities can be given for each recipient. MTAs must be changed to support priorities so Trustedbird provides a priority email gateway that works with Postfix using Qpsmtpd.

While most of these are features that may be of little interest to many, it is always nice to see governments taking advantage of the benefits of free software. In addition, some of the features—triple wrapping in particular—may well be of interest to those who regularly use email encryption. The fact that the French military is working with the Thunderbird project to get its code upstream is also rather novel for government-sponsored projects.

It seems likely that Trustedbird will find its way into more agencies and organizations with a need for a higher security level in their email handling; the fact that it's free software will likely save the taxpayers in those places some money—always a good thing. It also shows that free software ideas and ideals have rather wide applicability. It is not just monetary savings; there is something rather comforting in knowing what's in the code that is being used for security purposes.

Comments (4 posted)

Brief items

Chuck Norris Botnet Karate-chops Routers Hard (PC World)

PC World reports on a botnet based on poorly-secured Linux routers. "Right now Chuck Norris-infected machines can be used to attack other systems on the Internet, in what are known as distributed denial of service attacks. The botnet can launch a password-guessing dictionary attack on another computer, and it can also change the DNS (Domain Name System) settings in the router. With this attack, victims on the router's network who think they are connecting to Facebook or Google end up redirected to a malicious Web page that then tries to install a virus on their computers."

Comments (3 posted)

Researchers: Rootkits Work Nicely On Smartphones, Thank You (Dark Reading)

Dark Reading looks at research into rootkits on smartphones. It shouldn't come as a big surprise to those who pay attention to security issues, but it is a class of attacks that could be quite dangerous. "In one test, the researchers showed how a rootkit could turn on a phone's microphone without the owner knowing it happened. In such a case, an attacker would send an invisible text message to the infected phone, telling it to place a call and turn on the microphone, such as when the phone's owner is in a meeting and the attacker wants to eavesdrop."

Comments (1 posted)

New vulnerabilities

cronie: modification time changes

Package(s):	cronie	CVE #(s):	CVE-2010-0424
Created:	February 24, 2010	Updated:	March 22, 2012
Description:	The cronie tool suffers from a race condition which can allow a local user to modify the modification time of otherwise inaccessible files.
Alerts:	SuSE SUSE-SR:2010:007 2010-03-30 Fedora FEDORA-2010-4300 2010-03-12 Fedora FEDORA-2010-2751 2010-02-24 Red Hat RHSA-2012:0304-03 2012-02-21 Oracle ELSA-2012-0304 2012-03-07 Scientific Linux SL-vixi-20120321 2012-03-21

Comments (none posted)

ffmpeg: multiple vulnerabilities

Package(s):	ffmpeg	CVE #(s):	CVE-2009-4631 CVE-2009-4632 CVE-2009-4633 CVE-2009-4634 CVE-2009-4635 CVE-2009-4636 CVE-2009-4637 CVE-2009-4638 CVE-2009-4640
Created:	February 19, 2010	Updated:	July 18, 2011
Description:	From the Debian advisory: Several vulnerabilities have been discovered in ffmpeg, a multimedia player, server and encoder, which also provides a range of multimedia libraries used in applications like MPlayer: Various programming errors in container and codec implementations may lead to denial of service or the execution of arbitrary code if the user is tricked into opening a malformed media file or stream.
Alerts:	Mandriva MDVSA-2011:114 2011-07-18 Mandriva MDVSA-2011:112 2011-07-18 Mandriva MDVSA-2011:062 2011-04-01 Mandriva MDVSA-2011:061 2011-04-01 Mandriva MDVSA-2011:060 2011-04-01 Mandriva MDVSA-2011:059 2011-04-01 Mandriva MDVSA-2011:089 2011-05-16 Mandriva MDVSA-2011:088 2011-05-16 Ubuntu USN-931-2 2010-04-26 Ubuntu USN-931-1 2010-04-19 Debian DSA-2000-1 2010-02-18

Comments (none posted)

firefox: multiple vulnerabilities

Package(s):	firefox	CVE #(s):	CVE-2009-1571 CVE-2009-3988 CVE-2010-0159 CVE-2010-0160 CVE-2010-0162
Created:	February 18, 2010	Updated:	April 23, 2010
Description:	From the Red Hat alert: A use-after-free flaw was found in Firefox. Under low memory conditions, visiting a web page containing malicious content could result in Firefox executing arbitrary code with the privileges of the user running Firefox. (CVE-2009-1571) Several flaws were found in the processing of malformed web content. A web page containing malicious content could cause Firefox to crash or, potentially, execute arbitrary code with the privileges of the user running Firefox. (CVE-2010-0159, CVE-2010-0160) Two flaws were found in the way certain content was processed. An attacker could use these flaws to create a malicious web page that could bypass the same-origin policy, or possibly run untrusted JavaScript. (CVE-2009-3988, CVE-2010-0162)
Alerts:	Mandriva MDVSA-2010:071 2010-04-23 CentOS CESA-2010:0153 2010-03-26 CentOS CESA-2010:0154 2010-03-17 Red Hat RHSA-2010:0153-02 2010-03-17 Red Hat RHSA-2010:0154-02 2010-03-17 Pardus 2010-39 2010-03-09 Slackware SSA:2010-065-01 2010-03-08 SuSE SUSE-SA:2010:015 2010-03-04 Fedora FEDORA-2010-3230 2010-03-02 Fedora FEDORA-2010-3267 2010-03-02 Fedora FEDORA-2010-3230 2010-03-02 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Mandriva MDVSA-2010:042 2010-02-19 Ubuntu USN-895-1 2010-02-17 CentOS CESA-2010:0112 2010-02-18 Red Hat RHSA-2010:0112-01 2010-02-17 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1727 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1936 2010-02-19 Fedora FEDORA-2010-1932 2010-02-19 Debian DSA-1999-1 2010-02-18 Ubuntu USN-896-1 2010-02-17 CentOS CESA-2010:0113 2010-02-18 Red Hat RHSA-2010:0113-01 2010-02-17 Fedora FEDORA-2010-3267 2010-03-02 Mandriva MDVSA-2010:051 2010-03-01 CentOS CESA-2010:0112 2010-02-23 Fedora FEDORA-2010-1936 2010-02-19 Gentoo 201301-01 2013-01-07

Comments (none posted)

gnome-screensaver: unauthorized local access

Package(s):	gnome-screensaver	CVE #(s):	CVE-2009-4641
Created:	February 18, 2010	Updated:	February 24, 2010
Description:	From the Mandriva alert: gnome-screensaver 2.28.0 does not resume adherence to its activation settings after an inhibiting application becomes unavailable on the session bus, which allows physically proximate attackers to access an unattended workstation on which screen locking had been intended (CVE-2009-4641).
Alerts:	Mandriva MDVSA-2010:040 2010-02-17

Comments (none posted)

kernel: denial of service

Package(s):	linux-2.6	CVE #(s):	CVE-2010-0622
Created:	February 23, 2010	Updated:	October 8, 2010
Description:	From the Debian advisory: Jermome Marchand reported an issue in the futex subsystem that allows a local user to force an invalid futex state which results in a denial of service (oops).
Alerts:	Mandriva MDVSA-2010:188 2010-09-23 Mandriva MDVSA-2010:198 2010-10-07 Mandriva MDVSA-2010:088 2010-04-30 Pardus 2010-48 2010-04-09 Pardus 2010-64 2010-06-04 Red Hat RHSA-2010:0161-01 2010-03-23 SuSE SUSE-SA:2010:018 2010-03-22 Ubuntu USN-914-1 2010-03-17 CentOS CESA-2010:0504 2010-07-02 Red Hat RHSA-2010:0504-01 2010-07-01 Debian DSA-2012-1 2010-03-11 SuSE SUSE-SA:2010:016 2010-03-08 SuSE SUSE-SA:2010:014 2010-03-03 Debian DSA-2004-1 2010-02-27 Debian DSA-2003-1 2010-02-22

Comments (none posted)

konversation: remote crash

Package(s):	konversation	CVE #(s):
Created:	February 22, 2010	Updated:	February 24, 2010
Description:	Konversation through version 1.2.2 can be made to crash (with unknown consequences) by a message containing invalid Unicode characters.
Alerts:	Fedora FEDORA-2010-1935 2010-02-19 Fedora FEDORA-2010-1883 2010-02-19

Comments (none posted)

krb5: denial of service

Package(s):	krb5	CVE #(s):	CVE-2010-0283
Created:	February 19, 2010	Updated:	March 24, 2010
Description:	From the Red Hat bugzilla: A flaw was found in how the KDC processed invalid requests. An unauthenticated remote attacker could send an invalid request to a KDC process that would cause it to crash due to an assertion failure, resulting in a denial of service of the KDC. This flaw only affects MIT krb5 version 1.7 and later; earlier versions did not contain the vulnerable code.
Alerts:	Ubuntu USN-916-1 2010-03-23 SuSE SUSE-SR:2010:005 2010-02-23 Fedora FEDORA-2010-1722 2010-02-18 Gentoo 201201-13 2012-01-23

Comments (none posted)

moin: "major unspecified" vulnerabilities.

Package(s):	moin	CVE #(s):
Created:	February 22, 2010	Updated:	February 24, 2010
Description:	The MoinMoin wiki system suffers from a series of poorly described (but evidently serious) vulnerabilities. More information, such as it is, can be found in the 1.8.7 changelog and this Secunia advisory.
Alerts:	Fedora FEDORA-2010-1712 2010-02-19 Fedora FEDORA-2010-1743 2010-02-19

Comments (none posted)

netpbm: buffer overflow

Package(s):	netpbm	CVE #(s):	CVE-2009-4274
Created:	February 18, 2010	Updated:	April 30, 2010
Description:	From the Mandriva alert: Stack-based buffer overflow in converter/ppm/xpmtoppm.c in netpbm before 10.47.07 allows context-dependent attackers to cause a denial of service (application crash) or possibly execute arbitrary code via an XPM image file that contains a crafted header field associated with a large color index value (CVE-2009-4274).
Alerts:	Oracle ELSA-2011-1811 2011-12-13 Oracle ELSA-2011-1811 2011-12-13 Scientific Linux SL-netp-20111212 2011-12-12 CentOS CESA-2011:1811 2011-12-12 CentOS CESA-2011:1811 2011-12-12 Red Hat RHSA-2011:1811-01 2011-12-12 Ubuntu USN-934-1 2010-04-29 Debian DSA-2026-1 2010-04-02 SuSE SUSE-SR:2010:006 2010-03-15 Mandriva MDVSA-2010:039 2010-02-17

Comments (none posted)

pidgin: multiple vulnerabilities

Package(s):	pidgin	CVE #(s):	CVE-2010-0277 CVE-2010-0420 CVE-2010-0423
Created:	February 18, 2010	Updated:	November 15, 2010
Description:	From the Red Hat alert: An input sanitization flaw was found in the way Pidgin's MSN protocol implementation handled MSNSLP invitations. A remote attacker could send a specially-crafted INVITE request that would cause a denial of service (memory corruption and Pidgin crash). (CVE-2010-0277) A denial of service flaw was found in Finch's XMPP chat implementation, when using multi-user chat. If a Finch user in a multi-user chat session were to change their nickname to contain the HTML "br" element, it would cause Finch to crash. (CVE-2010-0420) Red Hat would like to thank Sadrul Habib Chowdhury of the Pidgin project for responsibly reporting the CVE-2010-0420 issue. A denial of service flaw was found in the way Pidgin processed emoticon images. A remote attacker could flood the victim with emoticon images during mutual communication, leading to excessive CPU use. (CVE-2010-0423)
Alerts:	Debian DSA-2038-3 2010-11-13 Debian DSA-2038-2 2010-05-17 Mandriva MDVSA-2010:085 2010-04-28 Debian DSA-2038-1 2010-04-18 SuSE SUSE-SR:2010:006 2010-03-15 Slackware SSA:2010-069-01 2010-03-11 Pardus 2010-34 2010-02-25 CentOS CESA-2010:0115 2010-02-20 Ubuntu USN-902-1 2010-02-22 Fedora FEDORA-2010-1383 2010-02-19 Fedora FEDORA-2010-1934 2010-02-19 Mandriva MDVSA-2010:041 2010-02-18 Red Hat RHSA-2010:0115-01 2010-02-18 CentOS CESA-2010:0115 2010-02-23 Fedora FEDORA-2010-1279 2010-02-19

Comments (none posted)

polipo: denial of service

Package(s):	polipo	CVE #(s):	CVE-2009-3305 CVE-2009-4413
Created:	February 19, 2010	Updated:	February 24, 2010
Description:	From the Debian advisory: Several denial of service vulnerabilities have been discovered in polipo, a small, caching web proxy. The Common Vulnerabilities and Exposures project identifies the following problems: A malicious remote sever could cause polipo to crash by sending an invalid Cache-Control header. CVE-2009-3305 A malicious client could cause polipo to crash by sending a large Content-Length value. CVE-2009-4413
Alerts:	Debian DSA-2002-1 2010-02-19

Comments (none posted)

squid: denial of service

Package(s):	squid	CVE #(s):	CVE-2010-0639
Created:	February 24, 2010	Updated:	January 19, 2012
Description:	Squid fails to properly handle malformed HTCP packets; as a result, a remote attacker can cause squid to crash with a null pointer dereference.
Alerts:	Gentoo 201110-24 2011-10-26 Mandriva MDVSA-2010:060 2010-03-10 Fedora FEDORA-2010-3064 2010-02-27 Ubuntu USN-904-1 2010-02-24 openSUSE openSUSE-SU-2012:0102-1 2012-01-19

Comments (none posted)

systemtap: denial of service

Package(s):	systemtap	CVE #(s):	CVE-2010-0411 CVE-2010-0412
Created:	February 19, 2010	Updated:	April 27, 2010
Description:	From the CVE entries: Multiple integer signedness errors in the (1) __get_argv and (2) __get_compat_argv functions in tapset/aux_syscalls.stp in SystemTap 1.1 allow local users to cause a denial of service (script crash, or system crash or hang) via a process with a large number of arguments, leading to a buffer overflow.
Alerts:	SuSE SUSE-SR:2010:010 2010-04-27 Fedora FEDORA-2010-1720 2010-02-18 Fedora FEDORA-2010-1373 2010-02-18 CentOS CESA-2010:0124 2010-03-02 CentOS CESA-2010:0125 2010-03-01 Red Hat RHSA-2010:0125-01 2010-03-01 Red Hat RHSA-2010:0124-01 2010-03-01

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The 2.6.33 kernel is out, released on February 24. Linus says:

Other interesting stuff in 2.6.33 includes dynamic tracing, the block I/O bandwidth controller, and the compressed cache mechanism.

See the KernelNewbies 2.6.33 page for more information on this release.

The current stable kernel is 2.6.32.9, released on February 23. There are 93 fixes in this update, many of which are security-related. See below for our detailed look at this release.

Comments (4 posted)

Quotes of the week

-- Zachary Amsden

-- Linus Torvalds

Comments (none posted)

Open by handle

Most Linux users never deal directly with file handles; indeed, most may not even know they exist. Of the rest, the bulk will have an experience limited to the cheery "stale file handle" message seen by NFS users at horribly inopportune times. In fact, a file handle is just a means by which a file can be uniquely identified within a filesystem. Handles are used in NFS, for example, to represent an open file in a way which allows the server to be almost entirely stateless. Handles are not normally used by, or even available to user-space applications.

Aneesh Kumar is trying to change that situation with a short patch series adding two new system calls:

    int name_to_handle(const char *name, struct file_handle *handle);
    int open_by_handle(struct file_handle *handle, int flags);

The first takes the given name and looks up the associated file handle, which is returned in the handle structure. That handle can then be passed to open_by_handle() to get an open file descriptor for the file. Only privileged users can call open_by_handle(); otherwise it could be possible for a malicious local user to bypass the normal permission checks on the directories in the path to a specific file.

Why would an application developer want to open a file in two steps instead of just calling open()? It comes down to the ability to write filesystem servers that run in user space. Such a server could use name_to_handle() to generate handles for files on the underlying filesystem; those handles are then passed to the filesystem's clients. At some future time, the client can pass the handle back to actually open the file. This type of feature is also already used with the XFS filesystem for backup and restore operations and with a hierarchical storage management system.

Discussion of these system calls has been minimal, thus far. It does seem that some work will be needed still to better describe what a file handle really is, and, in particular, what its expected lifetime will be. Without some clarity in that area, it will be hard to write applications which can make proper use of file handles.

Comments (6 posted)

Reserved network ports

It is not all that uncommon to have a network application which needs to be able to bind to a specific port. Often, such requirements result from a firewall configuration allowing incoming connections only to a specific port, but there can be other reasons as well. When running such an application, it can be unpleasant to discover that somebody else's long-running ssh connection happened to stumble onto the required port. It would be nice to be able to avoid this kind of conflict if at all possible.

This patch set from Octavian Purdila aims to make it possible. It adds a new sysctl knob (called ip_local_reserved_ports) under /proc/sys/net/ipv4. Should the system administrator write a comma-separated list of ports (or ranges of ports denoted by a hyphen) to this parameter, the networking layer will avoid those ports whenever it picks a port number for a new socket. Reserving ports in this manner will not interfere with any application which binds to those ports explicitly.

This patch has been through a surprising number of revisions; chances seem good that it will show up in the mainline once the 2.6.34 merge window opens.

Comments (16 posted)

Kernel development news

A Checkpoint/restart update

It has been exactly one year since LWN last checked up on the checkpoint/restart patch set. This code has just been reposted with a request for inclusion into the -mm tree, so it seems like an opportune time to restart our coverage of it. A lot of progress has been made on this front over the last year, but checkpoint/restart remains a difficult task which can probably never be implemented completely.

"Checkpointing" refers to the act of saving the state of a group of processes to a file, with the intent of restarting those processes at some future time. For many years, checkpointing has been used to save the state of long-running calculations to avoid losing work should the system fail. More recently, it has become a desired part of the virtualization toolkit, enabling the live migration of processes between physical hosts. The checkpoint/restart developers also see other potential advantages, such as the ability to quickly launch a set of processes on demand from a checkpoint image.

This patch set addresses checkpoint/restart in the containers context. In the context of full virtualization, checkpointing is relatively easy; the system just needs to save the entire memory image associated with the virtual machine and a bit of associated data. The "containers" model of virtualization tends to be messier in almost every way, and checkpointing is no exception. There is no memory image to be saved in one big chunk; instead, the kernel must track down every bit of state associated with the checkpointed processes and save it independently. When it works, it can be faster and more efficient than full virtual machine checkpointing; the checkpoint image will be much smaller. But getting it to work is a challenge. The complexity of this task can be seen in the current checkpoint/restart tree, which, despite being far from a complete solution of the problem, is a 27,000-line diff from 2.6.33-rc8.

Checkpointing

To checkpoint a group of processes, the following new system call is used:

    int checkpoint(pid_t pid, int fd, unsigned long flags, int logfd);

The pid parameter identifies the top-level process to be checkpointed; all children of that process will also be included in the checkpoint image, which will be written to the file indicated by fd. There is currently only one possible flag value, CHECKPOINT_SUBTREE, which turns off the normal requirement that an entire container be checkpointed as a whole. Checkpointing just a subtree is a bit riskier than checkpointing a full container because it is harder to ensure that all needed resources have been saved. The logfd parameter is file descriptor open for writing; the kernel will write relevant logging information there. There are vast numbers of possible ways for a checkpoint to fail; the log file is intended to help users figure out what is happening when a checkpoint refuses to work. If logging is not desired, logfd can be -1.

The set of processes to be checkpointed should be frozen prior to the call to checkpoint(). One exception to that rule is a process running in checkpoint() itself; this exception allows processes to checkpoint themselves.

Internally, the checkpointing process is implemented as a two-phase operation:

• The kernel traverses the tree of processes and "collects" every object which is to be a part of the checkpoint image. Essentially, "collecting" means building a hash table with an entry for every process, every open file, every virtual memory area, every open socket, etc. which must be saved. Scanning the tree in this way helps the kernel to abort the checkpoint process early if something which cannot be checkpointed is encountered. Just as importantly, the collecting process also lets the system track objects which have multiple references and ensure that they are only written to the image file once.

• The second pass then iterates over the collected objects and causes each to be serialized and written to the image file.

Once this is done, the checkpoint is finished. The just-checkpointed processes can either go on with their business or be killed, depending on the reason for the checkpoint.

These two phases are reflected in the changes made to the lower levels of the system. For example, the none-too-svelte file_operations structure gains two new operations:

    int collect(struct ckpt_ctx *ctx, struct file *filp);
    int checkpoint(struct ckpt_ctx *ctx, struct file *filp);

The collect() operation should identify every object which must be saved, eventually passing each to ckpt_obj_collect() (or one of several higher-level interfaces) for tracking. Later, a call to checkpoint() is made to request that the given filp be serialized for saving to the checkpoint image. Similar methods have been added to a number of other structure types, including vm_operations_struct and proto_ops.

The serialization process requires copying data from kernel data structures into a series of special structures intended to be written to the image file. So, for example, a file descriptor finds its way from struct fdtable into one of these:

    struct ckpt_hdr_file_desc {
	struct ckpt_hdr h;
	__s32 fd_objref;
	__s32 fd_descriptor;
	__u32 fd_close_on_exec;
    } __attribute__((aligned(8)));

Doing this copy requires a 75-line function which grabs the requisite information and very carefully checks that everything can be checkpointed successfully. In this case, the presence of locks on the file or an owner (to be notified with SIGIO) will cause the checkpoint to fail. In the absence of such roadblocks, the completed structure is handed to the checkpoint code for saving to the image file.

This serialization process is one of the scarier parts of the whole checkpoint/restart concept. Any changes to struct fdtable will almost certainly break this serialization, quite possibly in ways which will not be detected until some user runs into a problem. Even if a VFS developer cared about checkpointing, they might not think to look at the code in checkpoint/files.c to see if anything might require changing. Similar dependencies are created for every other kernel data structure which must be checkpointed. The whole setup looks like it could be a little fragile; keeping it working is almost certain to require significant ongoing maintenance.

Restarting

On the restart side, the application performing the restart is first expected to create a set of processes to be animated with the checkpointed information. That creation will be done with the much-reviewed "extended clone()" system call, which, in this iteration, looks like:

    int eclone(u32 flags_low, struct clone_args *cargs, int cargs_size,
	       pid_t *pids);

With eclone(), the processes can be created with specific pids and with an extended set of flags.

Once the process hierarchy exists, the restart() system call can be used:

    int restart(pid_t pid, int fd, unsigned long flags, int logfd);

The checkpoint image found at fd will be restored into the process hierarchy starting at pid. Once again, logfd can be used to gain information on how the process went. There are a number of flags defined: RESTART_TASKSELF (a single task is being restarted on top of the process calling restart()), RESTART_FROZEN (causes the restarted processes to be left frozen at the end), RESTART_GHOST (appears to be a debugging feature), RESTART_KEEP_LSM (restore security labels too), and RESTART_CONN_RESET (force the closing of open sockets). On a successful return from restart(), the process hierarchy should be ready to go.

Once again, restart requires support at the lower levels of the kernel. So our long-suffering file_operations structure gains another function:

    int restore(struct ckpt_ctx *, struct ckpt_hdr_file *);

This function (along with its analogs elsewhere in the kernel) is charged with reanimating the given object from the checkpoint file.

Security

It is not hard to imagine that these new system calls could have any of a number of security-related consequences, so it is surprising to see that, in the current implementation, both checkpoint() and restart() are unprivileged operations. This decision was made deliberately, with the idea of forcing the developers to think about security issues from the outset.

The biggest potential problem with checkpoint() is probably information disclosure. To avoid this problem, checkpoint() is only able to checkpoint processes which the caller would be able to call ptrace() on. So there should be no way for a hostile user to gain information from a checkpoint image which would not be available anyway.

The restart side is a little more frightening; it allows the caller to load vast amounts of potentially arbitrary data into kernel data structures. This risk is, one hopes, mitigated by causing all operations to be done in the context of the calling process. If the caller cannot open a file directly, that file cannot be opened via a corrupted checkpoint image either. Doing things this way will break certain use cases, such as checkpointing a setuid program which has since dropped its privileges, but there is probably no way to make that case work securely for unprivileged users.

For an added challenge, the checkpoint/restart developers have also implemented the checkpointing of security labels on objects. By default, these labels will not be used during the restart process, but the RESTART_KEEP_LSM flag can change that. Again, the labels are created in the context of the calling process, so the active security module should prevent the attachment of labels which compromise the security of the system.

Even with these measures in place, one still has to wonder about the security of the process as a whole. The kernel is populating a wide array of data structures from input which may be under the control of a hostile user; it is not hard to imagine that, somewhere in tens of thousands of lines of checkpoint/restart code, an important check has not been made. Perhaps as a result of this concern, the patch set adds a sysctl knob which can be set to disallow unprivileged checkpoint/restart operations.

Where things stand

According to the most recent patch posting:

So the patch set appears to be sufficiently functional to be minimally useful. There are, however, a lot of things which can stil prevent the creation of a successful checkpoint; they are summarized on this page. Problem areas include private filesystem mounts, network sockets in some states, open-but-unlinked files, use of any of the file event notification interfaces, open files on network or FUSE filesystems, use of netlink, ptrace(), asynchronous I/O, and more. There are patches in the works for some of these problems; others look hard.

As of this writing, there has been no response to the developers' request for inclusion in the -mm kernel. In the past, there have been concerns about how much work would be required to finish the job. Over the last year, much of that work is done, but checkpoint/restart looks like a job which is never truly finished. It's mostly a matter of whether what has been done so far appears to be good enough for real work, and whether the maintenance cost of this code is deemed to be worth paying.

Comments (10 posted)

2.6.32.9 Release notes

Stable kernel update announcements posted on LWN have a certain tendency to be followed by complaints about the amount of information which is made available. It seems that there is a desire for a description of the changes which is more accessible than the patches themselves, and for attention to be drawn to the security-relevant fixes. As an exercise in determining what kind of effort is being asked of the kernel maintainers, your editor decided to make a pass through the proposed 2.6.32.9 update and attempt to describe the impact of each of the changes - all 93 of them. The results can be found below.

Disclaimers: there is no way to review 93 patches in a finite time and fully understand each of them. So there are probably certainly errors in what follows. The simple truth of the matter is that it is very hard to say which fixes have security implications; a determined attacker can find a way to exploit some very obscure bugs.

Your editor would also like to discourage anybody from thinking that this will become a regular LWN feature. The amount of work required is considerable; it's not something we're able to commit to doing for every release.

That said, here's a look at what's in this update.

Security-related fixes

• #2: futex_lock_pi() key refcnt fix. It's possible to create dangling futex references, leading to a user-triggerable BUG_ON() oops. It's thus a denial of service vulnerability; it has been present since 2.6.28.

• #3: futex: Handle user space corruption gracefully. Malicious programs can cause a null pointer dereference or hijack somebody else's futex.

• #4: futex: Handle futex value corruption gracefully. User-space processes can cause warning floods, spamming the system logs.

• #5: Fix race in tty_fasync() properly. Possible (if unlikely) deadlock, and thus denial of service.

• #22: regulator: Fix display of null constraints for regulators. Fixes an information disclosure issue in the regulator code.

• #23: ALSA: hda-intel: Avoid divide by zero crash. Papers over a user-triggerable divide-by-zero crash; the real cause of the problem remains unknown.

• #26: cciss: Make cciss_seq_show handle holes in the h->drv[] array. Null pointer dereference in the cciss driver; probably not triggerable without privilege.

• #35: NFS: Fix an Oops when truncating a file. User-triggerable oops when truncating a file on an NFS filesystem.

• #36: NFS: Fix a umount race. Dangling pointer dereference on NFS filesystem unmount. Maybe triggerable in situations where users can cause mounts and unmounts to happen.

• #40: V4L/DVB: dvb-core: fix initialization of feeds list in demux filter. User-triggerable dereference of a corrupted pointer, with an oops being the best-case scenario.

• #47: netfilter: nf_conntrack: per netns nf_conntrack_cachep. Fixes a potential leak of information between network namespaces. Probably very hard to exploit in any useful way.

• #50: netfilter: nf_conntrack: fix hash resizing with namespaces. Changing the conntrack hash size in one namespace causes that size to be incorrect for all others, leading to unsightly kernel oops issues.

• #54: [S390] dasd: remove strings from s390dbf. Stale pointer dereference bugs in the S390 DASD driver.

• #57: dell-wmi, hp-wmi, msi-wmi: check wmi_get_event_data() return value. Fix a potential null pointer dereference on memory allocation failure.

• #75: ALSA: usb-audio - Avoid Oops after disconnect. Fixes a user-triggerable oops in the USB audio driver.

• #79: USB: usbfs: only copy the actual data received. Usbfs was copying more data than actually existed in some situations, leading to a potential information disclosure problem.

• #82: ACPI: Add NULL pointer check in acpi_bus_start. A null pointer dereference in the ACPI code.

• #92: dm log: userspace fix overhead_size calculations. A couple of structure-size miscalculations make both buffer overruns and information disclosure possible, though it's not at all clear that either is readily exploitable.

Other bug fixes

• #1: Fix potential crash with sys_move_pages. Fix an unreliable test which could cause a crash in the page migration code. [Update: as has been pointed out in the comments, this one is exploitable and should have been in the security list above.]

• #6: hwmon: (w83781d) Request I/O ports individually for probing. More robust access to hardware monitoring ports.

• #7: hwmon: (lm78) Request I/O ports individually for probing. More robust access to hardware monitoring ports.

• #8: hwmon: (adt7462) Wrong ADT7462_VOLT_COUNT. Fixes a bug which could cause one voltage measurement to be passed over.

• #9: ALSA: ctxfi - fix PTP address initialization. Fixes an alignment bug in the ctxfi sound driver.

• #10: drm/i915: disable hotplug detect before Ironlake CRT detect. Fixes a possible hang in the monitor detection code.

• #12: drm/i915: Disable SR when more than one pipe is enabled. Fixes a flicker-causing i915 bug.

• #13: drm/i915: Fix DDC on some systems by clearing BIOS GMBUS setup. Fixes a bug which can cause failure to detect some monitors.

• #15: drm/i915: Fix the incorrect DMI string for Samsung SX20S laptop. Incorrect identification information was returned to user space.

• #17: usb: r8a66597-hcd: Flush the D-cache for the pipe-in transfer buffers. Fixes a cache consistency problem.

• #18: i2c-tiny-usb: Fix on big-endian systems. An endianness bug in i2c-tiny-usb caused incorrect information to be returned to user space.

• #19: drm/i915: handle FBC and self-refresh better. Eliminates an i915 flicker problem.

• #20: drm/i915: Increase fb alignment to 64k. Fixes an obscure error in the i915 driver.

• #24: CPUFREQ: Fix use after free of struct powernow_k8_data. Fixes a use-after-free bug in the cpufreq code; does not appear to be user-triggerable.

• #25: freeze_bdev: dont deactivate successfully frozen MS_RDONLY sb. Fixes a boot-time crash in the block layer.

• #27: ioat: fix infinite timeout checking in ioat2_quiesce. Fixes a typo in the IOAT code.

• #29: fs/exec.c: restrict initial stack space expansion to rlimit. Fixes a bug which could cause process creation failures in the presence of tight stack limits.a

• #30: cifs: fix length calculation for converted unicode readdir names. Fixes a CIFS data consistency bug.

• #31: NFS: Fix a reference leak in nfs_wb_cancel_page(). Fixes a reference leak in the NFS cancellation code.

• #32: NFS: Try to commit unstable writes in nfs_release_page(). Looks like a fix for a potential data loss problem in the NFS code.

• #33: NFSv4: Dont allow posix locking against servers that dont support it. Be sure to notice if a server does not support POSIX locking.

• #34: NFSv4: Ensure that the NFSv4 locking can recover from stateid errors. Fix an NFSv4 locking problem with unknown effects.

• #37: NFS: Fix a bug in nfs_fscache_release_page(). Removes a spurious BUG_ON() call.

• #38: NFS: Fix the mapping of the NFSERR_SERVERFAULT error. Fix an incorrect error value returned to user space.

• #39: md: fix degraded calculation when starting a reshape. Some old code can cause the MD subsystem to be unclear on whether a given array is running in a degraded mode or not after a reshape.

• #42: kvmclock: count total_sleep_time when updating guest clock. Fix an error which could lead to incorrect wall clock time in KVM guests.

• #43: KVM: PIT: control word is write-only. Prevent attempts to read a write-only register.

• #44: tpm_infineon: fix suspend/resume handler for pnp_driver. Fixes a hang-on-suspend bug.

• #45: amd64_edac: Do not falsely trigger kerneloops. Remove a spurious warning in the amd64 EDAC code.

• #46: netfilter: nf_conntrack: fix memory corruption with multiple namespaces. Fixes a potential race condition which could lead to memory corruption. Requires the instantiation of a new namespace (and, thus, root privilege) to trigger.

• #48: netfilter: nf_conntrack: restrict runtime expect hashsize modifications. Don't allow the connection tracking expect_hashsize attribute to be modified, since the code isn't prepared to handle that.

• #49: netfilter: xtables: compat out of scope fix. Fixes a potential stack-based dangling pointer bug.

• #51: drm/i915: remove full registers dump debug. Removes an i915 debug option which could hang the machine.

• #52: drm/i915: add i915_lp_ring_sync helper. Code and performance improvement in the i915 driver.

• #53: drm/i915: Dont wait interruptible for possible plane buffer flush. The i915 DRM driver can corrupt the hardware state if a signal comes in at the wrong time. Could be seen as a denial of service problem, but that's a big stretch.

• #56: wmi: Free the allocated acpi objects through wmi_get_event_data. Fixes a memory leak in the WMI code.

• #58: /dev/mem: introduce size_inside_page(). Eliminates some duplicate code and fixes the alignment logic for /dev/kmem, which was described simply as "buggy." But who uses /dev/kmem anymore?

• #59: devmem: check vmalloc address on kmem read/write. A missing test for addresses in the vmalloc() space could cause an oops from the /dev/kmem code. Probably not triggerable by ordinary users, though, even on systems where /dev/kmem is enabled.

• #60: devmem: fix kmem write bug on memory holes. An attempt to write data to /dev/mem would get confused if a memory hole is hit, causing incorrect data to be written after the hole.

• #61: SCSI: mptfusion : mptscsih_abort return value should be SUCCESS instead of value 0. The mptfusion driver had an incorrect return value with unknown effects.

• #62: sh: Couple kernel and user write page perm bits for CONFIG_X2TLB. The SuperH architecture had a problem handling write faults for pages in the vmalloc() space, which could cause problems with drivers that map such pages into user space.

• #63: ALSA: hda - use WARN_ON_ONCE() for zero-division detection. Avoid spamming the log files if the hardware goes nuts.

• #64: dst: call cond_resched() in dst_gc_task(). The network destination cache code can process very long lists, leading to soft lockup warnings.

• #66: befs: fix leak. There is a memory leak in the BeFS mount code; one would not normally expect it to be user-triggerable.

• #67: rtc-fm3130: add missing braces. Missing braces in the rtc-fm3130 would cause spurious warnings to be emitted.

• #68: [libata] Call flush_dcache_page after PIO data transfers in libata-sff.c. Fix a cache coherency bug in the ATA code.

• #70: pktgen: Fix freezing problem. The packet generator could prevent the system from suspending or hibernating.

• #71: x86/amd-iommu: Fix IOMMU-API initialization for iommu=pt. Fix a boot-time initialization error in the IOMMU code.

• #72: x86/amd-iommu: Fix deassignment of a device from the pt_domain. Fix a KVM device assignment failure.

• #73: x86: Re-get cfg_new in case reuse/move irq_desc. Fix a bug in interrupt migration with unknown effect.

• #74: Staging: fix rtl8187se compilation errors with mac80211. Boring compilation problem fix.

• #76: serial: 8250: add serial transmitter fully empty test. Fixes a serial driver problem which could cause the loss of some transmitted data.

• #77: sysfs: sysfs_sd_setattr set iattrs unconditionally. An omitted initialization can cause sysfs attributes to have more restrictive permissions than desired.

• #78: class: Free the class private data in class_release. Fix a memory leak in the sysfs class code. Potentially user-exploitable if somebody were willing to dedicate a month of their life to repeatedly plugging and unplugging a device.

• #80: USB: usbfs: properly clean up the as structure on error paths. Fixes a memory leak in the usbfs error recovery paths.

• #83: ACPI: fix High cpu temperature with 2.6.32. Fixes behavior on a couple of laptops with problematic power management operation.

• #84: drm/radeon/kms: use udelay for short delays. Use of schedule_timeout() for short delays was slowing bootstrap considerably on some systems.

• #85: NFS: Too many GETATTR and ACCESS calls after direct I/O. Fixes a performance regression in the NFS code.

• #86: eCryptfs: Add getattr function. The eCryptfs filesystem would show incorrect file sizes.

• #87: b43: Fix throughput regression. Throughput on some BCM4311 devices is said to have dropped from 18Mb/s to 0.7Mb/s, which is a bit more of a penalty than some users wanted to pay.

• #88: ath9k: Fix sequence numbers for PAE frames. Fixes a protocol error in the ath9k driver.

• #89: mac80211: Fix probe request filtering in IBSS mode. The wireless code could reply to probe requests directed at a different SSID.

• #90: iwlwifi: Fix to set correct ht configuration. The iwlwifi driver was not configuring associations correctly, leading to dropped connections.

• #91: dm stripe: avoid divide by zero with invalid stripe count. Giving a bad stripe size to the device mapper code would cause a division by zero.

• #93: dm mpath: fix stall when requeueing io. Fixes a root-triggerable stall in the device mapper multipath code.

Enhancements

• #11: drm/i915: enable self-refresh on 965. Hardware feature enablement.

• #14: drm/i915: Add HP nx9020/SamsungSX20S to ACPI LID quirk list. Adds a quirk entry for buggy hardware.

• #16: drm/i915: Add MALATA PC-81005 to ACPI LID quirk list. Adds a quirk entry for more buggy hardware.

• #21: drm/i915: Update write_domains on active list after flush. Performance improvement in the i915 driver.

• #28: resource: add helpers for fetching rlimits. Adds helper functions to ensure that resource limit values are not fetched multiple times.

• #41: Export the symbol of getboottime and mmonotonic_to_bootbased. Adds a couple of symbol exports.

• #55: crypto: padlock-sha - Add import/export support. Improve interoperation with some HMAC code.

• #65: ALSA: hda - Improved MacBook (Pro) 5,1 / 5,2 support. Improves sound behavior on those systems.

• #69: ahci: add Acer G725 to broken suspend list. Note that Acer G725 laptops with old firmware have buggy suspend behavior.

• #81: rtl8187: Add new device ID. Recognize another device ID.

Conclusions

Out of 93 patches, 18 struck your editor as having clear security implications. Quite a few other patches fix crashes which could possibly be security problems; if they are not listed as such, it's because there was no immediately evident way that a user could trigger the problem. Doubtless people with more imagination will figure out ways to take advantage of some of these bugs.

What it comes down to is that the identification of security problems is often hard. In the kernel environment, almost any bug could potentially create some kind of vulnerability. So it is not surprising to see developers "silently fix" security bugs; they simply fix bugs without realizing the implications. It is also not surprising that some developers are reluctant to call attention to security-related fixes. The list above almost certainly includes "security fixes" for bugs that nobody can exploit while classifying true vulnerabilities as mere bug fixes. Any list of security-relevant patches is sure to be an incomplete and partially deceptive thing.

That said, it may well be that fixes which are truly known to have security implications should be marked as such. Attackers will make the effort to figure that out anyway; it's not clear that making life harder for everybody else has any benefits. Still, those who would complain about how the stable tree is managed would do well to remember that, a few years ago, we had no such tree. It came into being because people stepped up to do the work of maintaining it. There can be no doubt that a better job could be done here (as is the case almost everywhere else too); its just a matter of somebody finding the time and the energy to do it.

Comments (95 posted)

Huge pages part 2: Interfaces

In an ideal world, the operating system would automatically use huge pages where appropriate, but there are a few problems. First, the operating system must decide when it is appropriate to promote base pages to huge pages requiring the maintenance of metadata which, itself, has an associated cost which may or may not be offset by the use of huge pages. Second, there can be architectural limitations that prevent a different page size being used within an address range once one page has been inserted. Finally, differences in TLB structure make predicting how many huge pages can be used and still be of benefit problematic.

For these reasons, with one notable exception, operating systems provide a more explicit interface for huge pages to user space. It is up to application developers and system administrators to decide how they best be used. This chapter will cover the interfaces that exist for Linux.

1 Shared Memory

One of the oldest interfaces backs shared memory segments created by shmget() with huge pages. Today, it is commonly used due to its simplicity and the length of time it has been supported. Huge pages are requested by specifying the SHM_HUGETLB flag and ensuring the size is huge-page-aligned. Examples of how to do this are included in the kernel source tree under Documentation/vm/hugetlbpage.txt.

A limitation of this interface is that only the default huge page size (as indicated by the Hugepagesize field in /proc/meminfo) will be used. If one wanted to use 16GB pages as supported on later versions of POWER for example, the default_hugepagesz= field must be used on the kernel command line as documented in Documentation/kernel-parameters.txt in the kernel source.

The maximum amount of memory that can be committed to shared-memory huge pages is controlled by the shmmax sysctl parameter. This parameter will be discussed further in the next installment.

2 HugeTLBFS

For the creation of shared or private mappings, Linux provides a RAM-based filesystem called "hugetlbfs." Every file on this filesystem is backed by huge pages and is accessed with mmap() or read(). If no options are specified at mount time, the default huge page size is used to back the files. To use a different page size, specify pagesize=.

    $ mount -t hugetlbfs none /mnt/hugetlbfs -o pagesize=64K

There are two ways to control the amount of memory which can be consumed by huge pages attached to a mount point. The size= mount parameter specifies (in bytes; the "K," "M," and "G" suffixes are understood) the maximum amount of memory which will be used by this mount. The size is rounded down to the nearest huge page size. It can also be specified as a percentage of the static huge page pool, though this option appears to be rarely used. The nr_inodes= parameter limits the number of files that can exist on the mount point which, in effect, limits the number of possible mappings. In combination, these options can be used to divvy up the available huge pages to groups or users in a shared system.

Hugetlbfs is a bare interface to the huge page capabilities of the underlying hardware; taking advantage of it requires application awareness or library support. Libhugetlbfs makes heavy use of this interface when automatically backing regions with huge pages.

For an application wishing to use the interface, the initial step is to discover the mount point by either reading /proc/mounts or using libhugetlbfs. Finding the mount point manually is relatively straightforward and already well known for debugfs but, for completeness, a very simple example program is shown below:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/param.h>

char *find_hugetlbfs(char *fsmount, int len)
{
	char format[256];
	char fstype[256];
	char *ret = NULL;
	FILE *fd;

	snprintf(format, 255, "%%*s %%%ds %%255s %%*s %%*d %%*d", len);

	fd = fopen("/proc/mounts", "r");
	if (!fd) {
		perror("fopen");
		return NULL;
	}

	while (fscanf(fd, format, fsmount, fstype) == 2) {
		if (!strcmp(fstype, "hugetlbfs")) {
			ret = fsmount;
			break;
		}
	}

	fclose(fd);
	return ret;
}

int main() {
	char buffer[PATH_MAX+1];
	printf("hugetlbfs mounted at %s\n", find_hugetlbfs(buffer, PATH_MAX));
	return 0;
}

When there are multiple mount points (to make different page sizes available), it gets more complicated; libhugetlbfs also provides a number of functions to help with these mount points. hugetlbfs_find_path() returns a mount point similar to the example program above, while hugetlbfs_find_path_for_size() will return a mount point for a specific huge page size. If the developer wishes to test a particular path to see if it hugetlbfs or not, use hugetlbfs_test_path().

3 Anonymous mmap()

As of kernel 2.6.32, support is available that allows anonymous mappings to be created backed by huge pages with mmap() by specifying the flags MAP_ANONYMOUS|MAP_HUGETLB. These mappings can be private or shared. It is somewhat of an oversight that the amount of memory that can be pinned for anonymous mmap() is limited only by huge page availability. This potential problem may be addressed in future kernel releases.

4 libhugetlbfs Allocation APIs

It is recognised that a number of applications want to simply get a buffer backed by huge pages. To facilitate this, libhugetlbfs provides two APIs since release 2.0, get_hugepage_region() and get_huge_pages() with corresponding free functions called free_hugepage_region() and free_huge_pages(). These are all provided with manual pages distributed with the libhugetlbfs package.

get_huge_pages() is intended for use with the development of custom allocators and not as a drop-in replacement for malloc(). It is required that the size parameter to this API be hugepage-aligned which can be discovered with the function gethugepagesize().

If an application wants to allocate a number of very large buffers but is not concerned with alignment or some wastage, it should use get_hugepage_region(). The calling convention to this function is much more relaxed and will optionally fallback to using small pages if necessary.

It is possible that applications need very tight control over how the mapping is placed in memory. If this is the case, libhugetlbfs provides hugetlbfs_unlinked_fd() and hugetlbfs_unlinked_fd_for_size() to create a file descriptor representing an unlinked file on a suitable hugetlbfs mount point. Using the file-descriptor, the application can mmap() with the appropriate parameters for accurate placement.

Converting existing applications and libraries to use the API where applicable should be straightforward, but basic examples of how to do it with the STREAM memory bandwidth benchmark suite are available [gorman09a].

5 Automatic Backing of Memory Regions

While applications can be modified to use any of the interfaces, it imposes a significant burden on the application developer. To make life easier, libhugetlbfs can back a number of memory region types automatically when it is either pre-linked or pre-loaded. This process is described in the HOWTO documentation and manual pages that come with libhugetlbfs.

Once loaded, libhugetlbfs's behaviour is determined by environment variables described in the libhugetlbfs.7 manual page. As manipulating environment variables is time-consuming and error-prone, a hugectl utility exists that does much of the configuring automatically and will output what steps it took if the --dry-run switch is specified.

To determine if huge pages are really being used, /proc can be examined, but libhugetlbfs will also warn if the verbosity is set sufficiently high and sufficient numbers of huge pages are not available. See below for an example of using a simple program that backs a 32MB segment with huge pages. Note how the first attempt to use huge pages failed and some configuration was required as no huge pages were previously configured on this system.

The manual pages are quite comprehensive so this section will only give a brief introduction as to how different sections of memory can be backed by huge pages without modification.

  $ ./hugetlbfs-shmget-test 
  shmid: 0x2130007
  shmaddr: 0xb5e37000
  Starting the writes: ................................
  Starting the Check...Done.

  $ hugectl --shm ./hugetlbfs-shmget-test
  libhugetlbfs: WARNING: While overriding shmget(33554432) to add
                         SHM_HUGETLB: Cannot allocate memory
  libhugetlbfs: WARNING: Using small pages for shmget despite
                         HUGETLB_SHM shmid: 0x2128007
  shmaddr: 0xb5d57000
  Starting the writes: ................................
  Starting the Check...Done.

  $ hugeadm --pool-pages-min 4M:32M
  $ hugectl --shm ./hugetlbfs-shmget-test 
  shmid: 0x2158007
  shmaddr: 0xb5c00000
  Starting the writes: ................................
  Starting the Check...Done.

5.1 Shared Memory

When libhugetlbfs is preloaded or linked and the environment variable HUGETLB_SHM is set to yes, libhugetlbfs will override all calls to shmget(). Alternatively, launch the application with hugectl $--$shm. On setup, all shmget() requests will become aligned to a hugepage boundary and backed with huge pages if possible. If the system configuration does not allow huge pages to be used, the original request is honoured.

5.2 Heap

Glibc defines a __morecore hook that is is called when the heap size needs to be increased; libhugetlbfs uses this hook to create regions of memory backed by huge pages. Similar to shared memory, base pages are used when huge pages are not available.

When libhugetlbfs is preloaded or linked and the environment variable HUGETLB_MORECORE set to yes, libhugetlbfs will configure the __morecore hook, causing malloc() requests will use huge pages. Alternatively, launch the application with hugectl --heap.

Unlike shared memory, the page size can also be specified if more than one page size is supported by the system. The first example below uses the default page size (e.g. 16M on Power5+) and the second example explicitly overrides a default, using 64K pages.

    $ hugectl --heap ./target-application
    $ hugectl --heap=64k ./target-application

If the application has already been linked with libhugetlbfs, it may be necessary to specify --no-preload when using --heap so that an attempt is not made to load the library twice.

By using the __morecore hook and setting the mallopt() option M_MMAP_MAX to zero, libhugetlbfs prevents glibc from making use of brk() to expand the heap. An application that calls brk() directly will be using base pages.

If a custom memory allocator is being used, it must support the __morecore hook to use huge pages. An alternative may be to provide a wrapper around malloc() that called the real underlying malloc() or get_hugepage_region() depending on the size of the buffer. A heavy solution would be to provide a fully-fledged implementation of malloc() with libhugetlbfs that uses huge pages where appropriate, but this is currently unavailable due to the lack of a demonstrable use case.

5.3 Text and Data

Backing text or data is more involved as the application should first be relinked to align the sections to a huge page boundary. This is accomplished by linking against libhugetlbfs and specifying -Wl,--hugetlbfs-align -- assuming the version of binutils installed is sufficiently recent. More information on relinking applications is described in the libhugetlbfs HOWTO. Once the application is relinked, as before control is with environment variables or with hugectl.

    $ hugectl --text --data --bss ./target-application

When backing text or data by text, the relevant sections are copied to files on the hugetlbfs filesystem and mapped with mmap(). The files are then unlinked so that the memory is freed on application exit. If the application is to be invoked multiple times, it is worth sharing that data by specifying the --share-text switch. The consequence is that the memory remains in use when the application exits and must be manually deleted.

If it is not possible to relink the application, it is possible to force the loading of segments backed by huge pages by setting the environment variable HUGETLB_FORCE_ELFMAP to yes. This is not the preferred option as the method is not guaranteed to work. Segments must be large enough to overlap with a huge page and on architectures with limitations on where segments can be placed, it can be particularly problematic.

5.4 Stack

Currently, the stack cannot be backed by huge pages. Support was implemented in the past but the vast majority of applications did not aggressively use the stack. In many distributions, there are ulimits on the size of the stack that are smaller than a huge page size. Upon investigation, only the bwaves test from the SPEC CPU 2006 benchmark benefited from stacks being backed by huge pages and only then when using a commercial compiler. When compiled with gcc, there was no benefit, hence support was dropped.

6 Summary

There are a small number of interfaces provided by Linux to access huge pages. While cumbersome to develop applications against, there is a programming API available with libhugetlbfs and it is possible to automatically back segments of memory with huge pages without application modification. In the next section, it will be discussed how the system should be tuned.

Comments (6 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 2.6.33 released . (February 24, 2010)

• Thomas Gleixner: 2.6.33-rc8-rt1 . (February 21, 2010)

• Thomas Gleixner: 2.6.33-rc8-rt2 . (February 24, 2010)

• Greg KH: Linux 2.6.32.9 . (February 23, 2010)

• Thomas Gleixner: 2.6.31.12-rt21 . (February 18, 2010)

Build system

• Denys Vlasenko: Rename kernel's magic sections for compatibility with -ffunction-sections -fdata-sections . (February 20, 2010)

Core kernel code

• Oren Laadan: Linux Checkpoint-Restart - v19 . (February 22, 2010)

• Fabio Checconi: sched: use EDF to throttle RT task groups v2 . (February 23, 2010)

Development tools

• Paul E. McKenney: [PATCH tip/core/rcu 0/21] v6 add lockdep-based diagnostics to rcu_dereference() . (February 23, 2010)

• Arnd Bergmann: __rcu annotations, first draft . (February 24, 2010)

Device drivers

• Rafael J. Wysocki: PCI run-time PM support (rev. 4) . (February 18, 2010)

• Christopher Heiny: input/touchscreen: Synaptics Touchscreen Driver . (February 18, 2010)

• Matthew Garrett: backlight: Add support for exposing backlight types . (February 18, 2010)

• atrigent@ccs.neu.edu: [PATCH] Add a new VT mode which is like VT_PROCESS but doesn't require a VT_RELDISP ioctl call . (February 19, 2010)

• Marc Zyngier: [RFC][PATCH] genirq: Introduce IRQF_ALLOW_NESTED flag for request_irq() . (February 21, 2010)

• Dimitris Michailidis: cxgb4: new driver submission . (February 22, 2010)

• Sakari Ailus: V4L2 file handles and event interface . (February 22, 2010)

• Pawel Osciak: Multi-plane video buffer support for V4L2 API and videobuf . (February 22, 2010)

• Rodolfo Giometti: (version 2) LinuxPPS new functionalities . (February 23, 2010)

• Bruno Prémont: picolcd: driver for PicoLCD HID device . (February 24, 2010)

Documentation

• Luis R. Rodriguez: Firmware versioning best practices II . (February 20, 2010)

Filesystems and block I/O

• Dmitry Monakhov: RFC: introduce extended inode owner identifier v4 . (February 22, 2010)

• Aneesh Kumar K.V: Generic name to handle and open by handle syscalls . (February 22, 2010)

• Manish Katiyar: Online Hierarchical Storage Manager (OHSM v1.2) . (February 23, 2010)

Memory management

• Mel Gorman: Memory Compaction v3 . (February 18, 2010)

• Andrea Righi: [PATCH 0/2] memcg: per cgroup dirty limit . (February 21, 2010)

• Johannes Weiner: mm: used-once mapped file page detection . (February 22, 2010)

• KAMEZAWA Hiroyuki: memcg: page fault oom improvement . (February 23, 2010)

• Wu Fengguang: 512K readahead size with thrashing safe readahead v2 . (February 24, 2010)

Networking

• Simon Horman: SCTP Trasport Loadbalancing Support . (February 18, 2010)

• Andreas Petlund: net: TCP thin-stream latency-improving modifications . (February 18, 2010)

• Octavian Purdila: net: reserve ports for applications using fixed port numbers . (February 22, 2010)

• jamal: xfrm by MARK . (February 23, 2010)

• sjur.brandeland@stericsson.com: net-caif: introducing CAIF protocol stack . (February 23, 2010)

Architecture-specific

• Masami Hiramatsu: kprobes: Kprobes jump optimization support . (February 19, 2010)

• Luca Barbieri: x86-32: improve atomic64_t functions (v2) . (February 22, 2010)

• Grant Likely: Add basic ARM device tree support . (February 20, 2010)

• Martin Schwidefsky: s390 patch queue for 2.6.34 . (February 24, 2010)

Security-related

• john.johansen@canonical.com: AppArmor security module . (February 19, 2010)

Virtualization and containers

• Jean-Marc Pigeon: RFC containerized syslog and iptable log dispatching (working stage) . (February 23, 2010)

Benchmarks and bugs

• Rafael J. Wysocki: 2.6.33-rc8-git6: Reported regressions from 2.6.32 . (February 21, 2010)

• Rafael J. Wysocki: 2.6.33-rc8-git6: Reported regressions 2.6.31 -> 2.6.32 . (February 21, 2010)

Miscellaneous

• Karel Zak: util-linux-ng v2.17.1 . (February 22, 2010)

Page editor: Jonathan Corbet

Distributions

News and Editorials

Proxmox VE 1.5: combining KVM and OpenVZ

Now that virtualization is widespread, it becomes more important to have an easy-to-use, turnkey solution. The big hypervisor vendors are all working on their own virtualization management solutions, but one can also find interesting solutions in independent open source projects. One of these projects is Proxmox VE (Virtual Environment). Its unique selling point is that it marries the KVM hypervisor with OpenVZ virtual containers.

Proxmox VE makes it really easy to deploy a virtualization server. Users can download the ISO, burn it to a CD, put it in their server, answer a few questions the installer asks, and start serving virtual machines right away. Just be aware that the installer doesn't allow customizing the partitioning setup; it claims the whole hard drive. The system runs headless and can be configured using a web interface. Proxmox VE requires a 64-bit CPU with Intel VT or AMD-V instruction set and at least 2 GB of RAM if the user plans to run more than a handful of virtual machines.

Proxmox VE 1.5 is a heavily tweaked Debian Lenny (5.0) x86_64 distribution that partitions the hard drive with LVM2, installs a web-based management interface, and is optimized for use as a virtualization environment. Proxmox uses Debian Lenny's main and contrib repositories, as well as the security.debian.org repository with security updates. It adds its own download.proxmox.com repository that supplies custom kernels and some management tools. Proxmox VE is GPLv2-licensed and the German company Proxmox Server Solutions offers commercial support.

KVM and OpenVZ

Users can launch two sorts of virtual machines. First, one can use KVM for full virtualization. This allows the user to run a lot of operating systems, such as Windows, Linux, the BSDs, etc., because the operating system in the virtual machine runs on virtualized hardware that looks like real hardware. Proxmox VE also supports KVM with paravirtualization for device drivers to improve I/O performance, e.g. with paravirtualized network drivers for Windows guests.

The other flavor consists of OpenVZ containers, a form of operating system-level virtualization. Applications within a container appear to be on a standalone system, because processes in different containers are completely isolated from each other. Although each container appears as a standalone operating system, in reality they share a single instance of the Linux kernel: the Proxmox KVM/OpenVZ kernel. This also means that OpenVZ containers are lightweight. Users can download so-called OpenVZ templates, which are essentially gzipped tar files that contain a pre-installed image of a Linux system.

The combination of KVM and OpenVZ in one kernel is a major feat. It took an explanation by Proxmox VE's project lead Martin Maurer for your author to understand the challenges. Proxmox publishes three kernel branches, each one with minor feature differences. By default, Proxmox VE uses a 2.6.18 kernel, based on the latest Red Hat Enterprise Linux 5 kernel with the stable OpenVZ patches (which are also based on the 2.6.18 kernel) and backported KVM patches. The Proxmox developers added some minor driver updates and did a lot of integration and testing to combine both virtualization technologies in one kernel.

The second kernel version that Proxmox maintains is 2.6.24, based on the Ubuntu 8.04 (LTS) kernel but with driver updates and KVM patches. This kernel only exists for historical purposes, because Proxmox VE 0.9 beta until 1.4 used 2.6.24. However, the OpenVZ part is no longer maintained by the OpenVZ project, so while this kernel has KVM and OpenVZ support, all OpenVZ users should use the 2.6.18 kernel in Proxmox VE now. Last but not least, Proxmox VE also maintains a 2.6.32 kernel, based on the Ubuntu 10.04 (LTS) kernel but with some additional driver updates. It lacks OpenVZ support (because the stable OpenVZ version doesn't support this kernel), but it is interesting for users that want to have the latest KVM features, such as KSM (Kernel Samepage Merging).

Hands-on virtualization

Administering Proxmox is done by using the web interface, which is also lightweight. It doesn't use the Java-based Tomcat like VMware Server does, but just Apache with HTML pages that use the Prototype JavaScript library and the SOAP::Lite Perl library. The user can manage virtual machines, migrate them to other hosts, backup the system, monitor all running virtual machines, and do a slew of other management tasks.

Before starting a virtual machine, the user has to upload the ISO file or OpenVZ template. In the web interface, this can be done by clicking on "Virtual Machines - Create", selecting the image from local storage or from the host machine's CD-ROM drive, and then filling in the required information. The other solution is to scp the image over to /var/lib/vz/template/cache (for an OpenVZ template) or /var/lib/vz/template/iso (for an ISO image to start a KVM machine). But the user can also download pre-defined OpenVZ templates from within the web interface.

If the user wants a graphical window for a virtual machine, this can be done too: the Proxmox VE web interface has an embedded Java applet which launches a VNC console window on the client machine. Of course users are free to use their own favorite VNC client.

Proxmox VE has an integrated backup feature: VZDump, a utility that makes consistent backups of running virtual machines, be it OpenVZ containers or KVM guests. To make this possible for an online guest without downtime, VZDump uses the LVM2 storage to create a snapshot of the guest's virtual disk. The user can configure backups from within the web interface by defining the location of the remote storage and scheduling the cron job for the backup.

Users can also create a Proxmox VE Cluster that consists of one master node and one or more slave nodes, all running Proxmox VE. Users can then manage the virtual machines on all their Proxmox VE servers from one web interface, but they can also migrate their virtual machines between hosts, even live (that is, without recognizable downtime). Live migration is a standard feature of both KVM and OpenVZ, and both are supported in Proxmox.

OpenVZ containers have to be stored on local storage, but for KVM guests Proxmox VE adds a flexible storage model. From within the web interface, users can add iSCSI targets, NFS shares, LVM groups on shared storage, and locally mounted Samba shares. Thanks to the Debian base it also supports ATA over Ethernet (AoE) and replicated DRBD devices.

Ready-made appliances

The Proxmox developers have released several virtual appliances, which are ready-made OpenVZ templates that can be downloaded directly from within the Proxmox web interface. There are operating system appliances with a standard Debian 4.0, 5.0 or 6.0 installation, an Ubuntu 8.04 installation, a CentOS 4 or 5 installation or a Fedora 9 installation, but there are also so-called application appliances. These are tailored to a specific task, such as the Proxmox Mail Gateway, Acquia Drupal, SugarCRM, Zenoss, MediaWiki, Joomla!, WordPress, Zimbra, and so on. Proxmox has also created a script to automate the creation of custom OpenVZ appliances based on Debian, the Debian Appliance Builder.

Roadmap

Given that Proxmox VE has only been public for less than two years (0.9 was released in April 2008), the project has already an impressive feature list. Additionally, the developers have plans for much more: the roadmap for the 2.x releases mentions high-availability for KVM guests, advanced user management, more pre-built virtual appliance downloads, resource monitoring and a firewall. The future also promises a more pleasant marriage of KVM and OpenVZ, Martin explains:

All in all, Proxmox VE is a nicely integrated solution to run and manage virtual machines: what sets it apart from many other solutions is not only that it runs both KVM and OpenVZ, but also that it's not just a web interface or just a kernel that you install on an existing Linux installation. Instead it's a whole collection of components that gives the user a running system in a few minutes. So even for users that don't need both KVM and OpenVZ, Proxmox should be on their shortlist of virtualization platforms to try.

Comments (8 posted)

New Releases

Debian Installer 6.0 Alpha1 release

The Debian Installer Team has announced the Debian Installer 6.0 Alpha1. "As most people will have noticed, this release has taken more time than usual. This was for various reasons that go from technical (major changes in the installer itself and other components that affect us) to lack of manpower to manage all the work required quickly. We really need more people to help us and contribute; please contact us if you're interested in helping." Click below for a summary of new features and improvements.

Full Story (comments: none)

Number Two Always Tries Harder: openSUSE Milestone 2 released

The second milestone release of openSUSE 11.3 is available for testing. "Milestone 2 is part of the milestones where we track new releases in the open source universe and test the building of our various distribution images with them. While milestone 1 introduced various pre-release versions of free and open source projects (KDE 4.4 RC1, OpenOffice 3.2 Beta4 or VirtualBox 3.1 beta 1) into our development distribution openSUSE Factory, this milestone is characterized by final releases of those projects."

Full Story (comments: none)

PC-BSD 8.0 Released

The PC-BSD Team has announced the availability of PC-BSD 8.0 (Hubble Edition), running FreeBSD 8.0-RELEASE-P2, and KDE 4.3.5. "PC-BSD 8.0 contains a number of enhancements and improvements over the 7.x series. For a full list of changes, please refer to the changelog."

Comments (none posted)

Distribution News

Fedora

Fedora Board Recap 2010-02-18

Click below for a recap of the February 18, 2010 meeting of the Fedora Advisory Board. Topics include Improved metrics, Trademark approval for Security spin, Strategic Working Group outputs, What is Fedora the distro?, and Personas.

Full Story (comments: none)

Board SWG Meeting 2010-02-22 Recap

Click below for a recap of the February 22, 2010 meeting of the Fedora Board Strategic Working Group. Topics include Default Distribution Offering and Clarifying Issues Around Spins.

Full Story (comments: none)

Ubuntu family

identi.ca and twitter accounts setup to report the state of the Ubuntu archives

Robbie Williamson introduces identi.ca and twitter feeds to provide alerts and status when there are problems in the Ubuntu archives that are likely to cause serious problems for a large number of users.

Full Story (comments: none)

Ubuntu Netbook Remix enlightens ARM support (DesktopLinux)

DesktopLinux takes a look at Ubuntu's ARM port and other Ubuntu news. "Canonical is developing a 2D ARM interface based on Enlightenment Foundation Libraries for the upcoming Ubuntu 10.04 ("Lucid Lynx") version of Ubuntu Netbook Remix. In other Ubuntu news, Ubuntu Live CDs in Lucid Lynx will boot 33 percent faster, and The Linux Box will market Ubuntu."

Comments (1 posted)

Other distributions

Peter Tribble: OpenSolaris: Oracle, where art thou?

OpenSolaris developer Peter Tribble wonders about the fate of OpenSolaris. "That silence has continued. OpenSolaris has - publicly at least - been completely ignored by Oracle. It's as if we don't exist. Somewhat perturbed by this state of affairs, I asked for a communication channel to be established between Oracle and the OGB [OpenSolaris Governing Board]. This was ignored. The OGB pointed out (in pretty strong terms) to those Sun staff that we do have communication with that some level of contact was needed. We haven't heard back."

Comments (none posted)

Distribution Newsletters

Misc Debian developer news (#21)

This issue of miscellaneous Debian developer news covers DEHS, watching 10 000 packages, + Conversion of source packages to 3.0 (quilt/native), + DEHS now supports format 3.0 source packages, + userdir-ldap mail gateway updated, and + piuparts.d.o.

Full Story (comments: none)

DistroWatch Weekly, Issue 342

The DistroWatch Weekly for February 22, 2010 is out. "Now in its eighth incarnation, Linux Mint is no longer a new entry on the distribution list, but judging by its growing popularity, it's still marching from strength to strength. In this issue, we talk to Clement Lefebvre, the project's founder and lead developer, about the latest version, then take it for a quick test drive to see whether it justifies its label as being one of the most user-friendly operating systems available today. In the news section, Debian announces the availability of the first alpha release of its installer for "Squeeze", Ubuntu outlines plans for a new and lighter user interface for the ARM-based netbooks, OpenSolaris developers express disappointment over lack of communication from Oracle regarding their project, and Mandriva updates KDE to the recently released version 4.4 in its development branch, the "Cooker". Also in this issue we'll take a look at some of the disk mounting options that could increase the speed of accessing hard disks in modern Linux distributions. Happy reading!"

Comments (none posted)

Fedora Weekly News 214

The Fedora Weekly News for February 21, 2010 is out. "In Announcements, we start off with an invitation to contribute Fedora 13 Talking Points from the Fedora Marketing team. Also of note is the Fedora 13 branching and freezing that happened last week, and announcement of a new privilege escalation policy for Fedora packaging. In news from the Fedora Planet, details on how libvirt deals with different CPU models and passing on their capabilities to guests, a discussion on the value of having a target audience for Fedora, how to update your system BIOS without having to use Windows or a USB stick, and announcement of a new utility, gnome-speaker-setup. We're pleased to welcome Neville Cross to FWN to cover the Fedora Marketing Team activities for FWN. Neville's beat this week includes pointers to the Team's meeting last week as well as more internal detail on the Fedora 13 Talking Points, mentioned above. The Ambassador's beat includes coverage of last week's Southern California Linux Expo (SCALE 8X) con in California. In news from the Quality Assurance team, details on last week's team meetings, as well as the first Test Day in some time, this on Color Management, also updates on Fedora 13 images and a new Security Spin. In Translation news, announcement of Transifex v.0.7.4 for testing on the Fedora staging server, progress on the QA test of Fedora 13's L10n, and an update on Midnight Commander's availability for translations. In the Art/Design beat, updated details on Fedora 13 artwork including wallpapers, and testing results of the new Design Suite spin. Security Advisories provides last week's security patches for Fedora 11, 12, and for the first week, Fedora 13. This rounds out FWN 214, enjoy reading!"

Full Story (comments: none)

openSUSE Weekly News/111

This issue of the openSUSE Weekly News covers openSUSE News: Number Two Always Tries Harder: openSUSE Milestone 2, - Linux User & Developer/Kunal Deo: Another ten essential Python tips, - Gabriel Burt: Banshee Community Extensions, - How to Make sure I didn't get a Virus?, and - Jason McDonald: Qt 4.6.2 Released.

Comments (none posted)

Ubuntu Weekly Newsletter #181

The Ubuntu Weekly Newsletter for February 20, 2010 is out. "In this issue we cover: Debian Import Freeze in effect, Feature Freeze in place - Alpha 3 freeze ahead, New MOTU, Ubuntu single sign on service launched, Meet Jelmer Vernooij, Sikuli - scripting your use of GUIs, Global Bug Jam, Taking Lucid for a test spin, Opportunistic Developer Update, Ubuntu One Music Store, One Hundred Paper Cuts, Mark Shuttleworth to give keynote at PyCon 2010, Ubuntu UK Podcast returns, Ubuntu torrents are now IPv6 enabled, and much, much more!"

Full Story (comments: none)

Page editor: Rebecca Sobol

Development

Google releases "Living Stories" code

You are reading a standard-form news article, and when new information comes to light, the piece you're reading might just be referenced in a follow-up — but it won't be displayed in context or be easy to navigate. However, if Google's Living Stories experiment takes off following the release of its code, that won't always be the case.

Along with The New York Times and The Washington Post, Google worked on developing a new prototype for displaying news dynamically. The Living Stories project, announced in December of 2009, was an experiment on how to present news coverage online in a dynamic format rather than the staid and unchanging single-story per page mode. The project ran for about two months, and has been highly successful. According to the Google team that worked on the project, the feedback received has been extremely positive — with 75% of the people reporting that they preferred the Living Stories format. More importantly to publishers, who strive to keep time on site as high as possible, the readers spent "a significant amount of time exploring stories."

The problem with the online news

For all of the advances and changes brought about by online publishing, the way that news is reported online has changed remarkably little from print days. The speed of publishing has changed, and readers are able to access information on new and exciting devices: But the actual layout of a news story has remained more or less static.

Standard Web publishing layouts, like print, only work so well for telling ongoing stories. The best most publishing platforms can muster is a set of related links to earlier posts on the same topic. Navigating through this can be something of a nightmare when trying to dig through long-running stories. The standard presentation also leaves something to be desired in terms of seeing what the most current report is for any given topic.

Publishers and content management system designers have put more effort into mirroring the print experience online (making sites much prettier than the early days of online publishing) and paid little attention to how online publishing might better present the information at hand. With any luck, the Living Stories experiment and code release will push the envelope a bit and inspire publishers and developers to develop more efficient and intuitive ways to deliver news and other information.

The Living Stories Format

The page components of a Living Story are broken into four sections: A summary, update stream, timeline, and filter. The summary gives the gist of the topic and helps bring the reader into the story if they're unfamiliar with it, giving just the most important details. In addition, the Living Stories prototype has a navigable timeline that puts the story into context by displaying all the developments in a continuum. Readers can follow along with this and see just the headlines or drill down further into the complete updates at any point in the timeline.

The update stream, displayed in the middle column, shows updates in reverse order. Depending on the importance of an piece in this stream, it can be displayed with a larger or smaller font, or "collapsed" to show only the headline if a given update is of low priority. Major updates can be given more prominence.

Filters allow publishers to associate content with specific themes for readers to filter content by. For example, readers could drill down on specific elements like videos, graphics, quotes, or specific aspects of a story. If an LWN story was put into the Living Stories format, one might be able to filter by specific companies, or licenses, or by topics like distributions and development. This raises interesting questions for journalists as well as developers and publishers: The topics that are chosen as filters can shape the reader's interaction with a story. Someday setting the filters for a given topic on a major news site may be as much a part of the gatekeeping function of journalism as choosing the topics to be covered in the first place.

The final component is the right-hand timeline of events, which also link off to stories that are key elements in the story. Here only the most important pieces might be displayed, rather than every element that might be displayed in the overall stream. For example, if Oracle's acquisition of Sun were laid out as a Living Story, one might highlight some developments in the "Save MySQL" campaign.

Another part of the Living Stories design is to track the user's interaction with a story. On subsequent visits to a page, users would see new information highlighted. According to the data outline the Living Stories package would track users who are logged in and their last visit. It's not clear from the notes whether users would only be tracked if logged in.

So far, the new format has been used to hit a moderate range of topics. The Times used it for stories from global warming to the NFL Playoffs, and the Post test drove the format by looking at school reform in Washington D.C. and the embarrassing season the Washington Redskins just had. The stories are no longer being updated, but the existing content is still up for all to see.

Working with Living Stories

The code is also up for all to see as of February 17. The release is available under the Apache License 2.0, and includes documentation on the data structures, content types, and how to build and run the application. The code is written in Google's AppEngine Java SDK, but it may be possible to run Living Stories using AppScale on infrastructure other than Google's. AppScale allows running Google AppEngine applications on Amazon EC2, Eucalyptus, and on Xen and KVM systems.

The instructions provided so far require Eclipse, Google Plugin for Eclipse, the Google Web Toolkit SDK, and Google App Engine SDK. I didn't have much luck building the code following the instructions, but, to be fair, Java development in Eclipse is not something I have done previously. Perhaps it's user error. However, it was less than encouraging that three days after posting a question to the Living Stories discussion in Google Groups, it had not yet been moderated through to the list. In fact, no new posts have been approved or posted as of this writing (February 21st) since February 17th.

It's possible to get a sense of the workflow for Living Stories even without setting up an implementation. Google provides detailed documentation on the workflow for creating and editing content in the Living Stories Content Manager. Based on the instructions given, the content manager is a bit rough around the edges — at least from the viewpoint of editors and reporters who would have to manually insert the code required for some of the Living Stories features. The data structure and content types available in Living Stories are a bit more complex than the standard content management system. Living Stories allows for eight types of content ranging from Events (details related to the story that don't fit into other content types) to Data (for facts and data related to the story).

The specific implementation may not be as important, however, as the concept. As the core principles and best practices page notes, the package released by Google only represents "one possible implementation of these principles. Any news organization, however, can use the principles as a guide to implement their own version of living stories" as best suits the publication and its audience. With the examples and data structures that the project has developed out for all to work with, it should be possible to adapt the Living Stories concept to other content management systems and for use with all types of content.

Users who aren't looking to deploy on AppEngine may have hope. According to the Build and Run guide, alternate instructions are forthcoming for users who would prefer to deploy Living Stories with Apache and MySQL. I'm eager to see what the community develops based on Living Stories, and a simpler implementation that could be deployed on a standard LAMP setup would be welcome.

Whether the code is going to see much development from Google, the New York Times, or Washington Post at this point is unclear. The post on Google's News blog thanks both publications for their involvement so far, but suggests that the papers are moving away from working with the Google hosted code now that public development has started. The posts from Google so far indicate that the company does intend to keep developing Living Stories for the benefit of other news organizations. As yet, though, no other publications have announced plans to work with Living Stories.

Comments (3 posted)

System Applications

Audio Projects

Rivendell 1.7.0 released

Version 1.7.0 of Rivendell, a radio station automation system, has been announced. Changes include new log import format presets, a new AsPlayed report format, log import scripts, high-resolution timing, new translations and a database update.

Full Story (comments: none)

Database Software

MariaDB 5.1.42 is released

Version 5.1.42 of MariaDB has been announced. "MariaDB 5.1.42, a new branch of the MySQL database which includes all major open source storage engines, myriad bug fixes, and many community patches, has been released. We are very proud to have made our first final release, and we encourage you to test it out and use it on your systems."

Full Story (comments: none)

MySQL Community Server 5.1.44 has been released

Version 5.1.44 of MySQL Community Server has been announced. "MySQL Community Server 5.1.44, a new version of the popular Open Source Database Management System, has been released. MySQL 5.1.44 is recommended for use on production systems."

Full Story (comments: none)

PostgreSQL Weekly News

The February 21, 2010 edition of the PostgreSQL Weekly News is online with the latest PostgreSQL DBMS articles and resources.

Full Story (comments: 1)

Interoperability

Samba 3.5.0rc3 is available

Version 3.5.0rc3 of Samba has been announced. "This is the third release candidate of Samba 3.5.0. This is *not* intended for production environments and is designed for testing purposes only. Please report any defects via the Samba bug reporting system at https://bugzilla.samba.org/. Plans are to ship the final 3.5.0 release on March 1 if there are no major issues with 3.5.0rc3."

Comments (none posted)

Telecom

Announcing project OsmocomBB: Open Source GSM Stack

Harald Welte has announced project OsmocomBB, a project aimed at creating a Free Software GSM baseband firmware. "GSM has been deployed first 19 years ago. Despite billions of phones deployed world wide, all of them run a proprietary baseband firmware, consisting of proprietary drivers, RTOS and GSM protocol stack. OsmocomBB has set out to change this. We do not want our phones to be a black box connected 24/7 to a public network. We want to decide what kind of data our phone reveals about us or not."

Full Story (comments: 20)

Miscellaneous

gujin GPL bootloader version 2.8 released

Version 2.8 of gujin GPL bootloader has been announced. "The main improvements are in the handling of languages, Linux command line parameters description in /boot/gujin.cmd, GPT partition tables, and handling of file images like iso images. First, thanks to benny59 for the rewrite of italian messages; messages in russian, spanish, portugese, german, and dutch would probably improve with a human reader."

Full Story (comments: none)

LTSP 5.2, LDM 2.1 and LTSPfs 0.6 are out

New versions of LTSP, LDM and LTSPfs have been announced. LTSP is the Linux Terminal Server Project project. "Here's the quicky announcement kind-of blog post I made. It's now waiting for review on a few open source websites and will probably make some more noise over the days to come. http://www.stgraber.org/2010/02/21/ltsp-52-out Now it's time to start breaking things (but not too much, please !!!) and make LTSP event better."

Full Story (comments: none)

Desktop Applications

Audio Applications

Ecasound 2.7.1 released

Version 2.7.1 of Ecasound has been announced, it includes several new capabilities and some bug fixes. "Ecasound is a software package designed for multitrack audio processing. It can be used for simple tasks like audio playback, recording and format conversions, as well as for multitrack effect processing, mixing, recording and signal recycling. Ecasound supports a wide range of audio inputs, outputs and effect algorithms. Effects and audio objects can be combined in various ways, and their parameters can be controlled by operator objects like oscillators and MIDI-CCs. A versatile console mode user-interface is included in the package."

Full Story (comments: none)

Klactoveedsedstene 1.0.3 released

Version 1.0.3 of the Klactoveedsedstene song player has been announced. "I am pleased to announce that Klactoveedsedstene v1.0.3 has just been released. It doesn't include any new features as such, but has become more intelligent. This includes automatic detection of HTTP proxy, automatic import dialog after installation, and various other minor details."

Full Story (comments: none)

XMMS2 0.7 released

Version 0.7 of the XMMS2 music player has been announced. According to the release notes: "Noteworthy changes: * Waf build system updated to version 1.5.10 * Doesn't crash on OS X 10.6 (Snow Leopard) * Source preferences in collections serverside. * sc68 plugin * many nycli improvements".

Comments (none posted)

Desktop Environments

GNOME Software Announcements

The following new GNOME software has been announced this week:

• AT-SPI2 0.1.7 (new features, bug fixes and code cleanup)
• Brasero 2.29.91 (bug fix and translation work)
• Byzanz 0.2.1 (new features and bug fixes)
• Cheese 2.29.91 (new features, bug fixes and translation work)
• Empathy 2.29.91 (bug fixes, documentation and translation work)
• EOG Plugins 2.29.91 (bug fixes and translation work)
• Epiphany Extensions 2.29.91 (new features and translation work)
• Evince 2.29.91 (new features, bug fixes, documentation and translation work)
• Eye of GNOME 2.29.91 (bug fixes and translation work)
• GCalctool 5.29.91 (bug fixes, documentation and translation work)
• GLib 2.23.4 (new features, bug fixes and translation work)
• GNOME Activity Journal 0.3.3 (new features and code cleanup)
• GNOME Media 2.29.91 (bug fixes and translation work)
• GNOME Shell 2.29.0 (new features, bug fixes and translation work)
• Gnome Subtitles 1.0 (new features, bug fixes and translation work)
• gthumb 2.11.2.1 (build fix and translation work)
• GTK+ 2.19.6 (bug fixes and translation work)
• gtkmm 2.19.6 (new features, bug fixes and code cleanup)
• gtranslator 1.9.9 (bug fixes, code cleanup and translation work)
• libgdata 0.6.2 (bug fixes and translation work)
• libgweather 2.29.91 (translation work)
• libspectre 0.2.4 (bug fixes)
• mousetweaks 2.29.91 (new features and translation work)
• Mutter 2.29.0 (new features and bug fixes)
• Nanny 2.29.3 (bug fixes and translation work)
• Orca 2.29.91 (new features, bug fixes and translation work)
• Rygel 0.4.12 (new feature and bug fixes)
• Rygel 0.5.0 (new features and bug fixes)
• Sabayon 2.29.91 (bug fixes and translation work)
• seahorse-plugins 2.29.91 (bug fixes and translation work)
• tracker 0.7.21 (new features, bug fixes and translation work)

You can find more new GNOME software releases at gnomefiles.org.

Comments (none posted)

KDE Software Announcements

The following new KDE software has been announced this week:

• Clementine 0.1 (initial release)
• KDE Personal WebServer 0.1 (initial release)
• XbmcCover 0.1 (initial release)

You can find more new KDE software releases at kde-apps.org.

Comments (none posted)

The State of The X.Org Foundation 2010

Barton Massey, the (outgoing) secretary of the X.Org foundation, has posted a summary of the state of the Foundation in 2010. "In keeping with the X.Org goal of about one release per year, Release 7.5 of the X Window System occurred on October 26, 2009. This release featured the first official version of Multi-Pointer X, "E-EDID support", improved pointer acceleration, an XACE-based SELinux security module, and RandR version 1.3. It also included the kernel modesetting support developed over the last several years, with the goal of moving parts of X better handled by the host operating system into it." Also posted are the results of the 2010 board election, in which Alex Deucher, Keith Packard, Matthieu Herrb, Matthias Hopf, and Eric Anholt were elected.

Full Story (comments: none)

Desktop Publishing

Scribus 1.3.3.14 released

Version 1.3.3.14 of the Scribus desktop publishing system has been announced. "This represents the final version in the 1.3.3.x series, so now all developmental effort focuses on 1.3.5+ and beyond. Go to downloads to find your way to this latest stable version."

Comments (none posted)

Electronics

Kicad 2010-02-21-RC4 released

Version 2010-02-21-RC4 of Kicad, a circuit board CAD application, has been announced. Changes include: "Pcbnew: Lot of enhancements. Support of Netclasses (Please (re)read the on line documentation). Better DRC."

Comments (none posted)

Graphics

cairo release 1.8.10 is now available

Version 1.8.10 of the cairo graphics library has been announced. "This is the fifth update to cairo's stable 1.8 series. This release consists of about a dozen hand-picked fixes compared to 1.8.8 (which was released about 8 months ago). We recommend that everyone using cairo upgrade to 1.8.10."

Full Story (comments: none)

Inkscape enters chill phase for 0.48

The Inkscape SVG drawing editor is readying for the 0.48 release. "We are announcing chill phase and thus officially enter 0.48 release cycle. It means that development of new features now should be wrapping up, and focus should be shifting to bugfixing activity. So we are really serious about releasing 0.48 some time in May. An about screen contest will be announced separately. In the mean time you can help us by pointing out bugs and odd things in the documentation."

Comments (none posted)

Interoperability

Wine 1.1.39 announced

Version 1.1.39 of Wine has been announced. Changes include: "- Support for registry symbolic links. - Many MSI fixes. - Build process improvements. - MSXML cleanups and fixes. - A number of MSHTML improvements. - Various bug fixes."

Comments (none posted)

Mail Clients

Sylpheed 3.0rc released

Version 3.0rc of the Sylpheed mail client is available. Changes include: "* The Japanese manual was updated. * The 'Enable address auto-completion' setting was added. * Update check (curl command) timeout was set to 10 seconds. * The compile error with OpenSSL disabled was fixed. * Win32: multiple reference to a folder with different letter case is prohibited when creating a new one. * Some minor bugs were fixed."

Comments (none posted)

Office Applications

Leo 4.7 final released

Version 4.7 final of Leo has been announced. "Leo 4.7 final fixes all known bugs in Leo. Leo is a text editor, data organizer, project manager and much more."

Full Story (comments: none)

SyncEvolution 1.0 beta 2 released

Version 1.0 beta 2 of SyncEvolution, a PIM synchronizer, has been announced. "Major improvements: * easier setup of a phone with the sync-ui and GNOME Bluetooth plugin * prevention of slow sync enabled by default and integrated into sync-ui * recovery dialog and restore from backup in sync-ui * database dumps and comparisons more efficient".

Full Story (comments: none)

Science

Scilab 5.2.1 released

Version 5.2.1 of Scilab, an interactive platform for numerical computation, has been announced. See the whats new document for information on this version.

Full Story (comments: none)

Web Browsers

Apache turns 15

The Apache HTTP server is celebrating its 15th anniversary. "February 23, 1994: Individual patch authors around the world are invited to join the 'new-httpd' mailing list to discuss enhancements and future releases of NCSA httpd. The Apache name was chosen for this new effort within the first few days of discussion, along with basic rules for email-based collaboration and a mission to replace the existing server with a standards-based, open source, and extensible software system."

Comments (10 posted)

Mozilla announces new privacy policies

The Mozilla project has announced a new web site privacy policy. "Mozilla Foundation and Mozilla Corporation are updating the privacy policy that governs the websites and services they operate or are operated on their behalf. The new policy will appear shortly at http://www.mozilla.com/en-US/privacy-policy.html"

Full Story (comments: none)

Languages and Tools

Caml

Caml Weekly News

The February 23, 2010 edition of the Caml Weekly News is out with new articles about the Caml language.

Full Story (comments: none)

Python

BetterBatch 0.9.4 released

Version 0.9.4 of BetterBatch has been announced, a number of new capabilities have been added. "BetterBatch is designed as a middle ground between batch files and more powerful languages (Python, shell scripting, etc)."

Full Story (comments: none)

PiCloud 1.8 released

Version 1.8 of PiCloud has been announced. "PiCloud, a cloud-computing platform for the Python Programming Language, has released version 1.8 of its client library, cloud. PiCloud enables Python users to leverage the power of an on-demand, high performance, and auto scaling compute cluster with as few as three lines of code! No server management necessary."

Full Story (comments: 1)

posix_ipc 0.7.0 available

Version 0.7.0 of posix_ipc has been announced. "posix_ipc 0.7.0 is now available. This is the first version to include Python 3 support."

Full Story (comments: none)

DreamPie - The Python shell you've always dreamed about

Version 1.0 of DreamPie has been announced. "I'm pleased to announce DreamPie 1.0 - a new graphical interactive Python shell!"

Full Story (comments: none)

Page editor: Forrest Cook

Announcements

Non-Commercial announcements

The FSF's open letter to Google

The Free Software Foundation has laid down a challenge to Google in the form of an open letter. "With your purchase of On2, you now own both the world's largest video site (YouTube) and all the patents behind a new high performance video codec -- VP8. Just think what you can achieve by releasing the VP8 codec under an irrevocable royalty-free license and pushing it out to users on YouTube? You can end the web's dependence on patent-encumbered video formats and proprietary software (Flash)."

Comments (34 posted)

Legal Announcements

JMRI case settled

Andy Updegrove reports that Jacobsen v. Katzer (the JMRI case) has been settled on terms which uphold the enforceability of free software licenses in the US. "With the case now settled, there can be no further appeals - meaning that the rulings of the District and Appeals courts are now binding in their circuit. Although federal courts in other circuits will not be bound this court's decision, the California circuit is well respected, and other federal judges nationwide will be influenced by its legal conclusions. As a result, the results of the Jacobsen v. Katzer could eventually become the law of the land."

Comments (2 posted)

Articles of interest

Reeling in the hackers (Irish Times)

The Irish Times covers a study about hackers in films. "Also, the vast majority of hackers in films are actually portrayed as the good guys - a huge 73 per cent, with 10 per cent being somewhere in between, and 17 per cent portrayed as bad guys. "I was definitely surprised at the number of films showing hackers in a positive light," [Damian Gordon] says." (Thanks to John Britton)

Comments (6 posted)

New Books

97 Things Every Programmer Should Know--New from O'Reilly

O'Reilly has published the book 97 Things Every Programmer Should Know, edited by Kevlin Henney.

Full Story (comments: none)

Resources

LAC Times launched

The LAC Times has been launched. "I'm pleased to announce the first edition of the LAC Times, a newsletter as a prelude to the Linux Audio Conference 2010. We intend to release a new edition roughly every month with news, backgrounds and announcements regarding the Linux Audio Conference in May."

Full Story (comments: none)

Linux Foundation Newsletter

The February, 2010 edition of the Linux Foundation Newsletter has been published. "In this month's Linux Foundation newsletter: * Linux Foundation Hosts MeeGo Project * Free Linux Training Webinar Series Announced * New Collaboration Summit Program Details Revealed * 2010 "We're Linux" Video Contest Launches * Linux Foundation in the News * Upcoming Training Course from Linux Foundation".

Full Story (comments: none)

Hg Init: a Mercurial tutorial

Joel Spolsky has put together an extensive Mercurial tutorial with lots of examples and a humorous approach. "Finally, if you haven’t collapsed yet from exhaustion, before I finish this tutorial, I just want to show you one more tiny thing: you can use the hg update command to go backwards or forwards in time to any revision you want. Well, you can’t really go into the future per se, although that would be super-cool. If you only had four revisions you would just hg update -r 103994 and get some really cool anti-gravity sci-fi futuristic version of your source code. But of course, that is not possible."

Comments (75 posted)

Interviews

A handbook for the open source way (opensource.com)

Opensource.com talks with Karsten Wade about the project to write The Open Source Way, a book on creating communities. "The chapter that has me most excited and the one I want to dive into now that I have some breathing room is Business the open source way. There is potential to write some groundbreaking, new content here to identify practices that people haven't thought of in that way before. There is also an even greater chance to make one canonical location for references and learning derived from the best books and the best minds about applying these principles to business."

Comments (none posted)

Surveys

Arduino Usage Survey

The Arduino open hardware microprocessor project is conducting an online Usage Survey. "We're looking to gather data about how you use the Arduino hardware, software, and website. This survey should take about five to ten minutes. Please share it with all the Arduino users you know, so we can get a broad view of how you're using it."

Comments (none posted)

The Five Best Linux Video Players (Linux.com)

Here's a survey of Linux video players on Linux.com. "The video support for VLC on Linux is outstanding. If you can't watch it in VLC, odds are you can't watch it. Note that VLC also boasts support for quite a few subtitle and captioning formats, so it may be the best option for users who need or want subtitles with their video for accessibility reasons or just because they want to be able to watch their dialog."

Comments (35 posted)

Calls for Presentations

12th German Perl Workshop 2010 cfp (use Perl)

use Perl has announced the call for papers for the 12th German Perl Workshop 2010. "The 12th German Perl Workshop will be hosted from June 7th to 9th 2010 in Schorndorf near Stuttgart. The workshop is targeted towards all experienced and beginning Perl developers. Obviously, the workshop is only as good as its talks — your talks. We are interested in all talks about Perl itself or about Perl related topics, and especially talks that would apply to this year's motto “Modern Perl“. The submission deadline for your 5, 20 or 40 minute talks has been extended until March 5th, 2010."

Comments (none posted)

Rails 3 release explored at RailsConf

A call for participation has gone out for RailsConf. "Sebastopol, CA, February 18, 2010 - This year's RailsConf offers the Rails community the opportunity to delve into the possibilities created by the Rails 3 release. O'Reilly Media and Ruby Central, Inc. have opened registration for RailsConf on June 7-10, 2010, at the Baltimore Convention Center in Baltimore, MD. To ensure that the program includes the most timely content possible, the call for participation remains open until March 17, 2010."

Full Story (comments: none)

Upcoming Events

Collaboration Summit preliminary program agenda is now available

The preliminary program agenda for the 4th Annual Linux Foundation Collaboration Summit has been announced. The summit will be held in San Francisco, CA on April 14-16, 2010. "The Collaboration Summit is an exclusive, invitation-only gathering of the brightest minds in Linux, including core kernel developers, distribution maintainers, ISVs, end users, system vendors and other community organizations. It is the only conference designed to accelerate collaboration and encourage solutions by bringing together a true cross-section of leaders to meet face-to-face to tackle and solve the most pressing issues facing Linux today."

Full Story (comments: none)

OOoCon - September in Budapest

The next OpenOffice.org Conference has been announced, the call for papers will happen soon. "The OpenOffice.org community today announced the date for its anniversary OpenOffice.org Conference. This OOoCon in the year of the project's 10th birthday, will take place from August 31st to September 3rd in Budapest, Hungary."

Full Story (comments: none)

PostgreSQL Conference East talks announced

The PostgreSQL Conference East talks have been announced. "The PostgreSQL Conference, U.S. team is pleased to announce the current list of talks for PostgreSQL Conference East. Being held in Philadelphia from March 25th through March 28th, PostgreSQL Conference East is the PostgreSQL Conference for Developers, Users and Decision Makers who want to learn and network with all that is PostgreSQL. The list of talks can be found here."

Full Story (comments: none)

Events: March 4, 2010 to May 3, 2010

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
March 1 March 5	Global Ignite week	Online, Online
March 2 March 4	djangoski	Whistler, Canada
March 2 March 5	FOSSGIS 2010	Osnabrück, Germany
March 2 March 6	CeBIT Open Source	Hannover, Germany
March 5 March 6	Open Source Days 2010	Copenhagen, Denmark
March 7 March 10	Bossa Conference 2010	Recife, Brazil
March 13 March 19	DebCamp in Thailand	Khon Kaen, Thailand
March 15 March 18	Cloud Connect 2010	Santa Clara, CA, USA
March 16 March 18	Salon Linux 2010	Paris, France
March 17 March 18	Commons, Users, Service Providers	Hannover, Germany
March 19 March 20	Flourish 2010 Open Source Conference	Chicago, IL, USA
March 19 March 21	Panama MiniDebConf 2010	Panama City, Panama
March 19 March 21	Libre Planet 2010	Cambridge, MA, USA
March 22	OpenClinica Global Conference 2010	Bethesda, MD, USA
March 22 March 26	CanSecWest Vancouver 2010	Vancouver, BC, Canada
March 23 March 25	UKUUG Spring 2010 Conference	Manchester, UK
March 25 March 28	PostgreSQL Conference East 2010	Philadelphia, PA, USA
March 26 March 28	Ubuntu Global Jam	Online, World
March 30 April 1	Where 2.0 Conference	San Jose, CA, USA
April 9 April 11	Spanish DebConf	Coruña, Spain
April 10	Texas Linux Fest	Austin, TX, USA
April 12 April 14	Embedded Linux Conference	San Francisco, CA, USA
April 12 April 15	MySQL Conference & Expo 2010	Santa Clara, CA, USA
April 14 April 16	Linux Foundation Collaboration Summit	San Francisco, USA
April 14 April 16	Lustre User Group 2010	Aptos, California, USA
April 16	Drizzle Developer Day	Santa Clara, CA, United States
April 16 April 17	R/Finance 2010 Conference - 2nd Annual	Chicago, IL, US
April 23 April 25	FOSS Nigeria 2010	Kano, Nigeria
April 23 April 25	QuahogCon 2010	Providence, RI, USA
April 24	Festival Latinoamericano de Instalación de Software Libre	Many, Many
April 24	Open Knowledge Conference 2010	London, UK
April 24 April 25	OSDC.TW 2010	Taipei, Taiwan
April 24 April 25	BarCamb 3	Cambridge, UK
April 24 April 25	Fosscomm 2010	Thessaloniki, Greece
April 24 April 25	LinuxFest Northwest	Bellingham WA, USA
April 24 April 26	First International Workshop on Free/Open Source Software Technologies	Riyadh, Saudi Arabia
April 25 April 29	Interop Las Vegas	Las Vegas, NV, USA
April 28 April 29	Xen Summit North America at AMD	Sunnyvale, CA, USA
April 29	Patents and Free and Open Source Software	Boulder, CO, USA
May 1 May 2	OggCamp	Liverpool, England
May 1 May 2	Devops Down Under	Sydney, Australia
May 1 May 4	Linux Audio Conference	Utrecht, NL

If your event does not appear here, please tell us about it.

Page editor: Forrest Cook