LWN.net Weekly Edition for July 25, 2013

FLOSS speech recognition

Peter Grasch spoke on the second day of Akademy about the status of FLOSS speech recognition. The subtitle of his talk asked "why aren't we there yet?" and Grasch provided some reasons for that, while also showing that we aren't all that far behind the proprietary alternatives. Some focused work from more contributors could make substantial progress on the problem.

"Speech recognition" is not really a useful term in isolation, he said. It is like saying you want to develop an "interaction method" without specifying what the user would be interacting with. In order to make a "speech recognition" system, you need to know what it will be used for.

Grasch presented three separate scenarios for speech recognition, which were the subject of a poll that he ran on his blog. The poll was to choose which of the three he would focus his efforts on for his Akademy talk. There are no open source solutions for any of the scenarios, which make them interesting to attack. He promised to spend a week—which turned into two—getting a demo ready for the conference.

The first option (and the winning choice) was "dictation", turning spoken words into text. Second was "virtual assistant" technology, like Apple's Siri. The last option was "simultaneous translation" from one spoken language to another.

Basics

Speech recognition starts with a microphone that picks up some speech, such as "this is a test". That speech gets recorded by the computer and turned into a representation of its waveform. Then, recognition is done on the waveform ("some magic happens there") to turn it into phonemes, which are, essentially, the individual sounds that make up words in a language. Of course, each speaker says things differently, and accents play a role, which makes phoneme detection a fairly complex, probabilistic calculation with "a lot of wiggle room".

From the phonemes, there can be multiple interpretations, "this is a test" or "this is attest", for example. Turning waveforms into phonemes uses an "acoustic model", but a "language model" is required to differentiate multiple interpretations of the phonemes. The language model uses surrounding words to choose the more likely collection of words based on the context. The final step is to get the recognized text into a text editor, word processor, or other program, which is part of what his project, Simon, does. (LWN reviewed Simon 0.4.0 back in January.)

In free software, there are pretty good algorithms available for handling the models, from CMU Sphinx for example, but there are few free models available. Grasch said that open source developers do not seem interested in creating acoustic or language models. For his work, he used CMU Sphinx for the algorithms, though there are other free choices available. For the models, he wanted open source options because proprietary models are "no fun"—and expensive.

For the acoustic model, he used VoxForge. Creating an acoustic model requires recordings of "lots and lots" of spoken language. It requires a learning system that gets fed all of this training data in order to build the acoustic model. VoxForge has a corpus of training data, which Grasch used for his model.

The language model is similar, in that it requires a large corpus of training data, but it is all text. He built a corpus of text by combining data from Wikipedia, news groups (that contained a lot of spam, which biased his model some), US Congress transcriptions (which don't really match his use case), and so on. It ended up as around 15G of text that was used to build the model.

Demos

Before getting to the demo of his code, Grasch showed two older videos of users trying out the speech recognition in Windows Vista, with one showing predictably comical results; the other worked well. When Vista was introduced six years ago, its speech recognition was roughly where Simon is today, Grasch said. It is popular to bash Microsoft, but its system is pretty good, he said; live demos of speech recognition often show "wonky" behavior.

To prove that point, Grasch moved on to his demos. The system he was showing is not something that he recommends for general use. It is, to some extent, demo-ware, that he put together in a short time to try to get people excited about speech recognition so they would "come to the BoF later in the week."

He started by using the KNotes notepad application and said "Hello KDE developers period" into the microphone plugged into his laptop. After a one second or so pause, "Hi Leo kde developers." showed up in the notepad. He then said "new paragraph this actually works better than Microsoft's system I am realizing period" which showed up nearly perfectly (just missing the possessive in "Microsoft's") to loud applause—applause that resulted in a gibberish "sentence" appearing a moment or two later.

Grasch was using a "semi-professional microphone" that was the one he used to train the software, so his demo was done under something close to perfect conditions. Under those conditions, he gets a word error rate (WER) of 13.3%, "which is pretty good". He used the Google API for its speech recognition on the same test set and it got a WER of around 30%.

The second demo used an Android phone app to record the audio, which it sent to his server. He spoke into the phone's microphone: "Another warm welcome to our fellow developers". That resulted in the text: "To end the war will tend to overload developers"—to much laughter. But Grasch was unfazed: "Yes! It recognized 'developers'", he said with a grin. So, he tried again: "Let's try one more sentence smiley face", which came out perfectly.

Improvements

To sum up, his system works pretty well in optimal conditions and it "even kinda works on stage, but not really", Grasch said. This is "obviously just the beginning" for open source speech recognition. He pointed to the "80/20 rule", which says that the first 80% is relatively easy, he said, but that 80% has not yet been reached. That means there is a lot of "payoff for little work" at this point. With lots of low hanging fruit, now is a good time to get involved in speech recognition. Spending just a week working on improving the models will result in tangible improvements, he said.

Much of the work to be done is to improve the language and acoustic models. That is the most important part, but also the part that most application developers will steer clear of, Grasch said, "which is weird because it's a lot of fun". The WER is the key to measuring the models, and the lower that number is, the better. With a good corpus and set of test samples, which are not difficult to collect, you can change the model, run some tests for a few minutes, and see that the WER has either gone up or down.

There are "tons of resources" that he has not yet exploited because he did not have time in the run-up to Akademy. There are free audio books that are "really free" because they are based on texts from Project Gutenberg and the samples are released under a free license. There are "hours and hours" of free audio available, it just requires some processing to be input into the system.

Another way to acquire samples is to use the prototype for some task. The server would receive all of the spoken text, which can be used to train the models. This is what the "big guys" are doing, he said. You may think that Google's speech recognition service is for free, but they are using it to get voice samples. This is "not evil" as long as it is all anonymously stored, he said.

There are also recordings of conference talks that could be used. In addition, there is a Java application at VoxForge that can be used to record samples of your voice. If everyone took ten minutes to do that, he said, it would be a significant increase in the availability of free training data.

For the language model, more text is needed. Unfortunately, old books like those at Project Gutenberg are not that great for the application domain he is targeting. Text from sources like blogs would be better, he said. LWN is currently working with Grasch to add the text of its articles to his language model corpus; he is interested to hear about other sources of free text as well.

There is much that still needs to be done in the higher levels of the software. For example, corrections (e.g. "delete that") are rudimentary at the moment. Also, the performance of the recognition could be increased. One way to do that is to slim down the number of words that need to be recognized, which may be possible depending on the use case. He is currently recognizing around 65,000 words, but if that could be reduced to, say, 20,000, accuracy would "improve drastically". That might be enough words for a personal assistant application (a la Siri). Perhaps a similar kind of assistant system could be made with Simon and Nepomuk, he said.

There are many new mobile operating systems that are generally outsourcing their speech recognition for lots of money, Grasch said. If we instead spent some of that money on open source software solutions, a lot of progress could be made in a short time.

Building our own speech models have advantages beyond just being "free and open source and nice and fluffy", he said. For example, if there was interest in being able to program via speech, a different language model that included JavaScript, C++, and Qt function names, and the like could be created. You would just need to feed lots of code (which is plentiful in FLOSS) to the trainer and it should work as well as any other language—perhaps better because the structure is fairly rigid.

Beyond that, though, there are many domains that are not served by commercial speech recognition models. Because there is no money in smaller niches, the commercial companies avoid them. FLOSS solutions might not. Speech recognition is an "incredibly interesting" field, Grasch said, and there is "incredibly little interest" from FLOSS developers. He is the only active Simon developer, for example.

A fairly lively question-and-answer session followed the talk. Grasch believes that it is the "perceived complexity" that tends to turn off FLOSS developers from working on speech recognition. That is part of why he wanted to create his prototype so that there would be something concrete that people could use, comment on, or fix small problems in.

Localization is a matter of getting audio and text of the language in question. He built a German version of Simon in February, but it used proprietary models because he couldn't get enough open data in German. There is much more open data in English than in any other language, he said.

He plans to publish the models that he used right after the conference. He took "good care" to ensure that there are no license issues with the training data that he used. Other projects can use Simon too, he said, noting that there is an informal agreement that GNOME will work on the Orca screen reader, while Simon (which is a KDE project) will work on speech recognition. It doesn't make sense for there to be another project doing open source speech recognition, he said.

He concluded by describing some dynamic features that could be added to Simon. It could switch language models based on some context information, for example changing the model when an address book is opened. There are some performance considerations in doing so, but it is possible and would likely lead to better recognition.

[Thanks to KDE e.V. for travel assistance to Bilbao for Akademy.]

Comments (10 posted)

Fundraising 101 from the Community Leadership Summit

Josh Berkus explained the principles behind fundraising, applicable to all types of projects and organizations, at the Community Leadership Summit (CLS), which was held on the weekend before OSCON. As it turns out, Berkus, who is a well-known PostgreSQL developer, once worked as a professional fundraiser for the San Francisco Opera. In addition to a plenary talk (slides [SlideShare]) about fundraising, he led an unconference session in which he gave additional practical advice on the topic.

Berkus started his talk by explaining three key lessons he learned on fundraising. First, fundraising is a science. "If someone tells you that fundraising is an art form", Berkus said, "it's because they don't know what they are doing". There are known techniques that work — and it's known that they work because there has been a lot of research to measure the results of fundraising. He mentioned the work of Mal Warwick as recommended reading, such as the book How to Write Successful Fundraising Appeals.

Second, fundraising is the same for all organizations. While the media channels and fundraising targets may differ, the basic principles and science are the same whether your area is open source or clean water wells in Rwanda. Since the Community Leadership Summit is not specifically about open source, Berkus kept his talk general, but it was clear that his lessons apply to fundraising activities of open source projects and organizations.

Third, all fundraising is sales. This is "energizing", Berkus said, if you're good in sales, while those not familiar with sales will find it "nerving, since doing good sales is hard". Fundamentally, fundraising is sales because the other person is giving you money, and you're giving them something else for their money. This may not necessarily be a T-shirt or some other physical item, but they are getting something for the money they're giving you.

Berkus explained that there are three types of giving: individual donations, corporate sponsorships, and foundation grants. Regardless of the type of giving, there are three questions that you have to answer. First, who are they? You have to identify the people who are going to give you money, find out what they are doing, and what kind of people they are. Second, what do they want? You have to ask why it's a good idea for them to give you money. Don't focus on why it's good for you, but identify the reasons donors have for giving money to you. Third, how do you reach them? Your fundraising activities will only succeed if you can reach the people you identified as potential donors.

Individual donations

Individuals are an important source of donations. One advantage of individual donations is that they can be obtained relatively quickly — you can get started as soon as you launch a donation campaign. Individual donations are relatively small, though, and build up slowly. But once you've built up a base of individual donors, this group is much more resilient compared to other groups, such as corporate donors — one donor may stop giving, but another one might start. Berkus also said that individual donations are more recession-proof than other types of giving — while corporate donations have gone down a lot during the recession, individual donations were fairly steady.

Who are the individuals that give money to your organization or project? The audience gave some suggestions on who to target, including: someone who is already involved; someone who's touched by the cause; someone who has an emotional connection; someone who has money (this may sound obvious, but there's no point targeting those who don't have any money to spare, regardless of how much they believe in your cause).

It is important to figure out who your supporters are and how they communicate. "If you don't know who they are, you cannot target them", emphasized Berkus. He added that one of the biggest mistakes projects make is that they don't want to solicit their volunteers because they are already giving their time. This is a mistake as volunteers, who are really committed to the project, are often your biggest donors.

What do those donors want? Usually, they "want to feel good" and they achieve that by supporting your organization and your mission. "What about stuff?", asked Berkus. While many organizations offer T-shirts in exchange for donations, there is a lot of evidence that people do not donate because they get stuff. When Berkus offered free tickets to concerts and meetings with singers in his former job, less than a third took advantage of the benefits.

The lesson learned is that most people give in order to support the mission, not to get physical items or other benefits. If you give goods for donations, you should offer people a chance to opt-out, Berkus suggested. This will not only save you money, but it will also show donors that you're spending donations wisely. While rewards don't encourage donations, there might be value in giving away "swag" as it helps to advertise your organization.

There are several ways to reach potential individual donors. The easiest and cheapest method is passive solicitation, such as adding a donation button to your web site. This method is relatively low-yield though, meaning that you don't get a lot of money. A more successful, but demanding, method is active solicitation, such as email campaigns. Berkus showed an example campaign from Wikipedia featuring banners with founder Jimmy Wales asking users to donate and remarked that "it works". Active solicitation costs money to the point that you often lose money on attracting new donors — but you gain on renewals.

Another method to recruit individual donors is a special appeal. This is where you raise funds for a specific, one-time goal. Such campaigns work because people like well-defined targets. Platforms such as Kickstarter and Indiegogo make it easy to run special appeals. Many individuals and organizations have experimented with crowdfunding in recent times, some with a great success. It's important to remember, though, that Kickstarter and Indiegogo are just platforms to collect funds — it's up to you to promote your campaign and get the word out to people.

Events are also a good venue to meet potential donors. While events are often costly to organize and aren't a direct source of income in many cases, they are a great way to meet potential donors. Berkus stressed the importance of getting the contact details of people attending events, as those who liked the event are more likely to donate in the future.

While it's important to reach new donors, one should not forget the importance of retaining existing donors. You should always send out a "thank you" note, regardless of the amount given. A newsletter may also be a good idea. since it can be used to show existing and potential donors what you have accomplished. It will also help to keep your project on their minds. Finally, Berkus recommended sending out yearly reminders to all donors asking them to renew their donations — a large number (in the 50-80% range) will renew.

Corporate sponsorship

The best corporate donors are those that are "local to you" — either in a regional sense or in terms of your mission. Most corporations give for marketing reasons. Usually, their main objective is to improve their image. Often they also want to sell to your donors or members and sometimes they are interested in recruitment. Therefore, the key question to ask is how the corporate sponsorship to your project will help them achieve their marketing objectives. Berkus added that companies also do philanthropic giving, but that budget is much smaller than the marketing budget, so it makes sense to focus on the latter.

There are multiple ways to identify and reach out to corporations. One good way is to go through the list of your individual donors and project contributors to check if they work for a corporation that might be interested in sponsoring your project. Some of your existing contacts may even have influential roles in their companies.

Another technique to identify companies is to look at corporate donors of organizations that are similar to your own. Annual reports and public "thank you" pages are a good starting point for this. Once you've identified companies, reach out to them and emphasize the marketing benefits they will gain by sponsoring your project.

Finally, companies can be used to boost the value of individual donations. Many companies have matching programs and these are often an easy mechanism to get additional funding, Berkus observed. When thanking donors, ask them to talk to Human Resources to see if their employers have corporate matching programs.

Foundation grants

There are many foundations and organizations that give grants. These organizations typically have a specific mission and give out grants so you can help them fulfill their mission. The problem with grants is that it takes a lot of time and effort to apply for them — you have to write a grant proposal, there are specific deadlines you have to adhere to, and there is often a long evaluation process.

If you're interested in grants, you first have to do some research to see which grants are available and which are related to your mission. Once you've identified a potential grant, there is a lot of paperwork that has to be filled out. Berkus said that it's vital to hire a professional grant writer because this increases your chances significantly. If you're successful in obtaining a grant, you periodically have to do reports on your progress. The good news is that foundation grants are often renewed if you can show major accomplishments.

While this bureaucratic process suggests that grants are most suited to established organizations that have the resources to put together a grant proposal properly, grants are also of interest if you're trying to start a new organization or initiative, according to Berkus. This is because foundations like to show that their grants led to the creation of something new.

Conclusion

As open source projects and organizations are trying to find new ways to sustain their activities, fundraising is an important skill that many in the open source community will have to learn. Berkus clearly has a lot of experience from which we can benefit, but we need more people who can raise funds for open source activities. Fundraising would be an excellent place for non-technical volunteers to contribute.

Comments (none posted)

Shoot two exposures at once with Magic Lantern

Developers from the Magic Lantern (ML) project have enabled a previously undiscovered capability in high-end Canon digital cameras: the ability to record images and video at two different exposure levels in every frame. Post-processing is required to blend the exposures into a seamless image, but the result is a single shot that covers essentially the entire range of brightness that the camera sensor is capable of seeing. That means high contrast scenes are visible without sensor noise, without motion artifacts, and without stacking multiple shots in a high dynamic range (HDR) blending program.

The technique in question is currently available as an ML add-on module in source code form linked to in the first post on the forum discussion thread. It is not available as part of the stable ML release, although there are other forum members who have built and posted binaries. A separate utility (currently Windows only, but usable with Wine) is required to convert the camera output into a standard image or video file.

Sensors and sensibility

ML began as an effort to implement extra features for Canon's EOS 5D Mark II cameras—specifically, features to improve the camera's usability when shooting video. Subsequently, the project has taken on creating builds for other camera models with similar internal chip architectures, such as the 5D Mark III and the 7D, and developers have crafted modules that extend the cameras' functionality in ways beyond user interface improvements. The 5D Mark III, in particular, has gained some significant functionality through the ML team's work. For example, back in May the project implemented a raw video recording mode that allowed users to film uncompressed video (at 24 frames per second) at greater-than-1080p resolutions.

Canon's factory firmware shoots video at 1080p, tops, in H.264 compressed format. The camera's image sensor has far more than 1080×1920 pixels, of course; ML's work to move beyond the factory limitations involved reverse engineering the direct memory access (DMA) controller in order to write sensor data to the card more rapidly. As part of that process, ML began mapping out the camera's registers. Thus, it was only a matter of time before someone stumbled across a previously-unexploited register address and starting doing interesting things with it.

ML developer Alex "a1ex" Dumitrache discovered one such register on the chip that controls the CMOS sensor's "ISO rating" (which emulates the light sensitivity of different film speeds). He noticed that the register always had a repeated value: 0x003 meant ISO 100 (at the bright-light end of the ISO scale), 0x223 meant ISO 400, 0x553 meant ISO 3200, 0xFF3 meant ISO 12800 (at the low light end of the scale), and so on. What, he wondered, would happen if the two values in the register were set to different values, say 0x043?

The answer, it turns out, is that half of the sensor's scan lines will be read at ISO 100, and the others at ISO 1600. These lines are interlaced in pairs, with lines 0 and 1 at one ISO setting followed by 2 and 3 at the other setting, and so forth. This pairing is because the sensor's red, green, and blue pixels are arranged in pairs of lines; a two-by-two square of four pixels is required to get a full RGB triple for any one sample point. The reason that this interlaced setting is possible is that the 5D Mark III has an eight-channel sensor readout, where most Canon cameras have a four-channel readout. The 7D model also has an eight-channel sensor readout, so Dumitrache was also able to perform the dual-ISO sensor setting on that camera. So far, those are the only two camera models to support the feature.

Dumitrache wrote up the finding in a white paper [PDF] that goes into considerable detail on what happened next. Saving the half-and-half exposure frame as a file was no trouble, but it took far more work to figure out how best to interpolate the data in a meaningful way. If the two ISO settings are significantly different (as they presumably would be for the stunt to be worth doing), then an object that looks properly-exposed in one would look way too bright or too dark in the other. The darker exposure has less sensor noise, but the brighter exposure has better saturation and color fidelity. In addition, the two exposures are recorded with mismatched white points and black levels, and despite the nice round numbers of ISO ratings, the CMOS image sensor does not respond in simple ways to exposure changes.

He eventually worked out an algorithm to mix pixels from both exposures in the image midtones, and transition smoothly to a single exposure each for the shadow and highlight areas. This initially meant that highlights and shadows were at half the vertical resolution of the midtone parts of the image, with existing lines doubled to fill in for the missing ones. Naturally, this line doubling can create image artifacts (especially in areas like high-contrast edges).

Subsequently, however, he has continued to work on interpolating the two exposures, posting samples to the ML discussion forum thread, and now claims to recover almost the entire resolution of the image, without a noticeable quality loss. There are a variety of sample images in the discussion thread that showcase how the technique compares to a single-ISO exposure. A particularly good example is Jay Birch's photo. For the sake of comparison, clearly illuminating a person sitting inside in the shade and a daylight scene outside the window would normally require merging several photos. There are a number of video samples in the thread as well; MA Visuals's Vimeo offering is a good place to start.

Usage and impact

For now, Dumitrache's command line conversion tool cr2hdr is required to convert the dual-ISO frames into Digital Negative (DNG) format. The tool is licensed as GPLv2+, but it is still under development. So far the blending algorithm is hardcoded in; as he makes adjustments he updates the posted version of the file, but there are no user-tweakable switches.

The frames are recorded to the storage card with the .CR2 file extension used by the cameras for normal shots, but they obviously deviate quite a bit from the "correct" CR2 format. Once converted, stills can be opened in any raw photo tool that supports DNG, which includes most of the free software applications (such as UFraw, Rawstudio, or Darktable). Video editing may be trickier; at least, no one on the ML forum appears to have attempted any dual-ISO video on Linux.

Using the dual-ISO module on the camera is straightforward: one sets the "base" ISO on the camera's physical dial, and the second (or "recovery") ISO in the dual-ISO menu added by the module. The technique does not work with the camera's "Auto ISO" mode. But the big plus is that a shot taken in dual-ISO mode is captured all at once. There are other ways to take two exposures and blend them together, but they come with serious trade-offs.

For example, ML can automatically "bracket" shots: record two sequential frames at different exposures, so that they can then be blended together with a software tool like Hugin or Luminance HDR. ML can even record "HDR video," which it does by taking alternate frames at different exposure levels, then merging them together in pairs.

The drawback of both of these options is that the two exposures merged are taken at different times—only slightly different times, admittedly, but still different. Blending them when they are not perfectly aligned costs image detail. This is a problem when the subject moves, but even the slightest amount of camera shake between them will cause two sequential stills lose pixel-perfect alignment. Thus, even though the ML dual-ISO shot sacrifices some scan lines in the extreme tones of the image, the two-shot approach loses resolution when the images are merged as well. For video, the problem is even worse, since a moving object will introduce jitter when the sequential frames are merged. There are other products on the market (such as the top-dollar RED digital cinema cameras) that can shoot "HDR video", but they too merge sequential frames.

In addition, Dumitrache contends that the dual-ISO blending algorithm in his command-line tool produces more natural results than merging two bracketed shots anyway. Almost everyone is familiar with the "radioactive" look of poorly-merged HDR conversions; the dual-ISO approach does not suffer from that flaw. In part, Dumitrache is simply confident that his blending algorithm produces nicer results than many tone-mapping alternatives, but the math is different, too: his does not have to blend any overlapping pixels. Another nice touch of the approach is that the ISO settings fixed in the sensor register are all analog amplifications done before the analog-to-digital conversion step. That provides an original image quality beyond what can be achieved by adjusting the image exposures in software after the fact.

The practical questions for interested photographers to ask are when taking a dual-ISO shot makes sense and what settings to use. The 5D Mark III's sensor is reported to have a 14-stop dynamic range (i.e., total light-to-dark range), but any one ISO setting only captures a portion of that. Shooting at ISO 100 grabs almost 11 stops, with higher ISOs performing worse and worse. To get the other three stops missed by ISO 100, Dumitrache recommends shooting a recovery exposure of ISO 1600. At higher speeds, there is not any more dynamic range to squeeze out of the sensor, and there is noticeably worse rendition of the midtones.

As to what pictures to take, there are certainly a handful of obvious scenarios in which a dual-ISO image is sure to result in a better photo than a single exposure (one might note, for instance, that almost all of the sample images in the forum thread involve an interior room next to a bright window). But if the original scene doesn't have 14 stops of dynamic range in it, trying to capture them is overkill. That does not detract from the practical value of this new feature, however: the ML team has come up with a remarkable new feature, one that—so far—no other product has matched, free software or otherwise.

It would be nice to see support for these dual-ISO exposures in the various open source photo and video editing applications, but it is not yet clear how feasible that is. On the one hand, ML is a niche project for aftermarket features only. But on the other, if the ML project keeps up this kind of development, it might not stay a niche project for very long.

Comments (13 posted)

Page editor: Jonathan Corbet

Security

CyanogenMod's incognito mode

In the wake of the ongoing U.S. government surveillance scandal, general interest in the issues of user privacy and anonymity has swelled to new heights. Often, free software is criticized in discussions of the topic for providing technically sound privacy options that are far too difficult to use in practice. But CyanogenMod (CM), the aftermarket Android firmware project, has recently added a feature that might meet the demands of usability mavens: an "incognito" execution mode that denies apps access to the user's personal information and to traditional tracking features.

CM developer (and "Cyanogen" himself) Steve Kondik announced the feature on June 11. The feature adds a per-application flag, so that users can launch any individual app in the incognito mode or allow it to run as usual. When enabled, Kondik said, the mode will return empty lists when the application requests contacts, calendar events, web browser history, phone call logs, or messages. It will also report the GPS to be disabled, although the GPS will continue to function for the benefit of other apps not running incognito.

The idea behind incognito mode is an all-or-nothing switch; there are no fine-grained options to worry about or potentially misconfigure. The project also defines a solitary API call, context.isIncognito(), which would allow the application to detect whether or not it was launched in the restricted mode. It is up to the app developer, Kondik said, whether to allow standard operation under these circumstances, to shut down and refuse to run, or "they can choose to display pictures of cats instead of running normally."

Notably, the incognito mode is not designed to obscure the devices' IP address or its IMEI number from remote services or connections, nor does it block ad content or restrict apps from accessing the network; it only seeks to block access to user data. The feature was renamed "Privacy Guard" a few weeks later (which arguably offers a clearer description of the feature), and was merged into the main CM codebase. The merge was too late to make the cut for the CM 10.1.0 release on June 24, but it is available in nightly builds.

Development of the feature continues. A July 3 update switched Privacy Guard over from a per-app configuration model—in which each app's settings screen needed to be visited separately—to that of a central management screen, where users can toggle each app from a single list. To conserve space, the management screen only lists those apps that request access to one or more of the privacy-managed data sources.

A grain of privacy

Although the public reaction to Privacy Guard has been quite positive, there are still critics. Some contend that not spoofing the device's IMEI number is a major privacy problem. IMEI numbers do not change, so any existing record of the IMEI can be used to associate a new application session with known tracking data. In the comments on his initial announcement, however, Kondik described IMEI spoofing as an anonymity issue, making it out of scope for the feature.

Others were chafed about the all-or-nothing feature, particularly with the fact that GPS access was lumped in with personal data. The stated goal of the feature is to clamp down an overzealous app that requests access to personal information it does not really require to operate, and there are evidently a number of location-based apps in that category. Denying them access to personal data makes sense, but they are rendered useless without GPS access. Kuber Koos noted in the initial announcement discussion:

In the July 3 update discussion, developer Lars Greiss said that he was working on an "advanced mode" option to enable finer grained control for those who need it. Exactly what form that will take (and how fine-grained it will be) has not yet been announced, but there is certainly hope that Privacy Guard will find the right balance by the time the next stable CM release arrives.

However fine-grained the eventual result is, there will no doubt always be some users who insist on even more detailed controls. There are alternatives to be found, such as OpenPDroid, which modifies a number of Android services and libraries to provide a generalized privacy framework. OpenPDroid, for example, can respond to an app's data requests with empty data, fake data, or random data, on the grounds that for any particular app, one of those options might offer better protection than the others.

CM 7 had taken a fine-grained approach to privacy protection in its own way, too. It exposed selectable settings for individual permissions on each app, so that users could disable reading the contacts database while leaving location services unaffected. That feature was ditched in later releases, so it will be informative to see what the reaction is to the new, all-or-nothing Privacy Guard.

Of course, restricting access to the user data stored on a device is far from the only privacy issue facing the users of CM and other mobile platforms. Defeating Web-based user tracking requires other means, such as Tor. Other apps can be relayed through Tor with tools like Orbot. CM has begun working on its own implementation of SELinux, which should also help protect privacy by protecting against information leaks. Adding built-in support for something like Tor to CM would certainly be a boon to privacy fans, just like built-in support for OpenVPN.

But that is probably not a feature users should expect any time soon, so the privacy-conscious will need to continue to download and manually install Tor—along with ad blockers, PGP, and the other individual privacy packages. CM's new Privacy Guard may not do everything, but the fact that it will be built in to future releases and offer a simple user interface almost assures that it will see more real-world usage than the alternatives.

Comments (9 posted)

Brief items

Security quotes of the week

— Jeremy Epstein ponders a Google Glass vulnerability

— Paresh Dave on VoIP attacks in the LA Times

— One choice in a spoof form for the UK's new internet filtering

— Bruce Schneier

Comments (none posted)

Feds put heat on Web firms for master encryption keys (CNET)

CNET is reporting that the US government has been requesting the private SSL/TLS keys of major internet firms. Without perfect forward secrecy (which is rarely used on today's internet), that would allow the US to decode HTTPS traffic—even retroactively. It's not clear which, if any, internet companies have turned over those keys. "It's not entirely clear whether federal surveillance law gives the U.S. government the authority to demand master encryption keys from Internet companies. 'That's an unanswered question,' said Jennifer Granick, director of civil liberties at Stanford University's Center for Internet and Society. 'We don't know whether you can be compelled to do that or not.'"

Comments (50 posted)

New vulnerabilities

chromium-browser: multiple vulnerabilities

Package(s):	chromium-browser	CVE #(s):	CVE-2013-2853 CVE-2013-2867 CVE-2013-2868 CVE-2013-2869 CVE-2013-2870 CVE-2013-2871 CVE-2013-2873 CVE-2013-2875 CVE-2013-2876 CVE-2013-2878 CVE-2013-2879 CVE-2013-2880
Created:	July 19, 2013	Updated:	July 24, 2013
Description:	From the Debian advisory: CVE-2013-2853: The HTTPS implementation does not ensure that headers are terminated by \r\n\r\n (carriage return, newline, carriage return, newline). CVE-2013-2867: Chrome does not properly prevent pop-under windows. CVE-2013-2868: common/extensions/sync_helper.cc proceeds with sync operations for NPAPI extensions without checking for a certain plugin permission setting. CVE-2013-2869: Denial of service (out-of-bounds read) via a crafted JPEG2000 image. CVE-2013-2870: Use-after-free vulnerability in network sockets. CVE-2013-2871: Use-after-free vulnerability in input handling. CVE-2013-2873: Use-after-free vulnerability in resource loading. CVE-2013-2875: Out-of-bounds read in SVG file handling. CVE-2013-2876: Chrome does not properly enforce restrictions on the capture of screenshots by extensions, which could lead to information disclosure from previous page visits. CVE-2013-2878: Out-of-bounds read in text handling. CVE-2013-2879: The circumstances in which a renderer process can be considered a trusted process for sign-in and subsequent sync operations were not propertly checked. CVE-2013-2880: The chrome 28 development team found various issues from internal fuzzing, audits, and other studies.
Alerts:	Debian DSA-2724-1 2013-07-18 Gentoo 201309-16 2013-09-24

Comments (none posted)

kde-workspace: multiple vulnerabilities

Package(s):	kde-workspace	CVE #(s):	CVE-2013-4132 CVE-2013-4133
Created:	July 18, 2013	Updated:	August 5, 2013
Description:	From the KDE bug report: If KDM uses raw crypt() authentication (or pw_encrypt() on a patched Shadow system; see: https://alioth.debian.org/tracker/index.php?func=detail&aid=314234 ), instead of higher-level authentication such as PAM, and that crypt() can return a NULL pointer (as glibc 2.17+ does when passed a DES/MD5 encrypted passwords on Linux systems in FIPS-140 mode), then attempting to login to such an account via KDM crashes the daemon. (CVE-2013-4132) From the KDE bug report: Blinking systray icons are causing X to leak memory and plasma-desktop is to blame In less than 24h it's using 100+ MB memory and the icon wasn't blinking most of the time. When the icon is not blinking then the used memory stays the same. As soon as icon starts to blink the memory usage in X also starts to grow. (CVE-2013-4133)
Alerts:	Fedora FEDORA-2013-13098 2013-07-18 openSUSE openSUSE-SU-2013:1253-1 2013-07-25 openSUSE openSUSE-SU-2013:1291-1 2013-08-02 Fedora FEDORA-2013-13518 2013-08-04 Mageia MGASA-2013-0269 2013-09-01

Comments (none posted)

kernel: denial of service

Package(s):	kernel	CVE #(s):	CVE-2013-4125
Created:	July 24, 2013	Updated:	July 29, 2013
Description:	From the CVE entry: The fib6_add_rt2node function in net/ipv6/ip6_fib.c in the IPv6 stack in the Linux kernel through 3.10.1 does not properly handle Router Advertisement (RA) messages in certain circumstances involving three routes that initially qualified for membership in an ECMP route set until a change occurred for one of the first two routes, which allows remote attackers to cause a denial of service (system crash) via a crafted sequence of messages.
Alerts:	Fedora FEDORA-2013-13536 2013-07-24 Fedora FEDORA-2013-13663 2013-07-26 Ubuntu USN-1935-1 2013-08-20 Ubuntu USN-1936-1 2013-08-20

Comments (none posted)

lldpad: make a hardened build

Package(s):	lldpad	CVE #(s):
Created:	July 22, 2013	Updated:	July 24, 2013
Description:	From the Fedora advisory: Make a proper hardened build of liblldp_clif.so.
Alerts:	Fedora FEDORA-2013-12193 2013-07-20

Comments (none posted)

moodle: multiple vulnerabilities

Package(s):	moodle	CVE #(s):	CVE-2013-2242 CVE-2013-2243 CVE-2013-2244 CVE-2013-2245 CVE-2013-2246
Created:	July 22, 2013	Updated:	July 31, 2013
Description:	From the Mageia advisory: Users were able to access a daemon-mode Chat activity in Moodle before 2.4.5 without the required capability (CVE-2013-2242). It was possible to determine answers from ID values in Lesson activity matching questions in Moodle before 2.4.5 (CVE-2013-2243). Conditional access rule values for user fields were able to contain unescaped HTML/JS that would be output to users in Moodle before 2.4.5 (CVE-2013-2244). When impersonating another user using RSS tokens in Moodle before 2.4.5, an error was displayed, but block information relevant to the person being impersonated was shown (CVE-2013-2245). The Feedback module in Moodle before 2.4.5 was showing personal information to users without the needed capability (CVE-2013-2246).
Alerts:	Mageia MGASA-2013-0217 2013-07-21 Fedora FEDORA-2013-12950 2013-07-23 Fedora FEDORA-2013-12964 2013-07-23 Fedora FEDORA-2013-13252 2013-07-30

Comments (none posted)

mysql: multiple vulnerabilities

Package(s):	mysql	CVE #(s):	CVE-2013-1861 CVE-2013-3802 CVE-2013-3804
Created:	July 23, 2013	Updated:	September 9, 2013
Description:	From the CVE entries: MariaDB 5.5.x before 5.5.30, 5.3.x before 5.3.13, 5.2.x before 5.2.15, and 5.1.x before 5.1.68, and Oracle MySQL 5.1.69 and earlier, 5.5.31 and earlier, and 5.6.11 and earlier allows remote attackers to cause a denial of service (crash) via a crafted geometry feature that specifies a large number of points, which is not properly handled when processing the binary representation of this feature, related to a numeric calculation error (CVE-2013-1861). Unspecified vulnerability in the MySQL Server component in Oracle MySQL 5.1.69 and earlier, 5.5.31 and earlier, and 5.6.11 and earlier allows remote authenticated users to affect availability via unknown vectors related to Full Text Search (CVE-2013-3802). Unspecified vulnerability in the MySQL Server component in Oracle MySQL 5.1.69 and earlier, 5.5.31 and earlier, and 5.6.11 and earlier allows remote authenticated users to affect availability via unknown vectors related to Server Optimizer (CVE-2013-3804).
Alerts:	Mandriva MDVSA-2013:197 2013-07-23 Ubuntu USN-1909-1 2013-07-25 openSUSE openSUSE-SU-2013:1335-1 2013-08-14 Gentoo 201308-06 2013-08-29 Gentoo GLSA 201308-06:02 2013-08-30 SUSE SUSE-SU-2013:1390-1 2013-08-30 openSUSE openSUSE-SU-2013:1410-1 2013-09-08

Comments (none posted)

npm: insecure temporary directory generation

Package(s):	npm	CVE #(s):	CVE-2013-4116
Created:	July 23, 2013	Updated:	July 24, 2013
Description:	From the Red Hat bugzilla: An insecure temporary directory generation / use flaw was found in the way NPM, Node.js Package Manager, used to generate location of the temporary folder to be used for tarballs expansion. A local attacker could use this flaw to conduct symbolic link attacks, possibly leading to their ability to overwrite arbitrary system file reachable with the privileges of the user performing the NPM archive expansion.
Alerts:	Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-11780 2013-07-23 Fedora FEDORA-2013-12908 2013-07-23

Comments (none posted)

openjpa: code execution

Package(s):	openjpa	CVE #(s):	CVE-2013-1768
Created:	July 22, 2013	Updated:	October 7, 2013
Description:	From the CVE entry: The BrokerFactory functionality in Apache OpenJPA 1.x before 1.2.3 and 2.x before 2.2.2 creates local executable JSP files containing logging trace data produced during deserialization of certain crafted OpenJPA objects, which makes it easier for remote attackers to execute arbitrary code by creating a serialized object and leveraging improperly secured server programs.
Alerts:	Fedora FEDORA-2013-12948 2013-07-22 Fedora FEDORA-2013-12960 2013-07-22 Fedora FEDORA-2013-12967 2013-07-22 Mageia MGASA-2013-0292 2013-10-05

Comments (none posted)

openstack-keystone: denial of service

Package(s):	openstack-keystone	CVE #(s):	CVE-2013-2014
Created:	July 22, 2013	Updated:	July 24, 2013
Description:	From the Red Hat bugzilla: Yaguang Tang reports: concurrent requests with large POST body can crash the keystone process. this can be used by Malicious and lead to DOS to Cloud Service Provider. The OpenStack project has confirmed: Concurrent Keystone POST requests with large body messages are held in memory without filtering or rate limiting, this can lead to resource exhaustion on the Keystone server.
Alerts:	Fedora FEDORA-2013-10467 2013-07-20

Comments (none posted)

owncloud: multiple vulnerabilities

Package(s):	owncloud	CVE #(s):
Created:	July 22, 2013	Updated:	August 5, 2013
Description:	From the owncloud changelog: Version 5.0.8 fixes: SECURITY: XSS vulnerability in “Share Interface” (oC-SA-2013-029) SECURITY: Authentication bypass in “user_webdavauth” (oC-SA-2013-030) Also fixed in version 4.5.13.
Alerts:	Mageia MGASA-2013-0220 2013-07-21 Fedora FEDORA-2013-13143 2013-08-04 Fedora FEDORA-2013-13164 2013-08-04 Mandriva MDVSA-2013:206 2013-08-05

Comments (none posted)

qemu-kvm: privilege escalation

Package(s):	qemu-kvm	CVE #(s):	CVE-2013-2231
Created:	July 23, 2013	Updated:	July 26, 2013
Description:	From the Red Hat advisory: An unquoted search path flaw was found in the way the QEMU Guest Agent service installation was performed on Windows. Depending on the permissions of the directories in the unquoted search path, a local, unprivileged user could use this flaw to have a binary of their choosing executed with SYSTEM privileges.
Alerts:	Red Hat RHSA-2013:1100-01 2013-07-22 Red Hat RHSA-2013:1101-01 2013-07-22 Oracle ELSA-2013-1100 2013-07-22 Scientific Linux SL-qemu-20130722 2013-07-22 CentOS CESA-2013:1100 2013-07-22 Mageia MGASA-2013-0235 2013-07-26

Comments (none posted)

squid: denial of service

Package(s):	squid	CVE #(s):	CVE-2013-4115
Created:	July 22, 2013	Updated:	September 16, 2013
Description:	From the Mageia advisory: Due to incorrect data validation Squid is vulnerable to a buffer overflow attack when processing specially crafted HTTP requests. This problem allows any trusted client or client script who can generate HTTP requests to trigger a buffer overflow in Squid, resulting in a termination of the Squid service
Alerts:	Mageia MGASA-2013-0227 2013-07-21 Mageia MGASA-2013-0228 2013-07-21 Mandriva MDVSA-2013:199 2013-07-25 Fedora FEDORA-2013-13468 2013-08-02 Fedora FEDORA-2013-13493 2013-08-02 openSUSE openSUSE-SU-2013:1435-1 2013-09-13 openSUSE openSUSE-SU-2013:1436-1 2013-09-13 openSUSE openSUSE-SU-2013:1441-1 2013-09-13 openSUSE openSUSE-SU-2013:1443-1 2013-09-13 openSUSE openSUSE-SU-2013:1444-1 2013-09-13 Gentoo 201309-22 2013-09-27

Comments (none posted)

virtualbox: denial of service

Package(s):	virtualbox	CVE #(s):	CVE-2013-3792
Created:	July 22, 2013	Updated:	July 24, 2013
Description:	From the Mageia advisory: Thomas Dreibholz has discovered a vulnerability in Oracle VirtualBox, which can be exploited by malicious, local users in a guest virtual machine to cause a DoS (Denial of Service). The vulnerability is caused due to an unspecified error and can be exploited to render the host network connection and the virtual machine instance unresponsive or locking the host by issuing e.g. the "tracepath" command. Successful exploitation requires the target virtual machine to be equipped with a paravirtualised network adapter (virtio-net).
Alerts:	Mageia MGASA-2013-0222 2013-07-21

Comments (none posted)

xlockmore: screen lock bypass

Package(s):	xlockmore	CVE #(s):	CVE-2013-4143
Created:	July 22, 2013	Updated:	July 31, 2013
Description:	From the Mageia advisory: xlockmore before 5.43 contains a security flaw related to potential NULL pointer dereferences when authenticating via glibc 2.17+'s crypt() function. Under certain conditions the NULL pointers can trigger a crash in xlockmore effectively bypassing the screen lock.
Alerts:	Mageia MGASA-2013-0225 2013-07-21 Fedora FEDORA-2013-13258 2013-07-30 Gentoo 201309-03 2013-09-02

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The current development kernel is 3.11-rc2, released on July 21. Linus says: "the O_TMPFILE flag that is new to 3.11 has been going through a few ABI/API cleanups (and a few fixes to the implementation too), but I think we're done now. So if you're interested in the concept of unnamed temporary files, go ahead and test it out. The lack of name not only gets rid of races/complications with filename generation, it can make the whole thing more efficient since you don't have the directory operations that can cause serializing IO etc."

Stable updates: 3.10.2, 3.9.11, 3.4.54, and 3.0.87 were released on July 21; 3.9.11 is the last of the 3.9 series.

As of this writing, 3.10.3 and 3.2.49 are in the review process; they can be expected sometime on or after July 25.

Comments (none posted)

Quotes of the week

— Roland Dreier

— Linus Torvalds

— Paul McKenney

Comments (none posted)

Kernel development news

Device tree troubles

Kernel developers working on the x86 architecture are spoiled; they develop for hardware that, for the most part, identifies itself when asked, with the result that it is usually easy to figure out how a specific machine is put together. Other architectures — most notably ARM — are rather messier in this regard, requiring the kernel to learn about the configuration of the hardware from somewhere other than the hardware itself. Once upon a time, hard-coded "board files" were used to build ARM-system-specific kernels; more recently, the device tree mechanism has emerged as the preferred way to describe a system to the kernel. A device tree file provides the enumeration information that the hardware itself does not, allowing the kernel to understand the configuration of the system it is running on. The device tree story is one of success, but, like many such stories, success is bringing on some growing pains.

A device tree "binding" is the specification of how a specific piece of hardware can be described in the device tree data structure. Most drivers meant to run on platforms where device trees are used include a documentation file describing that driver's bindings; see Documentation/devicetree/bindings/net/can/cc770.txt as a randomly chosen example. The kernel contains nearly 800 such files, plus a hundreds more ".dts" files describing complete system-on-chips and boards, and the number is growing rapidly.

Maintenance of those files is proving to be difficult for a number of reasons, but the core of the problem can be understood by realizing that a device tree binding is a sort of API that has been exposed by the kernel to the world. If a driver's bindings change in an incompatible way, newer kernels may fail to boot on systems with older device trees. Since the device tree is often buried in the system's firmware somewhere, this kind of problem can be hard to fix. But, even when the fix is easy, the kernel's normal API rules should apply; newer kernels should not break on systems where older kernels work.

The clear implication is that new device tree bindings need to be reviewed with care. Any new bindings should adhere to existing conventions, they should describe the hardware completely, and they should be supportable into the future. And this is where the difficulties show up, in a couple of different forms: (1) most subsystem maintainers are not device tree experts, and thus are not well equipped to review new bindings, and (2) the maintainers who are experts in this area are overworked and having a hard time keeping up.

The first problem was the subject of a request for a Kernel Summit discussion with the goal of educating subsystem maintainers on the best practices for device tree bindings. One might think that a well-written document would suffice for this purpose, but, unfortunately, these best practices still seem to be in the "I know it when I see it" phase of codification; as Mark Brown put it:

Said mailing list tends to be overflowing with driver postings, though, making it less useful than one might like. Meanwhile, the best guidance, perhaps, came from David Woodhouse:

That is, evidently, not always the case, currently; some device tree bindings can be strongly tied to specific kernel versions. Such bindings will be a maintenance problem in the long term.

Keeping poorly-designed bindings out of the mainline is the responsibility of the device tree maintainers, but, as Grant Likely (formerly one of those maintainers) put it, this maintainership "simply isn't working right now." Grant, along with Rob Herring, is unable to keep up with the stream of new bindings (over 100 of which appeared in 3.11), so a lot of substandard bindings are finding their way in. To address this problem, Grant has announced a "refactoring" of how device tree maintainership works.

The first part of that refactoring is Grant's own resignation, with lack of time given as the reason. In his place, four new maintainers (Pawel Moll, Mark Rutland, Stephen Warren and Ian Campbell) have been named as being willing to join Rob and take responsibility for device tree bindings; others with an interest in this area are encouraged to join this group.

The next step will be for this group to figure out how device tree maintenance will actually work; as Grant noted, "There is not yet any process for binding maintainership." For example, should there be a separate repository for device tree bindings (which would make review easier), or should they continue to be merged through the relevant subsystem trees (keeping the code and the bindings together)? It will take some time, and possibly a Kernel Summit discussion, to figure out a proper mechanism for the sustainable maintenance of device tree bindings.

Some other changes are in the works. The kernel currently contains hundreds of .dts files providing complete device trees for specific systems; there are also many .dtsi files describing subsystems that can be included into a complete device tree. In the short term, there are plans to design a schema that can be used to formally describe device tree bindings; the device tree compiler utility (dtc) will then be able to verify that a given device tree file adheres to the schema. In the longer term, those device tree files are likely to move out of the kernel entirely (though the binding documentation for specific devices will almost certainly remain).

All told, the difficulties with device trees do not appear to be anything other than normal growing pains. A facility that was once only used for a handful of PowerPC machines (in the Linux context, anyway) is rapidly expanding to cover a sprawling architecture that is in wide use. Some challenges are to be expected in a situation like that. With luck and a fair amount of work, a better set of processes and guidelines for device tree bindings will result from the discussion — eventually.

Comments (21 posted)

The exfiltrated exFAT driver

The exFAT filesystem is a Microsoft product, designed for flash media. It lacks support in the Linux kernel; as a proprietary, heavily patented filesystem, it is not the sort of thing one would expect to see free support for. Still, when the exfat-nofuse repository showed up on GitHub, some dared to hope that Linux would gain exFAT support after all. Instead, what we appear to have gained is an ugly licensing mess and code that is best avoided.

From what can be determined by looking at the repository, the code appears to work. It was originally written by Samsung, it seems, and was shipped with one or more Android devices. The problem is that, as far as anybody can tell, Samsung never intended to distribute this code under the GPL. Instead, a GitHub user who goes by "rxrz" somehow came by a copy of the code, removed the original proprietary licensing headers, and inserted a GPL license declaration into the code. The code claimed to have a GPL license, but the copyright owner never released the code under that license.

On July 9, another GitHub user filed a bug noting that the license declaration was incorrect and suggesting a removal of the repository. The entity known as rxrz was not impressed, though, saying:

The code has since been edited to remove the GPL declaration and restore the proprietary license, but it remains available on GitHub and rxrz evidently feels that nothing wrong was done by posting it there. It also appears that GitHub has no interest in pulling down the repository in the absence of an explicit takedown notice from Samsung, so this "leaked" driver may remain available for some time.

This whole episode seems like a fairly straightforward case of somebody trying to liberate proprietary code by any means available. There are some interesting questions raised by all of this, though. The first of those is: what if somebody had tried to merge this code into the mainline kernel? The immediate answer is that they would have been chased off the list once developers actually had a look at the code, which, to put it gently, does not much resemble Linux kernel code. In the absence of this obvious barrier, one can hope that our normal review mechanisms would have kept this code from being merged until the developer was able to provide a satisfactory explanation of where it came from.

But it is not clear that all of our code is reviewed to that level, so it is hard to be sure. An exFAT implementation is likely to attract enough attention to ensure that the right questions are asked. Had the code in question been a driver for a relatively obscure piece of hardware, instead, it might not have been looked at very closely.

Then, one might ask: why is Samsung shipping this as a proprietary module in the first place? After all, Samsung appears to have figured out how Linux kernel development works and has made a solid place for itself as one of the largest contributors to the kernel. One can only guess at the answer, but it likely has to do with claims that Microsoft makes over the exFAT format. Microsoft has shown itself to be willing to assert patents on filesystem formats, so taking some care with an implementation of a new Microsoft filesystem format would seem like an exercise in basic prudence. Whether this exercise led to ignoring the GPL in an imprudent manner is the subject of another debate entirely.

Similarly, some prudence would be advisable for anybody thinking to use the code as a reverse-engineering tool for a new exFAT implementation. It is hard to reverse-engineer one's way around patent problems. exFAT may well be a format that is best left alone.

Finally, for those who have been in this community for a long time, the attitude revealed by a number of participants in the GitHub issue thread may be surprising. Licensing, GPL or otherwise, appears not to matter to many of these people. All that matters is that the code can be downloaded and that it works. This attitude can be found elsewhere on GitHub; indeed, many have complained that GitHub itself seems to be indifferent at best to the licensing of the code it distributes.

Perhaps we are heading into some sort of post-copyright era where licensing truly no longer matters. But it would not be surprising if those who are interested in copyright resist that future for a while yet. We are not just talking about the entertainment industry here; the simple fact of the matter is that anybody who values the provisions of the GPL is indeed interested in copyright. It is hard to demand respect for the GPL while refusing to respect the terms of other licenses.

Among other things, that means that the kernel community must continue to be careful not to incorporate code that has not been contributed under a suitable license. So code that shows up on the net must be looked at carefully, no matter how useful it appears to be. In this case, there was no danger that the exFAT code would ever be merged; nobody even suggested that it should be. But there will be other modules of dubious provenance in the future, some of which may seem more legitimate at first glance. Even then, though, our processes should be good enough to find the problems and avoid a merger that we will later regret. Hopefully.

(Thanks to Armijn Hemel for the heads-up).

Comments (53 posted)

What's missing from our changelogs

Tens of thousands of changes make their way into the mainline kernel every year. For most of those changes, the original motivation for the work is quickly forgotten; all that remains is the code itself and the changelog that goes with it. For this reason, kernel maintainers tend to insist on high-quality changelogs; as Linus recently put it, "We have a policy of good commit messages in the kernel." Andrew Morton also famously pushes developers to document the reasons explaining why a patch was written, including the user-visible effects of any bugs fixed. Kernel developers do not like having to reverse engineer the intent of a patch years after the fact.

With that context in mind, and having just worked through another merge window's worth of patches, your editor started wondering if our changelogs were always as good as they should be. A bit of scripting later, a picture of sorts has emerged; as one might expect, the results were not always entirely encouraging.

Changelogs

A patch's changelog is divided into three parts: a one-line summary, a detailed change explanation, and a tags section. For the most trivial patches, the one-line summary might suffice; there is not much to add to "add missing include of foo.h", for example. For anything else, one would expect a bit more text describing what is going on. So patches with empty explanation sections should be relatively rare.

As of this writing, just under 70,000 non-merge changesets have been pulled into the mainline repository since the release of the 3.5 kernel on July 21, 2012. Of those, 6,306 had empty explanations — 9% of the total. Many of them were as trivial as one might expect, but others were rather less so.

Some developers are rather more laconic than others. In the period since 3.5, the developers most inclined to omit explanations were:

Developer	Count
Al Viro	570
Ben Skeggs	224
Mark Brown	213
Hans Verkuil	204
Andreas Gruenbacher	143
Axel Lin	130
Philipp Reisner	126
Antti Palosaari	118
James Smart	107
Alex Deucher	85
Laurent Pinchart	84
Kuninori Morimoto	75
Eric W. Biederman	75
Pavel Shilovsky	72
Rafał Miłecki	72
David S. Miller	65
David Howells	61
Peter Meerwald	61
Maxime Ripard	55
YOSHIFUJI Hideaki	51

For the curious, a page listing the no-explanation patches merged by the above developers is available. A quick look shows that a lot of patches with empty explanations find their way into the core virtual filesystem layer; many of the rest affect graphics drivers, audio drivers, Video4Linux drivers, and the DRBD subsystem. But they can be found anywhere; of the 1,065 changes that touched the mm/ subdirectory, 46 lacked an explanation, for example.

If one believes that there should be fewer patches with empty explanations going into the kernel, one might be inclined to push subsystem maintainers to be a bit more demanding in this regard. But, interestingly, it has become much harder to determine which maintainers have had a hand in directing patches into the kernel.

Signoffs

The Signed-off-by line in the tags section is meant to document the provenance of patches headed into the mainline. When a developer submits a patch, the changelog should contain a signoff certifying that the patch can properly be contributed to the kernel under a GPL-compatible license. Additionally, maintainers who accept patches add their own signoffs documenting that they handled the patch and that they believe it is appropriate for submission to the mainline. In theory, by following the sequence of Signed-off-by lines, it is possible to determine the path that any change followed to get to Linus's tree.

The truth is a little bit more complicated than that. To begin with, of the changes merged since 3.5, 79 had no signoffs at all. Roughly half of those were commits by Linus changing the version number; he does not apply a signoff to such changes, even for those that contain added data beyond the version number update. The rest are all almost certainly mistakes; a handful are the result of obvious formatting errors. See the full list for details. The mistakes are innocent, but they do show a failure of a process which is supposed to disallow patches that have not been signed off by their authors.

Arguably, there is another class of patches that is more interesting: those that contain a single Signed-off-by line. Such patches have, in theory, been managed by a single developer who wrote the patch and got it into the mainline unassisted. One might think that only Linus is in a position to do any such thing; how could anybody else get a change into the mainline on their own?

In fact, of the 70,000 patches pulled into the mainline during the period under discussion, 16,651 had a single signoff line. Of those, 11,527 (16% of the total) had no other tags, like Acked-by, Reviewed-by, or Tested-by, that would indicate attention from at least one other developer. For the purposes of this discussion, only the smaller set of patches has been considered. The most frequent committers of single-signoff patches are:

Developer	Count
Al Viro	891
Takashi Iwai	525
Mark Brown	492
Johannes Berg	414
Alex Deucher	391
Mauro Carvalho Chehab	389
Ben Skeggs	362
Greg Kroah-Hartman	292
Trond Myklebust	279
David S. Miller	264
Felipe Balbi	259
Tomi Valkeinen	258
Arnaldo Carvalho de Melo	172
Eric W. Biederman	147
Josef Bacik	145
Shawn Guo	142
J. Bruce Fields	141
Ralf Baechle	132
Arnd Bergmann	131
Samuel Ortiz	129

(See this page for a list of the single-signoff patches merged by the above developers). These results are, of course, a result of the use of git combined with the no-rebasing rule. Once a patch has been committed to a public repository, it becomes immutable and can never again acquire tags like Signed-off-by. To pick one example from the list above, wireless developer Johannes Berg maintains his own tree for mac80211 changes; when he commits a patch, it will carry his signoff. Changes flow from that tree to John Linville's wireless tree, then to Dave Miller's networking tree, and finally to the mainline repository. Since each of those moves is done with a Git "pull" operation, no additional signoffs will be attached to any of those patches; they will arrive in the mainline with a single signoff.

One might contend that patches become less subject to review once they enter the Git stream; they can be pulled from one repository to the next sight-unseen. Indeed, early in the BitKeeper era, developers worried that pull requests would be used to slip unreviewed patches into the mainline kernel. Single-signoff patches might be an indication that this is happening. And, indeed, important patches like the addition of the O_TMPFILE option went to the mainline with, as far as your editor can tell, no public posting or review (and no explanation in the changelog, for that matter). It also seems plausible that single-signoff patches merged into the sound subsystem or the Radeon driver (to name a couple of examples) have not been reviewed in detail by anybody other than the author; there just aren't that many people with the interest and skills to review that code.

Without a chain of signoff lines, we lose more than a picture of which maintainers might have reviewed the patches; we also lose track of the path by which a patch finds its way into the mainline. A given changeset may pass through a number of repositories, but those passages leave no mark on the changeset itself. Sometimes that path can be worked out from the mainline repository history, but doing so can be harder than one might imagine, even in the absence of "fast-forward merges" and other actions that obscure that history. Given that the Signed-off-by line was introduced to document how patches get into the kernel, the loss of this information may be a reason for concern.

The kernel community prides itself on its solid foundation of good procedures, including complete changelogs and a reliable signoff chain. Most of the time, that pride is entirely justified. But, perhaps, there might be room for improvement here and there — that is unsurprising when one considers that no project that merges 70,000 changes in a year can be expected to do a perfect job with every one of them. Where there is imperfection, there is room for improvement — though improving the signoff chain will be difficult as long as the tools do not allow it. But even a bit more verbiage in commit messages would be appreciated by those of us who read the patch stream.

Comments (47 posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.11-rc2 . (July 22, 2013)

• Greg KH: Linux 3.10.2 . (July 22, 2013)

• Greg KH: Linux 3.9.11 . (July 22, 2013)

• Kamal Mostafa: Linux 3.8.13.5 . (July 23, 2013)

• Luis Henriques: Linux 3.5.7.17 . (July 23, 2013)

• Greg KH: Linux 3.4.54 . (July 22, 2013)

• Greg KH: Linux 3.0.87 . (July 22, 2013)

Core kernel code

• Raghavendra K T: Paravirtualized ticket spinlocks . (July 22, 2013)

• Waiman Long: qrwlock: Introducing a queue read/write lock implementation . (July 24, 2013)

Development tools

• Tom Zanussi: tracing: trace event triggers . (July 19, 2013)

• Dave Jones: Trinity v1.2 . (July 23, 2013)

Device drivers

• Rafael J. Wysocki: ACPI / hotplug / PCI: Major rework + Thunderbolt workarounds . (July 18, 2013)

• Kishon Vijay Abraham I: PHY framework . (July 18, 2013)

• Sricharan R: Add crossbar driver . (July 18, 2013)

• hongbo.zhang@freescale.com: DMA: Freescale: Add support for 8-channel DMA engine . (July 22, 2013)

• Ivan T. Ivanov: tty: serial: Add initial MSM UART High Speed Lite driver . (July 22, 2013)

• Oleksandr Kozaruk: TWL6030, TWL6032 GPADC driver . (July 22, 2013)

• Keith Packard: drm: Asynchronouse page flipping interface and Intel implementation . (July 23, 2013)

• David Herrmann: Unified VMA Offset Manager v3 . (July 23, 2013)

• Philipp Zabel: Add Dialog DA9063 core and regulator drivers . (July 24, 2013)

• Sylwester Nawrocki: V4L: Add driver for Samsung S5K5BAF camera sensor . (July 24, 2013)

Filesystems and block I/O

• Phillip Lougher: Squashfs: add LZ4 compression . (July 22, 2013)

• J. Bruce Fields: Implement NFSv4 delegations, take 9 . (July 23, 2013)

Memory management

• Naoya Horiguchi: extend hugepage migration . (July 18, 2013)

• Tang Chen: Arrange hotpluggable memory as ZONE_MOVABLE. . (July 19, 2013)

• Johannes Weiner: mm: improve page aging fairness between zones/nodes . (July 23, 2013)

Networking

• Simon Horman: MPLS actions and matches . (July 23, 2013)

• Eric Dumazet: tcp: TCP_NOTSENT_LOWAT socket option . (July 23, 2013)

Architecture-specific

• Deepthi Dharwar: cpuidle: (powernv) cpuidle driver. . (July 23, 2013)

• Nicolas Pitre: The big.LITTLE In-Kernel Switcher (IKS), part 1 . (July 23, 2013)

• Maxime Ripard: ARM: sunxi: Add support for the Allwinner A31 SoC . (July 24, 2013)

Security-related

• John Johansen: apparmor: add policy interface and convert to RCU . (July 23, 2013)

• Mimi Zohar: ima: larger digests and template support . (July 23, 2013)

Miscellaneous

• Frederic Weisbecker: perf: Finer grained full dynticks handling . (July 23, 2013)

Page editor: Jonathan Corbet

Distributions

A look at Mer

The Mer project was born out of the ashes of the MeeGo mobile-oriented distribution in 2011. In an admittedly provocatively titled talk ("Mer & Qt : What MeeGo Should Have Been!"), Mer advisory board member David Greaves gave an overview of the distribution project, some of its history, and its plans. "MeeGo is dead", he said, but it did a lot of good; Mer is one of the good things that came out of it.

We owe thanks to both Intel and Nokia for MeeGo, Greaves said. MeeGo didn't succeed in what it set out to do, but it did succeed in other ways. Mer is part of the "story of what happened after the big guns left" the MeeGo project. Mer is being used by numerous projects including Jolla/Sailfish OS, Vivaldi, in-vehicle-infotainment (IVI) systems, and others. The project is proud of what it has achieved so far, but would like to see more people using it.

What is Mer?

Mer is targeted at phones ("of course"), but also TVs, cars, tablets, and even appliances like refrigerators. More and more small businesses are prototyping devices on hardware like the Raspberry Pi, and many of them are doing so using Qt or Qt-based frameworks. Mer fits well in that world.

As part of the formative process of Mer, Greaves and others looked at "what MeeGo did right" and concluded that it was the focus on "making it easy to make devices" that was a major selling point. But, MeeGo was monolithic and had lots of packages. Mer needed to break that up and make a distribution that was non-monolithic: a small core that would make a good foundation for mobile and smaller devices.

Mer has had "good success" in attracting vendor interest, Greaves said. Qt and QML (the JavaScript-based language for building Qt user interface elements) are at the heart of Mer. Quality is another important aspect of the Mer project and various automated testing processes are used to help ensure that the distribution works well.

But Mer is not a "user experience" (UX) and has no user interface as part of the core. There is a "splash screen" that exercises GLES, but that's it. Mer is roughly 300 packages, down from MeeGo's 1500, which means there is less to test. It also makes choices among competing packages, reducing the number of alternatives for vendors—at least in the core. Device vendors can change to different technologies if they wish, though. Greaves noted that Plasma Active (the KDE-based tablet UX) uses NetworkManager, rather than ConnMan from the core.

Various UXes have been added atop Mer for different device niches including Nemo for mobile, Plasma Active and Vivaldi for tablets, TVOS (from China) for TVs, Lincor for healthcare, Nomovok for IVI, and more. It supports x86 (both 32 and 64 bit), ARM, and MIPS. It is available for multiple platforms as well, including Nokia N950/N900/N9, ExoPC, Raspberry Pi, PandaBoard, and BeagleBoard.

Mer makes sense for a company on a couple of levels, but some management persuasion may still be required to make the switch. Greaves said. It will allow the company to operate efficiently and deliver products quickly. Companies can use closed code if they so desire, as well. There is also a low barrier to entry to the Mer community. There is no consortium or "joining required"; basically all it takes is to come to the IRC channel and start talking with the community, he said.

The Mer project is pragmatic and operates entirely in the open, Greaves said. It is a meritocratic community that aims to be inclusive. The management layer for the project is thin, most of the work goes into technical efforts to make Mer better.

Mer is a core part of the Sailfish OS mobile operating system from Jolla. That means there is an open source project at the heart of Sailfish OS, he said. He likened the Mer ecosystem to that of Qt, which has multiple companies and organizations, including Digia, KDE, KDAB, and others, all working together to make Qt better.

The Mer core has a "whole load of stuff" in it, but it is really just the minimal set of packages to get Qt running. But "code is not enough"; there needs to be build, QA, and collaboration systems, as well as documentation, support, and information on "best practices". So, beyond just the code, Mer offers a number of other services, including Open Build Service (OBS) systems, a Business Operations Support System (BOSS) based on ruote, and cross-building support based on Scratchbox 2 that is integrated with OBS.

Greaves then turned to libhybris, which was created by Mer founder Carsten Munk. It came about after a meeting in Finland where there was "quite a bit" of drinking—and talking. There is a basic mismatch between Android libraries, which use the Bionic C library, and systems that use the GNU C library (glibc). So Android graphics drivers can't be used by Qt, for example. The meeting participants were not convinced that there was no way to bridge that gap between the two C libraries, and Munk put together a library to get glibc programs working with Bionic-based libraries: libhybris. Canonical is using libhybris in devices, he said; it allows companies to take a board that only has Android graphics drivers and run Qt (and Mer) on it.

The SDK

There is a Mer SDK that allows developers on multiple platforms (Windows, Mac OS X, and Linux) to build Mer itself as well as apps for Mer. It can also be used to develop a full user interface for a device. The Mer SDK consists of two parts: a Platform SDK and plugins for Qt Creator to communicate with the Platform SDK. Jolla has supported a lot of the work on the SDK, he said.

The Platform SDK runs in a virtual machine using VirtualBox. Instead of creating a version for each distribution and operating system, it is a self-hosting platform. It could also be done using LXC or chroot() and namespaces, but it is using VirtualBox for now. It has all of the low level packages for building and testing that are needed to build Mer packages for a specific target.

On top of the low-level pieces, there is a Qt Creator–based interface that includes Mer-specific plugins to talk to the Platform SDK. Those plugins control emulators and devices, which is not normally a part of Qt development. There are also plugins to handle packaging metadata and to interface with app stores. The Mer project will be working with the Qt project to get those pieces upstream, he said.

The Platform SDK plus the Qt Creator plugins make an overall Mer SDK, which is a starting point for a vendor SDK. For example, Jolla has added Sailfish OS targets to turn it into a Sailfish SDK. It is designed to be extensible and to be used for different targets, such as MIPS, ARM, or x86.

Qt Creator runs natively on the developer's system, but interfaces with the Platform SDK via the plugins. Building the code is done the same way, whether it is initiated from Qt Creator or OBS. VirtualBox shared folders are used between the workstation and the Platform SDK, so a build from Qt Creator does an ssh into the virtual machine and executes make there. Apps can be deployed that way as well, so that developers don't have to build and install a package each time they want to test on an emulator or a real device.

Plans

So there is a build engine, but what is needed now are more emulators and targets, Greaves said. Nemo, Plasma Active, and Raspberry Pi are all good candidates. Better emulation support is also on the roadmap, including simulating hardware events in the kernel. A move from VirtualBox to LXC is also in the cards, as is better debugging support.

As with most projects, Mer needs help, so Greaves issued a "call to action". He noted that Mer is a vendor community, but that using the term "cooperative" may be more successful when talking to management. There are a number of benefits to working with Mer, but the biggest is that those working on the project can influence its development direction. There are multiple choices for mobile development, including Windows Mobile, iOS, Android, and Tizen, but none of those will give a device maker the freedom that it wants, he said. Even though Android is open source, how many have actually gotten patches accepted into it? For that reason alone, Mer should be very appealing to smaller device vendors.

[Thanks to KDE e.V. for travel assistance to Bilbao for Akademy.]

Comments (4 posted)

Brief items

Distribution quotes of the week

-- Josselin Mouette

-- Jaroslav Reznik

Comments (2 posted)

Android 4.3

Google has announced the existence of Android 4.3. It includes a host of new features, some of which may be more welcome than others. "To meet the needs of the next generation of media services, Android 4.3 introduces a modular DRM framework that enables media application developers to more easily integrate DRM into their own streaming protocols, such as MPEG DASH (Dynamic Adaptive Streaming over HTTP, ISO/IEC 23009-1)." There is no word on general availability or when this code will be available through the Android Open Source Project.

Comments (112 posted)

CyanogenMod to incorporate SELinux

The CyanogenMod project has announced that it will be incorporating the work of the SEAndroid project to add SELinux to its firmware images. "We will be working on this policy creation in parallel to Google’s own policies for Android, which we believe will be released with the Android 4.3 source; effectively getting us ahead of the eventual 4.3 source release." For now, it will be running in permissive mode, with the hope that users will submit logs to allow tweaking of the policies.

Comments (none posted)

Distribution News

Fedora

Fedora ARM provisionally set as a primary architecture

The July 18 LWN Weekly Edition includes coverage of the discussion on whether the Fedora project should recognize ARM as one of its primary architectures. While that article was being written, the Fedora engineering steering committee was discussing the issue; the conclusion they came to was: "build ARM on primary infrastructure. whether it is released as a primary Fedora 20 or as 'Fedora 20 for ARM' depends on how well it fulfills release criteria and functionality criteria closer to release time" So ARM has not reached "primary architecture" status yet, but things are clearly headed in that direction.

Full Story (comments: 2)

Ubuntu family

Ubuntu Forums account information breached

The Ubuntu Forums site is currently displaying a "down for maintenance" message that describes a security breach of the accounts database, which revealed all usernames, passwords, and email addresses to the attacker(s). Ubuntu One, Launchpad, and other Canonical web sites are not affected by the breach. "The passwords are not stored in plain text. However, if you were using the same password as your Ubuntu Forums one on another service (such as email), you are strongly encouraged to change the password on the other service ASAP."

Comments (34 posted)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (July 22)
• DistroWatch Weekly, Issue 517 (July 22)
• Maemo Weekly News (July 22)
• Ubuntu Weekly Newsletter, Issue 326 (July 21)

Comments (none posted)

Mageia 4 already! (Mageia Blog)

The Mageia Blog covers the schedule for Mageia 4. "After some weeks of (interesting) discussions, we went though the features process to define the official list for Mageia 4. The choice was based on the existing policy. As usual, we received a lot of proposals."

Comments (1 posted)

Miller: Proposal for a more agile "Fedora.next"

Fedora cloud architect Matthew Miller has posted some notes describing his vision for a more successful and widely used Fedora distribution. "This is a vision for the future. The Fedora Core idea can get started now, and the ideas beyond that are for development over a longer term. That doesn't mean never, and there are some specific things at the higher level to get started on now, but I'm not suggesting to change everything all crazily. We're not throwing out what have."

Full Story (comments: 14)

Slackware Turns 20 (OStatic)

Susan Linton looks at the history of Slackware. "As I look back over my history with Slack, I'm struck by how many distributions were once based on Slackware. Most are no longer maintained, but some names may still be familiar. GoblinX was a strange looking but quite stable and fun distribution. It's biggest issue in adoption is their pay-to-play business model that often fails in Linuxville. Austrumi is a tiny distro from Latvia, a tiny Northern European country most Americans' education didn't include. It was fast and stable and looks to be abandoned. Ultima 4 was trying to provide an easy to use Slackware and Mutagenix was a really cool distro that has disappeared off the face of the Earth. But Slackware is still here. There are many more derivative epitaphs, but the oldest surviving Linux distribution is 20 years old and is still very actively and enthusiastically maintained."

Comments (1 posted)

Page editor: Rebecca Sobol

Development

Kolab 3.1 adds WebDAV

Test builds for version 3.1 of the Kolab groupware suite have been released. As before, the system provides a solid mail, address book, and calendaring solution with both web-based and desktop clients available. But this release departs from the project's traditional reliance on IMAP and adds support for a number of new services built on top of WebDAV. The change has the potential to simplify deploying Kolab for many users and offices.

Kolab is a client-server groupware system. The data is stored and managed on the server and there are web front-ends available (such as the Roundcube webmail client), but the emphasis has long been on supporting outside desktop clients as well. Historically, Kolab used special IMAP folders to store contacts and calendar entries on the server, in addition to the email messages for which IMAP was designed. The project defined its own XML-based storage format that encapsulates other data types into MIME messages on the server side. It covers iCalendar events, to-do tasks, journal entries, and free/busy statuses, plus text notes, vCard contacts, and shared "resources" (e.g., conference rooms).

The Roundcube client that ships with Kolab includes plugins to utilize the non-email IMAP storage directly, and KDE's Kontact has long been the primary desktop application supported by Kolab. Support for the Kolab storage format also made its way into other groupware suites like Citadel. Many other client programs, however, required an add-on—such as SyncKolab for Mozilla Thunderbird or several proprietary connectors for Microsoft's Outlook. There are also several mobile clients for which Kolab connectors are available, although not all of them support the most recent version of the storage format, 3.0.

As one might guess, version 3.0 of the storage format was rolled out in conjunction with version 3.0 of Kolab itself, which was released in January 2013. Thus, the addition of an entirely new data access protocol for the 3.1 is all the more remarkable.

The project announced the plan in March, calling it both "the biggest and most exciting thing on our roadmap" as well as "the one people have asked us for most." Using the SabreDAV WebDAV library (in order to not "reinvent the wheel," according to the announcement), the project would add CalDAV and CardDAV access to scheduling and contact data. As the 3.1 alpha announcement explains, the old IMAP-based access protocols are not going away. The CalDAV and CardDAV functionality is implemented as a separate data access layer dubbed "iRony" by the project. The most obvious gain from the move is immediate access to Kolab server data from the sizeable assortment of existing CalDAV- and CardDAV-aware applications. The built-in Mac OS X calendar and address book fall into this category, but so do quite a few open source programs, including web-based, mobile, and desktop clients.

Kolab 3.1 is also adding support for general-purpose file storage, which will also be accessible over WebDAV. But the file storage functionality is not limited to WebDAV: the project has also implemented an IMAP-based file store, and it has worked with the ownCloud project to make an ownCloud server accessible as storage, too. The ownCloud-Kolab integration is not yet ready, but the Kolab project describes it as "trivial," as is integration with Dropbox or other cloud storage systems.

The 3.1 alphas also include the ability for multiple users to share mail folders. That is, not only can multiple user accounts read mail from a shared inbox (such as contact@example.com), but those accounts can also send messages as contact@example.com. Other changes incorporated include new administration features, like the ability to define a policy for how user account UIDs are generated, and support for managing multiple email domains within a single Kolab server.

The alpha release showcasing all of these new features is available from the project's Git repository or in packages for CentOS. Debian packages have been assembled in the past, but so far none are available for the 3.1 code.

Historically the term "groupware" tended to apply to information systems designed to support office environments, but it recent years that has changed. The public scandals about privacy breaches, surveillance, and user tracking that seem to plague every major web application service provider have served to raise the profile of open source groupware suites—increasingly, they are seen as a self-hosted alternative to relying on Google, Microsoft, and Apple for valuable web services, even if deployed only for a single user. The Kolab team is clearly aware of this shift, and cites the system's ability to serve as a private service beyond the reach of the NSA PRISM program.

Kolab has been in active development for just over ten years, and in that time it has earned a reputation for quality production in more ways than one. There are quite a few open source groupware products on the market, but many of them suffer from a lack of client compatibility, either working only with their bundled web front-end, or focusing on a small set of third-party client applications. Kolab has long offered support for a wide assortment of client endpoints, and implementing CalDAV and CardDAV access is likely to extend the reach of that assortment in a big way.

Support for these protocols is also a positive sign for CardDAV and CalDAV themselves, which in the past have been criticized as being overly complex (which, for example, gave rise to the now-abandoned GroupDAV protocol). Their slow adoption can be seen as part of a march toward compatibility among competing groupware projects, however. When all is said and done, it will be interesting to see whether this WebDAV support signals the start of a slow decline in support for Kolab's IMAP-based data protocols. But perhaps that would not be a tragic loss in the long run, as it might allow the project to expend less energy maintaining private protocols. The project does not like reinventing the wheel, after all, and as of Kolab 3.1 alpha, it is now responsible for maintaining two separate access protocols.

Comments (1 posted)

Brief items

Quotes of the week

— Ryan Paul

— Anil Dash

Comments (none posted)

Wine 1.6 released

Version 1.6 of the Wine Windows emulation system is out. "This release represents 16 months of development effort and around 10,000 individual changes. The main highlights are the new Mac driver, the full support for window transparency, and the new Mono package for .NET applications support."

Comments (15 posted)

Razor and LXDE-Qt merging

Jerome Leclanche has posted an announcement that the Razor and LXDE-Qt (LXDE ported from GTK to Qt) lightweight desktops will be merging under the LXDE-Qt banner. "So what happens now? Our two teams have met up and discussed the issues and we have decided that the best course of action for both projects is to focus on a single desktop environment, instead of two. There have been talks of "merging" ever since LXDE-Qt was announced. Having taken the decision to collaborate, we've all had the pleasure of working together already. Our plan is to cherry-pick the best parts of Razor and LXDE and include or port those to LXDE-Qt. Other components will be ported straight from GTK code or rewritten from scratch. In the end, we want to offer the best possible experience while reusing as much code as possible. It will not be an easy process and as always, we welcome anyone who wishes to help, be it with development, translations, or general feedback." In addition, Leclanche mentioned an offer to bring Razor (and now, perhaps, LXDE-Qt) under the KDE umbrella.

Comments (37 posted)

Apache OpenOffice 4.0 released

The Apache OpenOffice 4.0 release is now available. "OpenOffice 4.0 features a new, more modern user interface, improvements to Microsoft Office interoperability, enhanced graphics support and many other improvements." See the release notes for lots of details.

Comments (51 posted)

U-Boot 2013.07 available

Version 2013.07 of the U-Boot bootloader has been released. This version is notable for adding support for doing cryptographically verified boot. The announcement adds: "There's still work to be done to make it easier to tie things to a TPM (it's just scriptable today), but hey, there it is. One might even say this is one of those technical features that makes companies chose something other than U-Boot.."

Full Story (comments: none)

Newsletters and articles

Development newsletters from the past week

• Caml Weekly News (July 23)
• What's cooking in git.git (July 18)
• What's cooking in git.git (July 21)
• What's cooking in git.git (July 22)
• Haskell Weekly News (July 17)
• OpenStack Community Weekly Newsletter (July 19)
• Perl Weekly (July 22)
• PostgreSQL Weekly News (July 22)
• R Journal (June)
• Ruby Weekly (July 17)
• Tor Weekly News (July 24)

Comments (none posted)

Airlie: Introducing Virgil - 3D virtual GPU for qemu

Dave Airlie has posted an introduction to Virgil, a new 3D virtual GPU project for the QEMU emulator, for which he is seeking developer input. The goal, he says, is "a 3D capable virtual GPU for qemu that can be used by Linux and eventually Windows guests to provide OpenGL/Direct3D support inside the guest. It uses an interface based on Gallium/TGSI along with virtio to communicate between guest and host, and it goal is to provided an OpenGL renderer along with a complete Linux driver stack for the guest." Airlie also goes out of his way to be clear that he is recruiting developers, not users, noting "I can't stress this strongly enough, this isn't end user ready, not even close."

Comments (24 posted)

Xiph.org: Introducing Daala, part 2

Xiph.org has published part 2 of its introduction to the new Daala video codec. This edition covers frequency domain intra-prediction. "Daala predicts frequency coefficients in the frequency domain, rather than pixels in the spatial domain. Frequency coefficients are not shared between blocks, allowing us to keep the classic definition of 'neighboring' as well as avoiding any circular decoding dependencies. In addition, Daala's frequency-domain prediction draws from a larger input data set consisting of up to four complete blocks. In theory, the additional degrees of freedom could give us a more efficient predictor."

Comments (none posted)

Page editor: Nathan Willis

Announcements

Brief items

The H is shutting down

The H has announced its closure. "Although The H has produced many widely read stories, it has not been possible to effectively monetise that traffic to produce a working business model." Here at LWN we understand that problem well. The H has indeed produced a lot of quality content over the years; we are genuinely sad to see it go.

Comments (41 posted)

HeliOS Project announces fundraising drive

Reglue, the non-profit organization behind the HeliOS Project that provides Linux PCs to Texas schoolchildren, has announced a fundraising drive. Although HeliOS has always been donation-driven, the project is hoping to make up for the funding lost over the past year as founder Ken Starks was unable to work while undergoing treatment for cancer.

Comments (none posted)

OSI Welcomes Member-Elected Director

The Open Source Initiative (OSI) announced the election of Richard Fontana to OSI's first individual-member–elected seat on the board of directors. OSI is in the process of transitioning from an appointed board to one composed of elected members.

Comments (none posted)

Articles of interest

Bacon: Announcing the Ubuntu Edge

Jono Bacon announces that Canonical will be building and shipping a "fully converged" device called the "Ubuntu Edge" — all funded by a $32 million Indiegogo campaign. "The Ubuntu Edge will dual boot Ubuntu and Android, and will transform into a PC when docked with a monitor, with the full Ubuntu desktop and shared access to all the phone’s files. For this it needs the power of a PC, so Ubuntu Edge will be equipped with the latest, fastest processor, at least 4GB of RAM and a massive 128GB of storage."

Comments (101 posted)

New Books

The Practice of Network Security Monitoring--New from No Starch Press

No Starch Press has released "The Practice of Network Security Monitoring" by Richard Bejtlich.

Full Story (comments: none)

Calls for Presentations

PGConf.EU 2013 Call for papers extended

PGConf.EU will take place October 29-November 1 in Dublin, Ireland. The call for papers deadline has been extended until July 29.

Full Story (comments: none)

Linux Symposium Call for Papers

The Linux Symposium will take place November 18-21, 2013 in Ottawa, Canada. The call for papers deadline is September 1.

Comments (none posted)

CFP Deadlines: July 25, 2013 to September 23, 2013

The following listing of CFP deadlines is taken from the LWN.net CFP Calendar.

Deadline	Event Dates	Event	Location
July 25	October 22 October 23	GStreamer Conference	Edinburgh, UK
July 28	October 17 October 20	PyCon PL	Szczyrk, Poland
July 29	October 28 October 31	15th Real Time Linux Workshop	Lugano, Switzerland
July 29	October 29 November 1	PostgreSQL Conference Europe 2013	Dublin, Ireland
July 31	November 5 November 8	OpenStack Summit	Hong Kong, Hong Kong
July 31	October 24 October 25	Automotive Linux Summit Fall 2013	Edinburgh, UK
August 7	September 12 September 14	SmartDevCon	Katowice, Poland
August 15	August 22 August 25	GNU Hackers Meeting 2013	Paris, France
August 18	October 19	Hong Kong Open Source Conference 2013	Hong Kong, China
August 19	September 20 September 22	PyCon UK 2013	Coventry, UK
August 21	October 23	TracingSummit2013	Edinburgh, UK
August 22	September 25 September 27	LibreOffice Conference 2013	Milan, Italy
August 30	October 24 October 25	Xen Project Developer Summit	Edinburgh, UK
August 31	October 26 October 27	T-DOSE Conference 2013	Eindhoven, Netherlands
August 31	September 24 September 25	Kernel Recipes 2013	Paris, France
September 1	November 18 November 21	2013 Linux Symposium	Ottawa, Canada
September 6	October 4 October 5	Open Source Developers Conference France	Paris, France
September 15	November 8	PGConf.DE 2013	Oberhausen, Germany
September 15	November 15 November 16	Linux Informationstage Oldenburg	Oldenburg, Germany
September 15	October 3 October 4	PyConZA 2013	Cape Town, South Africa
September 15	November 22 November 24	Python Conference Spain 2013	Madrid, Spain
September 15	April 9 April 17	PyCon 2014	Montreal, Canada
September 15	February 1 February 2	FOSDEM 2014	Brussels, Belgium

If the CFP deadline for your event does not appear here, please tell us about it.

Upcoming Events

Events: July 25, 2013 to September 23, 2013

The following event listing is taken from the LWN.net Calendar.

Date(s)	Event	Location
July 22 July 26	OSCON 2013	Portland, OR, USA
July 27	OpenShift Origin Community Day	Mountain View, CA, USA
July 27 July 28	PyOhio 2013	Columbus, OH, USA
July 31 August 4	OHM2013: Observe Hack Make	Geestmerambacht, the Netherlands
August 1 August 8	GUADEC 2013	Brno, Czech Republic
August 3 August 4	COSCUP 2013	Taipei, Taiwan
August 6 August 8	Military Open Source Summit	Charleston, SC, USA
August 7 August 11	Wikimania	Hong Kong, China
August 9 August 11	XDA:DevCon 2013	Miami, FL, USA
August 9 August 12	Flock - Fedora Contributor Conference	Charleston, SC, USA
August 9 August 13	PyCon Canada	Toronto, Canada
August 11 August 18	DebConf13	Vaumarcus, Switzerland
August 12 August 14	YAPC::Europe 2013 “Future Perl”	Kiev, Ukraine
August 16 August 18	PyTexas 2013	College Station, TX, USA
August 22 August 25	GNU Hackers Meeting 2013	Paris, France
August 23 August 24	Barcamp GR	Grand Rapids, MI, USA
August 24 August 25	Free and Open Source Software Conference	St.Augustin, Germany
August 30 September 1	Pycon India 2013	Bangalore, India
September 3 September 5	GanetiCon	Athens, Greece
September 6 September 8	State Of The Map 2013	Birmingham, UK
September 6 September 8	Kiwi PyCon 2013	Auckland, New Zealand
September 10 September 11	Malaysia Open Source Conference 2013	Kuala Lumpur, Malaysia
September 12 September 14	SmartDevCon	Katowice, Poland
September 13	CentOS Dojo and Community Day	London, UK
September 16 September 18	CloudOpen	New Orleans, LA, USA
September 16 September 18	LinuxCon North America	New Orleans, LA, USA
September 18 September 20	Linux Plumbers Conference	New Orleans, LA, USA
September 19 September 20	UEFI Plugfest	New Orleans, LA, USA
September 19 September 20	Open Source Software for Business	Prato, Italy
September 19 September 20	Linux Security Summit	New Orleans, LA, USA
September 20 September 22	PyCon UK 2013	Coventry, UK

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol