LWN.net Weekly Edition for January 19, 2012

LCA: A Samba 4 update

Andrew Bartlett took over to say that both he and Tridge think that the project is about ready for the Samba 4 release. The active directory (AD) domain controller (DC) support - a headline Samba 4 feature - is working well and is in production use in a number of sites; it is time to get it out there to the rest of the world. While they think that, at this point, things are ready for a release, the idea came as a shock to some of the other members of the team. Samba 4 had been seen by those developers as being far out on the horizon; they were not expecting talk of a release at this point.

The ensuing discussion was lively, but AD DC support was not the main point; everybody seems to agree that it is working well. The sticking point has to do with the long-time "bread and butter" features of Samba - little things like file serving. The new file server implementation in Samba 4 is missing a number of features that have gone into Samba 3 in recent years, so now the focus is on integration of Samba 4 with the Samba 3 file server. The developers have come up with a plan for this integration, and are now busily trying to implement it as quickly as possible. As Tridge put it, they ran into a social problem and came up with a technical solution because, in the end, coding is easier than arguing. The discussion has gone quiet because this coding is underway; they expect to present their solution soon, at which point the release discussion can be expected to restart.

Andrew spent some time talking about some of the things the Samba team has achieved with Samba 4. One of those is the new integrated build system - there is now "a single Samba." It is possible to build all binaries together; and there are a number of plugins to further integrate Samba's various pieces. As a result, Samba is now "one project," rather than a collection of related pieces.

Related to that is the new combined testing framework which is, according to Andrew, the most important thing that the Samba team has achieved. The framework can do full testing of all AD semantics. It is also set up to test Samba 3 and 4 against each other. A number of "rather embarrassing" interoperability problems between the two releases have been found and, naturally, fixed. This testing can now be done before every commit.

There is also a common security system that simplifies administration and fixes a lot of old "misunderstandings of Kerberos" that have been with the project for a long time. The Samba 3 and 4 security architectures have been merged.

All of this, Andrew said, has been good to make the new system work well, but it does not necessarily change the user's experience of Samba. There has been new feature work done as well, though. At the top of the list, according to Tridge, is subdomain support. Lots of sites do not work with a single domain at this point; instead, they have "forest" of domains organized into a hierarchy. Getting Samba to work in this mode has taken a lot of work over the last year. The 2011 plugfest event, where eight or so Samba developers went to Redmond to work on interoperability issues with Microsoft, was dedicated to firming up subdomain support and getting to a point where Samba can work at any level in an AD forest. It does work, but has not yet been designated ready for production; Tridge said he would like to see a couple of "brave" production sites deploy it and let them know how it works for them.

The project's relationship with Microsoft, they said, is quite good. They get quick answers to questions, even for detailed protocol history queries that require a fair amount of digging in the code to answer. Tridge said that he has been very impressed with the quality of the engineers that Microsoft has assigned to work with the project.

Another area of development is easing the process of upgrading from Samba 3 to Samba 4. Production sites, it seems, do not react well if you tell them that all of their users have to set new passwords before they can work under a new Samba release. At this point they have full user and group import into Samba 4, so users should not see the difference. The update is transparent to clients, except that they see the new AD support and start using it. There is still a bit of a flag day involved, though, in that clients, once they see an AD server, will not go back to talking to an older server release. So careful testing before deploying Samba 4 is still called for.

Amitay Isaacs and Kai Blin talked about their area of work: the built-in DNS server. Amitay has implemented one solution, whereby a new DLZ plugin for bind9 enables it to get its domain information from the AD database. It works, but it is "a bit clunky" as a result of the interactions between the two separate subsystems. So Kai is working on a new, internal DNS server. He had tried, he said, to get an existing DNS server project interested in closer integration, but found no takers. So he wrote a new server which, he said, was not that hard a problem. It is working now, with signed updates being the main missing feature at this point.

The "roadmap," according to Andrew, is that Samba 4 will probably be the next release from the project. It will include all of the expected features, including file and print servers, support for NT4-like domain controllers, and active directory support. It will also feature a number of improved tools and better usability in general. Samba has seen nearly 8,000 commits over the past year, changing 800,000 lines of code, and coming from some 70 authors. It has been, he said, a busy and important year. With a Samba 4 release likely, 2012 could be an even busier and more important year for this project, which quietly celebrated its 20th anniversary at the end of 2011.

[Your editor would like to thank the LCA organizers for assisting with his travel to Ballarat.]

Comments (13 posted)

Tizen releases source code and SDK previews

The nascent Tizen project unveiled its first set of materials on January 9, consisting of "preview" releases of the operating system source code and SDK, both intended to elicit feedback from developers. The announcement was accompanied by the launch of two new mailing lists and online documentation of the project's architecture and APIs. Tizen was announced in September 2011, but Intel took a markedly different approach to publicizing and promoting the project than it had with MeeGo. While MeeGo was launched with fanfare, the Tizen governance structure, policies, community resources, and ultimately code have been delivered at a much more measured — if not downright conservative — pace. Critics (particularly those upset over the demise of the MeeGo project) labeled the new project "vaporware" early on, but they — and everyone else — can now examine the source code up close.

Sources

The source code release comes in the form of a git repository, which hosts both Tizen packages and trees pulled from upstream projects. Tizen is often referred to as the successor to MeeGo, which was an Intel-Nokia collaboration on a "consumer electronics" Linux base layer derived from merging the two companies' existing projects. But even though Tizen's goals are the same, the actual software is a significant departure from MeeGo — the stack is based on the LiMo Foundation's work and is expected to integrate Samsung's Bada platform. More significantly for third-party developers, the application framework is centered around HTML5 rather than Nokia's Qt.

The components beneath that application framework are quite similar, of course (as they are in most embedded Linux distributions): the kernel (version 2.6.36), EGLIBC (2.13), Busybox, D-Bus, PulseAudio, GStreamer (0.10), ConnMan, and so forth. Among the other notable modules are a multi-lingual input framework derived from SCIM, hardware acceleration for OpenGL ES 1.1 and 2.0, and a WebKit-based web runtime. There is also a security module which provides Smack access control, secure data storage, and certificate management.

Perhaps the most significant departure from MeeGo is that Tizen's graphics toolkit is provided by the Enlightenment Foundation Libraries (EFL), rather than GTK+ or Qt. It has been a few years since Enlightenment and EFL were headline-makers on Linux desktops, but they have been in active development all the same, with support from Samsung, among others. LWN looked at EFL back in August.

EFL provides a canvas (called "Evas") that supports multiple back-ends; the Tizen project's is currently using the X11 back-end, which came as a mild disappointment to some observers, considering the work Intel had put into porting MeeGo to the Wayland display server. EFL's Carsten Haitzler told the mailing list that the hurdle was a lack of Wayland drivers for the system-on-chip video hardware targeted "at this stage," however, so there remains hope for Wayland support in the future.

EFL also provides application developers with a scripting environment called Embryo, a widget set (with touch support) called Elementary, as well as various utility libraries for timers, callbacks, event loop management, and data structures. Tizen includes the E17 version of EFL's Enlightenment window manager which supports compositing.

Web APIs

EFL does not encompass as much as Qt, however — richer APIs for hooking into system services and support for device features (like cameras and sensors) are both provided through Tizen's HTML5 application framework. The framework consists of three pieces: the "Tizen Web API," the Wholesale Application Community (WAC) "Device APIs," and the World Wide Web Consortium (W3C) "Widget Specification."

The Tizen Web API covers generic application objects, and interfaces to the communications, system information, and personal information management (PIM) subsystems. The supported communication frameworks include Bluetooth, NFC, cellular phone calls, and a general "messaging" module that encompasses email, IM, SMS, and MMS. The Tizen Web API's Contact, Calendar, and Messaging modules are each extensions of the corresponding WAC Device API modules, apparently with minor additions only. For example, The Tizen Web API's MessagingMessage adds message IDs and inResponseTo attributes, which allows an application to thread conversations.

WAC is an organization formed in 2010 by telecommunications companies and device OEMs. It publishes its own suite of OS-independent application development APIs, which are intended to provide a uniform HTML5 target for developers across the members' products. This effort is similar to the work produced by the W3C's Device APIs Working Group, but WAC does not develop its APIs in the open. Further confusing the situation, some of WAC's APIs require compliance with the corresponding W3C API, but some do not.

The Tizen alpha makes use of WAC's PIM APIs (the contacts, calendaring, and messaging modules mentioned above), its sensor APIs, its camera module, and its filesystem module. The sensor APIs provide interfaces to a mobile device's hardware sensors: accelerometer, digital compass, ambient light sensor, rotation/orientation sensor, and so on. The camera API covers both still and video capture, as well as displaying a preview image on screen. The filesystem module provides a single view of a device's storage, combining all on-board or removable storage into a unified virtual filesystem.

Finally, in spite of its name, the W3C Widget Specification is actually a standard for "installable" web applications — a storage format for compressed HTML5, CSS, and JavaScript applications, with an XML-based manifest. The Widget specification supports digital signatures and a device-controlled security policy to limit applications' access to network resources and local data.

Together, the Web API offerings may sound like a bit of a hodge-podge — after all, the WAC and W3C specifications overlap in many places, and conflict in others. Intel's Sakari Poussa responded to one developer's question by saying that Tizen was tracking the standards as best it could, considering that both APIs are still evolving, and that the project was committed to sticking with existing standards where possible.

Developing

In addition to the source code, the alpha release also includes the first look at the Tizen SDK. The SDK is available only for 32-bit Ubuntu (11.04 and newer), though the project says other distributions and operating systems will be supported in the final release. The SDK download is an installer script, which fetches and installs an Eclipse-based IDE, a QEMU-based emulator, testing tools, and support files.

The IDE includes project templates in two classes: "Tizen Web projects" and "native" Tizen projects. That is more-or-less in line with the messaging of the project, which emphasizes that HTML5 is the only supported application framework, but expresses tolerance for developers who wish to write code directly for the system APIs. The web project templates come in several flavors: WAC-only, generic HTML5, and jQuery (in several variations), and include Tizen packaging rules. The IDE experience also incorporates an event injector supporting mobile device hardware events, sensor data, and voice or SMS traffic.

Some web application debugging tools are also built in to the IDE, such as a WYSIWYG HTML5 editor and JavaScript/CSS syntax checkers. I was never able to get the IDE to launch the web project templates without crashing, so I cannot speak to their usefulness. There are known issues with the SDK mentioned in the release notes (including the fact that it is incompatible with OpenJDK — my experience suggests it may also be very picky about Sun Java 6); hopefully these concerns will be addressed between now and the final release.

The emulator has a few known issues of its own, of course, but the basic functionality is there. Currently only x86 targets can be created, with a handful of configuration options. The form factor of the virtual device is a keyboard-less mobile phone. The SDK documentation emphasizes that smartphones and tablets are the target devices for the alpha release, but that others will follow. Like MeeGo, Tizen is designed to eventually serve several distinct classes of hardware (including vehicles and smart-TVs).

The SDK also includes a documentation browser that encompasses an overview of the platform, the SDK tools themselves, and the various APIs. Some of this information is replicated on the Tizen developer site: the "Getting Started with Tizen" and "Tizen Web API Reference" sections in the documentation. But there is more in the SDK's browser, which covers packaging issues and hacking on the Tizen platform itself.

Community communications

Although it was not part of the developer preview per se, the SDK and code release coincided with the launch of two new Tizen mailing lists: one for application developers, and one for Tizen contributors. Up until now, the Tizen-General list and IRC channel had been the only communication channels for the project — very different from the approach taken with MeeGo, which had numerous mailing lists, forums, IRC channels, and a blog-aggregating Planet. Intel's we'll-announce-nothing-until-it-is-ready approach with Tizen has produced a slower trickle of discussion, but perhaps that is better than the publicity blitz that accompanied MeeGo's launch, when you consider that announcing the project and putting devices in consumers' hands can be many fiscal quarters apart.

A related development on the project management front was the sudden disappearance of the LiMo Foundation web site, which was replaced by the Tizen Association on or about January 1. The Tizen Association is essentially a re-branding of the LiMo Foundation, and, as yet, Intel itself has not finalized its membership. The Association's site describes its goal as enabling "key stakeholders to actively shape the industry role of Tizen and develop its market presence" by the "gathering of requirements, identification and facilitation of service models, and overall industry marketing and education." The project itself will continue to be hosted by the Linux Foundation.

The specifics of Tizen's project governance have not been fleshed out, but those are probably details that should come after the code itself has been released and developers have had a chance to work with it. In retrospect, the MeeGo project was very organization-heavy (as it was marketing-heavy), and in the end that did not help it make an impact in the marketplace. Tizen may still be a long way from shipping on commercial devices, but starting with the code rather than the other trappings of a large distributed project is a good first step.

Comments (3 posted)

LCA: Addressing the failure of open source

The open source (he always used that term) movement's goal, he said, was once to help a population that increasingly does everything - from entertainment to finances and voting - through computers. We wanted to enable people to do, not to be done to. But we find ourselves in a world where people are increasingly slaves to their tools. Yes, he said, tools like the iPhone empower their users, but they also constrain those users. They are designed not to allow their users to do things that might reduce the profits of their makers or of a number of related industries.

Why is this important? It matters to anybody who likes democracy. Almost all information is mediated through computers now - through computers controlled by a very small number of large corporations. That is a threat to democracy - and to freedom. But, he asked, why is it open source's job to address this problem? The answer is that the open source community is the only credible producer of software that is not bound to a single company's interest. History has given us absolutely no reason to believe that we can trust companies or governments to do that for us.

To be able to fill that role, we must continue to establish and defend our right to write and distribute open source software. In other words, he said, open source software should be legal. Things could be worse: we have been very lucky with software patents, for example. Patents could have been used to shut us down entirely by now; that has not happened - yet. But, increasingly, we find ourselves shut out of access to content, which does not help our cause.

There are also threats from new laws; increasingly, Bruce said, general-purpose computing is seen as a threat. Anti-hacking laws threaten the security of the net as a whole. SOPA-like laws threaten our tools and our infrastructure. The advent of 3D printers is going to inspire a whole new raft of laws as politicians figure out that they can be used to create weapons or other things that they do not want us to have.

We are very much on the defensive in these battles; why, he asked, do we have so much trouble being heard? The answer is that open source software has not built a relationship with "the common person" and does not have their sympathy. Instead, open source software has been co-opted and is being used to control people; thus we end up with locked-down products like the TiVo. We are, Bruce said, helping to make people into slaves of their tools. A project he worked on - busybox - is now shipped as an important component in any number of locked-down systems.

As an aside, Bruce said that, in retrospect, he should have continued to lead the busybox project.

It comes down to the fact that open source has not been able to protect its future through the reform of hostile laws. That is a result of our failure to reach the common man. We, as a community, have not achieved a sympathy for our users in the way that companies like Apple have. Some specific projects have had some success - he named Mozilla and Wikipedia - but, in general, our work is not particularly accessible to - or visible to - most users.

In fact, he said, most of us do not really even like users; we are making the software for ourselves. Demands from users who have not contributed anything are just annoying. But, he said, we need to get over that mindset; working only for ourselves is too limiting.

What about Ubuntu? It is, according to Bruce, demonstrating the necessary discipline to reach users. But, unfortunately, Ubuntu has turned itself into the intermediary between open source and its users; other open source companies, such as Red Hat, have done the same thing. We need, he said, to be more honest with ourselves about our partners. Commercial partners will always put their own business interests above any other consideration - in many countries, public companies are required to do that. They cannot work for the best interests of open source software.

So, he asked, where did we go wrong? There was a period where we held a moral high ground; that enabled political progress that we are not making any more. As for-profit corporations took over the foreground, we lost that high ground. Now we are just competitors, like any other. The commercial intermediaries represent us now, but they represent us in their own best interest. That is not surprising - it is what is expected of them - but it shows why they are wrong for this role. We really only have one tool for managing the behavior of these companies - copyright - and it is not helpful in this situation.

In dealing with companies, Bruce said, we always need to keep track of whether we are getting back something worthwhile for our efforts. Working for free to enrich Mark Shuttleworth is not a smart thing to do. We need to maintain a presence that is independent of organizations like Ubuntu. Debian, he said, does the real work of making Ubuntu; Canonical just adds the frosting. But does anybody in the wider world care about Debian at this point? We are, he said, failing at self promotion. Yes, tooting our own horn is tiresome, but if we don't do it, nobody else will do it for us.

Mobile apps are an especially big problem, he said. This is a commercial paradigm that has succeeded; that means, in particular, that the web failed. Now we are seeing open source software being sold as apps to people who don't know that they are buying something they could have for free. These apps, he said, duplicate every problem we have seen with proprietary software, only worse because these apps are meant to be ephemeral. We are going to have to find a way to change this paradigm, somehow.

Concluding this part of the talk, Bruce noted a strong trend away from generic platforms, toward proprietary, locked-down platforms. In the end, we will be left with nowhere to run free systems, and only jail environments for applications. Open source software, he said, has achieved tremendous things. But he is seeing signs that we have peaked; the locked-down platform is beating us. If we cannot find a way to address this problem, he said, it is our destiny to live in a constrained world.

Open hardware

There is a moment, far into Arlo Guthrie's epic "Alice's Restaurant," where he is required to remind the audience that, contrary to what they may think, they are listening to a song about Alice. Bruce's talk was similar; it was billed as being about open hardware, but it was only toward the end that he actually got around to talking about that subject. It is clearly Bruce's hope that a new revolution is building around open hardware, and that it may, in the end, help to save us from locked-down platforms. There are a lot of challenges in this area, he said, but he reminded the audience that the open source community has done many things that people thought were impossible.

Once upon a time, he said, people working on open hardware might have been seen as "isolated nutjobs." The good news is that, in the age of the Internet, there are no more isolated nutjobs. No matter how nutty you may be, there will always be fifty others like you on the net. In the open source world, those people got together and created tools that others could build on; things then took off from there. This history is, he said, repeating itself with open hardware.

There is a new set of enabling factors that are driving open hardware. We have the net, of course, but we also have the economic paradigms and collaboration methods developed by the open source community. But the big thing is that hardware is slowly being transformed into software; it can be developed - and shared - in much the same way. 3D printers, increasingly, let us print out solid objects that can be designed like software. Gate arrays allow the immediate implementation of complicated digital logic - at the level of a new CPU, for example. Throw in fast, cheap circuit board manufacturing and there is a lot that can be done by just about anybody.

So what is happening with open hardware? For many, the flagship project is certainly Arduino, which has found its way into many or most open hardware projects at this point. The boards can be cheap enough to (for example) throw into the LCA swag bag. There are over 250 "shields" adding interesting functionality; for any problem, there is likely to be a plugin solution out there somewhere. It has a development environment that makes it programmable by people like artists; as a result, Arduino is showing up in a lot of art applications now. And the whole thing is surrounded by an active community.

Arduino has some down sides. It does not use a 32-bit processor, so it cannot run complex software (like Linux, for example). There are a lot of alternatives out there, though, for those needing more power; the Beagleboard and Pandaboard offerings for example. Bruce also called out the Omap L138 processor in particular, calling it a dream for mobile software-defined radio applications. It has four CPUs, a lot of peripherals, and is quite affordable.

Bruce waved around a DSO Quad, a pocket-sized digital storage oscilloscope. It is all open hardware and software; it shows that one does not need a large corporation to create compelling mobile products. Then, there is the Bus Pirate, a general-purpose electronic hacking tool that can take on all kinds of tasks from programming FPGAs to sniffing data transfers from an I2C bus. Bruce also mentioned the Papilio gate array development board. This device can become almost any kind of circuit from a CPU emulator to a receiver for 20 simultaneous radio streams.

For those wanting more physical projects, there is the MakerBot, a plastic 3D printer. MakerBot leaves a lot to be desired; in particular, Bruce said, it will fall short until it has enough resolution to make high-quality Lego parts. It will be able to make anything one might want eventually; for now it is limited to "small, crappy-looking things." But it shows where things are headed. For the truly adventurous, there is OpenPCR, which allows genetic sequencing and duplication at home.

The list could probably have gone on for a long time, but Bruce had made his point - there is a lot happening in the open hardware area. There is the potential for amazing things to happen. But much depends on the legal climate, and it is clear that there are going to be legislative attacks on open hardware. We will have to be attentive if this revolution is not to be choked off before it can really begin.

[Your editor would like to thank the LCA organizers for assisting with his travel to Ballarat.]

Comments (157 posted)

Help wanted at LWN

Do you have an interest in helping to create the best free software technical information site on the net? Here at LWN.net, we have reached a point where we can hire another full-time editor if we can find the right person. We have put together a list of the skills we are looking and if you (or someone you know) meets them, please get in touch. The full job listing has more information.

Comments (none posted)

Page editor: Jonathan Corbet

Security

SOPA and PIPA

As this article is being written on January 18, many high-profile web sites have gone "black" in order to protest proposed legislation in the US that would, ostensibly, combat online "piracy". Lots of different sites are participating, from Wikipedia and Google to sites that cater to more technical audiences like Reddit and Bruce Schneier's security blog. The intent is to raise the profile of the two related pieces of legislation with the hope that users (both technical and less so) will recognize the threat to a "free and open internet" that the laws represent.

As seems to commonly be the case—at least in the US—the proposals have been given names that don't accurately reflect their scope. The "Stop Online Piracy Act" (SOPA) is the House of Representatives' entrant, while the Senate bill is called the "Protect Intellectual Property Act" (PIPA). Both are strongly backed by the content industries who make unsubstantiated claims about the financial and job losses caused by online piracy—and are undoubtedly pouring lots of money into lobbying for their passage. But, many internet sites and luminaries are extremely wary of the bills' contents because they could "break the internet".

The law of "unintended consequences" is often present in internet-targeted legislation, but it's a little hard to see some of the consequences from SOPA/PIPA as being unintended—at least by some of their proponents. The content industry has found it expensive and difficult to combat the copyright infringement of its works under the existing laws, and is always looking for a way to make others responsible for enforcement. SOPA/PIPA are just the latest salvo in that effort.

The biggest technical problem with the legislation is that it tries to fundamentally alter how the domain name system (DNS) works, so that "rogue" sites that carry infringing content—or even links to infringing content—can be "banished" from the internet. As pointed out by Paul Vixie, though, all of the technical measures that SOPA/PIPA want to use to blacklist these supposedly rogue sites won't do so. In addition, implementing these "features" will just increase the load on DNS servers as clients try to route around the censorship.

The argument from proponents is that many of the sites that they would like to shut down are foreign-owned, and thus are impossible to affect via US law. Aside from the irony of using US legislation to somehow do the impossible, mandating that US companies blacklist web sites via DNS or any other method is only likely to result in fragmenting the infrastructure of the internet. As an open letter from 83 internet inventors and engineers to the US Congress in December put it:

The bills would mandate that US companies enforce the blacklist, and provide penalties if a service allows users to circumvent the list. Not only does that put a large burden on ISPs, web application providers, internet startups, and others, it also adds a large uncertainty factor by the terms used. Rather than focusing strictly on sites that infringe copyrights (for which there are plenty of laws already available), these bills would be enforced against sites that are "enabling or facilitating" infringement (at least in SOPA). That kind of ambiguity leaves the door open to all sorts of abuse.

Those penalties can be severe. One of the actions that a copyright holder could take against a site deemed to have violated these acts is to get a court order that requires payment sites and advertising networks to cut the site off. So, a site that "facilitates" infringement (which could easily be applied to almost any site on the internet) could suddenly find itself cut off from its funding sources—or in court to show why it shouldn't be. For larger, well-established sites and service providers, that will be an expensive annoyance, but for smaller fish (including startups and sites like LWN) it could easily put the company out of business.

Proponents of these laws downplay the potential for widespread applicability, explaining that they are targeted at the "worst of the worst" offenders. But, as we have seen with almost any law (internet-focused or not), they is almost always some kind of overreach. Once these (or similar) laws are on the books, the content industries will be pushing the envelope. We've seen this overreach with the DMCA, anti-"hacking" laws, the PATRIOT act, and more. Meanwhile, the rest of the world (along with much of the US) will just find ways to route around the blockades.

Certainly copyright infringement is a problem. Copyright is, after all, the tool that the free software community uses to enforce its licenses. The question is how big of a problem infringement really is, and what the right solution is. The content industries would have us believe that infringement is stealing food from the mouths of babes—while often reporting record profits. It's not at all clear that passing more and more laws will "fix" the problem.

The only real way to completely prevent digital copyright infringement is to curtail general purpose computing (and networking) in ways that would be drastically detrimental to the worldwide economy (not to mention little things like personal freedom). Cory Doctorow recently spoke about the likelihood of an upcoming war against general-purpose computing. One could argue that we have already lost some battles in that war (e.g. DMCA) and SOPA/PIPA are yet another. Hopefully, the efforts of the high-profile sites protesting this legislation will start to wake people up about what kinds of things the content industries and their "pet" legislators are trying to do. If not, we will likely see even more draconian legislation down the road.

For readers wanting to know more, Techdirt's Mike Masnick has two analyses of SOPA/PIPA [1, 2] that are well worth a read.

Comments (5 posted)

Brief items

NSA releases security-enhanced Android (The H)

Comments (21 posted)

Berrangé: Building application sandboxes with libvirt, LXC & KVM

Comments (17 posted)

New vulnerabilities

acroread: code execution

Comments (none posted)

kernel: multiple vulnerabilities

Comments (none posted)

kernel: syscall instruction induces guest panic

    [bits 32]
    global _start
    SECTION .text
    _start: syscall

Comments (none posted)

kernel: denial of service

Comments (none posted)

libxml2: code execution

Comments (none posted)

openssl: private key disclosure

Comments (none posted)

plib: arbitrary code execution

Comments (none posted)

rubygem-rack: denial of service

Comments (none posted)

simplesamlphp: cross-site scripting

Comments (none posted)

t1lib: multiple vulnerabilities

Comments (none posted)

wordpress: cross-site scripting

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

3.2 is the most recent kernel, released on January 4. The 3.3 merge window is still open, see the article below for what's been merged in the last week. 3.3-rc1 can probably be expected sometime soon.

Stable releases: The 3.1.10 stable kernel was released on January 18. This is the last stable kernel in the 3.1.x series, so users should upgrade to the 3.2 series.

In addition, the 2.6.32.54, 3.0.17, 3.1.9, and 3.2.1 stable kernels were released on January 12.

Comments (none posted)

Quotes of the week

Comments (2 posted)

Kernel development news

3.3 merge window part 2

User-visible changes merged since last week include:

• The kernel has gained the ability to verify RSA digital signatures. The extended verification module (EVM) makes use of this capability.

• The slab allocator supports a new slab_max_order= boot parameter controlling the maximum size of a slab. Setting it to a larger number may increase memory efficiency at the cost of increasing the probability of allocation failures.

• The ALSA core has gained support for compressed audio on devices that are able to handle it.

• There have been some significant changes made to the memory compaction code to avoid the lengthy stalls experienced by some users when writing data to slow devices (USB keys, for example). This problem was described in this article, but the solution has evolved considerably. By making a number of changes to how compaction works, the memory management hackers (and Mel Gorman in particular) were able to avoid disabling synchronous compaction, which had the unfortunate effect of reducing huge page usage. See this commit for a lot of information on how this problem was addressed.

• There is a new "charger manager" subsystem intended for use with batteries that must be monitored occasionally, even when the system is suspended. The charger manager can partially resume the system as needed to poll the battery, then immediately re-suspend afterward. See Documentation/power/charger-manager.txt for more information.

• The Btrfs balancing/restriping code has been reworked to allow a lot more flexibility in how a volume is rearranged. Restriping operations can now be paused, canceled, or resumed after a crash.

• The audit subsystem is now supported on the ARM architecture.

• New device drivers include:
  • Systems and processors: Renesas R8A7740 CPUs, R-Car H1 (R8A77790) processors, NetLogic DB1300 boards, Ubiquiti Networks XM (rev 1.0) boards, Atheros AP121 reference boards, and Netlogic XLP SoC and systems.

  • Audio: Realtek ALC5632 codecs and Cirrus Logic CS42L73 codecs.

  • Block: Micron PCIe SSD cards and solid-state drives supporting the NVM Express standard.

  • Miscellaneous: TI TWL4030 battery chargers, Dialog DA9052 battery chargers, Maxim MAX8997 MUIC devices, Samsung Electronics S5M multifunction devices, and CSR SiRFprimaII DMA engines.

  • Video4Linux: Samsung S5P and EXYNOS4 G2D 2d graphics accelerators, remote controls using the Sanyo protocol, Austria Microsystems AS3645A and LM3555 flash controllers, Microtune MT2063 silicon IF tuners, Jellin JL2005B, JL2005C, or JL2005D-based cameras, HDIC HD29L2 demodulators, and Samsung S5P/Exynos4 JPEG codecs.

Changes visible to kernel developers include:

• The memory control group naturalization patches have been merged. These patches eliminate the double-tracking of memory and, thus, substantially reduce the overhead associated with the memory controller.

• The framebuffer device subsystem has a new FOURCC-based configuration API; see Documentation/fb/api.txt for details.

• The Btrfs filesystem has gained an integrity checking tool that monitors traffic to the storage device and looks for operations that could leave the filesystem corrupted if the system fails at the wrong time. See the comments at the top of fs/btrfs/check-integrity.c for more information.

The 3.3-rc1 release can be expected at almost any point; after that, the stabilization process begins for the 3.3 development cycle. If the usual timing holds (and it almost always does anymore), the final 3.3 kernel release can be expected in the second half of March.

Comments (none posted)

System call filtering and no_new_privs

We briefly covered a proposal for restricting system calls using the kernel packet filtering mechanism on the January 12 Kernel page, but, at that time, there hadn't been any comments on the proposal. Since then there have been several rounds of comments and revisions of the patch set, along with a revival of an older idea to let a process limit itself and its children to its current privilege level. So far, both sets of patches have received generally positive feedback, to the point where it seems like general-purpose system call filtering just might make it into the mainline sometime in the not-too-distant future.

For some time now, Will Drewry has been trying to find an acceptable way to enhance the seccomp ("secure computing") facility in the kernel so that more flexible system call filtering can be done. His target for the feature is the Chrome/Chromium web browser in order to sandbox untrusted code, but other projects (including QEMU, openssh, vsftpd, and others) have expressed interest in the feature as well. He (and others) have tried various approaches over the last few years without finding one that passed muster. His latest attempt, which uses the BPF (Berkeley Packet Filter) engine to filter the system calls, seems like it avoids many of the problems that were noted in the earlier attempts.

The basic idea is that instead of examining packet contents, the filters will examine system calls and any arguments passed in registers (that means that it won't follow pointers to avoid time-of-check-to-time-of-use races). The code will only allow those calls that pass the filter tests to be executed. The filtering fails "closed" so that any calls not listed in the filter, or whose arguments don't correspond to the filter rules, will return an EACCES error. The syntax for creating a filter, as described in the documentation file, is fairly painful, but Eric Paris has already started on a translator to turn a more readable form into the BPF rules needed.

In order to avoid a longstanding problem with the interactions between binaries that can change their privileges (e.g. setuid or file-based capabilities) and mechanisms to reduce privileges for a process, Drewry's initial patch would restrict the ability of a process to make an execve() call once a filter had been installed. The problem is that privilege-changing binaries can get confused when faced with an environment with fewer privileges than are expected. That confusion can lead to privilege escalation or other security holes. This is why things like chroot(), bind mounts, and, eventually, user namespaces are restricted to root-privileged processes.

If a filtered process can't successfully call execve(), though, all of the concerns about confusing those binaries is gone. It does make using the system call filtering a little clunky, however. One would expect that a parent could set up filters and then spawn a child that would be bound by those filters, but, without a way to exec, that won't work. That can be worked around for most existing programs with some LD_PRELOAD trickery, but in the discussion another potential solution was proposed.

Andrew Lutomirski pointed to his execve_nosecurity proposal as a possible solution. That would allow processes to set a flag so that they (and their children) would be unable to call execve() and would add a new variant (called, somewhat confusingly, execve_nosecurity()) that could be used instead but would not allow any security transitions for the executed program. That means that setuid, LSM context changes, changing capabilities, and so on would not be allowed. Linus Torvalds agreed that adding a way to restrict privilege changes would be useful:

That led Lutomirski to propose a flag in struct task_struct called no_new_privs that would be set via the PR_SET_NO_NEW_PRIVS flag to prctl(). It would be a one-way gate as there would be no way to unset the flag. If set, the flag would restrict executing binaries in much the same way that the nosuid mount flag works. In addition, it would disallow processes changing capabilities on exec or SELinux security context transitions.

But, Lutomirski's patch does not implement a sandbox, as it can still be subverted via ptrace() as Alan Cox points out. Cox was also concerned that preventing SELinux, AppArmor, or other LSMs from changing privileges could lead to other problems because those transitions may actually be changing the context to a less privileged state. Simply keeping the previous context, as Lutomirski's patch does, could lead to executing programs in a more-privileged context. But Eric Paris clarifies that SELinux, at least, will still make the same policy decision even without the transition (as it does for nosuid mounts), so that the execution will still fail if the process has the wrong context.

Lutomirski also notes that a sandbox will be much less useful if execve() has to fail when there is any kind of security transition, as Cox suggested. The presence of a policy on a particular binary would make that binary unusable from within a sandbox, no matter what the policy is. A better solution, Lutomirski said, is to set the no_new_privs bit, then set up a sandbox (using Drewry's seccomp system call filtering for example), then execute the binary, which will succeed or fail based on the actual mandatory access control (MAC) policy. That solves the problem of ptrace() and other circumvention methods as well because a sandbox requires both the no_new_privs patch and some other mechanism to filter system calls:

Meanwhile, Drewry has been revising his patches to take advantage of no_new_privs. One of those revisions brought about some other concerns regarding whether dropping privileges should be allowed after the bit is set. Torvalds is worried that allowing privilege dropping will somehow lead to confusing other programs: "We've had security bugs that were *due* to dropped capabilities - people dropped one capability but not another, and fooled code into doing things they weren't expecting it to do." Lutomirski's patches do not restrict things like calls to setuid() because they are not meant to implement a sandbox—that's what the existing seccomp, or an enhanced version from Drewry's patches (aka seccomp mode 2) will do. As Lutomirski explains:

It's clear that Lutomirski, at least, thinks the no_new_privs changes cannot lead to the problems that Torvalds and others (notably Smack developer Casey Schaufler) are concerned about. But, any program that uses no_new_privs needs to be aware of what it does (and doesn't) do. Coupling it with a system call filtering mechanism seems like it could only increase the security of the system. But, interactions between security mechanisms often have unforeseen effects, typically resulting in security holes, so it makes sense to be cautious.

So far, these changes are still being discussed, and no subsystem maintainer has volunteered to take them, but the two proposals seem to have support that other similar ideas have lacked. Whether Lutomirski can convince the other kernel hackers that no_new_privs can't lead to other problems, or whether he needs to figure out how to stop the dropping of privileges is unclear. But it does seem like there may now be a path for an enhanced seccomp to reach the mainline.

Comments (none posted)

The future calculus of memory management

Over the last fifty years, thousands of very bright system software engineers, including many of you reading this today, have invested large parts of their careers trying to solve a single problem: How to divide up a fixed amount of physical RAM to maximize a machine's performance across a wide variety of workloads. We can call this "the MM problem."

Because RAM has become incredibly large and inexpensive, because the ratios and topologies of CPU speeds, disk access times, and memory controllers have grown ever more complex, and because the workloads have changed dramatically and have become ever more diverse, this single MM problem has continued to offer fresh challenges and excite the imagination of kernel MM developers. But at the same time, the measure of success in solving the problem has become increasingly difficult to define.

So, although this problem has never been considered "solved", it is about to become much more complex, because those same industry changes have also brought new business computing models. Gone are the days when optimizing a single machine and a single workload was a winning outcome. Instead, dozens, hundreds, thousands, perhaps millions of machines run an even larger number of workloads. The "winners" in the future industry are those that figure out how to get the most work done at the lowest cost in this ever-growing environment. And that means resource optimization. No matter how inexpensive a resource is, a million times that small expense is a large expense. Anything that can be done to reduce that large expense, without a corresponding reduction in throughput, results in greater profit for the winners.

Some call this (disdainfully or otherwise) "cloud computing", but no matter what you call it, the trend is impossible to ignore. Assuming it is both possible and prudent to consolidate workloads, it is increasingly possible to execute those workloads more cost effectively in certain data center environments where the time-varying demands of the work can be statistically load-balanced to reduce the maximum number of resources required. A decade ago, studies showed that, on average, only 10% of the CPU in an average pizza box server was being utilized... wouldn't it be nice, they said, if we could consolidate and buy 10x fewer servers? This would not only save money on servers, but would save a lot on power, cooling, and space too. While many organizations had some success in consolidating some workloads "manually", many other workloads broke or became organizationally unmanageable when they were combined onto the same system and/or OS. As a result, scale-out has continued and different virtualization and partitioning technologies have rapidly grown in popularity to optimize CPU resources.

But let's get back to "MM", memory management. The management of RAM has not changed much to track this trend toward optimizing resources. Since "RAM is cheap", the common response to performance problems is "buy more RAM". Sadly, in this evolving world where workloads may run on different machines at different times, this classic response results in harried IT organizations all buying more RAM on most or all of the machines in a data center. A further result is that the ratio of total RAM in a data center vs. the sum of the "working set" of the workloads, is often at least 2x and sometimes as much as 10x. This means that somewhere between half and 90% of the RAM in an average data center is wasted, which is decidedly not cheap. So the question is begged: Is it possible to apply similar resource optimization techniques to RAM?

A thought experiment

Bear with me and open your imagination for the following thought experiment:

Let's assume that the next generation processors have two new instructions: PageOffline and PageOnline. When PageOffline is executed (with a physical address as a parameter), that (4K) page of memory is marked by the hardware as inaccessible and any attempts to load/store from/to that location result in an exception until a corresponding PageOnline is executed. And through some performance registers, it is possible to measure which pages are in the offline state and which are not.

Let's further assume that John and Joe are kernel MM developers and their employer "GreenCloud" is "green" and enlightened. The employer offers the following bargain to John and Joe and the thousands of other software engineers working at GreenCloud: "RAM is cheap but not free. We'd like to encourage you to use only the RAM necessary to do your job. So, for every page, on "average" over the course of the year, that you have offline, we will add one-hundredth of one cent to your end-of-year bonus. Of course, if you turn off too much RAM, you will be less efficient at getting your job done, which will reflect negatively on your year-end bonus. So it is up to you to find the right balance."

John and Joe quickly do some arithmetic: so, since my machine has 8GB RAM, if I on average keep 4GB offline, I will be $100 richer. They quickly start scheming ideas on how to dynamically measure their working set and optimize page offlining.

But the employer goes on: "And for any individual page that you have offline for the entire year, we will double that to two-hundredths of a cent. But once you've chosen the "permanent offline" option on a page, you are stuck with that decision until the next calendar year."

John, anticipating the extra $200, decides immediately to try to shut off 4GB for the whole year. Sure, there will be some workload peaks where his machine will get into a swapstorm and he won't get any work done at all, but that will happen rarely and he can pretend he is on a coffee break when it happens. Maybe the boss won't notice.

Joe starts crafting a grander vision; he realizes that, if he can come up with a way to efficiently allow others' machines that are short on RAM capacity to utilize the RAM capacity on his machine, then the "spread" between temporary offlining and permanent offlining could create a nice RAM market that he could exploit. He could ensure that he always has enough RAM to get his job done, but dynamically "sell" excess RAM capacity to those, like John, who have underestimated their RAM needs ... at say fifteen thousandths of a cent per page-year. If he can implement this "RAM capacity sharing capability" into the kernel MM subsystem, he may be able to turn his machine into a "RAM server" and make a tidy profit. If he can do this ...

Analysis

In the GreenCloud story, we have: (1) a mechanism for offlining and onlining RAM one page at a time; (2) an incentive for using less RAM than is physically available; and, (3) a market for load- balancing RAM capacity dynamically. If Joe successfully figures out a way to make his excess RAM capacity available to others and get it back when he needs it for his own work, we may have solved (at least in theory) the resource optimization problem for RAM for the cloud.

While the specifics of the GreenCloud story may not be realistic or accurate, there do exist some of the same factors in the real world. In a virtual environment, "ballooning" allows individual pages to be onlined/offlined in one VM and made available to other VMs; in a bare-metal environment, the RAMster project provides a similar capability. So, though primitive and not available in all environments, we do have a mechanism. By substantially reducing the total amount of RAM across a huge number of machines in a data center, both capital outlay and power/cooling would be reduced, improving resource efficiency and thus potential profit. So we have an incentive and the foundation for a market.

Interestingly, the missing piece, and where this article started, is that most OS MM developers are laser-focused on the existing problem from the classic single machine world which is, you recall: how to divide up a fixed amount of physical RAM to maximize a single machine's performance across a wide variety of workloads.

The future version of this problem is this: how to vary the amount of physical RAM provided by the kernel and divide it up to maximize the performance of a workload. In the past, this was irrelevant: you own the RAM, you paid for it, it's always on, so just use it. But in this different and future world with virtualization, containers, and/or RAMster, it's an intriguing problem. It will ultimately allow us to optimize the utilization of RAM, as a resource, across a data center.

It's also a hard problem, for three reasons: The first, is that we can't predict, but can only estimate, the future RAM demands of any workload. But this is true today, the only difference is whether the result is "buy more RAM" or not. The second, is that we need to understand the instantaneous benefit (performance) of each additional page of RAM (cost); my math is very rusty, but this reminds me of differential calculus, where "dy" is performance and "dx" is RAM size. At every point in time, increasing dx past a certain size will have no corresponding increase in dy. Perhaps this suggests control theory more than calculus but the needed result is a true dynamic representation of "working set" size. Third, there is some cost for moving capacity efficiently; this cost (and impact on performance) must be somehow measured and taken into account as well.

But, in my opinion, this "calculus" is the future of memory management. I have no answers and only a few ideas, but there's a lot of bright people who know memory management a lot better than I. My hope is to stimulate discussion about this very-possible future and how the kernel MM subsystem should deal with it.

Comments (35 posted)

Patches and updates

Kernel trees

• Greg KH: Linux 3.2.1 . (January 12, 2012)

• Con Kolivas: 3.2-ck1 . (January 16, 2012)

• Greg KH: Linux 3.1.9 . (January 12, 2012)

• Greg KH: Linux 3.1.10 . (January 18, 2012)

• Greg KH: Linux 3.0.17 . (January 12, 2012)

• Greg KH: Linux 2.6.32.54 . (January 12, 2012)

Core kernel code

• Frederic Weisbecker: cgroups: Task counter subsystem v7 . (January 16, 2012)

• Vaidyanathan Srinivasan: sched: unified sched_powersavings tunables . (January 17, 2012)

• Cyrill Gorcunov: fs, proc: Introduce /proc/<pid>/task/<tid>/children entry v6 . (January 17, 2012)

Development tools

• Xiao Guangrong: KVM: perf: a smart tool to analyse kvm events . (January 17, 2012)

• Jiri Olsa: perf tool: parser generator for events parsing . (January 17, 2012)

• Kees Cook: ramoops: use pstore interface . (January 18, 2012)

Device drivers

• Miklos Szeredi: cuse: implement mmap/munmap . (January 16, 2012)

• Bjørn Mork: Adding new drivers for Qualcomm MSM based LTE modems . (January 16, 2012)

• Adam Jackson: drm/vgem: virtual GEM provider . (January 16, 2012)

• Sakari Ailus: [PATCH 0/23] V4L2 subdev and sensor control changes, SMIA++ driver and N9 camera board code . (January 16, 2012)

• Sascha Hauer: Congatec CGEB base, i2c and watchdog driver support . (January 17, 2012)

• Rafael J. Wysocki: PM / Sleep: Introduce new phases of device suspend/resume . (January 17, 2012)

Documentation

• Christoph Hellwig: XFS status update for December 2011 . (January 13, 2012)

• Junio C Hamano: Using signed tag in pull requests . (January 18, 2012)

Filesystems and block I/O

• Andi Kleen: Updated btrfs/crypto snappy interface ready for merging . (January 13, 2012)

• Andrew Ayer: POSIX-compliant version of fchmodat with flag argument . (January 13, 2012)

Memory management

• KAMEZAWA Hiroyuki: [PATCH 0/7 v2] memcg: page_cgroup diet . (January 16, 2012)

• Minchan Kim: low memory notify . (January 18, 2012)

• Leonid Moiseichuk: Memory notification pseudo-device module . (January 18, 2012)

Networking

• Hans Schillstrom: NETFILTER new target module, HMARK . (January 16, 2012)

• Štefan Gula: [patch v1, kernel version 3.2.1] net/ipv4/ip_gre: Ethernet multipoint GRE over IP . (January 17, 2012)

Architecture-specific

• Kelvin Cheung: MIPS: Add support for Loongson1B . (January 18, 2012)

• Uri Yosef: ARM: OMAP4: Add support for Variscite OMAP4460 System-On-Module . (January 18, 2012)

Security-related

• Will Drewry: seccomp_filters: system call filtering using BPF . (January 13, 2012)

• Andy Lutomirski: Add PR_{GET,SET}_NO_NEW_PRIVS to prevent execve from granting privs . (January 13, 2012)

Virtualization and containers

• Raghavendra K T: kvm : Paravirt-spinlock support for KVM guests . (January 16, 2012)

• Wei Liu: New Xen netback implementation . (January 16, 2012)

Miscellaneous

• Lucas De Marchi: kmod 4 . (January 18, 2012)

Page editor: Jonathan Corbet

Distributions

LCA: The past, present, and future of Ubuntu on ARM

Ubuntu started making a distribution for ARM in 2008. This distribution was not based directly on Debian's effort for a simple reason: Debian was built around the ARMv4 architecture, but Ubuntu wanted to target the newer ARMv7. After some work, the first ARM-based release came out in April, 2009; it was targeted at the Freescale iMX51 development board.

Thereafter, ARM releases followed the same six-month schedule as the x86-based Ubuntu releases. The 9.10 release got them up to the ARMv6 instruction set and added support for a Marvell Armada board. 10.04 added support for an OMAP3 development board, could build on ARMv7, and was able to use the Thumb-2 instruction set. The Ubuntu ARM release also rebased itself onto the Netbook Edition at that time. 10.10 added initial device tree support; at this point, Ubuntu had started working with Linaro to improve ARM support in general. With the 11.04 release things started to get a little easier; in particular, TI had pushed support for the OMAP3 processor entirely upstream, so mainline kernels would just work on that architecture. The distribution was also targeting OMAP4 by this point; that support is arriving in the mainline but is not yet entirely there. Finally, 11.04 moved to using Qt for its desktop.

The current release, 11.10, supports the Beagleboard and Pandaboard systems, along with the iMX53 QuickStart board. There is a community-supported build for Toshiba's ARM-based AC-100 netbook system which, David said, is a lot nicer to work with than a development board. Ubuntu has also produced the first technical preview of the upcoming ARM server release.

The team is now working on the 12.04 release, due in April, which should include support for ARM server system-on-chip (SoC) processors from Marvell and Calxeda. The distribution is moving over to the ARMhf architecture, which offers hardware floating support (among other things), leading to a faster system overall. Ubuntu 12.04 will be fully multiarch-clean, which will make it much easier to do cross-development and testing on x86-based systems.

Life has been getting easier over time as the team gains experience and many of the software problems are hunted down and fixed. Creating the first ARMel archive, with 25,000 packages, took about three months; the ARMhf archive, instead, with 37,000 packages, was done in about three weeks. Lots of build issues have been overcome, and problematic assumptions in the code (such as "ARM systems will never be SMP") have been fixed.

Looking forward, David clearly sees a bright future for the ARM server architecture. That may be surprising to some; ARM has not generally been known as an architecture for servers, but that may be about to change. Power consumption is an increasingly important issue for data centers; many of them, at this point, have reached a point where no more power can be brought in and, thus, cannot accommodate any more systems. The industry spends some $45 billion per year on data center power and cooling. Anything that can reduce power consumption in this setting is welcome.

ARM is known for low power consumption, an attribute that makes it attractive in this situation. The architecture is adding a number of other features of interest to the server use case; we will be seeing 64-bit, multi-core processors with 2GHz clocks and support for virtualization. It will be possible to buy a quad-core SoC that can run flat-out with 5W of power, far less than any x86 chip. ARM servers, thus, have the potential to handle the server role while being much cheaper to buy and run.

So ARM servers may well find considerable success. Part of David's job, naturally, was to convince us that we would want to run Ubuntu Server on those ARM systems. It makes sense, he said, because Ubuntu has that experience with ARM going back to 2008; they have run into all the issues and know how things work. They have a distribution built natively for the Cortex A9 multi-core processor, ready to handle SMP workloads. And a lot of these systems will be running workloads like web serving - the sort of thing that Ubuntu has traditionally been good at.

Additionally, ARM-based server systems are going to bring some interesting system administration challenges. The density of these systems will be impressive, allowing the installation of thousands of servers into a single rack. Managing all of those systems - even at the level of ensuring they all get IP address when they boot - is not going to be straightforward. David thinks that Ubuntu's Juju configuration management system, originally developed for cloud deployments, will be well suited to the ARM server environment.

Looking forward, David anticipates that the first actual server release, targeting the Calxeda and Marvell SoCs, will happen in October. That will make Ubuntu the first operating system to support the ARM Cortex A15 processor. 12.10 will include support for virtualization with KVM or Xen. It will be a 32-bit release (64-bit work is getting started but is not ready yet), but it will support large amounts of physical memory using the (just merged for the 3.3 kernel) large physical address extension feature. Some of this support may also go into the 12.04 LTS release in later updates.

In 2013, the range of supported hardware will grow; there will also be support for the Cortex A7 processor and its interesting power management technologies. In 2013, he said, Ubuntu will be the first system to support the 64-bit ARMv8 architecture. UEFI boot will also be supported. Looking forward to 2014 and beyond, there will be a serious effort to create a unified kernel; running a single kernel on any ARM processor is a challenging goal, but it should be able to run on a wider range of processors than any kernel available today.

All told, it is an ambitious undertaking; ARM is not an architecture that has been congenial to general-purpose distributions in the past. But, if the ARM folks have their way, we may be about to see a significant change in the server market; Ubuntu clearly plans to be there if and when that happens.

[Your editor would like to thank the LCA organizers for assisting with his travel to Ballarat.]

Comments (3 posted)

Brief items

FreeBSD 9.0 released

Comments (4 posted)

PC-BSD 9.0 Released

Comments (none posted)

Newsletters and articles of interest

Distribution newsletters

• DistroWatch Weekly, Issue 439 (January 16)
• Maemo Weekly News (January 16)
• Ubuntu Weekly Newsletter, Issue 248 (January 15)

Comments (none posted)

Page editor: Rebecca Sobol

Development

Multi-touch support landing in X

X.Org 1.12, the next release of the reference X server, is currently in release candidate status. With it comes several new features, but the most anticipated is probably multi-touch support, courtesy of version 2.2 of the XInput2 extension. Peter Hutterer, maintainer of XInput2, has been writing about adding multi-touch support in his blog since December 2011 — including the architecture and what application developers will need to address before they can bring multi-touch and gesture support to users.

Of course, any discussion of multi-touch X begins by explaining what it is and what it isn't. Multi-touch refers specifically to the ability to recognize and use multiple input points on a single hardware device. For example, using more than one finger to manipulate objects on a touchscreen device, or multi-finger gestures on a touchpad. It is a different animal entirely from multi-pointer X (MPX), which is the ability to use two or more on-screen cursors at the same time, controlled by separate hardware devices. MPX support was added — also by Hutterer — to XInput2 in 2009, and was first released with X.Org 1.7.

Touchpad users are probably already familiar with two-finger scrolling and two- or three-finger mouse clicks. Although detecting multiple points-of-contact is involved, this is also not genuinely multi-touch — detecting the multiple taps or scrolling simply triggers a different event from the touchpad driver. A simple way to tell the difference is that with two-finger middle-clicks, the position of the user's fingers do not matter; the cursor stays (more or less) in one place. With a multi-touch gesture event like a pinch, however, tracking and interpreting the motion and relative positions of the fingers makes all the difference in the world.

Devices and events

XInput2.2 defines two distinct classes of multi-touch device that correspond to the two major modes of multi-touch user interaction. An XIDirectDevice is one where the touch event occurs on screen (as is the case with tablets, touch-screen monitors, and the like). In these cases, the coordinates where the event happens come directly from the position of the touch. The other class is an XIDependentDevice, which is typically a non-display input device like a touchpad. An XIDependentDevice controls a cursor in the "normal" fashion most of the time, but supports multi-touch events, too. The positions of the touch events on an XIDependentDevice are interpreted relative to the cursor position.

XInput2.2 also defines three event types that together describe a touch sequence in the wild: XI_TouchBegin, XI_TouchUpdate, and XI_TouchEnd. By definition, each touch event starts with an XI_TouchBegin, followed by zero or more XI_TouchUpdates, and ends with an XI_TouchEnd. Client applications that want to catch touch events must use the XISelectEvents method to register for all three event types.

To a client application listening for the new event types, touches appear no different from any other XInput event — with the addition of a "touch ID" that is returned in the event's detail field. This ID is a 32-bit value that the application must use to keep track of multiple, simultaneous touches. The touch events use the XIDeviceEvent structure that is also used for pointer and keyboard events; handling touches separately (including interpreting gestures) is left up to the application, library, or toolkit.

Device grabbing and ownership

Although the Begin, Update, and End events cover almost all touch event cases, there is a fourth touch event called XI_TouchOwnership defined by XInput2.2 in order to provide no-delay touch event processing in unusual situations.

The need arises because it is not always unambiguously clear which X client ought to process a series of touch events. For example, on a tablet, a window manager and an image viewer might both reasonably be expected to listen for touch events on the same window — the user could click and drag the window, or perhaps pinch and zoom in on the image. XInput allows a client to stake out a claim to the events that happen in a window by "grabbing" touch events.

Calling the grab API tells the X server that the grabby client wants exclusive access to the input device. But a client application that holds a grab on a device could begin to process a touch event sequence and only then discover that the sequence is not intended for it. To follow up on the example above, the image viewer might grab a touch event sequence, then process it and determine that it is not a gesture it recognizes. In that case, the grabby client sends a reject to the server, and the server then sends the sequence to the next client (i.e., the window manager), which hopefully does recognize the gesture and can react accordingly. When a client decides that it does want to process the touch sequence, it sends an accept instead.

That process works reasonably well when only one or two clients are involved, but every client that studies and rejects the gesture adds lag time, which is exacerbated because the X server re-sends the touch sequence to each subsequent client. If there are several involved, the UI suddenly becomes sluggish to the user. XInput2.2's solution is the XI_TouchOwnership event. A client can select to listen for this event in addition to the usual touch sequence events. When it does so, the X server sends a copy of each event sequence to the client immediately — even if another client has a grab on the device.

The result is that the client listening for XI_TouchOwnership can begin to process touch events immediately and lower its response times. However, because one of the other clients may accept the touch event sequence, whatever action the listening client takes needs to be either invisible and reversible, or hold off on execution until the X server tells the client that its turn has come by sending it an XI_TouchOwnership event.

Toolkit and application support

As the grabbing/ownership scenario illustrates, multi-touch support in X is difficult precisely because X handles multiple active applications running simultaneously. In 2010, Hutterer observed that this was a higher bar than that faced by Apple iOS or Microsoft's Surface project, both of which function in pre-defined environments with only a single full-screen application active at any one time. X is also expected to make multi-touch devices co-exist peacefully with keyboards and standard pointers, which touch-only devices generally do not support.

The result is that XInput2.2's multi-touch support arrives via a separate set of input device types and events, that applications will need to add support for manually. Consequently it may be several release cycles after X.Org 1.12 before window managers, GIMP, Firefox, and the rest fully support the new multi-touch features.

GUI Toolkit support is closer. Qt already has a stable multi-touch API that supports gestures, introduced in version 4.6. Ubuntu recommended Qt to application developers in 2011, while it maintained its own uTouch framework for the Unity environment during the XInput2.1 development cycle. Although uTouch included a patchwork of components in the past, the distribution is migrating it to XInput2.2 for its 12.04 release.

That leaves the GNOME side of field. Carlos Garnacho and others are working on adding multi-touch support to GTK+ and GDK in the multitouch branch. However, it is not clear when this work is expected to land. Garnacho noted his progress in his blog twice in early 2011, but there have been no formal updates there or on the mailing lists for months. Hutterer indicated on the Fedora wiki's multi-touch page that support may land as soon as GTK+ 3.4. Interested parties can keep up to date by watching the GNOME Shell Touchscreen wiki page.

The essential building blocks are lined up (if not all in place) for true multi-touch on X.org systems, so it is plausible that before 2013 arrives, desktop and mobile Linux users will consider multi-touch applications commonplace. But that does not mean that they will be happy about it. As Hutterer mentions on his blog, one of the most challenging aspects of multi-touch is coming up with good, intuitive user interface designs to fit together with the technical underpinnings. On that front, 2012 may also be the year of multi-touch experimentation.

Comments (4 posted)

Brief items

Quotes of the week

Comments (13 posted)

Cheese 3.3.4 released

Full Story (comments: none)

Media Goblin 0.2.0 released

Full Story (comments: none)

Salt 0.9.5 released

Comments (1 posted)

Newsletters and articles

Development newsletters from the last week

• Caml Weekly News (January 17)
• Perl Weekly (January 16)
• PostgreSQL Weekly News (January 15)
• Tahoe-LAFS Weekly News (January 14)

Comments (none posted)

Organizing Open Source Efforts at NASA (Linux.com)

Comments (none posted)

Kruisselbrink: Calligra on Android

Comments (2 posted)

Typing at 255 WPM shouldn't cost $4000: Plover, the open source steno system (Opensource.com)

Comments (23 posted)

Raghavan: PulseAudio vs. AudioFlinger: Fight!

Comments (28 posted)

Page editor: Jonathan Corbet

Announcements

Articles of interest

SFLC: Microsoft confirms UEFI fears, locks down ARM devices

Comments (43 posted)

Garrett: Why UEFI secure boot is difficult for Linux

Comments (41 posted)

Web Protests Piracy Bills, and 2 Senators Change Course (New York Times)

Comments (10 posted)

Second batch of FOSDEM 2012 speaker interviews

• Michael Meeks (LibreOffice)
• Carsten Munk (mobile Linux)
• Kir Kolyshkin (OpenVZ)
• Robert Dewar (commercial success)
• Carsten Haitzler (EFL).

Comments (none posted)

Google Code-in 2011-2012 Concludes

Comments (none posted)

New Books

Web Development Recipes--New from Pragmatic Bookshelf

Full Story (comments: none)

Calls for Presentations

PGCon 2012 Call for Papers

Full Story (comments: none)

2nd CFP: Open64 Workshop at PLDI'12

Full Story (comments: none)

Upcoming Events

SUSE Announces First Annual Enterprise Linux Conference

Comments (none posted)

SuperCollider 2012 program announced

Full Story (comments: none)

Events: January 19, 2012 to March 19, 2012

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol