LWN.net Weekly Edition for March 8, 2012

Getting multiarch dpkg into Debian

"Multiarch" is Debian's name for its approach to supporting multiple architecture ABIs on a single system; LWN recently covered the multiarch work in some detail. Multiarch support is one of the goals for the upcoming "wheezy" release; since that release is getting closer to its freeze time, Debian developers naturally want to have all the necessary code merged into the distribution and ready for testing. A disagreement over the uploading of one of the crucial pieces of the multiarch puzzle recently threatened that goal and provided an interesting view of the higher levels of the Debian project's governance mechanisms in action.

Clearly, a functioning multiarch implementation will require extensive support from the packaging mechanism. To that end, a number of developers have been working on extending the dpkg utility for some time; patch sets adding multiarch support to dpkg were posted as early as 2009. Much of this work has been supported by Canonical and Linaro, so it is not entirely surprising that it shipped first with the Ubuntu distribution; the 11.04 release included basic multiarch support. Until very recently, though, the necessary dpkg changes had not found their way into Debian - not even into the experimental repository.

A number of attempts have been made over time to push the dpkg changes into experimental. The holdup each time appears to have been the same: dpkg maintainer Guillem Jover blocked the push until he could do a full review of the relevant code. The problem, it seems, is that this review never happened; Guillem is a busy developer and has not been able to find the time to do that work. So the multiarch-compatible dpkg remained out of the repository; only those willing to build it from source were able to test the multiarch changes. Playing with fundamental packaging system changes is scary enough without having to build the tools, so this situation can only have led to less testing of multiarch support in Debian as a whole.

In October, 2011, Philipp Kern, representing the release team, expressed concern about the delay, saying:

Philipp requested that the new dpkg find its way into experimental immediately, with a move to unstable two weeks later. But things did not happen that way; instead, Guillem blocked another attempt to upload the work into experimental. Debian project leader Stefano Zacchiroli responded with some concerns of his own:

Guillem did not respond to this message until March 3, 2012 - over four months after it was sent. Meanwhile, the dpkg work languished outside of the Debian repositories. In January, 2012, Cyril Brulebois pushed the new dpkg using the Debian non-maintainer update (NMU) process. An NMU can be a way to route around an unresponsive or distracted maintainer, but the process has its limits; in this case, Guillem simply reverted the NMU, leaving the situation as it was before.

At the beginning of February, Stefano apparently decided that the situation had gone on for too long. The powers of the project leader are famously limited, though; there was nothing Stefano could do on his own to force a solution to the problem. The Debian Technical Committee, however, is a different story; it is empowered by the Debian Constitution as the final decision maker in the case of otherwise unresolvable technical disputes. So Stefano referred the problem to the committee, asking that it decide whether one of the dpkg maintainers can block progress indefinitely in such a situation.

The committee did not need long to reach a decision; it concluded unanimously that Guillem's blocking of the dpkg upload should be overridden. Guillem was given a set of deadlines by which to get the code into experimental (then unstable); if those deadlines were not met, one of the other dpkg maintainers (Raphaël Hertzog) was given the power to do the upload instead. As it happens, Guillem met the deadline and Debian now has a multiarch-capable package manager. This move has spawned another massive email thread - but this is Debian we're talking about, so it's really just business as usual at this point. Multiarch in wheezy is back on track.

While the Technical Committee resolved this particular conflict, it did not (and probably could not hope to) resolve the larger question. Debian, like many free software projects, tries to make its decisions by consensus whenever possible. But the project also gives developers something close to absolute power over the packages that they maintain. Occasionally, those two principles will come into conflict, as was the case here. Resolving such conflicts will never be easy.

In this case, everybody was working toward the same goals. Even those who were most critical of Guillem's behavior stopped far short of suggesting that he was deliberately trying to delay the multiarch work. He was simply trying to ensure the highest level of technical excellence for a package he maintains - a package that is critical to the functioning of the distribution as a whole. The problem is that he ran afoul of a project that has been trying - with significant success - to bring a bit more predictability to the release process in recent years. Overriding him was probably necessary if the release goals were to be met, but it must still feel harsh to a developer who has put a lot of time into improving the project's core infrastructure.

A large project like Debian, in the end, will have occasional conflicts involving developers who, for whatever reason, are seen as holding up important changes. So there needs to be some sort of mechanism for avoiding and dealing with such blockages. Debian's approach, which includes having teams of developers (instead of a single person) working on important packages and the Technical Committee as a court of last resort appears to work pretty well. The benevolent dictator model used by a number of other projects can also be effective, depending on the quality of the dictator. In the end, our community does not often have serious problems in this area; we are able to manage the interactions of thousands of dispersed developers with surprisingly little friction. But it's good to know that these conflicts can be resolved when they do arise.

Comments (30 posted)

The Perl 5 release process

LWN recently published an article, "The unstoppable Perl release train?", that raised some questions about the Perl 5 release process. In particular, the article questioned how the Perl 5 Porters handled a recent bug report from Tom Christiansen about Unicode support. This article aims to provide the reader with a better understanding of the current Perl release process, how it benefits Perl 5 users, and how it has helped invigorate Perl 5 development.

First, a disclaimer. I am decidedly pro-Perl. I am paid to program in Perl, and I've been doing so for over ten years now. I've contributed to the Perl core, mostly in the form of documentation, and I was also the release manager for the December 2011 development release of Perl, Perl 5.15.6.

Who are the Perl 5 Porters?

I'll be mentioning this group regularly throughout the article, so you should know who they are. The Perl 5 Porters are the volunteers who develop and maintain the Perl 5 language. They consist of anyone who cares enough about Perl 5 to participate in discussions on the perl5-porters email list. There's no formal membership, and influence is based on a combination of hacking skill and the ability to not be a jerk. The more you contribute and the more you communicate, the more control you have over the future of Perl 5. This should sound familiar if you know how other FOSS projects work.

While there are many people on the list, there are not that many people who actively contribute to the Perl core's C code. Currently, there are two paid developers, Nicholas Clark and Dave Mitchell, who are funded by grants from the Perl Foundation. In addition, there are usually another two to six people actively hacking on the C core at any time. Many more people contribute in the form of small patches, documentation, tests, and updates to core Perl modules.

Perl 5 and Perl 6

There's a reason this article's title specifically says Perl 5. Larry launched the Perl 6 project over ten years ago. When it was first being discussed, many Perl users thought that Perl 6 would supplant Perl 5 in much the same way that Perl 5 supplanted Perl 4.

Since then, that expectation has changed. Perl 5 and Perl 6 are both being actively developed, and there's no expectation that Perl 6 will replace Perl 5 any time soon. We now say that both Perl 5 and Perl 6 are part of the Perl language family.

The two languages co-exist peacefully, and ideas cross-pollinate between the two communities regularly.

A brief history of Perl 5

Larry Wall released Perl 1 in December 1987. He released Perl 5.000 in October 1994. Perl 5 is the Perl most of us know and love (or hate) today. It introduced features like object-orientation, the ability to dynamically load extensions written in C (using the "XS" mechanism), modules, closures, and more.

After the Perl 5.000 release, other Porters started making releases of Perl. Larry was no longer the sole keeper of Perl, and Perl development adopted the notion of a "Pumpking". Initially, the Pumpking (there has not been a Pumpqueen yet, unfortunately) was the person responsible for shepherding the development of each stable release. However, Larry was still Benevolent Dictator for Life, and Perl development still followed Rules 1 and 2, which can be paraphrased as "Larry has the final say on Perl, and can contradict himself whenever he feels like it."

As Larry started spending more time on Perl 6 development, he became less involved with Perl 5. These days, his focus is almost exclusively on Perl 6. While Rules 1 and 2 still apply in theory, in practice Larry does not make decisions involving Perl 5. Instead, the Pumpking is in charge of the language. The first Pumpking to take on this responsibility was Jesse Vincent. Recently, Ricardo Signes took over the Pumpking position.

For a long time Perl's release schedule was "release it when it's done". There was no specific list of features for a given release, and it wasn't always clear what should block a release from going out. In some cases, major releases ended up coalescing around a big feature, like threads (5.6.0) or Unicode (5.6.0 and 5.8.0), and that drove the release. In other cases, releases were repeatedly delayed due to a vicious cycle of adding a feature, fixing its bugs, and then adding a new feature during the bug fixing stage.

The development process also failed to clearly distinguish between major and minor releases. There was a nearly two year gap between the release of 5.10.0 and 5.10.1. In hindsight, the 5.10.1 release was not a minor release, and deserved to be called 5.12.0.

The current release process and support schedule

After the long gap between 5.10.0 and 5.10.1, many Porters wanted to streamline the development process. When Jesse Vincent took over as Pumpking for Perl 5.12.0, the Porters adopted a "timeboxed" release schedule. The theory is that in April of each year, the Pumpking will package up the code in the "blead" branch (Perl's name for "master") and ship it as the new major stable release.

Each major release is followed by a series of minor releases. Minor releases are minor! They include a small number of changes that are focused on fixing the most critical bugs. New features or incompatibilities are never introduced in a minor release.

Each major release series is supported for two years, except for critical security fixes, which are supported for three years. David Golden has a great write up of this support policy on his blog.

All release numbers start with "5.". The second number represents the major version number. The last number is the minor release number. Even major version numbers indicate stable releases. Odd major versions indicate development releases (e.g. 5.11.x). Minor development releases are made each month for community review and testing.

Minor stable releases are always backward compatible with other minor releases in the same stable series, down to binary compatibility for XS (C-based) extensions. Major releases may break backward compatibility, but Perl 5 is conservative, and backward incompatibilities are not introduced lightly.

The first release to follow this schedule was Perl 5.12.0, released on April 12, 2010.

Aiming for stability

The release schedule includes three well-defined freeze points. The first is the "contentious" code freeze, which happens in December before the stable release. This means that any changes that don't have widespread agreement from the Porters at this time must wait until after the stable release to be merged. The next freeze point is for user-visible changes, such as API changes, new features, backward incompatibilities, etc. This freeze occurs in February. In March, a "full" code freeze goes into effect in order to finalize the April Release Candidate and subsequent stable release.

There's a lot of work that goes into making this possible. While the number of people with commit bits has expanded, there is much stronger pressure to do work in per-feature or per-bugfix branches. This lets blead stay stable enough for releases to actually happen on schedule.

The Perl release train is only somewhat unstoppable. It stops for release blocking issues. Deciding whether an issue should block a release is more of an art than a science. Some questions that can help determine whether an issue is a blocker are:

• Is the issue a regression as compared to the last stable release of Perl?
• Does the issue represent an unintentional break in backward compatibility?
• Does the issue break a huge number of CPAN modules, or a few really important ones?
• Does the issue cause a major performance problem with commonly occurring code?
• Does the issue break the ability to install modules from CPAN, or the ability to install Perl itself?
• Is someone willing to commit to fixing this before the scheduled release?

The exact decision as to whether an issue is a blocker is made by the Pumpking in consultation with the Porters. Typically, a few months before the stable release, the Pumpking will ask the Porters to trawl through the Perl 5 issue tracker and nominate issues for blocker status. After discussion on the mailing list, the Pumpking comes up with a "final" list of issues. This list isn't really final. If someone found a major regression the day before the release, that could be declared a blocker.

So why weren't the bugs that Tom Christiansen found declared to be release blockers? This is answered by the first question above. These bugs exist in the current stable release of Perl. They are not regressions. Releasing Perl 5.16.0 with these bugs does not make Perl's Unicode support any worse than it already is. However, delaying Perl 5.16.0 does mean that users would not have access to many other useful bug fixes and features.

Timeboxed releases ensure that users can expect improvements and bug fixes on a predictable schedule, without the long gaps that were common in Perl's earlier history.

As for the security bug discussed in the same thread, a fix is being worked on. It's possible that the fix will be ready for the 5.16.0 release. If it's not ready in time for 5.16.0 then there will be a 5.16.1 release when the fix is ready, as well as 5.14.3 and 5.12.5 releases.

Perl 5 has come a long way in the last few years. Every month, a new development release comes out, shepherded by a different release manager. Once a year, a new major version is released. Releasing a new major version is no longer a months-long ordeal. When the new release schedule was first proposed, many of us wondered if such an ambitious plan could work. Now, it almost seems routine.

Comments (2 posted)

The life story of the XInput multitouch extension

The XInput multitouch extension provides for multitouch input events to be sent from the device to the appropriate window on the desktop. Multitouch events can then be used for gesture recognition, multi-user interactions, or multi-point interactions such as finger painting. While the general concepts behind delivering multitouch events through a window server are fairly well defined, there are many devils hiding in the details. Here, we provide a look into the development of the multitouch extension and many of the issues encountered along the way.

Motivations

For Henrik Rydberg, it began as an attempt to make the trackpad on his Apple Macbook work just as well on Ubuntu as it does on OS X. For Stéphane Chatty, it began as a research project to develop new user interface paradigms. For your author, it began as a quest to enable smooth scrolling from an Apple Magic Mouse.

Like many undertakings in open source, multitouch on the Linux desktop is the culmination of the many efforts of people with disparate goals. With the release of the X.org server 1.12, we now have a modern multitouch foundation for toolkits and applications to build upon.

The kernel

The beginning of multitouch support for Linux can be traced back to the 2.6.30 kernel. Henrik had just merged additions to the evdev input subsystem for multitouch along with support for the trackpads found in all Apple unibody Macbooks. Stéphane then added multitouch support to some existing Linux drivers, such as hid-ntrig, and some new drivers, such as hid-cando.

Some developers started playing around with the new Linux multitouch support. Over time, the libavg and kivy specialized media and user interface toolkits added multitouch support based on the evdev interface. However, there was a glaring issue: the absence of window-based event handling. Applications had to assume that they were full-screen, and all touch events were directed to them exclusively. This was a fair assumption for games, which was the main impetus for libavg touch support. However, it was clear we needed to develop a generic multitouch solution working through the X window system server.

The X.org server and the X gesture extension

Discussions began on how to incorporate touch events into the X input model shortly after kernel support was present. Initial work by Benjamin Tissoires and Carlos Garnacho extended XInput 2.0's new multi-device support for multitouch. Each time a touch began, a new pointing "device" would be created. Alternatively, a pool of pre-allocated touch "devices" could be used. However, this approach broke many assumptions about how devices and events should be handled. As a simple example, a "touch begin" event would appear to the client as though the pointer had moved to a new location. How would the client know that the previous touch hadn't simply moved, as opposed to a new touch starting? At this point Peter Hutterer, the X.org input subsystem maintainer, decided we needed completely new semantics for touch input through X.

Around the same time, Canonical was interested in adding multitouch interfaces to the Linux desktop. The uTouch team, of which your author is a member, was formed to develop a gesture system that could recognize and handle system-level and application-level gestures. Since X did not have touch support yet, the team focused on providing gestures through the X.org server using a server extension. The result was shipped in Ubuntu 10.10 and the extension was proposed for upstream X.org.

While many developers were enthusiastic about the potential for gesture support through the X.org server, it was not meant to be. X.org as a foundation holds backward compatibility in high regard. Applications written over 20 years ago should still function properly today, in theory. Though backward compatibility has benefits, it is a double-edged sword. Any new functionality must be thoroughly reviewed, and inclusion in one X.org release means inclusion in all future releases. Even to this day, gesture support is not a settled technology. It is highly probable that an X gesture extension created a year and a half ago would not be sufficient for use cases we are coming up with today, let alone potentially years from now. So the X developers are reluctant to include gesture support at this time.

XInput multitouch was born

Those concerns notwithstanding, the need for touch through the X server grew stronger. Peter and Daniel Stone developed a first draft of the XInput 2.1 protocol, which later became XInput 2.2, where touches send separate events from traditional pointer motion and button press events. Three event types ("touch begin," "update," and "end") were specified. However, the need to support system-level gestures added a requirement for a new method of event handling: the touch grab.

X11 input device grabs allow for one client to request exclusive access to a device under certain criteria. A client can request an active grab so that all events are sent to it. A client can also request a passive grab, where events are sent to it when a button on the mouse is pressed while the cursor is positioned over a window, or when a key is pressed on a keyboard while a window is focused. Passive grabs allow for raising a clicked window in a click-to-focus desktop environment, for example. When the user presses the mouse button over a lower window, the window manager receives the event first through a passive grab. It raises the window to the top and then replays the event so the application can receive the button press. However, X only allows for a passively grabbing client to receive one event before it needs to make a decision on whether to accept it and all future events until a release event, or to request that the server replay the event to allow another client to receive the events.

This mechanism has been adequate for decades, but doesn't quite work for system-level gestures. Imagine that the window manager wants to recognize a three-touch swipe. It is impossible to know if a three-touch swipe has been performed if the window manager can only view touch begin events; it must be able to receive the subsequent events to determine whether the user is performing a swipe or not. The idea behind touch grabs is that the grabbing client can receive all events until it makes a decision about whether to accept or reject a touch sequence. Now, the window manager can listen for all touches that begin around the same time and watch them as they move. If there are three touches and they all move in the same direction, the window manager recognizes a drag gesture and accepts the touch sequences. No one else will see the touch events. However, if the touches don't match for any reason, the window manager rejects the touch sequences so other clients, such as a finger painting application, can receive the events.

This works great for system-level gesture recognition. However, it necessarily imposes lag between a physical touch occurring and an application receiving the touch events if the the system is attempting to recognize gestures. At the X Developer Summit 2010, your author presented an overview of the vision for an XInput multitouch-based uTouch gesture stack. One afternoon, while eating lunch and discussing things over beer, the issue of the potential for lag came up. Between those at the table, including Peter, Kristian Høgsberg, and myself, the solution was elusive. However, at some point later in the conference the issue came up again on IRC. Keith Packard made the suggestion that touch events be sent to all clients, even before they become the owner of touch sequences. With the idea at hand, your author scurried home and drafted up the beginning of what would later become ownership event handling.

As Nathan Willis explained in his overview of the XInput 2.2 protocol, a client may elect to receive events for a touch sequence before it becomes the owner of the sequence by requesting touch ownership events alongside touch begin, update, and end events. The client will receive touch events without delay, but must watch for notification of ownership. Once a touch ownership event is received for a sequence, the client owns the sequence and may process it as normal. Alternatively, if a preceding touch grab is accepted, the client will receive a touch end event for the touch sequence without ever receiving a touch ownership event. This mechanism allows for a client to perform any processing as touch events occur, but the client must take care to undo any state if the touch sequences are ultimately accepted by some other client instead.

With the basic concepts hammered out, your author, with an initial base of work contributed by Daniel Stone, began a prototype implementation that shipped in Ubuntu 11.04 and 11.10. The uTouch gesture system based around XInput multitouch began to take form. This was enough to prove that the protocol was reasonably sound, and efforts began in earnest on an upstream implementation for the X.org server 1.12 release.

It is interesting to note how XInput multitouch compares to other window server touch handling. On one end of the spectrum are phones and tablets, which run most applications full screen. This, and the lack of support for indirect touch devices, e.g. touchpads, means mobile device window manager multitouch support is much simpler. On the other end of the spectrum are desktop operating systems. Windows 7 shipped with multitouch support, but only for touchscreens. For an unknown reason, Windows also only supports raw multitouch events or gesture events on a window, but not both. As an example of the consequences of this shortcoming, Qt had to build their own gesture recognition system so it could support both raw multitouch events and gestures at the same time. OS X only supports touchpads, but this simplification alone ensures that touches are only ever sent to one window at a time. The event propagation model they chose would not work for touchscreens. In comparison, the XInput multitouch implementation allows for system- and application-level gestures and raw multitouch events at the same time across both direct and indirect touch devices. In your author's biased opinion, this is a key advantage of Linux on the desktop.

A few bumps in the road

Although development of multitouch through X took more time than anyone wanted, it was shaping up well for the 1.12 X.org server release. Many complex issues, such as pointer emulation for touchscreens, were behind us. However, touchpad support had yet to be finalized. Two large issues surfaced involving scrolling and other traditional touchpad gestures.

The first issue involved the ability to scroll in two separate windows while leaving one finger on the touchpad at all times. Imagine there are two windows side by side. The user positions the cursor over one window and begins a two-touch scroll motion on the trackpad. The user then lifts one finger and uses the remaining finger to move the cursor over the second window. The second finger is then placed on the trackpad again, and a second scroll motion is performed. Under the XInput multitouch protocol, a touch sequence is locked to a window once it begins. If two-touch scrolling is performed through gesture recognition based on XInput touch events, the touch that began over the first window could not be used for a scroll gesture over the second window because the touch events would remain locked to the first. In order to resolve this difficulty, it was decided that, when only one touch is active on a touchpad, no touch events are sent. In order to not send two events for one physical action, pointer motion was also prevented when more than one touch was present on a touchpad.

This fix resolved pointer motion, but other traditional touchpad gestures are even more problematic. Particularly troublesome is two-finger scrolling. When mice with scroll wheels were first introduced, they had discrete scroll intervals. The wheels often clicked up and down. This led to an unfortunate API design for scroll events in the X server. The X core protocol cannot send pointer events with arbitrary values, such as a scroll amount. To provide for scrolling through the X core protocol buttons 4, 5, 6, and 7 were redefined from general purpose buttons to scroll up, down, left, and right. When the user scrolls up using a scroll wheel, the X server sends a button 4 press event and then a button 4 release event. As an aside, this is the reason why we don't yet have smooth scrolling on the Linux desktop.

The problem for multitouch lies in the possibility of sending two separate events for one physical action. This would occur if we sent touch events at the same time we sent scroll button events. It was decided that touch events may not be sent while the server is also sending other events derived from touches. This means that if the user enables two-finger scrolling, touch events are inhibited unless three touches are active on the touchpad. Likewise, if the user performs a two-finger tap to emit a right click, touch events are also inhibited unless three touches are active on the touchpad, and so on.

Many workarounds were considered, but nothing provided an air-tight solution. The double-edged sword of backward compatibility prevents X from supporting scroll events, click emulation, and touch events at the same time. Your author hopes this situation will end up hastening support for traditional trackpad gestures on the client side of X instead of the server side.

Wrapping up

The development of the multitouch extension finished with the release of the X.org server 1.12 on March 5th, 2012. Many upcoming distribution releases, including Ubuntu 12.04 LTS, will be shipping it soon. Although this is the end of the X.org multitouch story, it is only the beginning for toolkits and applications. GTK+ recently merged an API for handling raw touch events for 3.4, and your author hopes to merge raw touch support for Qt in the near future. Next on the roadmap will be gesture support included in standard toolkit widgets and APIs for application developers. There is still plenty of work to do, but the will of those hoping to bring smooth scrolling support to the Apple Magic Mouse and many other multitouch features is quite strong.

Comments (35 posted)

Page editor: Jonathan Corbet

Inside this week's LWN.net Weekly Edition

• Security: Github incident spawns Rails security debate; New vulnerabilities in imagemagick, kernel, lightdm, python-sqlalchemy, ...
• Kernel: Kernel memory usage accounting; Statistics for the 3.3 development cycle; The x86 NMI iret problem.
• Distributions: MINIX 3.2: A microkernel with NetBSD applications; Fedora, Kubuntu, Oracle Linux, ...
• Development: CUPS 1.6 shaking up Linux printing; PHP, Python, WINE, ...
• Announcements: OIN Expands Linux System Definition; Raspberry Pi; LPI; Cfp: LAC, Flossie, EuroPython, and LSM.

Next page: Security>>