"Multiarch" is Debian's name for its approach to supporting multiple
architecture ABIs on a single system; LWN recently covered the multiarch work
in some detail.
Multiarch support is one of the goals for the upcoming "wheezy" release;
since that release is getting closer to its freeze time, Debian developers
naturally want to have all the necessary code merged into the distribution
and ready for testing. A disagreement over the uploading of one of the
crucial pieces of
the multiarch puzzle recently threatened that goal and provided an
interesting view of the higher levels of the Debian project's governance
mechanisms in action.
Clearly, a functioning multiarch implementation will require extensive
support from the packaging mechanism. To that end, a number of developers
have been working on extending the dpkg utility for some time;
patch sets adding multiarch support to dpkg were posted as early as 2009. Much of this work
has been supported by Canonical and Linaro, so it is not entirely
surprising that it shipped
first with the Ubuntu distribution; the 11.04 release included basic
multiarch support. Until very recently, though, the necessary
dpkg changes had not found their way into Debian - not even into
the experimental repository.
A number of attempts have been made over time to push the dpkg
changes into experimental. The holdup each time appears to have been the
same: dpkg maintainer Guillem Jover blocked the push until he
could do a full review of the relevant code. The problem, it seems, is
that this review never happened; Guillem is a busy developer and has not
been able to find the time to do that work. So the multiarch-compatible
dpkg remained out of the repository; only those willing to build
it from source were able to test the multiarch changes. Playing with
fundamental packaging system changes is scary enough without having to
build the tools, so this situation can only have led to less testing of
multiarch support in Debian as a whole.
In October, 2011, Philipp Kern, representing the release team, expressed concern about the delay, saying:
If we want to carry on with multiarch in wheezy it really should
enter the archive now. The freeze deadline is approaching fast and
we must be able to shake out all the bugs in the time that's left.
Deprecating ia32-libs in a single cycle seems to be quite a bit of
Currently nobody can test multiarch with in-archive software. The
multi arch patches did not even land in experimental, despite some
pokes from fellow project members.
Philipp requested that the new dpkg find its way into experimental
immediately, with a move to unstable two weeks later. But things did not
happen that way; instead, Guillem blocked
another attempt to upload the work into experimental. Debian project leader
Stefano Zacchiroli responded with some
concerns of his own:
What worries me is that there is multi-arch work in dpkg, work that
has its origins in Debian. That work is ready enough to be deployed
in popular Debian derivatives such as Ubuntu, but is not in Debian
proper yet. That is bad for Debian morale and should be
avoided. Moreover, that work is also considered ready enough by
other dpkg co-maintainers, by the Release Team, and by various
porters, which have all asked multiple times to have that work in
the Debian archive.
Guillem did not respond to this message until March 3, 2012 - over four
months after it was sent. Meanwhile, the dpkg work languished
outside of the Debian repositories. In January, 2012, Cyril Brulebois pushed the new dpkg using the Debian
non-maintainer update (NMU) process. An NMU can be a way to route around
an unresponsive or distracted maintainer, but the process has its limits;
in this case, Guillem simply reverted the NMU, leaving the situation as it
At the beginning of February, Stefano apparently decided that the situation had gone
on for too long. The powers of the project leader are famously limited,
though; there was nothing Stefano could do on his own to force a solution
to the problem.
The Debian Technical
Committee, however, is a different story; it is empowered by the Debian
Constitution as the final decision maker in the case of otherwise
unresolvable technical disputes. So Stefano referred the
problem to the committee, asking that it decide whether one of the
dpkg maintainers can block progress indefinitely in such a
The committee did not need long to reach a decision; it concluded
unanimously that Guillem's blocking of the dpkg upload should be
overridden. Guillem was given a set of deadlines by which to get the code
into experimental (then unstable); if those deadlines were not met, one of
the other dpkg maintainers (Raphaël Hertzog) was given the power
to do the upload instead. As it happens, Guillem met the deadline and Debian now has a
multiarch-capable package manager. This move has spawned another massive
email thread - but this is Debian we're talking about, so it's really just
business as usual at this point. Multiarch in wheezy is back on track.
While the Technical Committee resolved this particular conflict, it did not
(and probably could not hope to) resolve the larger question. Debian, like
many free software projects, tries to make its decisions by consensus
whenever possible. But the project also gives developers something close
to absolute power over the packages that they maintain. Occasionally,
those two principles will come into conflict, as was the case here.
Resolving such conflicts will never be easy.
In this case, everybody was working toward the same goals. Even those who
were most critical of Guillem's behavior stopped far short of suggesting
that he was deliberately trying to delay the multiarch work. He was simply
trying to ensure the highest level of technical excellence for a package he
maintains - a package that is critical to the functioning of the
distribution as a whole. The problem is that he ran afoul of a project
that has been trying - with significant success - to bring a bit more
predictability to the release process in recent years. Overriding him was
probably necessary if the release goals were to be met, but it must still
feel harsh to a developer who has put a lot of time into improving the
project's core infrastructure.
A large project like Debian, in the end, will have occasional conflicts
involving developers who, for whatever reason, are seen as holding up
important changes. So there needs to be some sort of mechanism for
avoiding and dealing with such blockages. Debian's approach, which
includes having teams of developers (instead of a single person) working on
important packages and the Technical Committee as a court of last resort
appears to work pretty well. The benevolent dictator model used by a
number of other projects can also be effective, depending on the quality of
the dictator. In the end, our community does not often have serious
problems in this area; we are able to manage the interactions of thousands
of dispersed developers with surprisingly little friction. But it's good
to know that these conflicts can be resolved when they do arise.
Comments (30 posted)
LWN recently published an article, "The unstoppable Perl release train?", that raised some questions about the Perl 5 release process. In particular, the article questioned how the Perl 5 Porters handled a recent bug report from Tom Christiansen about Unicode support.
This article aims to provide the reader with a better understanding of the current Perl release process, how it benefits Perl 5 users, and how it has helped invigorate Perl 5 development.
First, a disclaimer. I am decidedly pro-Perl. I am paid to program in Perl, and I've been doing so for over ten years now. I've contributed to the Perl core, mostly in the form of documentation, and I was also the release manager for the December 2011 development release of Perl, Perl 5.15.6.
Who are the Perl 5 Porters?
I'll be mentioning this group regularly throughout the article, so
you should know who they are. The Perl 5 Porters are the volunteers who develop
and maintain the Perl 5 language. They consist of anyone who cares enough
about Perl 5 to participate in discussions on the perl5-porters email
list. There's no formal membership, and influence is based
on a combination of hacking skill and the ability to not be a jerk. The more you contribute and the more you communicate, the more control you have over the future of Perl 5. This should sound familiar if you know how other FOSS projects work.
While there are many people on the list, there are not that many people
who actively contribute to the Perl core's C code. Currently, there are
two paid developers, Nicholas Clark and Dave Mitchell, who are funded by
grants from the Perl Foundation. In addition, there are usually another two
to six people actively hacking on the C core at any time. Many more people contribute in the form of small patches, documentation, tests, and updates to core Perl modules.
Perl 5 and Perl 6
There's a reason this article's title specifically says Perl 5. Larry launched the Perl 6 project over ten years ago. When it was first being discussed, many Perl users thought that Perl 6 would supplant Perl 5 in much the same way that Perl 5 supplanted Perl 4.
Since then, that expectation has changed. Perl 5 and Perl 6 are both being actively developed, and there's no expectation that Perl 6 will replace Perl 5 any time soon. We now say that both Perl 5 and Perl 6 are part of the Perl language family.
The two languages co-exist peacefully, and ideas cross-pollinate between the two communities regularly.
A brief history of Perl 5
Larry Wall released Perl 1 in December 1987. He released Perl 5.000 in October 1994. Perl 5 is the Perl most of us know and love (or hate) today. It introduced features like object-orientation, the ability to dynamically load extensions written in C (using the "XS" mechanism), modules, closures, and more.
After the Perl 5.000 release, other Porters started making releases of Perl. Larry was no longer the sole keeper of Perl, and Perl development adopted the notion of a "Pumpking". Initially, the Pumpking (there has not been a Pumpqueen yet, unfortunately) was the person responsible for shepherding the development of each stable release. However, Larry was still Benevolent Dictator for Life, and Perl development still followed Rules 1 and 2, which can be paraphrased as "Larry has the final say on Perl, and can contradict himself whenever he feels like it."
As Larry started spending more time on Perl 6 development, he became
less involved with Perl 5. These days, his focus is almost exclusively on Perl 6. While Rules 1 and 2 still apply in theory, in practice Larry does not make decisions involving Perl 5. Instead, the Pumpking is in charge of the language. The first Pumpking to take on this responsibility was Jesse Vincent. Recently, Ricardo Signes took over the Pumpking position.
For a long time Perl's release schedule was "release it when
it's done". There was no specific list of features for a given
release, and it wasn't always clear what should block a release from
going out. In some cases, major releases ended up coalescing around a big
feature, like threads (5.6.0) or Unicode (5.6.0 and 5.8.0), and that drove
the release. In other cases, releases were repeatedly delayed due to a vicious cycle of adding a feature, fixing its bugs, and then adding a new feature during the bug fixing stage.
The development process also failed to clearly distinguish between major and minor releases. There was a nearly two year gap between the release of 5.10.0 and 5.10.1. In hindsight, the 5.10.1 release was not a minor release, and deserved to be called 5.12.0.
The current release process and support schedule
After the long gap between 5.10.0 and 5.10.1, many Porters wanted to streamline the development process. When Jesse Vincent took over as Pumpking for Perl 5.12.0, the Porters adopted a "timeboxed" release schedule. The theory is that in April of each year, the Pumpking will package up the code in the "blead" branch (Perl's name for "master") and ship it as the new major stable release.
Each major release is followed by a series of minor releases. Minor
releases are minor! They include a small number of changes that are focused on fixing the most critical bugs. New features or incompatibilities are never introduced in a minor release.
Each major release series is supported for two years, except for critical security fixes, which are supported for three years. David Golden has a great write up of this support policy on his blog.
All release numbers start with "5.". The second number represents the major
version number. The last number is the minor release number.
Even major version numbers indicate stable releases. Odd major versions
indicate development releases (e.g. 5.11.x). Minor development releases are
made each month for community review and testing.
Minor stable releases are always backward compatible with other minor
releases in the same stable series, down to binary compatibility for XS
(C-based) extensions. Major releases may break backward compatibility, but
Perl 5 is conservative, and backward incompatibilities are not introduced
The first release to follow this schedule was Perl 5.12.0, released on April 12, 2010.
Aiming for stability
The release schedule includes three well-defined freeze points. The
first is the "contentious" code freeze, which happens in December before the stable release. This means that any changes that don't have widespread agreement from the Porters at this time must wait until after the stable release to be merged. The next freeze point is for user-visible changes, such as API changes, new features, backward incompatibilities, etc. This freeze occurs in February. In March, a "full" code freeze goes into effect in order to finalize the April Release Candidate and subsequent stable release.
There's a lot of work that goes into making this possible. While the number of people with commit bits has expanded, there is much stronger pressure to do work in per-feature or per-bugfix branches. This lets blead stay stable enough for releases to actually happen on schedule.
The Perl release train is only somewhat unstoppable. It stops for
release blocking issues. Deciding whether an issue should block a release
is more of an art than a science. Some questions that can help determine
whether an issue is a blocker are:
- Is the issue a regression as compared to the last stable release of Perl?
- Does the issue represent an unintentional break in backward compatibility?
- Does the issue break a huge number of CPAN modules, or a few really important ones?
- Does the issue cause a major performance problem with commonly occurring code?
- Does the issue break the ability to install modules from CPAN, or the ability to install Perl itself?
- Is someone willing to commit to fixing this before the scheduled release?
The exact decision as to whether an issue is a blocker is made by the
Pumpking in consultation with the Porters. Typically, a few months before
the stable release, the Pumpking will ask the Porters to trawl through the
Perl 5 issue tracker and nominate issues for
blocker status. After discussion on the mailing list, the Pumpking comes up with a "final" list of issues. This list isn't really final. If someone found a major regression the day before the release, that could be declared a blocker.
So why weren't the bugs that Tom Christiansen found declared to be release blockers? This is answered by the first question above. These bugs exist in the current stable release of Perl. They are not regressions. Releasing Perl 5.16.0 with these bugs does not make Perl's Unicode support any worse than it already is. However, delaying Perl 5.16.0 does mean that users would not have access to many other useful bug fixes and features.
Timeboxed releases ensure that users can expect improvements and bug fixes on a predictable schedule, without the long gaps that were common in Perl's earlier history.
As for the security bug discussed in the same thread, a fix is being worked on. It's possible that the fix will be ready for the 5.16.0 release. If it's not ready in time for 5.16.0 then there will be a 5.16.1 release when the fix is ready, as well as 5.14.3 and 5.12.5 releases.
Perl 5 has come a long way in the last few years. Every month, a new development release comes out, shepherded by a different release manager. Once a year, a new major version is released. Releasing a new major version is no longer a months-long ordeal. When the new release schedule was first proposed, many of us wondered if such an ambitious plan could work. Now, it almost seems routine.
Comments (2 posted)
The XInput multitouch extension provides for multitouch input events to
be sent from the device to the appropriate window on the desktop.
Multitouch events can then be used for gesture recognition, multi-user
interactions, or multi-point interactions such as finger painting. While
the general concepts behind delivering multitouch events through a
window server are fairly well defined, there are many devils hiding in
the details. Here, we provide a look into the development of the
multitouch extension and many of the issues encountered along the way.
For Henrik Rydberg, it began as an attempt to make the trackpad on his
Apple Macbook work just as well on Ubuntu as it does on OS X. For
Stéphane Chatty, it began as a research project to develop new user
interface paradigms. For your author, it began as a quest to enable
smooth scrolling from an Apple Magic Mouse.
Like many undertakings in open source, multitouch on the Linux desktop
is the culmination of the many efforts of people with disparate goals.
With the release of the X.org server 1.12, we now have a modern
multitouch foundation for toolkits and applications to build upon.
The beginning of multitouch support for Linux can be traced back to the
Henrik had just merged additions to the evdev input subsystem for
multitouch along with support for the trackpads found in all Apple
unibody Macbooks. Stéphane then added multitouch support to some
existing Linux drivers, such as hid-ntrig, and some new drivers, such as
Some developers started playing around with the new Linux multitouch
support. Over time, the libavg and kivy specialized media and user interface
toolkits added multitouch
support based on the evdev interface. However, there was a glaring
issue: the absence of window-based event handling. Applications had to
assume that they were full-screen, and all touch events were directed
to them exclusively. This was a fair assumption for games, which was
the main impetus for libavg touch support. However, it was clear we
needed to develop a generic multitouch solution working through the X
window system server.
The X.org server and the X gesture extension
Discussions began on how to incorporate touch events into the X input
model shortly after kernel support was present. Initial work by
Benjamin Tissoires and Carlos Garnacho extended XInput 2.0's new
multi-device support for multitouch. Each time a touch began, a new
pointing "device" would be created. Alternatively, a pool of
pre-allocated touch "devices" could be used. However, this approach
broke many assumptions about how devices and events should be handled.
As a simple example, a "touch begin" event would appear to the client as though
the pointer had moved to a new location. How would the client know that
the previous touch hadn't simply moved, as opposed to a new touch
starting? At this point Peter Hutterer, the X.org input subsystem
maintainer, decided we needed completely new semantics for touch input
Around the same time, Canonical was interested in adding multitouch
interfaces to the Linux desktop. The uTouch team, of which your author
is a member, was formed to develop a gesture system that could
recognize and handle system-level and application-level gestures. Since
X did not have touch support yet, the team focused on providing
gestures through the X.org server using a server extension. The
result was shipped in Ubuntu 10.10 and the extension was proposed for
While many developers were enthusiastic about the potential for gesture
support through the X.org server, it was not meant to be. X.org as a
foundation holds backward compatibility in high regard. Applications
written over 20 years ago should still function properly today, in
theory. Though backward compatibility has benefits, it is a
double-edged sword. Any new functionality must be thoroughly reviewed,
and inclusion in one X.org release means inclusion in all future
releases. Even to this day, gesture support is not a settled
technology. It is highly probable that an X gesture extension created a
year and a half ago would not be sufficient for use cases we are coming
up with today, let alone potentially years from now. So the X developers
are reluctant to include gesture support at this time.
XInput multitouch was born
Those concerns notwithstanding, the need for touch through the X server
grew stronger. Peter and Daniel
Stone developed a first draft of the XInput 2.1 protocol, which later
became XInput 2.2, where touches send separate events from traditional
pointer motion and button press events. Three event types ("touch begin,"
"update," and "end") were specified. However, the need to support
system-level gestures added a requirement for a new method of event
handling: the touch grab.
X11 input device grabs allow for one client to request exclusive access
to a device under certain criteria. A client can request an active grab
so that all events are sent to it. A client can also request a passive grab,
where events are sent to it when a button on the mouse is pressed while
the cursor is positioned over a window, or when a key is pressed on a
keyboard while a window is focused. Passive grabs allow for raising a
clicked window in a click-to-focus desktop environment, for example.
When the user presses the mouse button over a lower window, the window
manager receives the event first through a passive grab. It raises the
window to the top and then replays the event so the application can
receive the button press. However, X only allows for a passively
grabbing client to receive one event before it needs to make a decision
on whether to accept it and all future events until a release event, or
to request that the server replay the event to allow another client to receive the events.
This mechanism has been adequate for decades, but doesn't quite work
for system-level gestures. Imagine that the window manager wants to
recognize a three-touch swipe. It is impossible to know if a three-touch
swipe has been performed if the window manager can only view touch
begin events; it must be able to receive the subsequent events to determine
whether the user is performing a swipe or not.
The idea behind touch grabs is that the grabbing client can receive all
events until it makes a decision about whether to accept or reject a
touch sequence. Now, the window manager can listen for all touches that
begin around the same time and watch them as they move. If there are
three touches and they all move in the same direction, the window
manager recognizes a drag gesture and accepts the touch sequences. No
one else will see the touch events. However, if the touches don't match
for any reason, the window manager rejects the touch sequences so other
clients, such as a finger painting application, can receive the events.
This works great for system-level gesture recognition. However, it
necessarily imposes lag between a physical touch occurring and an
application receiving the touch events if the the system is attempting
to recognize gestures. At the X Developer Summit 2010, your author
presented an overview of the vision for an XInput multitouch-based
uTouch gesture stack. One afternoon, while eating lunch and discussing
things over beer, the issue of the potential for lag came up. Between
those at the table, including Peter, Kristian Høgsberg, and myself, the
solution was elusive. However, at some point later in the conference
the issue came up again on IRC. Keith Packard made the suggestion that
touch events be sent to all clients, even before they become the owner
of touch sequences. With the idea at hand, your author scurried home
and drafted up the beginning of what would later become ownership event
As Nathan Willis explained in his overview of the XInput 2.2 protocol, a client may elect to
receive events for a touch sequence before it becomes the owner of the
sequence by requesting touch ownership events alongside touch begin,
update, and end events. The client will receive touch events without
delay, but must watch for notification of ownership. Once a touch
ownership event is received for a sequence, the client owns the
sequence and may process it as normal. Alternatively, if a preceding
touch grab is accepted, the client will receive a touch end event for
the touch sequence without ever receiving a touch ownership event. This
mechanism allows for a client to perform any processing as touch events
occur, but the client must take care to undo any state if the touch
sequences are ultimately accepted by some other client instead.
With the basic concepts hammered out, your author, with an initial base
of work contributed by Daniel Stone, began a prototype implementation
that shipped in Ubuntu 11.04 and 11.10. The uTouch gesture system based
around XInput multitouch began to take form. This was enough to prove
that the protocol was reasonably sound, and efforts began in earnest on
an upstream implementation for the X.org server 1.12 release.
It is interesting to note how XInput multitouch compares to other
window server touch handling. On one end of the spectrum are phones and
tablets, which run most applications full screen. This, and the lack of
support for indirect touch devices, e.g. touchpads, means mobile device
window manager multitouch support is much simpler. On the other end of
the spectrum are desktop operating systems. Windows 7 shipped with
multitouch support, but only for touchscreens. For an unknown reason,
Windows also only supports raw multitouch events or gesture events on a
window, but not both. As an example of the consequences of this
shortcoming, Qt had to build their own gesture recognition system so it
could support both raw multitouch events and gestures at the same time.
OS X only supports touchpads, but this simplification alone ensures
that touches are only ever sent to one window at a time. The event
propagation model they chose would not work for touchscreens. In
comparison, the XInput multitouch implementation allows for system- and
application-level gestures and raw multitouch events at the same time
across both direct and indirect touch devices. In your author's biased
opinion, this is a key advantage of Linux on the desktop.
A few bumps in the road
Although development of multitouch through X took more time than anyone
wanted, it was shaping up well for the 1.12 X.org server release. Many
complex issues, such as pointer emulation for touchscreens, were behind
us. However, touchpad support had yet to be finalized. Two large issues
surfaced involving scrolling and other traditional touchpad gestures.
The first issue involved the ability to scroll in two separate windows
while leaving one finger on the touchpad at all times. Imagine there
are two windows side by side. The user positions the cursor over one
window and begins a two-touch scroll motion on the trackpad. The user
then lifts one finger and uses the remaining finger to move the cursor
over the second window. The second finger is then placed on the trackpad
again, and a second scroll motion is performed. Under the XInput
multitouch protocol, a touch sequence is locked to a window once it
begins. If two-touch scrolling is performed through gesture recognition
based on XInput touch events, the touch that began over the first
window could not be used for a scroll gesture over the second window
because the touch events would remain locked to the first. In order to
resolve this difficulty, it was decided that, when only one touch is active on a
touchpad, no touch events are sent. In order to not send two events for
one physical action, pointer motion was also prevented when more than
one touch was present on a touchpad.
This fix resolved pointer motion, but other traditional touchpad
gestures are even more problematic. Particularly troublesome is
two-finger scrolling. When mice with scroll wheels were first
introduced, they had discrete scroll intervals. The wheels often
clicked up and down. This led to an unfortunate API design for scroll
events in the X server. The X core protocol
cannot send pointer events with arbitrary values,
such as a scroll amount. To provide for scrolling through the X core
protocol buttons 4, 5, 6, and 7 were redefined from general purpose
buttons to scroll up, down, left, and right.
When the user scrolls up using a scroll wheel, the X server
sends a button 4 press event and then a button 4 release event. As an
aside, this is the reason why we don't yet have smooth scrolling on the
The problem for multitouch lies in the possibility of sending two
separate events for one physical action. This would occur if we sent
touch events at the same time we sent scroll button events. It was
decided that touch events may not be sent while the server is also
sending other events derived from touches. This means that if the user
enables two-finger scrolling, touch events are inhibited unless three
touches are active on the touchpad. Likewise, if the user performs a
two-finger tap to emit a right click, touch events are also inhibited
unless three touches are active on the touchpad, and so on.
Many workarounds were considered, but nothing provided an air-tight
solution. The double-edged sword of backward compatibility prevents X
from supporting scroll events, click emulation, and touch events at the
same time. Your author hopes this situation will end up hastening
support for traditional trackpad gestures on the client side of X
instead of the server side.
The development of the multitouch extension finished with the release
of the X.org server 1.12 on March 5th, 2012. Many upcoming distribution
releases, including Ubuntu 12.04 LTS, will be shipping it soon.
Although this is the end of the X.org multitouch story, it is only the
beginning for toolkits and applications. GTK+ recently merged an API
for handling raw touch events for 3.4, and your author hopes to merge
raw touch support for Qt in the near future. Next on the roadmap will
be gesture support included in standard toolkit widgets and APIs for
application developers. There is still plenty of work to do, but the
will of those hoping to bring smooth scrolling support to the Apple
Magic Mouse and many other multitouch features is quite strong.
Comments (35 posted)
Page editor: Jonathan Corbet
Next page: Security>>