Getting multiarch dpkg into Debian

"Multiarch" is Debian's name for its approach to supporting multiple architecture ABIs on a single system; LWN recently covered the multiarch work in some detail. Multiarch support is one of the goals for the upcoming "wheezy" release; since that release is getting closer to its freeze time, Debian developers naturally want to have all the necessary code merged into the distribution and ready for testing. A disagreement over the uploading of one of the crucial pieces of the multiarch puzzle recently threatened that goal and provided an interesting view of the higher levels of the Debian project's governance mechanisms in action.

Clearly, a functioning multiarch implementation will require extensive support from the packaging mechanism. To that end, a number of developers have been working on extending the dpkg utility for some time; patch sets adding multiarch support to dpkg were posted as early as 2009. Much of this work has been supported by Canonical and Linaro, so it is not entirely surprising that it shipped first with the Ubuntu distribution; the 11.04 release included basic multiarch support. Until very recently, though, the necessary dpkg changes had not found their way into Debian - not even into the experimental repository.

A number of attempts have been made over time to push the dpkg changes into experimental. The holdup each time appears to have been the same: dpkg maintainer Guillem Jover blocked the push until he could do a full review of the relevant code. The problem, it seems, is that this review never happened; Guillem is a busy developer and has not been able to find the time to do that work. So the multiarch-compatible dpkg remained out of the repository; only those willing to build it from source were able to test the multiarch changes. Playing with fundamental packaging system changes is scary enough without having to build the tools, so this situation can only have led to less testing of multiarch support in Debian as a whole.

In October, 2011, Philipp Kern, representing the release team, expressed concern about the delay, saying:

Philipp requested that the new dpkg find its way into experimental immediately, with a move to unstable two weeks later. But things did not happen that way; instead, Guillem blocked another attempt to upload the work into experimental. Debian project leader Stefano Zacchiroli responded with some concerns of his own:

Guillem did not respond to this message until March 3, 2012 - over four months after it was sent. Meanwhile, the dpkg work languished outside of the Debian repositories. In January, 2012, Cyril Brulebois pushed the new dpkg using the Debian non-maintainer update (NMU) process. An NMU can be a way to route around an unresponsive or distracted maintainer, but the process has its limits; in this case, Guillem simply reverted the NMU, leaving the situation as it was before.

At the beginning of February, Stefano apparently decided that the situation had gone on for too long. The powers of the project leader are famously limited, though; there was nothing Stefano could do on his own to force a solution to the problem. The Debian Technical Committee, however, is a different story; it is empowered by the Debian Constitution as the final decision maker in the case of otherwise unresolvable technical disputes. So Stefano referred the problem to the committee, asking that it decide whether one of the dpkg maintainers can block progress indefinitely in such a situation.

The committee did not need long to reach a decision; it concluded unanimously that Guillem's blocking of the dpkg upload should be overridden. Guillem was given a set of deadlines by which to get the code into experimental (then unstable); if those deadlines were not met, one of the other dpkg maintainers (Raphaël Hertzog) was given the power to do the upload instead. As it happens, Guillem met the deadline and Debian now has a multiarch-capable package manager. This move has spawned another massive email thread - but this is Debian we're talking about, so it's really just business as usual at this point. Multiarch in wheezy is back on track.

While the Technical Committee resolved this particular conflict, it did not (and probably could not hope to) resolve the larger question. Debian, like many free software projects, tries to make its decisions by consensus whenever possible. But the project also gives developers something close to absolute power over the packages that they maintain. Occasionally, those two principles will come into conflict, as was the case here. Resolving such conflicts will never be easy.

In this case, everybody was working toward the same goals. Even those who were most critical of Guillem's behavior stopped far short of suggesting that he was deliberately trying to delay the multiarch work. He was simply trying to ensure the highest level of technical excellence for a package he maintains - a package that is critical to the functioning of the distribution as a whole. The problem is that he ran afoul of a project that has been trying - with significant success - to bring a bit more predictability to the release process in recent years. Overriding him was probably necessary if the release goals were to be met, but it must still feel harsh to a developer who has put a lot of time into improving the project's core infrastructure.

A large project like Debian, in the end, will have occasional conflicts involving developers who, for whatever reason, are seen as holding up important changes. So there needs to be some sort of mechanism for avoiding and dealing with such blockages. Debian's approach, which includes having teams of developers (instead of a single person) working on important packages and the Technical Committee as a court of last resort appears to work pretty well. The benevolent dictator model used by a number of other projects can also be effective, depending on the quality of the dictator. In the end, our community does not often have serious problems in this area; we are able to manage the interactions of thousands of dispersed developers with surprisingly little friction. But it's good to know that these conflicts can be resolved when they do arise.

---

to post comments