Debian contemplates patch management

Developers in the Debian project had a busy week cleaning up after the openssl vulnerability was disclosed. Once that was taken care of, they moved on to process-related issues. Clearly, some shortcomings in how Debian handles patches to the programs it ships have been revealed; now the project would like to face those problems and make things work better in the future. The resulting discussion shows Debian at its introspective best, and may well have results that other distributors will want to pay attention to. As a Fedora developer noted: "This bug could easily have been us on the receiving end." All distributors make changes to their packages, so all of them are potentially exposed to this kind of failure.

Debian's packaging policy resembles that of most other distributions. A Debian source package is supposed to contain a tarball of the upstream source distribution, without changes. Any distribution-specific patches are included separately and applied when the source package is prepared for building. There are couple of Debian-specific issues to be faced, though:

• From the discussion, it seems that the "pristine upstream tarball" rule is occasionally bent by developers. Sometimes there is no alternative: some upstream source distributions contain material which, due to its licensing, cannot be shipped by Debian. The justification for other cases is not always quite as clear.

• Debian's patches are all mashed together and included as a single diff file. So there is no metadata describing the patches, and they are difficult to separate from each other. In this regard, Debian differs from RPM-based distributions, which generally keep each patch separate.

The end result of all this is that Debian's patches are hard for others to review, hard for upstream projects to consider, and even hard for other Debian developers to get a handle on.

Raphaël Hertzog started a discussion on how to improve this situation. A key part of his approach (and an idea which others have been pursuing as well) is to make changes to the Debian source package format which would make the nature of each patch explicit. At a minimum, packagers would include a debian/patches directory with the source; that directory would contain each patch, broken out into a separate file. Some Debian packages are built this way already, though the practice is far from universal.

Beyond that, though, it would be nice to have the source package itself understand the patch stream and its associated metadata. There are a few proposals for this; Raphaël favors the "3.0 (quilt)" format, which keeps the patches (in a separate tarball) as a quilt series. This format seems to have a certain amount of support; among other things, its simplicity would make it easy for Debian developers to create packages in this format without having to learn new tools. The quilt series file - like the spec file used with RPM packages - makes it clear which patches must be applied, and in which order.

There are other variants of the 3.0 source package format, though. The "3.0 (git)" format contains a git repository containing the upstream source and a series of patches to it. This approach has the advantage of including the history of the patches along with the other metadata; it could also, arguably, make it easier for other distributors (and upstream) to cherry-pick patches of interest. On the other hand, a git-based package format requires the availability of git and has the potential to make those packages larger. The GitSrc FAQ has some more information on this format; there's also a "3.0 (bzr)" format variant out there.

Any of these new formats, if widely adopted, would bring a new level of transparency to Debian's patching activities. It would enable the creation of a "patches.debian.org" site (clearly inspired by patches.ubuntu.com) where anybody could quickly look at the changes which have been made to any given package. There are some developers who doubt the utility of this; they worry that upstream developers won't want to poll a site to see what changes have been made to their code. One developer at least (GNOME hacker Vincent Untz) thinks that a patches.debian.org site would be a step in the right direction, though.

Another quibble which has been heard is that Debian does not need any new infrastructure for patch management. The right place for patch tracking, it is said, is with the upstream project. Nobody seems to challenge the claim that more patches need to go back upstream, but there is also the fact that quite a few patches will never get there. The upstream developers for a number of projects seem to have different goals and are seen by the distribution maintainers as being overtly uncooperative. And some patches - such as those removing non-free material - may not be something that even cooperative upstream maintainers want. So there will always be a need for distribution-specific patches; the "track it upstream" approach will not solve the whole problem.

Meanwhile, Joey Hess brought a completely different idea to the discussion: just treat every divergence from upstream as a bug. Each patch would have a corresponding entry in the Debian bug tracking system (BTS) with a special tag. Anybody could then query the list of outstanding bugs, view the patches, and participate in the associated discussion. Using the BTS brings some real technical advantages, in that the system already exists. But, Joey says, the real benefit is elsewhere:

A separate patch tracking mechanism, instead, would be a mostly automatic subsystem on the side which might not bring the same sort of pressure to bear on developers.

The BTS approach is not universally acclaimed either. Some developers claim that most Debian-specific patches are not really Debian bugs - they are, instead, upstream bugs. Regardless of whether that is really true, distribution bug trackers generally carry a great many entries which, in the end, describe bugs in upstream packages. Another complaint is that creating and maintaining BTS entries would be just another bit of bureaucratic work imposed on Debian developers. Beyond any doubt, some developers would see it that way.

But this may be a place where a bit more bureaucracy makes some sense. The Linux distributors of the world (certainly not just Debian) are carrying thousands of patches against the free programs they distribute. Making the nature and extent of those patches more readily apparent can only be beneficial for users, reviewers, distributors, and upstream maintainers. One clear conclusion from recent events is that all distributors could do more to let the rest of the community know about the changes they are making.

A distributor's ability to patch a program is a crucial part of the whole ecosystem - it's the distributors' way of balancing their users' needs against the upstream maintainer's policies. But distributors should be clear about the changes they are making, willing to merge those changes upstream whenever possible, and wanting feedback on those patches. Any "bureaucracy" which helps to make that happen can only help our community as a whole in ways that go far beyond the avoidance of another openssl disaster.

One final note: the existence of source package formats which incorporate distributed version control system repositories shows that developers have been thinking about this problem for a while; it's not just a response to recent events. There is an effort underway to think about what the intersection of version control and packaging can really achieve for all distributors; the folks working on this project can be found at vcs-pkg.org. They are working on organizing a gathering this September in Extremadura. Vcs-pkg is worth watching; it has the potential to make things work better for developers and users of all distributions.

---

(Log in to post comments)

That Debian patches are all mashed up together is not true in general.  A lot of source
packages keep their patches nicely seperated under debian/patches and use one of the available
patch systems to apply/unapply them at build time etc.  This has been sort of recommended
behaviour for several years now.

The discussed problem is mostly with source packages maintained in a distributed version
control system, where it is unfeasable to break up the involved feature branches into mere
patches, so they are shipped as one big patch mashed together.  this is suboptimal for people
looking at the .diff.gz, but on the other hand, the source package control file should also
include information on where to find the revision control repository.  There is, however, no
central RCS for Debian source packages.

"There is, however, no central RCS for Debian source packages."

Apart from Alioth? (Which supports git,svn and hg at least... :O)

> Apart from Alioth? (Which supports git,svn and hg at least... :O)

First, "supports Subversion, git, Mercurial, Bazaar, etc." is not "central VCS", it's several
disparate VCSes. If Alioth supported only one VCS, it might qualify; but such a choice would
not be popular with those users whose VCS was not the one chosen :-)

Second, Alioth is not for *all* Debian packages, but only those free-software packages that
require collaborative maintenance. The Alioth admins reject requests to use it merely as
storage for a package without a need for ongoing collaborative maintenance.

> The Alioth admins reject requests to use it merely as storage for a package without a need
for ongoing collaborative maintenance.

Well, things can be kept in ~/public_{hg,git,whatever} and people are also free to use the
collab-maint repositories, which are often used as simple storage of a package's history.

>The Alioth admins reject requests to use it merely as storage for a package without a need
for ongoing collaborative maintenance.

This is wrong. See http://wiki.debian.org/Alioth/PackagingProject for details. You can host
your VCS repository in collab-maint even if you're alone working on your package.

Alioth is clearly the most used resource to host VCS for package management but it's not a
resource that has to be used by everybody. Each maintainer has the liberty to host it
whereever he prefers.

-- Raphael Hertzog, Alioth admin

My understanding of Alioth's policy was based on a rejection notice I received a while ago:

> looking at the description of your project request, it's not clear at all that you need a
full Alioth project to effectively co-maintain your package.

(This rejection was quite justified, IMO.)

From this I understood that Alioth doesn't accept "it's a Debian package" as the sole basis
for hosting a package. It was this that I referred to in my initial comment on the article.

I suppose that, although the above rejection notice doesn't mention Alioth's 'collab-maint'
project, it doesn't exclude the possibility either.

>My understanding of Alioth's policy was based on a rejection notice I received a while ago:

The subject of that rejection is "Please use collab-maint" and the second paragraph says:
"Please check out http://lists.debian.org/debian-devel-announce/2006/04/msg... and
http://wiki.debian.org/AliothPackagingProject"

It's that not enough hints to point you to collab-maint, I don't know what we can do. :-)

-- Raphael Hertzog

>Some Debian packages are built this way already, though the practice is far from universal.


As mentioned in the article, yes this is in place in Debian, where the diff.gz carries all the
patches that extract into the debian/patches directory and are applied at build time.

The problem with this approach is that there are half a dozen different "standard" ways to do
this plus probably a few homebrew solutions and they all operate differently.  My opinion is
that if this method of managing patches is to be really endorsed by the Debian Project, then
one mechanism needs to be mandated in policy and used project wide.


Overall, many of the "core" packages do tend to use a patch system of some sort, but of the
thousands of Debian packages, most do not and ship just a diff.gz that applies directly to the
source code.  Of course, most debian packages are also relatively small pieces of software and
probably do not need a full patch management system since the patches are mostly build system
fixups and small minor changes.

"That Debian patches are all mashed up together is not true in general."

Even if the patches are separated, they are _distributed_ mashed together with a bunch of
other files in a compressed diff. That raises the barrier to finding out what's in a Debian
package. Compare that to figuring out what's happening in a RH/Fedora package:

http://cvs.fedoraproject.org/viewcvs/devel/sed/

FWIW, I think patches.ubuntu.com is a vast improvement over a diff.gz, even with the nearly
nonexistent metadata.

Are all Fedora packages maintained in a central CVS?

The equivalent for openssl:

http://packages.qa.debian.org/findutils

You get a link to a repository and to a browser. In the browser:
http://svn.debian.org/wsvn/pkg-findutils/trunk/ 

No go to debian/patches .

Not all packages are team-maintained, and thus finding this is a bit more difficult.

> Are all Fedora packages maintained in a central CVS?

Yes, that's the only way to include packages in the distro, as *all* packages are built in
koji ( http://koji.fedoraproject.org/koji/ ) from the sources in cvs.

/me points to http://packages.qa.debian.org/findutils again
(it actually redirects to http://packages.qa.debian.org/f/findutils.html )

Here you can find all the information about the package. Including logs from the build
servers.

A package is rebuilt on build servers after it has been submitted by its maintainer. The
maintainer may maintain it in a version control system or may  write each and every new
version using butterflies. But once the maintainer uploads the candidate version to Debian, it
gets into the archive. This does not take a centralized VCS.

The package in that example is maintained in a separate subversion repository. It would still
be possible to use that even after other maintainers start using better methods such as
svn2.0, git, bzr and super-duper-future-vcs.

In fact, isn't the input to koji a source RPM?

> In fact, isn't the input to koji a source RPM?

It can be, however not in this case. 
Package is built from spec file and sources in cvs. 
koji builds the source and binary RPM.

>  There is, however, no central RCS for Debian source packages.
Indeed there is none of central one, but most of the packages now carry Vcs-{git,svn,...} and
Vcs-browser fields in their debian/control files. and very many keep their central VCS (or
some kind of clone) on alioth (although it is irrelevant indeed), especially if maintenance is
collaborative (ie by a team).

AND there is a tool now withing Debian's devscripts: "debcheckout -  checkout the development
repository of a Debian package" which makes use of those Vcs fields, so if I need a possibly
updated maintainer version of repository for fail2ban I just run 'debcheckout fail2ban' and I
see everything -- upstream, debian sources and branches which were merged into the release
branch, quite easy and quick. Sure thing not all packages provide Vcs- header fields but quite
a few

> That Debian patches are all mashed up together is not true in general.

So, what would be needed nowadays to get *reliably* all the patches that 
are being applied to Debian base packages, say for the ARM architecture?

At least few years ago that was about impossible task, only way to get 
automatically correct patches (applied), was building the Debian packages 
(which requires doing this in an environment supporting that) and because 
Debian packages do not support cross-building like some other 
distributions...

Debian developers are a diverse set of people, I don't want to tar them 
all with the same brush, but often they make no effort to collaborate 
with upstream, or even to tell them that their code has been packaged.

I have had some of my code debianized. The package hasn't been updated 
since August 2005! What can I do? As the upstream developer, I've no 
particular rights as regards the packaging of my software. I don't think 
I can request mantainership be passed to me.

It has one large patch .diff.gz, not the debian/patches/ style. Almost 
all of the patch file exists to create the debian-specific files, but 
buried deep in there is also a README.Debian, that includes useful 
information on how to use my code with Evolution. Its not debian 
specific, and I would be happy to accept documentation patches, but it 
was never offerred to me (I know now just because I went searching to see 
if any of my code had been debianized).

I like the suggestion to use the BTS. Whatever is done, every patch 
should have attached to it proof that it was at least offerred upstream 
(copy of email to author, bug# from upstream, something). All developers 
are busy, not just Debian developers, and Debian itself is glacially slow 
in accepting patches. Maybe its frustrating at times, but packagers 
should make at least a token effort!

You don't have to argue about who has whatever right. You can ask the original packager to
update the package. The formal way of doing it is to file a wishlist bug on the package. See
http://bugs.debian.org/ (this takes sending an email. No need to create an account or
anything).

If the old package is broken in interesting ways, mention it. If the old package is buggy it
may need to get more love and care, or maybe just get thrown away.

It also sounds like there are no debian/patches/ because the package does not patch anything
in the tarball itself. debian/patches/ are just some of the files created by applying the
.diff.gz .

I'm guessing you're talking about vpim :)
It looks like the vpim package was removed last year because it went unmaintained. You'll
still see references to it as it was part of a stable release, but it won't be included in
future releases. Unless somebody repackages it of course -- if you're interested in doing that
yourself, check out mentors.debian.net and the debian-mentors mailing list.

There are many examples of upstream developers who maintain the Debian packages of their work
within Debian.  What it does require, though, is that they take the time and effort to become
familiar with Debian's policies and procedures, and thus become a part of the Debian
development community.

Being upstream does not confer you any formal rights beyond what you would have otherwise, but
it does lend you some social capital that you can spend to getting the packages under your own
maintainment - assuming you are willing to consider Debian developers your peers and act
accordingly.

As a Debian developer of nearly ten years, I find it shameful that some Debian maintainers do
not cultivate good working relationship with their upstreams.  However, there isn't much I can
do about it, considering that I don't have the time or energy to devote to pushing this
matter.

> I have had some of my code debianized. The package hasn't been updated 
> since August 2005! What can I do? As the upstream developer, I've no 
> particular rights as regards the packaging of my software. I don't think 
> I can request mantainership be passed to me.

Maintainership is to some degree a formal affair. As a maintainer you get to decide, OTOH,
nothing prevents you as upstream to do the maintainer's work.

What you can do is apply the Debian diff.gz, include it upstream and improve it. That way, if
Debian's packaged version is in some respect lacking compared to upstream's, users can allways
download your version and build it directly with dpkg-buildpackage. And you can also offer
pre-built Debian packages.

That puts some competition/pressure into the game to offer the "best" package to the user and
can have a healthy effect on "stale" Debian packages. But beware: it can bruise egos, harm the
communication process etc.
*t

> What can I do? As the upstream developer, I've no particular rights as regards the packaging
of my software. I don't think I can request mantainership be passed to me.

Yes you can! Just contact the maintainer and ask to become maintainer or co-maintainer. You
will also need to get an official status in the Debian community. If the lengthy process of
becoming a Debian Developer bothers you (see http://www.debian.org/devel/join/newmaint), just
try to become Debian Maintainer. See http://wiki.debian.org/Maintainers, it's never been as
easy to contribute to Debian by maintaining your own package.

-- Raphael Hertzog

I like the idea of having patches attached to bugs. I'd think that most of them come out of
fixing reported issues anyway, so "fixed in Debian but not upstream" could be just a bug
status. Perhaps there should also be a field for a link to the upstream bug report that this
patch fixes, to prompt Debian developers to try to get it resolved at the other end. For
things where the upstream project and Debian differ on how things should be, this upstream bug
could be closed as something the upstream project doesn't want to do, but the pair would exist
as evidence that the Debian maintainer contacted the upstream project about it.

For that matter, this is something I think any good distributed bug tracker should support:
having the same bug in both the distro bug tracker and the upstream project bug tracker, with
different states. (So it could be fixed in the distro but open upstream if the upstream hasn't
deal with it, or it could be fixed upstream but open in the distro if the distro isn't yet
shipping a version where it's fixed.)

I'm upstream for pptp, pptpd, and a few other things.

A few years ago, the PPTP project support load came mainly from users who were using
distribution packages and not the raw source.  Often they would have an old version, or a
version changed from what I had released.

We weren't being told about changes made by distribution developers.

So pressing was this for me that at one linux.conf.au, after Mark Shuttleworth spoke of his
time in orbit, I went up to him and insisted that Ubuntu at least show people what they
change.  patches.ubuntu is presumably the result.

But still, it's a rare distribution developer who will engage with their upstream on trivial
issues that can be easily patched during packaging.

So what I do is hunt around the distributions looking for interesting patches, with my goal to
make the patches useless for the next version ... by merging them where possible.  Upstream
developers who don't bother with this are kinda missing the point, in my opinion.  Those users
still matter, even if they are hidden behind a DD.

I wish there was a clear list of places to hunt for these patches!

The "patch == bug" proposal would solve this.