Debian debate over tag2upload reaches compromise

Debian's proposed tag2upload service would be worthy of an article even if it wasn't so contentious; tag2upload promises a streamlined way for Debian developers using Git to upload packages to the Debian Archive. But tag2upload has been in limbo for years due to disagreement and a communication breakdown between the team behind tag2upload and the ftpmasters team. It took the threat of a General Resolution (GR), weeks of discussion, and more than 1,000 emails to finally move forward.

History

Work on tag2upload began in 2019. Sean Whitton wrote that he and Ian Jackson designed, implemented, and tested the prototype over a weekend. The system, he wrote, would allow Debian Developers to upload new versions of packages by using a new script ("git debpush") to push a signed and specially formatted Git tag (example here) to Debian's GitLab instance, called Salsa:

That's right: the only thing you will have to do to cause new source and binary packages to flow out to the mirror network is sign and push a git tag.

When a developer signs and pushes the Git tag, it triggers a GitLab webhook that sends the URI of the Salsa project repository and the name of the tag to the tag2upload service. The service then verifies the signature on the tag against Debian's keyring. Assuming that the signature checks out, tag2upload would produce a Debian source control file (.dsc) and a Debian changes (.changes) file, signs these, and uploads the results to the ftp-master.debian.org server queue. If all has gone well to this point, the source packages and binary packages would be built and then sent on their way to Debian's package pools. The access to that path runs through the ftpmasters.

The ftpmaster team members are appointed ("delegated" in Debian lingo) by the Debian Project Leader (DPL). This team is responsible for deciding what is allowed into the Debian Archive and may reject uploads for a variety of reasons including an unacceptable license, errors caught by Debian's package linter, missing source, and policy errors to name only a few. The reject FAQ provides a lengthy, but non-exhaustive, list and leaves the door open to rejection for other reasons. Because the team is delegated by the DPL, its decisions cannot be overridden by Debian's Technical Committee; they can only be overridden by a GR.

Jackson shared a draft of the service architecture to the debian-devel list in July 2019. Ansgar Burchardt, a member of the ftpmaster team, noted that the service as described would bypass various permission checks on the archive side. Jackson asked which checks would be bypassed, but no response was given. In August 2019, he posted a second version of the draft in response to suggestions from the list.

Debian developer Raphaël Hertzog replied that he had reviewed the thread and said that the "point of friction" between the groups was that the ftpmasters wanted to require a signed .dsc from the maintainer that would ensure that the source package created by tag2upload was what the maintainer intended to upload. Jackson, he said, doesn't want the maintainer to have to deal with the .dsc. He also proposed a workaround that might supply that information. There was no reply to that email on the list.

Another member of the ftpmaster team, Joerg Jaspert, replied to say that he was "a bit detached right now with anything Debian" and had not read most of the threads about tag2upload. He said he was in favor of something more Git-like, but the implementation as it stood was a no-go. The final check of the maintainer's key has to be performed on the ftpmasters systems. Other systems could perform the check as well, but the final one must be performed by the Debian Archive Kit (dak) system. Jackson replied that tag2upload would include a copy of the original uploader's signed Git tag object "as soon as dak supports it". The conversation seems to have sputtered to a stop there. Whitton, who is also part of the ftpmaster team as an FTP Assistant (a member with fewer privileges than members of the team with the FTP Masters role, such as Jaspert and Burchardt), did not reply on the list about Jaspert's concerns.

In 2020 there was a conversation on debian-devel about the status of tag2upload. Whitton said that the ftpmaster team had objections to its design and "we could not overcome the disagreement". There was some back-and-forth between Whitton and Burchardt, but nothing was resolved.

"All lies"

In 2023, Jackson and Whitton gave a talk at a Mini DebConf in Cambridge to promote tag2upload. In the draft text of the talk, they wrote that tag2upload would make uploading packages more convenient, and improve Debian's source code integrity too. Debian currently has several methods for uploading source packages; what they have in common is that they all require the source package to be created on the maintainer's system and then uploaded. In the talk text, however, Jackson and Whitton argue that the majority of Debian packagers are actually using Git and treating repositories in Salsa as the primary source of truth:

Theoretically, the Debian archive is the canonical source code repository. Theoretically, the .dscs are the preferred form for modification of our software. But, this is, nowadays, all lies, at least for the vast majority of packages.

Most packagers, they said, are treating the .dsc and "the whole of the official archive" as outputs rather than the principal work. But the primary working repositories in Salsa are not official, and they do not follow a standardized format. This makes life harder for people who test the packages, and for anyone making a non-maintainer upload (NMU). "You can't automatically grab the git source for an arbitrary package, and build it," they wrote. In contrast, using tag2upload, anyone could check out the tag that represents a version of a package and work with the same source that the maintainer used.

At the time of the talk, the pair said that the service was almost ready to deploy. It had received a security review by "some Debian security experts" including Russ Allbery and Jonathan McDowell, and there was some outstanding work to address before live deployment, but the tag2upload service also needed approval from the ftpmaster team. They expressed hope that the details covered in the tag2upload talk would help make that happen. What did not happen is a direct conversation between teams to attempt to unblock the service.

Call for a General Resolution

On June 12 of this year Whitton sent an email with a description of tag2upload, its benefits, the reasons for proposing a GR, and the draft of the GR to the debian-vote mailing list. Whitton said that he was posting the draft for review, rather than proposing it for a vote immediately, "because of the relative shortness of our official discussion periods". (LWN covered proposed changes to the GR discussion period in 2021, and a GR amending the process passed in 2022.)

The reason for a GR, Whitton wrote, is that the ftpmaster team had stated a requirement that the Debian Archive Kit (dak) system "be able to completely re-perform the verification of maintainer intent done by the tag2upload service". That requirement would fatally undermine the design and user-experience of tag2upload. Whitton also wrote that Allbery and others have tried, and failed, to get an explanation from the ftpmaster team "that we could understand as a strong technical objection". His conclusion was that the team is being conservative and favoring existing processes by default, rather than having an actual technical objection to tag2upload. The service was ready to deploy without any code changes by the ftpmaster team, he said, but it needed "a suitably trusted key" similar to the ones used by Debian's buildd autobuilders.

Since the designers and proponents of tag2upload had reached an impasse, Whitton said that the GR would be needed to get the project unstuck and into production. The text of the proposed GR stated that tag2upload should be deployed "in the form designed and implemented", which was by Allbery and McDowell. The ftpmaster team's objection would be overruled according to the Debian Constitution section 4.1.3. ("Make or override any decision authorised by the powers of the Project Leader or a Delegate.")

Jackson said that he and Whitton had been "very slow to reach for the GR hammer", and said that tag2upload had been blocked since 2019. He said that they had asked for help behind the scenes, including speaking to several Debian Project Leaders (DPLs) and others, "but sadly that has not been effective".

Allbery shared his security review of tag2upload to debian-vote shortly after Whitton's draft GR, with the caveat that he was not a neutral party "in the sense that I think tag2upload is a good idea and should be deployed". However, he said that he does security reviews professionally and tried to approach it the same way he would a major work project. He encouraged other Debian members with security expertise to check his work. The entire document is worth reading, but his conclusion was that tag2upload should be adopted:

Compared to the existing upload architecture, tag2upload provides additional defenses against injection of malicious code into source packages and better traceability of source package contents, at the cost of some minor additional security risk and infrastructure complexity. I believe tag2upload has somewhat stronger security properties than the current upload mechanism but not a profound advantage. I do not believe it introduces any significant security regressions.

Allbery added that there is "some irreducible security risk" that comes with introducing new features and a new attack surface, but whether the benefit outweighed the risk "is not a decision that can be made by a security review."

Surprise

On June 15, Jaspert joined the discussion. He wrote that he was taken by surprise by Whitton's draft GR and that the last communication "in my ftpmaster inbox" was 2019. "There had been mentionings on some mailing list somewhere, but nothing coming to us, that I can find." At that time, he said, the ftpmaster team raised several points of objection to tag2upload's design. The primary point Jaspert raised was that the design of tag2upload could "bypass/circumvent archive upload checks and restrictions". He said that, in five years since it was first discussed with the ftpmaster team, the concern had still not been addressed. "More like, entirely ignored." However, he said, the ftpmaster team was in favor of the service and wanted to see it running if the final verification and authorization of an upload was handled by dak without needing to trust another service.

Whitton replied that the tag2upload team has been "seeking help behind the scenes" for four years, without progress and that led to the GR. He said that Jaspert had not been very active with the ftpmaster team recently and "may be missing some things". He thanked Jaspert for how he had explained his position in the email, but said that the tag2upload team had argued against the ftpmaster team's opinion—they had not ignored it—and no one had responded. He directed Jaspert's attention to an email from Allbery, and said "I'm looking forward to your reply to Russ".

Allbery wanted to know why final verification and authorization was the ftpmaster team's red line. "Is it only that you don't want to add another system to the trusted set, or is there something more specific that you're concerned about?" He also proposed a hypothetical design for tag2upload that would add a signed hash of the Git tree, possibly in a separate file, that would then pass dak an object that has the hash of the tree and the uploader's signature. Would the ftpmaster team approve that design "modulo the normal sort of details that would need to be hashed out?"

Jaspert replied that, yes, there should be one point doing the verification and authorization, not many. He also said that tag2upload would be doing work that was delegated to the ftpmaster team, though that could be addressed by the ftpmaster team running tag2upload or adjusting delegations. But even if the ftpmaster team ran it, he said, it was still a loss given the current tag2upload design. For one thing, the current practice allowed users to verify the signature of the actual maintainer rather than having to trust the key from tag2upload or the ftpmaster team:

We want dak (and anyone else) to be able to say "Yes, [packager] $x has signed off this content". That only works, if dak (and later, the public, if they want to check too) have the signature for this in a way they can verify it. And not just a line somewhere "Sure, $service checked this for you, trust us, please".

In another reply on June 17, this one to Jackson, Jaspert said that the tag2upload proponents should have directly asked the ftpmaster team for its position instead of working behind the scenes. That, he said, would have led to an actual position from the team. He reiterated that he was interested in finding middle ground, but he had developed an eye condition that meant it hurt a lot to focus on anything. He asked for a pause on the GR "in the term of days, not months" to allow him time to return to the conversation.

Questions, objections, and clarifications

The conversation continued unabated for days in Jaspert's absence, as many still wanted to discuss the design and implications of tag2upload. Simon Richter asked whether tag2upload was "completely orthogonal to any efforts to move all packages to git-based team maintenance?" That was likely, at least in part, in reference to a conversation in April on the debian-devel mailing list. In short, current DPL Andreas Tille had opened a discussion about ending single-package maintainership and requiring Debian package source to be maintained in Salsa. As one might expect, that discussion did not lead to a consensus that all Debian packagers should adopt one true method of doing things. But the topic is clearly still fresh in the minds of some Debian contributors.

Jackson said that there were two answers to that question, one technical, one social. "The social answer is that there is absolutely no connection." The tag2upload team, he said, is not interested in trying to wrangle everyone to the same development platform. The aim is to provide tooling that can support anyone in their current development practices "insofar as we can". On a technical level, he said, tag2upload and efforts to mandate maintaining packages in Salsa are almost completely orthogonal, except that developers obviously can only use tag2upload if they are also using Git. As far as team versus individual maintainership, "tag2upload doesn't care at all about maintainership." It only cares about having a Git tag signed by a package maintainer that indicates an intent to upload.

Timo Röhling had a practical question: who would be responsible for the service once it is deployed? Jackson said that he expected to own the service together with Whitton, while the Debian System Administrators (DSA) team would manage the virtual machines.

One advantage to tag2upload, Matthias Urlichs wrote, is that the potential for discovering an attacker's modification is greater when using tag2upload. A signed tag, he noted, tracks the history of a set of files. A user could verify that an emergency NMU pushed to a package "only contains one commit on top of the maintainer's and changes only one file". Two files, if one counts the changelog. It is possible to do this with source packages, but he said it is more work, doesn't work well between upstream versions, and more: "all of which means that nobody's going to do it, much less notice said spurious change by accident."

One problem with tag2upload's design that Simon Josefsson called out is that it aggregates and amplifies the consequences of one successful compromise. While it may be easier to compromise, say, the laptop of a single packager, the reward (for an attacker) of compromising a tag2upload server would be much higher. However, he said that "adding another attack vector will not break this camel's back". The bigger problem, he said, was "how some people use their technical powers to limit what other people can do effectively."

Sprawling thread

On June 21, Allbery wrote that he was sure that many people had stopped reading because the discussion had become "a sprawling thread of nearly unreadable volume". He said that he was possibly the largest contributor to the volume, so he would "do penance and summarize the thread for everyone else".

Allbery said that it seemed that the ftpmaster team and tag2upload team had agreed that the authentication protocol for tag2upload should change. Instead of the tag2upload server performing the signature check and asking dak to trust that it had, it would include the signed Git tag and dak would redo the signature check. This would entail code changes for dak, and could be done with a proper API. He said that raised other questions about APIs between Debian project services but that should be deferred to later.

A remaining point of disagreement, he said, was that "tag2upload uploads will not contain an uploader signature over the exact files that comprise the source package". The transformation from a signed Git tree to source package might include "synthesizing or modifying files in ways that the uploader does not have in their Git tree in a hashable form". The ftpmaster team, Allbery said, wants the exact contents of every file in the source package to be covered by an uploader signature "other than the constructed *.dsc file" and possibly the *.orig.tar.gz file.

From a security standpoint, Allbery said, it makes little difference whether the source package construction happens before or after the signature "given that the uploader doesn't (and can't, realistically) check the output in either case". That seemed to be the remaining blocking issue, he said. Jackson thanked Allbery for the summary and said that he didn't spot anything he would disagree with in the summary.

Call for vote

On June 27, Whitton formally called for a GR to override the ftpmasters. The procedure for resolutions requires five additional sponsors, and a minimum of two weeks for discussion. Jaspert responded to the call for seconds with disappointment and asked why there was a hurry to run the GR now. "You waited 5 years without ever bringing this forward, and now it must be done right now, and can't wait more time? Oh come on." In another reply he disputed that the ftpmaster team had made a decision to reject uploads from tag2upload. He accused Whitton of taking a "my way or nothing" approach.

Allbery responded that he flatly did not believe Jaspert actually thought that ftpmaster had not made a decision:

There is literally no way that you could think, after all of this discussion, that you haven't been asked for a decision or what that decision is about.

He pointed out that Jaspert had asked for time ten days ago. When Whitton asked if the team was still considering options there was no response. Allbery said that the process was exhausting and that even though people are reluctant to use a GR to force a decision "at some point there *has to be an end*". If not, it would just be "decision by attrition".

Finally, compromise

Whitton did not reply directly to Jaspert, but on June 28 he wrote that he had put the GR forward for seconds because he wanted to minimize the impact of the GR on the upcoming DebConf and DebCamp. However, he said that he had received feedback that he had "misjudged the extent to which discussion is still ongoing" and appeared to hint at delaying or stopping the GR.

On June 30, Jaspert sent a proposal on how to move forward with tag2upload. It would require tag2upload to send a normal Debian source package, plus two files. He suggested using a Git command to generate a list of the file names, modes, and commit IDs that would be signed by the packager's key. The exact format, he said, could be worked out during implementation. The other file would be a shallow Git clone of the repository, put into a tarball, for the tag that is to be uploaded by tag2upload. That would be generated by the tag2upload service.

Allbery thanked Jaspert strongly for putting the proposal together, and they continued to hash out details on the list, with input from others. On July 1, Whitton cited the "productive discussion we're now having" and asked that Tille use his DPL powers to extend the discussion period for the GR to give the two camps time to nail down details with the anticipation that he would withdraw the GR once an agreement was reached. In his July 1 "Bits from the DPL" message Tille said that he would extend the discussion period to give the camps time to reach agreement and avoid a GR.

On July 2, Jackson seemed satisfied that an agreement had been reached, and said that he thought it was time to update the design of tag2upload with the agreed changes.

At this point, it seems that the parties will be able to move forward and eventually deploy tag2upload as a service for Debian's packagers. It is unfortunate that it took five years and the threat of a vote that would override the authority of a team to prod the parties into working together constructively to make what amounts to a small compromise and design change.