Python PGP proposal poses packaging puzzles
Sigstore is a project that is meant to simplify and improve the process of signing, verifying, and protecting software. It is a relatively new project, declared "generally available" in 2022. Python is an early adopter of sigstore; it started providing signatures for CPython artifacts with Python 3.11 in 2022. This is in addition to the OpenPGP signatures it has been providing since at least 2001. Now, Seth Michael Larson—the Python Software Foundation (PSF) security developer-in-residence—would like to deprecate the PGP signature and move to sigstore exclusively by next year. If that happens, it will involve some changes in the way that Linux distributions verify Python releases, since none of the major distributions have processes for working with sigstore.
PGP and sigstore
No doubt many readers already have some experience with using implementations of OpenPGP, Pretty Good Privacy (PGP) or GNU Privacy Guard (GPG), to sign and verify artifacts. PGP signatures have been used for decades to provide proof that tarballs, packages, ISO images, and so forth are genuine. The terms PGP and GPG are used interchangeably in the various discussion threads and elsewhere, but we will stick with "PGP" for simplicity's sake.
To sign an artifact, a developer needs to create a PGP key pair and publish the public key for interested parties to use to verify the signature. Anyone can create a key pair that claims to be from "Foo Project Release Manager <rm@foo-project.org>", use it to sign artifacts, and publish the public key near the artifact. Users have had to rely on published keys to judge whether the key used belongs to the person (or organization) that is supposed to have signed the artifact. The ideal scenario is being able to validate a key fingerprint via the "web of trust", but that can be difficult.
In some ways, PGP signing is part of open-source community culture. It used to be common to have signing parties at events and Linux User Group (LUG) meetings to help expand the web of trust. People would examine each others' passport or other identifying documentation, collect public-key fingerprints, and then sign each others' keys to affirm they belonged to the person in question. There are fewer key-signing parties at events these days, though they're still held at events like DebConf 2024.
The use of PGP has waned, but it is still actively used by upstream projects and Linux distributions. However, there are also many complaints about the complexity of managing and verifying PGP keys, as well as complaints about the deficiencies in the web-of-trust concept. The Linux kernel community has had to implement its own solution to maintain the kernel's web of trust in 2019, following a series of attacks on public keyservers.
Many developers would like PGP to go away, but there have been few alternatives to replace it. Sigstore is now being used by some projects to do just that. It was started by Red Hat to solve the problem of providing signatures for container images, it moved to the Linux Foundation in 2021, and then moved again to the foundation's Open Source Security Foundation (OpenSSF). The project offers several tools and services for signing, verifying, and enforcing policy based on sigstore metadata.
With sigstore, a developer does not need to create or maintain a key pair (though it is possible to do so). Instead, they only need to have an account with a provider, such as GitHub, GitLab, Google, or Microsoft, that uses OpenID Connect (OIDC) to verify identity. Note that those are the default providers, but it is possible to set up other OIDC providers. The developer then uses a client, such as cosign, that requests a certificate from the OpenID provider to verify the developer's identity. Assuming the developer authenticates successfully with the OpenID provider, a short-lived certificate is issued for signing the artifact and then the attestation is published to a public ledger called Rekor. Cosign is also used to verify signatures. The sigstore documentation has a detailed overview of the process and the various sigstore components. An open-access research paper is available for those who would like to go deeper into the concept.
In short, sigstore removes the need for developers to have, maintain, and distribute PGP keys. It replaces the web of trust with centralized authorities for verifying the identity of artifact signers. Some of the downsides are that it relies on a handful of providers and requires tooling changes for those who have automated PGP signature checking.
Proposal
Larson started
a "pre-PEP
" discussion on the Python forum about dropping PGP signatures on
September 25. Despite its newness, he made the case that sigstore
had been adopted by other major projects and services, such as the
Python Package Index (PyPI) (which dropped PGP support in 2023),
npm, and GitHub, and was "likely to stick around
". From a
technical perspective, he said it made sense for those signing
artifacts as well as those verifying them:
Sigstore has the benefit of not requiring release managers to maintain and protect separate long-lived expiring secret keys to maintain the integrity of Python artifacts, instead release managers only need to maintain control of an email account (something release managers already need to do). On the verifier side, Sigstore doesn't require discovering and fetching keys which might become unavailable, out of date, etc and doesn't require downloading an artifact to verify its integrity.
After some pre-PEP discussion, Larson published a draft of PEP 761 ("Deprecating PGP signatures for CPython artifacts") for discussion on October 9, and asked downstream users of the PGP signatures to weigh in.
If the PEP is accepted, CPython would phase out PGP signatures for the next major Python release, 3.14, due in October 2025. PGP signatures would still be published for updates to earlier CPython releases until those releases reach the end of their life. So, for example, the just-released Python 3.13 will continue to have PGP signatures for each update until it is discontinued in October 2029.
In the PEP, Larson argues that providing PGP and sigstore signatures fails to give downstream projects any incentive to adopt sigstore. So long as CPython continues to provide PGP signatures, there is little motivation to adopt sigstore.
Discussion
Larson's proposal drew support from Python contributors
on the forum. Hugo van Kemenade, a CPython core developer, said
that he would be the first release manager (RM) to be affected by the
change and was strongly in favor of it. "Sigstore looks a much
better solution, we already have it in place, and I'd much rather
spend my RM time on things that are more useful for our users
". He
also noted that he had not needed to use PGP for two decades:
"since Nokia (remember them?) used to mail us Symbian (remember
that?) SDKs encrypted on DVDs (remember those?)
".
Steve Dower, also a CPython core developer, weighed
in that PGP signatures for Windows and macOS Python builds are
"entirely redundant with the native OS embedded signatures we
use
", but that people complained when the project stopped providing
PGP signatures for those builds: "the only reason we still do them
is because people shouted last time we stopped
". Once signatures
were reinstated, he said, the shouting stopped.
Distributions weigh in
There has been no shouting in the discussions, but there have been concerns raised by maintainers from several Linux distributions. Miro Hrončok, Fedora's Python maintainer, said that Fedora verifies PGP signatures of Python tarballs when building packages, but said there were blockers to using sigstore to do the same. Specifically, he said that sigstore was missing offline verification and that the Python sigstore components had too many dependencies.
Larson pointed to the sigstore-go client as a potential solution to both problems. The discussion also spurred work on a longstanding issue of adding offline support to the sigstore-python tool. A release with offline support was uploaded to PyPI on October 10. Sigstore's cosign does support air-gapped or offline verification, though it requires downloading the trust root ahead of time to be able to use the --offline parameter.
Debian developer Stefano Rivera admitted that Debian had not been using PGP to verify CPython, because the package maintainers never got around to setting it up, or didn't consider it important. He said that he intended to ensure PGP signatures were checked in the future, and that a lot would need to happen for Debian to be able to support sigstore:
As to sigstore, we'd need to support it in our uscan tool, to have automated verification happen. To include the signature in our source packages, as PGP signatures are, we'd need to modify dpkg-source, and possibly the archive, to support storing them. It's hard to motivate that change for a single upstream. I know there are other ecosystems supporting it, but I've never run into any doing so actively.
Despite the extra work, though, Rivera said
that he thought there was agreement on goals, and it would be a matter
of "convincing people that sigstore is worth the effort of
implementation in Debian package tooling
." He started
a discussion on debian-devel about supporting sigstore, as well as
other mechanisms such as SSH signatures and signify.
Debian developer Guillem Jover provided some interesting thoughts on the matter. He was disappointed in the upstream discussion:
I find this usual conflation (in the upstream discussion) of GPG (or GnuPG) as if it was OpenPGP itself rather problematic, because the OpenPGP ecosystem is way richer than that, and as such way more active, and there has been and there is lots of work going on to implement and provide more usable, intuitive, secure and modern interfaces and code [...]
He was open to exploring other technologies for signing,
however. He suggested adding "partial
" support for other
methods, such as updating the uscan script that downloads sources and
verifies signatures, but not going so far as adding support to
dpkg. Rivera agreed and said that Debian might "talk about adding
support to dpkg later, when we can see significant use
".
Sam James, a Gentoo developer, recommended that Python continue using PGP signatures for one more release cycle to allow those downstream of Python to adapt and provide feedback:
We didn't have sigstore packaged at all before now and had to package a lot of new dependencies for it which took time. Others will surely be in the same position. We also have a lot of tooling around PGP and none for sigstore yet.
Larson thanked James for providing feedback and said that the final timeline was up to the steering council. He asked a few follow-up questions about Gentoo's timeline for starting to package 3.14, what blockers exist beyond offline verification, and when Gentoo might know what those blockers would be.
On October 12, Gentoo developer Michał Górny committed a series of patches to verify sigstore signatures for Gentoo, answering the question of when Gentoo might be able to adapt to sigstore. Hrončok said that Fedora would need to package cosign before it would be able to use it for verifying artifacts, but didn't say whether that is going to happen or not.
OpenSUSE developer Matěj Cepl brought up the topic on openSUSE's factory mailing list, and asked for thoughts on using sigstore. So far there have only been a few responses, and none have been enthusiastic about sigstore.
Next steps
The discussion is still ongoing, so it is possible that the steering council will decide to defer dropping PGP signatures to give downstream projects more time. Packagers from the major distributions have been quick to jump into this discussion, but it is asking a lot to require distributions to adapt to this change in less than 12 months. One thing that is clear from the discussions is that a few years is not enough time for a technology like sigstore to gain widespread acceptance and understanding. At least, not without forcing the conversation. Larson seems to be doing a good job of getting that ball rolling, while working collaboratively to clear obstacles from his goal of sigstore adoption.
Whether Python drops PGP signatures in 3.14 or a later release, it seems likely that it will drop them at some point in the not-so-distant future, and that Linux distributions will need to add sigstore verification to their repertoire. That will likely help drive its adoption further, which means that it's time for users of open-source software and Linux distributions to start paying closer attention to the technology and its implications.
| Index entries for this article | |
|---|---|
| Security | Python |
| Python | Packaging |
| Python | Security |
