Python cryptography, Rust, and Gentoo
There is always a certain amount of tension between the goals of those using older, less-popular architectures and the goals of projects targeting more mainstream users and systems. In many ways, our community has been spoiled by the number of architectures supported by GCC, but a lot of new software is not being written in C—and existing software is migrating away from it. The Rust language is often the choice these days for both new and existing code bases, but it is built with LLVM, which supports fewer architectures than GCC supports—and Linux runs on. So the question that arises is how much these older, non-Rusty architectures should be able to hold back future development; the answer, in several places now, has been "not much".
The latest issue came up on the Gentoo development mailing list; Michał
Górny noted
that the Python cryptography library
has started replacing some of its C code with Rust, which is now required
to build the library. Since the Gentoo Portage package manager
indirectly depends on cryptography, "we will
probably have to entirely drop support for architectures that are not
supported by Rust
". He listed five architectures that are not
supported by upstream Rust (alpha, hppa, ia64, m68k, and s390) and an additional five
that are supported but do not have Gentoo Rust packages (mips,
32-bit ppc, sparc, s390x, and riscv).
Górny filed a bug in the
cryptography GitHub repository, "but apparently
upstream considers Rust's 'memory safety' more important than ability to
actually use the package
". As might be guessed, the developers of
the library have a bit of a different way of looking at things. But the
enormous comment stream on the bug made it clear that many were taken by
surprise by the change that was made in version 3.4
of cryptography, which was released on February 7.
Christian
Heimes, one of the developers of contributors to the library, made
it clear that they would not be removing the dependence on Rust. He
pointed to an FAQ
entry on how to disable the Rust dependency for building the library,
but noted that it will not work when cryptography 3.5 is
released. He also pointed
out that Rust is solely a build-time dependency; there are no run-time
dependencies added.
But multiple people in the bug report complained about how the notice was given that the Rust dependency was being added; some thought that the project followed semantic versioning, which would mean that this kind of change should not come in a minor release. It turns out that the project has its own versioning scheme, which allows this kind of change (as does semantic versioning, actually). But Heimes did indicate that there may not have been sufficient communication about the change. He pointed to a pull request from July 2020 and a December 22 cryptography-dev mailing list announcement, both by Alex Gaynor, as places where the issue surfaced. Following links from those finds more discussion of the idea, but it is clear that news of the upcoming change did not reach far outside of the cryptography community. In part, that may be due to the usual way users get the library, as Heimes explained:
Many of the Alpine Linux users who were affected by the change, some of whom were loudly annoyed in comments on the bugs, have continuous integration and deployment (CI/CD) systems that update and build relevant packages frequently. In this case, though, the missing Rust compiler broke many of them. The most recent Alpine versions do have Rust support, though, so the fix there is fairly straightforward, or may be.
But for architectures that currently do not, and, in truth, likely never
will support Rust, there is no way forward except perhaps forking
cryptography and maintaining the C-based version going forward.
Górny suggested that in the gentoo-dev thread and in
the bug. Others were similarly inclined toward that, but it is unclear
if there is really enough wherewithal to support such a fork. Python 3.8
and 3.9 release manager Łukasz Langa challenged
Górny (and others) to proceed with a fork: "I invite you to do that. Please put
your money and time where your mouth is. Report back in a year's time how it went.
"
Langa also pointed out that the cryptography maintainers are volunteers as well, which means they get to allocate their efforts in whatever direction they wish, even if it makes it inconvenient for other volunteers elsewhere. Beyond that, those changes are being made for a reason:
[...] You expect those volunteers to keep their security-focused project dependent on inherently insecure technology because it would make your own job easier. Your goals and requirements might not be matching the goals and plans of the maintainers of cryptography. It might be unfortunate for you but it really is as simple as that.
The bug comments went on at length; there were some real problems that needed addressing in the way the CI/CD systems were handling versions in dependencies like cryptography, for example. But there was plenty of heat directed at the developers for "forcing" their Rust choice on others, and for "breaking" various systems. For their part, the developers have tried to help those with systems that can run Rust, but have shrugged their shoulders about the others.
Eventually, things boiled over and commenting was disallowed from anyone other than project contributors. Gaynor, in particular, felt that the problems were unavoidable for these, largely ancient, platforms. Once the thread had closed, he summarized what had been discussed and reiterated that the cryptography developers are not going to be held back by platforms that do not support Rust.
Back in Gentoo-land, it turned out that the cryptography dependency for Portage came because it was using urllib3 and requests. Those two packages in Gentoo are dependent on cryptography, but it turns out that they do not actually need it. A pull request to fix that was merged, so the problem for Portage, which is pretty fundamental to the operation of a Gentoo system, was averted.
At least it was averted for now. Górny is concerned
that the trustme TLS test-certificate
generator, which is used in the distribution's tests, does need
cryptography, so some platforms may not be able to be fully
tested. On the other hand, the cryptography developers have
decided may decide to create a 3.3 LTS release
that will maintain the pre-Rust version of the library until the end of
2021. Only fixes for CVEs will be made in that version, however.
But Górny has a bigger worry. He believes it is possible that some future version of Python itself will require Rust, though it is not entirely clear what he is basing that on. It would be devastating for Gentoo on the architectures that do not have Rust, since the distribution relies heavily on Python. It would seem likely to be problematic for other distributions as well, but the only real solution there is to get LLVM (thus Rust) working for those architectures—or for the gccrs GCC front-end for Rust (or a similar project) to make further progress.
While it may well be that Python itself does not go down that path, it is pretty clear that Rust is becoming more popular with every passing day. It would certainly be wonderful if it could be supported "everywhere", but it is going to take some real work to get there. The LLVM developers have been somewhat leery of taking on new architectures, unless they can be convinced there will be long-term support for them, which is understandable, but makes the problem even worse.
We saw a problem similar to that of Gentoo's back in 2018 with Debian and librsvg and we are likely to see it recur—frequently—over the coming years. It is not unreasonable for projects to use new tools, nor for projects to be uninterested in supporting ancient architectures. It is most certainly unfortunate, but we find ourselves between the proverbial rock and its friend, the hard place. Perhaps, with luck, something will change with that predicament.
| Index entries for this article | |
|---|---|
| Security | Python |
| Security | Rust |
| Python | Cryptography |
| Python | Other languages |