Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:36 UTC (Thu) by logang (subscriber, #127618)
In reply to: Python cryptography, Rust, and Gentoo by marcH
Parent article: Python cryptography, Rust, and Gentoo

>It seems straight-forward when you ignore all the work that distributions perform behind the scenes to achieve that result.

Absolutely right. It's a lot of work and a hard problem to deal with dependencies. Which is why we should pool the work in distributions and everyone should use and benefit from it.

>It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

That's an odd statement. I do that multiple times a week on more than a dozen machines.

>It seems straight-forward if you've never debugged CMake or (much worse) autotools.

I've done both. Not that hard.

>It seems straight-forward as long as you don't need different packages that require different versions of xyz.

If libraries are well maintained and care about not breaking their users, and support a range of their own dependencies (instead of essentially vendoring their own dependencies by insisting on a very specific version) this problem tends not to be that bad. Even in python, good well maintained libraries ensure they work on a wide range of python versions and with a range of versions of their own dependencies. But also, in general, long deep dependency trees should be avoided and pushed back against.

>It seems straight-forward as long as you don't try to naively "upgrade" the LTS version of your distro with packages from a newer version of the _same_ distro.

I've done this a lot. For the rare critical package, this is hard and should simply not be done. 9 times out of 10, it is easy.

> If it's so straight-forward, why have brand new projects like flatpak, snap etc. just been created?

No idea. But I avoid those like the plague. They don't solve any of my problems.

> Code re-use, software distribution and maintenance is hard, really hard. I'm not claiming rust or anything else cracked that nut, far from it and downloading random code from the Internet (in _any_ language_) is of course a security disaster[*] Pretending on the other hand that this problem has already been solved is either dishonest or incredibly naive and probably why the entire industry is still so bad at this. Have you never heard about "DLL Hell?". We should all keep open mind, take interest in any new approach and ignore anyone recommending to keep doing what they've have been always been doing.

Absolutely right. But the new languages don't seem to solve these problems, they just ignore them and try to vendor everything. From a security, maintenance and longevity perspective the distros have been doing far better, which is why I always go to them first and strongly resist the newer trends to vendor everything.

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:07 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> Absolutely right. It's a lot of work and a hard problem to deal with dependencies. Which is why we should pool the work in distributions and everyone should use and benefit from it.

You know how many users we have using distro-provided deployment mechanisms? Zero (that I hear from). I hear from distributor maintainers and we work to accomodate building with external deps (because *I* care and testing against external versions is the easiest way to set warning bells for API changes coming down the pipe).

Existing deployments are on bespoke machines with oddball dependencies not packaged by distros. They use custom MPI builds that are tuned for the hardware. External libraries compiled against those MPI libraries. And other things too.

I agree that distros do a lot of work and I'm grateful for it, but the "everyone deploys to Linux (or FreeBSD)" mentality (this is especially rampant in the web world too) is short-sighted to me. We vendor the "core" libraries we need. I even made sure we do it properly: no untracked patches to them, mangle the symbols, soname, and header include paths to avoid conflicts with external copies, provide options to *use* external copies, etc. It's a lot of work.

And after all that? I would really rather just drop a `Cargo.lock` file in for stability and have CI churn on new releases to let me know of what's up in the future.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:52 UTC (Thu) by roc (subscriber, #30627) [Link] (3 responses)

> But the new languages don't seem to solve these problems, they just ignore them and try to vendor everything

Effectively Rust wants developers to vendor everything, but a lot of work has gone into Rust+cargo to solve a lot of hard problems. For example:

cargo provides simple commands to update a dependency to the latest version, usually as simple as "cargo update" or "cargo update -p <library>".

cargo makes it easy to override a (possibly indirect) dependency with a patched version, via "[patch]".

rust-sec/advisory-db collects CVEs for Rust libraries and you can configure the cargo-deny tool to automatically break your build if one of your dependencies has an outstanding CVE.

Rust is designed so that by default linking multiple versions of the same library into a single binary works fine (always undesirable, but sometimes a necessary last resort).

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 23:31 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

Effectively Rust wants developers to vendor everything

I don't think this is quite right. As I understand it "vendoring" means copying the source code of libraries you used into your own source tree rather than linking to the distribution-provided library at run time. As I understand it, there are a few problems with vendoring:

Library fragmentation. When people on the project discover something wrong with the library- a bug or missing feature- there's a tendency to patch it in the local copy rather than pushing the fix to upstream. Even if the project attempts to push changes upstream, the project may keep them if upstream is uninterested, resulting in fragmentation of the library.
Patch delays. If something upstream gets patched, it takes extra time and effort to push the patch out to all the projects that have vendored the library compared to patching the single distribution provided version. This is annoying with ordinary bugs and a serious danger with security bugs.
Hidden copies. It can be difficult even to track down all the projects that have vendored the library to make sure their copy has been fixed. This further slows patch rollout.

What Rust (and many other languages with their own dependency resolution systems) does is slightly different. They incorporate libraries into a statically linked binary, but they still treat the library as an external dependency rather than copying it into the project wholesale. That means they still have problems with patch delays but much less of one with library fragmentation or hidden copies than projects which have truly vendored libraries.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:07 UTC (Fri) by marcH (subscriber, #57642) [Link]

> As I understand it "vendoring" means copying the source code of libraries you used into your own source tree [...] tendency to patch it in the local copy rather than pushing the fix to upstream

In other words forking the source.

> They incorporate libraries into a statically linked binary, but they still treat the library as an external dependency rather than copying it into the project wholesale.

In other words forking the binaries but not the source.

There are probably a few other (and incompatible...) "definitions" of vendoring, for instance those that (wrongly) care about where the copy is hosted, but I don't think any other vendoring definition matters besides the two ways of forking above. I suspect we can get rid of that new word and not lose anything - actually gain some clarity. Please prove me wrong!

Duplication is not bad in itself, it's bad only when it leads to Divergence.
https://doc.rust-lang.org/book/ch03-01-variables-and-muta...

I stopped saying "Copy/Paste", now I say Copy/Paste/Diverge. Even the least technical managers understand he latter.

Examples of Duplication that keeps Divergence under control: cache invalidation, RCU, version control, snapshot isolation, transactional memory,...

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:23 UTC (Fri) by roc (subscriber, #30627) [Link]

Yes, I used the term loosely. Sorry about that.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:13 UTC (Fri) by marcH (subscriber, #57642) [Link]

> > It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

(I meant "apt upgrade)

> That's an odd statement. I do that multiple times a week on more than a dozen machines.

Then pause once and try to gauge about how many persons and lines of code you trust every time you install or upgrade a few dozens packages. Maybe pip and cargo won't look that bad after all.