Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:16 UTC (Thu) by marcH (subscriber, #57642)
In reply to: Python cryptography, Rust, and Gentoo by logang
Parent article: Python cryptography, Rust, and Gentoo

> apt search xyz
> apt install libxyz
> How is that not straightforward?

It seems straight-forward when you ignore all the work that distributions perform behind the scenes to achieve that result.

It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

It seems straight-forward if you've never debugged CMake or (much worse) autotools.

It seems straight-forward as long as you don't need different packages that require different versions of xyz.

It seems straight-forward as long as you don't try to use a package from another distro because it's missing on yours.

It seems straight-forward as long as you don't try to naively "upgrade" the LTS version of your distro with packages from a newer version of the _same_ distro.

If it's so straight-forward, why have brand new projects like flatpak, snap etc. just been created?

Code re-use, software distribution and maintenance is hard, really hard. I'm not claiming rust or anything else cracked that nut, far from it and downloading random code from the Internet (in _any_ language_) is of course a security disaster[*] Pretending on the other hand that this problem has already been solved is either dishonest or incredibly naive and probably why the entire industry is still so bad at this. Have you never heard about "DLL Hell?". We should all keep open mind, take interest in any new approach and ignore anyone recommending to keep doing what they've have been always been doing.

[*] latest and greatest fun: https://www.theregister.com/2021/02/10/library_dependenci...

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:36 UTC (Thu) by logang (subscriber, #127618) [Link] (6 responses)

>It seems straight-forward when you ignore all the work that distributions perform behind the scenes to achieve that result.

Absolutely right. It's a lot of work and a hard problem to deal with dependencies. Which is why we should pool the work in distributions and everyone should use and benefit from it.

>It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

That's an odd statement. I do that multiple times a week on more than a dozen machines.

>It seems straight-forward if you've never debugged CMake or (much worse) autotools.

I've done both. Not that hard.

>It seems straight-forward as long as you don't need different packages that require different versions of xyz.

If libraries are well maintained and care about not breaking their users, and support a range of their own dependencies (instead of essentially vendoring their own dependencies by insisting on a very specific version) this problem tends not to be that bad. Even in python, good well maintained libraries ensure they work on a wide range of python versions and with a range of versions of their own dependencies. But also, in general, long deep dependency trees should be avoided and pushed back against.

>It seems straight-forward as long as you don't try to naively "upgrade" the LTS version of your distro with packages from a newer version of the _same_ distro.

I've done this a lot. For the rare critical package, this is hard and should simply not be done. 9 times out of 10, it is easy.

> If it's so straight-forward, why have brand new projects like flatpak, snap etc. just been created?

No idea. But I avoid those like the plague. They don't solve any of my problems.

> Code re-use, software distribution and maintenance is hard, really hard. I'm not claiming rust or anything else cracked that nut, far from it and downloading random code from the Internet (in _any_ language_) is of course a security disaster[*] Pretending on the other hand that this problem has already been solved is either dishonest or incredibly naive and probably why the entire industry is still so bad at this. Have you never heard about "DLL Hell?". We should all keep open mind, take interest in any new approach and ignore anyone recommending to keep doing what they've have been always been doing.

Absolutely right. But the new languages don't seem to solve these problems, they just ignore them and try to vendor everything. From a security, maintenance and longevity perspective the distros have been doing far better, which is why I always go to them first and strongly resist the newer trends to vendor everything.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:07 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> Absolutely right. It's a lot of work and a hard problem to deal with dependencies. Which is why we should pool the work in distributions and everyone should use and benefit from it.

You know how many users we have using distro-provided deployment mechanisms? Zero (that I hear from). I hear from distributor maintainers and we work to accomodate building with external deps (because *I* care and testing against external versions is the easiest way to set warning bells for API changes coming down the pipe).

Existing deployments are on bespoke machines with oddball dependencies not packaged by distros. They use custom MPI builds that are tuned for the hardware. External libraries compiled against those MPI libraries. And other things too.

I agree that distros do a lot of work and I'm grateful for it, but the "everyone deploys to Linux (or FreeBSD)" mentality (this is especially rampant in the web world too) is short-sighted to me. We vendor the "core" libraries we need. I even made sure we do it properly: no untracked patches to them, mangle the symbols, soname, and header include paths to avoid conflicts with external copies, provide options to *use* external copies, etc. It's a lot of work.

And after all that? I would really rather just drop a `Cargo.lock` file in for stability and have CI churn on new releases to let me know of what's up in the future.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:52 UTC (Thu) by roc (subscriber, #30627) [Link] (3 responses)

> But the new languages don't seem to solve these problems, they just ignore them and try to vendor everything

Effectively Rust wants developers to vendor everything, but a lot of work has gone into Rust+cargo to solve a lot of hard problems. For example:

cargo provides simple commands to update a dependency to the latest version, usually as simple as "cargo update" or "cargo update -p <library>".

cargo makes it easy to override a (possibly indirect) dependency with a patched version, via "[patch]".

rust-sec/advisory-db collects CVEs for Rust libraries and you can configure the cargo-deny tool to automatically break your build if one of your dependencies has an outstanding CVE.

Rust is designed so that by default linking multiple versions of the same library into a single binary works fine (always undesirable, but sometimes a necessary last resort).

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 23:31 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (2 responses)

Effectively Rust wants developers to vendor everything

I don't think this is quite right. As I understand it "vendoring" means copying the source code of libraries you used into your own source tree rather than linking to the distribution-provided library at run time. As I understand it, there are a few problems with vendoring:

Library fragmentation. When people on the project discover something wrong with the library- a bug or missing feature- there's a tendency to patch it in the local copy rather than pushing the fix to upstream. Even if the project attempts to push changes upstream, the project may keep them if upstream is uninterested, resulting in fragmentation of the library.
Patch delays. If something upstream gets patched, it takes extra time and effort to push the patch out to all the projects that have vendored the library compared to patching the single distribution provided version. This is annoying with ordinary bugs and a serious danger with security bugs.
Hidden copies. It can be difficult even to track down all the projects that have vendored the library to make sure their copy has been fixed. This further slows patch rollout.

What Rust (and many other languages with their own dependency resolution systems) does is slightly different. They incorporate libraries into a statically linked binary, but they still treat the library as an external dependency rather than copying it into the project wholesale. That means they still have problems with patch delays but much less of one with library fragmentation or hidden copies than projects which have truly vendored libraries.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:07 UTC (Fri) by marcH (subscriber, #57642) [Link]

> As I understand it "vendoring" means copying the source code of libraries you used into your own source tree [...] tendency to patch it in the local copy rather than pushing the fix to upstream

In other words forking the source.

> They incorporate libraries into a statically linked binary, but they still treat the library as an external dependency rather than copying it into the project wholesale.

In other words forking the binaries but not the source.

There are probably a few other (and incompatible...) "definitions" of vendoring, for instance those that (wrongly) care about where the copy is hosted, but I don't think any other vendoring definition matters besides the two ways of forking above. I suspect we can get rid of that new word and not lose anything - actually gain some clarity. Please prove me wrong!

Duplication is not bad in itself, it's bad only when it leads to Divergence.
https://doc.rust-lang.org/book/ch03-01-variables-and-muta...

I stopped saying "Copy/Paste", now I say Copy/Paste/Diverge. Even the least technical managers understand he latter.

Examples of Duplication that keeps Divergence under control: cache invalidation, RCU, version control, snapshot isolation, transactional memory,...

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:23 UTC (Fri) by roc (subscriber, #30627) [Link]

Yes, I used the term loosely. Sorry about that.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 1:13 UTC (Fri) by marcH (subscriber, #57642) [Link]

> > It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

(I meant "apt upgrade)

> That's an odd statement. I do that multiple times a week on more than a dozen machines.

Then pause once and try to gauge about how many persons and lines of code you trust every time you install or upgrade a few dozens packages. Maybe pip and cargo won't look that bad after all.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 11:15 UTC (Fri) by LtWorf (subscriber, #124958) [Link] (9 responses)

> It seems straight-forward if you ignore the incredibly large attack surface involved every time you run "apt update".

Uh?

How is this more risky than having 900 copies of libpng and hoping that all of them will be upgraded when inevitably the next buffer overflow is found?

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 17:03 UTC (Fri) by marcH (subscriber, #57642) [Link]

I'm not the one pretending to know which is more risky.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 22:12 UTC (Fri) by roc (subscriber, #30627) [Link] (7 responses)

Suppose we had a distro where all packages were built with Rust and statically linked the "png" crate, a CVE is issued for that crate, and a new minor version of "png" is available that fixes the bug. It would be very simple to scan all Cargo.lock files for all packages to see which ones are using a vulnerable version of "png". For each affected package, "cargo update -p png" would update to a non-vulnerable version. It would be easy to automate the entire process.

In this hypothetical distro you would also want to run 'cargo-deny' in CI to ensure that every time a package is built, the build fails if there is an outstanding CVE against one of its components.

The big picture here is that Rust+cargo standardize the build process and metadata to make managing dependencies much easier, more consistent and scalable.

(Of course we're ignoring the issue that you will have to do this much less frequently for a Rust PNG library because Rust code isn't prone to buffer overflows...)

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 2:14 UTC (Tue) by dvdeug (subscriber, #10998) [Link] (6 responses)

Let's compare that to what we have right now; we have a CVE in libpng, we upgrade the version of libpng in the distro, and fix all the packages without recompilation. That's already complex enough without literally recompiling almost every program on the system.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 2:55 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

And then some applications randomly break because of an ABI change.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 2:31 UTC (Wed) by dvdeug (subscriber, #10998) [Link] (2 responses)

Recompilation has been known to randomly break applications as well. The art of a good security patch is that it doesn't change anything besides making the security fix.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 4:59 UTC (Wed) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> Recompilation has been known to randomly break applications as well.
Uhh.... Whut? Recompilation can't break applications, especially with repeatable builds. A bad fix that changes the API can certainly do that.

But it's way better than random breakages because the ABI has subtly changed.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 8:33 UTC (Wed) by geert (subscriber, #98403) [Link]

If the compilation is needed due to a change in a dependency, it is not a repeat of the previous build. If it was, there was no point in recompiling it.
If the compiler has changed, the recompiled application may behave differently, too.

Python cryptography, Rust, and Gentoo

Posted Feb 16, 2021 16:40 UTC (Tue) by foom (subscriber, #14868) [Link] (1 responses)

If we had the ability to cross-compile for slow target architectures, and proper build automation, recompiling everything that depends on libpng wouldn't need to be a problem.

Distributing the update to users might need some adjustments, too, in order to avoid massive bandwidth usage -- a good mechanism to send just binary deltas for the affected files would be more important than it is now.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 22:50 UTC (Thu) by flussence (guest, #85566) [Link]

It's a bit ironic that the software I *most* need cross-compilation for is the stuff most resistant to being cross-compiled…

(Bought more RAM than I thought I'd ever need. The compiler crashes because it runs out of i686 registers now. *sigh*)