LWN: Comments on "Python cryptography, Rust, and Gentoo"

Python cryptography, Rust, and Gentoo

jengelh — Tue, 09 Jul 2024 18:27:48 +0000

>_This_ is "real" open-source: zero boundary between downloading/using

What you describe sounds more like "collaboration" than "open-source": a project can achieve open-source status with so little as having a OSI license, even if its sole author rejects all your PRs because he thinks the project only makes sense for him.

>Configuring and building C/C++ code at large is a nightmare and Linux distributions have been performing an amazing and critical job there. However to solve this they had to add layer(s) of indirection between software authors and users

It is thanks to this indirection - or rather: integration -, that gives projects visiblity in the first place. Things could have turned out quite differently: If librsvg could not be integrated again after the language switch, the distro would either sport no svg support, no browser or no users (or any combination).

Python cryptography, Rust, and Gentoo

mathstuf — Fri, 16 Apr 2021 22:23:49 +0000

> What is sad is that jerks are blindly following this without consideration for their own users.

That seems to be a lot of projection to me. If my target audience is mainstream platforms or the project's goal is to implement something securely, what obligations do I have to those who happen to find it useful in their niche without considering to make sure it's within my project's goal or target audience? If someone comes up to me and says "hey, I got it working on the Nintendo Switch last year, your latest release breaks it", what am I to do? I'm not getting (or potentially bricking) a Switch to test it and I never claimed to support such a thing anyways. Patches that don't interfere with other things are fine, but if you're asking me to resurrect some pre-refactoring code just for your setup, sorry. Feel free to fork though.

I suspect this will be used for new drivers. Existing drivers will need a lot of "oomph" to warrant a rewrite. It seems to also have put more interest into the GCC frontend, so if/when that reaches the finish line, it'll be just as good for all the other Linux platforms.

> With python cornering itself behind the sole list of platforms supported by rust and betraying its users, we're certain never to ever see python 4.

Python has done no such thing. The maintainers of cryptography may arguably have done this (I make no claim to either argument on that front here), but they are not the maintainers of the CPython project. Any Python 4 was known to never be a thing because the maintainers recognized something of the trainwreck that the Python3 migration ended up being.

Python cryptography, Rust, and Gentoo

wtarreau — Fri, 16 Apr 2021 21:35:26 +0000

> First, no language is going to show up with the platform support C has right out of the gate

You're confusing language and compiler. A language is platform-agnostic, it even works with paper and pen.

The problem here is that this so-called language knows a single compiler whose developers are not interested in porting to what they consider irrelevant platforms. It's their right, it's their project. What is sad is that jerks are blindly following this without consideration for their own users. With python cornering itself behind the sole list of platforms supported by rust and betraying its users, we're certain never to ever see python 4.

LLVM += new architecture

Gaelan — Fri, 26 Feb 2021 13:48:41 +0000

For what it's worth, there's bindgen, which automates the process of converting C headers to a Rust interface. It looks like the Rust people aren't using it for libc at the moment, but if you needed it to support a new architecture/libc, you could write a version of the libc crate that used bindgen, or just use bindgen's output as a starting point to add to the hand-written definitions in the libc crate.

Python cryptography, Rust, and Gentoo

myrrlyn — Thu, 25 Feb 2021 14:11:13 +0000

one of the reasons that i, personally, am looking forward to rust pushing out c is that i have a library that covers all of the points about bitfields you just laid out, and makes them trivial to correctly, conveniently, and performantly use in software that needs the space compaction

Python cryptography, Rust, and Gentoo

KZB — Sat, 20 Feb 2021 01:11:13 +0000

What the hack? Dropping Support for platforms because some people tend to use RUST, a language which is less well-known that C and less used?

That doesn't even make sense, even with the Security argumentation.
It's like "a complex password is more secure than a long password".
A false sense of security.

Somebody here has been trying to get into RUST by their bad documentation?
And they say "if you tend to use platforms WE DON'T SUPPORT.. you have to do it yourself, because WE ARE RUST".

The syntax of RUST and C is horrible. Lets switch to Python guys. * facepalm *

Also.. we should ask ourselves why RUST and these Python people trying to make it harder to get new platforms (and even older ones) supported. Hm... maybe "market reasons" ;-)

Python cryptography, Rust, and Gentoo

flussence — Thu, 18 Feb 2021 22:50:50 +0000

It's a bit ironic that the software I *most* need cross-compilation for is the stuff most resistant to being cross-compiled…

(Bought more RAM than I thought I'd ever need. The compiler crashes because it runs out of i686 registers now. *sigh*)

Python cryptography, Rust, and Gentoo

marcH — Thu, 18 Feb 2021 22:19:11 +0000

Imagine a few people on the Internet want to collaborate and start some experimental branch for a project hosted with subversion _without_ bothering the maintainers. They would most likely start by cloning the subversion project to git or similar - and submit back to subversion only in the end (if ever).

Python cryptography, Rust, and Gentoo

mathstuf — Thu, 18 Feb 2021 20:37:05 +0000

Sure. C is also 50 years old. Rust is what, 10? First, no language is going to show up with the platform support C has right out of the gate. Also, not all platforms are still relevant today. Those that are will gain Rust support if there's enough interest by users of those platforms to use the tools provided by Rust. If they're not, well, they won't get Rust software.

Python cryptography, Rust, and Gentoo

nybble41 — Thu, 18 Feb 2021 17:12:30 +0000

> In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Bitfields have some downsides, the biggest one being the lack of a standard, portable memory layout. The placement of individual bitfields within a structure is implementation-defined and varies in practice according to the target architecture, especially with respect to byte order. As a result, it is generally considered best practice to avoid directly sharing structures with bitfields between programs which may not share the same code-generation settings—for example, between user-space and kernel-space, or anything related to the layout of bits in a hardware register or persistent storage. Even for local data within the same process there are some ergonomic issues, such as the fact that one cannot take the address of a bitfield, make use of atomic operations, or easily set, clear, or test multiple bitfields within the same structure in the same operation in the same way that one can use bitwise AND/OR operations with a bitmask. The compiler can paper over some of these issues but this relies more heavily on the optimizer to merge together a series of one-bit read-modify-write sequences.

Python cryptography, Rust, and Gentoo

freem — Thu, 18 Feb 2021 11:39:56 +0000

>> which #defines can be ORed together

> Sure, at some point there is semantic that C doesn't provide. But that semantic is all the same for all Posix platforms...

Is it C which does not, or is it the source code?
In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Python cryptography, Rust, and Gentoo

lysse — Thu, 18 Feb 2021 07:38:51 +0000

Rust supports quite a few modern targets? That's great, but consider the number of targets supported by C: All of them.

Python cryptography, Rust, and Gentoo

rgmoore — Wed, 17 Feb 2021 19:15:07 +0000

I agree with LtWorf that "cannot really be considered open-source" is rather hyperbolic.

I think it's closer to true than you might expect. The GPL requires that source be provided in the preferred format for making modifications. That was meant to exclude things like generated code (rather than the material used to generate it) and obfuscated source, but it's not that far out to extend it to source being a copy of the version control system rather than just the raw source files.

Python cryptography, Rust, and Gentoo

mathstuf — Wed, 17 Feb 2021 16:38:05 +0000

Even then, you might be missing something due to an `#if` check you're not up-to-date with. I think the kernel providing its ABI via CTF or the like is *far* better in this realm (at least for the Linux-specific bits of the question). Of course, for libc/POSIX/etc., the headers *are* the definition, so that's what one should use there.

Python cryptography, Rust, and Gentoo

pizza — Wed, 17 Feb 2021 15:41:59 +0000

It's disingenuous to claim that "programmers choose to not write memory-safe" code. Bugs are (almost) never intentional.

But you're far more likely to get cut when playing with knives than with spoons.

Meanwhile, in the world I where I spend most of my F/OSS (and often, professional) time, the majority of the code I write is what other language consider inherently "unsafe". It's probably fair to say that C is the least-worst option.

Python cryptography, Rust, and Gentoo

Wol — Wed, 17 Feb 2021 15:37:34 +0000

Yup. Different languages, different strengths, different weaknesses. C *encourages* you to play with pointers, which means even experienced programmers use them when they're not necessary. And if you play with knives when you don't need to, you WILL, on average, get cut. Sometimes badly.

I'm sure Rust has its faults. My favourite language, DataBasic, has quite a few. But one of the biggest flaws in a language is using it in an environment for which it is not suited. C *was* brilliant as a low-level system language. Hardware has evolved. C is no longer low-level. People still use it as a low-level language and get badly sliced by the impedence mismatch between what C thinks the hardware is, and what the hardware really is. And it's the easy access to pointers that encourages this dangerous behaviour.

Cheers,
Wol

Python cryptography, Rust, and Gentoo

iainn — Wed, 17 Feb 2021 15:04:45 +0000

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Isn't that more of an argument to perform the autogeneration in a hermetic environment, like a container? Maintainers might also have Randomish environments.

(I agree the uploaded create should contain the autogenerated code; not needing libc headers, as discussed, is a good reason.)

Python cryptography, Rust, and Gentoo

mathstuf — Wed, 17 Feb 2021 14:28:24 +0000

Sure, writing secure C is *possible*, but I think that, as a whole, we programmers have proven to be pretty shitty at it. If the Linux kernel code review process with all the C veterans can't get it right (just look at the stable kernel patch queue!), what makes you think it's viable for the general coding population to use it? Sure, one can use clang-tidy, sanitizers, valgrind, etc. on it, but I see that as a failing of the language being propped up by expensive tooling rather than a benefit of the language itself.

Python cryptography, Rust, and Gentoo

kmweber — Wed, 17 Feb 2021 13:25:37 +0000

And, I mean, it's not like it's particularly difficult to write memory-safe code in C. I don't know where this myth that C is an "inherently insecure platform" comes from. It's not. It's exactly as easy to write safe code in C, as it is to write unsafe. It's programmers who choose not to, not the language forcing it on them.

Python cryptography, Rust, and Gentoo

MrWim — Wed, 17 Feb 2021 10:39:16 +0000

> This is why decentralized version control felt liberating.

Thanks for this analogy, there's definitely something to it. Something about not having to ask permission before acting, but instead being able to develop using the same tools as anyone, publish the results and have the results be judged instead.

Maybe what git is to a project, cargo is to a super project, or dependency graph. Hmm, that doesn't quite feel right because the versioning is still provided by git. It's the lockfile that extends git semantics to your entire dependency graph. Hmm, not sure if that's right, I'll have to think on it, but the analogy is food for thought.

As it is I'm a big fan of lockfiles, which can even be applied to whole distros and I believe that good tooling can unlock functional freedom, although I agree with LtWorf that "cannot really be considered open-source" is rather hyperbolic.

Python cryptography, Rust, and Gentoo

marcH — Wed, 17 Feb 2021 08:44:59 +0000

FYI, the usual behavior on this site when you don't understand something is to either ask or ignore.

Python cryptography, Rust, and Gentoo

geert — Wed, 17 Feb 2021 08:33:07 +0000

If the compilation is needed due to a change in a dependency, it is not a repeat of the previous build. If it was, there was no point in recompiling it.
If the compiler has changed, the recompiled application may behave differently, too.

Python cryptography, Rust, and Gentoo

Cyberax — Wed, 17 Feb 2021 04:59:44 +0000

> Recompilation has been known to randomly break applications as well.
Uhh.... Whut? Recompilation can't break applications, especially with repeatable builds. A bad fix that changes the API can certainly do that.

But it's way better than random breakages because the ABI has subtly changed.

Python cryptography, Rust, and Gentoo

dvdeug — Wed, 17 Feb 2021 02:31:57 +0000

Recompilation has been known to randomly break applications as well. The art of a good security patch is that it doesn't change anything besides making the security fix.

Python cryptography, Rust, and Gentoo

foom — Tue, 16 Feb 2021 16:40:30 +0000

If we had the ability to cross-compile for slow target architectures, and proper build automation, recompiling everything that depends on libpng wouldn't need to be a problem.

Distributing the update to users might need some adjustments, too, in order to avoid massive bandwidth usage -- a good mechanism to send just binary deltas for the affected files would be more important than it is now.

Python cryptography, Rust, and Gentoo

anton — Tue, 16 Feb 2021 15:28:43 +0000

As I wrote: "I mean C as used in many programs", and I actually point to a paper where I explain this in more detail. As for "pessimizing", it's certainly the case that advocates of adversarial C compilers claim that the adversarial behaviour is good for performance, invariably without giving any numbers to support these claims; maybe they think that repeating these claims and wishful thinking makes them true.

Wang et al. checked that for gcc-4.7 and clang-3.1 and found that the adversarial "optimizations" produced a minimal speedup on SPECint 2006, and that speedup could also be achieved by small changes to the source code in two places.

Yes, a performance advisor that points out places where changing the source code may improve performance would be a more productive way to spend both the compiler maintainer's and the compiler user's time than writing "optimizations" that break existing code, "sanitiziers" to find the places where the "optimizations" break it, and, on the programmer's side, "sanitizing" their code to withstand the latest attacks by "optimizers" (but not the next one). Moreover, such an advisor could point out optimizations that a programmer can do but a conformant compiler cannot (e.g., because there is an alias that even language lawyering cannot explain away). Of course, unlike maintained programs, benchmarks would not benefit from such a performance advisor, that's why no work goes into performance advisors; and conversely, "optimizations" don't break benchmarks (the compiler maintainers revert the "optimization" in that case), unlike other programs, and that's why we see "optimizations".

But what's more, by insisting on a very limited interpretation of what C means, the language lawyers remove opportunities for optimizations that programmers can make in the source code. I have discussed this at length.

I, too am skeptical that trying to change the C standard is the way to get rid of adversarial C compilers (not the least because you won't be able to achieve consensus with the implementors of these compilers on the committee), and I guess that's why advocates of adversarial compilers like to direct the blame for the misdeeds of these compilers at the standard, rather than at the compiler maintainers. It's not the standard that requires them to miscompile existing, tested and working, programs, it's the compiler maintainers' choice, so they alone are responsible.

Concerning architectures with other than two's complement representation of signed numbers, the last new such architecture was introduced over 50 years ago, and descendants of such architectures exist only in very few places and run select programs. There are far fewer of these machines than of the architectures (all two's-complement) that are not LLVM targets and that have brought about the parent article. And coding in a way that makes use of knowledge about the representation is one of the things that you can do for performance in a low-level language (and compilers do not perform all of these optimizations).

Python cryptography, Rust, and Gentoo

mathstuf — Tue, 16 Feb 2021 13:18:38 +0000

> I did not mean the language lawyer version of C.

What version of C are you talking about? The ISO standard? The image of C that you have in your head? My head? The C that GCC 2.95 accepted and worked with?

Let's imagine a world where C compilers magically stop doing "magic optimization" steps that tend to break code. What's going to happen is that C programmers that don't know this stuff already is going to have their code be pessimized and, presumably, slower in practice. What are they going to do? Start writing their C in such a way that the compiler was doing internal transformations during optimization passes anyways. They'll learn C more (and, I imagine, less satisfied with it), hopefully be using linters and tooling to tell them where their NULL checks are inverted with uses and such.

Rereading that, maybe it wouldn't be so bad. Maybe folks would migrate to better languages. Others might actually learn more about how loose C is in practice. The optimization passes could be migrated to the linters rather than the compiler to explain "hey, you could reorder your code to This Way and gain some performance". Maybe these passes would then gain some prose explaining what and why of them.

Then again, I have no idea such a C would be specified at ISO to disallow these optimizations while still allowing for architectures to not be forced into twos-complement representations or the like because "it's faster/easier for them".

Python cryptography, Rust, and Gentoo

mathstuf — Tue, 16 Feb 2021 13:07:16 +0000

> And what if they completely reject your changes or require substantial changes?

If we had gone with your proposal, now my distro is stuck with a patch rejected by upstream. Yay?

In this case, I rework the patch and adapt my code when I point to the next proposed patch. Same as step 1, just with a different baseline to diff from.

> Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

As if Linux is the only distribution platform for projects these days. You do realize that Linux (and the BSDs) are the oddballs out here, right? Pretty much everything else does vendoring or the like to a large extent. And if I want a turnkey release from my website, a tarball with dependencies embedded is the answer even for Linux without waiting for distros to churn on the new release (which generally takes a month or two to hit the unstable channels for our project; add a release cycle for the stable channel to have a chance).

Python cryptography, Rust, and Gentoo

LtWorf — Tue, 16 Feb 2021 10:54:34 +0000

Next time make one that makes sense?

Python cryptography, Rust, and Gentoo

anton — Tue, 16 Feb 2021 09:27:33 +0000

I did not mean the language lawyer version of C. That version is a bad fit for any architecture (including the PDP-11). However, it's great for adversarial compiler maintainers who want to do whatever they want (e.g., produce good benchmark results, grudgingly cater to requests by paying customers and tell other users that their bug reports are invalid), because this version allows them to always blame the programmer for something or other. After all, no terminating C program is a "strictly conformant program", and whenever someone mentions "conformant program" (the only other conformance level for program defined in the C standard), the adversaries produce advocacy why we should consider "conformant programs" as meaningless (interestingly, they claim that we should take the rest of the C standard at face value).

I mean C as used in many programs, which has a pretty simple correspondence to architectural features (and you see it easily in contexts where optimization does not set in, e.g., when you separately compile a function that performs just one thing).

The adversaries want us to consider C as a high-level language with no correspondence to the bare metal; that makes it easier to blame the programmers and absolve compiler maintainers of responsibility. The question is why any programmer would want that. We have plenty of high-level languages, often better than C in that capacity, but not that many low-level languages; basically C is the only popular one.

Concerning a totally defined C: I think that is at odds with a low-level language for multiple architectures, but as most (all?) C compilers have demonstrated for the first quarter-century of the language, that's no hindrance for implementing C in a benign rather than adversarial way. And for those who don't know how to do that, I have written a paper (which also explains why I consider totally defined C impractical).

I don't know what retpolines have to do with any of that. They are a workaround for a vulnerability in some microarchitectures and they cannot be implemented in C (there are limitations to C's low-level nature). The vulnerability should be fixed at the microarchitecture level, and I expect that the hardware manufacturers will come out with microarchitectures that do not have this vulnerability.

Python cryptography, Rust, and Gentoo

matthias — Tue, 16 Feb 2021 08:28:50 +0000

>> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.
> That's basically the opposite of Torvald's approach :D
No, it is exactly Torvalds' approach. The kernel policy is no regressions, yes. But Torvalds does not do the QA for all software that runs on linux. That would be simply impossible. He releases -rc versions of the kernel and asks everyone to test. If regressions are reported, the corresponding patches are reverted (or improved). But it is the job of the users (distros, cloud service providers, etc.) to do the testing and validation.

And it is definitely up to the distros to test whether a new kernel version works for them before they include it into the distribution.

The main difference between the kernel and many other projects is, that the kernel developers care much more about the regression reports received from the users and that regression reports have higher priority than new features.

Python cryptography, Rust, and Gentoo

marcH — Tue, 16 Feb 2021 07:38:59 +0000

Next time at least pretend to try to get the point.

Python cryptography, Rust, and Gentoo

LtWorf — Tue, 16 Feb 2021 06:57:22 +0000

> I don't need to wait for the maintainer to evaluate my patches for me to continue with my development.

And what if they completely reject your changes or require substantial changes?

This can be for several reasons.

What would you do at that point? You'd need to either give up using the feature you added or refactor your code.

> What I don't like about the traditional dynamically linked distro model is that this is enforced by technology, rather than policy.

The debian policy does not say anything against static linking vs dynamic linking. It does mandate that there can exist only 1 copy of a certain source within the archive. Browsers are granted exceptions because they are irreplaceable and do whatever they want.

> This level of effort from a library maintainer is reasonable to expect - but it would be unfair to ask that maintainer to do QA on all the packages that depend on their package to confirm the lack of breakage. They may not even know how some their dependencies are supposed to behave, so noting that they're not behaving correctly after the patch has been applied would be very difficult indeed.

That's basically the opposite of Torvald's approach :D

> we can more gracefully upgrade libraries without giving up on the **policy** that there should only be one version.

Libraries that are made to support it are normally available in multiple versions for a period of time. Typical example is Qt. Qt4 has been removed only very recently. But point releases replace the previous version, because we don't want to do the go way of depending on a specific commit.

> It seems that our disagreement is a matter of priorities. My understanding is that you believe in the primacy of the distro, and as such helping the distro has highest priority. Conversely my priorities are my application and users, the upstream library and only then the distro.

And when the user ends up with unpatched vulnerabilities because he downloaded some binary from some website because it wasn't included in a distribution because it violated every existing policy… how is the user served well by this?

He has to hope the author has a system in place to recompile when a CVE appears, that there is a repository set up to get the latest version. Using a distribution this would all be solved already rather than having to be solved 10000 times by every single project.

Python cryptography, Rust, and Gentoo

LtWorf — Tue, 16 Feb 2021 06:43:39 +0000

Open source has a precise definition that has absolutely nothing to do with your preferred version control system.

Python cryptography, Rust, and Gentoo

Cyberax — Tue, 16 Feb 2021 02:55:35 +0000

And then some applications randomly break because of an ABI change.

Python cryptography, Rust, and Gentoo

dvdeug — Tue, 16 Feb 2021 02:14:59 +0000

Let's compare that to what we have right now; we have a CVE in libpng, we upgrade the version of libpng in the distro, and fix all the packages without recompilation. That's already complex enough without literally recompiling almost every program on the system.

"fringe platforms which can't even run a Rust compiler"

sthibaul — Mon, 15 Feb 2021 22:47:51 +0000

> building an interesting new hardware platform

I'm not talking only about hardware platform, but also OS.

> That includes the resources to port all the significant languages to work with their new platform.

Yes, but that doesn't mean that projects shouldn't make reasonable attempts to make it not too hard.

Python cryptography, Rust, and Gentoo

rgmoore — Mon, 15 Feb 2021 22:46:41 +0000

GEB is not an easy read, but it is probably as easy and fun a read as any book that makes a serious pretense of explaining Gödel's Incompleteness Theorem is likely to be.

"fringe platforms which can't even run a Rust compiler"

rgmoore — Mon, 15 Feb 2021 22:45:06 +0000

The difference is that building an interesting new hardware platform is inherently a major undertaking, and it's only going to be done by an organization with substantial resources. That includes the resources to port all the significant languages to work with their new platform. Being able to do that is more or less required for a new platform to be interesting. A platform that can't run the languages people care about isn't very interesting.

Python cryptography, Rust, and Gentoo

Wol — Mon, 15 Feb 2021 19:44:35 +0000

Thing is, C *is* a bad fit for modern architectures. It has a whole bunch of features that are undefined, or implementation-defined, which are MEANT to be low-level "match the hardware". Except that they aren't.

Let's just get rid of all these so-called "low level" cock-ups, accept that C is now a high-level language and that undefined and implemetation-specific behaviours shouldn't exist, and move on.

Someone brought up retpolines - that monstrosity that tries to make sure that both the hardware and the software agree on the imaginary hardware interface that's where the CPU microcode and language macrocode meet ... wtf are we doing!

Cheers,
Wol