Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 8:52 UTC (Thu) by josh (subscriber, #17465)
In reply to: Python cryptography, Rust, and Gentoo by sthibaul
Parent article: Python cryptography, Rust, and Gentoo

> And Rust people don't seem to care about the situation since they have already ported to the only few platforms they care about...

We care a great deal about portability, and Rust supports quite a few target platforms, including both production and hobby platforms.

All of the mentioned platforms have not had hardware manufactured for years.

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 11:02 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (16 responses)

> > And Rust people don't seem to care about the situation since they have already ported to the only few platforms they care about...
>
> We care a great deal about portability

Then where is a tool that just picks up the libc headers to determine the sizes of structures etc. rather than having to hardcode everything in the rust repository? I did spend some time at some point to start writing them, copy/pasting from BSD files for a fair part, but everything has to be checked bit for bit to be sure that it's accurate, that is a very bug-prone process.

And... the pace at which Rust goes made me feel that probably what I wrote at the time is simply completely outdated since then and I'd better restart over from scratch next time I happen to find some time on this.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:28 UTC (Thu) by dbaker (guest, #89236) [Link]

Are you talking about bindgen?

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:32 UTC (Thu) by farnz (subscriber, #17727) [Link] (7 responses)

The tool that does that is called "bindgen". It's imperfect (necessarily so, because some of the descriptions in libc headers are in the form of human constraints on - e.g. - which #defines can be ORed together) but it does a decent first cut. The need for better is why the libc crate has hand-produced bindings instead, which have been human-read and the applicable extra constraints applied.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:19 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (6 responses)

> The tool that does that is called "bindgen".

Ok, thanks for the hint! At the time (novembre 2018) there was very little mention of this within rustc, and people had told me that they didn't think there was something like that.

> which #defines can be ORed together

Sure, at some point there is semantic that C doesn't provide. But that semantic is all the same for all Posix platforms...

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:50 UTC (Thu) by farnz (subscriber, #17727) [Link] (1 responses)

Unfortunately, it's not the same for all POSIX platforms - just about every platform has extensions to POSIX that aren't the same as other platforms :-(

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:55 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link]

> Unfortunately, it's not the same for all POSIX platforms - just about every platform has extensions to POSIX that aren't the same as other platforms :-(

Sure there are some extensions, but that's not much compared to the common Posix API.

Porting rust to a system that provides the Posix interface should just be a matter of running a script that looks for the ABI of the Posix interface (even if only to record it in the source tree). Like perl has been doing for decades.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:57 UTC (Thu) by josh (subscriber, #17465) [Link]

bindgen has been around for a while, but it's fiddly to build. I'm currently working on some things that may improve that.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 11:39 UTC (Thu) by freem (guest, #121851) [Link] (2 responses)

>> which #defines can be ORed together

> Sure, at some point there is semantic that C doesn't provide. But that semantic is all the same for all Posix platforms...

Is it C which does not, or is it the source code?
In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 17:12 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (1 responses)

> In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Bitfields have some downsides, the biggest one being the lack of a standard, portable memory layout. The placement of individual bitfields within a structure is implementation-defined and varies in practice according to the target architecture, especially with respect to byte order. As a result, it is generally considered best practice to avoid directly sharing structures with bitfields between programs which may not share the same code-generation settings—for example, between user-space and kernel-space, or anything related to the layout of bits in a hardware register or persistent storage. Even for local data within the same process there are some ergonomic issues, such as the fact that one cannot take the address of a bitfield, make use of atomic operations, or easily set, clear, or test multiple bitfields within the same structure in the same operation in the same way that one can use bitwise AND/OR operations with a bitmask. The compiler can paper over some of these issues but this relies more heavily on the optimizer to merge together a series of one-bit read-modify-write sequences.

Python cryptography, Rust, and Gentoo

Posted Feb 25, 2021 14:11 UTC (Thu) by myrrlyn (guest, #145084) [Link]

one of the reasons that i, personally, am looking forward to rust pushing out c is that i have a library that covers all of the points about bitfields you just laid out, and makes them trivial to correctly, conveniently, and performantly use in software that needs the space compaction

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:03 UTC (Thu) by josh (subscriber, #17465) [Link] (6 responses)

I would love to see much of the libc crate autogenerated; someone would need to write the necessary code to make that happen. It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need). That would also make the libc crate dependent on the version of headers you have installed. Today, the libc crate carefully avoids using features that are "too new" for the versions of libc that it supports. (Rust typically supports running with the version of glibc from the previous RHEL or Debian stable, for instance, rather than requiring the latest libc.)

So, autogenerating the libc crate would likely need to happen as part of constructing that crate (before publishing it), or alternatively the libc crate would need to ship with copies of appropriate versions of the libc headers for supported platforms. Either way, this would likely be welcome, but would require some substantial infrastructure to implement. If someone is interested in implementing that, I would suggest starting a conversation on the libc issue tracker about what the architecture and requirements for that would look like.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:11 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (5 responses)

> It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need).

I don't really understand that argument.

When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't. Making platform ports and libc bindings upgrade way more difficult just for this reason seems terribly weak to me.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:18 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (3 responses)

The problem is that it adds another axis to your "build environment matrix". In addition to target triple and compiler version, I now need to ask "what libc do you have installed?". And I need to document how to get that information for any given build environment. Oh, you're on macOS trying to cross-compile to Linux? Oof, sorry, try again next life.

Hand-coded bindings are annoying to keep in sync, but finding out someone is trying to target an older libc, a newer libc, or a completely different OS doesn't sound like less maintenance effort to me. Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:23 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link]

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Ok, fine. That's actually exactly the approach that perl has been using for decades.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 15:04 UTC (Wed) by iainn (guest, #64312) [Link] (1 responses)

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Isn't that more of an argument to perform the autogeneration in a hermetic environment, like a container? Maintainers might also have Randomish environments.

(I agree the uploaded create should contain the autogenerated code; not needing libc headers, as discussed, is a good reason.)

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 16:38 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Even then, you might be missing something due to an `#if` check you're not up-to-date with. I think the kernel providing its ABI via CTF or the like is *far* better in this realm (at least for the Linux-specific bits of the question). Of course, for libc/POSIX/etc., the headers *are* the definition, so that's what one should use there.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:39 UTC (Thu) by josh (subscriber, #17465) [Link]

> When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't.

Because that makes cross-compilation harder and more fragile. LLVM-based environments typically have all the code generation capabilities for every platform available through the same binary and toolchain. Having to install a specific cross-libc, and having the capabilities of the libc crate depend on your installed libc headers, would make cross-compilation harder, less reproducible, and more fragile.

This doesn't mean we can't do code generation based on those headers, but we'd want to ship either the headers or the generated code with the libc crate.

Python cryptography, Rust, and Gentoo

Posted Feb 15, 2021 8:44 UTC (Mon) by glaubitz (subscriber, #96452) [Link]

> All of the mentioned platforms have not had hardware manufactured for years.

Actually, that's not true. There is regularly new hardware developed for the Amiga and the Linux kernel even see support for new Amiga hardware:

> https://lore.kernel.org/lkml/20190811043253.24938-1-max@e...

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 7:38 UTC (Thu) by lysse (guest, #3190) [Link] (3 responses)

Rust supports quite a few modern targets? That's great, but consider the number of targets supported by C: All of them.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 20:37 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (2 responses)

Sure. C is also 50 years old. Rust is what, 10? First, no language is going to show up with the platform support C has right out of the gate. Also, not all platforms are still relevant today. Those that are will gain Rust support if there's enough interest by users of those platforms to use the tools provided by Rust. If they're not, well, they won't get Rust software.

Python cryptography, Rust, and Gentoo

Posted Apr 16, 2021 21:35 UTC (Fri) by wtarreau (subscriber, #51152) [Link] (1 responses)

> First, no language is going to show up with the platform support C has right out of the gate

You're confusing language and compiler. A language is platform-agnostic, it even works with paper and pen.

The problem here is that this so-called language knows a single compiler whose developers are not interested in porting to what they consider irrelevant platforms. It's their right, it's their project. What is sad is that jerks are blindly following this without consideration for their own users. With python cornering itself behind the sole list of platforms supported by rust and betraying its users, we're certain never to ever see python 4.

Python cryptography, Rust, and Gentoo

Posted Apr 16, 2021 22:23 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

> What is sad is that jerks are blindly following this without consideration for their own users.

That seems to be a lot of projection to me. If my target audience is mainstream platforms or the project's goal is to implement something securely, what obligations do I have to those who happen to find it useful in their niche without considering to make sure it's within my project's goal or target audience? If someone comes up to me and says "hey, I got it working on the Nintendo Switch last year, your latest release breaks it", what am I to do? I'm not getting (or potentially bricking) a Switch to test it and I never claimed to support such a thing anyways. Patches that don't interfere with other things are fine, but if you're asking me to resurrect some pre-refactoring code just for your setup, sorry. Feel free to fork though.

I suspect this will be used for new drivers. Existing drivers will need a lot of "oomph" to warrant a rewrite. It seems to also have put more interest into the GCC frontend, so if/when that reaches the finish line, it'll be just as good for all the other Linux platforms.

> With python cornering itself behind the sole list of platforms supported by rust and betraying its users, we're certain never to ever see python 4.

Python has done no such thing. The maintainers of cryptography may arguably have done this (I make no claim to either argument on that front here), but they are not the maintainers of the CPython project. Any Python 4 was known to never be a thing because the maintainers recognized something of the trainwreck that the Python3 migration ended up being.