Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 11:02 UTC (Thu) by sthibaul (✭ supporter ✭, #54477)
In reply to: Python cryptography, Rust, and Gentoo by josh
Parent article: Python cryptography, Rust, and Gentoo

> > And Rust people don't seem to care about the situation since they have already ported to the only few platforms they care about...
>
> We care a great deal about portability

Then where is a tool that just picks up the libc headers to determine the sizes of structures etc. rather than having to hardcode everything in the rust repository? I did spend some time at some point to start writing them, copy/pasting from BSD files for a fair part, but everything has to be checked bit for bit to be sure that it's accurate, that is a very bug-prone process.

And... the pace at which Rust goes made me feel that probably what I wrote at the time is simply completely outdated since then and I'd better restart over from scratch next time I happen to find some time on this.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:32 UTC (Thu) by farnz (subscriber, #17727) [Link] (7 responses)

The tool that does that is called "bindgen". It's imperfect (necessarily so, because some of the descriptions in libc headers are in the form of human constraints on - e.g. - which #defines can be ORed together) but it does a decent first cut. The need for better is why the libc crate has hand-produced bindings instead, which have been human-read and the applicable extra constraints applied.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:19 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (6 responses)

> The tool that does that is called "bindgen".

Ok, thanks for the hint! At the time (novembre 2018) there was very little mention of this within rustc, and people had told me that they didn't think there was something like that.

> which #defines can be ORed together

Sure, at some point there is semantic that C doesn't provide. But that semantic is all the same for all Posix platforms...

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:50 UTC (Thu) by farnz (subscriber, #17727) [Link] (1 responses)

Unfortunately, it's not the same for all POSIX platforms - just about every platform has extensions to POSIX that aren't the same as other platforms :-(

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:55 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link]

> Unfortunately, it's not the same for all POSIX platforms - just about every platform has extensions to POSIX that aren't the same as other platforms :-(

Sure there are some extensions, but that's not much compared to the common Posix API.

Porting rust to a system that provides the Posix interface should just be a matter of running a script that looks for the ABI of the Posix interface (even if only to record it in the source tree). Like perl has been doing for decades.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:57 UTC (Thu) by josh (subscriber, #17465) [Link]

bindgen has been around for a while, but it's fiddly to build. I'm currently working on some things that may improve that.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 11:39 UTC (Thu) by freem (guest, #121851) [Link] (2 responses)

>> which #defines can be ORed together

> Sure, at some point there is semantic that C doesn't provide. But that semantic is all the same for all Posix platforms...

Is it C which does not, or is it the source code?
In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 17:12 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (1 responses)

> In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Bitfields have some downsides, the biggest one being the lack of a standard, portable memory layout. The placement of individual bitfields within a structure is implementation-defined and varies in practice according to the target architecture, especially with respect to byte order. As a result, it is generally considered best practice to avoid directly sharing structures with bitfields between programs which may not share the same code-generation settings—for example, between user-space and kernel-space, or anything related to the layout of bits in a hardware register or persistent storage. Even for local data within the same process there are some ergonomic issues, such as the fact that one cannot take the address of a bitfield, make use of atomic operations, or easily set, clear, or test multiple bitfields within the same structure in the same operation in the same way that one can use bitwise AND/OR operations with a bitmask. The compiler can paper over some of these issues but this relies more heavily on the optimizer to merge together a series of one-bit read-modify-write sequences.

Python cryptography, Rust, and Gentoo

Posted Feb 25, 2021 14:11 UTC (Thu) by myrrlyn (guest, #145084) [Link]

one of the reasons that i, personally, am looking forward to rust pushing out c is that i have a library that covers all of the points about bitfields you just laid out, and makes them trivial to correctly, conveniently, and performantly use in software that needs the space compaction

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:03 UTC (Thu) by josh (subscriber, #17465) [Link] (6 responses)

I would love to see much of the libc crate autogenerated; someone would need to write the necessary code to make that happen. It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need). That would also make the libc crate dependent on the version of headers you have installed. Today, the libc crate carefully avoids using features that are "too new" for the versions of libc that it supports. (Rust typically supports running with the version of glibc from the previous RHEL or Debian stable, for instance, rather than requiring the latest libc.)

So, autogenerating the libc crate would likely need to happen as part of constructing that crate (before publishing it), or alternatively the libc crate would need to ship with copies of appropriate versions of the libc headers for supported platforms. Either way, this would likely be welcome, but would require some substantial infrastructure to implement. If someone is interested in implementing that, I would suggest starting a conversation on the libc issue tracker about what the architecture and requirements for that would look like.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:11 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (5 responses)

> It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need).

I don't really understand that argument.

When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't. Making platform ports and libc bindings upgrade way more difficult just for this reason seems terribly weak to me.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:18 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (3 responses)

The problem is that it adds another axis to your "build environment matrix". In addition to target triple and compiler version, I now need to ask "what libc do you have installed?". And I need to document how to get that information for any given build environment. Oh, you're on macOS trying to cross-compile to Linux? Oof, sorry, try again next life.

Hand-coded bindings are annoying to keep in sync, but finding out someone is trying to target an older libc, a newer libc, or a completely different OS doesn't sound like less maintenance effort to me. Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:23 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link]

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Ok, fine. That's actually exactly the approach that perl has been using for decades.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 15:04 UTC (Wed) by iainn (guest, #64312) [Link] (1 responses)

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Isn't that more of an argument to perform the autogeneration in a hermetic environment, like a container? Maintainers might also have Randomish environments.

(I agree the uploaded create should contain the autogenerated code; not needing libc headers, as discussed, is a good reason.)

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 16:38 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Even then, you might be missing something due to an `#if` check you're not up-to-date with. I think the kernel providing its ABI via CTF or the like is *far* better in this realm (at least for the Linux-specific bits of the question). Of course, for libc/POSIX/etc., the headers *are* the definition, so that's what one should use there.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:39 UTC (Thu) by josh (subscriber, #17465) [Link]

> When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't.

Because that makes cross-compilation harder and more fragile. LLVM-based environments typically have all the code generation capabilities for every platform available through the same binary and toolchain. Having to install a specific cross-libc, and having the capabilities of the libc crate depend on your installed libc headers, would make cross-compilation harder, less reproducible, and more fragile.

This doesn't mean we can't do code generation based on those headers, but we'd want to ship either the headers or the generated code with the libc crate.