Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:03 UTC (Thu) by josh (subscriber, #17465)
In reply to: Python cryptography, Rust, and Gentoo by sthibaul
Parent article: Python cryptography, Rust, and Gentoo

I would love to see much of the libc crate autogenerated; someone would need to write the necessary code to make that happen. It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need). That would also make the libc crate dependent on the version of headers you have installed. Today, the libc crate carefully avoids using features that are "too new" for the versions of libc that it supports. (Rust typically supports running with the version of glibc from the previous RHEL or Debian stable, for instance, rather than requiring the latest libc.)

So, autogenerating the libc crate would likely need to happen as part of constructing that crate (before publishing it), or alternatively the libc crate would need to ship with copies of appropriate versions of the libc headers for supported platforms. Either way, this would likely be welcome, but would require some substantial infrastructure to implement. If someone is interested in implementing that, I would suggest starting a conversation on the libc issue tracker about what the architecture and requirements for that would look like.

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:11 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (5 responses)

> It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need).

I don't really understand that argument.

When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't. Making platform ports and libc bindings upgrade way more difficult just for this reason seems terribly weak to me.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:18 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (3 responses)

The problem is that it adds another axis to your "build environment matrix". In addition to target triple and compiler version, I now need to ask "what libc do you have installed?". And I need to document how to get that information for any given build environment. Oh, you're on macOS trying to cross-compile to Linux? Oof, sorry, try again next life.

Hand-coded bindings are annoying to keep in sync, but finding out someone is trying to target an older libc, a newer libc, or a completely different OS doesn't sound like less maintenance effort to me. Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:23 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link]

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Ok, fine. That's actually exactly the approach that perl has been using for decades.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 15:04 UTC (Wed) by iainn (guest, #64312) [Link] (1 responses)

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Isn't that more of an argument to perform the autogeneration in a hermetic environment, like a container? Maintainers might also have Randomish environments.

(I agree the uploaded create should contain the autogenerated code; not needing libc headers, as discussed, is a good reason.)

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 16:38 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Even then, you might be missing something due to an `#if` check you're not up-to-date with. I think the kernel providing its ABI via CTF or the like is *far* better in this realm (at least for the Linux-specific bits of the question). Of course, for libc/POSIX/etc., the headers *are* the definition, so that's what one should use there.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:39 UTC (Thu) by josh (subscriber, #17465) [Link]

> When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't.

Because that makes cross-compilation harder and more fragile. LLVM-based environments typically have all the code generation capabilities for every platform available through the same binary and toolchain. Having to install a specific cross-libc, and having the capabilities of the libc crate depend on your installed libc headers, would make cross-compilation harder, less reproducible, and more fragile.

This doesn't mean we can't do code generation based on those headers, but we'd want to ship either the headers or the generated code with the libc crate.