Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 0:27 UTC (Thu) by sthibaul (✭ supporter ✭, #54477)
Parent article: Python cryptography, Rust, and Gentoo

> fringe platforms which can't even run a Rust compiler.

Did he actually try to port Rust to a new platform?

That sentence is completely upside down. It is Rust that can't even
build&run on a Posix system.

It really is a beast. And I haven't seen anything in Rust to make it
easy. Basically you have to go over all your libc headers and describe
them structure by structure, bit by bit to rust. And Rust people don't
seem to care about the situation since they have already ported to the
only few platforms they care about...

to post comments

LLVM += new architecture

Posted Feb 11, 2021 1:56 UTC (Thu) by marcH (subscriber, #57642) [Link] (2 responses)

> Basically you have to go over all your libc headers and describe them structure by structure, bit by bit to rust.

"All your libc headers" are not all architecture-specific, are they? Can you elaborate?

> > fringe platforms which can't even run a Rust compiler.

I can't wait the next LWN article on the real issue: how much effort is this really. Plural starts at two so LLVM must have some solid architecture abstractions, no?

> > The LLVM developers have been somewhat leery of taking on new architectures, unless they can be convinced there will be long-term support for them, which is understandable, but makes the problem even worse.

This could/should start in an unofficial branch. Don't distributions routinely tweak toolchains already? This caused most C portability issues across Linux distributions I've met. I understand rebasing such a branch would be more work (how much more?) than merely "tweaking" a toolchain but at least the tooling and workflow should be there already.

You may consider Rust "exotic" but LLVM is of course not.

LLVM += new architecture

Posted Feb 11, 2021 10:58 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (1 responses)

> "All your libc headers" are not all architecture-specific, are they? Can you elaborate?

For a given OS, not all headers are arch-specific, but a large part of them is. Think about dirent structure, poll structure, fenv details, pthread structures, signal structures, stat structures, time structures, etc. ad nauseam.

And for a different OS, it's basically all the headers that need to be ported. Making basic free software libraries depend on rust basically means excluding from the free software game any non-Linux OS unless somebody takes up the daunting task of explaining the libc headers to Rust, while Rust could simply act like all other such languages (perl, python, haskell) do: just interpret the libc headers at configure time.

LLVM += new architecture

Posted Feb 26, 2021 13:48 UTC (Fri) by Gaelan (guest, #145108) [Link]

For what it's worth, there's bindgen, which automates the process of converting C headers to a Rust interface. It looks like the Rust people aren't using it for libc at the moment, but if you needed it to support a new architecture/libc, you could write a version of the libc crate that used bindgen, or just use bindgen's output as a starting point to add to the hand-written definitions in the libc crate.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 8:52 UTC (Thu) by josh (subscriber, #17465) [Link] (22 responses)

> And Rust people don't seem to care about the situation since they have already ported to the only few platforms they care about...

We care a great deal about portability, and Rust supports quite a few target platforms, including both production and hobby platforms.

All of the mentioned platforms have not had hardware manufactured for years.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 11:02 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (16 responses)

> > And Rust people don't seem to care about the situation since they have already ported to the only few platforms they care about...
>
> We care a great deal about portability

Then where is a tool that just picks up the libc headers to determine the sizes of structures etc. rather than having to hardcode everything in the rust repository? I did spend some time at some point to start writing them, copy/pasting from BSD files for a fair part, but everything has to be checked bit for bit to be sure that it's accurate, that is a very bug-prone process.

And... the pace at which Rust goes made me feel that probably what I wrote at the time is simply completely outdated since then and I'd better restart over from scratch next time I happen to find some time on this.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:28 UTC (Thu) by dbaker (guest, #89236) [Link]

Are you talking about bindgen?

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:32 UTC (Thu) by farnz (subscriber, #17727) [Link] (7 responses)

The tool that does that is called "bindgen". It's imperfect (necessarily so, because some of the descriptions in libc headers are in the form of human constraints on - e.g. - which #defines can be ORed together) but it does a decent first cut. The need for better is why the libc crate has hand-produced bindings instead, which have been human-read and the applicable extra constraints applied.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:19 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (6 responses)

> The tool that does that is called "bindgen".

Ok, thanks for the hint! At the time (novembre 2018) there was very little mention of this within rustc, and people had told me that they didn't think there was something like that.

> which #defines can be ORed together

Sure, at some point there is semantic that C doesn't provide. But that semantic is all the same for all Posix platforms...

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:50 UTC (Thu) by farnz (subscriber, #17727) [Link] (1 responses)

Unfortunately, it's not the same for all POSIX platforms - just about every platform has extensions to POSIX that aren't the same as other platforms :-(

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:55 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link]

> Unfortunately, it's not the same for all POSIX platforms - just about every platform has extensions to POSIX that aren't the same as other platforms :-(

Sure there are some extensions, but that's not much compared to the common Posix API.

Porting rust to a system that provides the Posix interface should just be a matter of running a script that looks for the ABI of the Posix interface (even if only to record it in the source tree). Like perl has been doing for decades.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:57 UTC (Thu) by josh (subscriber, #17465) [Link]

bindgen has been around for a while, but it's fiddly to build. I'm currently working on some things that may improve that.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 11:39 UTC (Thu) by freem (guest, #121851) [Link] (2 responses)

>> which #defines can be ORed together

> Sure, at some point there is semantic that C doesn't provide. But that semantic is all the same for all Posix platforms...

Is it C which does not, or is it the source code?
In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 17:12 UTC (Thu) by nybble41 (subscriber, #55106) [Link] (1 responses)

> In many places, I see long lists of #defines which are only powers of 2. I never understood why they don't use bitfields which, if I am not wrong, are in C since at least C89?

Bitfields have some downsides, the biggest one being the lack of a standard, portable memory layout. The placement of individual bitfields within a structure is implementation-defined and varies in practice according to the target architecture, especially with respect to byte order. As a result, it is generally considered best practice to avoid directly sharing structures with bitfields between programs which may not share the same code-generation settings—for example, between user-space and kernel-space, or anything related to the layout of bits in a hardware register or persistent storage. Even for local data within the same process there are some ergonomic issues, such as the fact that one cannot take the address of a bitfield, make use of atomic operations, or easily set, clear, or test multiple bitfields within the same structure in the same operation in the same way that one can use bitwise AND/OR operations with a bitmask. The compiler can paper over some of these issues but this relies more heavily on the optimizer to merge together a series of one-bit read-modify-write sequences.

Python cryptography, Rust, and Gentoo

Posted Feb 25, 2021 14:11 UTC (Thu) by myrrlyn (guest, #145084) [Link]

one of the reasons that i, personally, am looking forward to rust pushing out c is that i have a library that covers all of the points about bitfields you just laid out, and makes them trivial to correctly, conveniently, and performantly use in software that needs the space compaction

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:03 UTC (Thu) by josh (subscriber, #17465) [Link] (6 responses)

I would love to see much of the libc crate autogenerated; someone would need to write the necessary code to make that happen. It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need). That would also make the libc crate dependent on the version of headers you have installed. Today, the libc crate carefully avoids using features that are "too new" for the versions of libc that it supports. (Rust typically supports running with the version of glibc from the previous RHEL or Debian stable, for instance, rather than requiring the latest libc.)

So, autogenerating the libc crate would likely need to happen as part of constructing that crate (before publishing it), or alternatively the libc crate would need to ship with copies of appropriate versions of the libc headers for supported platforms. Either way, this would likely be welcome, but would require some substantial infrastructure to implement. If someone is interested in implementing that, I would suggest starting a conversation on the libc issue tracker about what the architecture and requirements for that would look like.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:11 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link] (5 responses)

> It shouldn't just autogenerate at build time, because that would require an installed version of the libc headers for a target in order to cross-compile for that target (which you don't currently need).

I don't really understand that argument.

When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't. Making platform ports and libc bindings upgrade way more difficult just for this reason seems terribly weak to me.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:18 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (3 responses)

The problem is that it adds another axis to your "build environment matrix". In addition to target triple and compiler version, I now need to ask "what libc do you have installed?". And I need to document how to get that information for any given build environment. Oh, you're on macOS trying to cross-compile to Linux? Oof, sorry, try again next life.

Hand-coded bindings are annoying to keep in sync, but finding out someone is trying to target an older libc, a newer libc, or a completely different OS doesn't sound like less maintenance effort to me. Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:23 UTC (Thu) by sthibaul (✭ supporter ✭, #54477) [Link]

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Ok, fine. That's actually exactly the approach that perl has been using for decades.

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 15:04 UTC (Wed) by iainn (guest, #64312) [Link] (1 responses)

> Autogeneration is fine, but I'd like *that* generated code committed because it's just too damn important to leave up to the wild west of Random Developer Machine.

Isn't that more of an argument to perform the autogeneration in a hermetic environment, like a container? Maintainers might also have Randomish environments.

(I agree the uploaded create should contain the autogenerated code; not needing libc headers, as discussed, is a good reason.)

Python cryptography, Rust, and Gentoo

Posted Feb 17, 2021 16:38 UTC (Wed) by mathstuf (subscriber, #69389) [Link]

Even then, you might be missing something due to an `#if` check you're not up-to-date with. I think the kernel providing its ABI via CTF or the like is *far* better in this realm (at least for the Linux-specific bits of the question). Of course, for libc/POSIX/etc., the headers *are* the definition, so that's what one should use there.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:39 UTC (Thu) by josh (subscriber, #17465) [Link]

> When you cross-build something for another target, you'd just install the libc headers for that other target, yes! I don't see why you wouldn't.

Because that makes cross-compilation harder and more fragile. LLVM-based environments typically have all the code generation capabilities for every platform available through the same binary and toolchain. Having to install a specific cross-libc, and having the capabilities of the libc crate depend on your installed libc headers, would make cross-compilation harder, less reproducible, and more fragile.

This doesn't mean we can't do code generation based on those headers, but we'd want to ship either the headers or the generated code with the libc crate.

Python cryptography, Rust, and Gentoo

Posted Feb 15, 2021 8:44 UTC (Mon) by glaubitz (subscriber, #96452) [Link]

> All of the mentioned platforms have not had hardware manufactured for years.

Actually, that's not true. There is regularly new hardware developed for the Amiga and the Linux kernel even see support for new Amiga hardware:

> https://lore.kernel.org/lkml/20190811043253.24938-1-max@e...

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 7:38 UTC (Thu) by lysse (guest, #3190) [Link] (3 responses)

Rust supports quite a few modern targets? That's great, but consider the number of targets supported by C: All of them.

Python cryptography, Rust, and Gentoo

Posted Feb 18, 2021 20:37 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (2 responses)

Sure. C is also 50 years old. Rust is what, 10? First, no language is going to show up with the platform support C has right out of the gate. Also, not all platforms are still relevant today. Those that are will gain Rust support if there's enough interest by users of those platforms to use the tools provided by Rust. If they're not, well, they won't get Rust software.

Python cryptography, Rust, and Gentoo

Posted Apr 16, 2021 21:35 UTC (Fri) by wtarreau (subscriber, #51152) [Link] (1 responses)

> First, no language is going to show up with the platform support C has right out of the gate

You're confusing language and compiler. A language is platform-agnostic, it even works with paper and pen.

The problem here is that this so-called language knows a single compiler whose developers are not interested in porting to what they consider irrelevant platforms. It's their right, it's their project. What is sad is that jerks are blindly following this without consideration for their own users. With python cornering itself behind the sole list of platforms supported by rust and betraying its users, we're certain never to ever see python 4.

Python cryptography, Rust, and Gentoo

Posted Apr 16, 2021 22:23 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

> What is sad is that jerks are blindly following this without consideration for their own users.

That seems to be a lot of projection to me. If my target audience is mainstream platforms or the project's goal is to implement something securely, what obligations do I have to those who happen to find it useful in their niche without considering to make sure it's within my project's goal or target audience? If someone comes up to me and says "hey, I got it working on the Nintendo Switch last year, your latest release breaks it", what am I to do? I'm not getting (or potentially bricking) a Switch to test it and I never claimed to support such a thing anyways. Patches that don't interfere with other things are fine, but if you're asking me to resurrect some pre-refactoring code just for your setup, sorry. Feel free to fork though.

I suspect this will be used for new drivers. Existing drivers will need a lot of "oomph" to warrant a rewrite. It seems to also have put more interest into the GCC frontend, so if/when that reaches the finish line, it'll be just as good for all the other Linux platforms.

> With python cornering itself behind the sole list of platforms supported by rust and betraying its users, we're certain never to ever see python 4.

Python has done no such thing. The maintainers of cryptography may arguably have done this (I make no claim to either argument on that front here), but they are not the maintainers of the CPython project. Any Python 4 was known to never be a thing because the maintainers recognized something of the trainwreck that the Python3 migration ended up being.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 11, 2021 16:20 UTC (Thu) by clugstj (subscriber, #4020) [Link] (33 responses)

That wording struck me as odd also. It's not the platforms that can't run Rust, it's that Rust doesn't support those platforms.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 11, 2021 22:24 UTC (Thu) by moltonel (subscriber, #45207) [Link] (32 responses)

Round peg, square hole, semantically one is not more responsible than the other. We tend to feel that the other team should do the job, so do you identify more with the platform user inconvenienced by the new dependency on a language that seems unnecessary, or with the developer inconvenienced by the obsolete platform he'll never use ? Is the incompatibility a bigger issue for the platform or for the language ?

We all have an instinctive answer to those questions but it doesn't really matter, we all agree that the incompatibility is a problem, and it can be approached from either side.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 12, 2021 8:24 UTC (Fri) by sthibaul (✭ supporter ✭, #54477) [Link] (31 responses)

> Is the incompatibility a bigger issue for the platform or for the language ?

When the dependency is put on librsvg, which is a dependency for gtk, yes it does hurt strongly.
If the dependency is added to python itself, which is a dependency for so much free software, that could as well just kill the platform. So much for the dream of free software.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 12, 2021 17:14 UTC (Fri) by nix (subscriber, #2304) [Link] (5 responses)

The dream is still alive: the software is free, so you are free to port LLVM and Rust to any platform of your choice (and maintain it yourself forever, probably, because I kinda doubt the LLVM maintainers want to drag around m68k support in the upstream tree).

The free software dream was never about *having* stuff. It was always about *being able to do* stuff, and that is still there.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 12, 2021 23:22 UTC (Fri) by Wol (subscriber, #4433) [Link] (2 responses)

Actually, following the llvm mailing list, I think m68k support is being welcomed into the tree.

Granted, it's on the basis "you want it in, you need to make sure it's maintained", but the llvm attitude to architectures seems to be similar to the linux attitude to device drivers - better in than out!

Cheers,
Wol

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 22:25 UTC (Sun) by rodgerd (guest, #58896) [Link] (1 responses)

The problem has never been that these things can't be done: it's that the people making noise about them either want someone else to do it, for free; or a concern-trolling because they're looking for an excuse to block a tool that they don't like, while failing to offer a meaningful alternative.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 8:51 UTC (Mon) by glaubitz (subscriber, #96452) [Link]

> The problem has never been that these things can't be done: it's that the people making noise about them either want someone else to do it, for free; or a concern-trolling because they're looking for an excuse to block a tool that they don't like, while failing to offer a meaningful alternative.

That's not true. I'm one of these loud voices and I'm actually also one of the people who did lots of contributions to Rust to make it more portable:

> https://lwn.net/Articles/771355/

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 8:49 UTC (Mon) by glaubitz (subscriber, #96452) [Link] (1 responses)

> The dream is still alive: the software is free, so you are free to port LLVM and Rust to any platform of your choice

What would the Rust developers say if the LLVM project hypothetically changed its code in a way that it could no longer be used with Rust?

Would your answer also be "Go maintain your own LLVM fork!"?

Or if the kernel developers decided to drop support for anything but large IBM mainframes and POWER servers?

Would you also say "No problem, I'll maintain my own kernel fork!"?

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 11:30 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> What would the Rust developers say if the LLVM project hypothetically changed its code in a way that it could no longer be used with Rust?
What if GCC tomorrow decides to drop all languages except Ada?

> Would your answer also be "Go maintain your own LLVM fork!"?
That's actually what Rust had been doing for a while. They used to maintain a private fork of LLVM with Rust-specific patches. So yes, "go and maintain your fork".

> Or if the kernel developers decided to drop support for anything but large IBM mainframes and POWER servers?
> Would you also say "No problem, I'll maintain my own kernel fork!"?
Yup.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 12, 2021 17:26 UTC (Fri) by moltonel (subscriber, #45207) [Link] (24 responses)

Python isn't getting a rust dependency, it's "just" a (pretty important) python package that is. Python *is* dropping support for the s390 platform, as support had already been dropped left and right, including in the Linux kernel.

IMHO the "dream of free software" is intact: all the tools are available to resolve the incompatibilities and we all agree on that goal. We "just" need somebody motivated enough to do or sponsor the work. It's already happening to some extent, but it can't be done overnight and it can't delay much the vast majority of users who are up to date.

I suppose another lens to analyze the situation with is whether these incompatibilities are the cause or the symptom of a platform's death.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 12, 2021 22:14 UTC (Fri) by sthibaul (✭ supporter ✭, #54477) [Link] (23 responses)

> I suppose another lens to analyze the situation with is whether these incompatibilities are the cause or the symptom of a platform's death.

Yes, but making ports difficult can also kill in the egg possibly interesting new projects.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 13:54 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (17 responses)

Sure, any little thing could "kill in the egg possibly interesting new project". Economic recession, war, etc. Less hyperbolicly, I don't know that LLVM, Rust, et al. are actively making ports more difficult other than raising the bar that needs to be met to qualify as a Supported Platform. If those platforms are not able to support more than C or C++, why do *they* get to hold the rest of the world hostage with such a low bar of offered features?

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 19:38 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (16 responses)

I'm not saying that platform shouldn't bother porting more than a C/C++ compiler.

There is room for a middle ground between "you have to do a lot of work to port Rust to your platform" and "you only need a C compiler and Posix environment to port Rust to your platform".

That said, the free software ecosystem has historically only roughly needed a C compiler to get perl, python, etc. which helped a lot various platform projects to get stuff working.

If the bar is raised too much, the possibility of interesting projects lowers.

If bash/gcc/gdb/etc. were not easy to port in early 90s, we would just not have Linux today.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:48 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (15 responses)

These days it's more like "you need LLVM for your platform". This will automatically give you a C/C++ compiler, Rust and soon Go. If anything, it's actually easier to port non-trivial userspace to new platforms today.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:50 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (14 responses)

No, LLVM won't immediately give you Rust, see other comments about the workload to port libc.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 21:51 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (13 responses)

Rust doesn't need libc.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 21:58 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (12 responses)

But cargo does, and building rust needs cargo?

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:12 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

It depends on what you WANT to do. If you're bringing up a small embedded platform then you don't need to run cargo on it. You just do cross-compilation, which only requires LLVM for the target.

If you want to bring up a full large-scale OS (think Fuchsia or Linux) then you'll need a libc for Python and other tools anyway. You'll need to port the libc crate to that OS as well, but this will be a pretty small task overall compared to the amount of work you have to do.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:20 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (10 responses)

> You'll need to port the libc crate to that OS as well, but this will be a pretty small task overall compared to the amount of work you have to do.

As I mentioned in other comments, porting the libc create is currently *not* a small task, you have to go over all system-provided libc headers to check for precise bits and bytes. This step deserves some automation, like perl/python/etc. have.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:29 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

Here's an example of libc for VxWorks (a non-Unix OS): https://github.com/rust-lang/libc/blob/master/src/vxworks...

It's 2000 lines and most of them can be trivially generated by cut&pasting headers and doing string replaces. You can write it in within a day.

Automating it would be nice, but different OSes can have wildly different header conventions.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:37 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (6 responses)

> most of them can be trivially generated by cut&pasting headers and doing string replaces.

cut&pasting+string replace from the C .h headers??

> You can write it in within a day.

For somebody that doesn't even know Rust from the start?

> Automating it would be nice, but different OSes can have wildly different header conventions.

I'm not saying to automate string replacements, but simply do like all other languages implementations I have seen do: interpret the C headers in C, i.e. bindgen.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 23:13 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> cut&pasting+string replace from the C .h headers??
Yup. Get your platform's headers, copy the definitions and then massage them until they are valid Rust.

If you want something minimalistic, you can even skip most of them (I'd estimate that around 500 lines are truly needed).

> For somebody that doesn't even know Rust from the start?
Add one more day for cargo-culting from an existing file. If you ever had to debug autohell, you're qualified enough to do it.

> I'm not saying to automate string replacements, but simply do like all other languages implementations I have seen do: interpret the C headers in C, i.e. bindgen.
That is possible, but you'd spend quite a bit of time debugging the generated code for your custom platform anyway. And given that these files are pretty much static once they are written, they don't need a lot of maintenance.

In reality, bringing up a non-Unix platform is such a huge undertaking that you'd likely be better off by writing the libc in _Rust_.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 23:20 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (4 responses)

> Get your platform's headers, copy the definitions and then massage them until they are valid Rust.

glibc, for instance, has ~500 headers, the definitions are split among arch-specific and non-arch-specific files. Just to give an instance for the st_dev member of the stat structure: it is actually defined in bits/stats.h, with type __dev_t. Type __dev_t is defined in bits/types.h from __DEV_T_TYPE. That type is defined in bits/typesizes.h from __UQUAD_TYPE. That type is defined in bits/types.h from __uint64_t.

> That is possible, but you'd spend quite a bit of time debugging the generated code for your custom platform anyway.

What is left to debug, when the C interpretation is *by definition* the correct answer?

> In reality, bringing up a non-Unix platform is such a huge undertaking that you'd likely be better off by writing the libc in _Rust_.

Why Rust?

Really, that's exactly the point: it looks like Rust people want to bring the whole world into Rust, and not bother with the rest of the world that isn't Rust.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 23:25 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> glibc, for instance, has ~500 headers, the definitions are split among arch-specific and non-arch-specific files.
If you have a libc from your device vendor, then just do "#include <socket.h>" and then pre-process the file. If you are making a completely new arch, then YOU have to write these 500 files.

> What is left to debug, when the C interpretation is *by definition* the correct answer?
For example, Rust has special definitions to navigate the socket headers (these are macros in C, not functions). C also doesn't specify the error returns.

> Why Rust?
Because it's safer than C and faster to write.

> Really, that's exactly the point: it looks like Rust people want to bring the whole world into Rust, and not bother with the rest of the world that isn't Rust.
Honestly, that would be a great outcome for the world.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 0:20 UTC (Sun) by sthibaul (✭ supporter ✭, #54477) [Link] (2 responses)

> just do "#include <socket.h>" and then pre-process the file.

Yes, but that still doesn't magically collapse all the typedefs. With just socket.h that produces 500 lines of C, that a C compiler and scripts shared by *all* ports would much better grok than doing seds by hand.

> For example, Rust has special definitions to navigate the socket headers (these are macros in C, not functions).

Ok, but that's only a very tiny part of the Posix interface.

> C also doesn't specify the error returns.

You mean the set of errors that a function may return? Yes, and a kernel never makes such a promise as the exact set of errors it might ever return in the coming decades.

> > Why Rust?
> Because it's safer than C and faster to write.

Let me rephrase: why Rust in particular? And not the myriad of other languages that have been invented by mankind?

> > Really, that's exactly the point: it looks like Rust people want to bring the whole world into Rust, and not bother with the rest of the world that isn't Rust.
> Honestly, that would be a great outcome for the world.

So that's exactly what I wrote in another comment: so much for the free software dream, if you're free to use any language, as long as it is Rust.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 16:05 UTC (Sun) by nix (subscriber, #2304) [Link]

> Yes, but that still doesn't magically collapse all the typedefs. With just socket.h that produces 500 lines of C, that a C compiler and scripts shared by *all* ports would much better grok than doing seds by hand.

Aha, thank you for giving me another use case for CTF! The upcoming CTF format v4 work will include an objdump option that emits a collection of C header files given a .ctf section. They won't be the same as the inputs, of course (no comments, for starters, and with the default linker options you'll get one big header for non-ambiguous types and then small headers #including the big one for all the ambiguously-defined ones), but it'll be better than nothing -- but until now I didn't have an answer for why this would ever be better than using the original .h files. Now I have one: transformations on the headers to help people doing work like this. (A transformation that collapses away all typedefs, leaving everything textually defined in terms of the base types, and leaving the typedefs around but with no actual users, should be trivial to implement. It's probably a bad idea to *use* those headers, but for human analysis for FFIs it's probably likely to be useful.)

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 0:28 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> Ok, but that's only a very tiny part of the Posix interface.
You actually don't need much to bootstrap Rust. The now-removed CloudABI was about 500 lines of code for the libc, which is probably about the minimal amount.

> Let me rephrase: why Rust in particular? And not the myriad of other languages that have been invented by mankind?
For stuff like libc you need a language that has no runtime (no GC, no threads created behind your back, etc.), natively compiled, memory-safe and preferably with additional type-safety-enforced guarantees.

Not many languages fit this bill. Out of modern-ish ones only Rust and Swift fit the bill.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 20:25 UTC (Sun) by geert (subscriber, #98403) [Link] (1 responses)

IIUIC, Rust expects to call into snprintf() on VxWorks?

Last time I developed software for VxWorks (15y ago), its C library didn't have snprintf(), so we had to use our own implementation.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 21:57 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

> IIUIC, Rust expects to call into snprintf() on VxWorks?
Nope. Rust has its own string formatting support (in stdlib) that is independent of the OS.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:03 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link] (4 responses)

Yes, but making ports difficult can also kill in the egg possibly interesting new projects.

The problem in this case is not so much the difficulty of making the port but the lack of resources to do the porting. Many of the platforms in question have been out of production for a decade or more and were in serious decline long before that. The only people supporting them are a handful of hobbyists who like maintaining obsolete hardware. We're far more likely to kill interesting new projects in the egg by making support for those platforms a requirement than by failing to support them.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:10 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (3 responses)

> The problem in this case is not so much the difficulty of making the port but the lack of resources to do the porting.

That's the original question of the article, yes. But the same difficulty will be faced by new platforms, the difference being that you won't see them not-appear.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 18:59 UTC (Sun) by laarmen (subscriber, #63948) [Link]

I don't know, this reasoning sound weird to me. If I make my language hard to port on ENIAC, it will most likely make it harder to port it on new platforms that behave like ENIAC, thus limiting the opportunities of building new platforms that differ from the currently supported platforms in the same way that ENIAC does.

If it's a new platform shouldn't we assume that it will behave more like current platforms than older ones? Otherwise, why do you need a new platform as the older one already exist? Of course you can say that the new platform is a variant of the older one but running much much faster, but that's only because it doesn't and its properties are custom built to support your argument. An empty set can have any property you want it to.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 22:45 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

The difference is that building an interesting new hardware platform is inherently a major undertaking, and it's only going to be done by an organization with substantial resources. That includes the resources to port all the significant languages to work with their new platform. Being able to do that is more or less required for a new platform to be interesting. A platform that can't run the languages people care about isn't very interesting.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 22:47 UTC (Mon) by sthibaul (✭ supporter ✭, #54477) [Link]

> building an interesting new hardware platform

I'm not talking only about hardware platform, but also OS.

> That includes the resources to port all the significant languages to work with their new platform.

Yes, but that doesn't mean that projects shouldn't make reasonable attempts to make it not too hard.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 21:44 UTC (Fri) by rodgerd (guest, #58896) [Link]

I guess everyone should have less security so a few nerds can keep running processors that haven't been built this century.