"fringe platforms which can't even run a Rust compiler" [LWN.net]

"fringe platforms which can't even run a Rust compiler"

Posted Feb 12, 2021 17:26 UTC (Fri) by moltonel (subscriber, #45207)
In reply to: "fringe platforms which can't even run a Rust compiler" by sthibaul
Parent article: Python cryptography, Rust, and Gentoo

Python isn't getting a rust dependency, it's "just" a (pretty important) python package that is. Python *is* dropping support for the s390 platform, as support had already been dropped left and right, including in the Linux kernel.

IMHO the "dream of free software" is intact: all the tools are available to resolve the incompatibilities and we all agree on that goal. We "just" need somebody motivated enough to do or sponsor the work. It's already happening to some extent, but it can't be done overnight and it can't delay much the vast majority of users who are up to date.

I suppose another lens to analyze the situation with is whether these incompatibilities are the cause or the symptom of a platform's death.

to post comments

"fringe platforms which can't even run a Rust compiler"

Posted Feb 12, 2021 22:14 UTC (Fri) by sthibaul (✭ supporter ✭, #54477) [Link] (23 responses)

> I suppose another lens to analyze the situation with is whether these incompatibilities are the cause or the symptom of a platform's death.

Yes, but making ports difficult can also kill in the egg possibly interesting new projects.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 13:54 UTC (Sat) by mathstuf (subscriber, #69389) [Link] (17 responses)

Sure, any little thing could "kill in the egg possibly interesting new project". Economic recession, war, etc. Less hyperbolicly, I don't know that LLVM, Rust, et al. are actively making ports more difficult other than raising the bar that needs to be met to qualify as a Supported Platform. If those platforms are not able to support more than C or C++, why do *they* get to hold the rest of the world hostage with such a low bar of offered features?

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 19:38 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (16 responses)

I'm not saying that platform shouldn't bother porting more than a C/C++ compiler.

There is room for a middle ground between "you have to do a lot of work to port Rust to your platform" and "you only need a C compiler and Posix environment to port Rust to your platform".

That said, the free software ecosystem has historically only roughly needed a C compiler to get perl, python, etc. which helped a lot various platform projects to get stuff working.

If the bar is raised too much, the possibility of interesting projects lowers.

If bash/gcc/gdb/etc. were not easy to port in early 90s, we would just not have Linux today.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:48 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (15 responses)

These days it's more like "you need LLVM for your platform". This will automatically give you a C/C++ compiler, Rust and soon Go. If anything, it's actually easier to port non-trivial userspace to new platforms today.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:50 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (14 responses)

No, LLVM won't immediately give you Rust, see other comments about the workload to port libc.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 21:51 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (13 responses)

Rust doesn't need libc.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 21:58 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (12 responses)

But cargo does, and building rust needs cargo?

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:12 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (11 responses)

It depends on what you WANT to do. If you're bringing up a small embedded platform then you don't need to run cargo on it. You just do cross-compilation, which only requires LLVM for the target.

If you want to bring up a full large-scale OS (think Fuchsia or Linux) then you'll need a libc for Python and other tools anyway. You'll need to port the libc crate to that OS as well, but this will be a pretty small task overall compared to the amount of work you have to do.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:20 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (10 responses)

> You'll need to port the libc crate to that OS as well, but this will be a pretty small task overall compared to the amount of work you have to do.

As I mentioned in other comments, porting the libc create is currently *not* a small task, you have to go over all system-provided libc headers to check for precise bits and bytes. This step deserves some automation, like perl/python/etc. have.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:29 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (9 responses)

Here's an example of libc for VxWorks (a non-Unix OS): https://github.com/rust-lang/libc/blob/master/src/vxworks...

It's 2000 lines and most of them can be trivially generated by cut&pasting headers and doing string replaces. You can write it in within a day.

Automating it would be nice, but different OSes can have wildly different header conventions.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 22:37 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (6 responses)

> most of them can be trivially generated by cut&pasting headers and doing string replaces.

cut&pasting+string replace from the C .h headers??

> You can write it in within a day.

For somebody that doesn't even know Rust from the start?

> Automating it would be nice, but different OSes can have wildly different header conventions.

I'm not saying to automate string replacements, but simply do like all other languages implementations I have seen do: interpret the C headers in C, i.e. bindgen.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 23:13 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> cut&pasting+string replace from the C .h headers??
Yup. Get your platform's headers, copy the definitions and then massage them until they are valid Rust.

If you want something minimalistic, you can even skip most of them (I'd estimate that around 500 lines are truly needed).

> For somebody that doesn't even know Rust from the start?
Add one more day for cargo-culting from an existing file. If you ever had to debug autohell, you're qualified enough to do it.

> I'm not saying to automate string replacements, but simply do like all other languages implementations I have seen do: interpret the C headers in C, i.e. bindgen.
That is possible, but you'd spend quite a bit of time debugging the generated code for your custom platform anyway. And given that these files are pretty much static once they are written, they don't need a lot of maintenance.

In reality, bringing up a non-Unix platform is such a huge undertaking that you'd likely be better off by writing the libc in _Rust_.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 23:20 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (4 responses)

> Get your platform's headers, copy the definitions and then massage them until they are valid Rust.

glibc, for instance, has ~500 headers, the definitions are split among arch-specific and non-arch-specific files. Just to give an instance for the st_dev member of the stat structure: it is actually defined in bits/stats.h, with type __dev_t. Type __dev_t is defined in bits/types.h from __DEV_T_TYPE. That type is defined in bits/typesizes.h from __UQUAD_TYPE. That type is defined in bits/types.h from __uint64_t.

> That is possible, but you'd spend quite a bit of time debugging the generated code for your custom platform anyway.

What is left to debug, when the C interpretation is *by definition* the correct answer?

> In reality, bringing up a non-Unix platform is such a huge undertaking that you'd likely be better off by writing the libc in _Rust_.

Why Rust?

Really, that's exactly the point: it looks like Rust people want to bring the whole world into Rust, and not bother with the rest of the world that isn't Rust.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 23:25 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> glibc, for instance, has ~500 headers, the definitions are split among arch-specific and non-arch-specific files.
If you have a libc from your device vendor, then just do "#include <socket.h>" and then pre-process the file. If you are making a completely new arch, then YOU have to write these 500 files.

> What is left to debug, when the C interpretation is *by definition* the correct answer?
For example, Rust has special definitions to navigate the socket headers (these are macros in C, not functions). C also doesn't specify the error returns.

> Why Rust?
Because it's safer than C and faster to write.

> Really, that's exactly the point: it looks like Rust people want to bring the whole world into Rust, and not bother with the rest of the world that isn't Rust.
Honestly, that would be a great outcome for the world.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 0:20 UTC (Sun) by sthibaul (✭ supporter ✭, #54477) [Link] (2 responses)

> just do "#include <socket.h>" and then pre-process the file.

Yes, but that still doesn't magically collapse all the typedefs. With just socket.h that produces 500 lines of C, that a C compiler and scripts shared by *all* ports would much better grok than doing seds by hand.

> For example, Rust has special definitions to navigate the socket headers (these are macros in C, not functions).

Ok, but that's only a very tiny part of the Posix interface.

> C also doesn't specify the error returns.

You mean the set of errors that a function may return? Yes, and a kernel never makes such a promise as the exact set of errors it might ever return in the coming decades.

> > Why Rust?
> Because it's safer than C and faster to write.

Let me rephrase: why Rust in particular? And not the myriad of other languages that have been invented by mankind?

> > Really, that's exactly the point: it looks like Rust people want to bring the whole world into Rust, and not bother with the rest of the world that isn't Rust.
> Honestly, that would be a great outcome for the world.

So that's exactly what I wrote in another comment: so much for the free software dream, if you're free to use any language, as long as it is Rust.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 16:05 UTC (Sun) by nix (subscriber, #2304) [Link]

> Yes, but that still doesn't magically collapse all the typedefs. With just socket.h that produces 500 lines of C, that a C compiler and scripts shared by *all* ports would much better grok than doing seds by hand.

Aha, thank you for giving me another use case for CTF! The upcoming CTF format v4 work will include an objdump option that emits a collection of C header files given a .ctf section. They won't be the same as the inputs, of course (no comments, for starters, and with the default linker options you'll get one big header for non-ambiguous types and then small headers #including the big one for all the ambiguously-defined ones), but it'll be better than nothing -- but until now I didn't have an answer for why this would ever be better than using the original .h files. Now I have one: transformations on the headers to help people doing work like this. (A transformation that collapses away all typedefs, leaving everything textually defined in terms of the base types, and leaving the typedefs around but with no actual users, should be trivial to implement. It's probably a bad idea to *use* those headers, but for human analysis for FFIs it's probably likely to be useful.)

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 0:28 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

> Ok, but that's only a very tiny part of the Posix interface.
You actually don't need much to bootstrap Rust. The now-removed CloudABI was about 500 lines of code for the libc, which is probably about the minimal amount.

> Let me rephrase: why Rust in particular? And not the myriad of other languages that have been invented by mankind?
For stuff like libc you need a language that has no runtime (no GC, no threads created behind your back, etc.), natively compiled, memory-safe and preferably with additional type-safety-enforced guarantees.

Not many languages fit this bill. Out of modern-ish ones only Rust and Swift fit the bill.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 20:25 UTC (Sun) by geert (subscriber, #98403) [Link] (1 responses)

IIUIC, Rust expects to call into snprintf() on VxWorks?

Last time I developed software for VxWorks (15y ago), its C library didn't have snprintf(), so we had to use our own implementation.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 21:57 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link]

> IIUIC, Rust expects to call into snprintf() on VxWorks?
Nope. Rust has its own string formatting support (in stdlib) that is independent of the OS.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:03 UTC (Sat) by rgmoore (✭ supporter ✭, #75) [Link] (4 responses)

Yes, but making ports difficult can also kill in the egg possibly interesting new projects.

The problem in this case is not so much the difficulty of making the port but the lack of resources to do the porting. Many of the platforms in question have been out of production for a decade or more and were in serious decline long before that. The only people supporting them are a handful of hobbyists who like maintaining obsolete hardware. We're far more likely to kill interesting new projects in the egg by making support for those platforms a requirement than by failing to support them.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 13, 2021 20:10 UTC (Sat) by sthibaul (✭ supporter ✭, #54477) [Link] (3 responses)

> The problem in this case is not so much the difficulty of making the port but the lack of resources to do the porting.

That's the original question of the article, yes. But the same difficulty will be faced by new platforms, the difference being that you won't see them not-appear.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 14, 2021 18:59 UTC (Sun) by laarmen (subscriber, #63948) [Link]

I don't know, this reasoning sound weird to me. If I make my language hard to port on ENIAC, it will most likely make it harder to port it on new platforms that behave like ENIAC, thus limiting the opportunities of building new platforms that differ from the currently supported platforms in the same way that ENIAC does.

If it's a new platform shouldn't we assume that it will behave more like current platforms than older ones? Otherwise, why do you need a new platform as the older one already exist? Of course you can say that the new platform is a variant of the older one but running much much faster, but that's only because it doesn't and its properties are custom built to support your argument. An empty set can have any property you want it to.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 22:45 UTC (Mon) by rgmoore (✭ supporter ✭, #75) [Link] (1 responses)

The difference is that building an interesting new hardware platform is inherently a major undertaking, and it's only going to be done by an organization with substantial resources. That includes the resources to port all the significant languages to work with their new platform. Being able to do that is more or less required for a new platform to be interesting. A platform that can't run the languages people care about isn't very interesting.

"fringe platforms which can't even run a Rust compiler"

Posted Feb 15, 2021 22:47 UTC (Mon) by sthibaul (✭ supporter ✭, #54477) [Link]

> building an interesting new hardware platform

I'm not talking only about hardware platform, but also OS.

> That includes the resources to port all the significant languages to work with their new platform.

Yes, but that doesn't mean that projects shouldn't make reasonable attempts to make it not too hard.