Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 8:06 UTC (Thu) by glaubitz (subscriber, #96452)
Parent article: Python cryptography, Rust, and Gentoo

Copying my comment from HN for more visibility:

The Rust portability issue is the reason I have been pushing forward with the new M68k backend for LLVM which is due to be merged shortly:

> https://github.com/M680x0/M680x0-mono-repo

Patches for the M68k backend are discussed on reviews.llvm.org.

I have also started a Rust port for m68k already that depends on the LLVM backend work above:

> https://github.com/glaubitz/rust/tree/m68k-linux

For other architectures, the goal is to get gccrs moving forward and merged upstream:

> https://github.com/philberty/gccrs/

For anyone interested in helping, there are Bountysource campaigns I started to support both efforts:

> https://www.bountysource.com/issues/90829856-llvm-complet...

> https://www.bountysource.com/issues/86138921-rfe-add-a-fr...

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 9:09 UTC (Thu) by josh (subscriber, #17465) [Link] (20 responses)

> For other architectures, the goal is to get gccrs moving forward and merged upstream:

I would much rather see rustc_codegen_gcc upstreamed: https://github.com/antoyo/rustc_codegen_gcc

A GCC backend for rustc would provide all the code generation of GCC without duplicating the Rust frontend.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 11:06 UTC (Thu) by jrtc27 (subscriber, #107748) [Link] (19 responses)

Both have value. A separate backend is the simpler solution, but having an entirely separate frontend means you have an independent implementation which is good for the language, just like we see with GCC and LLVM.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 12:29 UTC (Thu) by moltonel (subscriber, #45207) [Link] (17 responses)

It's not as clear-cut as you think.

Looking at the GCC/LLVM/MSVC example, implementations are able to compete on C/C++ language features, build/runtime performance, supported targets, analyzers, etc. The story is similar with Javascript engines, for example. That's both a frontend and backend win.

But looking at something like Python, none of the alternate implementations compete on language features, just aspects like performance (I put stackless in the performance category rather than feature) or target. Language-wise, they all lag behind CPython. That's a backend win but a frontend loss.

A rust gcc frontend will always lag behind rustc, like mrustc currently does. Keeping compatibility with gccrs will be a (hopefully small) burden for rust developers. Is the frontend/backend tradeoff that Python made a good bet for Rust too ? Turns out that rustc has an advantage over cpython: it already supports multiple backends (llvm and cranelift). It will hopefully support a gcc backend "soon". So rust could get its backend win while avoiding a frontend loss.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 12:48 UTC (Thu) by glaubitz (subscriber, #96452) [Link] (16 responses)

> A rust gcc frontend will always lag behind rustc, like mrustc currently does.

Rust will probably (and hopefully) slow down with their process of changing the language. At some point, they will want to stabilize and then alternative implementations are able to catch up.

It's Rust's very own interest that alternative, more portable implementations exist. This will help Rust's adoption in the community because the portability is no longer a concern.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:21 UTC (Thu) by moltonel (subscriber, #45207) [Link] (15 responses)

Python is 30 years old and just added pattern matching. C++17 and C++20 feel like completely new languages. The alternative implementations are constantly lagging. If you mean "stability" as "no new language features" then I don't know of any general-purpose language that reached that stage without being considered dead. OTOH, Rust prides itself on "stability without stagnation" meaning that new features get added without breaking backward compatibility, and they do an exemplary job of it.

Rust clearly needs to reach niche and new platforms faster, to gain wider acceptance and not cause anguish when the next library starts using it. If it can do that without inflicting frontend differences on the user, it'll have done better at portability than C and C++.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:35 UTC (Thu) by glaubitz (subscriber, #96452) [Link] (14 responses)

> Python is 30 years old and just added pattern matching. C++17 and C++20 feel like completely new languages. The alternative implementations are constantly lagging. If you mean "stability" as "no new language features" then I don't know of any general-purpose language that reached that stage without being considered dead.

The difference is that these other two languages have a specification and alternative implementations that can keep up with each other, aren't they? Also, especially C/C++ are evolving at a much slower pace than Rust where almost every new upstream release contains new language features.

I mean, the whole reason this article exists is because Rust is special in this regard and causes problems for downstream projects. You can't really deny that, can you?

> Rust clearly needs to reach niche and new platforms faster, to gain wider acceptance and not cause anguish when the next library starts using it. If it can do that without inflicting frontend differences on the user, it'll have done better at portability than C and C++.

Sure, but the problem is that this hasn't happened yet and one of the main reasons is the fact that the language is moving so fast that alternative implementations have trouble keeping up.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 18:59 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

I don't know how well the various C++ implementations are keeping up with C++20 (at least). Specifically with Concepts, Coroutines, and Modules implementation progress.

> Also, especially C/C++ are evolving at a much slower pace than Rust where almost every new upstream release contains new language features.

I think they have a longer release cadence, but the difference between Rust c. 2017 and Rust c. 2020 is, IMO, *far* smaller than the difference between C++17 and C++20 (IMO). Rust added async syntax and language features, tweaked the module import spelling (in a backwards compatible way via the editions mechanism), added const compile-time evaluation support, and other things. On the other hand, C++ got consteval, a completely new module system (that needs build systems to chase actual use of it in practice), concepts, coroutines, and many many more things.

> Sure, but the problem is that this hasn't happened yet and one of the main reasons is the fact that the language is moving so fast that alternative implementations have trouble keeping up.

I think the main thing is that Rust is a hard language to implement. mrustc skips lifetime checking, but handles the rest of the language (as of 1.29 or so). The main developer of it went onto other tasks (IIRC, academic pursuits). A community could certainly get behind it to push it further.

I don't know that it's so much "have trouble keeping up" as getting over the initial hurdle to getting a lifetime-tracking system that is independently implemented. Once that happens, I don't think much in the language would be *too* difficult to keep up with (most of the changes I'm seeing in release notes are stdlib, perf changes, or incremental language changes that are likely way easier than C++ incremental updates (as long as we're comparing)).

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 19:09 UTC (Thu) by roc (subscriber, #30627) [Link] (4 responses)

I would say Rust is adding features at a lower rate than C++ is, currently. C++ just added modules, concepts and coroutines which are each *huge* language features.

I agree with you that multiple popular Rust frontends, in conjunction with a Rust specification, would be a good thing at some point, to clarify implementation bugs vs language features. However it's a huge investment to keep all three C++ frontends going and Rust isn't yet at the point where that could be sustained.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 12:34 UTC (Fri) by khim (subscriber, #9252) [Link] (3 responses)

It's just so happened that C++20 got huge number of new features (thus, I think it would be the same story as with C++11/C++14: many users would just skip C++20 completely and go from C++17 to C++23 where rough edges of all these new things would be somewhat softened).

But these things were developed for more than ten years! C++ concepts, in particular, were conceived before Rust even got it's name!

It's just so happened that C++20 is huge release… time will show if Rust would ever have such a huge language-bending moment like C++11 and C++20 were.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 14:02 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (2 responses)

And it's not like C++23 or C++26 are looking any smaller either. Networking, modularized stdlib, pattern matching, templated `this` (to avoid rewriting the same body for a const, non-const, ref, and rvalue-ref variants of a method), metaprogramming, reflection, contracts, and others I can't even remember right now are all on the docket for the next two releases.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 17:56 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

The things which you have listed are not larger than what Rust gets in three years, though.

Small additions like these are added rust basically every release.

But concepts and especially modules… they change things. Not just allow you to write less code in some cases, but allow you to do things which weren't possible (or a least weren't feasible) before on the level of them whole program design.

Even metaprogramming doesn't change C++ as much as modules or concepts: you just get “for free” something which was already available before — just with IDL and tools like capnproto.

Sure that's nice simplification, but it doesn't imply insane amount of work which would be needed to make modules or concepts work. They would require changes in literally everything: from standard library to third-party libraries and many other things before they would become truly useful.

All the things that you listed are, actually, pretty minor in comparison: they only need local changes for you to benefit from them.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 18:41 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

> The things which you have listed are not larger than what Rust gets in three years, though.

Maybe not. These are not things that I think any one person's view is going to give the same size and scope for any individual feature as another's view anyways. Personally, these are not "small" things to be adding to C++ (in aggregate).

Pattern matching, at least, adds a decent amount of syntax to the language. Contracts does too (though, thankfully, both are via context-sensitive keywords). Template-this also has some additional syntax (though that feels like a cleanup at least).

Modularized stdlib probably is going to have knock-on effects for projects not using IWYU where types aren't leaking out over entire #include trees.

I'm also aware of the scope of modules pretty intimately: I'm working with the ISO committee to make sure they can be built at all :) .

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 19:52 UTC (Thu) by rgmoore (✭ supporter ✭, #75) [Link] (6 responses)

Sure, but the problem is that this hasn't happened yet and one of the main reasons is the fact that the language is moving so fast that alternative implementations have trouble keeping up.

I don't think it's as much a matter of alternative implementations having trouble keeping up as it is Rust not yet having reached critical mass to justify an alternative implementation. C and C++ have enough users to justify multiple FOSS implementations and multiple independent proprietary ones. Rust doesn't have that kind of user base yet.

My impression is that the perception of Rust as changing rapidly is driven by the way the compiler is developed. The developers have often used features new to release N when writing release N+1, so anyone who wants to bootstrap from another language will have to build every version of the compiler to get to the current version. That leads to the perception that the language as a whole is developing at a breakneck pace, even though it has a good history of preserving backward compatibility.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 19:57 UTC (Thu) by glaubitz (subscriber, #96452) [Link] (4 responses)

> That leads to the perception that the language as a whole is developing at a breakneck pace, even though it has a good history of preserving backward compatibility.

It may have backward compatibility, but the main problem is that many upstream projects adopt new language feature rather quickly which means that distributions with longer release cycles such as Debian, RHEL or SLE will have to backport new versions of the Rust compiler when updating packages like Firefox.

I have not observed this particular problem with C/C++ code while maintaining packages for an LTS distribution.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:01 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (3 responses)

Eh, the "core" libraries that "everyone" depends on tends to keep the MSRV (minimum suitable Rust version) pretty low, or at least documented. Sure, things get bumped occasionally, not that even that pace is suitable for LTS distro cadences, but it isn't a mad rush to use the latest-and-greatest for the entire ecosystem.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 4:54 UTC (Fri) by plugwash (subscriber, #29694) [Link] (2 responses)

The reality in Debian is that due to the huge number of security holes that web browsers acrete and the fact that even the LTS release cycles of firefox are much shorter than those of a stable release they are practically forced to upgrade to new release series of firefox within a stable release.

Before firefox/thunderbird started using rust, this wasn't a huge problem, the firefox package could be updated largely independently of everything else. Nowadays though, a new release series of firefox basically means a new release of rustc.

And that breaks things. Sure the rustc developers have defined a stable subset, but a bunch of crates depend on less stable features. Some of this through the officially endorsed "feature gates" mechanism, others through a certain flag that is not supposed to be used by user code but is anyway that lets you use "nightly" features on a stable compiler.

Python cryptography, Rust, and Gentoo

Posted Feb 12, 2021 13:52 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

> Some of this through the officially endorsed "feature gates" mechanism, others through a certain flag that is not supposed to be used by user code but is anyway that lets you use "nightly" features on a stable compiler.

Feature gates can only be opened on a nightly compiler. The stable compiler does not let you use them (unless you masquerade as the stdlib with some internal flags, but then you're in a "you get to keep both pieces" situation anyways).

What you're probably actually seeing is the crates using the newer language/stdlib features of *stable* rust releases.

Python cryptography, Rust, and Gentoo

Posted Feb 15, 2021 8:56 UTC (Mon) by glaubitz (subscriber, #96452) [Link]

> What you're probably actually seeing is the crates using the newer language/stdlib features of *stable* rust releases.

I'm pretty sure I have seen crates that required Rust nightly in the past.

Building rustfmt is one of such cases: https://github.com/rust-lang/rustfmt#installing-from-source

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:29 UTC (Thu) by josh (subscriber, #17465) [Link]

> Rust not yet having reached critical mass to justify an alternative implementation

Rust is permissively licensed, which removes much of the motivation for any alternative implementation.

And personally, I'm hopeful that mrustc will meet any regulatory or similar requirements for "alternative implementation", so that we don't need another one for that reason either.

I hope that we go as long as possible *without* an alternative implementation, to avoid fragmenting the ecosystem by expecting crate authors to use the subset of Rust provided by that implementation.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 23:06 UTC (Thu) by moltonel (subscriber, #45207) [Link]

> The difference is that these other two languages have a specification and alternative implementations that can keep up with each other, aren't they?

My point is that the alternative implems are *not* keeping up with each other and each new feature. I'm sure the spec is useful, but it clearly doesn't solve the problem.

> I mean, the whole reason this article exists is because Rust is special in this regard and causes problems for downstream projects. You can't really deny that, can you?
> Sure, but the problem is that this hasn't happened yet and one of the main reasons is the fact that the language is moving so fast that alternative implementations have trouble keeping up.

The problem is caused by Rust not being available on some platforms. You seem to imply that if it had a spec or evolved more slowly, rust would have alternate implementations an be available everywhere, but it's not that easy. Specs and lack of evolution don't give you alternate implementations, time does. The language isn't moving that fast, as other have pointed out. Alternates implementations don't guarantee availability on more platforms (see mrustc). And more platforms can be reached without the need of alternate implementations (see rustc_codegen_gcc).

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:24 UTC (Thu) by josh (subscriber, #17465) [Link]

Rust has a permissively licensed frontend (and backends). Duplicating that code seems like a substantial waste, compared to collaborating on the existing implementation.

There were many competing proprietary C compilers. GCC provided a FOSS C compiler and compiler infrastructure, and there wasn't a strong need for a *second* such infrastructure until LLVM came along with substantially different design goals (permissive license and easy extension).

I don't think there's value in a second frontend, and I think it would cause ecosystem fragmentation.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 9:26 UTC (Thu) by roc (subscriber, #30627) [Link] (1 responses)

That's awesome.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 10:00 UTC (Thu) by glaubitz (subscriber, #96452) [Link]

Feel free to join #llvm-m68k on OFTC IRC and our monthly m68k online meeting: http://m68k.info/

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 14:38 UTC (Thu) by tlamp (subscriber, #108540) [Link] (1 responses)

So another commenter stated[0] how painful it is to port rust to another platform - can you, which is doing an actual port, please comment on that? (out of pure interest).

[0]: https://lwn.net/Articles/845637/

Python cryptography, Rust, and Gentoo

Posted Feb 15, 2021 8:58 UTC (Mon) by glaubitz (subscriber, #96452) [Link]

> So another commenter stated[0] how painful it is to port rust to another platform - can you, which is doing an actual port, please comment on that? (out of pure interest).

If the new platform is just a new Linux architecture, the port is pretty much straight-forward and easy provided that LLVM has a backend for the architecture.

I cannot comment onto ports for new operating systems as I haven't worked on this myself.