Python cryptography, Rust, and Gentoo [LWN.net]

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 11:56 UTC (Thu) by Sesse (subscriber, #53779)
In reply to: Python cryptography, Rust, and Gentoo by moltonel
Parent article: Python cryptography, Rust, and Gentoo

Yes, I wanted it to be mutable after init; it's a cache of state between multiple HTTP requests. So I need a Mutex, and how do you initialize a Mutex safely without lazy_static?

to post comments

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 14:07 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (15 responses)

You could do what `lazy_static` is doing behind the scenes. It's not compiler magic or anything, but plain Rust code. The crate just packages it up in a nicer API than stamping out the manual code all the time.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 14:13 UTC (Thu) by Sesse (subscriber, #53779) [Link] (14 responses)

Sure, but when I searched around for this, people were “do not reimplement lazy_static, you'll be doing it wrong, use the crate”.

It's a bit like Turing completeness. It's _possible_ to do without a crate, but it's definitely much harder, and it doesn't really matter what you do as a single developer long as the entire ecosystem goes the other way.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 14:25 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (13 responses)

Sure, I didn't say it'd be easy. The fact that lazy_static makes it so easy behind its easy-to-use API is a *benefit*, not a downside.

Global mutable state is a tricky thing in C and C++ too. It's a mistake that they make it so easy to appear that one got it right. The fact that one can easily add a dependency that does something as "trivial" as making it does is a vast improvement. C and C++ dependency management is a PITA on a good day with reasonable projects as dependencies. Rarely do you get one, nevermind both.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 14:44 UTC (Thu) by Sesse (subscriber, #53779) [Link] (12 responses)

What? In C++, I can have a global std::mutex with no external dependencies. There's absolutely no reason why Rust couldn't have a simple way of initializing one in std, without requiring a crate.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 15:37 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (11 responses)

Sure, you can make the mutex, but what data is it guarding? Who ensures the mutex is *used* when accessing the guarded data (spoiler: no one)? When does any of it get initialized at runtime? C++ doesn't guarantee any of this stuff (other than "it happens before it's needed" for the initialization). For example, static initializers might only be run when other code in the .o is used. If the guarded data lives in the data section and is looked up in some way other than through the TU that contains the mutex (e.g., `dladdr` or something), the mutex initializer might not be run at the right time, so good luck with that.

In the case of lazy_static specifically, there is another way that looks better and is also more performant[1]: once_cell. It doesn't use a macro and looks cleaner anyways (no weird macro-syntax of `static ref`). So in this specific instance, the stdlib would have gained a subpar API for such a thing anyways. This is, IMO, a vast improvement over the Python way which enshrines bad APIs because "they were available first" and is how one ends up with urllib, urllib2, urllib3, and requests.

[1]https://github.com/async-rs/async-std/issues/406

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 15:58 UTC (Thu) by Sesse (subscriber, #53779) [Link] (10 responses)

> Sure, you can make the mutex, but what data is it guarding? Who ensures the mutex is *used* when accessing the guarded data (spoiler: no one)?

Please stop the whataboutism.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 16:13 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (7 responses)

You're arguing that Rust makes global mutable state harder to do. Yes, it does. It is in service of helping to point out improper use of such things. That is, IMO, important in such a discussion of comparing the ease of making such declarations in each language.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 16:30 UTC (Thu) by Sesse (subscriber, #53779) [Link] (6 responses)

No, I am arguing that Rust makes even simple things hard to do _without pulling in crates_. I am notably not making a comparison against C++.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 16:46 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

I'm arguing that what seems "simple" is not as simple as you might think it is based on how "easy" C and C++ have made it in the past.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 19:36 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

Why do you even need a global state? That's an uncommon thing to use, so having users to write code is perfectly fine for that.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:30 UTC (Thu) by marcH (subscriber, #57642) [Link] (3 responses)

If you think a global mutable state is a "simple thing" then you have a serious memory safety problem.

It took a very long wait, but in 2011 even C finally got a memory model that realizes concurrency has to be part of the language

https://en.wikipedia.org/wiki/C11_(C_standard_revision)

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:41 UTC (Thu) by Sesse (subscriber, #53779) [Link] (2 responses)

I think initializing a mutex should be simple thing!

I'm giving up this discussion; too many people are interested in arguing against strawmen, and too few people are interested in discussing the actual problem. It's pretty off-putting when a community's reaction to criticism is “who needs to do that anyway”.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 20:59 UTC (Thu) by mathstuf (subscriber, #69389) [Link]

> I think initializing a mutex should be simple thing!

And it is, mechanically. Semantically, it is *not* a simple thing. These kinds of issues are what Rust is aiming to tackle as a whole.

Could once_cell or lazy_static be added to the stdlib? Sure. Why not yet? Maybe the API isn't sufficiently nailed down, soundness cases considered, etc. enough for the stdlib. Until then, crates.io is a handy place for these things to mature *while getting real world (ab)use*.

Some context for the C++ side of things. Improvements living in some random P paper on the ISO C++ standard committee mailings isn't going to get battle-hardened by anyone other than the author without the heroic work of making it available on existing language specs (e.g., Eric Neibler's Ranges library). This kind of stuff is nigh impossible with language features too. There is still errata coming in for `for (auto i : expr)` for crying out loud because this is undefined behavior:

std::vector<std::string> func();
// ...
for (auto i : func()[0]) // oops, you're iterating on a temporary that just got destructed
{}

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:58 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

> I think initializing a mutex should be simple thing!
It's actually not. For example, on some systems mutexes can require a system call.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 16:14 UTC (Thu) by laarmen (subscriber, #63948) [Link]

Actually, this is relevant to the discussion. A Rust mutex isn't standalone but a container, and it must be able to guarantee that its contents are a valid value for the contained type, whereas a C++ std::mutex doesn't have any knowledge of the data it protects. The former approach makes it possible to guarantee protected access, but it makes the mutex implementation that much difficult.

I agree that it's a bit of a shame to have to resort to a thirdparty crate to easily have a global mutex-protected variable, but as pointed by someone else in the thread, it turns out the approach used by the proeminent crate for this might not be the best after all, which makes me glad it has not been imported into std after all :)

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 17:24 UTC (Thu) by farnz (subscriber, #17727) [Link]

All of those are lists of why what appears to be simple in C++ ("just" create a std::mutex at global scope) are in fact not simple at all once you consider the details. It's just that C++'s (and C's) way of doing things relies on you knowing that it's not that simple, and that you have a whole boatload of other complexity to consider when doing this.

And Rust globals are writable - you have to use the unsafe marker with writes to a global, because writing to a global without sufficient consideration of concurrency results in memory unsafety (in C, Rust, or C++ - this is a common thing to all systems languages). C++ just doesn't force you to confront that front-and-centre, where Rust does.

Python cryptography, Rust, and Gentoo

Posted Feb 11, 2021 21:19 UTC (Thu) by josh (subscriber, #17465) [Link]

> Yes, I wanted it to be mutable after init; it's a cache of state between multiple HTTP requests. So I need a Mutex, and how do you initialize a Mutex safely without lazy_static?

To answer your question:

We're currently adding an equivalent to lazy_static in the standard library: https://doc.rust-lang.org/std/lazy/index.html . It's currently available on nightly, and folks are working towards marking it stable.

We're generally very careful before adding something to the Rust standard library. We have strong stability guarantees for anything in the standard library, and once we add something it's subject to those same guarantees. The Python project has the philosophy that "the standard library is where code goes to die", for much the same reason; there are various pieces of the Python standard library for which the standard wisdom is "don't use it, use this third-party module instead". We want to avoid that situation in Rust whenever we can, even if that means that some common functionality requires a crate. It's very easy to add a dependency on a crate from the crates.io ecosystem.

Now, separate from the answer to your question, there are two reasons you might not want to use a global mutex-guarded variable as the cache for your HTTP requests. First, you might want to use a concurrent data structure instead, such as "dashmap", a fast concurrent hashmap. And second, you might consider putting that data structure in one of your library's objects instead, so that you (or other code calling into your code) can use multiple such objects concurrently without global state. All that said, you *can* use a global mutex-guarded variable if you want to, and std::lazy should let you do that using just the standard library.