LWN: Comments on "A discussion of Rust safety documentation"

Why this code possibly has undefined behavior using strict providence

hkBst — Sun, 06 Oct 2024 18:13:31 +0000

For others wondering how this code can be UB: this code triggers a UB error from Miri due to provenance issues, the exact rules of which are still being worked out, but which do have a simple superset, called strict provenance. The exact rule which is relevant for this example is that a pointer derived from a reference/slice only has valid provenance if it points to where that reference/slice can point to. Given this analysis it is simple to satisfy Miri with a small change to the code[1]:

```rust
use std::{mem,ptr};

fn bad_load_int_le(buf: &[u8; 32], i: usize) -> u32 {
assert!(i + mem::size_of::<u32>() <= buf.len());
let mut data = 0u32;
unsafe {
ptr::copy_nonoverlapping(
buf.get_unchecked(i), // provenance problem because only references the first byte
//buf.get_unchecked(i..i + mem::size_of::<u32>()).as_ptr(), // reference all four bytes
//buf.as_ptr().add(i), // reference entire buf
&mut data as *mut _ as *mut u8,
mem::size_of::<u32>(),
);
}
data.to_le()
}

// this let's you use Miri
fn main() {
let data = [0u8; 32];
bad_load_int_le(&data, 0);
}
```

[1]:https://play.rust-lang.org/?version=stable&mode=debug...

Slice to `[u8;4]`

Wol — Fri, 04 Oct 2024 13:12:42 +0000

> So... you're saying that the job _isn't_ done when the Rust compiler stops complaining, and that the resultant code might not actually _work_ even though the compiler "proved" it to be "correct"?

But isn't that *always* true? Just because the *maths* is correct, doesn't mean the *logic* is sound.

I have a production system that uses bell-curve statistics on a skewed distribution. It mostly works because the data is USUALLY a strong peak with a weak tail on one side. But feed it a double-peak, or a strong tail, and the results will be rubbish. That's a risk we've chosen to accept. The mathematical model is sound. It's just the fit to reality isn't quite right ...

Cheers,
Wol

Slice to `[u8;4]`

farnz — Fri, 04 Oct 2024 12:21:05 +0000

As with anything, it depends what you require the compiler to prove. My experience is that Rust code is more likely to use enums, structs and other such type system to make it impossible to represent impossible states, rather than clever code that allows for this sort of bug to slip past.

In other words, the problem here is clever code where you're trying to beat the optimizer, rather than trusting that the optimizer will get it right. And my experience of cheap coders (offshore outsourcing body shops) is that they trust the optimizer instead of trying to be clever, so they'll do the right thing in Rust; I have, however, had a pile-on for expressing that in the past.

Slice to `[u8;4]`

pizza — Fri, 04 Oct 2024 12:08:48 +0000

> You're right - that's the other fun of using "clever" code, in that you can confuse someone trying to refactor it, and the compiler won't complain.

So... you're saying that the job _isn't_ done when the Rust compiler stops complaining, and that the resultant code might not actually _work_ even though the compiler "proved" it to be "correct"?

(Yeah, yeah, I know you're not one of the obnoxious evangelists making that idiotic claim, but I've been piled upon here multiple times here for making this point..)

Slice to `[u8;4]`

farnz — Fri, 04 Oct 2024 11:55:40 +0000

The reason there's so much code for one asm instruction is that most of the code is concerned with things that asm doesn't care about - asm doesn't care if you're reading out of bounds (that's a CPU fault at worst, and a program bug at best), or if you're asking it to read from the "wrong" location, or a whole host of other things that programming languages care about.

And it's similar in size to the dance you need to do with standard C to get the same instruction.

Slice to `[u8;4]`

geert — Fri, 04 Oct 2024 11:41:01 +0000

About your original: Ah, adding -Wunused helps.
About the new version: Wow, writing that much code to generate a function with a single asm instruction...

Slice to `[u8;4]`

farnz — Fri, 04 Oct 2024 10:42:01 +0000

You're right - that's the other fun of using "clever" code, in that you can confuse someone trying to refactor it, and the compiler won't complain.

A better translation would be this Godbolt link, which makes the use of unsafe very clearly about not panicking on error, and uses safe functionality for all of the data transformation. It also makes it clear that this function should be marked as unsafe, since as a precondition it requires buf to be long enough.

Slice to `[u8;4]`

geert — Fri, 04 Oct 2024 08:19:49 +0000

Seems you have dropped using the "i" input parameter value?

Unsafety can be subtle

ralfj — Sat, 21 Sep 2024 06:45:28 +0000

khim, I understand you are passionate about Rust-for-Linux and want to see that project succeed, but I think unnecessarily aggressive comments like this are hurting the cause. It is not necessary to repeat the point ("one element" vs "entire buffer") so many times in your answer, nor is it appropriate to suggest people forgot basic things taught early in school. Such a comment will not manage to convince anyone of your position, it will just make people uninterested in engaging with you. Please re-think your communication strategy and try to be a little more empathetic towards the position of others. :)

Kind regards,
Ralf

Unsafety can be subtle

ralfj — Sat, 21 Sep 2024 06:22:31 +0000

> Ralf Jung has also explored the idea of modifying Stacked Borrows to make it legal [2].

Unfortunately those modifications wouldn't make this particular code legal. I don't know any way to achieve that without going to something like Tree Borrows.

comments can end up being very good at _hiding_ bugs

ralfj — Sat, 21 Sep 2024 06:21:36 +0000

Personally, I find such comments to be invaluable when reviewing tricky unsafe code -- I can compare the comments (expressing the authors intent in a fine-grained way) against the actual code in a fairly local way, which is a lot easier than trying to figure out how everything in the current function works together to make this line correct. It's the difference between things being checkable in a local way vs having to globally keep everything in my head at once.

Unsafety can be subtle

ralfj — Sat, 21 Sep 2024 06:18:05 +0000

Note that the error Miri shows for this code has the following:

= help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
= help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/... for further information

It is hard to find a set of rules for these aliasing requirements that is precise, can be automatically checked, and allows the desired optimizations. We don't have the final answer yet. That's why there are no strong warnings against this kind of code in the docs: It is deep inside the "gray area", and it is not something we actually already use for optimizations. Restricting references to only access the field/element they point to is something I strongly think we should relax (and that's what the discussion at https://github.com/rust-lang/unsafe-code-guidelines/issue... is about). This is especially true for your case, where the reference points into an array, for the reasons you give -- pointer arithmetic inside arrays is legal even in C, so we should avoid such pitfalls in that space as much as we can. (Unlike C, Rust also permits pointer arithmetic in structs if you use raw pointers instead of references.) But it's hard to make Stacked Borrows accept this code without making Stacked Borrows accept *too much* code.

Nobody forces you to make your code compatible with Stacked Borrows, it is an *experimental* model after all. But if you want to be as sure as you can currently be that your code is fine with whatever the future aliasing model ends up being, then it is a good idea to do the extra work and fix Stacked Borrows errors. Think of it like adding a safety margin -- we don't know where the edge between "UB" and "okay" will be, so it is better to err on the side of staying a bit conservative and avoiding the gray area. If you are okay with living a bit more "on the edge" and an increased risk of having to adjust your code in the future as the aliasing model solidifies, you can use "-Zmiri-tree-borrows" to switch to a different, even more experimental aliasing model that *will* accept your code. This model should still be able to catch all UB that is exploited by today's optimizer (but the optimizer can change in the future, and we're not ready yet to commit to a hard set of rules here).

> so I am fairly confident that this is actually undefined behavior, and not just extra paranoia

It is Undefined Behavior under *experimental* rules, as the error clearly indicates. The rules are deliberately conservative, so there is some extra paranoia involved here. We don't currently instruct the optimizer to actually make us of this particular UB, but there is other Stacked Borrows UB that we *do* use in the optimizer, and a bunch of Stacked Borrows UB that we *may want to* use in the optimizer. It's unclear how to tell these apart, hence this error.

Unsafety can be subtle

apoelstra — Fri, 20 Sep 2024 23:02:42 +0000

This idea of using `unreachable_unchecked` to just tell the compiler what I know to be true, then using safe unwrapping methods from there on out (since the optimizer will now definitely be able to make the error paths go away), is really cool, and very general!

But I will add my voice to several others asking that you take a kinder tone and offer others the benefit of the doubt (or even, just stop assuming the worst). It makes these technical insights much harder to read and less likely to be noticed.

Unsafety can be subtle

daroc — Fri, 20 Sep 2024 11:20:30 +0000

Let's stop discussing this here. It doesn't look like you're likely to change NYKevin's mind, and it seems like this comment doesn't really contribute any new technical information. Sometimes the best thing you can say is just "I still don't agree with you, but thanks for having this discussion with me."

Unsafety can be subtle

comex — Fri, 20 Sep 2024 02:48:16 +0000

FYI, the main pattern you cited - where you take a reference to part of an allocation, then convert it to a pointer and index out of bounds of the referenced type (but still in bounds of the allocation) - may end up becoming defined behavior.

It is undefined behavior according to Stacked Borrows. But it's not undefined behavior [1] according to Tree Borrows, which is an alternative model that Rust may end up officially adopting instead of Stacked Borrows. Ralf Jung has also explored the idea of modifying Stacked Borrows to make it legal [2]. See also the canonical issue report about this pattern [3].

Broadly speaking, this pattern is something that would be nice to make legal at least for immutable references / reads, because there aren't any current optimizations, or highly-desired future optimizations, that would break it.

The other pattern you cited - aliasing with mutable references - is different. It needs to be UB in _some_ form, because it's used to justify an important optimization: marking function parameters of reference type with LLVM's `noalias` attribute, equivalent to C `restrict`. Since most C code does not use `restrict`, this is and will remain an area where Rust unsafe is harder to reason about than C.

That said, Tree Borrows affects this pattern as well, and at least reduces the amount of UB compared to Stacked Borrows. Specifically, if you create an aliasing mutable reference but don't actually write through it (at least not to the same bytes that you later read from), then you're probably okay under Tree Borrows. [5]

(For pedantry's sake, I should mention that even Stacked Borrows' rule is less strict than how you phrased it, but only slightly.)

[1] https://github.com/BoxyUwU/rust-quiz/issues/11
[2] https://rust-lang.zulipchat.com/#narrow/stream/136281-t-opsem/topic/Making.20Stacked.20Borrows.20better.20with.20ugly.20hacks
[3] https://github.com/rust-lang/unsafe-code-guidelines/issue...
[4] https://github.com/rust-lang/unsafe-code-guidelines/issue...
[5] https://perso.crans.org/vanille/treebor/protectors.html#n...

Unsafety can be subtle

khim — Fri, 20 Sep 2024 00:51:39 +0000

> I literally just told you a story like that, and you responded by making up a bunch of things I didn't say,

How is it different from how you make up piece of documentation that never existed and write “slice::get_unchecked is documented to have the same semantic meaning as C pointer arithmetic” then use that “a bunch of things [others] didn't say” as justification for your assertions?

I have even gave you benefit of doubt and asked just where it was documented that way.

And in the end, instead of showing us something like that you decided to blame the documentation for your inability to read: “it is perfectly reasonable to interpret the two as synonymous when reading documentation casually”.

And when I ridiculed it editor only had to say this: “This is an on-topic discussion, but please remember to keep things polite”.

And yes, I agree, I wasn't polite, but I, at least, stick to the facts.

> and then ridiculing those things.

You want to say that only you can lie (that one even you admitted as a lie: “but literally nobody follows the strict provenance rules anyway, since they're explicitly marked as experimental and non-normative“) and then use these lies to misrepresent things? Others couldn't do that?

Let me quite myself, for a change (you haven't misquoted me, just ignored what I say): Rust developer would like to hear “hey, I don't know how to implement this kind of design while staying in the boundaries outlined by the strict provenance approach” stories. Not stories related to someone's inability to read or understand the documentation (documentation writers do accept patches for such cases, but, as noted, it's not clear how to change the documentation and force someone to actually try to read and understand it… I'm not even sure that's possible at all)!

And you tell me that we are discussing exactly such story here. But that's big fat lie, author of the original example even admitted that original code that was discussed was brought to compliance with stacked borrows (and the whole example that we are discussing here really-really doesn't need any pointers and thus stacked borrows or other such horrors, if you want to eliminate bounds checking then call to one unsafe function with literally zero arguments is absolutely enough).

Thus it doesn't see an example of design that is limited by strict provenance. Not even remotely close to it. Sorry.

Unsafety can be subtle

khim — Fri, 20 Sep 2024 00:38:17 +0000

> In this case your code appears to be correct, but I don't think "avoid pointers, always reach for transmute first" is a reasonable policy for authors of unsafe code.

Why? Generally the advice is to use the safest approach if possible and while, technically, transmute have pretty-much unlimited chaos potential, but it have fewer failure modes than transmute_copy (compiler can at least verify that sizes of objects match) and, of course, transmute_copy have fewer failure modes then copy_nonoverlapping (in addition to all failure modes that transmute_copy you now have to deal with possibilities of having two objects overlapping and pointer being invalid, etc).

> You have replaced code which "narrowly" uses unsafe to do an unchecked array index with code that uses `transmute`, which has pretty-much unlimited chaos potential!

That's still a good change. Heck, C++20 even added their own version of transmute citing precisely pifalls of correct use of memcpy (C/C++ version of copy_nonoverlapping).

P.S. I wanted to avoid discussion about why your approach to the whole thing is more-or-less ridiculous by assuming that you just wanted to show how it's not easy to use pointers in Rust, but if your goal is to just read integer then Rust documentation, of course, includes pretty good example which explains how to do what you want to do properly, without any unsafe code here. And if you want to avoid panic there, then the simplest way is to just literally tell the compiler that you know what you are doing:

pub fn read_le_u32(input: &[u8]) -> u32 {
    if input.len() < 4 {
        unsafe { std::hint::unreachable_unchecked() }
    }
    let (int_bytes, _) = input.split_at(std::mem::size_of::<u32>());
    u32::from_le_bytes(int_bytes.try_into().unwrap())
}

That's it. It would be compiled into one instruction and there are really no need to invoke horrors of transmute, transmute_copy or copy_nonoverlapping if you really just want to eliminate the bounds check.

Unsafety can be subtle

NYKevin — Thu, 19 Sep 2024 23:03:04 +0000

> Rust developers are ready to accommodate certain requests to help these people, but “fuck everyone without C experience and make their life miserable for the sake of 10% of Rust users who are using it as C replacement” doesn't sound like a reasonable request.

Please stop making quotes up and attributing them to other people. It is extremely rude.

> Nope. This doesn't work like this. “I will ignore any and all rules that you may place on me” is an attitude problem, not documentation problem.

Ibid.

> It's one thing to say “hey, I don't know how to implement this kind of design while staying in the boundaries outlined by the strict provenance approach”. Such stories (if justified and without any simple solution that would make them strict-provenance compliant) are always welcome on the URLO and IRLO or even just the bugtracker.

I literally just told you a story like that, and you responded by making up a bunch of things I didn't say, and then ridiculing those things.

Since this conversation is clearly going nowhere, I'm going to bow out.

Unsafety can be subtle

khim — Thu, 19 Sep 2024 19:35:34 +0000

> This is an incredible inference from example code I posted to illustrate pointer provenance rules and which has never actually been deployed in the form I described.

That's not not inference from the example code, but more of inference from your comments. You are using unsafe function to do some manipulations that usually are done with normal, safe, code in an unsafe way by using function that's sole purpose is to save a tiny amount of execution time by bypassing the usual safety checks. And then you add debug_assert that, essentially, restores error-checking that is normally used by Rust programs unconditionally in debug build. And then complain that function that you used to bypass normal safety checks have poor documentation – while asking for clarifications that are already present in the sibling version of that same function, that's already referred as “safe alternative” there and which is one click away. Which means that, you probably, have never read the documentation for the safe variant: how else would you not notice that this variant already addresses all you concerns?

Forgive me, but I fail to see how one would end up in that situation except if one have apriori decided for oneself that one is not interested in writing normal, safe code and would stick to the removal of Rust-provided guardrails as much as possible.

Note: I'm not even saying that's it's wrong to write code that way. Maybe, for the kernel needs, it's even the best way and you haven't forgotten to erect manual check after you removed the normal Rust-provided provided check there.

But that definitely puts you into a different usage mode: normally it's expected that Rust developer would stick to normal, “safe” interface as much as possible and would only remove guardrails where that's truly justified.

Going down that “normal” road it's almost impossible to miss detailed specification that explains what get function works, how is it supposed to be used, there are examples with a single argument and a range, etc.

Unsafety can be subtle

apoelstra — Thu, 19 Sep 2024 19:09:06 +0000

>Your situation is a bit unusual because you have immediately decided to skip all the range-checking without testing and without verifying that these are the bottlenecks and are worth eliminating

This is an incredible inference from example code I posted to illustrate pointer provenance rules and which has never actually been deployed in the form I described.

Slice to `[u8;4]`

atnot — Thu, 19 Sep 2024 17:02:47 +0000

Personally I've never needed this because usually if you're reading bytes from a file, you'll be dealing with the `Read` trait, which means you can just `reader.read_exact(&mut array[..])` which will let you handle it as an EOF.

I do sometimes wish there was a function to return an array directly in the standard library and save that line of code, but honestly there's enough utility crates like bytemuck, binrw etc. that you'll probably want to be using anyway if you're doing a lot of reading bytes into filetypes.

Unsafety can be subtle

khim — Thu, 19 Sep 2024 14:20:46 +0000

> Though having said that, `get_unchecked` is a very old method and one of the classic "here's a case where you'd need to use unsafe" methods, so I'm skeptical that many users will carefully (or even casually) read the docs.

Note that get_unchecked is, actually, range-check-skipping version of get and it even explicitly tells you: For a safe alternative see get (with clickable link!), which actually then tells you that you may either access one element or range of elements with this method.

Your situation is a bit unusual because you have immediately decided to skip all the range-checking without testing and without verifying that these are the bottlenecks and are worth eliminating (note that if they are easy to eliminate for human then often compiler can easily eliminate them, too, after inlining, which means these checks are not, necessarily, a performance bottlenecks and often where they are performance bottlenecks they are, often, desirable because if compiler couldn't prove that they are not needed then chances are high that it's not actually guaranteed by the structure of your code, either).

That's why you have missed all these explanations (that are there for get, but not for it's range-check-skipping sibling).

That's not how slice is supposed to be used, normally! I guess documentation was just not written with people which would try to remove as many guardrails as possible as early as possible.

Are you even sure in your case use of get_unchecked actually brings measurable performance wins over the use of get?

Slice to `[u8;4]`

farnz — Thu, 19 Sep 2024 13:46:21 +0000

Note that you can use try_from to convert a slice of the correct size into an array. Assuming that you've verified that the slice is big enough, this can't actually fail - and you'd expect the compiler to optimize out the failure path as a result.

And, indeed, when I go that route (and yes, this is slightly obscure), I get the optimal form that cannot panic at runtime:


use std::convert::TryInto as _;

pub fn load_int_le(buf: &[u8; 32], i: usize) -> u32 {
    let slice = &buf[0..4];
    u32::from_le_bytes(slice.try_into().expect("load_int_le slicing issue Q1234Qload_int_leQ32Z32Z"))
}

The string "Q1234Qload_int_leQ32Z32Z" is in there as something that's unlikely to appear in the compiled binary by mistake; you can thus have release-mode CI grep for that string and fail if it appears, because you want the failure path to be resolved at compile time; if it does appear, you know where to check.

Unsafety can be subtle

apoelstra — Thu, 19 Sep 2024 13:42:32 +0000

>So what's so special about get_unchecked() that makes it different, exactly?

To restate khim's post much more briefly: the difference is that as_ptr returns a raw pointer while get_unchecked returns a reference.

I agree that it'd be helpful to say this in the docs (as well as to mention that you can call get_unchecked with a range if you want access to more than one element, which is how I ended up fixing my original code). I don't even think you need to mention C.

Though having said that, `get_unchecked` is a very old method and one of the classic "here's a case where you'd need to use unsafe" methods, so I'm skeptical that many users will carefully (or even casually) read the docs.

Unsafety can be subtle

khim — Thu, 19 Sep 2024 13:18:00 +0000

> It is quite unreasonable to assume that people writing unsafe Rust have no preconceptions from C whatsoever.

Yet it would be even more unreasonable, some would even say preposterious, to assume that everyone who uses Rust and even unsafe Rust is a C programmer.

People who are using Rust and who never used C do exists and there would be more of them over the time.

I wouldn't be surprised to find out that there are more of them already than Rust users that know C.

The desire to bring Rust in kernel is driven, in large part, by the desire to bring precisely these people into the mix.

> The bytes that make up the element are also the bytes that make up the buffer, so it is perfectly reasonable to interpret the two as synonymous when reading documentation casually.

That's not true for the majority of the languages. If we only include popular languages then it's, essentially, true only for C and C++. C#, Go, Java, JavaScript, Python… nowhere would you find such equation. Most of these languages don't give you access to the part of the object, but if such ability is there then you get to touch that part and nothing else.

> Rust cannot escape the shadow of C, as you can see by the the functions as_ptr() and friends, since they exist primarily for interoperation with C (as_ptr_range() says this explicitly).

That's true, for some degree, but you can write lots of apps for years in Rust and never hit the need to know that dark corner of the language. Pushing its existence in the description of various perfectly normal, “safe” function would definitely be strange.

Equally as strange as saying that you have to go and learn C before you would attempt Rust.

Remember that Rust wasn't even imagined as a replacement for C and C++, it was originally not developed for that audience and such work is still not it's primary focus.

Rust developers are ready to accommodate certain requests to help these people, but “fuck everyone without C experience and make their life miserable for the sake of 10% of Rust users who are using it as C replacement” doesn't sound like a reasonable request.

> Either you tell everyone what the rules are in a highly explicit and unambiguous way, or some programmers will misunderstand them,

Nope. This doesn't work like this. “I will ignore any and all rules that you may place on me” is an attitude problem, not documentation problem. And the only solution is social. Rust community does well there thus there's a chance.

Unsafe Rust is hard. You just have to accept it. Yes, in an ideal world it would be easy. Yes, people know it's a mess and try to fix that. But right now the story is: you have to be careful.

Sticking with strict provenance is your best bet right now (after you have looked on safe sound wrapper around unsafety and couldn't find a suitable one for your usecase, of course). If you can do that. These are very strict, yet sensible, rules and they allow 99% of code that needs unsafe to be written. And the remaining 1%… they are still working on it.

Complaining about the fact that something that is known to be underdocumented and hard and just saying that it's underdocumented and hard wouldn't help you if you have no constructive ideas about how to change that situation.

> and then compiler writers will once again be stuck having to stick flags all over their shiny new optimizer because it breaks some legacy code that was wrong when it was written.

Not if they would proactively kick out people who are writing code that ignores the rules.

It's one thing to say “hey, I don't know how to implement this kind of design while staying in the boundaries outlined by the strict provenance approach”. Such stories (if justified and without any simple solution that would make them strict-provenance compliant) are always welcome on the URLO and IRLO or even just the bugtracker.

But the first line of defense should always be an attempt to not use the unsafe code in the first place! And the second line is to use sound API. And third line is strict provenance. And only after all these possibilities are exhausted you go to these forums in a search of a solution.

You tell us: nobody is going to carefully scrutinize every word of docs.rust.org trying to figure out whether Professor Plum did it with the misaligned pointer in the .text section – but that's exactly how fully conforming C code is supposed to be written! You already have to look on subtle nuiances of wording of the C standard and defect reports and other such things… Rust just makes it easier because there are various forums where you can ask Rust language developers for clarification.

Unfortunately “we code for the hardware” people don't even stop to think about whether it's possible to do something while staying within boundaries – but that's social problem and can not be solved by technical means.

Technology may just make conformance easier. And Rust language developers are trying to help with documentation and tooling (Miri is a must-have for someone who develops unsafe code in Rust), but, ultimately, only developer can be responsible for the conformance with the rules. Part where compiler does it's magic and where you don't have to carefully scrutinize every word exists on the other side of unsafe!

Unsafety can be subtle

apoelstra — Thu, 19 Sep 2024 12:55:35 +0000

>As a bonus, the bounds check will apply in release mode too

This isn't a bonus. The point of the original code was to avoid the bounds check and associated panic path.

>you can implement it with `unsafe` but without any tricky references or pointers:

Aside from your new code now having `try_into` and an explicit panic path, it also illustrates my comment perfectly :). You have replaced code which "narrowly" uses unsafe to do an unchecked array index with code that uses `transmute`, which has pretty-much unlimited chaos potential! In this case your code appears to be correct, but I don't think "avoid pointers, always reach for transmute first" is a reasonable policy for authors of unsafe code.

Unsafety can be subtle

apoelstra — Thu, 19 Sep 2024 12:50:38 +0000

The method you are quoting takes a [u8; 4], not a slice or a larger array.

Unsafety can be subtle

daroc — Thu, 19 Sep 2024 11:53:32 +0000

This is an on-topic discussion, but please remember to keep things polite. Specifically:

> You couldn't tell the difference between “one” and “many”? That's explained pretty early in most schools. Maybe you forgot?

This is not polite or respectful, and doesn't add anything interesting to your technical point. Please avoid insulting other commenters.

Provenance rules not yet formalised

farnz — Thu, 19 Sep 2024 09:11:00 +0000

You're right to say that the rules for provenance haven't yet been fully formalised, and I suspect that the operation you're talking about is in that set.

Right now, the "real" rules for pointer provenance are "whatever LLVM happens to do on this machine at this version, given that LLVM doesn't want to do anything that will shock C or C++ programmers". Strict provenance is a set of rules that are guaranteed to be stricter than whatever the final rules will be; the intention is that the final formalization of provenance rules will end up being equivalent to "strict provenance, except that these things banned by strict provenance are actually allowed in the final rules".

It'll probably not be written that way, though; there will be strict provenance rules (as today), and then a set of rules that are verified to be a superset of strict provenance rules. There may even be multiple tiers of this, where each tier is both less strict and harder to verify your code against than the tier above.

The idea remains, though, that if your code is compliant with strict provenance rules, it will definitely be compliant with the final rules for pointer provenance; whether those be PVNI-ae-udi from draft N2676 for the C language, or something with more subtleties in it. And there's thus a promise from the Rust compiler developers that they'll avoid breaking unsafe code that complies with strict provenance, even though the final provenance rules haven't yet been determined.

Unsafety can be subtle

donald.buczek — Thu, 19 Sep 2024 09:01:41 +0000

> Can we, please, stop the language lawyering.

I don't understand why you have to devalue your otherwise interesting answers by using unnecessary fighting words.

> You don't say “can you identify a concrete rule which is violated?”
> This asserts that only documentation and rules included in it matters and everything outside of it just simply doesn't exist or could be ignored.

This interpretation of my question is wrong.

Unsafety can be subtle

NYKevin — Thu, 19 Sep 2024 03:34:41 +0000

> If Rust would have been a new version of C then sure. But Rust is not a new version of C and, in general, in Rust, you don't get the permission to access something that lies outside of that object that function gives you and, most importantly, nowhere near that point raw pointers are even discussed.

Rust cannot escape the shadow of C, as you can see by the the functions as_ptr() and friends, since they exist primarily for interoperation with C (as_ptr_range() says this explicitly). It is quite unreasonable to assume that people writing unsafe Rust have no preconceptions from C whatsoever.

> There are big difference betwee a reference to an element or subslice and pointer to the slice’s buffer.

The bytes that make up the element are also the bytes that make up the buffer, so it is perfectly reasonable to interpret the two as synonymous when reading documentation casually. If this is really an important distinction, then it needs an entire section of the 'nomicon, not just a couple of stray words in std::ptr and then a minor difference in phrasing here or there in other parts of std. This is not a game of Clue - nobody is going to carefully scrutinize every word of docs.rust.org trying to figure out whether Professor Plum did it with the misaligned pointer in the .text section.

Either you tell everyone what the rules are in a highly explicit and unambiguous way, or some programmers will misunderstand them, and then compiler writers will once again be stuck having to stick flags all over their shiny new optimizer because it breaks some legacy code that was wrong when it was written.

Unsafety can be subtle

atnot — Thu, 19 Sep 2024 00:03:41 +0000

Not to call the pot black as a kettle, but I would greatly appreciate if you could stop having the same near-identical heated argument under every C or Rust related post, and presumably so would others that haven't muted you yet.

Unsafety can be subtle

khim — Wed, 18 Sep 2024 23:26:09 +0000

> Because the reference was returned by slice::get_unchecked, and slice::get_unchecked is documented to have the same semantic meaning as C pointer arithmetic (but with the added constraint that it is illegal to construct a "one past the end" pointer, unlike in C where that is legal).

Where is it documented that way? Official documentation is pretty clear: returns a reference to an element or subslice, without doing bounds checking. And subslice there means not any random subslice but specifically subslice specified by something like Range. That's not how it was used in the discussed example, thus we can ignore that case.

Nowhere does it tell is that you get permission to access anything outside of that single element (or slice).

> C pointer arithmetic is generally understood to return a pointer with provenance over the whole array, so if you're going to change that rule, you really ought to do so explicitly.

If Rust would have been a new version of C then sure. But Rust is not a new version of C and, in general, in Rust, you don't get the permission to access something that lies outside of that object that function gives you and, most importantly, nowhere near that point raw pointers are even discussed.

Why function from not C, that returns something that C doesn't support and in a fashion that is typical for that language should include warning in the form of “hey, if you know about C then remember that this function works like a Rust function and not like a C function”. That would be really very strange. Obnoxious and repetitive (remember that most Rust developers don't, actually, know C, they come from Java or Python background).

> Note also that a completely literal interpretation of your argument would equally well apply to slice::as_ptr()

Sure, but result would be different. Because that one is different: it returns a raw pointer to the slice’s buffer.

There are big difference betwee a reference to an element or subslice and pointer to the slice’s buffer.

Element is, well, element. Singular. One byte. Buffer is not one element, it's content of the entire slice.

> Nothing in either function's documentation clarifies that there is a meaningful distinction.

You couldn't tell the difference between “one” and “many”? That's explained pretty early in most schools. Maybe you forgot?

> So what's so special about get_unchecked() that makes it different, exactly?

Well… the fact that it does different thing?

> Unfortunately, in this case, we have the option of saying that it is indeed legal to do pointer arithmetic on pointers derived from get_unchecked(),

Whoa, whoa, whoa. Of course you can do pointer aritmetic – as long as you don't go beyond boundaries of that one, single, byte that you have got access to!

What else do you expect if you have got reference to one byte and not reference to the whole buffer?

> or we have the option of saying that it is not possible to use as_ptr_range() for anything other than accessing the first element.

Whoa, whoa, whoa. Why would that be the case? One function returns reference to one, single, element (or subslice), another function returns pointer to buffer and third one returns two raw pointers spanning the slice.

Why would they behave identically if they return references and/or pointers that deal with different entities. How is that ever logical?

> If it is the second one (stricter provenance for references than for raw pointers), what exactly are those rules, and where can I read about them?

Why would you need any such rules? If your pointer or reference are pointing to one element, then you can touch that element and nothing else. If your pointer or reference are pointing to the entire buffer then the whole buffer is fair game.

It's really as simple as that: there are no radical difference between pointer and reference in the [naïve] approach to their provenance (experimental models relax the restrictions, don't add a new ones), but these functions return references (or pointers) to a different objects, why should they behave identically?

Unsafety can be subtle

NYKevin — Wed, 18 Sep 2024 22:39:24 +0000

> How is this even relevant? There are no “shrinkage of provenance” in that example. You have reference that's pointing to one, single byte. That means that you get to access just one, single, byte and nothing more.

Because the reference was returned by slice::get_unchecked, and slice::get_unchecked is documented to have the same semantic meaning as C pointer arithmetic (but with the added constraint that it is illegal to construct a "one past the end" pointer, unlike in C where that is legal). C pointer arithmetic is generally understood to return a pointer with provenance over the whole array, so if you're going to change that rule, you really ought to do so explicitly.

Note also that a completely literal interpretation of your argument would equally well apply to slice::as_ptr(), because the return type of that function is const *T, not some nonexistent unsafe slice type (and so a pedantically literal interpretation is that you may only access the first element of the slice through said pointer, because it is a pointer to T, not a pointer to unsafe-slice-ish of T).

In fact, I cannot find a solid reason to allow one and not the other. Nothing in either function's documentation clarifies that there is a meaningful distinction. At the same time, it is obvious (by the existence of e.g. as_ptr_range()) that this is at least meant to be allowed in the case of as_ptr(). So what's so special about get_unchecked() that makes it different, exactly?

> That's direct consequence of refusal to adopt insanity of C/C++ TBAA: if we no longer consider memory as typed where different pieces of memory turn into differently typed objects magically, then boundaries of objects are now parts of pointers and references, for how else can we reason about them?

It sounds as if you're saying that the rules for pointers are the same as the rules for references, which I agree makes logical sense. Unfortunately, in this case, we have the option of saying that it is indeed legal to do pointer arithmetic on pointers derived from get_unchecked(), we have the option of saying that references have stricter provenance rules than raw pointers, or we have the option of saying that it is not possible to use as_ptr_range() for anything other than accessing the first element. The latter is obviously ridiculous and untenable, so which of the first two is your interpretation of this case? If it is the second one (stricter provenance for references than for raw pointers), what exactly are those rules, and where can I read about them?

Unsafety can be subtle

khim — Wed, 18 Sep 2024 21:37:16 +0000

> I still cannot find a single, authoritative source which explicitly documents all of the operations that can shrink provenance, and I'm not 100% convinced that it has even been formalized in the first place.

How is this even relevant? There are no “shrinkage of provenance” in that example. You have reference that's pointing to one, single byte. That means that you get to access just one, single, byte and nothing more.

That's direct consequence of refusal to adopt insanity of C/C++ TBAA: if we no longer consider memory as typed where different pieces of memory turn into differently typed objects magically, then boundaries of objects are now parts of pointers and references, for how else can we reason about them?

This automatically and immediately makes this example illegal. Which is good for everyone concerned: given the fact that out-of-object accesses are core issue with 90% of all exploits we don't want to declare such accesses “normal”.

> But if something is marked as "non-normative and experimental" in the documentation, I personally am going to ignore it until such time as that label goes away.

Then you couldn't do many things then these “non-normative and experimental” models give you. You are left, essentially, with this: The result of casting a reference to a pointer is valid for as long as the underlying object is live and no reference (just raw pointers) is used to access the same memory.

All other usages (in particular the potential ability to go from a single byte to something larger) need these “non-normative and experimental” models because without them they all these operations fall under the the precise rules for validity are not determined yet clause.

This reading, most definitely, doesn't give you the ability to access anything outside of that one, single, byte that you received from get_unchecked

Unsafety can be subtle

NYKevin — Wed, 18 Sep 2024 18:37:22 +0000

OK, "nobody" was maybe an exaggeration. But if something is marked as "non-normative and experimental" in the documentation, I personally am going to ignore it until such time as that label goes away.

The other problem is that you have to read these rules extremely carefully to even notice that this is illegal in the first place, because they're almost exclusively focused on the usize -> ptr conversion, and barely acknowledge that "shrinking provenance" is a thing that can happen through other operations. I still cannot find a single, authoritative source which explicitly documents all of the operations that can shrink provenance, and I'm not 100% convinced that it has even been formalized in the first place.

Unsafety can be subtle

khim — Wed, 18 Sep 2024 13:03:27 +0000

> ...And what do you do if the rules are ambiguous or do not match reality?

The important thing is what you don't do. You don't say “can you identify a concrete rule which is violated?”

This asserts that only documentation and rules included in it matters and everything outside of it just simply doesn't exist or could be ignored.

Unsafety can be subtle

pizza — Wed, 18 Sep 2024 12:35:52 +0000

> No. The “entire point” of Rust is to make program safer. Highly-opinionated rules (as well as any other rules) help as long as they are followed.

...And what do you do if the rules are ambiguous or do not match reality?

while (<undesireable behavior occurs>) {
if (<behavior allowed by spec>) {
if (!<we care>) {
break; // The typical answer of C language stewards
} else {
// bug in specification
fix_spec(); // The typical answer of Rust language stewards
}
} else {
// bug in compiler/tooling
fix_implementation();
}
}

Unsafety can be subtle

khim — Wed, 18 Sep 2024 12:19:37 +0000

> Language lawyering is how "problems" with your language are identified

Sure. That's how you cause dissent and doubt. Why is that needed here?

> Strongly enforcing a still-growing set of highly opinionated rules is the *entire point* of Rust.

No. The “entire point” of Rust is to make program safer. Highly-opinionated rules (as well as any other rules) help as long as they are followed.

And language lawyering (attempt to use the documentation as holy gospel while forcibly ignoring things that exist outside of it) is not how rules are followed, but the exact opposite: it's the way to dig extra rules from the exact same text.

It's useful activity only as long as you are planning to improve the documentation itself. To show that documentation contradicts the implementation or is deficient for some other reason (e.g. if it have self-contradictions or something that couldn't be supported), etc.

When you are doing that to find an excuse to do or not do something… this just splinters the community, because different people may “discover” different things in the exact same “holy” text.

Unsafety can be subtle

pizza — Wed, 18 Sep 2024 11:51:17 +0000

> Can we, please, stop the language lawyering. That's how C turned into the disaster, there are no need to repeat that story.

Language lawyering is how "problems" with your language are identified [1]. How those "problems" are handled/resolved is another matter entirely.

[1] Strongly enforcing a still-growing set of highly opinionated rules is the *entire point* of Rust.