|
|
Subscribe / Log in / New account

Too complex for my small brain

Too complex for my small brain

Posted Nov 11, 2024 12:14 UTC (Mon) by daroc (editor, #160859)
In reply to: Too complex for my small brain by khim
Parent article: Safety in an unsafe world

Do you happen to have any code demonstrating your approach published? I feel like I have a good intuitive understanding of Fuchsia's approach, but I don't think I understand your approach as well.


to post comments

Too complex for my small brain

Posted Nov 11, 2024 15:49 UTC (Mon) by khim (subscriber, #9252) [Link] (4 responses)

You mean the code in production? That's not “my” code, just code I helped to develop. And I would need to spend a lot of time what dependencies we are tracking, how and why (it wasn't about lock tracking, rules there were different, that's why simple pair of before and after traits were not enough).

I think it would be better to concentrate on that simplified example that started the whole discussion.

I kinda assumed that it would be trivial to follow yet, apparently, it's not trivial… thus I would try to explain the ingredients that are used there and show how can they be combined together to detect violations of locking ordering.

  1. Zero-sized types, in Rust, may carry compile-time payload. Specifically:
    1. They can carry information about some other type
    2. They can carry information about lifetime
    3. They can carry values (integers or sequences of integers)
  2. To check the correctness of locking we would use all three properties. Specifically:
    1. We would be putting information about all currently taken locks in the context (by using 1.1 property, obviously)
    2. We would do that by putting zero-sized reference to the previous context: if you take &'a mut T and convert into PhantomData<&'a mut T> then this zero-sized type acts precisely like a real reference… except you couldn't use it to derefence anything (but we don't have anything interesting in the context itself, the only interesting things are in the context type, thus we don't need to dereference anything) – in particular as long as PhantomData<&'a mut T> exists the original context is not accesible (by using 1.2, obviously)
    3. And we also would add some constant to know what we have put in the that context, e.g. we may put all locks in one enum and then simply use LockPriority::IPV6LOCK us usize (by using 1.3, obvious)
  3. And with all that placed in our context we only just need to verify that lock that we are taking right now is valid. In Rust 1.79+ that's trivial: you just used const { assert!(…) } or const { if … { panic!("…") } and that's it. That's a bit strange because it looks as if that's a runtime check, but no, that's compiler-time check. So… you have two constants, compare and you would know if lock is Ok to take or not (ask me if you need to backport it to older versions of Rust, there you needed somewhat more complicated dance, but it's also doable).
    Note that you have access to all types in the chain here, even if indirect, and could implement arbitrarily complex verification scheme… it's only hindered by the fact that const functions couldn't be part of the trait, so you would probably want to put your machinery in separate module to not fill the namespace with helper functions.

And that's… essentially it. We are done. Just need some syntax sugar to make it more pretty. I guess you would want to add something like LockPriority trait:

pub trait LockPriority : sealed::LockPriority {
    const PRIORITY: usize;
}

impl<'a, T: ?Sized, const P: usize> LockPriority
for TakenLockPriority<'a, T, P> {
    const PRIORITY: usize = P;
}

mod sealed {
    pub trait LockPriority {}
    impl<'a, T: ?Sized, const P: usize> LockPriority
    for super::TakenLockPriority<'a, T, P> {}
}

Would also need some macro because you would be doing things like
let (mut ctx, guard) = mutex.lock(UsePriority(& mut ctx));
pretty often. Or maybe not, maybe it's better to keep that explicit. Attempts to use wrong context would be detected and stopped by the compiler, anyway.

P.S. The while thing relies on the fact that correctness checking for the const expression and detection of errors happens not when you define some generic function, but when it's instantiated. And it's obvious why it have to happen like this: correctness checking of something likes const { assert!(…); } couldn't happen during definition time because what goes into assert!(…); may define on constants that ass passed as arguments. And doing that in runtime wouldn't make any sense: this would mean that your constant value would, suddenly, depend on something in runtime…

Too complex for my small brain

Posted Nov 11, 2024 16:18 UTC (Mon) by daroc (editor, #160859) [Link] (3 responses)

Thank you for the explanation; I see what I was missing. I didn't understand that the assertions in the const block were actually being evaluated at compile time — with that understanding, the scheme makes more sense.

It seems like the big difference between the two schemes is really just about the use of a partial order [1] versus a total order. Fuchsia's scheme also involves carrying around a zero-sized context that gets compared to see whether one element is less than another element at compile time — the difference is that the comparison has to be encoded into traits, because Rust doesn't support using custom types as const generics (yet).

I personally think that there are reasons to prefer a partial order, but that probably a scheme that used const generics to pass around a type with a custom PartialOrd instance would be the best of both worlds.

[1]: https://en.wikipedia.org/wiki/Partially_ordered_set#Parti...

Too complex for my small brain

Posted Nov 11, 2024 17:24 UTC (Mon) by khim (subscriber, #9252) [Link] (2 responses)

> It seems like the big difference between the two schemes is really just about the use of a partial order versus a total order.

That's actually pretty minor issue. const functions can do arbitrary calculations, if you are worried that locks would be accepted in some random arbitrary order while you want the compile-time error (and explicit decision by developer who would declare whether storage or network have higher priority, instead) then this can be done via simple proc-macro which would generate an appropriate table. netstack3 is defining their locks in once place with pile of macros, anyway.

As I have said: I developed this scheme to validate more complicated situation, where nether total nor partial order work. I just used simple number for illustration purposes.

And yes, we had function that accepted two enum numbers and then answered whether they are compatible. Pretty obvious modification.

> the difference is that the comparison has to be encoded into traits

The main difference is when and how it's checked, not how it's encoded. With netstack3 function declares what it accepts in it's signature… and that's it.

But how would it know? Think about function that stores something… if it would send it to USB storage, if it would save it on SSD then there would be different locks, if that file on the NFS server then you would hit yet another situation.

And you need to specify one, single markup that would cover everything. Sure, instead of simple callback you may accept another trait which embeds information about what locks that callback would need… but what if you have two callbacks? What if they could be called in a different order?

Instead of one, simple, rule you start develop elaborated schemes that may describe what you are really trying to achieve… but you are doing in a pretty limited world of traits and generics… which is total: if generic function accepts some type of callback – then it always accepts that type of callback! And if the calling function could't prove at the declaration time that it have the right to call that callback… oh, well… now we need another trait… and if we use two callbacks… we need to decide (and declare) order that we plan to use in the function declaration, too! And if order can be different… well that's another indirection! Traits are multiplying and fill your code with something that it doesn't really need, that's only there to help with lock order checking!

As I have said: this leads to insane mess… and insane compilation times.

Now, constants are validated after instantiation. That means that you simply just don't even need to care about combinations that are not happening in real code. And if you use function with a callback… callback would determine validity of your function automatically. You don't need to create an additional trait, put callback and lock in there and pass that around. Any generic function or generic callback already carries the required data, you don't need to duplicate it in separate, manually written, traits!

> I personally think that there are reasons to prefer a partial order, but that probably a scheme that used const generics to pass around a type with a custom PartialOrd instance would be the best of both worlds.

Sadly that's not possible in Rust because traits couldn't carry const functions and PartialOrd couldn't be used in const context, too.

You may create helper function, though. If your locks are defined in one place then it's trivial and if they are not defined in one place then RED/BLACK scheme can be used (it's more-or-less inveitable with netstack3 approach, too, as was already discussed).

P.S. One mode that's trivial with const-checking and which is, essentially, impossible to implement (without additional const-coding, heh) with netstack3 approach: “just make me happy”. I have 2 (3, 4, 5, N) locks… how can I lock them all without causing the deadlock. How would you implement that? That's powerful capability, but it's not compatible with “just leave this machinery out” compile-speedup enhancement (where you just map everything to () and compiler accepts everything till you would compiler with checking-enabled flag)…

Too complex for my small brain

Posted Nov 11, 2024 18:18 UTC (Mon) by daroc (editor, #160859) [Link] (1 responses)

Ah, I see now. I can't speak for anyone else, but you've managed to convince me.

As for why the Fuchsia team went with a trait-based approach — your comment about when panics in const blocks stabilized raises another theory: netstack3 was started in 2019. I'm not sure exactly when the LockBefore solution was added, but it could have been before the approach you described became possible.

Too complex for my small brain

Posted Nov 11, 2024 19:21 UTC (Mon) by khim (subscriber, #9252) [Link]

> I'm not sure exactly when the LockBefore solution was added, but it could have been before the approach you described became possible.

I think that was possible even in 2019, but it sure looked awfully ugly back then.

Panic in const were always possible (just divide by zero and you get panic at compile time) and, indeed, in Rust 1.51 everything already works. But without const generics all these checks would have to be based on One and Zero types and associated consts in traits and that looks pretty awful. When Rust 1.57 stabilized panics it became easier and with Rust 1.79 the need for strange dance with extra struct and const implementation for it is no longer needed.

I don't remember where I discovered the use of const for invariants checking (I think it was on URLO, but could be reddit, too), but I remember that current version of Rust was 1.5x, which meant that I don't need to do crazy dance with types to encode number, can just simply use const generics, but still needed to use “divide by zero” trick. Discussion about the fact that panic in const would be stabilized “soon”, that I vaguely recall, means that was middle of 2021.

Now suspect that without const generics with the need to encode numbers in One and Zero types and then use pile associated constants for calculations it was just too ugly to really contemplate and one needed a leap of faith to believe that eventually it all would become much simpler and more usable to bet an implementation on that.

But for me, since I knew about that approach essentially from my first attempt to help some of my friends to write their first large Rust program… it just feels much more natural than traits-based checks. Closer to how one would do that in C++17 and thus more… “traditional”, I would say.

I guess is never occurred to me that it may be very obscure technique that is not known to everyone.


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds