Progress in wrangling the Python C API

(I mean, yes, it is a little more complicated than that. Python actually tells you all sorts of metadata about how the buffer is laid out in memory, whether it is multi-dimensional, etc., and can even hand out non-contiguous buffers. But in terms of what you are allowed to do with it, it's pretty much just "a pointer and a writable flag.")

In general, Python's attitude is that, if you want to fiddle with the buffer protocol, you should be doing it as an implementation detail of some object that you entirely control, or else you should be using some kind of explicit locking which is not managed by the interpreter. But Python is also totally fine with a library writing "lol this is not thread-safe, don't even try" in their documentation and calling it a day. And frankly, that insouciance cannot be made compatible with "let's have a sound Rust wrapper type that can safely interact with arbitrary Python buffer protocol objects." Python does not provide the guarantees that Rust wants, nor does Python provide a standardized mechanism for Rust to ask for such guarantees.

What I think needs to happen is one (or more) of the following:

* Rust code may only use the buffer protocol to interact with objects that it owns or mutably borrows (in Rust terms) and that have a (Python) refcount of 1 (or which have not returned from their Python constructors yet). Additionally, the object must not be compatible with (Python) weak references (so that the refcount is correct). This combination of requirements is very limiting. I'm not sure how easy it is to prove all of these requirements are satisfied at compile time, but you could dynamically check them, so this could be used to build a safe primitive out of unsafe code.
* Python starts offering flags that indicate a buffer is const or exclusive. const is a stronger condition than read-only, because it means the object won't change of its own accord (or by some means other than the buffer protocol). But const can probably only be offered for objects that Python considers immutable, and exclusive raises questions of how the locking is supposed to work. What happens if a legacy client asks for a (writable) buffer when the new API considers the request unacceptable? Does it just get a BufferError? That probably breaks something, but there's no obvious alternative behavior.
* Python requires only one client to have mutable access to a given buffer at a given time. It is technically possible to do this without breaking ABI compatibility, because buffer accessors are already required to call PyBuffer_Release when they are done with the buffer (so you can keep track of who is reading or writing the buffer at any given time). However, this would be semantically incompatible with the existing behavior and would likely result in problems anyway. For example, there's currently nothing wrong with a client eagerly acquiring all of its buffers up-front, figuring out which ones alias, and then managing those aliases with a separate set of locks on the C side, without informing Python of the locks at all. If you added a lock to the buffer protocol, then that client would break because it would be unable to acquire the same buffer more than once. It is, frankly, unreasonable to break that client just to make things easier for Rust - they used the API as advertised, they're not doing anything particularly sketchy or unusual, and there's currently no way to inform Python of the locks even if you want to (nor does the current version of Python have the slightest idea what to do with that information).
* Python offers an advisory locking mechanism for the buffer protocol. This doesn't really solve anything because clients can ignore it, but maybe you deprecate non-locked access and try to force everybody to take locks in a future version. Single-threaded clients will probably be unimpressed by that.
* Rust treats Python buffers as equivalent to raw pointers, and stops trying to build safe wrappers for the protocol as a whole. Instead, build wrappers specific to each type you want to interact with. Unfortunately, that's probably not a whole lot easier than trying to wrap the whole protocol, but at least you can somewhat depend on the semantics of the specific C extension, which might be able to offer stronger guarantees than Python does.