A missed invariant

I suspect nor did the maintainer, because he wasn't paying attention to what I was saying, and he was stuck in his mental model of how other drivers work, and that's why he kept rejecting my ideas.

> The driver may not be torn while I the hardware is doing something

This is obvious. However, the scheduler is not the driver and in my driver there is not one global scheduler as there is in most drivers. In my driver, a scheduler is bound to a graphics queue and gets torn down when the queue is destroyed, for example when you kill the process that owns it. There are many schedulers running in parallel. This is because the actual global scheduler is implemented in GPU firmware and the DRM schedulers are just one layer on top of the firmware's scheduling primitives, used for dependency control and to wait for hardware resources (not to actually schedule across multiple queues since each only handles one). As strange as this might sound at first glance, this is how firmware-scheduled GPUs are supposed to be implemented in DRM, and I had long conversations with DRM people about this before jumping into the implementation, so the concept is sound.

Therefore schedulers are created and destroyed constantly, and it is not acceptable to block scheduler destruction on unnecessary things, since that would create the need for a bunch of extra tracking that would not otherwise be necessary.

> The schedulable entities must exist while they have state in the gpu to manage.

This is not true. It might be true in some drivers, perhaps drivers that have a better match between the drm_sched design and the GPU design, perhaps even the AMD driver (it is not a coincidence that the drm_sched maintainer is an AMD employee, that code came from the AMD driver).

But it is not true in my driver. My driver has to match the firmware API (which Apple defines and in fact isn't even stable and changes across versions and GPU generations) to DRM's model and the UAPI that I defined, which in turn is what the userspace code speaks to. As such, the driver's code consists of lower layers that manage firmware resources such as its idea of command queues, and higher layers that aggregate them into UAPI objects such as the UAPI command queue. In fact, in this driver, a single UAPI command queue maps to up to three firmware command queues (compute, vertex, fragment).

Since it is critical that firmware resources are always kept alive as long as is necessary (otherwise the whole GPU firmware crashes irrecoverably), the underlying firmware job objects are what hold references to the resources they need (and so on for resources that depend on others). These job objects are only cleaned up when the firmware itself signals job completion (successful or not).

This is perhaps very different from the typical C GPU driver design. C encourages you to have big objects and top-down locking and ownership management. That sort-of works but runs into all kinds of issues where everything needs to wait for everything else for cleanup, and you can easily end up with ordering issues.

Rust instead encourages fine-grained reference counting and fine-grained division of responsibility. The outcome of doing things this way is very positive, because it means that I can use Rust lifetimes to track firmware object lifetimes and ensure they meet the expectations of the firmware itself. This is in fact *the* reason why I chose Rust for this driver, because I knew getting all this right in C (and I have to get it right for the driver to not crash the whole system all the time) would be a nightmare.

Put another way, I do not trust drm_sched or any high-level code to be the ultimate decision maker of the lifetime of GPU resources. The underlying resources "track themselves" instead, and that works a lot better.

And so, when a process dies or destroys a GPU queue, I really don't care if there are jobs in flight. I just want to tear down the DRM scheduler layer and its concept of jobs. The underlying firmware resources will continue to run (there is no "cancel" command I can use to abort them, macOS never does that either), and will continue to hold references to any needed resources such as the associated GPU VM page tables, firmware command queues, kernel-allocated temporary GPU buffers, etc. until they complete, and then all the reference counts drop to zero and everything gets freed, possibly over one minute (compute jobs can be long-running) after the DRM scheduler and indeed the whole process that created this GPU job is long gone.

Not coincidentally, this is also how macOS behaves at least at the UAPI layer (you can ^C a compute app running a long job and the process dies, but the compute job continues to run to completion or failure in the firmware and kernel driver), though there are definitely big differences in our implementations. It's how the hardware is intended to be driven.

To paraphrase the Red Queen, hardware has the remarkable ability of triggering six supposedly-impossible situations before breakfast, invariably wrecking your nice clean/consistent/clever abstractions in the process.

(I have no opinion on if that applies to this particular situation...)