Not-a-GPU accelerator drivers cross the line

The GPU-driver rule is the result of a "line in the sand" drawn by direct-rendering (DRM) maintainer Dave Airlie in 2010. The kernel side of most GPU drivers is a simple conduit between user space and the device; it implements something similar to a network connection. The real complexity of these drivers is in the user-space component, which uses the kernel-provided channel to control the GPU via a (usually) proprietary protocol. The DRM maintainers have long taken the position that, without a working user-space implementation, they are unable to judge, maintain, or test the kernel portion of the driver. They have held firm for over a decade now, and feel that this policy is an important part of the progress that this subsystem has made over that time.

At its core, a GPU is an accelerator that is optimized to perform certain types of processing much more quickly than even the fastest CPU can. Graphics was the first domain in which these accelerators found widespread use, but it is certainly not the last. More recently, there has been a developing market in accelerators intended to perform machine-learning tasks; one of those, the Habana Gaudi, is supported by the Linux kernel.

The merging of the Gaudi driver has raised a number of questions about how non-GPU accelerators should be handled. This driver did not go through the DRM tree and was not held to that subsystem's rules; it went into the mainline kernel while lacking the accompanying user-space piece. That was later rectified (mostly — see below), but the DRM developers were unhappy about a process that, they felt, bypassed the rules they had spent years defending. Just over one year ago, the arrival of a couple of other accelerator drivers spurred a discussion on whether those drivers should be treated like GPUs or not; no clear conclusions resulted.

The Habana driver has been the source of a few similar discussions over the last few months, with bursts in late June and early July. The problem now is an expansion of that driver's capabilities that requires using the kernel's DMA-BUF and P2PDMA subsystems to move data between devices. These subsystems were developed to work with GPU drivers and are clearly seen by some DRM developers as being part of the kernel's GPU API; drivers using them should, by this reasoning, be subject to the GPU subsystem's merging rules. Or, as Airlie phrased it in his objection to merging the Gaudi changes:

NAK for adding dma-buf or p2p support to this driver in the upstream kernel. There needs to be a hard line between "I-can't-believe-its-not-a-drm-driver" drivers which bypass our userspace requirements, and I consider this the line.

This driver was merged into misc on the grounds it wasn't really a drm/gpu driver and so didn't have to accept our userspace rules.

Adding dma-buf/p2p support to this driver is showing it really fits the gpu driver model and should be under the drivers/gpu rules since what are most GPUs except accelerators.

The interesting twist here, as acknowledged by DRM developer Daniel Vetter, is that there is, indeed, a free user-space implementation of the Gaudi driver. What is still not available is the compiler used to generate the instruction streams that actually drive this device. Without the compiler, Vetter said, the available code is "still useless if you want to actually hack on the driver stack". He elaborated further:

Can I use the hw how it's intended to be used without it?

If the answer is no, then essentially what you're doing with your upstream driver is getting all the benefits of an upstream driver, while upstream gets nothing. We can't use your stack, not as-is. Sure we can use the queue, but we can't actually submit anything interesting.

Over the course of the discussions, the DRM developers have tried to make it clear that they want a working, free implementation of the user-space side. It does not have to be the code that is shipped to customers, as long as it is sufficient to understand how the driver as a whole works. To some, though, the compiler requirement stretches things a bit far. Habana developer Oded Gabbay has described the DRM subsystem's requirements this way:

I do think the dri-devel merge criteria is very extreme, and effectively drives-out many AI accelerator companies that want to contribute to the kernel but can't/won't open their software IP and patents.

I think the expectation from AI startups (who are 90% of the deep learning field) to cooperate outside of company boundaries is not realistic, especially on the user-side, where the real IP of the company resides.

Cooperating outside of company boundaries is, of course, at the core of the Linux kernel development process. The DRM subsystem is not alone in making such requirements; Vetter responded by pointing out, among other things, that the kernel community will not accept a new CPU architecture without a working, free compiler.

Over the years, there has been no shortage of problems with vendors that want their hardware to work with Linux while keeping their "intellectual property" to themselves. This barrier has been overcome many times, resulting in wider and better hardware support in the kernel we all use. Getting there has required at least two things: demand from customers for free drivers and a strong position in the development community against proprietary drivers. The demand side must develop on it own (and often does), but the kernel community has worked hard to maintain and communicate a unified position on driver code; consider, for example, the position statement published in 2008. As a result there is a consensus in the community covering a number of areas relevant to proprietary drivers; one will have to work hard to find voices in favor of exporting symbols to benefit such drivers, for example.

This ongoing series of discussions makes it clear that the kernel community has not yet reached a consensus when it comes to the requirements for drivers for accelerator devices. That creates a situation where code that is subject to one set of rules if merged via the DRM subsystem can avoid those rules by taking another path into the kernel. That, of course, will make it hard for the rules to stand at all. Concern about this prospect extends beyond the DRM community; media developer Laurent Pinchart wrote:

I can't emphasize strongly enough how much effort it took to start getting vendors on board, and the situation is still fragile at best. If we now send a message that all of this can be bypassed by merging code that ignores all rules in drivers/misc/, it would be ten years of completely wasted work.

Avoiding that outcome will require getting kernel developers and (especially) subsystem maintainers to come to some sort of agreement — always a challenging task.

In the case of the Gaudi driver, Greg Kroah-Hartman replied that he had pulled the controversial code into his tree. In response to the subsequent objections, he dropped that work and promised to "write more" once time allows. Dropping the patches for now helps to calm the situation, but it has not resolved the underlying disagreement. At some point, the kernel community will have to reach some sort of conclusion regarding its rules for accelerator drivers. Failing that, we are likely to see a steady stream of not-a-GPU drivers finding their way into the kernel — and a lot of unhappiness in their wake.