LWN: Comments on "Free user space for non-graphics drivers"

Free user space for non-graphics drivers

mathstuf — Thu, 11 Jun 2020 18:15:24 +0000

> I'm not certain what applications people are running in Docker Windows containers

CI processes :) . It's really nice to have a known-clean state to start from for CI. Without the unbearable slowdown/latency VMs tend to have.

Free user space for non-graphics drivers

nye — Thu, 11 Jun 2020 16:58:22 +0000

> One of the major use cases seems to be CUDA (e.g. for developing and testing CUDA applications on Windows with WSL, before deploying to a Linux cloud), which is only supported on Linux by the proprietary NVIDIA drivers, so Mesa is irrelevant there. For 3D graphics, I assume there's normally some tight integration between user-mode and kernel-mode drivers, and the real kernel-mode driver (in the Windows kernel) is going to be the proprietary NVIDIA/AMD/Intel one, so you'll need the matching proprietary user-mode driver on Linux - you wouldn't be able to use Nouveau with it even if they all supported DRM.

> D3D12 support on WSL doesn't sound directly useful for applications, because WSL seems like a development platform rather than a deployment platform

The initial motivation for WSL appears to be just to turn Windows into a viable development platform (very successfully IMO), but I rather suspect that this specific feature is intended primarily for running real production workloads in Docker Linux containers on the same servers that are running Docker Windows containers, and with minimal (plausibly really could be negligible) performance impact.

I'm not certain what applications people are running in Docker Windows containers, or why you would actually want to do that, but it's definitely a thing so clearly there are some use cases out there.

Free user space for non-graphics drivers

marcH — Tue, 09 Jun 2020 20:22:08 +0000

Has such dummy user space code already been released? I guess it could still be useful for people using kernel code not merged by Greg KH. I heard rumours that not everyone uses "pure upstream" kernel code all the time (unbelievable, I know)

Maybe that limited but non-zero experience could even provide some useful lessons for all parties involved, who knows.

Free user space for non-graphics drivers

pabs — Tue, 09 Jun 2020 03:18:50 +0000

Hmm, I can't find anything that uses as10k1. It seems instead the linux.git sound/pci/emu10k1/emufx.c driver has a couple of functions called by the OP/A_OP defines that use instruction set defines in the include/uapi/sound/emu10k1.h header.

Free user space for non-graphics drivers

pizza — Tue, 09 Jun 2020 02:40:07 +0000

> https://perso.m-labs.hk/sb/prism54/freemac.html

I'm not sure this really belongs on the list, as they don't know how to control anything beyond the basic ARM9 core and other off-the-shelf peripherals (eg PCI and USB cores). Hardware timers and the radio itself are complete unknowns, So while technically there is "open firmware" it's good for little more than blinking LEDs.

(In a former life, I had access to the firmware source code and low-level hardware documentation for the entire p54 "Arm MAC" family..)

> I wasn't able to find anything about the EMU10K1 work you mentioned, only about the GPLed but sourceless blobs (sigh) in the alsa-firmware git repository

That firmware appears to be FPGA bitstreams for certain pro-grade E-MU hardware, and isn't needed for the prosumer SBLive/Audigy cards built on the EMU10K1/10K2 chips.

(IIRC, if you plug in a SBLIve/Audigy card, it does not rely on any external firmware, though the driver uploads little DSP snippets to implement driver-exposed functionality such as multichannel mixing and other such things. Those DSP programs were written using the 'as10k1' assembler, users could upload arbitrary programs if they so chose..)

Free user space for non-graphics drivers

pabs — Tue, 09 Jun 2020 02:09:51 +0000

openfwwf was already on the wiki, along with Prism54 FreeMAC, and the carl9170fw and open-ath9k-htc-firmware projects released by Atheros.

http://netweb.ing.unibs.it/~openfwwf/
https://perso.m-labs.hk/sb/prism54/freemac.html
https://perso.m-labs.hk/sb/prism54/snapshots/prism54_free...
https://github.com/qca/open-ath9k-htc-firmware
https://github.com/chunkeey/carl9170fw

I added fx2lib (and sigrok fx2lafw) to the wiki.

https://github.com/djmuhlestein/fx2lib/
https://sigrok.org/wiki/Fx2lafw

I wasn't able to find anything about the EMU10K1 work you mentioned, only about the GPLed but sourceless blobs (sigh) in the alsa-firmware git repository:

https://github.com/alsa-project/alsa-firmware/tree/master...

If you have any links to the EMU10K1 work or the others, that would be welcome.

Free user space for non-graphics drivers

mfuzzey — Mon, 08 Jun 2020 19:43:22 +0000

> I suspect the kernel <-> GPU interfaces are generally much more complex than the interfaces between the kernel and stuff in /lib/firmware, hence the reluctance.

I think it's also a question of exposure of the interface to an attacker.

On the general /lib/firmware side the issue is more potentially malicious firmware.

Unpriveleged users can't (or shouldn't) be able to change the firmware files. Most of the time the firmware won't be receiving commands directly from userspace without the kernel sanitizing them first. In fact normally the uapi will be something standard on Linux for that subsystem and the commands sent to the firmware will be completely determined by the kernel driver.

On the GPU side though it's a bit different as unpriveleged applications can submit commands to the GPU via /dev/dri (otherwise stuff like games wouldn't work). The kernel is mostly a pipe. So a malicious actor who has knowledge of a GPU exploit could submit crafted buffers to compromise the system.

The kernel can't assume that userspace is Mesa or something trusted.

But the existence of an open source userspace, even if it is bypassed by an attacker, means that the knowledge is available for the kernel to sanitize the commands. Up to a point at least because there could still exist undiscovered holes. But it's way better than an open source kernel driver accepting opaque command buffers with no idea what they do or the security implications.

Free user space for non-graphics drivers

marcH — Mon, 08 Jun 2020 19:05:57 +0000

> Normally it will be loaded onto some other device so the risk depends on the level of hardware access that device has to memory and the degree to which Linux can restrict it (by an IOMMU for example).

IOMMU is apparently hit-and-miss: https://lwn.net/Articles/782381/

> On the other hand firmare for hardware with bus mastering DMA capabilities not restricted by an IOMMU basically has full system access.

Thanks, so let's stop pretending the kernel cares about attack vectors from co-processors when it doesn't even log the filenames loaded from /lib/firmware/ by default [*]

These attack vectors and pseudo-security concerns being out of the picture, that leaves only the maintenance (troubleshooting and refactoring) questions. I suspect the kernel <-> GPU interfaces are generally much more complex than the interfaces between the kernel and stuff in /lib/firmware, hence the reluctance. Is
that higher GPU complexity why there is a "line in the sand" somewhere in the middle, do I get that right? Or because /lib/firmware has been causing fewer maintenance headaches in the past? Or both.

[*] As the distributed system bootloader that it is, in an ideal world request_firmware() should check signatures. I digress.

Free user space for non-graphics drivers

nybble41 — Mon, 08 Jun 2020 16:19:54 +0000

Yes, TCP/IP works from WSL1 so you can use GUI apps with Cygwin/X if you're OK with no hardware acceleration or even XSHM. Presumably the focus of the new development is on making it possible to accelerate the rendering of GUIs within WSL and efficiently composite the results onto the Windows desktop.

Free user space for non-graphics drivers

pizza — Mon, 08 Jun 2020 14:26:25 +0000

Offhand I can only think of three that saw success:

* Broadcom wifi f(openfwwf)
* Cypress FX/FX2 (fx2lib)
* EMU10K1 (Sound Blaster Live/Audigy)

But there are others out there in varying stages of functionality.

Free user space for non-graphics drivers

pabs — Mon, 08 Jun 2020 12:42:02 +0000

Do you have some examples of the projects you mentioned?

Free user space for non-graphics drivers

pizza — Mon, 08 Jun 2020 12:40:19 +0000

As a general response, nobody is working on reverse-engineering the stuff in /lib/firmware, but there are some folks working on specific devices with varying degrees of success.

The thing is, each device has to be independently reverse-engineered, a process so involved that by the time the effort yields something usable, the device will have been obsolete for a decade.

(At least these days the embedded CPUs are usually an ARM variant, or some other licensed/documented core...)

Free user space for non-graphics drivers

pabs — Mon, 08 Jun 2020 02:32:48 +0000

Is anyone working on reverse engineering /lib/firmware? There is precious little open firmware out there and it is mainly for increasingly obsolete hardware.

https://wiki.debian.org/Firmware/Open

Free user space for non-graphics drivers

mfuzzey — Sun, 07 Jun 2020 22:11:43 +0000

>I'm not sure how that compares from a security perspective to loading a dozen /lib/firmwares

That depends what is done with the stuff loaded from lib/firmware.

Normally it will be loaded onto some other device so the risk depends on the level of hardware access that device has to memory and the degree to which Linux can restrict it (by an IOMMU for example).

So for example firmware loaded into an I2C connected touchscreen controller is unlikely to be able to do much harm since that bus provides no DMA capacity, worst case it could simulate clicks to enter false data (assuming it knows what is on the screen).

On the other hand firmare for hardware with bus mastering DMA capabilities not restricted by an IOMMU basically has full system access.

Free user space for non-graphics drivers

marcH — Sun, 07 Jun 2020 21:49:00 +0000

> loading a dozen /lib/firmwares without even a debug statement in the firmware loader showing every file loaded.

Of course /lib/firmware doesn't include a number of other closed-source bits loaded even before the kernel (generally because they play a role in... booting the kernel).

https://www.ptsecurity.com/ww-en/analytics/disabling-inte...

Now the security of these is different because they can't be unsigned code, unlike most stuff in /lib/firmware I guess or the GPU/AI workloads mentioned above. But they're not open-source either.

Free user space for non-graphics drivers

marcH — Sun, 07 Jun 2020 08:26:40 +0000

> These compute/gpu things run touring complete code with hw security features you have to take on faith, ...

So this is basically Asymetric Multi Processing (CPU and [GP]GPU) with one side running open-source (Linux) and the other side running random closed source code with little or no security boundary between the two processors. Correct?

To be honest I'm not sure how that compares from a security perspective to loading a dozen /lib/firmwares without even a debug statement in the firmware loader showing every file loaded. Which is worse? There are has been for a long time more than two processors running closed source in any modern [D]SoC; how much isolation between them? Can the kernel even tell?

[D] Distributed

Free user space for non-graphics drivers

marcH — Sun, 07 Jun 2020 08:10:28 +0000

> There is work going on to support Linux GUI applications on WSL, but that would be done using the Remote Desktop Protocol (RDP) and a Wayland compositor;

BTW Xming worked with WSL1 last time I checked (< 1 year ago). Just in case this sentence gave someone a wrong impression.

Free user space for non-graphics drivers

jiiksteri — Sat, 06 Jun 2020 19:47:39 +0000

> Couldn't you create a subdirectory for "vendorpipedrivers", and it's made abundantly clear that anything in that directory is (a) the vendor's problem, and (b) will not be enabled by default.

People will ignore such abundant clarity and enable it anyway. And if _anything_ breaks they will report the breakage to their distribution / upstream, only to be asked to reproduce the problem without the offending driver. Which they can't, and since nobody apart from the vendor can debug the problem, time is wasted and nobody's better off after the exercise.

Isn't this what already happens / happened with the binary nvidia drivers and those weren't even shipped with the kernel :)

Free user space for non-graphics drivers

Wol — Sat, 06 Jun 2020 18:51:57 +0000

Couldn't you create a subdirectory for "vendorpipedrivers", and it's made abundantly clear that anything in that directory is (a) the vendor's problem, and (b) will not be enabled by default.

Hopefully stuff will be upgraded as kernels change around it, but it can be freely redistributable GPL code, just not the kernel dev's problem - it's down to users and vendors.

Or could that be setting a dangerous precedent ...

Cheers,
Wol

Free user space for non-graphics drivers

Cyberax — Fri, 05 Jun 2020 19:43:45 +0000

> That way the linux guest could use existing Mesa userspace drivers
That's not how DRM works. DRM basically is a "dumb pipe" for commands from userspace to the end device. So you still need a Mesa driver for that device.

Free user space for non-graphics drivers

excors — Fri, 05 Jun 2020 14:31:50 +0000

Using DRM sounds like a lot of extra work for little benefit. The user-mode components which are being ported from Windows-only to cross-platform would have to support both DRM and D3DKMT kernel APIs, which I guess will have a lot of subtle differences and probably some major differences as they evolve and new GPU features are supported in one but not the other; then the Windows side would need to translate DRM back into D3DKMT. By using D3DKMT throughout, it's a simpler and much less bug-prone task of just proxying commands between the two kernels and recompiling the existing user-mode components for Linux with minimal API changes.

One of the major use cases seems to be CUDA (e.g. for developing and testing CUDA applications on Windows with WSL, before deploying to a Linux cloud), which is only supported on Linux by the proprietary NVIDIA drivers, so Mesa is irrelevant there. For 3D graphics, I assume there's normally some tight integration between user-mode and kernel-mode drivers, and the real kernel-mode driver (in the Windows kernel) is going to be the proprietary NVIDIA/AMD/Intel one, so you'll need the matching proprietary user-mode driver on Linux - you wouldn't be able to use Nouveau with it even if they all supported DRM.

D3D12 support on WSL doesn't sound directly useful for applications, because WSL seems like a development platform rather than a deployment platform, so games on Windows will use the native D3D12 API and games on native Linux can't use D3D12 at all (or at least Microsoft hasn't announced any plans for that yet). But D3D12 support means Microsoft can port their OpenGL-to-D3D12 translation layer (https://devblogs.microsoft.com/directx/in-the-works-openc...) from Windows to Linux with little extra effort, meaning OpenGL-based applications can be developed on WSL then deployed on Linux. (Apparently that translation layer does use the higher-level parts of Mesa, with a driver that talks to libd3d12 instead of libdrm.)

Free user space for non-graphics drivers

blackwood — Fri, 05 Jun 2020 13:39:43 +0000

The main use-case is to run the nvidia blob for CUDA and that stuff. And to run that they dig a direct path from the windows kernel gpu driver to linux, but the code that runs in linux isn't a linux driver at all, it's a windows driver. Simply recompiled so it's an ELF .so instead of PE .dll.

Even the mesa gl driver that could run on top of this is the brand new windows gl mesa driver that runs on top of directx. So really this is all about running the windows graphics stack, in linux, on top of windows. It'll never run on bare metal, and it won't look like a linux graphics stack. Hence why the wayland compositor is completely custom, and will use RDP to shovel the buffers over the wire to windows. Essentially microsoft is working on a wayland compositor for windows, using the windows directX stack and windows paravirt RDP desktop integration stuff, but then accidentally run that on linux. But still a windows graphics stack (just now also with wayland protocol support) end to end.

Whether drivers/hyperv wants that or not in upstream is entirely different question.

Free user space for non-graphics drivers

mfuzzey — Fri, 05 Jun 2020 07:28:03 +0000

Sure but in that case why implement /dev/drx at all?

Microsoft could, if the aim were GPU support in Linux guests running on a Windows host (though apparently it's not about that but just compute for the moment) implement a real Linux DRM kernel driver that talks to the Windows host kernel instead of hardware directly.

That way the linux guest could use existing Mesa userspace drivers

Free user space for non-graphics drivers

tzafrir — Fri, 05 Jun 2020 06:37:54 +0000

Because Microsoft DRM would be even more confusing. See comment above: https://lwn.net/Articles/822207/

Free user space for non-graphics drivers

flussence — Fri, 05 Jun 2020 04:54:54 +0000

I just can't see a use case for a proprietary DirectX API inside a Linux VM running on a Windows 10 box. We already have Vulkan, OpenGL 4.6, OpenCL, CUDA, and an *open* D3D stack in Mesa and Wine that regularly outperforms the one microserfs play with their toys on.

Why would anyone waste the effort to write apps for a frankensteined-together mishmash of OSes that exists at the sole whim of a company infamous for its architectural ADHD? This'll be quietly left to bitrot in 3 years and all five downstream users will end up with a hard dep on Ubuntu 20.04 long after it's out of support.

Free user space for non-graphics drivers

blackwood — Thu, 04 Jun 2020 21:26:09 +0000

Even the hw spec doesn't give you security, the hw might not live up to it, see smeltdown.

What an open stack allows you is to fix up the mess once you are aware of a security hole. Worst case you might end up with something like CONFIG_ENABLE_ROOTHOLES to not break existing uapi if the old one is unfixable. But going forward you can change both kernel and userspace driver parts to come up with something to work around the security holes. But often the breakage is really minor (like more gpu hangs in some corner case), so good enough if you just ship bugfixes on both sides. If one part is an opaque blob you can't really reason about, much less change, then that's all impossible. And vendors abandon their closed source stacks real fast, better if they sell you new hw with new drivers.

Free user space for non-graphics drivers

kpfleming — Thu, 04 Jun 2020 17:41:18 +0000

Since nothing else in tree interacts with this driver, and that situation seems unlikely to change in the foreseeable future, requiring users to obtain the driver separately and install it using DKMS seems completely reasonable to me.

Free user space for non-graphics drivers

nivedita76 — Thu, 04 Jun 2020 15:48:58 +0000

I'm not really close to this stuff, but I don't see how even getting open-source userspace is really enough for security. That tells you if the vendor-provided user-space library can be compromised, but you really need to get specifications for what the hardware is capable of doing, not just what the vendor-provided userspace uses it for, since a real attacker could do something completely different with the provided ioctl's.

Free user space for non-graphics drivers

brigdh — Thu, 04 Jun 2020 13:44:00 +0000

I proposed that, and was rejected. GregKH wants the actual library we use. I'm running it up the chain to figure out what we can do, but I expect it to take time.

Free user space for non-graphics drivers

daniels — Thu, 04 Jun 2020 10:22:48 +0000

The Microsoft /dev/dxg interface is not really a 'DirectX driver' in the way that most people would interpret it. It is a direct port of their Windows DirectX kernel interfaces, but in analogue to Linux, this is DRM rather than OpenGL/Vulkan. Those interfaces allow you to allocate GPU hardware buffers (as in DRM), submit pre-compiled hardware bytecode shaders for execution (as in DRM), and synchronise against that execution (also as in DRM).

It doesn't stream DirectX client API commands over the device boundary (as VirGL does for GL/GLES). WSL2 will use a proprietary libdirectx.so, and accompanying hardware-vendor drivers, to implement the DirectX graphics API, translate that into hardware-specific commands, and then submit those commands to the kernel interface.

Pushing the comparison further, it would be entirely possible to port any of the Mesa hardware drivers - say Intel's ANV or AMD's RADV Vulkan drivers, or any of the Gallium OpenGL/GLES drivers - to run on top of /dev/dxg instead of on top of DRM kernel support.

Free user space for non-graphics drivers

blackwood — Wed, 03 Jun 2020 22:08:32 +0000

Well this is why drivers/gpu requires fully open stack in the userspace side of things. Allows us to review the entire thing, figure out whether the security model has a reasonable chance of working. Backstop it with kernel-side validation if necessary, which has huge uapi impact since you can't validate Turing complete stuff, so you need to figure out a way to limit the Turing completeness to where it doesn't matter for security, without impacting the functionality exposed to userspace in a way that users care about.

Plus when (this isn't an if in a post smeltdown world) the hw security features turn out to be more leaky than presumed, we have a fighting chance to fix up the mess going forward.

But without all that it's indeed hopeless, and you might as well assume your kernel is compromised.

Free user space for non-graphics drivers

roc — Wed, 03 Jun 2020 21:49:06 +0000

Too bad. I guess that means when one of these drivers is accessible you may as well assume complete kernel compromise.

Being able to bound the user-space-visible effects of each system call seems like a good place to draw a line in the sand.

Free user space for non-graphics drivers

blackwood — Wed, 03 Jun 2020 21:44:55 +0000

Not even close, which is the entire problem. These compute/gpu things run touring complete code with hw security features you have to take on faith, and between the ioctl stuff and the actual things you run there's generally a few layers of abstraction. E.g. on most gpus you just tell the kernel to run a buffer, which contains a cmdlist, which contains state packets, which contains pointers to more stuff, most of these containing the data, but some of it also containing the actual shaders/kernels you run on the gpu/compute hw. And this entire hairball is directly execute by hw.

Reverse-engineering this with humans and specialized fuzzing/discover tools takes a few years. Forget doing any kind of security audit without that work being done, or having proper access to both the runtime and all the compilers in userspace. Ideally also hw specs.

And all the kernel<->userspace uapi is entirely custom to the hw, and the most leaky abstraction you can think of. Stuff like which automatic clock gating bits the kernel sets impacts which instruction scheduling workarounds the compiler needs to apply to prevent the entire thing from keeling over and crashing the compute workloads userspace submits.

Free user space for non-graphics drivers

iabervon — Wed, 03 Jun 2020 21:28:28 +0000

It seems like Qualcomm ought to release open source user space code that programs the DMA engine and does some useless example computation. That wouldn't let people do deep learning techniques without using Qualcomm's proprietary code, but would let people verify that the driver and device are harmless.

Free user space for non-graphics drivers

roc — Wed, 03 Jun 2020 21:05:10 +0000

I hope these drivers at least expose enough information to know exactly which userspace memory is read or written by each ioctl (and in between ioctls as well). That information is essential for fuzzing, and also for rr.