LWN: Comments on "ioctl() forever?"

ioctl() forever?

developer122 — Sat, 18 Jun 2022 21:33:35 +0000

There is a stunning level of hostility, regardless of licence.

ioctl() forever?

paulj — Tue, 14 Jun 2022 12:54:07 +0000

It is archived by sci-hub.se, depending on what you think of that kind of thing.

ioctl() forever?

ejr — Mon, 13 Jun 2022 21:57:04 +0000

AUGH. Drat. Sorry! I was hoping, and I didn't see how I was already logged in. There is no RTL in the book at all but rather proofs based around integers. That's kinda how I think of circuits, so I like it.

ioctl() forever?

farnz — Sun, 12 Jun 2022 16:06:19 +0000

And underlying that is that electrical safety is a mature and well-understood field. The physics involved in a GFCI or an overcurrent breaker have been well-understood since the 19th century, and the physics required for an AFCI were fully settled before World War I in 1914. There's nothing in any NEC-compliant installation that a good physicist from 1920 couldn't fully understand and explain - although they'd be amazed by the manufacturing techniques involved (and would be seriously shocked by the IC in an AFCI, even though they could explain the physical principles that underpin its operation).

In contrast, logic design (both software and hardware) is a rapidly evolving field even today - Rust's type system is based atop affine logic from the 1970s, and there's mathematics I'm aware of that's probably relevant to programming language design that was only formulated rigorously in the last 20 years, and where there's active leading-edge research trying to determine if it's useful, and if so, how.

I'd also agree on the "throw it at the wall and see if it sticks" thing - the only time, IME, that engineers actually seriously bother with the rigorous analysis that's possible for something like a GFCI is when you're cost-optimizing it. Otherwise, it's considered too much work, when you can take a design that's probably overkill but meets requirements and ship it.

ioctl() forever?

mpr22 — Sun, 12 Jun 2022 15:37:00 +0000

You were in fact directed there by an institutional subscription :(

For me, that link goes to a website run by Springer Nature, where I am informed that, as a private individual in the United Kingdom of Great Britain and Northern Ireland, I would have to pay £19.95 for a PDF of the Logical Operations chapter (pp 35-44) of ISBN 978-3-030-87181-9, £95.50 for the whole of ISBN 978-3-030-87181-9 as an eBook, or £119.99 for the whole of ISBN 978-3-030-87181-9 as a physical hardcover book (all prices inclusive of applicable VAT).

ioctl() forever?

ejr — Sun, 12 Jun 2022 15:01:36 +0000

The main problem is that the market for electrical engineers actually *designing* these things is relatively static or slowly growing. Meanwhile, companies are absolutely desperate to fill programming seats.

And if it ain't Python or Matlab/Octave, many students don't know it until maybe their third year. Maybe. If type checking is not mentally established, the core of current formal methods that include interactive proofs is going nowhere. (For the audience: Making a program compile without type errors *is* working with an interactive proof system called the compiler. The correct program is a proof of its being well-typed given the language's rules / axioms.)

I also suspect you underestimate how often "throw it at the wall and see if it sticks" applies even in engineering fields. Or more appropriately "this worked for something kinda similar, so it'll work here."

I recently learned of a wonderful text that can illustrate the kind of knowledge needed on the practical hardware side: D. M. Russinoff, Formal Verification of Floating-Point Hardware Design, https://doi.org/10.1007/978-3-030-87181-9_3 . It *appears* to be available gratis; I don't think I was redirected through an institutional subscription.

Software potentially could be simpler in some areas, but the OS/device level requires a similar level of skill in finding the right model. That's a polite way of saying the kind of infinite bike-shedding that was happening with btrfs. If some research group can hit it out of the park like the formalization of RISC-V, sure, but people with product-like deadlines don't have that time. (I've been on both sides.)

ioctl() forever?

farnz — Sun, 12 Jun 2022 09:01:14 +0000

The NEC (and its international equivalents) are guides to safe installation, not to electrical engineering. It says that if you purchase products that have been engineered to a suitable standard, and then install them following this guide, you will be safe enough.

The components you install when you buy the things that the NEC requires are engineered with things like Maxwell's laws as guidance - a GFCI or an AFCI or an ELCB or an overcurrent breaker, or even a simple switch is not just empirically designed to work as specified, but rather designed around the known laws of physics and then tested against the specification.

It's just that we need the NEC rules because the possible range of components is not as wide as we'd like - and thus we need to allow for known physical issues when we install. For example, we need overcurrent breakers because wires have resistance and thus heat up when carrying current; a significant chunk of the NEC is describing the different ways you can install wires that meet a given standard, then place a breaker on the source end so that the wire cannot heat up enough to start a fire.

ioctl() forever?

Cyberax — Sun, 12 Jun 2022 01:15:01 +0000

> Electrical engineering stopped electrocuting frogs and developed Maxwell's laws.

Open the NEC and look for any mentions of Maxwell's equations. If anything, modern safety engineering is built on installing as many "speedbumps" as possible, rather than on making everything perfectly safe.

ioctl() forever?

developer122 — Sun, 12 Jun 2022 00:16:51 +0000

Such is the pervading wisdom in software.

It was yesterday I was commented that modern security practice is to layer incomplete speedbumps one by one and call it "defense in depth." This was in response to new bypass of the M1's hardware pointer checking, a "last line of defense."

None of these half-baked measures qualify as defense in depth. Anyone who calls it that is kidding themselves. They're deployed one by one with large time spans inbetween, reactively to whatever attackers currently favor. It should come as no surprise that attackers have no problem paying a low continual cost to work around each new measure, in return for continued ability to exploit hosts.

Meanwhile, we plaster over the undefined behavior that makes C/C++ static analysis impractical. We don't even talk about how CPUs work. We continue to ignore that actual exploit *theory* exists, like the Weird Machine whose wikipedia page starts: "Exploits exist empirically, but were not studied from a theoretical perspective prior to the emergence of the framework of weird machines." This makes it impossible to analyze systems holistically, and to find and shut out entire classes of exploits.

Civil engineering eventually got past "I think we can cut a small hole in this wall" and reached "lets calculate the loads and strains." Electrical engineering stopped electrocuting frogs and developed Maxwell's laws. Right now computing is still far behind all others and until formal methods and theory take over, we won't ever make statements like "this bridge will stand for 100 years."

ioctl() forever?

pm215 — Sat, 11 Jun 2022 12:26:59 +0000

If you're the kernel maintainers you absolutely can force people to write documentation -- just refuse to merge any patch that adds a new syscall or ioctl and doesn't include documentation as part of the patchset. You can set the quality bar at any level you like. If stuff slips through the code review stage without sufficient documentation, you can revert it.

In other words, allowing or not allowing undocumented new interfaces is a choice, just as allowing or not allowing changes that break userspace is a choice.

ioctl() forever?

pm215 — Sat, 11 Jun 2022 07:50:44 +0000

How does CTF help here? I read https://lwn.net/Articles/795384/ which seems to describe it as basically a more compact debug info format, which leaves me unsure how it would be useful for describing the kernel ABI in a more machine-readable way. I guess in theory you could build the whole kernel and then fish out the debug info, but that would take forever and only have the info for the specific binary that got built...

ioctl() forever?

wahern — Thu, 09 Jun 2022 23:14:22 +0000

With CTF symbols you could easily write a simple function to recursively copy a data structure into kernel space. So you end up in the same place, but with a fraction of the complexity and code. For C string members you can trivially implement constraint checking during the copy. Similarly, for dependent pointer and size members you can include simple qualifiers in the structure definition that the copy routine can validate. (There was a proposal, effectively implementing a simple dependent type system for this case, which came close to passing muster for the next C2x revision. It seems to have failed for lack of real-world implementation experience, but strictly speaking it should be quite simple to implement.)

Alternatively, you could implement a type-safe unpacking API. (If you only care about memory safety, you might not even need CTF.) For example, "give me the next member, which I expect to be a NUL-terminated string". This may be more or less ergonomic than the above, but either way would provide the same effective interface as generic deserializers--even "zero copy" serialization formats typically cannot be exposed as plain C data structures, even when the serialization and deserialization code is generated a la Cap'n Proto. The difference between `string_t getNamedField(object_t)` and `string_t copy_in_string(userstruct_t *, cursor_t *)` is purely syntactical when you can't completely trust the input to be well formed, which we can't.

In any event, simple intermediations, with or without CTF, provide avenues to enjoying all the same safety guarantees, without creating both kernel and userland dependencies on enormous (10-100+k SLoC) libraries. Microkernels tend to heavily rely on serialized message passing and/or RPC code generators. I'm sure they work well, but there's a steep upfront cost, especially when it comes to tooling.

Maybe even a simpler alternative is it avoid variable length strings entirely. Briefly looking at existing ioctls, it seemed like most string fields within structures are fixed-sized arrays. Variable length strings seemed more likely to be passed directly as an argument. (But please feel free to set me straight on that account as I didn't look very hard and certainly didn't write any tools to analyze the types.) To the extent the latter are a problem, they could be fixed with a very thin intermediating interface. There's something to be said for simply using fixed-sized arrays. During the heyday of writing GNU replacements for proprietary tools, there was an emphasis on removing arbitrary limitations and permitting every conceivable type of input to be variable length and unbounded; that emphasis became pervasive and reflected in even trivial, internal interfaces removed from user input. That's rarely needed, and rarely warranted given the resulting complexity, *especially* for kernel interfaces. Even file paths (or more specifically, file path arguments) have a fixed upper bound in the Linux kernel. Recently I discovered this was true even in Solaris, despite Solaris have a complicated kernel syscall facility for unbounded-length input and output syscall arguments. If you step away from trying to making everything configurable and unbounded, then it becomes much easier to limit complexity. Usually you can set an upper bound that is good enough and move on; and for many of the exceptions, you can switch to semantics that let you trade time for space (i.e. trade serial processing for discrete buffers). Or to put it another way, for the rare cases where setting fixed-bound arguments is too cumbersome, use netlink instead of ioctl. Problem solved. One doesn't need to become the other, or replaced with something fancier; just make the choice more clear.

If you want to make it easier to prevent people from accidentally doing the wrong thing, and to identify places where that might be likely, you can add type annotations to structures and other types used by kernel ioctl interfaces, complemented with a GCC module pass that identifies code that directly access members. IOW, you can implement something akin to Rust's type checker that prevents normal code from reading pointers in an unstructured manner, forcing that code to go through a "safe" API.

ioctl() forever?

Cyberax — Thu, 09 Jun 2022 19:47:55 +0000

It works fine for IOCTLs that don't need input or output parameters.

ioctl() forever?

camhusmj38 — Thu, 09 Jun 2022 19:27:55 +0000

I looked it up - the rules are documented. Which combination of nulls are allowed - if you go the overlapped (async) route you can leave the number of bytes out null and vice versa.

ioctl() forever?

mathstuf — Thu, 09 Jun 2022 19:25:15 +0000

I like how the metadata is "optional" when it probably means "nullable". I doubt calling this without those arguments at all works out that well in practice.

ioctl() forever?

camhusmj38 — Thu, 09 Jun 2022 19:02:34 +0000

I imagine they would rather eat their own hats.

ioctl() forever?

Cyberax — Thu, 09 Jun 2022 18:44:48 +0000

That's why just marshalling all the data into variable-length structures that can be simply copied into the kernel space is the right way to do it. Yes, you have some overhead duet to packing and copying, but both are pretty efficient.

Also, ioctl()s shouldn't be performance-critical anyway.

ioctl() forever?

nix — Thu, 09 Jun 2022 17:33:09 +0000

This sort of thing is why CTF was added to the toolchain :) abigail has support for it now, too... of course ioctls don't all specify which structures they manipulate, but for those that *do*...

ioctl() forever?

NYKevin — Thu, 09 Jun 2022 17:04:30 +0000

You can't force people to write documentation. If you try, you just end up with something like the following:

> int FrobnicateSprocket(Sprocket *s)
> Frobnicates a sprocket.
>
> Arguments:
> s: The sprocket to frobnicate.
>
> Returns:
> Zero if no error, or nonzero and sets errno.
>
> Errno values:
> EPERM - Frobnicating this sprocket is not permitted.

IMHO if you're going to do something like this, it should be a machine-readable enumeration of commands, not a man page.

ioctl() forever?

karim — Thu, 09 Jun 2022 16:39:00 +0000

Very good question. I can't say I thought it all the way through. Maybe another inspiration could be online help output from commands that print options when you provide "-h" ... i.e. just some free-form text to be printed on screen when invoked. It's a rough cut of an idea. Definitely requires some more forethought to be useful. But, personally, I'd love to have a way to ask about the ioctl()s available for any /dev/foo and even have a tool to invoke them from the command line if it makes sense.

ioctl() forever?

mathstuf — Thu, 09 Jun 2022 16:27:17 +0000

From `ioman(1)`, sure. But what will `man_ioctl(2)` use to communicate whatever gets converted into groff format (or is the kernel going to speak groff directly?).

ioctl() forever?

karim — Thu, 09 Jun 2022 15:54:10 +0000

man pages may have good examples. If you do a "man 2 open", for instance, it lists all possible open flags with a description. Maybe something like that?

ioctl() forever?

mathstuf — Thu, 09 Jun 2022 15:43:24 +0000

So, uh, what format will this communicate the requirements in? Snippets of C headers? ;)

ioctl() forever?

ejr — Thu, 09 Jun 2022 15:11:49 +0000

To paraphrase the bard, "That way [formal methods] lie; let me shun that."

ioctl() forever?

karim — Thu, 09 Jun 2022 15:00:39 +0000

Just on the discoverability front, it would be great if there was a way to query any ioctl() interface for the commands it accepts and the parameters expected. A sort of obligatory man_ioctl() call on any kernel party that wants to expose ioctl(). I could then do something like "$ ioman /dev/foobar" and get a list of list of parameters, etc. Maybe even gate access to register an ioctl() on having said man_ioctl() registered along with it.

ioctl() forever?

nix — Thu, 09 Jun 2022 13:37:26 +0000

Yeah, but that's neither CTF nor BTF's fault: that's the same problem with dereferencing pointers (and finding a way to do that while not allowing attackers to modify the thing and sneak bad stuff past you via TOCTTOU races, *without* incurring horrible overhead by forcing every such pointed-to thing to be CoWed on modification).

ioctl() forever?

khim — Thu, 09 Jun 2022 13:33:42 +0000

Ability to avoid clashes. Even if out-of-tree device uses the same number as another out-of-tree device they don't affect each other because, well, they are different devices.

Syscall numbers are globals, on the other hand.

ioctl() forever?

jazzy — Thu, 09 Jun 2022 11:42:38 +0000

Has someone considered copying Windows?

BOOL DeviceIoControl(
[in] HANDLE hDevice,
[in] DWORD dwIoControlCode,
[in, optional] LPVOID lpInBuffer,
[in] DWORD nInBufferSize,
[out, optional] LPVOID lpOutBuffer,
[in] DWORD nOutBufferSize,

Basically it allows for invoking an IOCTL and pass in and/or pass out a continous buffer. To me this seems extendable and avoids the need to embed pointers. It also clearly specifies what is readable and writable.

ioctl() forever?

adobriyan — Thu, 09 Jun 2022 11:38:35 +0000

And ship it with kernel: mount -t abi ... !

ioctl() forever?

jengelh — Thu, 09 Jun 2022 11:02:33 +0000

>that netlink could not be used because it depends on networking being configured into the kernel

Perhaps when someone ponders its use for a component, what they really had in mind was just the serialization format, without the AF_NETLINK socket. You could still hand in a netlink-formatted stream via ioctl.

Netlink has its own share of issues. One is the asynchronous model and the programming that this entails; you need to do two at least two calls (send/recv). Another is the 16-bit size fields in its serialization format. If you can't fit your stuff into one message because of that limit, you have to go on a multi-roundtrip endeavour from userspace. The kernel part meanwhile may need to keep extra state to logically tie the individual netlink messages together. That's kinda terrible.

Speaking of ioctl, setsockopt is just the same.

ioctl() forever?

roc — Thu, 09 Jun 2022 07:35:43 +0000

Syzkaller has an abstract description of syscalls: https://github.com/google/syzkaller/blob/master/docs/sysc...
It's fairly fuzzer-specific of course.

ioctl() forever?

roc — Thu, 09 Jun 2022 07:33:06 +0000

We really should come up with a shared description of the syscall ABI that can be used by QEMU, rr, fuzzers, ASAN, Valgrind, etc. I think we're all duplicating work.

ioctl() forever?

pm215 — Thu, 09 Jun 2022 06:55:57 +0000

I think a lot of the problem is that the kernel refuses to define its ABI in any other way than "here's a pile of header files, and if you're lucky also some documentation". This is a pain for any case except "I'm a C program". ioctl isn't the only offender here either -- setsockopt is another "feed arbitrary data structures via a generic-looking interface" mechanism.

QEMU's user-mode emulation runs into trouble with these things because we have to convert from guest architecture struct layout and endianness to host layout and endianness; without a machine-readable definition of what's being passed around by ioctl and similar syscalls we have to hand-roll support for every new ioctl somebody cares about. And every so often we run into one that's just straight-up not even documented.

ioctl() forever?

roc — Thu, 09 Jun 2022 06:39:36 +0000

Many ioctls were invented before rr or fuzzers or security mattered, and then all the other ioctls were added because that's just how things are done.

ioctl() forever?

wahern — Thu, 09 Jun 2022 05:43:56 +0000

Unix has had a standardized serialization format for decades: XDR. https://en.wikipedia.org/wiki/External_Data_Representation
An encoder/decoder has been in the Linux source tree since almost the beginning: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/...

If better serialization formats sufficiently resolved these issues then presumably ioctl would already be gone, replaced by something using XDR or one of the other dozen similar formats that have come and gone over the years. The marginal benefit of these formats over plain C data structures is manifestly not that great, at least for local IPC.

ioctl() forever?

atnot — Thu, 09 Jun 2022 01:35:22 +0000

This wasn't at all meant to be a dig at snprintf(), feel free to insert your preferred mechanism. I just don't think string formatting is something that should be regularly involved at the very core of OS APIs at all, even if C's string handling wasn't as uniquely terrible as it is. Although that fact doesn't help of course.

ioctl() forever?

NYKevin — Thu, 09 Jun 2022 00:53:32 +0000

Meh, snprintf isn't the kernel's fault, it's C's fault for being terrible at finagling strings. Stringly-typed interfaces are bad because they are difficult or impossible to validate, not because snprintf is painful.

ioctl() forever?

abatters — Wed, 08 Jun 2022 21:32:52 +0000

In case it's useful, here is a concrete example of a complex ioctl:

https://sg.danny.cz/sg/sg_v40.html

The sg driver has ioctls for sending generic SCSI commands, so it supports read-type, write-type, and bidirectional commands, direct I/O, pointers to arrays of iovecs of userspace buffers like readv/writev, pointers to buffers to contain error information, async I/O, and lots of other complex features, and it has to be high-performance. It's sort of like io_submit() and io_getevents() for SCSI commands in an ioctl().

ioctl() forever?

koverstreet — Wed, 08 Jun 2022 21:24:38 +0000

hey, I was just trying rr for my first time today!

I'm not sure how practical it'll be to solve this problem in general. My approach was to try and make ioctls more like syscalls - i.e. something that looks like a normal function call. That would make the simple cases simpler, by getting rid of ioctl structs entirely for the simple cases.

But this won't do anything for the more complex cases you brought up. The problem in those cases is just that C is a very low level language, and anything that's complex enough to handle those cases isn't going to feel natural and simple in C. This is my complaint with those advocating netlink as a wholesale replacement for ioctls - it's more complicated than it needs to be for the simple cases. But for the more complicated cases maybe it's the right approach.

There's another possibility that just occurred to me because I saw you were using it today - cap'n proto. If the problem is that defining complex data structures in a portable, ABI independent way sucks, this is exactly what cap'n proto is intended for. When I last looked at it the story for using it from C seemed incomplete, but maybe this has changed - might be worth another look.

For those unfamiliar: cap'n proto [1] is a schema language for defining ABI independent types. Crucially, unlike things like protobufs, it doesn't have pack/unpack operations - the wire format is the in memory format. It's like defining types in C using only standard sized integers, only without all the razor sharp edges and some really useful features.

1: https://capnproto.org/

ioctl() forever?

roc — Wed, 08 Jun 2022 20:32:40 +0000

For rr we really want to be able to determine, from the parameters and results of a syscall, all the userspace memory locations that a syscall writes to. Other tools (e.g. fuzzers) need the memory locations read as well. This is particularly hard for ioctls. The current ioctl op format is supposed to tell us the size of the data pointed to by the pointer, and whether it's read or written or both, but this information is often incomplete or plain wrong. In the worst cases, the data in that struct is just the root of some elaborate data structure with many other userspace pointers in it. So it would be great if an ioctlv2 design solves this. One way would be to provide metadata allowing us to parse the layout *and making the correctness of that metadata testable by automated tests*. Another way would just be to require that the data be a single contiguous buffer.