LWN: Comments on "Avoiding blocking file-name lookups"

Avoiding blocking file-name lookups

ksandstr — Sun, 07 Feb 2021 10:49:28 +0000

An issue that's thusfar gone unraised is, "what do applications do when RESOLVE_CACHED doesn't find all the path resolution gubbins in the cache?" This has all the visual and olfactory signs of a "try" feature, and the question is not a difficult one to ask.

Say git-repack, crawling over a hojilliard objects; what'll it do when RESOLVE_CACHED returns its "path not cached, won't open"? Should the program revisit those files later without that flag, incurring the synchronous I/O delay it just took pains to avoid?

Moreover there's the question of how an application would be structured to use this feature, aside from the non-POSIX io_uring interface. It's not obvious where the benefit lay besides the marginal case where synchronous path resolution evicts path gubbins that could've resolved later files immediately; and even then a "try" syscall to probe the cache costs a TLB flush in the post-Meltdown world, so the bar to measurable benefit is rather high.

Unless there's a planned feature where the kernel would spawn a speculative path resolution on calls to open(2) w/ RESOLVE_CACHED when that didn't succeed, so that future synchronous open(2) would complete faster, it seems like this feature is completely useless to ordinary userspace programs. Isn't this, then, a feature purely for io_uring's async syscall wrapper, which wouldn't otherwise merit altering open(2)? And if the intent is to announce to the kernel that userspace would like to open the file either immediately or (failing that) sometime later, shouldn't the flag be named RESOLVE_FUTURE?

It seems a reasonable counter-proposal would identify the true problem as the file descriptor returned by the open(2) family having not been created until the call returns, by which time the file is expected to have been opened -- or alternatively, errors related to path resolution, access control, etc. to have been raised[0]. Since the file descriptor doesn't exist, no unthreaded multiplexing mechanism can be applied to sniff out its eventual result. A solution would be a set of three new features: a syscall to create an "unassociated" file descriptor[1] which could only pop errors related to descriptor creation; another to detach-open a filesystem object on that descriptor; and a third to maybe-wait and retrieve open(2) status; with the assumption that a completing open(2) would trigger select, poll, and epoll.

The kernel's implementation might resolve those opens that're immediately available and return them to userspace in the space of a single select/poll/epoll_wait call, and more importantly send off batches of I/O requests for those parts that are still required. Therefore this proposed alternative would allow a single-threaded process to saturate an I/O bus (or more likely, the kernel's concurrent access model) while avoiding the thread-switching overhead implied by resorting to kernel threads behind the scenes.

[0] or kept their silence forevermore
[1] more likely several to reduce average syscall overhead per fd; that's an implementation detail

Avoiding blocking file-name lookups

dcoutts — Thu, 04 Feb 2021 17:01:25 +0000

Indeed. That extends the existing "MIO" I/O manager to use io_uring for I/O readiness notification. So it is basically using io_uring as a slightly faster and simpler version of epoll. It's used only for sockets, pipes etc that can block indefinately.

Of course with io_uring we can do proper async I/O for disk files, as well as the existing readiness notification style for sockets etc. Whereas right now, disk I/O still blocks all the lightweight threads running on a GHC capability / HEC. So the next stage would be to use io_uring in an I/O manager for all files, which would allow GHC lightweight threads to issue I/O ops concurrently without blocking each other.

Avoiding blocking file-name lookups

zdzichu — Thu, 04 Feb 2021 13:13:36 +0000

Looks like GHC backend for io_uring is complete:
https://gitlab.haskell.org/ghc/ghc/-/issues/18390 (New backend for event manager based on io_uring )
As usual, CI tests fail, because they are run in a sandbox without new syscalls whitelisted. The bane of linux sandboxing :(

Avoiding blocking file-name lookups

dcoutts — Thu, 04 Feb 2021 12:32:26 +0000

For some high level languages this will be relatively easy in that the applications will not need to change, only the runtime system and/or libraries. This will be the case in Haskell for example, and I imagine the same will be true for Erlang.

In Haskell we multiplex many lightweight threads on a single OS thread. These lightweight threads use a blocking I/O style, but only blocking individual lightweight threads. To provide this blocking I/O abstraction -- without actually blocking the whole OS thread -- the runtime system uses one of non-blocking or async I/O OS interfaces.

In particular, GHC has I/O manager backends for select, epoll, kqueue (for some BSDs & Darwin) and Windows IOCP. Adding an I/O manager backend for iouring is a fair bit of work of course, but it only has to be done once, in one place. Existing applications will not need to be changed.

Will this eventually bottleneck if filesystems are mounted without noatime?

njs — Thu, 04 Feb 2021 06:01:53 +0000

As I understand it, when you call openat2 with the new flag, there are two things that might happen

Option 1 (if the data is all in cache): it does exactly the same thing as old-style openat2 would have done

Option 2 (otherwise): it does nothing and returns an error

So I don't see how it could introduce any new IO load – the new functionality is a strict subset of the old functionality.

Avoiding blocking file-name lookups

miquels — Thu, 28 Jan 2021 11:29:10 +0000

Then at least build something on epoll(), which is almost 20 years old, which was the replacement for poll() which is more than 30 years old which was a replacement for select() which is ancient :)

Anyway, the filedescriptor that is a handle to a io_uring instance, is actually pollable. So it would be possible to write an abstraction on top of io_uring that gives you asynchronous disk I/O that integrates with your existing select/poll/epoll loop. As I understand it, that is exactly how libraries like Rust's Tokio are planning on integrating io_uring, at least initially.

Mike.

Avoiding blocking file-name lookups

epa — Mon, 25 Jan 2021 06:51:25 +0000

Yes, I am thinking of the programming language interface. Since most languages developed on a Unix-like platform over the past thirty years will have select() in their standard library.

Avoiding blocking file-name lookups

dtlin — Sun, 24 Jan 2021 16:41:21 +0000

io_uring is the API to "open lots of files and then read then in whatever order they become available from cache and disk". As the article says, this just optimizes the "open" part to avoid bouncing threads when possible.

Avoiding blocking file-name lookups

epa — Sun, 24 Jan 2021 15:59:11 +0000

Maybe ye olde select() could be adapted so you can open lots of files and then read then in whatever order they become available from cache and disk? (Reading sequentially within each file, I mean.) There are even more parallelized ways of doing it, but this would be a start, and select() is a widely known and supported API.

Will this eventually bottleneck if filesystems are mounted without noatime?

buck — Sun, 24 Jan 2021 06:44:35 +0000

It's all well and good to be able to read files with abandon, but will the system have to pay the price in backlogged metadata updates? Will that somehow be able to trash caches enough that the happy path gets closed off at some point?

Sorry if this question is way off-base, but i don't know enough (anything, really) about the caches involved to know if it's preposterous.

In the ideal case, yes, it sounds great that a workload that's managed to garner a complete working set in cache could sidestep block I/O latency.

Avoiding blocking file-name lookups

ncm — Sat, 23 Jan 2021 23:36:58 +0000

The overwhelming majority of projects using Linux run only on one or a few carefully-curated hosts. You don't hear much about them, for largely the same reason.

Some in this class you do hear about run supercomputers and spacecraft.

Avoiding blocking file-name lookups

jugglerchris — Sat, 23 Jan 2021 15:10:10 +0000

This sounds useful for something that walks a whole subtree - like tar or a recursive grep.
You'd be able to start reading cached files without being blocked on name lookups for the ones whose name lookup needs I/O if you happen to try opening them in the wrong order.

Avoiding blocking file-name lookups

walters — Fri, 22 Jan 2021 14:32:03 +0000

This is very cool indeed. The mind-bending thing for me though is imagining rewriting vast chunks of userspace code (has anyone written a basic "io-uring tar/rsync equivalent"?) to make use of it; and the problem with that is that almost every project will need to support older kernels too for quite a while, so the code duplication there will be painful.

I think the William Gibson quote "The future is already here — it's just not very evenly distributed." will apply for a while =)