Kernel Summit: Asynchronous I/O [LWN.net]

This article is part of LWN's 2004 Kernel Summit coverage.

Suparna Bhattacharya has done much of the recent work on asynchronous I/O, so she was the logical person to lead a session on that subject. She noted that numerous patches exist which address many of the current shortcomings of mainline AIO; these patches include the retry mechanism, support for asynchronous buffered filesystem I/O, pipe I/O, and the poll() system call. Vector AIO (a mechanism for joining multiple asynchronous requests and submitting them together) is also in the works.

Beyond simply completing the implementation, Linux AIO could use a number of other changes. One idea calls for separating the current aio_read() file_operations method into submit_aio_read() and complete_aio_read() (and similarly for the write side). Splitting the AIO method in that way would apparently make retry-based implementations easier to implement. The kernel currently supports "synchronous kiocbs," which are a way of requesting synchronous operations via the AIO paths; it may come as little surprise that nobody is actually using this mechanism, so it can come out.

In a more general sense, it is perhaps time to reexamine the relationship between synchronous and asynchronous I/O. One could look at synchronous operations as really just being the asynchronous variety with a wait added at the end. If the kernel were restructured along those lines, some things would be simplified, but it would imply that retry-based schemes (used for buffered filesystem AIO - see LWN's coverage) would be used for synchronous I/O as well.

The topic turned to asynchronous fsync(); Linus noted that, internally, this call is implemented by starting the requisite I/O, then waiting for completion. It would not be hard to export a separate, asynchronous fsync() operation. Why not, asked Suparna, simply use the existing AIO interface for that? The answer, it seems, is that nobody likes the current AIO interface. Few people use it; even glibc avoids it.

So, one might ask, how can this interface be fixed? One idea was to add an aio_suspend() system call which could wait for (one or more of) multiple I/O contexts to complete. Other ideas included using signals to notify processes of completion, or simply to use the epoll calls. One way of doing that could be to create a new file descriptor for each outstanding AIO operation which could then be passed to epoll.

The real problem with AIO, however, remains the (perceived) lack of users for this capability. The existing AIO code is used by a few high-end databases for their file I/O, and MySQL is apparently being extended to make use of it - but that is about it. Really pushing AIO forward will require that more users step up, and that the performance benefits of working in this mode be better demonstrated.

>>Next: Multipath I/O.

Index entries for this article
Kernel	Asynchronous I/O

to post comments

One happy user

Posted Jul 22, 2004 14:04 UTC (Thu) by avik (guest, #704) [Link] (4 responses)

We're happily using aio (in a RHEL 3 2.4 based kernel), with some added patches
for vector aio, and copyless network transmits. aio is very important to us, and
we'd really like to see increased support, especially where it enables copyless
operation.

One happy user

Posted Jul 22, 2004 16:03 UTC (Thu) by cliffman (guest, #13144) [Link] (3 responses)

Can you tell us more about the application?

One happy user

Posted Jul 22, 2004 18:56 UTC (Thu) by avik (guest, #704) [Link] (2 responses)

Sure. It's a clustered filesystem (a hot topic these days). It's
implemented in userspace, so efficient and copyless I/O are essential.

We're using Red Hat's Enterprise kernel, which provides aio poll and
blockdev read/write. We hacked aio readv and writev (a must for
performance) and copyless aio udp sendmsg. We'll probably do aio tcp
sendmsg as well.

We would dearly love the receive counterparts as well, especially if they
are copyless, though more sophisticated networking hardware is probably
needed.

I'd like to emphasize that aio poll is absolutely crucial for mixing
networking and blockdev I/O, at least until full aio networking support is
available.

Avi

One happy user

Posted Jul 23, 2004 23:25 UTC (Fri) by giraffedata (guest, #1954) [Link] (1 responses)

What I'd like to hear is how it's better than buffered I/O and/or multiple threads doing synchronous I/O. That's what it's competing with.

One happy user

Posted Jul 24, 2004 7:10 UTC (Sat) by avik (guest, #704) [Link]

Buffered I/O is out of the question. It involves copying which reduces throughput. Synchronous reads, buffered or not, will block, throwing parallelism out the window. We do our own caching (distributed filesystem), so the kernel cache is very small and ineffective; it only adds overhead,

Our previous solution was nonblocking network I/O, plus slave threads doing synchronous direct disk I/O. This had two problems:

- less efficient due to context switches and scheduling latencies
- no way to do copyless network I/O.

The last point is only partially addressed by aio, but it's better than nothing.

Kernel Summit: Asynchronous I/O

Posted Jul 22, 2004 16:01 UTC (Thu) by cliffman (guest, #13144) [Link]

We definately need a 'call-to-arms' from the aio users. How about we collect experiences from AIO users? We ( OSDL ) are very interested, and we'd be glad to know of as many applications for AIO as possible, especially people who could give us a sample workload for testing purposes.

Kernel Summit: Asynchronous I/O

Posted Aug 2, 2004 6:42 UTC (Mon) by goaty (guest, #17783) [Link] (1 responses)

I'd love to use asynchronous file I/O. Unfortunately, the APIs have "this was designed to make Oracle go fast" written all over them. I couldn't think of a single thing they were useful for except a big old-fashioned database. They can't even do open() asynchronously.

The application I was interested in was GUI file managers. To call up a directory in a file manager, you first need the file list, then you need to stat every file, and then ideally you need to read the file's magic number. At present all this is done synchronously, so the disk goes crazy seeking all over the damn place, and it takes forever. If the I/O requests could be submitted asynchronously, then the kernel could sort them using its normal elevator algorithm, and performance would suck less.

What it comes down to is that the current AsyncIO API has few users because it has few uses. If there was an AsyncIO API "for the peoples", that made things like web servers and "ls -l" faster, then the users would come.

Kernel Summit: Asynchronous I/O

Posted Sep 13, 2004 8:57 UTC (Mon) by oberholtzer (guest, #24680) [Link]

We use asynchronous I/O from network, serial and keyboard devices. We do
this in a single thread because we do not believe that multiple thread
responsiveness is as good. This is via SIGIO.

We have not used the 'aio' named functions. Not due to lack of interest. I
think a more general description of the functions is needed.

aio vs sync io

Posted May 28, 2005 22:20 UTC (Sat) by Nir (guest, #27440) [Link]

i have been testing aio vs synchronous io and i came to conclude that aio is not doing the job. the test was as follows, i have created big buffers for fetching data from the disks ( 4 MB ) and submitted 200 requests . I have done it synchronously and asynchnrously and the async io seems to be 5 times slower, at least. Also, and this is just a side remark, linux seems to wasting cpu when it comes to io. no matter whether it is sync io or async io. Raz.