The rapid growth of io_uring [LWN.net]

The rapid growth of io_uring

Posted Feb 9, 2020 23:05 UTC (Sun) by andresfreund (subscriber, #69562)
In reply to: The rapid growth of io_uring by dcoutts
Parent article: The rapid growth of io_uring

The submission/completion queues do not contain entries for requests currently being processed. I.e. there can be more requests being processed, than fit in either of the max queue sizes.

Early on that could lead to the completion queue overflowing, but now those completions are saved (but no new submissions are allowed, basically).

to post comments

The rapid growth of io_uring

Posted Feb 10, 2020 1:33 UTC (Mon) by dcoutts (subscriber, #5387) [Link] (3 responses)

So if I understand what you mean then in principle we could have a completion ring of e.g. 1k entries, and submit 10k IORING_OP_POLL_ADD operations on sockets, and this can't lead to loosing any completions. At most, if the completion ring is empty we have to collect completions before submitting new operations (which is completely reasonable). If so, that's excellent.

So is this pattern of use with sockets expected to perform well, e.g. compared to epoll? If so, that's fantastic, we really can use just one system for both disk I/O and network I/O. The one-shot behaviour of IORING_OP_POLL_ADD is entirely adequate (covering epoll's EPOLLONESHOT which is its most useful mode).

The rapid growth of io_uring

Posted Feb 10, 2020 3:29 UTC (Mon) by andresfreund (subscriber, #69562) [Link] (1 responses)

> So if I understand what you mean then in principle we could have a completion ring of e.g. 1k entries, and submit 10k IORING_OP_POLL_ADD operations on sockets, and this can't lead to loosing any completions.

Yes, that's correct to my knowledge. That wasn't the case in the first kernels with support for io_uring however (you need to test for IORING_FEAT_NODROP to know). But for poll like things it's also not that hard to handle the overflow case - just treat all of the sockets as ready. But that doesn't work at all for asynchronous network IO (rather than just readiness).

And obviously, you'd need to submit the 10k IORING_OP_POLL_ADDs in smaller batches, because the submission queue wouldn't be that long. But that's fine.

FWIW, I am fairly sure you can submit an IORING_OP_POLL_ADD on an epoll fd. Signal readiness once epoll_pwait would return (without returning the events of course). So if you need something for which oneshot style behaviour isn't best suited, or which you might want to wait on only some of the time, you can also compose io_uring and epoll. Caveat: I have not tested this, but I'm fairly sure I've seen code doing so, and a quick look in the code confirms that this should work: eventpoll_fops has a .poll implementation, which is what io_uring (via io_poll_add) relies on to register to get notified for readiness.

The rapid growth of io_uring

Posted May 25, 2020 23:25 UTC (Mon) by rand0m$tring (guest, #125230) [Link]

the completion queue size is allocated to approx 2x the submission queue.

https://github.com/torvalds/linux/blob/444565650a5fe9c63d...

so that's the point at which it would reach capacity and fall over.

The rapid growth of io_uring

Posted Mar 5, 2022 23:06 UTC (Sat) by smahapatra1 (guest, #157268) [Link]

Sorry for the naive question: why is EPOLL_ONESHOT the most useful mode? Does it not require another EPOLL_CTL and hence less efficient?
Thank you.