LWN: Comments on "sys_epoll - making poll fast"

I want some of them callbacks!

bronson — 2002-11-02T21:15:33+00:00

Maybe IkeTo gave too comprehensive an answer...

Is it too tricky to have the kernel change to user context and invoke a function pointer that I have given it?

Yes.

Whenever the kernel calls user code directly, non-obvious and non-trivial race conditions result. It's a very difficult problem. Much easier to just have the system call return, and then the user can deal with the callbacks however she/he wants.

I want some of them callbacks!

zooko — 2002-10-31T20:56:08+00:00

I think there is an interesting disconnect between the low-level hackers and the higher level network/application level hackers here. A lot of the interesting projects that I know of (not to mention the ones that I lead myself: Mnet), use select()/poll() interfaces exactly as I've described above -- as nothing more than a way to get at the callbacks which they then immediately de-multiplex back into callbacks. The projects I'm thinking of here are things like Twisted, and E (although the current E implementation doesn't use kernel calls, as it is in Java...).

Regards,

Zooko

I want some of them callbacks!

IkeTo — 2002-10-31T13:24:18+00:00

Hmm... that seems to be as difficult to use as signals. The function that you want to use is never in the list of async signal-safe functions, and you application will always have race condition using signals unless your application uses the call in a very limited way: your main application must do basically nothing, and your application must block basically every signal (and wait for signals using sigwait() or sigsuspend()). That means... you'll have to create system calls similar to those to deal with signals, and then to setup a huge size set for suspending file descriptor callbacks. Not to mention that pushing some code to the user is not something that a kernel developers would like to do unless absolutely needed. Doesn't seem to be exactly exciting for this type of optimization which is constant-time anyway.

I want some of them callbacks!

zooko — 2002-10-31T11:01:49+00:00

I don't understand why the callbacks aren't propagated to userland. All of my code which uses select() or poll() style interfaces immediately invokes callbacks based on which fds have changed, so if my code ran on top of epoll(), then the sequence of events would look like this:

the socket activity would cause callbacks
which would be intercepted by the epoll() system
which would update the data structure and then return from epoll_wait(),
where my userland code would crawl the data structure and invoke callbacks for each changed thing,
where higher-level application code would then receive the callbacks.

Doesn't this seem unnecessarily complicated, not to mention less efficient? Is it too tricky to have the kernel change to user context and invoke a function pointer that I have given it?

Regards,

Zooko

sys_epoll - making poll fast

johnchx — 2002-10-31T08:08:30+00:00

I'd like to hear more about how this approach compares with asynchronous reads and writes from/to sockets. Doesn't the ability to do async i/o reduce the need for a scalable select()-style mechanism?

sys_epoll - making poll fast

dododge — 2002-10-31T07:22:13+00:00

LKML has had a bunch of messages this week about the API documentation, and it's clear there are some subtle issues with how you go about using sys_epoll correctly. If you try to use it as a drop-in replacement for existing select and poll code, you can end up with races. The setup and waiting has to be done with an understanding of what actually causes epoll_wait to report an event. A good/quick explanation of how to use it:

http://www.uwsg.indiana.edu/hypermail/linux/kernel/0210.3/1520.html

sys_epoll - making poll fast

cpeterso — 2002-10-31T04:02:35+00:00

This sounds a lot like NT's IO Completion Ports, but without NT's automatic thread pool management. IO Completion Ports are nice because the thread pool balances the work items (IO callbacks) to threads pinned to each CPU. There is usually one one thread per CPU servicing IO callbacks. This avoids extra context switches overhead and avoids trashing the CPU cache with multiple threads. I'm sure thread pooling could be added with userland libraries on top of the sys_epoll system call.

sys_epoll - making poll fast

yanfali — 2002-10-31T02:09:16+00:00

I've personally used the old version /dev/epoll and it scales beautifully,
especially when combined with NAPIfied network drivers. In an internal
test application I was able to handle 60K sockets opened and over
12K active per second on a 1Ghz pentium III. This API is going to rocket
the performance of linux based TCP/IP servers through the ceiling.

Congrats to Davide for a job well done.