EPOLL_CTL_DISABLE and multithreaded applications

ONESHOT is the obvious and easiest way to handle lockless multithreaded processing of an epoll queue. In fact, on Solaris ONESHOT is the only option. There are no persistent events. The kernel, of course, is free to optimize for persistence, but userspace threads don't need to worry about a loaded gun lying around.

Edge-triggered signaling only provides a nominal percentage improvement in performance. If you're already going multithreaded and attempting lockless, than you're already massively multicore. Why bother adding all the complexity of edge-triggered events? Also, it's worth pointing out that with *BSD kqueue, ONESHOT automatically removes the decriptor from the queue. If epoll followed this excellent example then using ONESHOT would be end of story.

Seems to me the simplest solution to starvation is to ask the kernel to return events in FIFO order, i.e. the last one installed will also be the last in the next reported pending queue. That way you can use ONESHOT and still guarantee that there's only ever be a single owner of the object, e.g. exactly one of the threads or the kernel.

For the early termination cases (e.g. a second thread walking a shared queue and destroying sockets), just call shutdown on the socket and let the kernel report it via the normal queue processing.

The root of the problem here is that people want to use both a message passing pattern via epoll messaging, as well as allow arbitrary threads to jump into the fray and manipulate shared contexts. That's just asking for trouble.