Recent Features
| |||||||||||||
EPOLL_CTL_DISABLE and multithreaded applicationsEPOLL_CTL_DISABLE and multithreaded applicationsPosted Oct 19, 2012 14:05 UTC (Fri) by mkerrisk (subscriber, #1978)In reply to: EPOLL_CTL_DISABLE and multithreaded applications by kjp Parent article: EPOLL_CTL_DISABLE and multithreaded applications
Better solution: since epoll lets you store a generic 64 bit cookie, just use a 64 bit sequence that increments for each new file descriptor. I haven't thought through your solution very far, but it seems unfortunate to have to chew up the cookie to solve this problem. User space might very want to user the epoll_event.data field for other purposes.
(Log in to post comments)
EPOLL_CTL_DISABLE and multithreaded applications Posted Oct 19, 2012 15:25 UTC (Fri) by kjp (subscriber, #39639) [Link]
To clarify, my solution is all user space, no kernel changes. It's just changing the kernel from holding a 'strong reference' to a 'weak reference'. (You know what they say about adding a layer of indirection...)
When I did epoll, I used it in edge_triggered mode (and also not 'oneshot') and had a single thread processing the epoll events and scheduling workers. I had worker threads that just pumped data to and from the kernel. I put direct pointers in the epoll data (i.e. strong references) to my internal structures since I had only one thread calling epoll.
But if I had multiple threads calling epoll, I think the solution I outlined would work fine. I don't see what else I would need the cookie for... as long as its still doing its job, pointing to a real data structure of mine.
And the nice thing is, it works with all epoll modes. What's very distasteful about the kernel patch is that it requires ONESHOT (yuck!).
The epoll designer(s) had their thinking caps on with this api. Storing arbitrary cookies + edge triggered mode = Insanely good.
EPOLL_CTL_DISABLE and multithreaded applications Posted Oct 19, 2012 15:35 UTC (Fri) by mkerrisk (subscriber, #1978) [Link] But if I had multiple threads calling epoll, I think the solution I outlined would work fine. I don't see what else I would need the cookie for... as long as its still doing its job, pointing to a real data structure of mine. Yes, but other people may want to use the cookie in a quite different way, and it seems a shame to limit the generality of the API by requiring it to be used for this task. And the nice thing is, it works with all epoll modes. What's very distasteful about the kernel patch is that it requires ONESHOT (yuck!). Yes, requiring the use of EPOLLONESHOT is rather unfortunate. I strongly suspect that there could be a solution quite similar to the EPOLL_CTL_DISABLE approach that doesn't require EPOLLONESHOT. I have something in mind, but I need to think about it a little more.
EPOLL_CTL_DISABLE and multithreaded applications Posted Oct 19, 2012 21:39 UTC (Fri) by wahern (subscriber, #37304) [Link]
ONESHOT is the obvious and easiest way to handle lockless multithreaded processing of an epoll queue. In fact, on Solaris ONESHOT is the only option. There are no persistent events. The kernel, of course, is free to optimize for persistence, but userspace threads don't need to worry about a loaded gun lying around.
Edge-triggered signaling only provides a nominal percentage improvement in performance. If you're already going multithreaded and attempting lockless, than you're already massively multicore. Why bother adding all the complexity of edge-triggered events? Also, it's worth pointing out that with *BSD kqueue, ONESHOT automatically removes the decriptor from the queue. If epoll followed this excellent example then using ONESHOT would be end of story.
Seems to me the simplest solution to starvation is to ask the kernel to return events in FIFO order, i.e. the last one installed will also be the last in the next reported pending queue. That way you can use ONESHOT and still guarantee that there's only ever be a single owner of the object, e.g. exactly one of the threads or the kernel.
For the early termination cases (e.g. a second thread walking a shared queue and destroying sockets), just call shutdown on the socket and let the kernel report it via the normal queue processing.
The root of the problem here is that people want to use both a message passing pattern via epoll messaging, as well as allow arbitrary threads to jump into the fray and manipulate shared contexts. That's just asking for trouble.
EPOLL_CTL_DISABLE and multithreaded applications Posted Oct 30, 2012 0:23 UTC (Tue) by normalperson (subscriber, #47508) [Link]
I agree ONESHOT is awesome for MT and probably should've been the default.
epoll_wait() already returns ONESHOT events in FIFO order based on my reading of fs/eventpoll.c (and my own testing/experience). kqueue also seems to return in FIFO order with ONESHOT.
I wrote a server based on this behavior (along with concurrent calls to epoll_wait(...,maxevents=1,...)) for getting fair distribution between threads stuck in I/O wait a while back: http://bogomips.org/cmogstored.git
|
|||||||||||||