It sounds like the issue is a timeout case. The diagram shows one thread sees the file descriptor as not ready (no events) and decides to delete it. But, then suddenly an event for it comes in and starts processing on another thread. I don't see how your solution addreses that. Your pipe wakeup could happen at the same time as a 'real' socket wakeup event.
Posted Oct 28, 2012 17:59 UTC (Sun) by kjp (subscriber, #39639)
[Link]
My comment was imprecise at best. I'll clarify what I think you are doing:
Thread 1 decides the fd is no longer needed, due to no events
Thread 2 gets a wakeup for a real event, but is then scheduled out and does not progress
Thread 1 deletes the socket from epoll, marks fd as needing deletion, and signals via a pipe.
Another thead 3 then reads the pipe and deletes the fd
That does nothing to address the race with thread 2. There's still a race, all you've added is the essence of a sleep() which delays things. (Like the article mentioned, the solution of adding an arbitrary delay).