My comment was imprecise at best. I'll clarify what I think you are doing:
Thread 1 decides the fd is no longer needed, due to no events
Thread 2 gets a wakeup for a real event, but is then scheduled out and does not progress
Thread 1 deletes the socket from epoll, marks fd as needing deletion, and signals via a pipe.
Another thead 3 then reads the pipe and deletes the fd
That does nothing to address the race with thread 2. There's still a race, all you've added is the essence of a sleep() which delays things. (Like the article mentioned, the solution of adding an arbitrary delay).