Clearly the way to avoid the memory shortage at the time that you're OOM is to pre-create the GUI dialog, pre-allocating all it's memory and then have it tied into some IPC (even an open FS on a named PIPE, a la /dev/initctl). Even a signal handler might work)
Then the OOM code simply posts an event to the IPC or sends the signal.
Now the GUI un-hides itself (this might trigger some memory utilization in the X server's backing store but that's very likely to already be available from X' heap and if any malloc() fails I'd hope that the X server would be robust enough to simply throw away the backing. Backing store is a caching feature that should fail gracefully).
The trick now is for the code filling in the GUI dialog to traverse the process table, displaying entries and allowing the selection of death all within the memory it pre-allocated. It must be prepared to page through the process listing in relatively small (let's say 4KB) fragments.