As you note, memory is a problem with glibc implementation with large directories. Your proposed scheme for readdir looks good, except that I find it a bit restrictive to tie a top layer to a bottom layer. I guess with you scheme, once you do a union mount, you can't re-union the top layer on top of any other layer.