Hmm. When the program is started, can't you just call getrlimit to find out the RLIMIT_AS limit is? Then, if that is less than 16 GB, allocate only that amount of pages rather than the full 16 GB.
Of course, if that's *all* you do, you can't get any more pages by other means, such as mmap.
I guess you can work around that too. Whenever you are about to call mmap, first reduce your original reservation by the size of the reservation you're about to make using mremap(MREMAP_FIXED).