It seems like anonymous mmap might as well be implemented using shm under the covers. Maybe it is.
The sample code looks plausible, but you omitted code to check whether backing store for an address range has been reclaimed. Actually we don't need a new flag; MADV_MREMOVE suffices. The docs don't say what the process would find in an MREMOVEd page, were it to look, or what marking a previously MREMOVEd range MADV_NORMAL does. Mark it MADV_NORMAL to be see if everything written is still there. If that fails, check progressively smaller ranges. Or unmap the lot and forget all about it. Or mark it DONTNEED, equivalent to munmapping and re-mmapping the address range.