Per-file OOM badness

> Can't the malicious process just create files in /dev/shm (either directly, or via shm_open(3)) instead?

Files created in /dev/shm/ reside in a named tmpfs filesystem, which sets an upper bound on the memory consumed (50% of RAM by default—Debian doesn't appear to override this). The files created with memfd_create() are also tmpfs files, from an internal, unmounted tmpfs, but they behave more like anonymous shared memory mappings (mmap() with MAP_ANONYMOUS)—so far as I can tell the limits on /dev/shm/ do not apply. There is a limit on the total amount of shared memory, the kernel.shmall sysctl knob, which I think would also affect memfd_create(), but this defaults to "unlimited" (~2**64 pages).