Keeping secrets in memfd areas

That depends on the hardware details - e.g. I believe some common ARM TrustZone implementations can mark regions of memory as inaccessible to the DMA controller, the GPU, the CPU, etc, which sounds like it could be useful here. (Sometimes used for DRM video decoding, where the decrypted bitstream and decoded frames are accessible to the VPU/GPU/display and not to the CPU, but it could be configured the other way round.)