LWN: Comments on "Blocking userfaultfd() kernel-fault handling"

Blocking userfaultfd() kernel-fault handling

tobin_baker — Wed, 17 Jun 2020 00:48:47 +0000

How about implementing COW private mappings of shared memory with true snapshot semantics?

Blocking userfaultfd() kernel-fault handling

smooth1x — Sun, 17 May 2020 08:54:35 +0000

What happens if the VM contains a database server? I can see this for that use case.

Blocking userfaultfd() kernel-fault handling

nilsmeyer — Wed, 13 May 2020 08:48:28 +0000

> Client workflows often can't be interrupted at will and even asking clients nicely to reboot their instances (so they can migrate to other hardware nodes) can take months. It's much easier to involuntarily migrate client VMs to different hardware.

That is true in a lot of environments, especially when yo u are dealing with software that manages state. It's easy to say that one can design an application so this isn't necessary (though a lot of the container/cloud-native crowd completely ignores stateful systems), but the reality is very different.

(de-)compression and view are different layers

gus3 — Mon, 11 May 2020 07:02:26 +0000

If the kernel handles compression/decompression matters, it's to save on paging space/speed. The user space sees nothing different.

If the user space handles compression, the kernel doesn't care about it at all.

They aren't related.

Blocking userfaultfd() kernel-fault handling

meyert — Sat, 09 May 2020 22:33:11 +0000

A bit OT, but I recently learned that quintessential come from Latin "quinta essentia" and means literally "fifths element" and indeed does mean the fifth element, i.e. "ether".

Blocking userfaultfd() kernel-fault handling

NYKevin — Sat, 09 May 2020 20:33:00 +0000

> Live migration is ABSOLUTELY justified for cloud computing providers to protect against hypervisor vulnerabilities.

I don't understand how this contradicts anything that I said...

Blocking userfaultfd() kernel-fault handling

Cyberax — Sat, 09 May 2020 15:59:23 +0000

I worked at Amazon, but I've heard about T2/T3 migration publicaly at AWS re:Invent multiple times. These instance types are severely oversubscribed and migration is used to balance the load.

Blocking userfaultfd() kernel-fault handling

roc — Sat, 09 May 2020 13:00:00 +0000

I *think* a UFFD_USER_MODE_ONLY flag/mode would work fine for us. We don't actually allow this fake process to execute syscalls normally; we catch its syscalls with ptrace and emulate them.

Blocking userfaultfd() kernel-fault handling

roc — Sat, 09 May 2020 12:58:37 +0000

Our Pernosco omnisicient, record-and-replay debugger uses userfaultfd() in a way that's neither 1 nor 2.

We have a giant omniscient database which lets us reconstruct the memory state of a process at any point in its recorded history. Sometimes we want to execute an application function "as if" the process was at some point in that history. So we create a new process, ptrace it, create mappings in it corresponding to the VMAs that existed at that point in history, and enable userfaultfd() for those mappings. Then we set the registers into the right state for the function call and PTRACE_CONT. Every time the process touches a new page, we reconstruct the contents of that page from our database. Works great.

Blocking userfaultfd() kernel-fault handling

Sesse — Sat, 09 May 2020 07:59:10 +0000

You're assuming data is paged out to begin with. :-) A prime candidate for this is if you want to mmap a compressed file (and have your application see uncompressed data).

Blocking userfaultfd() kernel-fault handling

wahern — Sat, 09 May 2020 07:52:38 +0000

Interesting. Any sources which I could share? All I could find in a quick Google search is an HN comment, "T2 and T3 use live migration to get around this, but it's not public knowledge." https://news.ycombinator.com/item?id=17815806

Blocking userfaultfd() kernel-fault handling

Cyberax — Sat, 09 May 2020 05:37:34 +0000

The virtual machine that runs client's code (KVM) looks like a regular process to the host Linux.

Blocking userfaultfd() kernel-fault handling

kccqzy — Sat, 09 May 2020 05:27:26 +0000

I don't understand the cloud provider argument. It does seem like this feature can help with live VM migration, but when you are a cloud provider, you don't necessarily require all users to run unmodified Linux kernels. If a user runs a non-Linux VM, how can the cloud provider migrate that VM?

Blocking userfaultfd() kernel-fault handling

Cyberax — Sat, 09 May 2020 05:02:43 +0000

> AWS doesn't support live migration.
It actually does behind the scenes with T2 and T3 instances.

Live migration is very useful to move client software out of a failing node. So really this makes sense only for large cloud providers.

Blocking userfaultfd() kernel-fault handling

wahern — Sat, 09 May 2020 04:58:33 +0000

AWS doesn't support live migration. Live migration is useful, but not for cloud computing, where state is kept outside the node. It's useful for traditional architectures where state is maintained on the node, with only backups (hopefully!) elsewhere. Not just useful but critical, because you're packing more work on the same piece of hardware, so reboots are more disruptive than with dedicated hardware.

Blocking userfaultfd() kernel-fault handling

Cyberax — Sat, 09 May 2020 02:00:41 +0000

Live migration is ABSOLUTELY justified for cloud computing providers to protect against hypervisor vulnerabilities.

Client workflows often can't be interrupted at will and even asking clients nicely to reboot their instances (so they can migrate to other hardware nodes) can take months. It's much easier to involuntarily migrate client VMs to different hardware.

Blocking userfaultfd() kernel-fault handling

josh — Sat, 09 May 2020 01:36:55 +0000

There are other use cases for this. Fastly's Lucet uses it for their WebAssembly VM, to catch out-of-bounds memory accesses.

Blocking userfaultfd() kernel-fault handling

NYKevin — Sat, 09 May 2020 01:30:32 +0000

My 2 cents: If you are not one a cloud provider, then you *probably* don't need userfaultfd() at all. It's the low-level equivalent of fiddling with the garbage collection algorithm, or writing your own malloc(). Basically, there are two use cases for this:

1. You're doing live migrations of VMs.
2. You can dynamically regenerate paged-out data faster than the OS can page it in.

(1) makes very little sense if you control all of the code in the VM, because it's far easier to just use a container instead of a VM, and start/stop instances as required (with all state living in some kind of database-like-thing, or perhaps a networked filesystem, depending on your needs). Sure, this is slightly more upfront design work, but live migration consumes an incredible amount of bandwidth once you try to scale it up, whereas container orchestration is a mature and well-understood technology. Unless you are making money per VM, it's difficult to justify the cost of live migration.

(Granted, if all of your VMs are very similar to one another, you might be able to develop a clever compression algorithm that shaves a lot of bytes off of that cost, but you're still not going to beat containers on size.)

That leaves (2). What's happening in case (2) is that you're using the page fault mechanism as a substitute for some kind of LRU cache for data that is expensive to compute, but cheaper than actually hitting the disk. But you can build an LRU cache in userspace, and it'll probably be a lot more efficient and easier to tune, since you can design it to exactly fit your specific use case. Trying to rope page faults into that problem makes no logical sense.

So, in conclusion, I'd tentatively suggest that distros consider turning the whole feature off and see if anything breaks. Perhaps they should teach their package managers to enable this setting if, and only if, one or more installed packages really need it.

Blocking userfaultfd() kernel-fault handling

Paf — Fri, 08 May 2020 23:25:01 +0000

The cost is awfully small, and while another option isn’t perfect, distros can enable it if desired. If it doesn’t break much, many or most will do so.

It’s not perfect, but this option is low cost.

Blocking userfaultfd() kernel-fault handling

dvdeug — Fri, 08 May 2020 22:57:05 +0000

I see the argument for it, but it's yet another obscure option. By default, it can't be turned on, so it won't provide any defense to most users. Considering "the existing vm/unprivileged_userfaultfd knob", is the cost of a feature to hobble but not disable userfaultfd really worth it?