|
|
Subscribe / Log in / New account

Blocking userfaultfd() kernel-fault handling

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 1:30 UTC (Sat) by NYKevin (subscriber, #129325)
In reply to: Blocking userfaultfd() kernel-fault handling by Paf
Parent article: Blocking userfaultfd() kernel-fault handling

My 2 cents: If you are not one a cloud provider, then you *probably* don't need userfaultfd() at all. It's the low-level equivalent of fiddling with the garbage collection algorithm, or writing your own malloc(). Basically, there are two use cases for this:

1. You're doing live migrations of VMs.
2. You can dynamically regenerate paged-out data faster than the OS can page it in.

(1) makes very little sense if you control all of the code in the VM, because it's far easier to just use a container instead of a VM, and start/stop instances as required (with all state living in some kind of database-like-thing, or perhaps a networked filesystem, depending on your needs). Sure, this is slightly more upfront design work, but live migration consumes an incredible amount of bandwidth once you try to scale it up, whereas container orchestration is a mature and well-understood technology. Unless you are making money per VM, it's difficult to justify the cost of live migration.

(Granted, if all of your VMs are very similar to one another, you might be able to develop a clever compression algorithm that shaves a lot of bytes off of that cost, but you're still not going to beat containers on size.)

That leaves (2). What's happening in case (2) is that you're using the page fault mechanism as a substitute for some kind of LRU cache for data that is expensive to compute, but cheaper than actually hitting the disk. But you can build an LRU cache in userspace, and it'll probably be a lot more efficient and easier to tune, since you can design it to exactly fit your specific use case. Trying to rope page faults into that problem makes no logical sense.

So, in conclusion, I'd tentatively suggest that distros consider turning the whole feature off and see if anything breaks. Perhaps they should teach their package managers to enable this setting if, and only if, one or more installed packages really need it.


to post comments

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 1:36 UTC (Sat) by josh (subscriber, #17465) [Link]

There are other use cases for this. Fastly's Lucet uses it for their WebAssembly VM, to catch out-of-bounds memory accesses.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 2:00 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (6 responses)

Live migration is ABSOLUTELY justified for cloud computing providers to protect against hypervisor vulnerabilities.

Client workflows often can't be interrupted at will and even asking clients nicely to reboot their instances (so they can migrate to other hardware nodes) can take months. It's much easier to involuntarily migrate client VMs to different hardware.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 4:58 UTC (Sat) by wahern (subscriber, #37304) [Link] (3 responses)

AWS doesn't support live migration. Live migration is useful, but not for cloud computing, where state is kept outside the node. It's useful for traditional architectures where state is maintained on the node, with only backups (hopefully!) elsewhere. Not just useful but critical, because you're packing more work on the same piece of hardware, so reboots are more disruptive than with dedicated hardware.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 5:02 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

> AWS doesn't support live migration.
It actually does behind the scenes with T2 and T3 instances.

Live migration is very useful to move client software out of a failing node. So really this makes sense only for large cloud providers.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 7:52 UTC (Sat) by wahern (subscriber, #37304) [Link] (1 responses)

Interesting. Any sources which I could share? All I could find in a quick Google search is an HN comment, "T2 and T3 use live migration to get around this, but it's not public knowledge." https://news.ycombinator.com/item?id=17815806

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 15:59 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

I worked at Amazon, but I've heard about T2/T3 migration publicaly at AWS re:Invent multiple times. These instance types are severely oversubscribed and migration is used to balance the load.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 20:33 UTC (Sat) by NYKevin (subscriber, #129325) [Link]

> Live migration is ABSOLUTELY justified for cloud computing providers to protect against hypervisor vulnerabilities.

I don't understand how this contradicts anything that I said...

Blocking userfaultfd() kernel-fault handling

Posted May 13, 2020 8:48 UTC (Wed) by nilsmeyer (guest, #122604) [Link]

> Client workflows often can't be interrupted at will and even asking clients nicely to reboot their instances (so they can migrate to other hardware nodes) can take months. It's much easier to involuntarily migrate client VMs to different hardware.

That is true in a lot of environments, especially when yo u are dealing with software that manages state. It's easy to say that one can design an application so this isn't necessary (though a lot of the container/cloud-native crowd completely ignores stateful systems), but the reality is very different.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 5:27 UTC (Sat) by kccqzy (guest, #121854) [Link] (1 responses)

I don't understand the cloud provider argument. It does seem like this feature can help with live VM migration, but when you are a cloud provider, you don't necessarily require all users to run unmodified Linux kernels. If a user runs a non-Linux VM, how can the cloud provider migrate that VM?

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 5:37 UTC (Sat) by Cyberax (✭ supporter ✭, #52523) [Link]

The virtual machine that runs client's code (KVM) looks like a regular process to the host Linux.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 7:59 UTC (Sat) by Sesse (subscriber, #53779) [Link] (1 responses)

You're assuming data is paged out to begin with. :-) A prime candidate for this is if you want to mmap a compressed file (and have your application see uncompressed data).

(de-)compression and view are different layers

Posted May 11, 2020 7:02 UTC (Mon) by gus3 (guest, #61103) [Link]

If the kernel handles compression/decompression matters, it's to save on paging space/speed. The user space sees nothing different.

If the user space handles compression, the kernel doesn't care about it at all.

They aren't related.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 12:58 UTC (Sat) by roc (subscriber, #30627) [Link] (1 responses)

Our Pernosco omnisicient, record-and-replay debugger uses userfaultfd() in a way that's neither 1 nor 2.

We have a giant omniscient database which lets us reconstruct the memory state of a process at any point in its recorded history. Sometimes we want to execute an application function "as if" the process was at some point in that history. So we create a new process, ptrace it, create mappings in it corresponding to the VMAs that existed at that point in history, and enable userfaultfd() for those mappings. Then we set the registers into the right state for the function call and PTRACE_CONT. Every time the process touches a new page, we reconstruct the contents of that page from our database. Works great.

Blocking userfaultfd() kernel-fault handling

Posted May 9, 2020 13:00 UTC (Sat) by roc (subscriber, #30627) [Link]

I *think* a UFFD_USER_MODE_ONLY flag/mode would work fine for us. We don't actually allow this fake process to execute syscalls normally; we catch its syscalls with ptrace and emulate them.

Blocking userfaultfd() kernel-fault handling

Posted May 17, 2020 8:54 UTC (Sun) by smooth1x (guest, #25322) [Link]

What happens if the VM contains a database server? I can see this for that use case.

Blocking userfaultfd() kernel-fault handling

Posted Jun 17, 2020 0:48 UTC (Wed) by tobin_baker (guest, #139557) [Link]

How about implementing COW private mappings of shared memory with true snapshot semantics?


Copyright © 2025, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds