"User-space data may not be resident in RAM"

Posted Jul 9, 2020 6:28 UTC (Thu) by epa (subscriber, #39769)
In reply to: "User-space data may not be resident in RAM" by Fowl
Parent article: Sleepable BPF programs

Yes, the fork-exec idiom is a longstanding pain point for conservative memory management. If only posix_spawn() weren't so kitchen-sink-looking compared with the classical elegance of separate fork and exec primitives.

As for ordinary fork() where you won't immediately exec() afterwards, that might not spoil things, as it's only *writing* to user-space memory that can cause a copy-on-write fault, not reading it. And a system call to read bytes from a file into some memory may as well reserve the physical space in advance, rather than allocating it only once data comes back from the disk.

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 6:35 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link] (3 responses)

> As for ordinary fork() where you won't immediately exec() afterwards, that might not spoil things, as it's only *writing* to user-space memory that can cause a copy-on-write fault, not reading it.
Unless you're writing from other threads...

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 8:55 UTC (Thu) by epa (subscriber, #39769) [Link] (2 responses)

You've reached the limits of my knowledge. Could you explain how writing from other threads would cause a copy-on-write page fault to happen when a page is *read* rather than when it is written?

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 13:28 UTC (Thu) by Fowl (subscriber, #65667) [Link]

The fault wouldn't occur on a read, that's true, but even if the child process doesn't write, the parent is still running with all its threads, which will likely write eventually, and those can then unexpectedly fault.

So you'd have to not write in *both* the parent and the child until the child execs.

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 17:54 UTC (Thu) by Cyberax (✭ supporter ✭, #52523) [Link]

The child and parent both become CoW, and fork() does not stop any other threads in the parent. These threads can very well keep on writing to the newly shared memory.

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 13:13 UTC (Thu) by Fowl (subscriber, #65667) [Link] (2 responses)

Ha, that's an interesting idea - fork_nocow(). Exist in the parent's address space until exec().

Hey, clone(2) already exists! A trampoline to spill all your registers to memory and you're done. Except for having to be careful not to leak any memory, as the memory would leak in the parent too, it's probably a friendlier environment than using regular fork - all your threads are still there. Oh and locks, who knows what's going on with that.

Oh and how does thread-local storage work? Can you move it between threads? Is it even common? Does it depends on the thread ID? Is it possible to fake the TID? I don't know

Hmm and how do you know when you can free the stack you've just allocated for this new out-of-process-but-same-address-space thread you've just started? And I bet python and other runtime-y things are not going to be happy with two threads that think they're the same. Maybe block the parent until the child exec s? I'm going to wave my magic wand and proclaim that if you did this you could then share the same stack on both sides of this new fork_nocow.

Anyway, I had fun spending 15 minutes thinking about it.

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 13:20 UTC (Thu) by Fowl (subscriber, #65667) [Link] (1 responses)

Oh I'm an idiot. fork has to return - of course you can't share a stack. Copying/COWing just the stack would mess up all those RAII and scope based resource cleanup things. More likely to work with a callback, but then you've thrown away the almost-compatibility with existing code that was the whole point.

Still, fun.

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 17:06 UTC (Thu) by JGR (subscriber, #93631) [Link]

Your fork_nocow() is probably closest to vfork(2), which is approximately the same as forking but without a new stack or any CoW. However vfork is only really useful for the child to immediately call exec().

"User-space data may not be resident in RAM"

Posted Jul 9, 2020 21:39 UTC (Thu) by ibukanov (subscriber, #3942) [Link]

posix_spawn is a kitchen-sink because there is no way to create a suspended process, apply all necessary settings/syscalls to it and start it. Hence the need to communicate a lot of possible settings through a single system call.