Preserving guest memory across kexec

Posted May 20, 2022 16:32 UTC (Fri) by developer122 (guest, #152928)
Parent article: Preserving guest memory across kexec

Reminds me a bit of the problem of preserving userspace across reboots of a microkernel.

The solution tends to look a lot like a filesystem. When a computer is rebooted, contents on disk are still there and are rediscovered by the new instance of the filesystem driver. So, extend that metaphor to memory, where the "files" and "directories" are old processes and memory mappings. Page tables <-> filesystem trees.

Of course, there are some details like making sure the previous version's mappings can be updated to reflect a new "on disk format" and that calls to now-nonexistant system calls can be handled appropriately. Also that some ram is available for the new kernel to set itself up before walking the old page and process tables.

The beauty of a microkernel is that the userspace drivers probably still remember the state of hardware devices, even after the microkernel itself is rebooted.

to post comments

Preserving guest memory across kexec

Posted May 20, 2022 16:34 UTC (Fri) by developer122 (guest, #152928) [Link] (5 responses)

I wonder, if at some point in the future we will be asking how the linux kernel might preserve a select few processes across reboots?

Preserving guest memory across kexec

Posted May 20, 2022 16:47 UTC (Fri) by mb (subscriber, #50428) [Link] (4 responses)

Checkpoint + restore?
https://en.wikipedia.org/wiki/CRIU

Preserving guest memory across kexec

Posted May 21, 2022 6:30 UTC (Sat) by developer122 (guest, #152928) [Link] (3 responses)

That saves it to disk, so I guess you could fetch it from the filesystem after rebooting, but the point is really to leave processes in memory and largely unmolested while the OS reboots around them.

Preserving guest memory across kexec

Posted May 21, 2022 12:37 UTC (Sat) by mb (subscriber, #50428) [Link] (2 responses)

Yes. Processes have lots of kernel state that's not easily retained across reboots. That's basically what checkpoint/restore solves.
If you don't want to access the disk, then the only additional thing needed would be a ramdisk that survives reboots where the checkpoints are stored. Such a ramdisk sounds way easier to implement than reinventing checkpoint/restore just for the kexec case.

Preserving guest memory across kexec

Posted May 26, 2022 2:35 UTC (Thu) by developer122 (guest, #152928) [Link]

checkpoint and restore means a ton of copying application data around and packing and unpacking. But I guess you have to do it at all before you can do it zero-copy.

Preserving guest memory across kexec

Posted May 31, 2022 14:51 UTC (Tue) by simcop2387 (subscriber, #101710) [Link]

That actually sounds a lot like a big use case for the whole NVDIMM stuff that has been appearing lately. Faster than disk, not as fast as memory but it's also made to persist between reboots, power offs, etc.

Preserving guest memory across kexec

Posted May 21, 2022 15:05 UTC (Sat) by developer122 (guest, #152928) [Link] (4 responses)

The extreme example is one where the OS can replace itself while userspace continues on running CPU-bound load. Having the microkernel replace itself and pick back up without any downtime at all, unless perhaps a program tries to make a call.

Dunno that it would be possible on current hardware though, with existing interrupt controllers and memory paging.

Preserving guest memory across kexec

Posted May 22, 2022 7:37 UTC (Sun) by Wol (subscriber, #4433) [Link] (3 responses)

Bear in mind you are describing 1980s technology ...

I've stood next to a system where my friend said "if I pull out this board no-one will notice except the engineers monitoring the system ..." he was pointing at the CPU ...

and the OS was just another replaceable component same as ...

Cheers,
Wol

Preserving guest memory across kexec

Posted May 26, 2022 2:54 UTC (Thu) by developer122 (guest, #152928) [Link] (2 responses)

Perhaps, but we stopped building anything to support that *in the 80's.* All the Dell and HP and Supermicro boxes are glorified PCs.

At the same time, the only reason that was possible was because of virtualization. All the "OSs" were running in logical partitions aka hardware VMs. If you lost the hypervisor then the system went down, and the base hypervisor firmware was impossible to replace (often because it's in ROM).

Preserving guest memory across kexec

Posted May 26, 2022 9:19 UTC (Thu) by Wol (subscriber, #4433) [Link] (1 responses)

Well, yes there was a lot of - not exactly custom - hardware in there, but I guess to upgrade the hypervisor they could quite possibly just have replaced the CPU boards. I don't remember any reference to rom-based hypervisors, though ...

Cheers,
Wol

Preserving guest memory across kexec

Posted May 30, 2022 0:42 UTC (Mon) by developer122 (guest, #152928) [Link]

From what I recall, the LPARS on IBM or domains on Sun SPARC were managed by firmware loaded from the system board itself, into either a main CPU or a service processor.