The real realtime preemption end game

Posted Nov 16, 2023 21:44 UTC (Thu) by itsmycpu (guest, #139639)
Parent article: The real realtime preemption end game

That has me wondering... in general, is it possible to create a protected memory space that survives rebooot?

The real realtime preemption end game

Posted Nov 16, 2023 22:53 UTC (Thu) by mjg59 (subscriber, #23239) [Link] (18 responses)

In general? No. You have no guarantees about the behaviour of the firmware over reboot, and it's legitimate for it to just wipe the entire contents of RAM before booting anything else. But there are specific cases where this can be guaranteed - check the various pstore backends for examples.

The real realtime preemption end game

Posted Nov 17, 2023 0:47 UTC (Fri) by itsmycpu (guest, #139639) [Link] (1 responses)

Maybe something like this could preserve a range of memory? However I wouldn't know if that could be interesting in this context...

https://en.wikipedia.org/wiki/Reboot#Warm

"The Linux family of operating systems supports an alternative to warm boot; the Linux kernel has optional support for kexec, a system call which transfers execution to a new kernel and skips hardware or firmware reset. The entire process occurs independently of the system firmware. The kernel being executed does not have to be a Linux kernel.[citation needed]"

The real realtime preemption end game

Posted Nov 17, 2023 5:33 UTC (Fri) by mjg59 (subscriber, #23239) [Link]

kexec certainly provides a mechanism for preserving memory ranges over kernel switches, but it's not really what most people would describe as reboot

The real realtime preemption end game

Posted Nov 18, 2023 5:36 UTC (Sat) by mirabilos (subscriber, #84359) [Link] (10 responses)

It works in practice on BSD, dmesg shows the previous run’s messages as well.

In practice here means x86 hardware like Thinkpads and other assorted PCs and servers whose BIOS will not overwrite the entire memory during warm reboot, as well as SPARCstations whose OpenBoot will similarily not clear the high-up memory used for the kernel log buffer.

The real realtime preemption end game

Posted Nov 19, 2023 3:20 UTC (Sun) by Paf (subscriber, #91811) [Link] (7 responses)

“ It works in practice on BSD, dmesg shows the previous run’s messages as well.”

And surely this is only possible through the retention of data in memory over reboot! What other magic could do this?

Sorry, but I’d lay a lot of money this is done with storage.

The real realtime preemption end game

Posted Nov 19, 2023 5:15 UTC (Sun) by mirabilos (subscriber, #84359) [Link] (6 responses)

How much money are you willing to hand over?

https://mbsd.evolvis.org/cvs.cgi/src/sys/kern/subr_log.c?...
(I’m using a somewhat beefier mirror here to not get the main server slashdotted)
look for initmsgbuf near the beginning of the file, which gets a pointer to the RAM region.

It is called for SPARC from:
https://mbsd.evolvis.org/cvs.cgi/src/sys/arch/sparc/sparc...
(initmsgbuf called with an almost fixed (only the oldest systems avoid the first page) address…)

For i386, the call is at…
https://mbsd.evolvis.org/cvs.cgi/src/sys/arch/i386/i386/m...
… where msgbufp comes from…
https://mbsd.evolvis.org/cvs.cgi/src/sys/arch/i386/i386/p...
(the __OpenBSD__ ifdef) which sets the virtual address. The physical address (MMU mapping) is done somewhere between locore.s and there, and it looks to me like its location depends on the size of the kernel image, so you’d only get the log messages if you boot the same or a very similar-sized kernel after warm reboot.

And yes, it’s purely memory-based. It helps immensely in copying e.g. the remainder of a ddb(4) session (in-kernel debugger) out if you don’t have a serial console.

The real realtime preemption end game

Posted Nov 19, 2023 5:17 UTC (Sun) by mirabilos (subscriber, #84359) [Link]

Heh, and of course just as I posted this, the hoster fell off the ’net (including not only the hoster’s own homepage but also their status page, which is not hosted at a different site as is usual).

Oh well, it’ll be back at some point.

The real realtime preemption end game

Posted Nov 19, 2023 18:29 UTC (Sun) by kreijack (guest, #43513) [Link] (4 responses)

My understanding is that (at least in the x86 world), the memory is cleaned during the reboot.

So the problem is not to find a fixed area where store the data, but avoid that this area is cleaned up during a reboot.
And this cannot be done in a generic way.

The kind of reboot that I am talking, is the one that allow you to exit from a "crash", so I think that we are talking about an hard reboot. And an hard reboot implies the memory cleanup.
Think if this wouldn't exists: this would allow to extract from the memory some secret with a simple reboot at the "right time"; it would be a giant security hole.

The pstore back-ends in the x86 are mostly two: the first one relies on the UEFI variable storage; the second one relies on the ACPI-ERST, which is like a flash memory.

The real realtime preemption end game

Posted Nov 19, 2023 18:35 UTC (Sun) by mirabilos (subscriber, #84359) [Link] (3 responses)

I’m afraid your understanding has always been wrong, then.

The real realtime preemption end game

Posted Nov 19, 2023 18:40 UTC (Sun) by mirabilos (subscriber, #84359) [Link] (2 responses)

Hm, perhaps a bit more elaborating.

Yes, it’s not a persistent storage like the BIOS (or EFI) settings.

No, a boot does not imply memory cleaning (except for memory used during boot, of course). It usually does imply some kind of memory test, and several kinds of memory amount probing by different places in the boot process, but these are often nōn-intrusive enough to keep the memory contents.

A cold boot does have empty memory simply because the memory had no power and the memory controller likewise did not refresh the memory banks.

A warm reboot does not have a period of such, so the memory is *usually* retained.

A hard reboot can fall into either category, depending on how it is executed and wired. The usual power button long-press will be a poweroff followed by a mostly-cold boot; a watchdog reboot, or if the kernel crashed but is still able to reboot-ish (even if just by causing a triple-fault) can be warm reboots (this mostly depends on the memory controller to continue refreshing the memory during that, and of course the firmware not overwriting it).

The real realtime preemption end game

Posted Nov 20, 2023 19:57 UTC (Mon) by kreijack (guest, #43513) [Link] (1 responses)

> A warm reboot does not have a period of such, so the memory is *usually* retained.

I think that the key word is "*usually*". On my UEFI system I build a UEFI program that dump the first 4 bytes of the following address:
- 3GB
- 7GB
- 14GB

Then it sets these bytes to a specific value, and then it dump again.

What I saw is:
1) the first time that I run the program, I saw "random values", like 0 and other non 0 values.
2) the 2nd time that I run the program, I saw the same values that I set in the first iteration.

This proof that UEFI doesn't reset the memory between different program invocation.

Then I "warm rebooted" the system, and I saw the "random values" at 1). So it seemed that in my system the memory is cleared between the reboot.

What I'm telling is that at least some bios clears the memory. In may case (a ASUS B550 desktop mainboard) it seems that the BIOS clear the memory.

What I found is that it is possible to force the BIOS to not clear the memory after a reset [1]. But again this is not typically what happens after a crash; after a crash you push the reset physical buttons.

[1] https://stackoverflow.com/questions/36608101/does-a-soft-...

The real realtime preemption end game

Posted Nov 21, 2023 23:31 UTC (Tue) by mirabilos (subscriber, #84359) [Link]

Yes, that’s precisely what I meant with “usually”: a sufficient amount of systems keeps sufficient amounts of memory alive to make this feature worth being in existence, even if counter-examples exist and no spec supports this usage.

The reset button as the only way out of a crash is such a PC thing though. Some machines have watchdogs, and some have something like ddb(4) on BSD or SysRq on Linux that allow for warm reboots even in the face of a crash.

The real realtime preemption end game

Posted Nov 19, 2023 5:40 UTC (Sun) by mjg59 (subscriber, #23239) [Link] (1 responses)

It works as long as your firmware behaves in a specific way, something no specification requires of it.

The real realtime preemption end game

Posted Nov 19, 2023 18:22 UTC (Sun) by mirabilos (subscriber, #84359) [Link]

Yes, of course. Which is why I said that “it works in practice”: it works on a sufficiently large array of machines that OpenBSD (and probably NetBSD before) chose to implement it so, and even if it doesn’t work on one machine it’s no big loss.

The real realtime preemption end game

Posted Nov 19, 2023 9:40 UTC (Sun) by DemiMarie (subscriber, #164188) [Link] (4 responses)

In my expierience, pstore is the only reliable way to get a stack trace on panic, unless one is in a VM. Serial consoles don’t work because end-user systems don’t have them. Graphical consoles don’t work because the user is in an X11 or Wayland session.

How does Windows manage to display the BSOD message?

The real realtime preemption end game

Posted Nov 19, 2023 14:10 UTC (Sun) by Wol (subscriber, #4433) [Link]

> How does Windows manage to display the BSOD message?

I guess it just seizes control of the graphics card, or puts it into text mode, or whatever.

Cheers,
Wol

The real realtime preemption end game

Posted Nov 19, 2023 20:38 UTC (Sun) by ballombe (subscriber, #9523) [Link]

It displays the BSOD at start up, and then prints the normal screen as an overlay on top of it. This way, when something goes wrong, the overlay disappears and you see the BSOD. That is why the option "customize the BSOD" requires you to reboot. That is also why, if you set the background to an image with transparency, you get the BSOD.

(just jocking of course)

The real realtime preemption end game

Posted Nov 19, 2023 21:07 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (1 responses)

> How does Windows manage to display the BSOD message?

Windows drivers are much more resilient than the drivers in Linux. A surprising amount of functionality remains working in Windows even if half the kernel is going haywire.

In particular, modesetting and simple framebuffer access have always been a part of the kernel driver. And each driver is also responsible for pre-allocating its object pools, so there's much less dependency on memory allocation. The IRQL system also has a side effect of forcing driver writers to avoid putting anything too involved inside the critical pathways.

The real realtime preemption end game

Posted Dec 8, 2023 17:45 UTC (Fri) by pawel44 (guest, #162008) [Link]

Wishful thinking. USB driver failure will bring Windows down. Not to mention Windows Driver Model and Windows Driver Frameworks are holey like Swiss cheese.

The real realtime preemption end game

Posted Nov 24, 2023 21:06 UTC (Fri) by mtthu (subscriber, #123091) [Link]

This would maybe be doable if you have control over all stages of the boot process and the environment the kernel runs in. I guess it would be easier to be introduced in virtualized environments as a memory region could be mapped into a file on the host where it could be synced to disk before a restart. Memory integrity could be checked on that level as well, for example for the case that the host has an uncontrolled reboot.