LWN: Comments on "The real realtime preemption end game"

The real realtime preemption end game

pawel44 — Fri, 08 Dec 2023 17:45:45 +0000

Wishful thinking. USB driver failure will bring Windows down. Not to mention Windows Driver Model and Windows Driver Frameworks are holey like Swiss cheese.

The real realtime preemption end game

mtthu — Fri, 24 Nov 2023 21:06:55 +0000

This would maybe be doable if you have control over all stages of the boot process and the environment the kernel runs in. I guess it would be easier to be introduced in virtualized environments as a memory region could be mapped into a file on the host where it could be synced to disk before a restart. Memory integrity could be checked on that level as well, for example for the case that the host has an uncontrolled reboot.

The real realtime preemption end game

mirabilos — Tue, 21 Nov 2023 23:31:25 +0000

Yes, that’s precisely what I meant with “usually”: a sufficient amount of systems keeps sufficient amounts of memory alive to make this feature worth being in existence, even if counter-examples exist and no spec supports this usage.

The reset button as the only way out of a crash is such a PC thing though. Some machines have watchdogs, and some have something like ddb(4) on BSD or SysRq on Linux that allow for warm reboots even in the face of a crash.

The real realtime preemption end game

kreijack — Mon, 20 Nov 2023 19:57:41 +0000

> A warm reboot does not have a period of such, so the memory is *usually* retained.

I think that the key word is "*usually*". On my UEFI system I build a UEFI program that dump the first 4 bytes of the following address:
- 3GB
- 7GB
- 14GB

Then it sets these bytes to a specific value, and then it dump again.

What I saw is:
1) the first time that I run the program, I saw "random values", like 0 and other non 0 values.
2) the 2nd time that I run the program, I saw the same values that I set in the first iteration.

This proof that UEFI doesn't reset the memory between different program invocation.

Then I "warm rebooted" the system, and I saw the "random values" at 1). So it seemed that in my system the memory is cleared between the reboot.

What I'm telling is that at least some bios clears the memory. In may case (a ASUS B550 desktop mainboard) it seems that the BIOS clear the memory.

What I found is that it is possible to force the BIOS to not clear the memory after a reset [1]. But again this is not typically what happens after a crash; after a crash you push the reset physical buttons.

[1] https://stackoverflow.com/questions/36608101/does-a-soft-...

The real realtime preemption end game

Cyberax — Sun, 19 Nov 2023 21:07:14 +0000

> How does Windows manage to display the BSOD message?

Windows drivers are much more resilient than the drivers in Linux. A surprising amount of functionality remains working in Windows even if half the kernel is going haywire.

In particular, modesetting and simple framebuffer access have always been a part of the kernel driver. And each driver is also responsible for pre-allocating its object pools, so there's much less dependency on memory allocation. The IRQL system also has a side effect of forcing driver writers to avoid putting anything too involved inside the critical pathways.

The real realtime preemption end game

ballombe — Sun, 19 Nov 2023 20:38:44 +0000

It displays the BSOD at start up, and then prints the normal screen as an overlay on top of it. This way, when something goes wrong, the overlay disappears and you see the BSOD. That is why the option "customize the BSOD" requires you to reboot. That is also why, if you set the background to an image with transparency, you get the BSOD.

(just jocking of course)

The real realtime preemption end game

mirabilos — Sun, 19 Nov 2023 18:40:28 +0000

Hm, perhaps a bit more elaborating.

Yes, it’s not a persistent storage like the BIOS (or EFI) settings.

No, a boot does not imply memory cleaning (except for memory used during boot, of course). It usually does imply some kind of memory test, and several kinds of memory amount probing by different places in the boot process, but these are often nōn-intrusive enough to keep the memory contents.

A cold boot does have empty memory simply because the memory had no power and the memory controller likewise did not refresh the memory banks.

A warm reboot does not have a period of such, so the memory is *usually* retained.

A hard reboot can fall into either category, depending on how it is executed and wired. The usual power button long-press will be a poweroff followed by a mostly-cold boot; a watchdog reboot, or if the kernel crashed but is still able to reboot-ish (even if just by causing a triple-fault) can be warm reboots (this mostly depends on the memory controller to continue refreshing the memory during that, and of course the firmware not overwriting it).

The real realtime preemption end game

mirabilos — Sun, 19 Nov 2023 18:35:22 +0000

I’m afraid your understanding has always been wrong, then.

The real realtime preemption end game

kreijack — Sun, 19 Nov 2023 18:29:23 +0000

My understanding is that (at least in the x86 world), the memory is cleaned during the reboot.

So the problem is not to find a fixed area where store the data, but avoid that this area is cleaned up during a reboot.
And this cannot be done in a generic way.

The kind of reboot that I am talking, is the one that allow you to exit from a "crash", so I think that we are talking about an hard reboot. And an hard reboot implies the memory cleanup.
Think if this wouldn't exists: this would allow to extract from the memory some secret with a simple reboot at the "right time"; it would be a giant security hole.

The pstore back-ends in the x86 are mostly two: the first one relies on the UEFI variable storage; the second one relies on the ACPI-ERST, which is like a flash memory.

The real realtime preemption end game

mirabilos — Sun, 19 Nov 2023 18:22:14 +0000

Yes, of course. Which is why I said that “it works in practice”: it works on a sufficiently large array of machines that OpenBSD (and probably NetBSD before) chose to implement it so, and even if it doesn’t work on one machine it’s no big loss.

The real realtime preemption end game

Wol — Sun, 19 Nov 2023 14:10:54 +0000

> How does Windows manage to display the BSOD message?

I guess it just seizes control of the graphics card, or puts it into text mode, or whatever.

Cheers,
Wol

The real realtime preemption end game

DemiMarie — Sun, 19 Nov 2023 09:40:41 +0000

In my expierience, pstore is the only reliable way to get a stack trace on panic, unless one is in a VM. Serial consoles don’t work because end-user systems don’t have them. Graphical consoles don’t work because the user is in an X11 or Wayland session.

How does Windows manage to display the BSOD message?

The real realtime preemption end game

mjg59 — Sun, 19 Nov 2023 05:40:34 +0000

It works as long as your firmware behaves in a specific way, something no specification requires of it.

The real realtime preemption end game

mirabilos — Sun, 19 Nov 2023 05:17:17 +0000

Heh, and of course just as I posted this, the hoster fell off the ’net (including not only the hoster’s own homepage but also their status page, which is not hosted at a different site as is usual).

Oh well, it’ll be back at some point.

The real realtime preemption end game

mirabilos — Sun, 19 Nov 2023 05:15:42 +0000

How much money are you willing to hand over?

https://mbsd.evolvis.org/cvs.cgi/src/sys/kern/subr_log.c?...
(I’m using a somewhat beefier mirror here to not get the main server slashdotted)
look for initmsgbuf near the beginning of the file, which gets a pointer to the RAM region.

It is called for SPARC from:
https://mbsd.evolvis.org/cvs.cgi/src/sys/arch/sparc/sparc...
(initmsgbuf called with an almost fixed (only the oldest systems avoid the first page) address…)

For i386, the call is at…
https://mbsd.evolvis.org/cvs.cgi/src/sys/arch/i386/i386/m...
… where msgbufp comes from…
https://mbsd.evolvis.org/cvs.cgi/src/sys/arch/i386/i386/p...
(the __OpenBSD__ ifdef) which sets the virtual address. The physical address (MMU mapping) is done somewhere between locore.s and there, and it looks to me like its location depends on the size of the kernel image, so you’d only get the log messages if you boot the same or a very similar-sized kernel after warm reboot.

And yes, it’s purely memory-based. It helps immensely in copying e.g. the remainder of a ddb(4) session (in-kernel debugger) out if you don’t have a serial console.

The real realtime preemption end game

Paf — Sun, 19 Nov 2023 03:20:15 +0000

“ It works in practice on BSD, dmesg shows the previous run’s messages as well.”

And surely this is only possible through the retention of data in memory over reboot! What other magic could do this?

Sorry, but I’d lay a lot of money this is done with storage.

The real realtime preemption end game

mirabilos — Sat, 18 Nov 2023 05:36:14 +0000

It works in practice on BSD, dmesg shows the previous run’s messages as well.

In practice here means x86 hardware like Thinkpads and other assorted PCs and servers whose BIOS will not overwrite the entire memory during warm reboot, as well as SPARCstations whose OpenBoot will similarily not clear the high-up memory used for the kernel log buffer.

The real realtime preemption end game

donald.buczek — Fri, 17 Nov 2023 06:23:23 +0000

Oh, and of course everybody had to wait for you, while you carefully read through the secret scripture, which nobody but you could understand, until you decide to enter "b <ret>", which initiated thr printing of another few pages of startup messages during the next 10 minutes :-)

The real realtime preemption end game

donald.buczek — Fri, 17 Nov 2023 05:51:49 +0000

Nothing tops the feeling when a bugcheck on vax/vms system causes all these hex dumps to be printed slowly and noisily to the paper of your hardcopy console. Physical printout and slowness signifies importance while stackdumps with hex values signify secret knowledge. This is not for mortals.

The real realtime preemption end game

mjg59 — Fri, 17 Nov 2023 05:33:39 +0000

kexec certainly provides a mechanism for preserving memory ranges over kernel switches, but it's not really what most people would describe as reboot

The real realtime preemption end game

areilly — Fri, 17 Nov 2023 01:30:55 +0000

I remember a microVax that I used as an undergrad that had a real teletype set up as the (serial) console. Had the effect of (a) rebooting the machine if it was ever accidentally turned off, and (b) halting the machine whenever it ran out of paper. I was always amazed that the Ultrix would keep going as though nothing had happened as soon as more paper was loaded and it was put back on line.

The real realtime preemption end game

itsmycpu — Fri, 17 Nov 2023 00:47:50 +0000

Maybe something like this could preserve a range of memory? However I wouldn't know if that could be interesting in this context...

https://en.wikipedia.org/wiki/Reboot#Warm

"The Linux family of operating systems supports an alternative to warm boot; the Linux kernel has optional support for kexec, a system call which transfers execution to a new kernel and skips hardware or firmware reset. The entire process occurs independently of the system firmware. The kernel being executed does not have to be a Linux kernel.[citation needed]"

The real realtime preemption end game

mjg59 — Thu, 16 Nov 2023 22:53:37 +0000

In general? No. You have no guarantees about the behaviour of the firmware over reboot, and it's legitimate for it to just wipe the entire contents of RAM before booting anything else. But there are specific cases where this can be guaranteed - check the various pstore backends for examples.

The real realtime preemption end game

itsmycpu — Thu, 16 Nov 2023 21:44:13 +0000

That has me wondering... in general, is it possible to create a protected memory space that survives rebooot?

The real realtime preemption end game

IanKelling — Thu, 16 Nov 2023 16:17:21 +0000

https://wiki.linuxfoundation.org/realtime/start link to latest development version patch, cloc says:

Language                     files          blank        comment           code
-------------------------------------------------------------------------------
diff                             1           1899           5676           8032

That is pretty small. I've enjoyed reading about this over the years.

The real realtime preemption end game

grawity — Thu, 16 Nov 2023 14:17:23 +0000

> printk(), Gleixner said, is fully synchronous in current kernels; a call will not return until the message has been sent to all of the configured destinations.

I remember when I was managing a large Linux-based gateway, and I configured serial console (over IPMI), and later I added some iptables LOG rules, and it turned out that just a few matching packets per second would DoS it because it wasn't processing any packets while waiting for each log message to go out through ttyS1...