> I'm not sure requiring HW is a non-starter. It would
> have to be cheap---say $20 or less---to start with.
Here's an idea that I had a while ago: make a USB dongle that appears to be a USB-to-serial converter. Data that is sent to it is recorded in its flash; you configure your system to send log messages to it like a serial console.
My aim was to have a "dying breath" log for machines where networked logging wasn't feasible, e.g. a single co-located server. The idea is that if there is a kernel panic and the critical last log messages don't make it to the disk, they might still make it to this device. You could then re-mount it, potentially on another machine, where it would appear as a storage device containing the log files.
You could make this relatively secure by not allowing a transition from logging mode to read mode without re-plugging.
This could be implemented by a simplish microcontroller with USB device functionality. Maybe a bit more than a PIC, but not much more.
One thing that I was unsure of was how likely it would be that the host's USB system would keep running for long enough after a kernel crash. Perhaps a real serial device, or a console video recorder, would work in more cases. This device would be rather pointless if it were no more reliable than the disk.