The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 2:58 UTC (Thu) by raven667 (subscriber, #5198)
In reply to: The Grumpy Editor's guide to surviving the systemd debate by nix
Parent article: The Grumpy Editor's guide to surviving the systemd debate

> Maybe you lose a bunch of other lines. But recovery from that is automatic; you only lose a line of syslog data around that touched by the corruption.

Maybe I misread but isn't that what journalctl does, skip invalid records and start back up at the next valid record, without touching the original file so you always have a pristine copy with the corruption intact? Logs are different than filesystems in that they are write once, read many, you never want to rewrite history, even if that history is corrupt due to a bug.

In fact if it were me, if you detect corruption on reading a journal log I would want to immediately kill journald and let systemd restart it, on startup the new copy can start a new log file Treat it like a microkernel server, die and reload quickly on error.

to post comments

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 3:47 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

with a binary file format, finding the start of the next message is not that easy.

As for killing journald when you _read_ a bad message, that sure isn't the right thing to do because you can read the same message many times, and do it years after it was written. killing journald when you read a message that's written a significant time in the past is the WRONG thing to do.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 4:28 UTC (Thu) by raven667 (subscriber, #5198) [Link]

sorry, I maybe should have been more clear, journalctl reads both the currently writing log and all previous ones that have been rotated and are still around, it should only restart journald if the corruption is in the log that journald is currently writing. The reason to implement this in the reader is that the I don't expect the log writer be reading often enough to notice if something has gone wrong.

I don't know if that's how it actually works, that's just my first idea of how I think it should work.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 21, 2014 9:15 UTC (Fri) by zlynx (guest, #2285) [Link]

Finding stuff in "binary" files is no more difficult than text files. It depends on the particular format.

People tend to forget that text files ARE A BINARY FORMAT. You have your 7-bit ASCII message terminated by a new-line character.

You could also have an array of 4 byte little-endian unsigned integers, terminated by 4 bytes of zero. It would be just as easy to find the start of the next message as it is in text.

We're pretty spoiled these days and don't have to worry about our messages getting off by a bit or two. I recall modems where if you weren't using parity (or was it stop bits) you might lose sync and anything could happen.

Anyway, there might be valid complaints about the particular design of journald's binary format. But just because it is binary doesn't necessarily make it unusually hard to recover or read.