The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 23:45 UTC (Mon) by nix (subscriber, #2304)
In reply to: The Grumpy Editor's guide to surviving the systemd debate by niner
Parent article: The Grumpy Editor's guide to surviving the systemd debate

Now, our strategy to rotate-on-corruption is the safest thing we can do, as we make sure that the internal corruption is frozen in time, and not attempted to be "fixed" by a tool, that might end up making things worse. After all, in the case the often-run writing code really fucks something up, then it is not necessarily a good idea to try to make it better by running a tool on it that tries to fix it up again, a tool that is necessarily a lot more complex, and also less tested.

I can't believe anyone thinks this argument has any merit at all. You could use the same words to argue that filesystems (a complex mass of binary data frequently written to which can suffer corruption) should never be repaired, because you can never get a fsck that always repairs all problems and never makes anything worse. Instead, we consider a filesystem sans fsck to be substandard.

Meanwhile, the format it's replacing, if corrupted, well, let's see -- you lose a line of syslog. Maybe you lose a bunch of other lines. But recovery from that is automatic; you only lose a line of syslog data around that touched by the corruption. You certainly don't lose a whole file. You *certainly* don't just rotate it away and, oh, I guess we should hope it wasn't an important log then.

This is not a sensible attitude to log data unless you consider log data fundamentally unimportant -- and in that case, why on earth are you recording it at all?

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 2:58 UTC (Thu) by raven667 (subscriber, #5198) [Link] (3 responses)

> Maybe you lose a bunch of other lines. But recovery from that is automatic; you only lose a line of syslog data around that touched by the corruption.

Maybe I misread but isn't that what journalctl does, skip invalid records and start back up at the next valid record, without touching the original file so you always have a pristine copy with the corruption intact? Logs are different than filesystems in that they are write once, read many, you never want to rewrite history, even if that history is corrupt due to a bug.

In fact if it were me, if you detect corruption on reading a journal log I would want to immediately kill journald and let systemd restart it, on startup the new copy can start a new log file Treat it like a microkernel server, die and reload quickly on error.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 3:47 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

with a binary file format, finding the start of the next message is not that easy.

As for killing journald when you _read_ a bad message, that sure isn't the right thing to do because you can read the same message many times, and do it years after it was written. killing journald when you read a message that's written a significant time in the past is the WRONG thing to do.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 4:28 UTC (Thu) by raven667 (subscriber, #5198) [Link]

sorry, I maybe should have been more clear, journalctl reads both the currently writing log and all previous ones that have been rotated and are still around, it should only restart journald if the corruption is in the log that journald is currently writing. The reason to implement this in the reader is that the I don't expect the log writer be reading often enough to notice if something has gone wrong.

I don't know if that's how it actually works, that's just my first idea of how I think it should work.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 21, 2014 9:15 UTC (Fri) by zlynx (guest, #2285) [Link]

Finding stuff in "binary" files is no more difficult than text files. It depends on the particular format.

People tend to forget that text files ARE A BINARY FORMAT. You have your 7-bit ASCII message terminated by a new-line character.

You could also have an array of 4 byte little-endian unsigned integers, terminated by 4 bytes of zero. It would be just as easy to find the start of the next message as it is in text.

We're pretty spoiled these days and don't have to worry about our messages getting off by a bit or two. I recall modems where if you weren't using parity (or was it stop bits) you might lose sync and anything could happen.

Anyway, there might be valid complaints about the particular design of journald's binary format. But just because it is binary doesn't necessarily make it unusually hard to recover or read.