The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 23:47 UTC (Mon) by nix (subscriber, #2304)
In reply to: The Grumpy Editor's guide to surviving the systemd debate by niner
Parent article: The Grumpy Editor's guide to surviving the systemd debate

File systems such as ext4 have an fsck tool since they don't have the luxury to just rotate the fs away and fix the structure on read: they have to use the same file system for all future writes, and they thus need to try hard to make the existing data workable again.

This is, of course, a fallacious argument too. File systems such as ext4 have a fsck tool because it would be an appalling implementation which said, oh, the FS is corrupted, we'll hit you with an empty one and let you get at the old one with some special commands! Which is more-or-less what systemd's journal does.

to post comments

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:07 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link]

That is not what the bug report claims:

"of course, having corrupted files isn't great, and we should make sure the files even when corrupted stay as accessible as possible. Hence: the code that reads the journal files is actually written in a way that tries to make the best of corrupted files, and tries to read of them as much as possible, with the the subset of the file that is still valid. We do this implicitly on every access."

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:15 UTC (Tue) by mchapman (subscriber, #66589) [Link] (7 responses)

> oh, the FS is corrupted, we'll hit you with an empty one and let you get at the old one with some special commands! Which is more-or-less what systemd's journal does.

No, it doesn't end up empty.

In my experience, a corrupted journal file is still completely readable and searchable, except for its final entry. Since the tools don't expose how individual journal files are being accessed, this corruption simply appears as if you'd lost a single log message.

Yes, it's possible some header corruption could mean an entire file is unreadable. I haven't personally seen that, but I can imagine that it's a concern for some people. You can limit the damage by ensuring the files get rotated frequently. There's {System,RunTime}MaxFileSize config options available; perhaps corresponding {System,RunTime}MaxRotateInterval might be useful. Worst comes to worst, you could just SIGUSR2 the journald process regularly from Cron or a timer unit.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:59 UTC (Tue) by dlang (guest, #313) [Link] (6 responses)

the instance of journald corruption that I saw ended up with the binary pointers going backwards and so walking the journal ended up with an endless loop

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 2:34 UTC (Tue) by mchapman (subscriber, #66589) [Link] (5 responses)

Right. I haven't seen that myself, but I could imagine it could happen.

But it sounds like something easily detectable by the libraries that read the journal files (so other apps, not just journalctl, would get the fix), and it still wouldn't result in an "empty" journal.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 4:03 UTC (Tue) by dlang (guest, #313) [Link] (4 responses)

It does make it so that all log entries after the corruption are lost though.

This is one of the problems with binary logs

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 5:04 UTC (Tue) by mchapman (subscriber, #66589) [Link] (3 responses)

Only up until the next rotation... which would presumably happen when you update systemd to fix the bug, if not earlier.

Look, I'm not denying that there are bugs that can corrupt the logs or lose large numbers of log messages. But the original claim -- that corruption yields an "empty" journal that can only be made non-empty through the use of "special commands" -- is patently false.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 11:59 UTC (Tue) by notninjaz (guest, #99725) [Link] (2 responses)

As I read it, the original "empty" statement was referring to the new journal which would be empty as in a clean slate, comparing it to offering the user a new clean slate filesystem rather than trying to repair a corrupted one.

This thread has brought out some of the tradeoffs, though, such as a recommendation to rotate the log frequently to minimize the impact of possible corruption. For the servers I administer, I would prefer logging to a text file and risking loss of a fraction of a second of logs if an entry is corrupt instead of rotating a journal hourly and potentially losing 59 minutes should there be corruption.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 13:14 UTC (Tue) by mchapman (subscriber, #66589) [Link] (1 responses)

> As I read it, the original "empty" statement was referring to the new journal which would be empty as in a clean slate, comparing it to offering the user a new clean slate filesystem rather than trying to repair a corrupted one.

Maybe that was what it was referring to. At any rate, it's just not true. When a journal file is rotated (whether that be because journald has detected corruption in it, or because it's reached its maximum size, or because the admin has simply asked for it by sending SIGUSR2 to the journald process), its contents are still read automatically by journalctl when searching and iterating through the logs.

The journal is not a single file; it is the sum total of *all* journal files. As far as I know, the only way to create an empty journal is to literally remove all of these journal files.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 14:51 UTC (Tue) by notninjaz (guest, #99725) [Link]

Right, I was referencing the statement: "Yes, it's possible some header corruption could mean an entire file is unreadable. I haven't personally seen that, but I can imagine that it's a concern for some people. You can limit the damage by ensuring the files get rotated frequently. There's {System,RunTime}MaxFileSize config options available; perhaps corresponding {System,RunTime}MaxRotateInterval might be useful. Worst comes to worst, you could just SIGUSR2 the journald process regularly from Cron or a timer unit."

I meant to express a design preference rather to critique any particular software with regard to text vs. binary logs.