Thank you...

Posted Nov 21, 2011 8:59 UTC (Mon) by niner (guest, #26151)
In reply to: Thank you... by pilif
Parent article: That newfangled Journal thing

Do you really store your logfiles in plain text for all time? No logrotated with enabled compression?

Speaking for myself, my logfiles are already stored in a binary format. Even more, this format has changed over time! It used to be gzip, has been bzip2 for quite a while and nowadays is xz. Oh and yes, if these files are corrupt in some way, there's just no way to recover. But this has never been a problem. If random files are corrupted, I have a far greater problem than logfiles.

So I even nowadays need a special program to access the binary formatted logfiles: xzcat. On my wireless router it's called logread and I have no idea where logfiles even are stored...

to post comments

Thank you...

Posted Nov 21, 2011 19:23 UTC (Mon) by cmccabe (guest, #60281) [Link] (1 responses)

A lot of people have compared using the journald binary format to using gzip to compress your logfiles. However, this comparison may be misleading.

gzip is a block compressor. You don't have to have access to the entire file to recover some of the data. You can actually seek to the beginning of a gzip block and start reading without having read all the data that came before.

A lot of media formats have this property as well-- mp3 for one. If the beginning of the mp3 is lost, you can still access the last part. Video formats often have 'key frames' that can be used to recover synchronization if the decoding got screwed up.

Being able to recover something, even if part of the file is messed up, is a very important property for log files. Disks can and do fail and start returning garbage-- I just had one turn this week. I hope that the journald designers think about issues like this, at least in later versions of the format.

Framing with FEC.

Posted Nov 21, 2011 19:43 UTC (Mon) by ejr (subscriber, #51652) [Link]

Indeed. A simple framing including some forward error correction code around some well-known, extensible encoding (maybe text, maybe protocol buffers, dunno) could prove even more resilient to failures than the plain-text logs. Same can be said for the text files used by dpkg, etc. VMS FTW?