Thank you...

Posted Nov 21, 2011 6:19 UTC (Mon) by pilif (subscriber, #3857)
In reply to: Thank you... by jamesh
Parent article: That newfangled Journal thing

Reading the proposal, I would say that the structure doesn't come from the application that's doing the logging, but from the Journal itself. This is things like process id, user, date and everything else that can be inferred from an external standpoint (these are the _ prefixed fields in the spec)

What journal allows in addition to that is that the application can provide additional arbitrary key/value pairs (without the beginning _) in addition to the traditional free form message.

So a traditional syslog() sent message would still gain some structure, but if you usr the journal-native API, you even get to log application-specific values.

That said, I'm really torn on this proposal. I DO see many advantages, but on the other hand, NOTHING beats the accessibility of cat/grep and tail -f

Needing a special tool, just to look at logs is a huge increase in complexity. Especially if we are using a binary format. What if that file gets corrupted? You can still read a corrupted textfile, but can those future tools still read a corrupt binary file? Can the developers predict any possible type of corruption?

As I said: I'm torn.

I am looking forward however to see how this is going to play out.

Philip

to post comments

Thank you...

Posted Nov 21, 2011 8:59 UTC (Mon) by niner (guest, #26151) [Link] (2 responses)

Do you really store your logfiles in plain text for all time? No logrotated with enabled compression?

Speaking for myself, my logfiles are already stored in a binary format. Even more, this format has changed over time! It used to be gzip, has been bzip2 for quite a while and nowadays is xz. Oh and yes, if these files are corrupt in some way, there's just no way to recover. But this has never been a problem. If random files are corrupted, I have a far greater problem than logfiles.

So I even nowadays need a special program to access the binary formatted logfiles: xzcat. On my wireless router it's called logread and I have no idea where logfiles even are stored...

Thank you...

Posted Nov 21, 2011 19:23 UTC (Mon) by cmccabe (guest, #60281) [Link] (1 responses)

A lot of people have compared using the journald binary format to using gzip to compress your logfiles. However, this comparison may be misleading.

gzip is a block compressor. You don't have to have access to the entire file to recover some of the data. You can actually seek to the beginning of a gzip block and start reading without having read all the data that came before.

A lot of media formats have this property as well-- mp3 for one. If the beginning of the mp3 is lost, you can still access the last part. Video formats often have 'key frames' that can be used to recover synchronization if the decoding got screwed up.

Being able to recover something, even if part of the file is messed up, is a very important property for log files. Disks can and do fail and start returning garbage-- I just had one turn this week. I hope that the journald designers think about issues like this, at least in later versions of the format.

Framing with FEC.

Posted Nov 21, 2011 19:43 UTC (Mon) by ejr (subscriber, #51652) [Link]

Indeed. A simple framing including some forward error correction code around some well-known, extensible encoding (maybe text, maybe protocol buffers, dunno) could prove even more resilient to failures than the plain-text logs. Same can be said for the text files used by dpkg, etc. VMS FTW?