Thank you...

Posted Nov 21, 2011 1:42 UTC (Mon) by jamesh (guest, #1159)
In reply to: Thank you... by khim
Parent article: That newfangled Journal thing

I doubt journald will be able to force users to produce structured messages.

There are many applications and services that produce messages through the syslog() C library call. Either journald will have to accept those messages and deal with some amount of unstructured messages, or you'll need to keep a traditional syslog daemon around and deal with two system logs.

I would be quite surprised if they decided to ignore messages generated by syslog() calls, so there will probably be a lot of unstructured messages for the foreseeable future.

That doesn't reduce the usefulness of converting a few key messages over to a new API though.

to post comments

Thank you...

Posted Nov 21, 2011 6:19 UTC (Mon) by pilif (subscriber, #3857) [Link] (3 responses)

Reading the proposal, I would say that the structure doesn't come from the application that's doing the logging, but from the Journal itself. This is things like process id, user, date and everything else that can be inferred from an external standpoint (these are the _ prefixed fields in the spec)

What journal allows in addition to that is that the application can provide additional arbitrary key/value pairs (without the beginning _) in addition to the traditional free form message.

So a traditional syslog() sent message would still gain some structure, but if you usr the journal-native API, you even get to log application-specific values.

That said, I'm really torn on this proposal. I DO see many advantages, but on the other hand, NOTHING beats the accessibility of cat/grep and tail -f

Needing a special tool, just to look at logs is a huge increase in complexity. Especially if we are using a binary format. What if that file gets corrupted? You can still read a corrupted textfile, but can those future tools still read a corrupt binary file? Can the developers predict any possible type of corruption?

As I said: I'm torn.

I am looking forward however to see how this is going to play out.

Philip

Thank you...

Posted Nov 21, 2011 8:59 UTC (Mon) by niner (guest, #26151) [Link] (2 responses)

Do you really store your logfiles in plain text for all time? No logrotated with enabled compression?

Speaking for myself, my logfiles are already stored in a binary format. Even more, this format has changed over time! It used to be gzip, has been bzip2 for quite a while and nowadays is xz. Oh and yes, if these files are corrupt in some way, there's just no way to recover. But this has never been a problem. If random files are corrupted, I have a far greater problem than logfiles.

So I even nowadays need a special program to access the binary formatted logfiles: xzcat. On my wireless router it's called logread and I have no idea where logfiles even are stored...

Thank you...

Posted Nov 21, 2011 19:23 UTC (Mon) by cmccabe (guest, #60281) [Link] (1 responses)

A lot of people have compared using the journald binary format to using gzip to compress your logfiles. However, this comparison may be misleading.

gzip is a block compressor. You don't have to have access to the entire file to recover some of the data. You can actually seek to the beginning of a gzip block and start reading without having read all the data that came before.

A lot of media formats have this property as well-- mp3 for one. If the beginning of the mp3 is lost, you can still access the last part. Video formats often have 'key frames' that can be used to recover synchronization if the decoding got screwed up.

Being able to recover something, even if part of the file is messed up, is a very important property for log files. Disks can and do fail and start returning garbage-- I just had one turn this week. I hope that the journald designers think about issues like this, at least in later versions of the format.

Framing with FEC.

Posted Nov 21, 2011 19:43 UTC (Mon) by ejr (subscriber, #51652) [Link]

Indeed. A simple framing including some forward error correction code around some well-known, extensible encoding (maybe text, maybe protocol buffers, dunno) could prove even more resilient to failures than the plain-text logs. Same can be said for the text files used by dpkg, etc. VMS FTW?

Thank you...

Posted Nov 24, 2011 7:30 UTC (Thu) by kleptog (subscriber, #1183) [Link]

I'm really looking forward to what comes out of this project. The problem with structured data in your syslog log now is that every program does it differently and is therefore useless. We now have the opportunity you make a standard structure that can be supported everywhere.

A seriously, many many applications don't use syslog directly but some kind of wrapper, which allows you to say:

logfunction(severity, fmtstring, args...)

Look! Structure! Imagine being able to search log messages like: "all message of level NOTICE with 'sda' as an argument". Use the MD5 of the format string as ID and you've got something that people using translated programs can do something with. You could even translate your logs to another language after the fact.

Or, show the IP fields of all records with "Traceback" in the message.

Hurrah!