That newfangled Journal thing

Posted Nov 20, 2011 18:48 UTC (Sun) by ejr (subscriber, #51652)
Parent article: That newfangled Journal thing

"Changing our grandest traditions" in one person's terminology is "breaking all our existing mechanisms" in another's. Some of the goals within "the Journal" could be met by producing formatted, textual syslog messages. Current systems wouldn't break, and not everyone need jump to a new system all at once. It has been tried and has failed repeatedly.

IME, it often has failed because having to decide on formats, etc. ahead of time places more work on the developer. Logging / diagnostics aren't supposed to be user-visible features, so they're treated like cost centers. Unformatted messages can be parsed and understood later, pushing the cost out to the future. I don't see how yet another interface changes that relationship.

Also, the developers' history of replacing everything wholesale and suffering various levels breakage during the initial time period has followed them into an area where various levels of breakage is not well tolerated. Some of those developers appear unwilling to listen to others' feedback on their methods regardless of differences in goals.

to post comments

Thank you...

Posted Nov 20, 2011 19:05 UTC (Sun) by khim (subscriber, #9252) [Link] (13 responses)

Wow. You just explaind in a few sentences why Journal is needed (but now why some particular design decisions are needed, of course).

Some of the goals within "the Journal" could be met by producing formatted, textual syslog messages. Current systems wouldn't break, and not everyone need jump to a new system all at once. It has been tried and has failed repeatedly.

Well, this is the best-possible pro-journald argument. If "let's all be friends" approach does not work then it's time for "it's my way or the highway" approach. If developers can not be convinced produce formatted syslog messages then perhaps they can be forced to produce formatted journald messages?

You are right when you say that developers don't think much about logging - but this is when they are left alone. If something is a requirement for admission to some group (repository, certification mark, etc) and if work is not too onerous then they will grumble but do that. If work becomes too hard then they may rethink the decision.

We'll see if journald will manage the right balance, but the fact that voluntary approaches failed in pro-journald argument, not counter-journald one!

Thank you...

Posted Nov 20, 2011 21:15 UTC (Sun) by zlynx (guest, #2285) [Link] (5 responses)

It doesn't work for Windows.

Sure, developers are forced to report their event log messages in the event log binary format. They are also supposed to produce DLLs that contain text identifiers for the log messages.

Guess how many of them do this?

I'm as guilty as anyone. I just used one identifier and stuffed a free-form text message into the binary data field. I wasn't about to go through every debug log print and make a unique ID for it. Especially since C++ iostreams don't lend themselves to that in any way.

Thank you...

Posted Nov 21, 2011 2:07 UTC (Mon) by alankila (guest, #47141) [Link]

I read that the uuid for journal message is optional. The recommendation, probably, is to add it, but of course if you do then you promise not to change the circumstances and meaning of the message, as you are basically declaring it as an API if you do.

Thank you...

Posted Nov 21, 2011 19:00 UTC (Mon) by faramir (subscriber, #2327) [Link] (3 responses)

Microsoft can't stop you from shipping your program if you don't follow the spirit of the rules.

Ubuntu/Fedora/Debian/etc. can effectively do this by refusing to include your program their core distribution.

That's not to say that the distributions would actually do this or that they wouldn't get push back from the developers of "important" packages. Still the differences between the Microsoft and Free Software worlds may make enforcement of good practices easier in the Free Software world.

Thank you...

Posted Nov 23, 2011 13:12 UTC (Wed) by sorpigal (subscriber, #36106) [Link]

Distributions don't have any more power than Microsoft to control what gets used on their OS. Distro packages are not the entire world.

Thank you...

Posted Dec 7, 2011 5:36 UTC (Wed) by Klavs (guest, #10563) [Link] (1 responses)

I will bet that a program, starting with logging everything with free form text to the same ID would, if used by sysadmins, quickly be improved upon, in regards to id logging. As a sysadmin I often have to look through source code to find the real reason for an error message (that's one big reason I prefer Open Source) - and if the program already used something with support for proper logging, like journald idea - I would easily whip up a patch for upstream to improve logging for the instance - helping myself in the future as well.

Thank you...

Posted Dec 7, 2011 10:35 UTC (Wed) by dlang (guest, #313) [Link]

what is it about the journal 'proper logging' that makes it any easier for you to write a patch to fix bad logging than with normal syslog?

normal syslog can deal with structured logging already via RFC 5324 structured logging, and the CEE specs standardize this further http://cee.mitre.org/

why are these not as good as the journal proposal?

Thank you...

Posted Nov 21, 2011 1:42 UTC (Mon) by jamesh (guest, #1159) [Link] (5 responses)

I doubt journald will be able to force users to produce structured messages.

There are many applications and services that produce messages through the syslog() C library call. Either journald will have to accept those messages and deal with some amount of unstructured messages, or you'll need to keep a traditional syslog daemon around and deal with two system logs.

I would be quite surprised if they decided to ignore messages generated by syslog() calls, so there will probably be a lot of unstructured messages for the foreseeable future.

That doesn't reduce the usefulness of converting a few key messages over to a new API though.

Thank you...

Posted Nov 21, 2011 6:19 UTC (Mon) by pilif (subscriber, #3857) [Link] (3 responses)

Reading the proposal, I would say that the structure doesn't come from the application that's doing the logging, but from the Journal itself. This is things like process id, user, date and everything else that can be inferred from an external standpoint (these are the _ prefixed fields in the spec)

What journal allows in addition to that is that the application can provide additional arbitrary key/value pairs (without the beginning _) in addition to the traditional free form message.

So a traditional syslog() sent message would still gain some structure, but if you usr the journal-native API, you even get to log application-specific values.

That said, I'm really torn on this proposal. I DO see many advantages, but on the other hand, NOTHING beats the accessibility of cat/grep and tail -f

Needing a special tool, just to look at logs is a huge increase in complexity. Especially if we are using a binary format. What if that file gets corrupted? You can still read a corrupted textfile, but can those future tools still read a corrupt binary file? Can the developers predict any possible type of corruption?

As I said: I'm torn.

I am looking forward however to see how this is going to play out.

Philip

Thank you...

Posted Nov 21, 2011 8:59 UTC (Mon) by niner (guest, #26151) [Link] (2 responses)

Do you really store your logfiles in plain text for all time? No logrotated with enabled compression?

Speaking for myself, my logfiles are already stored in a binary format. Even more, this format has changed over time! It used to be gzip, has been bzip2 for quite a while and nowadays is xz. Oh and yes, if these files are corrupt in some way, there's just no way to recover. But this has never been a problem. If random files are corrupted, I have a far greater problem than logfiles.

So I even nowadays need a special program to access the binary formatted logfiles: xzcat. On my wireless router it's called logread and I have no idea where logfiles even are stored...

Thank you...

Posted Nov 21, 2011 19:23 UTC (Mon) by cmccabe (guest, #60281) [Link] (1 responses)

A lot of people have compared using the journald binary format to using gzip to compress your logfiles. However, this comparison may be misleading.

gzip is a block compressor. You don't have to have access to the entire file to recover some of the data. You can actually seek to the beginning of a gzip block and start reading without having read all the data that came before.

A lot of media formats have this property as well-- mp3 for one. If the beginning of the mp3 is lost, you can still access the last part. Video formats often have 'key frames' that can be used to recover synchronization if the decoding got screwed up.

Being able to recover something, even if part of the file is messed up, is a very important property for log files. Disks can and do fail and start returning garbage-- I just had one turn this week. I hope that the journald designers think about issues like this, at least in later versions of the format.

Framing with FEC.

Posted Nov 21, 2011 19:43 UTC (Mon) by ejr (subscriber, #51652) [Link]

Indeed. A simple framing including some forward error correction code around some well-known, extensible encoding (maybe text, maybe protocol buffers, dunno) could prove even more resilient to failures than the plain-text logs. Same can be said for the text files used by dpkg, etc. VMS FTW?

Thank you...

Posted Nov 24, 2011 7:30 UTC (Thu) by kleptog (subscriber, #1183) [Link]

I'm really looking forward to what comes out of this project. The problem with structured data in your syslog log now is that every program does it differently and is therefore useless. We now have the opportunity you make a standard structure that can be supported everywhere.

A seriously, many many applications don't use syslog directly but some kind of wrapper, which allows you to say:

logfunction(severity, fmtstring, args...)

Look! Structure! Imagine being able to search log messages like: "all message of level NOTICE with 'sda' as an argument". Use the MD5 of the format string as ID and you've got something that people using translated programs can do something with. You could even translate your logs to another language after the fact.

Or, show the IP fields of all records with "Traceback" in the message.

Hurrah!

Thank you...

Posted Dec 21, 2011 7:15 UTC (Wed) by topher (guest, #2223) [Link]

If developers can not be convinced produce formatted syslog messages then perhaps they can be forced to produce formatted journald messages?

**buzz**

Thanks for playing, better luck next time.

This will not work. You can't force developers into something like that. Instead, they'll just dump their log data to a local file and ignore "the journal". Now you're left in a much worse situation than you were before, because you don't even have the logs hitting syslog.