That newfangled Journal thing

Posted Nov 20, 2011 21:27 UTC (Sun) by dlang (guest, #313)
Parent article: That newfangled Journal thing

> This proposal is a request for comments; it has garnered no end of those. What is interesting to see, though, is that a lot of these comments take the form of "syslog has been good for Unix for 40 years, this proposal is trying to change one of our grandest traditions." There is a lot to value in the Unix tradition, but we do need to be careful about sticking to it just because That's How Things Have Always Been Done. Even longstanding mechanisms are subject to improvement.

Yes there is room to improve log handling, but there is no need to throw away everything that is good about the existing systems and force everything to re-write it's logging in the process.

If you look at rsyslog and syslog-ng you will see that there has been a lot of improvement in 'traditional' logging over the last several years. Most of this improvement is invisible to desktop users because they just don't care about it, but it is very visible to people who run server farms and actually use their logs on a regular basis

There have been many people who have advocated forcing all applications (and the kernel) to change their logs to make them easier to identify and parse. There are even standards on how to do this (the latest syslog RFC defines key-value pair formatting of log messages and CEE defines standards for what to put in the log messages), but application and kernel programmers do not choose to make the effort to standardize their output. I don't see any way that yet another logging mechanism would make it any more likely that the messages would be well standardized.

One thing that systemd is in a unique position to do, is to define a custom /dev/log for each service that it starts, so that it could insert a tag into each message generated by the program claiming that systemd has validated what service it came from, this could be done without any need to introduce any new logging mechanism, just create a simple pass-though that tweaks the message before handing it off the the existing syslog daemon.

If the messages are well formatted, it is very easy for modern syslog tools to do very powerful things with them. logwatch and simple event correlator (SEC) have very complex rules most of the time because they are dealing with poorly formatted log messages, but if you can feed them well formatted log messages they can have rather simple configurations.

to post comments

That newfangled Journal thing

Posted Nov 20, 2011 23:19 UTC (Sun) by HelloWorld (guest, #56129) [Link] (13 responses)

> Yes there is room to improve log handling, but there is no need to throw away everything that is good about the existing systems and force everything to re-write it's logging in the process.
The proposal mentions that one can use syslog(3) to generate log messages for journald.

That newfangled Journal thing

Posted Nov 20, 2011 23:42 UTC (Sun) by dlang (guest, #313) [Link] (12 responses)

and if someone does so, what is going to make the logs any more structured than the logs created by syslog(3) today?

That newfangled Journal thing

Posted Nov 21, 2011 0:04 UTC (Mon) by elanthis (guest, #6227) [Link] (8 responses)

Why does it have to be all or nothing?

The inability or unwillingness of most LWN users to think like a damn programmer always confuses me.

That newfangled Journal thing

Posted Nov 21, 2011 0:16 UTC (Mon) by dlang (guest, #313) [Link]

well, there's nothing preventing any programmer from writing good, well structured logs today (there are even RFC standards for how to do so)

but the claim is that this new tool will 'solve' the problem of poorly structured logs. If it doesn't actually prevent the programmer from writing poorly structured logs, then that needs to be called out as that claim is false.

That newfangled Journal thing

Posted Nov 21, 2011 0:40 UTC (Mon) by jubal (subscriber, #67202) [Link] (1 responses)

Ah, I always tried to to think like a saved programmer, you see. So you're saying that damnation is the key to understand a developer?

That newfangled Journal thing

Posted Nov 21, 2011 1:51 UTC (Mon) by elanthis (guest, #6227) [Link]

:) The games industry is basically all Windows, so I'm only really familiar with damned programmers these days. :)

That newfangled Journal thing

Posted Nov 21, 2011 9:05 UTC (Mon) by tpo (subscriber, #25713) [Link] (4 responses)

> Why does it have to be all or nothing?

Indeed. The comments to this article seem to be arguing:

I want syslog
I want structured logging

And there's a few that argue:

we can run syslog in parallel

However, the synthesis can go further than that:

Journal could also be accepting unstructured log messages and just stuff them into
```
   UUID: crap
   Structure: none at all
   Here's-the-content: bla bla
```
entries. In API language that would be:
```
   Journal.log_no_questions_asked("just the message here").
```
All other params would be reset to default (or "untidy crap").
on the other hand syslog could also receive messages directly from Journal. Of course journal could have a nice configuration facility where you can pick what gets output from Journal so that you only get what is of interest to you in your syslog log.

That way we have the best of both worlds (with additional HD space cost):

If needed one can do highly structured messages, which are well machine processible.
And we get at the same time an easily human readable/grep-able text.

And both contain all the information that are expected. With the bonus, that I can go and grep my syslog, and if I find out that I indeed need razor sharp data, then I can additionaly go and query the Journal.

I can't see why it's not possible to have this and why such a solution would be problematic for any of the "camps" above?

That newfangled Journal thing

Posted Nov 21, 2011 10:37 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

you can have structured logging with syslog today, all you have to do is to structure the messages you write from your application (or from the kernel). Syslog even defines the format in the most recent RFC, so no changes are needed to do structured logging via syslog.

no need to run two logging systems in parallel, no need to figure out how to deal with logs that only get written to one of the two systems, just define the format of your logs are you are good to go.

That newfangled Journal thing

Posted Nov 21, 2011 12:46 UTC (Mon) by tpo (subscriber, #25713) [Link] (2 responses)

Are you refering to RFC5424 [1] when you are affirming that syslog defines the format of structured log entries? Or are there more comprehensive syslog standarts wrt structuring?

AFAI understand Journal identifies and solves the following additional problems:

*
there's no standartised API for reading or writing log entries (so I'd guess that each application that wants to do structured logging has to reinvent how to implement that (escaping, ordering of entries, sub-entries dependencies etc.))

*
there's no standartization on what fields (key/values) are mandatory and what the individual fields mean

Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

Is my view of affairs correct?

Also you write:

> no need to figure out how to deal with logs that only get written to one of the two

As far as I can see, if both systems were set up to feed into each other, as suggested, there would be no such thing as "logs that only get written to one of the two"? So I think this is not a base for critique?

I can see the additional cost of running both systems. However they would also yield a gain from standartized interfaces and semantics and unambiguous usage, as offered by the Journal?

[1] http://datatracker.ietf.org/doc/rfc5424/?include_text=1

That newfangled Journal thing

Posted Nov 23, 2011 23:06 UTC (Wed) by dlang (guest, #313) [Link]

RFC5424 defines a standard syntax, but not the semantics

look at the CEE standards process for standardisation of what the individual fields are named and what they should mean.

the sort of thing the CEE is doing only helps in a small subset of the cases, by definition you can only standardise what people have already thought of, so if you need to say something different (because your system works differently, so the meaning is a bit different), then you can't use the standardised semantics, but you can still use the standardised syntax

That newfangled Journal thing

Posted Nov 23, 2011 23:10 UTC (Wed) by dlang (guest, #313) [Link]

> Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

this is going to be the case no matter what structured format you use. there will always be programs that generate data that is almost, but not quite correct. In many cases such systems will get widely deployed before people start noticing (or caring ) that they are doing the wrong thing. In many cases the result is going to be learning to tolerate the wrong thing instead of refusing to interoperate with it.

This isn't just at the hardware level. Look at the things that Cisco has done 'differently' than any other vendor and you will see that the other vendors have had to adapt to Cisco or they got shut out entirely.

That newfangled Journal thing

Posted Nov 21, 2011 0:41 UTC (Mon) by HelloWorld (guest, #56129) [Link] (2 responses)

> and if someone does so, what is going to make the logs any more structured than the logs created by syslog(3) today?
You can't make something out of nothing.

That newfangled Journal thing

Posted Nov 21, 2011 1:17 UTC (Mon) by dlang (guest, #313) [Link] (1 responses)

I'm not understanding your comment. are you agreeing that the journal won't help enforce formatting? or are you claiming that syslog messages cannot ever be well formatted? (or something else)

That newfangled Journal thing

Posted Nov 21, 2011 3:23 UTC (Mon) by HelloWorld (guest, #56129) [Link]

The point I was trying to make is that you can't expect the journal to generate structured logging messages from programs which weren't written with structured logging in mind.