That newfangled Journal thing

Posted Nov 21, 2011 10:37 UTC (Mon) by dlang (guest, #313)
In reply to: That newfangled Journal thing by tpo
Parent article: That newfangled Journal thing

you can have structured logging with syslog today, all you have to do is to structure the messages you write from your application (or from the kernel). Syslog even defines the format in the most recent RFC, so no changes are needed to do structured logging via syslog.

no need to run two logging systems in parallel, no need to figure out how to deal with logs that only get written to one of the two systems, just define the format of your logs are you are good to go.

That newfangled Journal thing

Posted Nov 21, 2011 12:46 UTC (Mon) by tpo (subscriber, #25713) [Link] (2 responses)

Are you refering to RFC5424 [1] when you are affirming that syslog defines the format of structured log entries? Or are there more comprehensive syslog standarts wrt structuring?

AFAI understand Journal identifies and solves the following additional problems:

*
there's no standartised API for reading or writing log entries (so I'd guess that each application that wants to do structured logging has to reinvent how to implement that (escaping, ordering of entries, sub-entries dependencies etc.))

*
there's no standartization on what fields (key/values) are mandatory and what the individual fields mean

Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

Is my view of affairs correct?

Also you write:

> no need to figure out how to deal with logs that only get written to one of the two

As far as I can see, if both systems were set up to feed into each other, as suggested, there would be no such thing as "logs that only get written to one of the two"? So I think this is not a base for critique?

I can see the additional cost of running both systems. However they would also yield a gain from standartized interfaces and semantics and unambiguous usage, as offered by the Journal?

[1] http://datatracker.ietf.org/doc/rfc5424/?include_text=1

That newfangled Journal thing

Posted Nov 23, 2011 23:06 UTC (Wed) by dlang (guest, #313) [Link]

RFC5424 defines a standard syntax, but not the semantics

look at the CEE standards process for standardisation of what the individual fields are named and what they should mean.

the sort of thing the CEE is doing only helps in a small subset of the cases, by definition you can only standardise what people have already thought of, so if you need to say something different (because your system works differently, so the meaning is a bit different), then you can't use the standardised semantics, but you can still use the standardised syntax

That newfangled Journal thing

Posted Nov 23, 2011 23:10 UTC (Wed) by dlang (guest, #313) [Link]

> Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

this is going to be the case no matter what structured format you use. there will always be programs that generate data that is almost, but not quite correct. In many cases such systems will get widely deployed before people start noticing (or caring ) that they are doing the wrong thing. In many cases the result is going to be learning to tolerate the wrong thing instead of refusing to interoperate with it.

This isn't just at the hardware level. Look at the things that Cisco has done 'differently' than any other vendor and you will see that the other vendors have had to adapt to Cisco or they got shut out entirely.