That newfangled Journal thing

Posted Nov 21, 2011 9:05 UTC (Mon) by tpo (subscriber, #25713)
In reply to: That newfangled Journal thing by elanthis
Parent article: That newfangled Journal thing

> Why does it have to be all or nothing?

Indeed. The comments to this article seem to be arguing:

I want syslog
I want structured logging

And there's a few that argue:

we can run syslog in parallel

However, the synthesis can go further than that:

Journal could also be accepting unstructured log messages and just stuff them into
```
   UUID: crap
   Structure: none at all
   Here's-the-content: bla bla
```
entries. In API language that would be:
```
   Journal.log_no_questions_asked("just the message here").
```
All other params would be reset to default (or "untidy crap").
on the other hand syslog could also receive messages directly from Journal. Of course journal could have a nice configuration facility where you can pick what gets output from Journal so that you only get what is of interest to you in your syslog log.

That way we have the best of both worlds (with additional HD space cost):

If needed one can do highly structured messages, which are well machine processible.
And we get at the same time an easily human readable/grep-able text.

And both contain all the information that are expected. With the bonus, that I can go and grep my syslog, and if I find out that I indeed need razor sharp data, then I can additionaly go and query the Journal.

I can't see why it's not possible to have this and why such a solution would be problematic for any of the "camps" above?

to post comments

That newfangled Journal thing

Posted Nov 21, 2011 10:37 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

you can have structured logging with syslog today, all you have to do is to structure the messages you write from your application (or from the kernel). Syslog even defines the format in the most recent RFC, so no changes are needed to do structured logging via syslog.

no need to run two logging systems in parallel, no need to figure out how to deal with logs that only get written to one of the two systems, just define the format of your logs are you are good to go.

That newfangled Journal thing

Posted Nov 21, 2011 12:46 UTC (Mon) by tpo (subscriber, #25713) [Link] (2 responses)

Are you refering to RFC5424 [1] when you are affirming that syslog defines the format of structured log entries? Or are there more comprehensive syslog standarts wrt structuring?

AFAI understand Journal identifies and solves the following additional problems:

*
there's no standartised API for reading or writing log entries (so I'd guess that each application that wants to do structured logging has to reinvent how to implement that (escaping, ordering of entries, sub-entries dependencies etc.))

*
there's no standartization on what fields (key/values) are mandatory and what the individual fields mean

Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

Is my view of affairs correct?

Also you write:

> no need to figure out how to deal with logs that only get written to one of the two

As far as I can see, if both systems were set up to feed into each other, as suggested, there would be no such thing as "logs that only get written to one of the two"? So I think this is not a base for critique?

I can see the additional cost of running both systems. However they would also yield a gain from standartized interfaces and semantics and unambiguous usage, as offered by the Journal?

[1] http://datatracker.ietf.org/doc/rfc5424/?include_text=1

That newfangled Journal thing

Posted Nov 23, 2011 23:06 UTC (Wed) by dlang (guest, #313) [Link]

RFC5424 defines a standard syntax, but not the semantics

look at the CEE standards process for standardisation of what the individual fields are named and what they should mean.

the sort of thing the CEE is doing only helps in a small subset of the cases, by definition you can only standardise what people have already thought of, so if you need to say something different (because your system works differently, so the meaning is a bit different), then you can't use the standardised semantics, but you can still use the standardised syntax

That newfangled Journal thing

Posted Nov 23, 2011 23:10 UTC (Wed) by dlang (guest, #313) [Link]

> Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

this is going to be the case no matter what structured format you use. there will always be programs that generate data that is almost, but not quite correct. In many cases such systems will get widely deployed before people start noticing (or caring ) that they are doing the wrong thing. In many cases the result is going to be learning to tolerate the wrong thing instead of refusing to interoperate with it.

This isn't just at the hardware level. Look at the things that Cisco has done 'differently' than any other vendor and you will see that the other vendors have had to adapt to Cisco or they got shut out entirely.