That newfangled Journal thing

Posted Nov 21, 2011 6:54 UTC (Mon) by k8to (guest, #15413)
Parent article: That newfangled Journal thing

Structured logging is a failure.

There's been so many attempts to build systems with structured logging (sometimes called 'messaging' or 'events') and always the system proves unable to anticipate the needs of real applications, and builds an inflexible overcomplicated system that no one wants.

That's why we still have text-stream-logging to this day.

Text stream logging offers some other benefits, such as simplicity, and possibility to track the stream without tight coupling.

Moreover most applications want to log to the filesystem, not to a special service. This means that most logging will be in the style of the text stream. Therefore most tools around logging will work with that, and this special journald binary format will just be in the way.

It's the wrong answer to a set of problems that don't really exist.

That newfangled Journal thing - some thinking required

Posted Nov 21, 2011 19:16 UTC (Mon) by ndye (guest, #9947) [Link] (1 responses)

Structured logging is a failure.

There's been so many attempts to build systems with structured logging (sometimes called 'messaging' or 'events') and always the system proves unable to anticipate the needs of real applications, and builds an inflexible overcomplicated system that no one wants.

That's why we still have text-stream-logging to this day.

can be, and has been, said many times:

XYZ has never worked, so it never will

Often over the "40 years" of UNIX, rethinking the fundamentals took place in a proprietary cathedral. The artificial limitation of market share limited the pool of engineers thinking about the problem and its interactions. The non-engineering folks required to think of "market" share so the engineers could eat were also driving anti-engineering efforts to limit the implementation to a segment of brain power, by which I mean at least these two forms of intellect segmentation:

people who, having learned one product, can't cross the chasm of artificial differences to adapt to a new environment (I'm looking at you, AIX)
people who must keep unnecessary differences in their head as they rove between vendors

Because we're in the shared ecospace of Linux, the preponderance of designers and users have opportunity to influence, beginning at design rather than late beta.

Not that I'm advocating this particular reinvention -- I find lots of this distasteful: UUID's, binary encoding, the coupling with systemd (solved?), overloading the "journal" name.

As someone else suggested, maybe Lennart should stop publishing his own thoughts and let someone more diplomatic front his ideas, vetting them for obvious mistakes like the name overloading.

Just in the last month I experienced an outage measured in days instead of hours precisely because of name overloading: It *matters*.

Here's hoping this topic stops reminding me of a maze of twisty passages, nearly alike but not inconsequentially different, strewn with pits, tar-filled and otherwise . . . .

That newfangled Journal thing - some thinking required

Posted Nov 24, 2011 8:19 UTC (Thu) by k8to (guest, #15413) [Link]

Yeah, except sometimes it's like poo-pooing people who try to reinvent bicycle wheels. Or people who try to put a CVT on a bicycle, or eliptical gear-wheels. None of them are better at all. In fact the bicycle wheel is one of the strongest per weight human structures ever designed. And it's cheap and easy to fix! But yet some people keep trying to add on carbon struts and such and sell it as better...

In this case, using a structured format means creating code that knows about the structured format. Now you've either got to do something gross like add all the journald interfaces to the kernel, or you have to do something else gross like add a big library between the emitting code and the fwrite() call.

You want to get the logged data out of the process space immediately, and the reason you want to do that is that in the crash scenario, you want to know what was happening at the time of the crash, not 2 seconds earlier when the buffering got last pushed.

Meanwhile, you want other systems to be able to tell what's logged, so you want to be able to watch the datastream as it's landing on disk. With a binary format all of this gets fairly complicated. Everyone has to link in this clunky library, or (worse) go through another daemon so that you have another place to lose buffers when the expected failure happens, to handle which is the first driver for adding logging.

So you have to have a clunky library and it has to be race condition free with its binary format, and it has to be carefully vetted to handle all unexpected binary structures correctly, or else tail -f becomes a security attack surface -- well it does regardless but at least if you carefully vet it the probability of a problem is decreased.

And the benefit of all of this is.. what exactly?

That newfangled Journal thing

Posted Nov 22, 2011 0:09 UTC (Tue) by alankila (guest, #47141) [Link] (3 responses)

Just because not every place needs structured logging doesn't mean that nobody needs it. And those places that do need it do benefit from journal format that can handle it and provide search facilities on top of it.

The *potential* to generate and find structured messages easily is valuable. There are immediate use cases, from usage statistics on services to automatically showing latest messages from log if daemon fails to start---it's not just more work for programmers for no good reason.

We wrote something like this at work because we needed structured log stream: events our users did, and variable messagetype-specific metadata attached to every log message. It made life simpler for programmers, helpdesk and the beancounters. From this, I would predict: the people who want structured messages are probably more than willing to peek into code to add them, because the alternative is continuing the current regex parsing regime, which is inelegant and error prone.

That newfangled Journal thing

Posted Nov 22, 2011 6:53 UTC (Tue) by k8to (guest, #15413) [Link] (2 responses)

You're talking about the case of the logging application producing messages with some kind of internal structure.

I'm talking about structure imposed by the logging interface.

It's a crucial difference. With text stream logging you can easily produce structured messages as needed by the application, whether it's simple and easily readable key-value pairs, or more elaborate and designed-for-program consumption formats like JSON or (ick) XML. Text stream logging lets you do this, it doesn't impose the limitation.

With a structured interface, you're locked into what that interface provides, and cannot grow beyond it or handle things it did not anticipate.

It's a classic case of trying to apply structure from the wrong side.

That newfangled Journal thing

Posted Nov 24, 2011 16:28 UTC (Thu) by alankila (guest, #47141) [Link] (1 responses)

I disagree & find the entire argument bizarre.

The proposed format is key-value storage. Any value can be your "text stream", thereby showing that the new approach is a superset of the old one.

That newfangled Journal thing

Posted Nov 24, 2011 19:45 UTC (Thu) by k8to (guest, #15413) [Link]

And if you need to create nested enclosures that open and close?

A single blob of text inside an enclosure is not a stream.