That newfangled Journal thing

Posted Nov 21, 2011 0:03 UTC (Mon) by hjh (guest, #4352)
Parent article: That newfangled Journal thing

The biggest problem I see from the comments and the proposals and what is understandably irritating people seems to be the unnecessary monolitic design, with all the resulting problems of:

* no way to generate pure text files
* no network service
* undocumented binary files
* no way to interface, but api calls
* not UNIX style

One obvious separation into a frontend, for receiving, annotating and generating log records, integrated into systemd, and a configurable back end would make good on the biggest concerns mentioned over and over.

The proposed storage design could still be the default backend and might still be integrated in systemd. But by additionally providing a pipe with a general text based standard record format to other external backends, the whole idea would become much more traditional UNIX style and less irritating, because it wouldn't force unnecessary policy on users.

One can envision: live monitoring, network transport, database, null, filter, live monitoring, alarm trigger, statistic and many other backends.

I can not belief that this would be much of a burden for the developers.

The second, often mentioned concern is missing speaking message ids. I can clearly see the benefits of UUIDs as well as those of speaking ids like reverse domain ids. But I think this could be solved, by allowing speaking ids in addition to required UUIDs as labels.

The third concern, security design, probably can be improved by some additional administrative measures, but the basic idea seem sane and is much better than no security at all. Security is a very tricky area and needs experts to comment, which I am not.

Despite those solvable problems, the journal proposal looks very very promising and has many valuable advantages which shouldn't be overlooked and the the argument we managed without is plain ..
..emmm... unconvincing.

to post comments

That newfangled Journal thing

Posted Nov 21, 2011 9:04 UTC (Mon) by nim-nim (subscriber, #34454) [Link] (1 responses)

> One obvious separation into a frontend, for receiving, annotating and
> generating log records, integrated into systemd, and a configurable back
> end would make good on the biggest concerns mentioned over and over.

This has been tried with gconf and didn't work too well.

The developpers wrote(and rewrote) the frontend so it was fast and easy to integrate in apps (their pet peeve) and the user-friendly backend that anyone could use without reading the source code never materialised.

Developpers don't want to worry about the backend, if you let them sell you an unfinished backend (we'll do better later, promise) you'll never see the replacement before the app dies.

That newfangled Journal thing

Posted Nov 21, 2011 11:56 UTC (Mon) by hjh (guest, #4352) [Link]

> The developers wrote(and rewrote) the frontend so it was fast and easy
> to integrate in apps (their pet peeve) and the user-friendly backend that
> anyone could use without reading the source code never materialised.

not if there is a simple documented streaming interface, to the backends that could even just as well stream into a file. Somthing with log messages like:

#time: <date> #id: org.kernel.subsys.facility.xyz #...
#uuid:<UUID> #other: <other> #future1: <future1>
_ mesg_line_1
_ mesg_lin_e2
_ ...
_ mesg_line_n

#time:<date> # .....
...

such a format should be possible to commit to for developers as it's expandable with new records elements in the future, and as it does not require to commit to more complicated backend storage design. Binaries like core dump shouldn't go to the log anyway, only as references to a file location. Admins concerned can opt out of the provided backend and
use files or a database instead to store the messages with some more
storage space required. I even think that using a database for indexing
purpose is the better solution, than yet another special database for logging only to better with compression.

The best would be, to separate the backend into a second process, which supports the compression, hashing and indexing stuff. The second process could then easily be located on a different host, and gets the records synchronously or in batch mode.

That newfangled Journal thing

Posted Jan 10, 2012 11:13 UTC (Tue) by spongy (guest, #59953) [Link]

you forgot a big bullet: no documentation.

That newfangled Journal thing

Posted Dec 6, 2013 8:49 UTC (Fri) by daniel (guest, #3181) [Link] (1 responses)

Let me tag onto your list of deficiences with a big, read, block letters, flashing, bold warning: might prove irresistible for developers to use log messages for messaging instead of their true purpose of error logging and audit. Which would be a fine recipe for a perfrect mess.

That newfangled Journal thing

Posted Dec 6, 2013 13:42 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

With all the filtering power journalctl has, wouldn't finding the errors be pretty much the same with and without lots of logging? Hell, you even get colors so you can just look for red and yellow messages. Sometimes sub-INFO-level debugging is useful; I don't see why supporting or encouraging it would be a downside.