The journald design is horrible to the point of useless

Posted Dec 2, 2011 4:36 UTC (Fri) by dlang (guest, #313)
In reply to: The journald design is horrible to the point of useless by anselm
Parent article: That newfangled Journal thing

I have never said that LP is an idiot, If he was I wouldn't care what he said as such people don't influence Linux much.

I do believe that he is wrong, and I give him the benefit of the doubt and assume that he failed to research the current state of the art around syslog
(and the fact that Rainer says that LP never talked to him about the journal reinforces this), because otherwise he is being deliberately misleading in his statements.

The fact that anyone would propose such a drastic change without learning the state of the art significantly undermines the credibility of their paper as far as I am concerned.

syslog-ng and rsyslog are both commonly used on *BSD and solaris, so while you may want to do research to see if there are missing tweaks, others have done so already.

there are many different ways of storing logs, different ways have different benefits and drawbacks when accessing the logs

the traditional text files rolled periodically are great for looking at what happened around a particular time (including especially, what is happening now, or has happened just recently)

However, they are horrible at doing searches for previously unexpected information over a large volume of logs.

If you know what you are expecting to be looking for, you can tailor your log store to make it easy to find that, but assuming that you don't want to do so, a couple 'off the shelf' approaches to rapidly finding things in large log volumes are:

1. log to a postrgesql database and enable the full-text indexing (I believe it's called gist indexing), this will let you search rapidly for any text

2. log to hadoop and use it's search capabilities (including clustering) to search the logs.

in both cases this is a little more than just 'apt-get magic-tool', because you need to configure the database and then configure rsyslog to write to the database, but all the pieces are there, it really is just a matter of configuring them.

this is more than just "yes we could do that if we wanted", it's "we already created the tools to do that, and people are using the tools to do that, you can do the same"

to post comments

The journald design is horrible to the point of useless

Posted Dec 2, 2011 9:04 UTC (Fri) by anselm (subscriber, #2796) [Link] (8 responses)

syslog-ng and rsyslog are both commonly used on *BSD and solaris, so while you may want to do research to see if there are missing tweaks, others have done so already.

Yes, but these are the open-source Unices. I'd be more worried about the likes of MacOS X, AIX, or HP/UX.

the traditional text files rolled periodically are great for looking at what happened around a particular time (including especially, what is happening now, or has happened just recently)

Great in that the messages in question do end up near one another. Not so great in that you need to scan a file (or a whole bunch of files) sequentially in order to find where that »particular time« is in the log. (One could probably do per-file binary search based on the log time stamps but if there is a tool to do that it is definitely not in widespread use.) Also it turns out that, like in the e-mail case, related log messages do not always happen to occur around a particular time.

I still like the systemd/journald idea of including the last few lines of log output produced by a system service in a »service status« output. This is wildly impractical under the current scheme since there is no way to tell in which file(s) a service's output even ends up (short of parsing the syslogd configuration), and even then you need to scan the file(s) sequentially to find what you're interested in. You end up having to tweak all your init scripts and/or reinvent large parts of your logging infrastructure. Systemd and journald suddenly start looking pretty sweet by comparison. (And yes, the stock answer to that is »We didn't need this for the last 30 years, it is unnecessary, nobody should bother, systemd sucks.«)

in both cases this is a little more than just 'apt-get magic-tool', because you need to configure the database and then configure rsyslog to write to the database, but all the pieces are there, it really is just a matter of configuring them.

And if the pieces weren't there, it would really just be a matter of putting them in? Seriously, I think you're making it a bit too easy on yourself here. Especially since the real problem is not logging stuff to a database (which rsyslogd can do if you ask it nicely), it is getting at the stuff afterwards which again is something that Lennart and Kay do acknowledge in their proposal. Getting rid of old stuff would also be something one would have to consider at some point. Full-text search in databases is nice but it would be even better to be able to exploit the structure that some types of log do have, that RFC 5424 suggests, and that journald pretty much enforces. Right now this requires considerable manual work, and »we created the tools to do that« is over-optimistic »we created the tools to enable people to build the tools to do that on their own« would be closer to the truth (and is already not a bad thing, but a lot less than what you claim).

The journald design is horrible to the point of useless

Posted Dec 2, 2011 14:18 UTC (Fri) by dlang (guest, #313) [Link] (7 responses)

When I talk about rsyslog running on Solaris, I'm not talking opensolaris. as for the other flavors of Unix, if any users of them care enough about rsyslog or syslog-ng running on that platform it will happen (either from contributed code or from paying for support). But in any case the existing syslog daemons do just fine receiving messages from those systems (and various other proprietary systems like Cisco) and then handling those logs just like all the other logs. This means that you can combine them in whatever log store you choose to use. You won't get things like trusted attributes from such systems, but that's because those systems don't offer that capability, and journald sure isn't going to solve this since it won't run on those systems (and can't make such systems provide info that isn't there in the first place)

finding which file in a series of time-rotated files a particular time is in is pretty trivial, as is finding the time in the log file once you have opened it (at least if you use a normal test editor), you can either do the binary search yourself by jumping to various line numbers, or just search for the timestamp you are looking for. You are really overstating the difficulty here. Yes this can be done wrong (rolling the logs daily, even if you have gigs of logs in a day is usually not a wise thing to do for example)

the idea that you can get the log messages when you ask for the status of a program only works in the trivial case where all the logs are written by the one pid that was started by the program you are asking about the status. If that pid started other processes that then wrote logs, systemd (or equivalent) isn't going to have a way of knowing for sure which 'service' those log messages are for.

I don't believe that the Journal is going to enforce programs writing well structured logs any more than the windows event log does (see Rainer's comment that in spite of the 'structure' being enforced by windows event log, the problem of analyzing logs on windows is the same mess that it is on *nix)

It sounds as if you have decided that anything that LP writes is the Pony that you want and any criticism of it just means the person doing the criticizing is against all progress.

The journald design is horrible to the point of useless

Posted Dec 2, 2011 17:07 UTC (Fri) by raven667 (subscriber, #5198) [Link] (1 responses)

If that pid started other processes that then wrote logs, systemd (or equivalent) isn't going to have a way of knowing for sure which 'service' those log messages are for.

I think this particular statement is probably not true. systemd uses the cgroups feature to associate all processes as part of a generic "service" or "session" which is not fooled by daemonization so it could reliably associate all logs across multiple PIDs with a particular "service".

The journald design is horrible to the point of useless

Posted Dec 2, 2011 19:05 UTC (Fri) by dlang (guest, #313) [Link]

this is a point I didn't consider, but identifying what cgroup something is in is going to be another racy lookup isn't it?

in any case, I've suggested to Rainer that he add this lookup to the trusted properties that rsyslog provides. Once that is there then it will be trivial to split the logs by service (and therefor trivial to do a tail of the most recent logs from any service)

The journald design is horrible to the point of useless

Posted Dec 2, 2011 17:16 UTC (Fri) by anselm (subscriber, #2796) [Link]

the idea that you can get the log messages when you ask for the status of a program only works in the trivial case where all the logs are written by the one pid that was started by the program you are asking about the status.

AFAIK this is wrong. Systemd puts each service into its own control group. Processes that the service generates stay in the same control group. The system status command can then consider all messages from the same control group, or the same systemd service, in order to catch everything from the service process and its children. This is actually in the proposal.

That this is a lot more difficult to do with the current infrastructure (SysV init and (r)syslogd) doesn't mean that it is, in fact, impossible in general. Systemd is not a complete exercise in futility.

It sounds as if you have decided that anything that LP writes is the Pony that you want and any criticism of it just means the person doing the criticizing is against all progress.

No. As far as I am concerned it is absolutely OK to criticise Lennart's and Kay's proposal, just as it is absolutely OK to criticise the existing syslog infrastructure and protocol. Progress results not from hanging on to the existing stuff at all costs, but from critical evaluation of both the old and any newly proposed stuff, and from putting the best ideas of both together to create something that is better than what we had before. Whether that is, in the end, a beefed-up syslogd or something entirely different is irrelevant as long as it solves the problems at hand and there is a reasonable compatibility path to the existing infrastructure. You will note that this also seems to be Rainer's attitude, although he is (for understandable reasons) somewhat biased towards the »beefed-up syslogd« end of the spectrum of possible solutions. (Which is of course OK.)

I've been using various syslogd implementations for a very long time and have usually been able to get them to do what I need, but that doesn't mean I'm blind to possible improvements that may come in from elsewhere. This IMHO is a better approach than dismissing all of a number of possible improvements outright just because one does not like the people who came up with them. No one (not even Lennart or Kay) believes that the journald proposal solves all possible problems with logging, but pretending that there are no problems with logging at all, or no serious problems, or that if there are in fact problems then the journald proposal doesn't solve them either, doesn't actually lead to progress.

The journald design is horrible to the point of useless

Posted Dec 2, 2011 17:28 UTC (Fri) by raven667 (subscriber, #5198) [Link] (3 responses)

It sounds as if you have decided that anything that LP writes is the Pony that you want and any criticism of it just means the person doing the criticizing is against all progress.

I can't speak for the OP but from from my perspective, as someone who is not personally invested in the outcome, about 80% of the comments seem to be baseless negative personal animosity against LP or an appeal to tradition against progress. Half of the other 20% are making actual technical arguments pointing out flaws in the proposal and the other half are defending LP, pointing out positives about the proposal or just advocating keeping an open mind. Those that are defending tend to be responding more the 80% than to the actual constructive criticism of the 10%

Some of the constructive criticism is very pursuasive and I'm not at all sure that the journal is the right way to go, unlike with systemd which was so obviously the right, UNIX, way to do things that I wish it was written decades ago. I can't help of think about daemontools supervise and multilog, which I used very successfully for many years, and see LP as this generation's DJB.

The journald design is horrible to the point of useless

Posted Dec 2, 2011 19:06 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

> and see LP as this generation's DJB

I think that may people would agree with you, including a LOT of the people who are being critical of LP and this proposal.

there are good reasons (and not just personal animosity or his license choice) that DJB's software did not take over the world.

LP: the new DJB?

Posted Dec 2, 2011 22:50 UTC (Fri) by anselm (subscriber, #2796) [Link] (1 responses)

there are good reasons (and not just personal animosity or his license choice) that DJB's software did not take over the world.

However, some of DJB's ideas did become popular on technical merit. Maildir comes to mind.

I don't think Lennart and Kay are doing badly in comparison at all. Systemd in particular is likely to go a lot farther than Qmail or DJBDNS ever did, simply because, whatever its detractors may say, it does have all kinds of advantages compared to SysV init (the situation is a lot more obvious than with, say, Qmail vs. other MTAs) and the two don't go for DJB-style rest-of-the-world-be-damned my-way-or-the-highway backwards incompatibility in quite the same way. After all, systemd still interfaces with SysV init scripts and traditional syslogd but in a manner that introduces interesting and useful new features.

As far as journald is concerned, we'll have to see; maybe the future will just be a closer association between systemd and rsyslogd, which wouldn't be a bad thing either.

LP: the new DJB?

Posted Dec 3, 2011 3:39 UTC (Sat) by raven667 (subscriber, #5198) [Link]

Actually when I read some of the first design docs for systems I was greatly reminded of daemontools which I have used in all my big, 24/7 systems over the last decade. I chose daemontools because of its technical merits, because of obvious deficiencies in sysv init and despite the fact the software comes from Mars with its own filesystem layout and other needless complications. We even tried using svscan for pid 1. Systemd has all the technical merits and none of the deficiencies.

I hope this journal thing leads to some improvements but I'm less certain that the journal as described is the be all and end all of logging. Let's see where it goes.