The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 5:14 UTC (Fri) by cdmiller (guest, #2813)
In reply to: The Grumpy Editor's guide to surviving the systemd debate by notninjaz
Parent article: The Grumpy Editor's guide to surviving the systemd debate

As far as SLES goes we are locked to RHEL on some systems due to vendor support, though it sounds well worth a look at for our Oracle servers.

For your misfeature request I'll focus on a short list for the journal. These are already well known and discussed. The themes however are telling and appear to exist in other portions of the systemd space.

Binary log format.
Custom binary log format.
Lack of log shipping.
Log corruption.
The requirement to run multiple log daemons.
Web access to log files.

The binary format of the journal is an unnecessary non feature. When we want binary format we already ship to a central server and a proven database of some sort, with a web accessible interface. At the very least the journal could use an existing vetted database back end. At the point of origin venerable grep friendly text files are usually enough. A structured format is great but in the end it's application dependent, binary or not. Having to run two logging daemons to ship logs for anything started under systemd is a poor design. Adding non features like the web accessibility while treating the rest as non issues due to band aids existing is unfortunate. These items truly need to be fixed, time will tell.

to post comments

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 11:24 UTC (Fri) by pbonzini (subscriber, #60935) [Link]

> The binary format of the journal is an unnecessary non feature.

Are you aware that on RHEL7 the binary format is just in-memory? The on disk persistent log uses syslog.

> Having to run two logging daemons to ship logs for anything started under systemd is a poor design

journald aggregates log from the services's stdout/stderr, and keeps it in memory for fast access in "systemctl status" (one of my favorite systemd features). syslogs persists it. "Do one thing, do it well".

> Web access to log files.

Not mandatory, and a separate component from journald. Again, "do one thing, do it well".

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 14:15 UTC (Fri) by notninjaz (guest, #99725) [Link]

Thanks for the reply!

I had the same feeling about the approach to log reporting - we solve it in a different place, too, using kibana/td-agent/fluentd from a central location with the logs also kept as text. I remember situations, though, in shops without that kind of setup wanting to look at e.g., 15 minutes around a SAN or network event on a number of servers with the benefit of a pipe to egrep to get rid of known background chatter. So, keeping an indexed binary copy of the log isn't a dealbreaker for me as long as syslog is directly written to as well. I see it as similar to having find and locatedb - I wouldn't want to have to depend on locatedb not being corrupt, but locate can be handy to find a file more quickly than find or ls alone.

The main thing I am concerned about with a dependency-based init is having a situation where a service that isn't critical for the server's purpose ends up blocking a business-critical application from being started due to failed dependencies. I suppose that could be worked around by marking the dependency as non-critical as we do with clusters, but vendors tend to have a way of declaring "UNSUPPORTED!" the moment any distribution-maintained stuff is changed.

On the other hand, one might instead simply go through service-by-service and make a service depend on only the 20 out of 140 services it actually needs (as someone else replied on a thread here about NetworkManger: rather than using simple text files / udev rules to start the network, one might simply make NetworkManager depend on the em1 interface using the dependency-based init system... yikes!), but that starts making what was an easy thing require quite a bit more work.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 21, 2014 16:28 UTC (Fri) by Wol (subscriber, #4433) [Link] (5 responses)

> At the very least the journal could use an existing vetted database back end.

As a database guy myself, I consider this about the most stupid option possible!!! By all means use a database to INDEX the log file, but the log itself should be a streaming write - any other choice is !insane!

I know there's argument about what journald does, but the reason for doing it is sound - to maximise data retention in the case of a problem, and minimise the possibility of data loss. To store the log data itself in a database is simply ASKING for trouble.

Okay, I can only speak for Pick-like databases, but every log message would need to be allocated a sequential or GUID key. On a multi-CPU system collisions are almost inevitable. I can't imagine a Pick system keeping up under the load, and given that they kick seven bells out of relational for speed, no relational engine would stand a chance.

Then there's the matter of actually writing the data. Step one is to write a transaction log entry. Step two, hash the key and append the record to the appropriate block. Step three, is the table overflowing, and if so expand it, logging that as you do so. Step four, close off the transaction log and commit.

Now how many opportunities are there in that database write for a screw-up to happen? That's even before you give the OS level and the disk subsystem the opportunity to screw things up for you! Even worse, step three presents a marvellous opportunity for trashing old log messages as the database block gets split into two. I don't know how other DB engines handle it, but they're not going to be any better at it!

At the end of the day, the ONLY way that makes any sense is to stream the messages straight to disk. You're still relying on the OS and disk not to screw you up, but there's no way round that.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 25, 2014 22:24 UTC (Tue) by nix (subscriber, #2304) [Link] (4 responses)

Then there's the matter of actually writing the data. Step one is to write a transaction log entry. Step two, hash the key and append the record to the appropriate block. Step three, is the table overflowing, and if so expand it, logging that as you do so. Step four, close off the transaction log and commit.
Now how many opportunities are there in that database write for a screw-up to happen? That's even before you give the OS level and the disk subsystem the opportunity to screw things up for you! Even worse, step three presents a marvellous opportunity for trashing old log messages as the database block gets split into two. I don't know how other DB engines handle it, but they're not going to be any better at it!

Oh yes they are. Non-corruption under conditions of power loss and -ENOSPC is routinely tested in any half-competent database system (hell, even sqlite. Perhaps especially sqlite, their test harness is awesome).

Unfortunately these things slow everything down, quite a lot, and in the end you end up writing everything at least twice (once to a log of some kind, once to the DB) and taking noticeably more space to do it. I concur, writing syslogs to a database if you don't have a reason to do it (e.g. streaming structured logs there for later querying) is nuts.

But, then, existing projects like syslog-ng have been able to do this for many years. So this too isn't a justification for journald.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 25, 2014 22:37 UTC (Tue) by dlang (guest, #313) [Link]

> Unfortunately these things slow everything down, quite a lot, and in the end you end up writing everything at least twice (once to a log of some kind, once to the DB) and taking noticeably more space to do it. I concur, writing syslogs to a database if you don't have a reason to do it (e.g. streaming structured logs there for later querying) is nuts.

> But, then, existing projects like syslog-ng have been able to do this for many years. So this too isn't a justification for journald.

Rsyslog will let you insert a bunch of messages at once (batch mode), which does wonders for the performance of inserting logs into databases.

But it's generally not the right thing to do because of the performance overhead (both inserting and retrieving)

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 20:36 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

> Oh yes they are. Non-corruption under conditions of power loss and -ENOSPC is routinely tested in any half-competent database system (hell, even sqlite. Perhaps especially sqlite, their test harness is awesome).

You're confusing DATA CORRUPTION with DATA LOSS. Yep, a database promises you won't get data CORRUPTION. In order to provide that guarantee, it also pretty much guarantees you WILL get data LOSS.

A database guarantees that if you have problems with the write, the database is ROLLED BACK to a consistent state. Which is totally at odds with the requirements of a logging system. If your computer hits a crash-state, a database guarantees that all your "in flight" log entries (ie the ones you are most interested in) will be THROWN AWAY.

A simple streaming write is the best way to ensure the information you actually want makes its way to disk before everything goes pear-shaped. The reason I explained how Pick would save the record is to stress the point that ANY problem, at ANY stage of the process, that causes a write failure, will cause the corresponding log entry to be thrown away!!! NOT what is wanted!!!

The only thing that's going to screw up a streaming write is corruption in the kernel or a screwed-up disk. Those will also affect a database, but there's a far bigger window for those problems (and other problems too) to screw up a database. And I repeat - the time at which you most need logging to successfully save log entries, is exactly the time at which a database is MOST likely to GET screwed up, and throw "writes in flight" away in the name of data CONSISTENCY.

Use a database for INDEXING the logs, not for storing them. Oh - and as for sqlite's vaunted robustness, I believe Firefox uses it. Why oh why then does my wife's Firefox give me so much grief with data issues? (She has mental health issues, and her login session gets a lot of abuse ...) My Firefox, on the same system, doesn't give me grief at all.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 20:41 UTC (Wed) by Wol (subscriber, #4433) [Link] (1 responses)

Oops - I think I've missed a trick - if the initial log entry gets saved successfully, then the transaction can be rolled forward, but you've still got an unnecessary window for data loss.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 23:36 UTC (Wed) by dlang (guest, #313) [Link]

the key is that the database doesn't consider the transaction complete until it gets confirmation that the log entry has been written successfully.

so there is no data loss once the database acks the transaction.