The Grumpy Editor's guide to surviving the systemd debate

Are you aware that on RHEL7 the binary format is just in-memory? The on disk persistent log uses syslog.

> Having to run two logging daemons to ship logs for anything started under systemd is a poor design

journald aggregates log from the services's stdout/stderr, and keeps it in memory for fast access in "systemctl status" (one of my favorite systemd features). syslogs persists it. "Do one thing, do it well".

> Web access to log files.

Not mandatory, and a separate component from journald. Again, "do one thing, do it well".

I had the same feeling about the approach to log reporting - we solve it in a different place, too, using kibana/td-agent/fluentd from a central location with the logs also kept as text. I remember situations, though, in shops without that kind of setup wanting to look at e.g., 15 minutes around a SAN or network event on a number of servers with the benefit of a pipe to egrep to get rid of known background chatter. So, keeping an indexed binary copy of the log isn't a dealbreaker for me as long as syslog is directly written to as well. I see it as similar to having find and locatedb - I wouldn't want to have to depend on locatedb not being corrupt, but locate can be handy to find a file more quickly than find or ls alone.

The main thing I am concerned about with a dependency-based init is having a situation where a service that isn't critical for the server's purpose ends up blocking a business-critical application from being started due to failed dependencies. I suppose that could be worked around by marking the dependency as non-critical as we do with clusters, but vendors tend to have a way of declaring "UNSUPPORTED!" the moment any distribution-maintained stuff is changed.

On the other hand, one might instead simply go through service-by-service and make a service depend on only the 20 out of 140 services it actually needs (as someone else replied on a thread here about NetworkManger: rather than using simple text files / udev rules to start the network, one might simply make NetworkManager depend on the em1 interface using the dependency-based init system... yikes!), but that starts making what was an easy thing require quite a bit more work.

As a database guy myself, I consider this about the most stupid option possible!!! By all means use a database to INDEX the log file, but the log itself should be a streaming write - any other choice is !insane!

I know there's argument about what journald does, but the reason for doing it is sound - to maximise data retention in the case of a problem, and minimise the possibility of data loss. To store the log data itself in a database is simply ASKING for trouble.

Okay, I can only speak for Pick-like databases, but every log message would need to be allocated a sequential or GUID key. On a multi-CPU system collisions are almost inevitable. I can't imagine a Pick system keeping up under the load, and given that they kick seven bells out of relational for speed, no relational engine would stand a chance.

Then there's the matter of actually writing the data. Step one is to write a transaction log entry. Step two, hash the key and append the record to the appropriate block. Step three, is the table overflowing, and if so expand it, logging that as you do so. Step four, close off the transaction log and commit.

Now how many opportunities are there in that database write for a screw-up to happen? That's even before you give the OS level and the disk subsystem the opportunity to screw things up for you! Even worse, step three presents a marvellous opportunity for trashing old log messages as the database block gets split into two. I don't know how other DB engines handle it, but they're not going to be any better at it!

At the end of the day, the ONLY way that makes any sense is to stream the messages straight to disk. You're still relying on the OS and disk not to screw you up, but there's no way round that.

Cheers,
Wol