The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 21, 2014 16:28 UTC (Fri) by Wol (subscriber, #4433)
In reply to: The Grumpy Editor's guide to surviving the systemd debate by cdmiller
Parent article: The Grumpy Editor's guide to surviving the systemd debate

> At the very least the journal could use an existing vetted database back end.

As a database guy myself, I consider this about the most stupid option possible!!! By all means use a database to INDEX the log file, but the log itself should be a streaming write - any other choice is !insane!

I know there's argument about what journald does, but the reason for doing it is sound - to maximise data retention in the case of a problem, and minimise the possibility of data loss. To store the log data itself in a database is simply ASKING for trouble.

Okay, I can only speak for Pick-like databases, but every log message would need to be allocated a sequential or GUID key. On a multi-CPU system collisions are almost inevitable. I can't imagine a Pick system keeping up under the load, and given that they kick seven bells out of relational for speed, no relational engine would stand a chance.

Then there's the matter of actually writing the data. Step one is to write a transaction log entry. Step two, hash the key and append the record to the appropriate block. Step three, is the table overflowing, and if so expand it, logging that as you do so. Step four, close off the transaction log and commit.

Now how many opportunities are there in that database write for a screw-up to happen? That's even before you give the OS level and the disk subsystem the opportunity to screw things up for you! Even worse, step three presents a marvellous opportunity for trashing old log messages as the database block gets split into two. I don't know how other DB engines handle it, but they're not going to be any better at it!

At the end of the day, the ONLY way that makes any sense is to stream the messages straight to disk. You're still relying on the OS and disk not to screw you up, but there's no way round that.

Cheers,
Wol

to post comments

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 25, 2014 22:24 UTC (Tue) by nix (subscriber, #2304) [Link] (4 responses)

Then there's the matter of actually writing the data. Step one is to write a transaction log entry. Step two, hash the key and append the record to the appropriate block. Step three, is the table overflowing, and if so expand it, logging that as you do so. Step four, close off the transaction log and commit.
Now how many opportunities are there in that database write for a screw-up to happen? That's even before you give the OS level and the disk subsystem the opportunity to screw things up for you! Even worse, step three presents a marvellous opportunity for trashing old log messages as the database block gets split into two. I don't know how other DB engines handle it, but they're not going to be any better at it!

Oh yes they are. Non-corruption under conditions of power loss and -ENOSPC is routinely tested in any half-competent database system (hell, even sqlite. Perhaps especially sqlite, their test harness is awesome).

Unfortunately these things slow everything down, quite a lot, and in the end you end up writing everything at least twice (once to a log of some kind, once to the DB) and taking noticeably more space to do it. I concur, writing syslogs to a database if you don't have a reason to do it (e.g. streaming structured logs there for later querying) is nuts.

But, then, existing projects like syslog-ng have been able to do this for many years. So this too isn't a justification for journald.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 25, 2014 22:37 UTC (Tue) by dlang (guest, #313) [Link]

> Unfortunately these things slow everything down, quite a lot, and in the end you end up writing everything at least twice (once to a log of some kind, once to the DB) and taking noticeably more space to do it. I concur, writing syslogs to a database if you don't have a reason to do it (e.g. streaming structured logs there for later querying) is nuts.

> But, then, existing projects like syslog-ng have been able to do this for many years. So this too isn't a justification for journald.

Rsyslog will let you insert a bunch of messages at once (batch mode), which does wonders for the performance of inserting logs into databases.

But it's generally not the right thing to do because of the performance overhead (both inserting and retrieving)

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 20:36 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

> Oh yes they are. Non-corruption under conditions of power loss and -ENOSPC is routinely tested in any half-competent database system (hell, even sqlite. Perhaps especially sqlite, their test harness is awesome).

You're confusing DATA CORRUPTION with DATA LOSS. Yep, a database promises you won't get data CORRUPTION. In order to provide that guarantee, it also pretty much guarantees you WILL get data LOSS.

A database guarantees that if you have problems with the write, the database is ROLLED BACK to a consistent state. Which is totally at odds with the requirements of a logging system. If your computer hits a crash-state, a database guarantees that all your "in flight" log entries (ie the ones you are most interested in) will be THROWN AWAY.

A simple streaming write is the best way to ensure the information you actually want makes its way to disk before everything goes pear-shaped. The reason I explained how Pick would save the record is to stress the point that ANY problem, at ANY stage of the process, that causes a write failure, will cause the corresponding log entry to be thrown away!!! NOT what is wanted!!!

The only thing that's going to screw up a streaming write is corruption in the kernel or a screwed-up disk. Those will also affect a database, but there's a far bigger window for those problems (and other problems too) to screw up a database. And I repeat - the time at which you most need logging to successfully save log entries, is exactly the time at which a database is MOST likely to GET screwed up, and throw "writes in flight" away in the name of data CONSISTENCY.

Use a database for INDEXING the logs, not for storing them. Oh - and as for sqlite's vaunted robustness, I believe Firefox uses it. Why oh why then does my wife's Firefox give me so much grief with data issues? (She has mental health issues, and her login session gets a lot of abuse ...) My Firefox, on the same system, doesn't give me grief at all.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 20:41 UTC (Wed) by Wol (subscriber, #4433) [Link] (1 responses)

Oops - I think I've missed a trick - if the initial log entry gets saved successfully, then the transaction can be rolled forward, but you've still got an unnecessary window for data loss.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 23:36 UTC (Wed) by dlang (guest, #313) [Link]

the key is that the database doesn't consider the transaction complete until it gets confirmation that the log entry has been written successfully.

so there is no data loss once the database acks the transaction.