The Grumpy Editor's guide to surviving the systemd debate [LWN.net]

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 17:12 UTC (Thu) by cdmiller (guest, #2813)
In reply to: The Grumpy Editor's guide to surviving the systemd debate by bferrell
Parent article: The Grumpy Editor's guide to surviving the systemd debate

"It is just a system initialization utility". Really? A simple system initialization utility in one large software suite that includes 69 individual binaries. From all indications that number is set to continue growing. There are some indications that tight coupling of those binaries and forced exclusion of their alternatives is increasingly preferred.

IMHO, the real systemd debate will be an ongoing conversation between developers and customers, and it's going to last a while.

Someone in these debates made a statement to the effect, if systemd is really too problematic systems administrators would appear with torches and pitchforks (kind of amusing as it's usually admins facing the pitchforks of their users). Well guess what? Seasoned systems admins are voicing legitimate technical complaints. There are some bug or anti-feature reports being ignored with snide attitudes. Recently there are a few grousing sessions on LOPSA IRC. Within my own team we now have occasional discussions about what to do systemd wise (journal appears to be a current focus, along with stunned disbelief and sarcasm about some anti-features). At the extreme do we stick with RHEL 6.5 and it's extended support cycle for some of our systems? We have 4 years left on Ubuntu Trusty LTS server edition. Almost sounds like waiting for the next Windows Desktop version and that's pretty sad.

In my view experienced sys admins are generally slow and cautious in response to major system changes rather than an unruly pitchfork wielding mob. We are primarily users, with occasional patches or bug reports and internal glue code as our contributions to free and open source. If legitimate complaints from the most prolific customers of linux are brought up and ignored over a sufficient period of time, some fracturing of the community will eventually occur. Linux is dominant in the server and cloud space and doing well in embedded. Vendors and projects who ignore or attempt to quash the requirements of those environments will eventually suffer the consequences.

Philosophically, about freedom, bferrell is correct. One reason we choose free and open source in my shop is freedom of choice. Being able to choose from a plethora of tools to find the best combination for any given job has been a boon for free software acceptance. If that changes to an Apple like vendor lock-in attitude other solutions will be sought or created. We will eventually route around systemd *if* it appears too unusable for our needs. The next year or two will tell how well it will eventually work out in our purposely heterogeneous shop.

The real systemd debate is going to last a while. In fact, in the systems admin space, it's barely getting started.

to post comments

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 17:36 UTC (Thu) by jspaleta (subscriber, #50639) [Link] (21 responses)

There are bug reports? Can you point me to your favorite ones?

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 4:20 UTC (Fri) by cdmiller (guest, #2813) [Link] (20 responses)

Since I mentioned the journal I'll take your bait and point you at one. Probably the most rehashed, bug 64116.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 9:10 UTC (Fri) by niner (guest, #26151) [Link] (19 responses)

Can you please explain to me why the following comment from the bug report you mentioned, does not explain the rationale behind this design decision sufficiently?

Lennart Poettering 2014-10-08 20:27:49 UTC
Since this bugyilla report is apparently sometimes linked these days as an example how we wouldn't fix a major bug in systemd:

Journal files are mostly append-only files. We keep adding to the end as we go, only updating minimal indexes and bookkeeping in the front earlier parts of the files. These files are rotated (rotation = renamed and replaced by a new one) from time to time, based on certain conditions, such as time, file size, and also when we find the files to be corrupted. As soon as they rotate they are entirely read-only, never modified again. When you use a tool like "journalctl" to read the journal files both the active and the rotated files are implicitly merged, so that they appear as a single stream again.

Now, our strategy to rotate-on-corruption is the safest thing we can do, as we make sure that the internal corruption is frozen in time, and not attempted to be "fixed" by a tool, that might end up making things worse. After all, in the case the often-run writing code really fucks something up, then it is not necessarily a good idea to try to make it better by running a tool on it that tries to fix it up again, a tool that is necessarily a lot more complex, and also less tested.

Now, of course, having corrupted files isn't great, and we should make sure the files even when corrupted stay as accessible as possible. Hence: the code that reads the journal files is actually written in a way that tries to make the best of corrupted files, and tries to read of them as much as possible, with the the subset of the file that is still valid. We do this implicitly on every access.

Hence: journalctl implicitly does on read what a theoretical journal file fsck tool would do, but without actually making this persistent. This logic also has a major benefit: as our reader gets better and learns to deal with more types of corruptions you immediately benefit of it, even for old files!

File systems such as ext4 have an fsck tool since they don't have the luxury to just rotate the fs away and fix the structure on read: they have to use the same file system for all future writes, and they thus need to try hard to make the existing data workable again.

I hope this explains the rationale here a bit more.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 23:45 UTC (Mon) by nix (subscriber, #2304) [Link] (5 responses)

Now, our strategy to rotate-on-corruption is the safest thing we can do, as we make sure that the internal corruption is frozen in time, and not attempted to be "fixed" by a tool, that might end up making things worse. After all, in the case the often-run writing code really fucks something up, then it is not necessarily a good idea to try to make it better by running a tool on it that tries to fix it up again, a tool that is necessarily a lot more complex, and also less tested.

I can't believe anyone thinks this argument has any merit at all. You could use the same words to argue that filesystems (a complex mass of binary data frequently written to which can suffer corruption) should never be repaired, because you can never get a fsck that always repairs all problems and never makes anything worse. Instead, we consider a filesystem sans fsck to be substandard.

Meanwhile, the format it's replacing, if corrupted, well, let's see -- you lose a line of syslog. Maybe you lose a bunch of other lines. But recovery from that is automatic; you only lose a line of syslog data around that touched by the corruption. You certainly don't lose a whole file. You *certainly* don't just rotate it away and, oh, I guess we should hope it wasn't an important log then.

This is not a sensible attitude to log data unless you consider log data fundamentally unimportant -- and in that case, why on earth are you recording it at all?

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 23:59 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link]

As far as I remember, X.org's repositories were corrupted beyond recovery after power failed multiple times during fsck.

I've also had a similar experiences (albeit with NTFS on Windows).

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 2:58 UTC (Thu) by raven667 (subscriber, #5198) [Link] (3 responses)

> Maybe you lose a bunch of other lines. But recovery from that is automatic; you only lose a line of syslog data around that touched by the corruption.

Maybe I misread but isn't that what journalctl does, skip invalid records and start back up at the next valid record, without touching the original file so you always have a pristine copy with the corruption intact? Logs are different than filesystems in that they are write once, read many, you never want to rewrite history, even if that history is corrupt due to a bug.

In fact if it were me, if you detect corruption on reading a journal log I would want to immediately kill journald and let systemd restart it, on startup the new copy can start a new log file Treat it like a microkernel server, die and reload quickly on error.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 3:47 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

with a binary file format, finding the start of the next message is not that easy.

As for killing journald when you _read_ a bad message, that sure isn't the right thing to do because you can read the same message many times, and do it years after it was written. killing journald when you read a message that's written a significant time in the past is the WRONG thing to do.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 20, 2014 4:28 UTC (Thu) by raven667 (subscriber, #5198) [Link]

sorry, I maybe should have been more clear, journalctl reads both the currently writing log and all previous ones that have been rotated and are still around, it should only restart journald if the corruption is in the log that journald is currently writing. The reason to implement this in the reader is that the I don't expect the log writer be reading often enough to notice if something has gone wrong.

I don't know if that's how it actually works, that's just my first idea of how I think it should work.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 21, 2014 9:15 UTC (Fri) by zlynx (guest, #2285) [Link]

Finding stuff in "binary" files is no more difficult than text files. It depends on the particular format.

People tend to forget that text files ARE A BINARY FORMAT. You have your 7-bit ASCII message terminated by a new-line character.

You could also have an array of 4 byte little-endian unsigned integers, terminated by 4 bytes of zero. It would be just as easy to find the start of the next message as it is in text.

We're pretty spoiled these days and don't have to worry about our messages getting off by a bit or two. I recall modems where if you weren't using parity (or was it stop bits) you might lose sync and anything could happen.

Anyway, there might be valid complaints about the particular design of journald's binary format. But just because it is binary doesn't necessarily make it unusually hard to recover or read.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 17, 2014 23:47 UTC (Mon) by nix (subscriber, #2304) [Link] (9 responses)

File systems such as ext4 have an fsck tool since they don't have the luxury to just rotate the fs away and fix the structure on read: they have to use the same file system for all future writes, and they thus need to try hard to make the existing data workable again.

This is, of course, a fallacious argument too. File systems such as ext4 have a fsck tool because it would be an appalling implementation which said, oh, the FS is corrupted, we'll hit you with an empty one and let you get at the old one with some special commands! Which is more-or-less what systemd's journal does.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:07 UTC (Tue) by rahulsundaram (subscriber, #21946) [Link]

That is not what the bug report claims:

"of course, having corrupted files isn't great, and we should make sure the files even when corrupted stay as accessible as possible. Hence: the code that reads the journal files is actually written in a way that tries to make the best of corrupted files, and tries to read of them as much as possible, with the the subset of the file that is still valid. We do this implicitly on every access."

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:15 UTC (Tue) by mchapman (subscriber, #66589) [Link] (7 responses)

> oh, the FS is corrupted, we'll hit you with an empty one and let you get at the old one with some special commands! Which is more-or-less what systemd's journal does.

No, it doesn't end up empty.

In my experience, a corrupted journal file is still completely readable and searchable, except for its final entry. Since the tools don't expose how individual journal files are being accessed, this corruption simply appears as if you'd lost a single log message.

Yes, it's possible some header corruption could mean an entire file is unreadable. I haven't personally seen that, but I can imagine that it's a concern for some people. You can limit the damage by ensuring the files get rotated frequently. There's {System,RunTime}MaxFileSize config options available; perhaps corresponding {System,RunTime}MaxRotateInterval might be useful. Worst comes to worst, you could just SIGUSR2 the journald process regularly from Cron or a timer unit.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 0:59 UTC (Tue) by dlang (guest, #313) [Link] (6 responses)

the instance of journald corruption that I saw ended up with the binary pointers going backwards and so walking the journal ended up with an endless loop

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 2:34 UTC (Tue) by mchapman (subscriber, #66589) [Link] (5 responses)

Right. I haven't seen that myself, but I could imagine it could happen.

But it sounds like something easily detectable by the libraries that read the journal files (so other apps, not just journalctl, would get the fix), and it still wouldn't result in an "empty" journal.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 4:03 UTC (Tue) by dlang (guest, #313) [Link] (4 responses)

It does make it so that all log entries after the corruption are lost though.

This is one of the problems with binary logs

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 5:04 UTC (Tue) by mchapman (subscriber, #66589) [Link] (3 responses)

Only up until the next rotation... which would presumably happen when you update systemd to fix the bug, if not earlier.

Look, I'm not denying that there are bugs that can corrupt the logs or lose large numbers of log messages. But the original claim -- that corruption yields an "empty" journal that can only be made non-empty through the use of "special commands" -- is patently false.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 11:59 UTC (Tue) by notninjaz (guest, #99725) [Link] (2 responses)

As I read it, the original "empty" statement was referring to the new journal which would be empty as in a clean slate, comparing it to offering the user a new clean slate filesystem rather than trying to repair a corrupted one.

This thread has brought out some of the tradeoffs, though, such as a recommendation to rotate the log frequently to minimize the impact of possible corruption. For the servers I administer, I would prefer logging to a text file and risking loss of a fraction of a second of logs if an entry is corrupt instead of rotating a journal hourly and potentially losing 59 minutes should there be corruption.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 13:14 UTC (Tue) by mchapman (subscriber, #66589) [Link] (1 responses)

> As I read it, the original "empty" statement was referring to the new journal which would be empty as in a clean slate, comparing it to offering the user a new clean slate filesystem rather than trying to repair a corrupted one.

Maybe that was what it was referring to. At any rate, it's just not true. When a journal file is rotated (whether that be because journald has detected corruption in it, or because it's reached its maximum size, or because the admin has simply asked for it by sending SIGUSR2 to the journald process), its contents are still read automatically by journalctl when searching and iterating through the logs.

The journal is not a single file; it is the sum total of *all* journal files. As far as I know, the only way to create an empty journal is to literally remove all of these journal files.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 14:51 UTC (Tue) by notninjaz (guest, #99725) [Link]

Right, I was referencing the statement: "Yes, it's possible some header corruption could mean an entire file is unreadable. I haven't personally seen that, but I can imagine that it's a concern for some people. You can limit the damage by ensuring the files get rotated frequently. There's {System,RunTime}MaxFileSize config options available; perhaps corresponding {System,RunTime}MaxRotateInterval might be useful. Worst comes to worst, you could just SIGUSR2 the journald process regularly from Cron or a timer unit."

I meant to express a design preference rather to critique any particular software with regard to text vs. binary logs.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 18, 2014 22:52 UTC (Tue) by cdmiller (guest, #2813) [Link] (2 responses)

The technical side is still debatable as evidenced below. There is apparent confusion over common failure modes of the binary log and what log data is available or potentially lost after a failure. Folks appear to not fully understand or agree with the explanation. The timing of the more detailed response is troubling. What you posted above appeared over a year after the initial bug report. Suffice it to say I'm not convinced systemd-journald log corruption handling is sufficient for servers, or that anything more robust is being pursued.

According to one post below, interesting defaults are being set for logging configuration in RHEL7. The configuration described looks like a band aid to side step this and possibly other issues.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 19, 2014 0:35 UTC (Wed) by rodgerd (guest, #58896) [Link] (1 responses)

TO be honest, journald *features* are nice, but the implementation detail still seems like the main weakness in systemd-the-suite-of-programs. I don't quite understand why the binary blob DB isn't based on an already extant DB, for example (Gnu DB, Berkely, whatever), and the corruption and lack of a remote logging mechanism undermine the work Lennart did to try and make it more useful as an audit tool.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 19, 2014 11:07 UTC (Wed) by mchapman (subscriber, #66589) [Link]

> I don't quite understand why the binary blob DB isn't based on an already extant DB

The official answer to that can be found at http://thread.gmane.org/gmane.comp.sysutils.systemd.devel....

I have read through the journal file format specification [1] and the code that implements it [2], and from what I can tell it looks reasonably straight-forward. I have no doubt bugs will be discovered. That is inevitable with any non-trivial code. But there's really no reason to believe that these bugs will go unfixed. It's not as if it's "new" code, anyway -- people have been using it now for years.

[1] http://www.freedesktop.org/wiki/Software/systemd/journal-...
[2] http://cgit.freedesktop.org/systemd/systemd/plain/src/jou...

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 13, 2014 20:01 UTC (Thu) by notninjaz (guest, #99725) [Link] (9 responses)

I would be interested to read about the misfeatures your team has identified. It was welcome news here that log lines are being shown complete instead of abbreviated with ..., which leads me to believe that the developers are addressing this kind of thing based on sysadmin feedback.

For a medium-term strategy, I have been looking pretty hard at SLES 11 SP3. It addresses some of the major shortcomings in RHEL 6 - it ships with pacemaker, has KVM image locking via virtlockd, includes XFS at no additional cost, and is certified for Oracle 10/11/12

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 5:14 UTC (Fri) by cdmiller (guest, #2813) [Link] (8 responses)

As far as SLES goes we are locked to RHEL on some systems due to vendor support, though it sounds well worth a look at for our Oracle servers.

For your misfeature request I'll focus on a short list for the journal. These are already well known and discussed. The themes however are telling and appear to exist in other portions of the systemd space.

Binary log format.
Custom binary log format.
Lack of log shipping.
Log corruption.
The requirement to run multiple log daemons.
Web access to log files.

The binary format of the journal is an unnecessary non feature. When we want binary format we already ship to a central server and a proven database of some sort, with a web accessible interface. At the very least the journal could use an existing vetted database back end. At the point of origin venerable grep friendly text files are usually enough. A structured format is great but in the end it's application dependent, binary or not. Having to run two logging daemons to ship logs for anything started under systemd is a poor design. Adding non features like the web accessibility while treating the rest as non issues due to band aids existing is unfortunate. These items truly need to be fixed, time will tell.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 11:24 UTC (Fri) by pbonzini (subscriber, #60935) [Link]

> The binary format of the journal is an unnecessary non feature.

Are you aware that on RHEL7 the binary format is just in-memory? The on disk persistent log uses syslog.

> Having to run two logging daemons to ship logs for anything started under systemd is a poor design

journald aggregates log from the services's stdout/stderr, and keeps it in memory for fast access in "systemctl status" (one of my favorite systemd features). syslogs persists it. "Do one thing, do it well".

> Web access to log files.

Not mandatory, and a separate component from journald. Again, "do one thing, do it well".

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 14, 2014 14:15 UTC (Fri) by notninjaz (guest, #99725) [Link]

Thanks for the reply!

I had the same feeling about the approach to log reporting - we solve it in a different place, too, using kibana/td-agent/fluentd from a central location with the logs also kept as text. I remember situations, though, in shops without that kind of setup wanting to look at e.g., 15 minutes around a SAN or network event on a number of servers with the benefit of a pipe to egrep to get rid of known background chatter. So, keeping an indexed binary copy of the log isn't a dealbreaker for me as long as syslog is directly written to as well. I see it as similar to having find and locatedb - I wouldn't want to have to depend on locatedb not being corrupt, but locate can be handy to find a file more quickly than find or ls alone.

The main thing I am concerned about with a dependency-based init is having a situation where a service that isn't critical for the server's purpose ends up blocking a business-critical application from being started due to failed dependencies. I suppose that could be worked around by marking the dependency as non-critical as we do with clusters, but vendors tend to have a way of declaring "UNSUPPORTED!" the moment any distribution-maintained stuff is changed.

On the other hand, one might instead simply go through service-by-service and make a service depend on only the 20 out of 140 services it actually needs (as someone else replied on a thread here about NetworkManger: rather than using simple text files / udev rules to start the network, one might simply make NetworkManager depend on the em1 interface using the dependency-based init system... yikes!), but that starts making what was an easy thing require quite a bit more work.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 21, 2014 16:28 UTC (Fri) by Wol (subscriber, #4433) [Link] (5 responses)

> At the very least the journal could use an existing vetted database back end.

As a database guy myself, I consider this about the most stupid option possible!!! By all means use a database to INDEX the log file, but the log itself should be a streaming write - any other choice is !insane!

I know there's argument about what journald does, but the reason for doing it is sound - to maximise data retention in the case of a problem, and minimise the possibility of data loss. To store the log data itself in a database is simply ASKING for trouble.

Okay, I can only speak for Pick-like databases, but every log message would need to be allocated a sequential or GUID key. On a multi-CPU system collisions are almost inevitable. I can't imagine a Pick system keeping up under the load, and given that they kick seven bells out of relational for speed, no relational engine would stand a chance.

Then there's the matter of actually writing the data. Step one is to write a transaction log entry. Step two, hash the key and append the record to the appropriate block. Step three, is the table overflowing, and if so expand it, logging that as you do so. Step four, close off the transaction log and commit.

Now how many opportunities are there in that database write for a screw-up to happen? That's even before you give the OS level and the disk subsystem the opportunity to screw things up for you! Even worse, step three presents a marvellous opportunity for trashing old log messages as the database block gets split into two. I don't know how other DB engines handle it, but they're not going to be any better at it!

At the end of the day, the ONLY way that makes any sense is to stream the messages straight to disk. You're still relying on the OS and disk not to screw you up, but there's no way round that.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 25, 2014 22:24 UTC (Tue) by nix (subscriber, #2304) [Link] (4 responses)

Then there's the matter of actually writing the data. Step one is to write a transaction log entry. Step two, hash the key and append the record to the appropriate block. Step three, is the table overflowing, and if so expand it, logging that as you do so. Step four, close off the transaction log and commit.
Now how many opportunities are there in that database write for a screw-up to happen? That's even before you give the OS level and the disk subsystem the opportunity to screw things up for you! Even worse, step three presents a marvellous opportunity for trashing old log messages as the database block gets split into two. I don't know how other DB engines handle it, but they're not going to be any better at it!

Oh yes they are. Non-corruption under conditions of power loss and -ENOSPC is routinely tested in any half-competent database system (hell, even sqlite. Perhaps especially sqlite, their test harness is awesome).

Unfortunately these things slow everything down, quite a lot, and in the end you end up writing everything at least twice (once to a log of some kind, once to the DB) and taking noticeably more space to do it. I concur, writing syslogs to a database if you don't have a reason to do it (e.g. streaming structured logs there for later querying) is nuts.

But, then, existing projects like syslog-ng have been able to do this for many years. So this too isn't a justification for journald.

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 25, 2014 22:37 UTC (Tue) by dlang (guest, #313) [Link]

> Unfortunately these things slow everything down, quite a lot, and in the end you end up writing everything at least twice (once to a log of some kind, once to the DB) and taking noticeably more space to do it. I concur, writing syslogs to a database if you don't have a reason to do it (e.g. streaming structured logs there for later querying) is nuts.

> But, then, existing projects like syslog-ng have been able to do this for many years. So this too isn't a justification for journald.

Rsyslog will let you insert a bunch of messages at once (batch mode), which does wonders for the performance of inserting logs into databases.

But it's generally not the right thing to do because of the performance overhead (both inserting and retrieving)

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 20:36 UTC (Wed) by Wol (subscriber, #4433) [Link] (2 responses)

> Oh yes they are. Non-corruption under conditions of power loss and -ENOSPC is routinely tested in any half-competent database system (hell, even sqlite. Perhaps especially sqlite, their test harness is awesome).

You're confusing DATA CORRUPTION with DATA LOSS. Yep, a database promises you won't get data CORRUPTION. In order to provide that guarantee, it also pretty much guarantees you WILL get data LOSS.

A database guarantees that if you have problems with the write, the database is ROLLED BACK to a consistent state. Which is totally at odds with the requirements of a logging system. If your computer hits a crash-state, a database guarantees that all your "in flight" log entries (ie the ones you are most interested in) will be THROWN AWAY.

A simple streaming write is the best way to ensure the information you actually want makes its way to disk before everything goes pear-shaped. The reason I explained how Pick would save the record is to stress the point that ANY problem, at ANY stage of the process, that causes a write failure, will cause the corresponding log entry to be thrown away!!! NOT what is wanted!!!

The only thing that's going to screw up a streaming write is corruption in the kernel or a screwed-up disk. Those will also affect a database, but there's a far bigger window for those problems (and other problems too) to screw up a database. And I repeat - the time at which you most need logging to successfully save log entries, is exactly the time at which a database is MOST likely to GET screwed up, and throw "writes in flight" away in the name of data CONSISTENCY.

Use a database for INDEXING the logs, not for storing them. Oh - and as for sqlite's vaunted robustness, I believe Firefox uses it. Why oh why then does my wife's Firefox give me so much grief with data issues? (She has mental health issues, and her login session gets a lot of abuse ...) My Firefox, on the same system, doesn't give me grief at all.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 20:41 UTC (Wed) by Wol (subscriber, #4433) [Link] (1 responses)

Oops - I think I've missed a trick - if the initial log entry gets saved successfully, then the transaction can be rolled forward, but you've still got an unnecessary window for data loss.

Cheers,
Wol

The Grumpy Editor's guide to surviving the systemd debate

Posted Nov 26, 2014 23:36 UTC (Wed) by dlang (guest, #313) [Link]

the key is that the database doesn't consider the transaction complete until it gets confirmation that the log entry has been written successfully.

so there is no data loss once the database acks the transaction.