LWN: Comments on "Enabling the persistent journal in Debian"

Wrong decisions

jezuch — Tue, 25 Feb 2020 12:22:20 +0000

Well, the boogeymen evolve with time. Slavery is once again (still, after all these years?) a hot topic in the US, so it's easier for USians to relate to it. Hitler, in comparison, is becoming more and more abstract as historical education falters and the numbers of Nazi sympathizers grow... In some places you'll also see variants involving Stalin or Pol Pot etc. so perhaps what is needed is a generic/parameterized law template ;)

Wrong decisions

flussence — Sun, 23 Feb 2020 04:10:56 +0000

…especially since Godwin himself repealed that law in the face of interesting times.

Wrong decisions

jrigg — Fri, 21 Feb 2020 21:21:10 +0000

> The abolition of chattel slavery in the United States caused trouble for longer than that.

I think we need a replacement for Godwin's Law: As an online discussion grows longer, the probability of a comparison involving slavery approaches 1.

Wrong decisions

Wol — Fri, 21 Feb 2020 18:57:23 +0000

> > I would say that a decision that has caused trouble for 6 years has obviously, scientifically, provably, been a wrong decision.

> Your principle of scientific obviousness holds up about as well as one of Uri Geller's bent spoons.

The OP clearly has no clue what science is, or what a scientific proof proves. ALL scientific proofs come in the form "X can NOT be true, because here is an observation Y that refutes it". You can NOT come to a 100% positive scientific conclusion, because that would not be science.

The only way the OP could come up with scientific proof would be to re-run the same scenario except with different decisions, and *show* that they caused "less trouble", whatever that means. He would then have evidence to back his statement, but that's still not *proof*.

ALL scientific "proofs positive" basically consist of the statement "we cannot find a counter-example" - often because the investigators subconsciously avoid searching in the obvious places ... :-(

That's actually why it's almost invariably the young who make scientific advances - they are dis-satisfied with the wisdom of their elders, and the old who can't make advances because they are entrenched in the beliefs of their youth. History is replete with examples of "Grand Old Men" of Science who, because they were revered for their discoveries as youngsters, they were absolute menaces in old age.

Cheers,
Wol

Wrong decisions

branden — Fri, 21 Feb 2020 01:52:54 +0000

> I would say that a decision that has caused trouble for 6 years has obviously, scientifically, provably, been a wrong decision.

The abolition of chattel slavery in the United States caused trouble for longer than that. Eventually, political and business leaders grew tired of Reconstruction and Rutherford B. Hayes bargained it away in 1876 as the price of the presidency.

Thus the U.S. South entered the period of "Redemption"; it implemented Jim Crow, meaning slavery by another name, the enshrinement of white supremacist ideology into the law to go along with its cultural norms, and that quieted things down (along with lynchings that grew in number for decades, peaking in the 1890s. This sort of violence was not "trouble" in the political sense; the rest of the country left the racist South to its own devices, and frequently emulated its example.

Your principle of scientific obviousness holds up about as well as one of Uri Geller's bent spoons.

6 years !

mpr22 — Thu, 20 Feb 2020 19:04:07 +0000

Which of the other outcomes that were credibly on offer (Further Discussion aka passive sysvinit as default; affirmative sysvinit as default; openrc as default; upstart as default) do you believe would have caused a shorter and/or less severe period of bad relations?

Enabling the persistent journal in Debian

JFlorian — Thu, 20 Feb 2020 18:43:01 +0000

I simply meant that our computers are writing files constantly. I don't worry about corruption in my log files more than any other file. I have lots of value held in databases and those writes are going to be far more complex than log files, but I trust the job will get done.

As for FSS, I'm referring to it's Forward Secure Sealing feature as discussed here: https://lwn.net/Articles/512895/. I'll admit right now I don't know if any distro has this enabled by default or even if I'm using it, but I like the concept and thought it relevant here.

In the end, I too like plain text log files for their simplicity and versatility but I'll never be able to query them as powerfully and easily as I can with journalctl and all that power doesn't scare me off for the modest amount of complexity that I imagine it adds. That and having never been bitten by journal corruption despite plenty of invitations for disaster, I welcome the advancement in logging.

Enabling the persistent journal in Debian

mathstuf — Thu, 20 Feb 2020 14:33:03 +0000

> FSS

Forward secure sealing.

https://lwn.net/Articles/512895/

6 years !

seyman — Thu, 20 Feb 2020 10:59:25 +0000

> I would say that a decision that has caused trouble for 6 years has obviously, scientifically, provably, been a wrong decision.

IIRC, Debian TC had the choice between 3 options (keep sysvinit the default, choose upstart as the new default, choose systemd as the new default). No matter which way they would have voted, the result would have been years of bad relations so I'm glad they choose to focus on technical merits.

Enabling the persistent journal in Debian

HelloWorld — Thu, 20 Feb 2020 10:53:32 +0000

> Systems with no remote logging and no resources to analyse those logs can probably switch to journald without too much pain. I am biased against systemd on servers because it requires dbus and I can't see any reason to want dbus (or udev, udiscs2, etc) on an internet facing server.

systemctl uses the D-Bus protocol to talk to systemd, but systemd does not require dbus-daemon (or dbus-broker) to be running. It also doesn't require udev or udisks2 [sic].

6 years !

Zolko — Thu, 20 Feb 2020 10:36:03 +0000

"it's been almost exactly six years since the Debian technical committee voted to make systemd the default for the "jessie" release."

I would say that a decision that has caused trouble for 6 years has obviously, scientifically, provably, been a wrong decision. No matter how "democratic" the process to come to that bad decision has been. The decision has, observably, led to 6 years of bad relations. Therefore, it was a bad decision.

Enabling the persistent journal in Debian

qtplatypus — Thu, 20 Feb 2020 08:32:52 +0000

"In the greater context of everything our computers do for us on a daily basis, that seems an overly trivial argument."

I am unsure what you mean here can you expand in this argument?

The perspective I am coming from is where a system is needed for something important (like logging) its persistant state should change from consistant state to consistant state atomically. So there is no possible window where corruption can occure.

I prefer systems that are intrinsically safe to systems where you have to be lucky.

Also can you expand FSS in this context, i am thinking 'Fair Share Schedular' but that is clearly wrong.

Enabling the persistent journal in Debian

davidstrauss — Tue, 18 Feb 2020 19:45:45 +0000

The systemd journal *does* log text in key/value pairs. It's just in an indexed, binary format on disk. Various key/value output formats, including JSON, are available via journalctl, HTTP (by enabling the journal's REST-like API server), or various libraries for popular languages.

Enabling the persistent journal in Debian

JFlorian — Tue, 18 Feb 2020 13:27:46 +0000

In the greater context of everything our computers do for us on a daily basis, that seems an overly trivial argument. Again, I've yet to encounter corruption despite no end of improper shutdowns and other poor circumstances. Maybe it's just an awesome file system (mix of ext4 and xfs) or something else, but the cons of fearing a journal corruption simply do not outweigh all the pros, IMHO. Furthermore, I suspect FSS will make very obvious any potential corruption, does your plain text log file guarantee that?

Enabling the persistent journal in Debian

qtplatypus — Tue, 18 Feb 2020 04:43:21 +0000

My understanding is that journald fsync's its content to disk periodically and that adding to the journal requires two writes. One to the end of the file with the logged content and one to the start to update the header. So there is more to go wrong compared to a text file that is being appended to.

Enabling the persistent journal in Debian

mathstuf — Mon, 17 Feb 2020 22:22:14 +0000

Issues I can think of off-hand with that:

- escaping protocols need to be defined (at least newline and the escape character; control characters likely need help too)
- searching performance sucks (without an index at least); grep gets me the one line I'm interested, but how much context do I need for the related field I'm *actually* interested in?
- delimiting records needs some kind of separator (and can't be crafted by another log message)
- getting a whole record for some single field match is not easier than journalctl's filtering (I can think of awk doing it, but that doesn't sound nicer than `-u` and the like)
- encoding? everybody loves encoding questions (yeah, utf-8 is the Right Answer, but since when has that stopped anyone?)
- timezones are probably going to be in ASCII and I know no one has ever messed up ASCII timezone conversions before, so maybe that's OK too ;)

Yeah, journalctl is slow in a number of cases, but I think that could probably be avoided by using its query interface rather than piping to less, using its search and buffer stuff rather than asking journalctl for the thing I'm looking for then crafting a --since and --until parameter to bracket some reasonable timeframe around the event (loses easy perusing, but less' buffering is…not stellar).

Enabling the persistent journal in Debian

k8to — Mon, 17 Feb 2020 21:28:42 +0000

Logging in text in k-v pairs has been best practice for decades. It's a shame journald doesn't just do it.

Enabling the persistent journal in Debian

mbiebl — Mon, 17 Feb 2020 09:39:11 +0000

As I was curious I pulled the ISO from https://coreos.com/os/docs/latest/booting-with-iso.html and booted into that CoreOS system.
I could read journal files from a Debian 10 and Debian sid system without any problems.

I would have loved to find out what specific problems you had, but given your reply it's probably too late for that now.

Enabling the persistent journal in Debian

niner — Sun, 16 Feb 2020 08:07:13 +0000

You mean the rate limiting? That's configurable. Simply add RateLimitIntervalSec=0 to your /etc/systemd/journald.conf

Enabling the persistent journal in Debian

liam — Sun, 16 Feb 2020 07:48:32 +0000

Those are, or were, in some cases, legitimate issues, so, no, I don't take you for a hater:)
Looks like you ackd the fix for 3, and, as you said 2 is supposed to be fixed. The docs for journal-remote.conf are a bit sparse, but the lack of solution for unit tab completion is pretty damned annoying. Occasional indexing may not be elegant but the alternative of 1m+ tab completion times shouldn't be acceptable.

Enabling the persistent journal in Debian

mbunkus — Sat, 15 Feb 2020 11:57:28 +0000

I ended up using stunnel on both sides. Nothing against nginx; I use it everywhere else. stunnel is simply quite a bit less complex and a tool suited for the job at hand.

Note that Michael Biebl re-opened my but report about "journalctl --vacuum-* --directory=…" not working (probably after reading my post here) and asked us to verify if this is still the case. My tests showed that it does work now. I haven't tested automatic cleanup (meaning whether settings in /etc/systemd/journald.conf affect the contents of /var/log/journal/remote or any of its sub-directories), but at least you can run "journalctl --vacuum-* --directory=…" in a timer for similar effect.

Enabling the persistent journal in Debian

tedd — Sat, 15 Feb 2020 10:55:52 +0000

Kang for lyfe.

Enabling the persistent journal in Debian

fwiesweg — Sat, 15 Feb 2020 10:00:14 +0000

Yes I can confirm those issues, especially the missing auto-vacuuming of journal-remote is sorely being missed.

Concerning the certificate verification issue, have you tried routing it through an https proxy? We're running nginx on the same machine for other services incapable of serving via https anyway. It'd be pretty straightforward to have the journal bind to localhost only, delegating all the certificate verification to nginx.

Enabling the persistent journal in Debian

ju3Ceemi — Fri, 14 Feb 2020 21:50:49 +0000

Journald randomly "misses" logement en triés, which is not acceptable

Enabling the persistent journal in Debian

kreijack — Fri, 14 Feb 2020 19:48:44 +0000

Even I like the idea behind journald, I suffered for its poor performance when used on a BTRFS filesystem.
I wrote even a little blog about that
http://kreijack.blogspot.com/2014/06/btrfs-and-systemd-jo...

In order to solve this issue and to not lost the advantages of journald, I configured it in order to forward the messages to rsyslog in "structured format" (or to be more correct, I configured rsyslog to import). Then I wrote a tool in order to parse/query the log stored by rsyslog (formatted in json-like format).

This I give me the advantages:
- a clean file format which doesn't fight with btrfs
- the possibility to query the log using the key/value filter

Enabling the persistent journal in Debian

ibukanov — Fri, 14 Feb 2020 18:18:26 +0000

That CoreOS server was decommissioned couple of month ago. Note that I was able eventually to look at the log from a Fedora laptop so it was not corrupted.

Enabling the persistent journal in Debian

tchernobog — Fri, 14 Feb 2020 15:27:11 +0000

I do have a job that includes administering servers, and the journal saved my life more than once where grep could not cut it. See? Subjective.

Enabling the persistent journal in Debian

JFlorian — Fri, 14 Feb 2020 14:39:44 +0000

Citations please. I oversee nearly a thousand deployments and have yet to encounter a corrupted journal. I'm not saying it can't happen but I have zero reason to believe systemd-journal to be any more vulnerable to mangled data than any other app that writes files.

Enabling the persistent journal in Debian

Baughn — Fri, 14 Feb 2020 14:32:57 +0000

I've never once seen one.

I wonder if they're more common on filesystems like ext4? I use ZFS across my entire fleet.

Enabling the persistent journal in Debian

Kamiccolo — Fri, 14 Feb 2020 12:16:26 +0000

Well, corrupted journald log files are pretty common. It would be nice to dig through the specifics and percentages on different use cases....

Enabling the persistent journal in Debian

mbiebl — Fri, 14 Feb 2020 11:52:02 +0000

Could you by any chance provide the output of journalctl --version from that CoreOS installation?

Enabling the persistent journal in Debian

ibukanov — Fri, 14 Feb 2020 10:59:31 +0000

The journal was created on Debian 10. I inspected it the last summer on CoreOS with automatic updates enabled.

Enabling the persistent journal in Debian

mbunkus — Fri, 14 Feb 2020 10:35:31 +0000

I've tried using systemd-journal-{upload,remote} for more than a year and was less than impressed. In my scenario I had to log over the open internet, meaning I had to implement encryption at some level.

In no particular order:

1. The documentation (man pages) are copy-pasted and wrong wrt. the configuration of encryption (certificates). See [1].
2. A real killer: systemd-journal-remote neither requires client certificates, nor does it actually validate them against the CA I've told it to validate against. See [2]. This is supposed to be fixed, but by that time I had moved away from the solution to verify.
3. The configuration wrt. resource usage (how much disk space journal files may use) does not affect the remote journal direct. Worse, running "journalctl --vacuum-* --directory…" does not actually remove anything. See [3]. The bug is closed, but that's wrong; several people in the bug asked for re-opening as the issue is still present. The result is the remote storage growing huge without manual usage of "rm …". This leads me to…
4. journalctl is dog slow in a lot of scenarios as it has to open & read all relevant files. Even on SSDs. Read [4] for details; I think it was mentioned earlier in this thread or over on the debian-devel list. Couple that with a central log host being, well, the collector of logs of all clients and you end up with a huge amount of files with a huge amount of data to read through, and journalctl becomes really unpleasant to use.
5. systemd-journal-upload remembers how much it has already uploaded. If the connection between it and -remote had been down for a while, -upload tries to upload everything since that last position _before it signals readiness to systemd_. The result is that "systemctl start systemd-journal-upload.service" may time out (e.g. after the default 1m30s) if -upload is still in the process of uploading. Then systemd kills the process, and you have to try starting it again. This is mostly an issue with the default unit configuration for -upload. I worked around it by adding "Restart=always" and "TimeoutStartSec=<some big value>" in a unit drop-in for it.

[1] https://github.com/systemd/systemd/issues/4093
[2] https://github.com/systemd/systemd/issues/4092
[3] https://github.com/systemd/systemd/issues/2376
[4] https://github.com/systemd/systemd/issues/6238

As workarounds for the encryption problems I experimented with using ssltunnel and a real VPN between the machines and running systemd-journal-{remote,upload} unencrypted over those

I realize I sound like a hater. Which I'm totally not. I'm a huge fan of the journal storing all that meta data, the JSON format is perfectly fine, the filtering mechanisms are great, and systemd in general has brought so much joy to me as a sysadmin. The journal, or more precisely, remote journal just not being quite there in quality & ease of use & polish is just hugely disappointing to me.

Enabling the persistent journal in Debian

mbiebl — Fri, 14 Feb 2020 09:29:17 +0000

I'd be interested to know more about that:
Which systemd version (and which distro) was used to create the journal?
Which systemd version (and which distro) was used to inspect the journal?

Enabling the persistent journal in Debian

ibukanov — Fri, 14 Feb 2020 08:11:32 +0000

I used to like journald since it’s UI allowed to quickly look only at relevant things (especially after systemd fixed annoying bugs that lead to log entries with missing tags).

But then one day I needed to analyze a log from a crashed server. Surely journalctl supports viewing individual log files just fine. Except that time I got a message about unsupported journal version. Apparently the crashed server had a newer systemd that changed the journal format.

That showed that designers of journald did not understand the logging and the need for forward compatibility in the log file format.

Enabling the persistent journal in Debian

rvolgers — Fri, 14 Feb 2020 07:32:09 +0000

(To be clear, those would both be perfectly valid, I'm just wondering if there's more to it because I'm curious. I've used both "journald only", "rsyslog only", and hybrid solutions, and reliability has not really been a huge factor except in the sense that having remote logging at all is a big factor.)

Enabling the persistent journal in Debian

rvolgers — Fri, 14 Feb 2020 07:26:32 +0000

I'm curious what kind of problem journald has with reliability. Is it just that you need features provided by rsyslog and don't want an extra layer involved in logging, or that you simply consider rsyslog to be the more mature solution, or something more specific?

Enabling the persistent journal in Debian

liam — Fri, 14 Feb 2020 00:54:32 +0000

What are the problems with systemd-journal-(upload & remote) (for networking) and the journal export format (for readability)?

Enabling the persistent journal in Debian

dps — Thu, 13 Feb 2020 23:37:12 +0000

If I had a job administering servers then I would want remote logging which both syslog and rsyslog provide in a readable format. There are lots of tools, both free and commercial, which understand this format. Using jounald instead would be a huge move in the wrong direction.

If a server has been owned then I don't want to be relying on local logs, which might have been doctored. Remote logs which the attacker can't doctor are worth much more.

Systems with no remote logging and no resources to analyse those logs can probably switch to journald without too much pain. I am biased against systemd on servers because it requires dbus and I can't see any reason to want dbus (or udev, udiscs2, etc) on an internet facing server.

I might also decide modules are bloat on servers and build a kernel with everything required built in and everything else left out. This might be because one the first things one did in the days of 0.9pl11 and fast 486DX2/50 boxes was to build a kernel which supported only the hardware one actually had.

Enabling the persistent journal in Debian

mbiebl — Thu, 13 Feb 2020 21:00:47 +0000

If you have buster system which has rsyslog installed and you upgrade that to bullseye, rsyslog will *not* be removed during the upgrade. Rsyslog will continue to work as before.

It's under discussion whether to lower the priority of rsyslog from important to optional. The effect of that change would be that for system that is setup from scratch, rsyslog would no longer be installed during the intial setup.