The Journal - a proposed syslog replacement

So it replaces text files that can be read and processed with the standard UNIX tools with an undocumented binary format that can only be read by a single tool?

Think I'll pass.

Let's favor low-tech solutions over high-tech crypto-heavy ones here.

and in any case, crypto signing the logs does nothing to prevent the attacker from erasing the files. If each log entry is signed individually then you don't even prevent the attacker from erasing individual log entries. you would have to sign each log entry on top of the prior one to detect gaps.

OBVIOUSLY¸ no sane sysadmin could even imagine a situation where this would cause any problems. Especially when trying to analyse logs from, lemme think, a minimal rescue boot image on a host where the filesystem has been damaged in some way.

Brilliant.

It's easy to add complexity to software. What's hard, and what really requires discipline and experience, is creating a simple, yet sufficiently powerful system. Simplicity is its own reward because it makes modification, composition, and troubleshooting easier. When you add complexity, you need to justify it by citing the benefits it will bring, and the benefits of this journal thing clearly aren't that important.

We already have a simple solution. Let's not ruin it.

But then I don't get why the logs need to be in a undocumented binary format, inaccessible to anything else. We all know that security though obscurity is a bad thing, and text format can accomodate anything. I don't buy the argument of "easier to analyze programmatically" argument: we can all use XML or JSON or some other organized text format if that is one of the needs. It also facilitates ease of recovery in case of, say, a disk failure.

However, overall, their hash chain idea is cute, but just sending everything to another machine seems far more effective, since instead of just detecting that someone erased the logs, you can prevent them from doing so (as long as they don't compromise the other machine too, that is).

Most of the points they raises are very valid.

Instead of using UUIDs, however, I would suggest using (an hash of?) the format string; if it needs to be changed, a special API can be used which takes both the original format string for ID purposes and the new one for formatting.

Note that cryptographic hashes are _not_ digital signatures.

That sounds about as good of an idea as the journal does from a high level in every environment _sans_ high integrity / audit environments such as those that need heavy Information Assurance and SELinux lockdown.

This seems to need a Google account to read :-(

> the top-most hash is regularly saved to a secure write-only location

Does anyone know what that "secure write-only location" is, in practice? Does my existing computer have one, or is this a new piece of hardware?

I'm also not certain how it avoid this:

blah blah [hash=1]
foo foo [hash=2]
root login from evil.com [hash=3]

Now the "secure write only location" stores 3. The attacker truncates the last entry from the log, and writes 2 to the "secure write-only location".

What have I missed?

It's really too bad that the OpenBSD folks have such a bad attitude toward Linux; when they set about making a more secure replacement for some bit of Unix infrastructure, they're quite good at doing it in a way that respects the way people expect (and want) Unix to work.

Lots of Unix flavors have had binary syslogs. Fortunately, they are mostly dead or dying.

Firstly, I'm not sure I share the author's conviction that syslog is profoundly broken. Drastically changing core tools in a system administrator's toolkit is something that needs to happen from time to time, but the very change itself will have a negative impact, as well as numerous unforseen consequences, and the justification will have to be a compelling one. The article mentions message source validation, log rotation, and uneven disk usage due to chronological log rotation as drawbacks of the current rsyslogd system. None of these things require a wholesale replacement of the tool or even its output format. The cryptographic hash chain also seems to be a solution in search of a problem; the primary problem with the kernel.org compromise AFAIK was overly permissive access policies, and log files are only a small part of the post-mortem forensics. I'm far too young to be against all changes to the world for any reason, but the issues presented in the proposal seem to be tangential to the changes in journald itself.

Secondly, the rigid bidirectional coupling between systemd and journald smells very bad to me. I like systemd because it is actually good at decoupling things and being filesystem-centric, two traits that are held up as integral parts of good UNIX design. Systemd's site-specific overrides that can exist independently of unit files in software distributions, configuration of system policy by symlink manipulation, and a restricted declarative unit language with easily-modelled semantics are points in its favour. However, journald is a step backwards; this bidirectional coupling just creates a monolithic entity with questionable design, and this decision does not come with any justification. Conceptually a logging service ought to exist independently of a service manager, though the service manager could be highly dependent on the unique feature set of a logging system. Similarly, the telescreen-like inability to turn it off instead of merely turning it down is just downright bizarre, and I can't think of any good reason for that. Monitoring systems should be passive; I should be able to remove journald from my system entirely and suffer no loss of functionality. This is a key design principle of just about every other logging system out there.

Unique and globally-namespaced identifiers for different classes of events together with machine-readable key-value tags are an excellent idea, and definitely serve as a good justification for making this change, but I'm not sure about using UUIDs. The use of Java-style reversed domain name prefixes has compelling advantages over UUIDs, because 8fa69db4-0a89-4c7b-b715-7e7ea93233c7 doesn't exactly roll off the tongue, nor is it at all memorable. On the other hand, I could potentially search for "journald bad sector event" and discover that the tag for this message is "org.kernel.blockdev.media.bad_sector". I can then search for that string online and find far more posts, log fragments, cross-references, and diagnostic scripts referencing that particular kind of event than I would find just searching for a UUID, since people are far more likely to remember the text string and cite it in discussion groups than they are to post a UUID. The only disadvantage to this style of identifier given in the article is that it is inefficient and that namespaces would have to be explicitly managed. This focus on execution efficiency and namespace collisions comes from a programmer's perspective, not from a system administrator's perspective, and far more system administrators will have to deal with this system than programmers will have to maintain journald's implementation. That, and, there is of course the old quote about sins being committed in the name of efficiency.

Finally, I wish the authors would choose a different name than "The journal". That word is already taken in the sysadmin's lexicon: a journal is a filesystem data structure, for heaven's sake. Then again, databases refer to journals as logs, so I suppose I can't get too upset about systemd referring to logs as journals :)

(FWIW I'm a programmer, not an administrator as such)

Rather than reinvent the wheel, why not resuse what is already there?

I've even persuaded it to create MySQL based logs of emails through Exim + Spam Assassin + SA-Exim + ClamAV.

Good luck making that easier via Journal.

At least Rsyslog has been around for ages and developed by someone who is _somewhat_ focused on one thing.

As MS et al have found to their cost as well, logging is not easy and does not lend itself to many standards.

Facilitating parsing and passing is useful however - and for that RSyslog is a bit good.

Cheers
Jon

Security is a constantly changing field so vulnerabilities like these have to be examined. But we have to consider the impact on performance. Generating hash codes is not a simple or quick operation when you are considering it in relation of a heavily used transaction processor. Hash codes on the syslog may not noticeably impact a low demand but it will never be used on a high-demand server.

In any case, I am against use of binary files for system administration of UNIX/Linux systems. Human readable ASCII files preserve compatibility, make it easier for system operators to tell what is going on, and make it easier for new users to learn how the systems work. What do you do when you binary log reader is hacked? How can you tell the difference when it is telling you the truth or is lying to you?

I'm not completely comfortable having the file format undocumented, though I understand why you want to do so _initially_ for agility and the ability to make quick fixes/improvements.

At the very least a format version header is needed so that tools written to read the format (which there *will* be, documented or no and whether or not there's a good reason) can at least detect newer and unrecognised versions of the format.

FWIW, I like the direction systemd is going in as well, so thanks for the good work so far. Linux needs a real core overhaul if it's going to be maintainable for future workloads as they move away from the desktop and to multiple interconnected sometimes-on battery-powered devices, and systemd provides a useful foundation for some of that. Thanks for enduring the hate mail from change resistant types.

As to the notion of a binary only, undocumented, non-standardized logging format...I don't know where to begin so in the interest of my blood pressure, I won't comment further on that.

- Structured and more paresable syslog. That is known as RFC5424, and is already supported by rsyslog. It might help to get applications to take better advantage of the new format but a better backwards compatible syslog has already been done and is standardized.

Can we please just take advantage of the tools we have rather than embarking on yet another NIH adventure?

Every piece you write you have to maintain in the future. Even more so for such core tools. This adds up.
Will Lennart be able to support and maintain all the things he has developed (and will develop if it continues like that) in the next 10 or 20 years ?

Alex

This might enable extra security with existing file systems and utilities, and would perhaps be suitable ftp file archive. Logs are more difficult, they are often compressed which does not work without being able to delete uncompressed file. I guess to overcome this choosing a file system that supports transparent compression is the way to go.

The Journal proposal sounds interesting. I understand the proposal breaks old ways of working, and I am not worried about that; sometimes old bad habits has to go to give room for new generation of solution. That is in align with Hume's (is-ought) Law. More importantly it is easy to defend what is, and tell old must not break, without really contributing how thing ought to be. If it is unclear I am aside of study how things ought to be, including intuitively wrong and crazy ideas. People who abandon alternatives without careful thinking are not thinking, and therefore their opinion is less important.

p.s. For people who do not read old philosophers; the Hume's Law can be expressed many ways, and here is two 'you cannot determining how things ought to be from how they are', or 'how things are is not necessarily how they ought to be'.

Syslog already supports different sinks like databases, network. Is it hard to add authentication to those ? I have seen 0MQ with rsyslog being implemented -- so basically you have all the freedom to add all the boilerplate auth you need.

Lastly, I don't see this being used on any desktops, only on servers perhaps.

Also, one of the comments states -- instead of gzcat i will use <tool> .. But for uncompressed text based logs, picking one tool from a plethora is what makes it best.

The first rule about security is ACCESS.. If they have access to delete the logs or destroy the system, that's all they need.

Heh, a write-only location for the initial seed of the logs is silly... How is it going to be read in order to verify the first entry? Why wouldn't root just write a new value?
Again, I re-iterate what has been known for YEARS: to secure logs of events, things should be sent to a remote syslog (/rsyslog) server.

This custom binary rubbish is just plain madness.

(BTW, I can see a great amount of sense and reasoning behind systemd/puleaudio - which is why I'm so surprised)

In some cases your logs are important enough to do this, but even in many places where security is very important, availability is still more important than guaranteeing that every log message gets saved.

with many TB of real-world logs, I'm getting the following results

gzip -9 give me 10:1 compression

bzip2 -9 gives me 20:1 compression but is significantly slower

doing a zgrep or zcat from a compressed file is actually faster than grep or cat from the uncompressed file (on a fairly sophisticated disk array.

I haven't yet done tests on lzma compression

"oh, crap, it seems like my hard disk broke down. Let's quickly check the logs to see what happened."

"error: could not read log file: log file corrupt."

I keep hearing the same pattern in this thread over and over again, which is something like this:

"Once we get the binary file, we'll use tool X to convert it into a text file so that we can actually make use of the thing.."

Am I the only one that appreciates the irony here?

Also I can not see how you can use the same methods as before to clean the logs, and to backup these logs. For example what happened if you wanted to view that log file in Windows, how would you do this. From a tool point of view you need the correct security tokens (which can be easily bypassed), etc.

You also have to remember routers use the syslogd protocol to send messages to a unix systems in case of DDOS, DOS and port scans etc. How will this be handled?

I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file. Anyone can add to it and anyone can remove lines from it. It is up to the kernel and the logging program to control who can do what. The point about the syslog files being text only is mute as normally only root can write to the file and a binary file has the same "problem".

Why not use XML or something like that so that tools can still read and parse it, the log files can still be read even if the system can not boot. and provide a tool that can control access to the log file.

Lets not follow some stupid Windows way of doing it when we have a tried and test way we have used for years.

On a side note, should we even be looking at the security of the log file when only root can change it and if the hacker has root access. they can do a lot more dangerous things then change a log file.

Somehow I doubt this will pose a problem to an attacker who is so stealthy he/she manipulates logs. Most attackers just wipe them. That's why remote logging was invented.

After all, you will need a toolset to handle these logs for reading, searching and writing. I'm sure there will be a tool in this toolset which rewrites logs (just like there is for git).

The part where you store the root seed which authenticates the whole logs is also quite opaque. I guess you have to store a new seed every time you rotate logs, otherwise you'll be completely unable to authenticate anything when (parts of) an older log goes missing. So you'd have to have a remote logging protocol which logs these seeds continously, at which point you could just let it log everything and be done with it.

I'm all for enforcing stricter inter-application log formats to make them easier to parse, but this is just solving made up problems instead of the real ones just because it's easier.

A very big strength of syslog is that it can be read by a large number of tools - from cat to Libreoffice. The multitude of tools have saved me on several occasions when I have been rootkited.

Somebody could setup a public logging server that accepts a new hash every minute for a registered system and creates a hash chain of all entries it receives. The storage needed would be pretty small. This is the "write once" media that is practical because it needs to store only one hash per minute.

No information is disclosed to this service other than "i am logging".

Then there could be a recovery cdrom from the distribution which reads the journal of the public service and automatically validates all logs on the system.