The Journal - a proposed syslog replacement

So it replaces text files that can be read and processed with the standard UNIX tools with an undocumented binary format that can only be read by a single tool?

Think I'll pass.

The systemd crew rocks on. Onto the next target.

Why not just send syslog entries to another machine where they can't be changed? Or use the classic approach of sending syslog entries to a line printer, where an attacker clearly can't get at the old entries.

Let's favor low-tech solutions over high-tech crypto-heavy ones here.

this will not scale well to large numbers of events. It's far better to send the logs elsewhere.

and in any case, crypto signing the logs does nothing to prevent the attacker from erasing the files. If each log entry is signed individually then you don't even prevent the attacker from erasing individual log entries. you would have to sign each log entry on top of the prior one to detect gaps.

Please, Lennart, stop messing with the system. You are over-thinking and over-complicating everything. And your unneeded changed are arriving at my systems undocumented. Enough is enough.

Ah, so basically:
– the logs won't be readable with regular unix tools
– the on-disk format will be binary and won't be documented
– there won't be any third-party tools to process the data
– the daemon won't be portable

OBVIOUSLY¸ no sane sysadmin could even imagine a situation where this would cause any problems. Especially when trying to analyse logs from, lemme think, a minimal rescue boot image on a host where the filesystem has been damaged in some way.

Brilliant.

What the authors of this proposal miss is that *simplicity is a feature*, and it's more important than the minor features listed in the FAQ, the lack of which clearly can't be that important because people get along with syslog just fine today.

It's easy to add complexity to software. What's hard, and what really requires discipline and experience, is creating a simple, yet sufficiently powerful system. Simplicity is its own reward because it makes modification, composition, and troubleshooting easier. When you add complexity, you need to justify it by citing the benefits it will bring, and the benefits of this journal thing clearly aren't that important.

We already have a simple solution. Let's not ruin it.

First off, I like the idea of a hash chain, so that all log entries can be check for authenticity. The issue with the first hash can be solved easily by, for example, printing it out and tape it to the machine (maybe with a seal and some holographic prints, so that even a physical attacker cannot change it).

But then I don't get why the logs need to be in a undocumented binary format, inaccessible to anything else. We all know that security though obscurity is a bad thing, and text format can accomodate anything. I don't buy the argument of "easier to analyze programmatically" argument: we can all use XML or JSON or some other organized text format if that is one of the needs. It also facilitates ease of recovery in case of, say, a disk failure.

I wonder why JSON couldn't be used instead of a binary format. It's structured, easily understandable, and can still be processed with tools (sure, maybe simple grep, sed, and awk are out, but I can't imagine there aren't already perl tools for doing XPath/XQuery-like on JSON files yet).

Binary formats are a non-issue, since you can just use a tool that writes it out in text format, and pipe that to whatever.

However, overall, their hash chain idea is cute, but just sending everything to another machine seems far more effective, since instead of just detecting that someone erased the logs, you can prevent them from doing so (as long as they don't compromise the other machine too, that is).

Most of the points they raises are very valid.

Instead of using UUIDs, however, I would suggest using (an hash of?) the format string; if it needs to be changed, a special API can be used which takes both the original format string for ID purposes and the new one for formatting.

I see the discussion that the method might be cumbersome to use, but I don't see any discussion on why this method is secure. My reading of it is that it doesn't add any security at all. If the only requirement is to keep the top hash in a read only location, then I just keep the top entry and recalculate the hashes of the next messages excluding the ones I want to delete.

Note that cryptographic hashes are _not_ digital signatures.

Perhaps syslog should just be replaced with git. systemd could git commit /var/log/messages after every log line. What could go wrong?

That sounds about as good of an idea as the journal does from a high level in every environment _sans_ high integrity / audit environments such as those that need heavy Information Assurance and SELinux lockdown.

<sarcastic mode = ON>
Would sone one PLEASE shoot Lennart?
<sarcastic mode = OFF>

Why not just use a log filesystem and only allow history changes from someone with the right keys? Maybe i'm simplifying things too much.

> posted a detailed document

This seems to need a Google account to read :-(

> the top-most hash is regularly saved to a secure write-only location

Does anyone know what that "secure write-only location" is, in practice? Does my existing computer have one, or is this a new piece of hardware?

I'm also not certain how it avoid this:

blah blah [hash=1]
foo foo [hash=2]
root login from evil.com [hash=3]

Now the "secure write only location" stores 3. The attacker truncates the last entry from the log, and writes 2 to the "secure write-only location".

What have I missed?

I'd like to propose that Lennart get together with Daniel J. Bernstein to do their own distribution of Linux that works completely differently from the way the rest of us want.

It's really too bad that the OpenBSD folks have such a bad attitude toward Linux; when they set about making a more secure replacement for some bit of Unix infrastructure, they're quite good at doing it in a way that respects the way people expect (and want) Unix to work.

The proposed use-case is silly. Everyone uses remote logs after a break-in.

Lots of Unix flavors have had binary syslogs. Fortunately, they are mostly dead or dying.

Alright, since this is an RFC of sorts I'll add my 2c

Firstly, I'm not sure I share the author's conviction that syslog is profoundly broken. Drastically changing core tools in a system administrator's toolkit is something that needs to happen from time to time, but the very change itself will have a negative impact, as well as numerous unforseen consequences, and the justification will have to be a compelling one. The article mentions message source validation, log rotation, and uneven disk usage due to chronological log rotation as drawbacks of the current rsyslogd system. None of these things require a wholesale replacement of the tool or even its output format. The cryptographic hash chain also seems to be a solution in search of a problem; the primary problem with the kernel.org compromise AFAIK was overly permissive access policies, and log files are only a small part of the post-mortem forensics. I'm far too young to be against all changes to the world for any reason, but the issues presented in the proposal seem to be tangential to the changes in journald itself.

Secondly, the rigid bidirectional coupling between systemd and journald smells very bad to me. I like systemd because it is actually good at decoupling things and being filesystem-centric, two traits that are held up as integral parts of good UNIX design. Systemd's site-specific overrides that can exist independently of unit files in software distributions, configuration of system policy by symlink manipulation, and a restricted declarative unit language with easily-modelled semantics are points in its favour. However, journald is a step backwards; this bidirectional coupling just creates a monolithic entity with questionable design, and this decision does not come with any justification. Conceptually a logging service ought to exist independently of a service manager, though the service manager could be highly dependent on the unique feature set of a logging system. Similarly, the telescreen-like inability to turn it off instead of merely turning it down is just downright bizarre, and I can't think of any good reason for that. Monitoring systems should be passive; I should be able to remove journald from my system entirely and suffer no loss of functionality. This is a key design principle of just about every other logging system out there.

Unique and globally-namespaced identifiers for different classes of events together with machine-readable key-value tags are an excellent idea, and definitely serve as a good justification for making this change, but I'm not sure about using UUIDs. The use of Java-style reversed domain name prefixes has compelling advantages over UUIDs, because 8fa69db4-0a89-4c7b-b715-7e7ea93233c7 doesn't exactly roll off the tongue, nor is it at all memorable. On the other hand, I could potentially search for "journald bad sector event" and discover that the tag for this message is "org.kernel.blockdev.media.bad_sector". I can then search for that string online and find far more posts, log fragments, cross-references, and diagnostic scripts referencing that particular kind of event than I would find just searching for a UUID, since people are far more likely to remember the text string and cite it in discussion groups than they are to post a UUID. The only disadvantage to this style of identifier given in the article is that it is inefficient and that namespaces would have to be explicitly managed. This focus on execution efficiency and namespace collisions comes from a programmer's perspective, not from a system administrator's perspective, and far more system administrators will have to deal with this system than programmers will have to maintain journald's implementation. That, and, there is of course the old quote about sins being committed in the name of efficiency.

Finally, I wish the authors would choose a different name than "The journal". That word is already taken in the sysadmin's lexicon: a journal is a filesystem data structure, for heaven's sake. Then again, databases refer to journals as logs, so I suppose I can't get too upset about systemd referring to logs as journals :)

(FWIW I'm a programmer, not an administrator as such)

Most of the 14 points made in the link are covered in some way by rsyslog.

Rather than reinvent the wheel, why not resuse what is already there?

I've even persuaded it to create MySQL based logs of emails through Exim + Spam Assassin + SA-Exim + ClamAV.

Good luck making that easier via Journal.

At least Rsyslog has been around for ages and developed by someone who is _somewhat_ focused on one thing.

As MS et al have found to their cost as well, logging is not easy and does not lend itself to many standards.

Facilitating parsing and passing is useful however - and for that RSyslog is a bit good.

Cheers
Jon

Why do you think that it is necessary to modify the established log file format. The features described here can be kept in their own tracking file maintained by the syslog system without a single modification to any of the existing files or formats. If people do not want this performance wasting feature then they can do without. I haven't seen a wide spread problem with these log files. Hardened systems track entry in multiple ways using redundancy and isolation in security mechanisms. They try to avoid single-points of failure whenever possible. It has been suggested that log files were modified to hide the tracks of hackers but how often have they succeeded? They didn't in the case mentioned since they got caught.

Security is a constantly changing field so vulnerabilities like these have to be examined. But we have to consider the impact on performance. Generating hash codes is not a simple or quick operation when you are considering it in relation of a heavily used transaction processor. Hash codes on the syslog may not noticeably impact a low demand but it will never be used on a high-demand server.

In any case, I am against use of binary files for system administration of UNIX/Linux systems. Human readable ASCII files preserve compatibility, make it easier for system operators to tell what is going on, and make it easier for new users to learn how the systems work. What do you do when you binary log reader is hacked? How can you tell the difference when it is telling you the truth or is lying to you?

... as a sysadmin I'm DELIGHTED to see someone looking at fixing the painfully unstructured, horrid and braindead logging infrastructure used on Linux and UNIXes. Thanks for having the guts to tackle this task - despite surely knowing the hostility and change-resistance you'd face. The state of *nix logging is miserable and I'm delighted to see some progress in this area after endless years of logging so crap it makes the Java logging fiasco look good in comparison.

I'm not completely comfortable having the file format undocumented, though I understand why you want to do so _initially_ for agility and the ability to make quick fixes/improvements.

At the very least a format version header is needed so that tools written to read the format (which there *will* be, documented or no and whether or not there's a good reason) can at least detect newer and unrecognised versions of the format.

FWIW, I like the direction systemd is going in as well, so thanks for the good work so far. Linux needs a real core overhaul if it's going to be maintainable for future workloads as they move away from the desktop and to multiple interconnected sometimes-on battery-powered devices, and systemd provides a useful foundation for some of that. Thanks for enduring the hate mail from change resistant types.

Just to comment on the specific quote in the original post. There is absolutely no reason syslog entries we have today could not include a cryptographic hash. Therefore, calling that out as a "feature" needs to be countered promptly less some become confused that only by replacing another fundamental piece of Unix technology can we be "secure" from kernel.org breakins in the future.

As to the notion of a binary only, undocumented, non-standardized logging format...I don't know where to begin so in the interest of my blood pressure, I won't comment further on that.

-Forensics. The biggest problem I have had recently is the default logrotate configuration that deletes everything after a month. You don't have to be a hacker and delete the logs lograte is configured to delete
the logs for you.

- Structured and more paresable syslog. That is known as RFC5424, and is already supported by rsyslog. It might help to get applications to take better advantage of the new format but a better backwards compatible syslog has already been done and is standardized.

Can we please just take advantage of the tools we have rather than embarking on yet another NIH adventure?

Another practical concern: Lennart is developing a lot of new stuff, supposed to sit at the core of a Linux system.
PA, systemd, bluez, now this, was there more ?

Every piece you write you have to maintain in the future. Even more so for such core tools. This adds up.
Will Lennart be able to support and maintain all the things he has developed (and will develop if it continues like that) in the next 10 or 20 years ?

Alex

I don't understand why there are no way to allow {,u}mount only if answer to private key challenge is known. Block device layer could be modified the way that only file read, create and append are passed to file system. Additionally allowing rename might be sane, but not across file system. Basically that would make block device to be something in between traditional read-write & read-only.

This might enable extra security with existing file systems and utilities, and would perhaps be suitable ftp file archive. Logs are more difficult, they are often compressed which does not work without being able to delete uncompressed file. I guess to overcome this choosing a file system that supports transparent compression is the way to go.

The Journal proposal sounds interesting. I understand the proposal breaks old ways of working, and I am not worried about that; sometimes old bad habits has to go to give room for new generation of solution. That is in align with Hume's (is-ought) Law. More importantly it is easy to defend what is, and tell old must not break, without really contributing how thing ought to be. If it is unclear I am aside of study how things ought to be, including intuitively wrong and crazy ideas. People who abandon alternatives without careful thinking are not thinking, and therefore their opinion is less important.

p.s. For people who do not read old philosophers; the Hume's Law can be expressed many ways, and here is two 'you cannot determining how things ought to be from how they are', or 'how things are is not necessarily how they ought to be'.

Is it not possible to achieve consistency of log files without making the log files binary ? They say it may increase complexity but won't this increase complexity too. People will be really unforgiving if high CPU etc on heavily loaded servers is due to logging.

Syslog already supports different sinks like databases, network. Is it hard to add authentication to those ? I have seen 0MQ with rsyslog being implemented -- so basically you have all the freedom to add all the boilerplate auth you need.

Lastly, I don't see this being used on any desktops, only on servers perhaps.

Also, one of the comments states -- instead of gzcat i will use <tool> .. But for uncompressed text based logs, picking one tool from a plethora is what makes it best.

Wouldn't it be a lot simpler to use git to store todays logfiles. AFAICS it could provide the same level of security and every Unix person needs to learn git anyways (IMO).

Has Lennart gone mad!?
You secure logs by sending them to a remote host (or two) and making sure they can only be administered by a handful of people.

The first rule about security is ACCESS.. If they have access to delete the logs or destroy the system, that's all they need.

Heh, a write-only location for the initial seed of the logs is silly... How is it going to be read in order to verify the first entry? Why wouldn't root just write a new value?
Again, I re-iterate what has been known for YEARS: to secure logs of events, things should be sent to a remote syslog (/rsyslog) server.

This custom binary rubbish is just plain madness.

(BTW, I can see a great amount of sense and reasoning behind systemd/puleaudio - which is why I'm so surprised)

the idea that rate limiting is a defense against a DOS is invalid, if the logging infrastructure can't keep up and therefor slows down the rate at which other systems can generate logs, it doesn't prevent a DOS, it creates one by making the other systems stop responding while they wait for the logs to be written.

In some cases your logs are important enough to do this, but even in many places where security is very important, availability is still more important than guaranteeing that every log message gets saved.

the compression that they talk about here sounds very impressinve and sophisticated, but looking for identical strings of text and replacing them with shorter placeholders (i.e. pointers) is exactly what gzip, bzip2, etc do today. I would be surprised if the journal software was really able to do much better.

with many TB of real-world logs, I'm getting the following results

gzip -9 give me 10:1 compression

bzip2 -9 gives me 20:1 compression but is significantly slower

doing a zgrep or zcat from a compressed file is actually faster than grep or cat from the uncompressed file (on a fairly sophisticated disk array.

I haven't yet done tests on lzma compression

struct journal_log_attribute
{
    enum journal_attribute_type type;
    union {
        journal_attribute_guid guid;
        journal_attribute_keyval keyval;
        journal_attribute_module module;
        /* etc */
    };
};

void
journal_vsyslog(
    int priority, 
    const char* msg, 
    va_list args, 
    struct journal_log_attribute* attributes[] /* NULL-terminated */)
{
    if (journald_is_active()) {
        journal_internal_vsyslog(priority, msg, args, attributes);
    }

    vsyslog(priority, msg, args); /* Ignore attributes */
}

Binary logging is a very bad idea.

"oh, crap, it seems like my hard disk broke down. Let's quickly check the logs to see what happened."

"error: could not read log file: log file corrupt."

Wow, if only all of this energy were spent solving the real problem (how the attacker gets in) rather than the tangential problem (how do we know he got there), perhaps all of this would be rendered... moot?

I keep hearing the same pattern in this thread over and over again, which is something like this:

"Once we get the binary file, we'll use tool X to convert it into a text file so that we can actually make use of the thing.."

Am I the only one that appreciates the irony here?

One of the big problems with implementing a binary base log file system is as mentioned already, only one tool can read it correctly. And you loose the security if someone finds out how the data is stored. It's the exact same thing "Problem" as a text based log file. The other problem is a lot has to change in order for programs to work with the new system, i'e dmesg needs to be changed, boot loaders (including GRUB) have to add tools to allow people to read logs in case something goes wrong with the system and it can't boot.

Also I can not see how you can use the same methods as before to clean the logs, and to backup these logs. For example what happened if you wanted to view that log file in Windows, how would you do this. From a tool point of view you need the correct security tokens (which can be easily bypassed), etc.

You also have to remember routers use the syslogd protocol to send messages to a unix systems in case of DDOS, DOS and port scans etc. How will this be handled?

I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file. Anyone can add to it and anyone can remove lines from it. It is up to the kernel and the logging program to control who can do what. The point about the syslog files being text only is mute as normally only root can write to the file and a binary file has the same "problem".

Why not use XML or something like that so that tools can still read and parse it, the log files can still be read even if the system can not boot. and provide a tool that can control access to the log file.

Lets not follow some stupid Windows way of doing it when we have a tried and test way we have used for years.

On a side note, should we even be looking at the security of the log file when only root can change it and if the hacker has root access. they can do a lot more dangerous things then change a log file.

One last thing, your already told who logged the message. as the first part of the message that gets logged is the process that logged it and the processes ID.

Each log entry authenticates the previous ones? So in order to remove a few lines from the log, you have to rewrite the log since that point in time?

Somehow I doubt this will pose a problem to an attacker who is so stealthy he/she manipulates logs. Most attackers just wipe them. That's why remote logging was invented.

After all, you will need a toolset to handle these logs for reading, searching and writing. I'm sure there will be a tool in this toolset which rewrites logs (just like there is for git).

The part where you store the root seed which authenticates the whole logs is also quite opaque. I guess you have to store a new seed every time you rotate logs, otherwise you'll be completely unable to authenticate anything when (parts of) an older log goes missing. So you'd have to have a remote logging protocol which logs these seeds continously, at which point you could just let it log everything and be done with it.

I'm all for enforcing stricter inter-application log formats to make them easier to parse, but this is just solving made up problems instead of the real ones just because it's easier.

Binary formats require tools to read them. Usually you have a single library for a specific format. This creates s Single Point of Failure. It becomes easy to target the tool rather than the data for the bad guy.

A very big strength of syslog is that it can be read by a large number of tools - from cat to Libreoffice. The multitude of tools have saved me on several occasions when I have been rootkited.

The property that a hash validates the whole logging chain of all entries before could be used for some interesting things.

Somebody could setup a public logging server that accepts a new hash every minute for a registered system and creates a hash chain of all entries it receives. The storage needed would be pretty small. This is the "write once" media that is practical because it needs to store only one hash per minute.

No information is disclosed to this service other than "i am logging".

Then there could be a recovery cdrom from the distribution which reads the journal of the public service and automatically validates all logs on the system.