LWN: Comments on "The Journal - a proposed syslog replacement"

The Journal - a proposed syslog replacement

topher — Tue, 20 Dec 2011 08:03:33 +0000

That's not an argument against binary logging.

Actually, it is. And it's a valid one. Take the case of a corrupted block on your filesystem. We're not caring how it got there (physical problem, filesystem problem, whatever), but it's there. If you've got a text file with a corrupted chunk in it, you can generally recover everything except for the corrupted part with no special tools.

Now imagine the same scenario, but with a binary file. You better hope and pray that whoever wrote the tools for that (currently undocumented) binary format have specialized tools for analyzing and recovering from file corruption. Otherwise, there's an excellent chance you just lost that entire file. Depending on the the internal format of the file, you might be screwed no matter what (especially if the file is doing internal compression functionality, which could mean your corruption just "infected" your entire file).

The Journal - a proposed syslog replacement

topher — Tue, 20 Dec 2011 07:46:14 +0000

For the first, I'm not sure requiring HW is a non-starter. It would have to be cheap---say $20 or less---to start with.

Yes, it is a non-starter. There is no computer (or parts) manufacturer that is going to start including specialized hardware, even if it only cost $0.01us, for a system that doesn't exist yet, and that hasn't been adopted.

Especially when a lot of people, including some of us who have spent years dealing with logging retention, access, security, processing, alerting, etc, look at this and think it's a bad idea.

The Journal - a proposed syslog replacement

ndecker — Mon, 05 Dec 2011 14:06:45 +0000

The property that a hash validates the whole logging chain of all entries before could be used for some interesting things.

Somebody could setup a public logging server that accepts a new hash every minute for a registered system and creates a hash chain of all entries it receives. The storage needed would be pretty small. This is the "write once" media that is practical because it needs to store only one hash per minute.

No information is disclosed to this service other than "i am logging".

Then there could be a recovery cdrom from the distribution which reads the journal of the public service and automatically validates all logs on the system.

The Journal - a proposed syslog dropout?

gvy — Thu, 01 Dec 2011 17:06:45 +0000

It's weird that neither msyslogd or the preceding ssyslogd are even mentioned -- it's much better to reinvent the wheel at least knowing of prior art, /and/ to criticize the process thereof either. :)

Thanks Solar Designer, here's the link (in Russian).

making the logs temper evident through git like hash chains

dlang — Sun, 27 Nov 2011 05:47:19 +0000

however, since systems don't actually include WORM memory, and are very unlikely to (except for very specialized systems), how does that actually help?

remember that WORM memory needs to be a replaceable thing since by default you can't erase it to make room for new data.

The Journal - a proposed syslog replacement

HelloWorld — Sun, 27 Nov 2011 02:23:44 +0000

> with plain text formats I can use the *same* collection of tools for everything, and every new thing i learn about sh or sed or grep or awk or perl or whatever is automatically useful in hundreds of other contexts, not just the context in which i originally learnt it.
I found that in most cases, it is easier to learn a new tool that is specialized for the job at hand than to try to get "standard unix" tools to do what you want, especially if you want to do it in a robust and maintainable way. For example, people keep asking again and again how to handle some XML format with things like sed or awk, which is just a bad idea given the existence of specialized tools like xmlstarlet.

making the logs temper evident through git like hash chains

rgmoore — Sun, 27 Nov 2011 02:18:03 +0000

But the big security benefit in that case is from the existence of the WORM memory, since any data written to it is inherently tamper-proof. You could stick to an un-hashed text log and still have confidence that it hadn't been rewritten by an intruder. The benefit of the hash chain is that you can provide tamper evident recording by keeping only a fraction of the hashes, which is most important if the WORM storage is expensive or difficult to deal with. Of course keeping only a fraction of the hashes leaves open a potential window if the attacker can break in an alter the records between writes to WORM.

Not very thought through proposal

dlang — Fri, 25 Nov 2011 23:31:30 +0000

one argument could be that remote seed logging is low bandwith compared to full remote logging.

I don't think it's a compelling argument (especially in light of all the complexity involved here, etc), but it's an argument.

by the way, it turns out that there is a RFC on how to properly secure logs, RFC5848 that has been through the mill of analysis, both from a crypto point of view and it's limitations (http://www.gerhards.net/download/log_hash_chaining.pdf)

Not very thought through proposal

job — Fri, 25 Nov 2011 22:27:51 +0000

Indeed there will be such tools. It is required as the format is undocumented binary.

The reason some attackers manipulate logs with sed today is simple because it's possible. They will use another tool when it's not.

I hope some of the proponents of this proposal would answer the details about remote seed logging and what problems this would solve as opposed to simply remote logging. This part is completely left out, and I still think the misunderstanding is on my part as it is simply a much too ill thought out proposal otherwise.

The Journal - a proposed syslog replacement

HelloWorld — Fri, 25 Nov 2011 16:26:27 +0000

> Unfortunately, Lennart has a fairly bad track record.
Well, Lennart always cared about having a working migration path. PulseAudio supports the old ESD protocol and features an ALSA plugin. Systemd supports traditional init scripts just fine, and the journal can be used via the traditional syslog(3) api.
Also, I'd say that having someone like Lennart who has the guts to try to get rid of some established but broken/limited practise is an asset, not a liability.

The Journal - a proposed syslog replacement

jeremiah — Fri, 25 Nov 2011 15:02:05 +0000

those are on my list. It's mostly an issue of upgrading servers at this point to version that support them.

The Journal - a proposed syslog replacement

lindi — Fri, 25 Nov 2011 13:51:56 +0000

Doesn't that mean that the attacker can choose which tool to attack and always pick the weakest? For example if you use "cat" then you can be subject to issues described in the 2003 paper titled "TERMINAL EMULATOR SECURITY ISSUES".

Not very thought through proposal

dlang — Fri, 25 Nov 2011 10:11:15 +0000

there will need to be distro bundled tools to manipulate the logs.

there are just too many cases where you need to deal with parts of the logs and you don't want to have to deal with all of the logs, so you need to do the equivalent of grep or grep -v

the only way it will slow the attacker down is as it is first introduced via 'security by obscurity', If it ever does become the standard, the attackers will just use the appropriate tools.

Not very thought through proposal

intgr — Fri, 25 Nov 2011 09:57:56 +0000

> Somehow I doubt this will pose a problem to an attacker who is so stealthy
> he/she manipulates logs. Most attackers just wipe them. That's why remote
> logging was invented.

Nope. I've seen intruders simply using 'sed' to delete the lines they want to hide. I do think that Journal will defeat log manipulation by many simpler attackers, simply because there are no distro-bundled tools to manipulate them.

The Journal - a proposed syslog replacement

jacob22 — Thu, 24 Nov 2011 22:16:26 +0000

Binary formats require tools to read them. Usually you have a single library for a specific format. This creates s Single Point of Failure. It becomes easy to target the tool rather than the data for the bad guy.

A very big strength of syslog is that it can be read by a large number of tools - from cat to Libreoffice. The multitude of tools have saved me on several occasions when I have been rootkited.

The Journal - a proposed syslog replacement

cas — Thu, 24 Nov 2011 08:06:35 +0000

because it increases the design and manufacturing costs of the end product without increasing sales of the end product.

and the really annoying thing is that it would just be a short-term once-off design cost until the chipsets for various device types all had ECC support (all AMD CPU chipset motherboards have ECC support and have had for several years - the catch is that ECC RAM is much more expensive). And the economies of scale for producing just one kind of RAM instead of "server RAM" and "desktop RAM", would quickly offset even those costs within a HW design generation

which is, of course, the reason why it hasn't happened - artificial market segmentation is extremely profitable. You can only charge more for "server-class" hardware if they have a few things which don't exist or are uncommon on "consumer" motherboards - e.g. ECC being uncommon outside of AMD chipsets, consumer motherboards having SATA rather than SAS (which should just replace SATA entirely), and consumer drives being SATA interface rather than SAS.

The Journal - a proposed syslog replacement

dlang — Wed, 23 Nov 2011 23:38:06 +0000

even multi-terabyte disks are expensive if you need a lot of them.

I store my logs at 10:1 compression (or better) and I still have 10's of TB of logs to deal with.

The Journal - a proposed syslog replacement

dlang — Wed, 23 Nov 2011 23:36:26 +0000

splunk is great for searching logs (I have a large cluster of machines for doing exactly this), but in terms of gathering and transporting logs, it's far from the best.

take a look at syslog-ng and rsyslog and the options they have to gather data from log files written by other apps.

The Journal - a proposed syslog replacement

dlang — Wed, 23 Nov 2011 23:34:51 +0000

even with today's disk sizes, nobody runs a fully logging filesystem. The inability to overwrite data will fill any disk up very quickly.

a 'logging filesystem' will give you a few older versions, depending on settings, but hardly every older version.

The Journal - a proposed syslog replacement

dlang — Wed, 23 Nov 2011 23:31:38 +0000

that program name and pid is data provided as part of the log written by the application, so the application can lie about both.

However,fixing this doesn't require making changes that are nearly this drastic.

systemd is already planning to create a new container for each application (with cgroups, etc), have it create a new filesystem namespace as well with a different /dev/log and the existing modern syslog daemons (rsyslog and syslog-ng) can record which container the log came from.

The Journal - a proposed syslog replacement

cas — Wed, 23 Nov 2011 22:42:58 +0000

Here's an idea that I had a while ago: make a USB dongle that appears to be a USB-to-serial converter. Data that is sent to it is recorded in its flash; you configure your system to send log messages to it like a serial console.

This bit of the idea is good

You could make this relatively secure by not allowing a transition from logging mode to read mode without re-plugging.

but this bit isn't. It would make more sense and be far more usable if the USB dongle presented two devices.

The first device being a (perhaps serial) output device for writing log entries with maybe a control option for rotating log files by YYYYMMDD or whatever. each line sent to the device should have a "filename" (or syslog facility, or some other identifier) as the first word/field, with the remainder of the line being the log entry

The second a *read-only* USB storage device for reading the logs whenever you like.

so, the one device would provide write-once/append-only logging, and random read access to those logs

such a device could be made dirt cheap, too. it's just a USB flash disk with a slightly more capable processor & USB interface

The Journal - a proposed syslog replacement

cas — Wed, 23 Nov 2011 22:12:37 +0000

But then, once you have a standardised text format, why not save some space and processing time with an efficient binary serialization of XML/JSON/YAML/what-have-you?

space is irrelevant these days. multi-terabyte disks are cheap, readily available consumer products
in my experience, XML etc *greatly* complicates most jobs, increasing processing time, difficulty of programming, difficulty of understanding WTF is going on. it turns what should be a quick and simple one liner to extract information into a multi-hour programming effort reading API docs, parsing the data in whatever obscured format it's in (and possibly parsing other things like the DTD).
it's completely missing the point of XML, JSON, YAML etc - they're data *transfer* protocols, not data *storage* methods. their purpose is to unambiguosly transfer data from one system to another, not to store data in yet another obscure special purpose file format
it violates the KISS principle. but, then, everything Lennart is involved in does that.

The Journal - a proposed syslog replacement

cas — Wed, 23 Nov 2011 22:03:22 +0000

what is wrong with binary formats is that I have to use a *different* tool for every different binary format. i'll never use any of them often enough to truly master them and, worse, anything i learn about their usage is trapped within that single usage context.

with plain text formats I can use the *same* collection of tools for everything, and every new thing i learn about sh or sed or grep or awk or perl or whatever is automatically useful in hundreds of other contexts, not just the context in which i originally learnt it.

binary formats suck.

The Journal - a proposed syslog replacement

jd — Wed, 23 Nov 2011 21:50:47 +0000

If the syslog files are in a fully logging filesystem, every version is retained, allowing you to recover the missing data without needing any specialist anythings.

The Journal - a proposed syslog replacement

ziggyfish — Wed, 23 Nov 2011 21:40:34 +0000

Not by default, however it is still possible to add this information into the log.

The Journal - a proposed syslog replacement

jeremiah — Wed, 23 Nov 2011 21:14:07 +0000

> I honestly haven't seen any systems actually configured for remote syslog in my time in the industry.

um... We do this on all of our servers. I love being able to tail one single file and get everything going on. Now, we don't do this outside of our local network. But a gig network seems to be able to handle the bandwidth just fine. I'd love a nice tool that would allow us to securely archive log files. And to me it seems that Journal is trying to do both, Archive, and transport. I don't think these two processes should effect each other. We also find that Splunk works pretty for the archiving and searching. As well as getting a hold of log files that don't output to syslog ie. request.log.

Not very thought through proposal

job — Wed, 23 Nov 2011 12:07:34 +0000

Each log entry authenticates the previous ones? So in order to remove a few lines from the log, you have to rewrite the log since that point in time?

Somehow I doubt this will pose a problem to an attacker who is so stealthy he/she manipulates logs. Most attackers just wipe them. That's why remote logging was invented.

After all, you will need a toolset to handle these logs for reading, searching and writing. I'm sure there will be a tool in this toolset which rewrites logs (just like there is for git).

The part where you store the root seed which authenticates the whole logs is also quite opaque. I guess you have to store a new seed every time you rotate logs, otherwise you'll be completely unable to authenticate anything when (parts of) an older log goes missing. So you'd have to have a remote logging protocol which logs these seeds continously, at which point you could just let it log everything and be done with it.

I'm all for enforcing stricter inter-application log formats to make them easier to parse, but this is just solving made up problems instead of the real ones just because it's easier.

The Journal - a proposed syslog replacement

nye — Wed, 23 Nov 2011 11:59:14 +0000

>You made a threat against someones life. I fail to see how you think that is acceptable. I think a police report should be filed.

There are actually laws against willfully wasting police time.

The Journal - a proposed syslog replacement

nye — Wed, 23 Nov 2011 11:28:31 +0000

>People who hate change and want to keep things like they were several decades ago can always use the BSDs...

Almost everyone dislikes user-visible change, unless there's a very slow migration path one step at a time (see KDE4, Vista, Gnome 3, etc.). Yes, this community has a larger proportion of heatseekers than the world at large, but even the FOSS world tends not to like huge revolutionary changes in one go.

More transparent changes - by which I mean things that don't change the way the user interacts with the machine on a daily basis - tend to be more welcome as there's less of a downside, so if it improves some functionality in some obvious way then it's an easier sell. For example, systemd has been far more positively received than PulseAudio since it solves real known problems without much of an effect on the end user.

Unfortunately, Lennart has a fairly bad track record. His projects tend to involve a grandiose scheme to replace some way of doing things entirely, which he then gets bored of once they reach the 90% stage.

Presumably some up-and-coming new star will come along again in a few years and decide to rewrite sound systems or init systems or syslog (or display servers, or desktop environments, or...), get them 90% done, and then get bored of them, and the cycle will begin anew.

Unfortunately, the options are either to stick with systems that are permanently 90% done, or be dismissed as a greybearded old has-been who 'hates change'.

The Journal - a proposed syslog replacement

nye — Wed, 23 Nov 2011 11:14:58 +0000

>It seems now that once per 3 months there is an announcement "Lennart replaces another core UNIX utility", so in the not too distant future we would have Linux boxes where I don't know a thing

ITYM 'Lennux boxes'.

The Journal - a proposed syslog replacement

mpr22 — Wed, 23 Nov 2011 10:09:51 +0000

A quick look at /var/log/syslog on my desktop Ubuntu box suggests that this turns out to only partly be the case. There are several programs (at a minimum: acpid, NetworkManager, modem-manager, dhcpd), which do not have a PID in their syslog messages.

The Journal - a proposed syslog replacement

anselm — Wed, 23 Nov 2011 09:06:27 +0000

There are various interesting things one could straightforwardly do with a well-designed binary log file format that are difficult to do with the current free-for-all text files. Efficient search according to (a combination of) different criteria is perhaps the most obvious candidate.

I agree that it is important to have a tool that will dump the binary file to a readable text file if required, but it seems to me that much of the »binary format sucks, over my dead body« we hear here is due to a Pavlov reflex.

The Journal - a proposed syslog replacement

chuckles — Wed, 23 Nov 2011 05:12:06 +0000

Please keep your politics out of this.

Not interested in hearing them.

thank you.

The Journal - a proposed syslog replacement

ziggyfish — Wed, 23 Nov 2011 05:01:15 +0000

One last thing, your already told who logged the message. as the first part of the message that gets logged is the process that logged it and the processes ID.

The Journal - a proposed syslog replacement

ziggyfish — Wed, 23 Nov 2011 04:54:55 +0000

One of the big problems with implementing a binary base log file system is as mentioned already, only one tool can read it correctly. And you loose the security if someone finds out how the data is stored. It's the exact same thing "Problem" as a text based log file. The other problem is a lot has to change in order for programs to work with the new system, i'e dmesg needs to be changed, boot loaders (including GRUB) have to add tools to allow people to read logs in case something goes wrong with the system and it can't boot.

Also I can not see how you can use the same methods as before to clean the logs, and to backup these logs. For example what happened if you wanted to view that log file in Windows, how would you do this. From a tool point of view you need the correct security tokens (which can be easily bypassed), etc.

You also have to remember routers use the syslogd protocol to send messages to a unix systems in case of DDOS, DOS and port scans etc. How will this be handled?

I don't like the move, it defeats the whole point of UNIX. Every thing in UNIX is a text file. Anyone can add to it and anyone can remove lines from it. It is up to the kernel and the logging program to control who can do what. The point about the syslog files being text only is mute as normally only root can write to the file and a binary file has the same "problem".

Why not use XML or something like that so that tools can still read and parse it, the log files can still be read even if the system can not boot. and provide a tool that can control access to the log file.

Lets not follow some stupid Windows way of doing it when we have a tried and test way we have used for years.

On a side note, should we even be looking at the security of the log file when only root can change it and if the hacker has root access. they can do a lot more dangerous things then change a log file.

The Journal - a proposed syslog replacement

Baylink — Wed, 23 Nov 2011 01:54:13 +0000

Yes, we have enough political and religious wars in computing about topics having nothing to do with politics and religion; I don't see that we need to discuss the actual items, either.

The Journal - a proposed syslog replacement

Baylink — Wed, 23 Nov 2011 01:52:14 +0000

> You can enable services by symlinking files /lib into /etc, not by the chkconfig API.

You *do* know what chkconfig (and insserv under it) actually does, right?

The Journal - a proposed syslog replacement

raven667 — Tue, 22 Nov 2011 20:27:10 +0000

Wow, if only all of this energy were spent solving the real problem (how the attacker gets in) rather than the tangential problem (how do we know he got there), perhaps all of this would be rendered... moot?

If only the world were that simple. You are never going to be able to prevent a motivated attacker from getting into your system. That's not really practically possible, new vulnerabilities will be discovered at the same rate or faster than you can plug them. What you can do, and what is a better use of resources, is a strong audit capability so that you can throw the bums out as soon as they get in, hopefully before they can cause serious damage. Not that you shouldn't patch as well but all the patching in the world will never be enough.

The world is a harsh place, you can't prevent all bad things from happening all the time but you can respond quickly with strength and resilience when they do.

The Journal - a proposed syslog replacement

JesseDyer — Tue, 22 Nov 2011 19:28:19 +0000

Wow, if only all of this energy were spent solving the real problem (how the attacker gets in) rather than the tangential problem (how do we know he got there), perhaps all of this would be rendered... moot?

I keep hearing the same pattern in this thread over and over again, which is something like this:

"Once we get the binary file, we'll use tool X to convert it into a text file so that we can actually make use of the thing.."

Am I the only one that appreciates the irony here?

The Journal - a proposed syslog replacement

mathstuf — Tue, 22 Nov 2011 16:38:29 +0000

I was pointing out that deleting files to disable things is more EIAF than running sed on a sysconfig file.

Out of curiosity, what makes it djb-ish?