Interesting idea, but...

Posted Nov 20, 2011 19:27 UTC (Sun) by tshow (subscriber, #6411)
Parent article: That newfangled Journal thing

From what I've seen of the journal proposal, I have only two real problems with it, and one of them isn't really a reasonable problem. The other problem is a deal breaker, though; it's a fundamental failure.

The lesser (and perhaps unreasonable) problem I have with the journal scheme is that I'm not convinced the security is going to work in most cases. Most people are going to keep the signing keys in a stupid, obvious place (like /root, or ~), so script-kiddying a complete regeneration of the log hashes isn't going to be hard. The people who take proper precautions will be fine, but that's true now; if you run syslog out a network port or a serial line (or to a line printer...) it's nearly impossible to tamper with it after the fact.

On the other hand, extra security is extra security. Maybe sometimes it will make things just that little bit harder.

The major problem I have is with the undocumented binary format. I have no problem with "binary format", it's "undocumented" that isn't doing it for me. There should be a well documented "dump log to raw text" tool, and a reasonable commitment to format stability. I should be able to read the dump utility source, break out the programming language of my choice, and write my own log data mining tools without having to go through the magic log translator.

There is nothing useful to be gained by obscuring the format; it's the worst form of improper incentives. Attackers have lots of incentive to figure the format out in detail, and can take the time to do so. Defenders just want the damned thing to work, and don't want to have to waste time on understanding the fiddly details until they've been compromised and suddenly have no choice.

Furthermore, unless it's a very simple format with no internal referencing, there's always the chance that an attacker can find some way to make the log file skip parts of itself or lie. Or, for that matter, buffer overflow the log reading tool and do yet more damage. If you could get a buffer overflow into the log data that invoked a "they're on to us, clean everything up" script...

Fundamentally, though, this misses the worst part of the design. Here's the deal breaker: We now have a single point of failure; if the attacker can compromise the log reader program, they have you. Everything goes through the log reader program. The attacker doesn't even have to tamper with the logs any more, they just have to replace the log reader program with something that lies.

With text-based logging, the attacker would have to replace nearly every binary in the system to prevent you from reading the real log (though they could probably guess and just hit less, vi and tail...), so it's easier to rewrite the log. With binary logging, your selection of available tools for reading the log becomes much smaller, and the log reading tools become a potential target rather than the log itself. When there's only one tool (or a small suite of standard tools), the tools themselves become an extremely attractive target.

If I were attacking a system with this kind of logging, what I would be trying to do is (1) replace the log reader with something that lies, and (2) try to get an overflow attack into the actual log data so that if someone tries uncorrupted tools I can corrupt them from within.

If the format were documented and I could write my own tools, the attacker can no longer assume that they can compromise anything that could hit the log, and are back to having to tamper directly with the log.

So, to me, hash-chain log messages? Sure, sounds good, should be worth 10% more security. Data mining messages? Also good. Binary format? No problem, as long as it's simple, stable, well documented, heavily audited for potential malicious data attacks, and has a tool that will dump it. Undocumented log format? Biiiiig red flag.

That undocumented format takes this from a fairly good idea to a very bad one.

to post comments

Interesting idea, but...

Posted Nov 20, 2011 19:44 UTC (Sun) by vrfy (guest, #13362) [Link] (12 responses)

Stabilizing the format is just not on the table right now. We need to get the overall model sorted out first, and that will take its time. As soon as we reach a point where stuff works as expected, there is no problem to make that format versioned, formally defined, and provide backwards read compatibility.

For now, the journal files are just an indexed ring buffer that follow store/forward logic. Anybody who depends on any data/log archival or classic tools to run, should run the syslog daemon at the same time, which gets exactly the same data and writes the same files as before.

The journal data is a zero maintenance buffer, that just takes the maximum configure disk space an no other ressources, it does not change any aspect of classic syslog. Local tools like systemd, or other agents will rely on the indexed data from the journal during runtime, and can not work with syslog files.

Journal files are not a log archive in the same sense as syslog files are for some time. That's just impossible to promise from where we are right now, hence the warning about it.

Interesting idea, but...

Posted Nov 20, 2011 19:59 UTC (Sun) by quotemstr (subscriber, #45331) [Link] (1 responses)

> Stabilizing the format is just not on the table right now.

Having a fluid format during development is acceptable. Shipping a critical component that writes with an unstable format is not.

> the journal files are just an indexed ring buffer that follow store/forward logic

Have you guys looked at ETW, the Windows event logging facility? The designers encountered and solved many of the issues that journal will run into. I still prefer plain-text logs to binary ring-buffer abominations, but if you're going to go down the latter route, you might as well learn from the past.

One convenient feature of ETW is a configurable and flexible approach to log management. ETW lets you decide on a per-log basis whether you want ring or serial logging, whether you want new log files to be created as old ones fill up, how often you want logs flushed to disk, and so on. ETW also has a "real-time consumer" option that basically pipes log messages to a process instead of writing them to disk, allowing programs to perform actions when certain system events happen. (You can even start a service when a certain log message appears.) The kernel also uses ETW to log its messages, allowing you to configure all the logging in one place.

Now, the details of ETW are disgusting: there are too many overlapping log options (you can do log rotation two different ways), too many log-message formats, too much metadata for each message (do we really need separate "keywords" and "categories" and such?), and too baroque an API. It's too damned hard to get simple plain-text output from the Windows event-logging system.

But beneath all the grime, there are some interesting ideas.

Interesting idea, but...

Posted Nov 23, 2011 14:41 UTC (Wed) by skorgu (subscriber, #39558) [Link]

Having recently written some C# code I took one look at the windows event log API, uttered some curse words and used log4net to write to a normal file instead.

The idea of highly structured output appeals to me but it seems more like tracing than logging. I'd rather have a kickass perf and a kickass syslog than try to combine the two.

Interesting idea, but...

Posted Nov 20, 2011 20:03 UTC (Sun) by tshow (subscriber, #6411) [Link] (1 responses)

Hmm. What's been put forth so far has strongly implied a replacement for syslog, but you're describing it as a supplement.

At any rate, if the plan is to have a properly documented format for the log data when the project hits 1.0, I withdraw my objection; my concern was that it appeared that undocumented was the overall plan, not simply a consequence of the early development phase.

Interesting idea, but...

Posted Nov 20, 2011 20:19 UTC (Sun) by vrfy (guest, #13362) [Link]

The format might be stabilized when it's reasonable stable, there is no version number, or any specific plan to tell now. We will get there over time. Until that, nothing should expect to be able to read/write the raw files directly. They need to get dumped with journalctl, or read with the library API, that's all what we can promise now.

The classic syslog model is not touched at all. If anything relies on any of the syslog features, the format, the remote logging, the files, it should run syslog like it always did.

This is something that runs on the local machine, and serves as the base for tools that need to make decisions or provide the 'history' of services. If syslog is the model to look at, journald is just a proxy.

The journald design is network-aware, but in no way network-transparent. All that can be done pretty efficiently by additional tools, but these tools will probably not be part of the core installation.

Interesting idea, but...

Posted Nov 20, 2011 22:19 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

Have you considered using reverse domain names for message ids?
UUID are just too unwieldy. Imagine that I'm a developer writing:

class SomeClass
{
    void someMethod()
    {
         while(true)
         FILE *fl = fopen(...);
         if (!fl)
             log(DEBUG, "File with data is not found, trying to create it");
    ...

At that moment you realize that you need a UUID for this message. Ok, easy enough:

class SomeClass
{
    void someMethod()
    {
         while(true)
         FILE *fl = fopen(...);
         if (!fl)
             log(DEBUG, "d92bc8ba-7a98-4e49-8384-8ee013e2f773", "File with data is not found, trying to create it");
    ...

But damn! Now my string wraps around 80 character limit! And if I'm unlucky it can wrap around inside the UUID itself, so I need to reformat my code.

Alternatively, I need to create a file with #defines of UUIDs and write things like this: "log(DEBUG, FILE_DATA_NOT_FOUND_UUID, "File with data is not found, trying to create it");" which leads to duplications. So most lazy programmers (such as myself) would just define a couple of UUIDs and use them with free-form text fields for verbose messages.

So please, consider making a good API for developers to be your first priority. And using UUIDs in API is definitely doubleplus ungood.

Actually I think it's good idea to have two types of logs

Posted Nov 20, 2011 23:43 UTC (Sun) by khim (subscriber, #9252) [Link] (6 responses)

I think Google does it right. They have two separate types of logs:
1) Binary structured logs which are moved to centralized storage, kept for a long time and are used for a lot of things.
2) Debug free-text logs which are only kept around till server is restarted.

Sometimes messages generated in logs of second type are deemed valuable enough and are "promoted": at this point unique message type should be added, message should be documented, etc. Adding unique UUID is trivial in comparison :-)

I doubt it makes much sense to force structure on all log messages which program can ever generate. If log record is supposed to be analyzed exclusively by humans then it's Ok to use free-form text for it. If it can be used by other software then it's time to organize and document it.

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 1:22 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

Well, if it can be made easy to use structured logging everywhere, then why not do it?

I don't really see how reverse domain names are worse, except of handwavy 'it's faster to use UUIDs'.

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 1:31 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

what is making it hard to use structured logging with syslog?

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 1:49 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Well, the lack of structure, for one thing. There are even tools (like http://www.splunk.com/ ) that exist to parse at least some of the logs.

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 10:31 UTC (Mon) by dlang (guest, #313) [Link]

I am very familiar with splunk, it doesn't parse the logs, it just indexes every word in the logs.

but back to my point. you can send structured logs via syslog today, there is even a standard to do so. People choose not to do this today, but that's not the fault of the syslog mechanism, that's the fault of the programmers.

As has been noted elsewhere, you can still make unstructured logs through this new mechanism, so the new mechanism doesn't give you structured logs any more than syslog does

Actually I think it's good idea to have two types of logs

Posted Nov 23, 2011 17:01 UTC (Wed) by sam-williams (guest, #57470) [Link]

Splunk is really more about using the data in the logs for enterprise systems management. Suggesting its purpose is solely for reformatting syslog data is inaccurate.

Structure would improve things a bit, but no self-respecting systems administrator would suggest they can't do their job without a bit of binary hand-holding. The binary fileformat could cause more problems then it cures. Care should be used in providing an ability to access this information with simply tools.

As I've already said...

Posted Nov 21, 2011 8:08 UTC (Mon) by khim (subscriber, #9252) [Link]

Any conclusion made from false premise will be true so you rant is certainly valid.

Well, if it can be made easy to use structured logging everywhere, then why not do it?

The only way known to humanity is to put fixed value in all fields except textual "details" field and then write free-form description there. A lot of peoples tried to make "structured logging everywhere" work, yet none succeeded. This means it's time to stop trying to push "structured logging everywhere" idea and think about different question "do we really need structured logging?" and the answer is "probably not". A lot of logs only make sense for someone who has detailed knowledge of the program. If you don't think long and hard about what your log is trying to convey and to whom then then no amount of structure applied will help. And not all log messages deserve such attention. At least this is what developers usually think - and if you system will not be accepted by developers then it may as well not exist.

Just a small note...

Posted Nov 20, 2011 19:51 UTC (Sun) by khim (subscriber, #9252) [Link] (1 responses)

With text-based logging, the attacker would have to replace nearly every binary in the system to prevent you from reading the real log (though they could probably guess and just hit less, vi and tail...), so it's easier to rewrite the log.

Actually attacker only needs to inject small kernel module and that's it. There are plenty of material for script kiddied on the internet.

If I were attacking a system with this kind of logging, what I would be trying to do is (1) replace the log reader with something that lies, and (2) try to get an overflow attack into the actual log data so that if someone tries uncorrupted tools I can corrupt them from within.

Well, overflow in log reader tool is valid concern at least. I guess feasibility will depend on size of these tools, but yes, this is pretty scary thing. Of course there are plenty of sandbox options for you to use (seccomp, nacl, etc), but still, this is valid concern.

Just a small note...

Posted Nov 20, 2011 19:57 UTC (Sun) by tshow (subscriber, #6411) [Link]

> Well, overflow in log reader tool is valid concern at least.

It can be (mostly) mitigated by ensuring that the log format is simple and the log reader is paranoid, but it's definitely worth paying attention to. The more self-referential the log format is, the more nightmarish it gets to ensure the reader is bulletproof.

Interesting idea, but...

Posted Nov 20, 2011 19:53 UTC (Sun) by butlerm (subscriber, #13312) [Link]

I agree, an undocumented format for critical data is insane. It is like bringing everything wrong with black-box-proprietary-binary-goo, fall-over-and-die-on-the-slightest-change software to Linux for no good reason.

Who cares if the file format changes? A good format ought to be extensible so that existing readers do not break. And if tools do process the format directly instead of relying on a library as they should, then naturally it is their responsibility to keep up with it.

There are tools that can and do process raw filesystem data, and we don't use that as an excuse for trying to hide the format in a black hole, or to never change it in any substantial respect either.

What if someone wants to write a different library that accesses the same data, possibly in a different language, after the fact, many years later, on a different operating system? Is the log data just supposed to be some sort of impenetrable format that is good for nothing without a compiled version of the original tools? Suggesting that people _reverse engineer_ it is laughably absurd. All the extra cost, repeated multiple times over, without the benefit of actual compatibility - all the bugs, weaknesses, and oversights of the original source base, if that.