The Journal - a proposed syslog replacement [LWN.net]

The Journal - a proposed syslog replacement

Posted Nov 18, 2011 19:39 UTC (Fri) by jmorris42 (guest, #2203)
In reply to: The Journal - a proposed syslog replacement by willnewton
Parent article: The Journal - a proposed syslog replacement

More importantly it is of a piece with the rest of this cast of fools past work. They despise UNIX, the philosophy behind it and the horse it rode in on. Each project tends to be designed to attack a core UNIX idea. This is a direct attack on Everything is a File. They prefer the Windows philosophy of Everything is an API.

These incidents point to a common problem in Linux land. We have allowed immigrants commit access to key parts of our cultural heritage before we assimilated them to our ways. Unless we find a way to reverse this trend we are soon going to find ourselves strangers in our own land, doing things the Windows Way because they fled Windows and remade Linux in it's image due to their superior numbers.

Not trying to get political, but this notion does have a real world analogy. Look at population and voting patterns in the Western US. People are fleeing the failed Blue coastal states like CA yet bringing the same failed Blue state ideas with them, turning the inner band of states Blue as the migrate in and repeating the cycle as they flee in ever increasing numbers. Bringing it back, these Windows refugees are doing the same thing. Windows doesn't suck because Microsoft is incompetent (they have really bright folks) the ideas that underlie it suck. They are now bringing the suck into Linux because they won't face the fact the ideas THEY hold suck. That THEY are the problem.

Credit where it is due, systemd does actually work and is documented, unlike, example at random..., pulseaudio. But this syslog replacement is a defective solution in search of a problem that doesn't even exist. Syslog supports logging to the network, that is the only truly secure solution to any security problems.

That Fedora is picking this turkey up just confirms that as a current Fedora user that it is time to GO.

to post comments

The Journal - a proposed syslog replacement

Posted Nov 18, 2011 20:14 UTC (Fri) by raven667 (guest, #5198) [Link] (1 responses)

We have allowed immigrants commit access to key parts of our cultural heritage...[snip]

Really? We can't discuss ideas based on their technical merit because that would be offensive to our "UNIX cultural heritage"? Really?! That's the most asinine thing I have read all day.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 3:52 UTC (Sat) by wmf (guest, #33791) [Link]

I wouldn't use the word "heritage" because that sounds like a fallacious appeal to tradition, but Unix is definitely a culture and it can be useful to think in those terms. (Check out http://www.faqs.org/docs/artu/philosophychapter.html if you haven't already.)

There are now several projects that are using Linux but aren't really part of Unix culture (OLPC, Android, Ubuntu, to a lesser extent Fedora with all its Lennartisms). I think it's arrogant to assume that Unix has already gotten everything right so I support all this exploration, but perhaps it deserves a warning label like "Fedora's not Linux as you know it; Slackware is that way".

The Journal - a proposed syslog replacement

Posted Nov 18, 2011 20:17 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (4 responses)

> They prefer the Windows philosophy of Everything is an API.

That's odd. I found systemd to be more EIAF than SysV init. To change the default TTYs from 1-6 to just 1 and 2 with SysV init, I had to sed a sysconfig file. For F15, I deleted the tty3-6 files from systemd's /etc directory. Of course, now it defaults to just 1 and auto spawns, which I like much better. You can enable services by symlinking files /lib into /etc, not by the chkconfig API.

> Credit where it is due, systemd does actually work and is documented, unlike, example at random..., pulseaudio.

Pulseaudio works for me (though the first audio-using app may not work due to a race condition, it's my fault for wanting to do it closer to on-demand than running things I don't need...should probably file a bug). True, the documentation could be better, but what project doesn't have that problem?

I am a little disappointed that it's aiming for F17 instead of F18, but I'll see how it works out on my Rawhide box at home first. Hopefully the binary format is dropped first...

The Journal - a proposed syslog replacement

Posted Nov 22, 2011 8:37 UTC (Tue) by Seegras (guest, #20463) [Link] (2 responses)

> For F15, I deleted the tty3-6 files from systemd's /etc directory.

That sounds more like D.J.Bernsteinish...

And...

Posted Nov 22, 2011 12:58 UTC (Tue) by khim (subscriber, #9252) [Link]

You said it like it's a bad thing.

I actually respect D.J.Bernsteinish very much. I refuse to use his creations not because they are bad, but because he insists that he, alone, knows the truth. He's right in about 95% or may be even 99% cases, but because sometimes he is wrong and you he does not tolerate deviations at all... well, it's safer to not even try.

But I fail to see why these same approaches used by people who don't share DJB "you have no right to touch my baby" should be bad...

The Journal - a proposed syslog replacement

Posted Nov 22, 2011 16:38 UTC (Tue) by mathstuf (subscriber, #69389) [Link]

I was pointing out that deleting files to disable things is more EIAF than running sed on a sysconfig file.

Out of curiosity, what makes it djb-ish?

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 1:52 UTC (Wed) by Baylink (guest, #755) [Link]

> You can enable services by symlinking files /lib into /etc, not by the chkconfig API.

You *do* know what chkconfig (and insserv under it) actually does, right?

The Journal - a proposed syslog replacement

Posted Nov 18, 2011 20:59 UTC (Fri) by AndreE (guest, #60148) [Link] (1 responses)

Immigrants who try to improve things are far more valuable than self-proclaimed cultural guardians who rely on slinging mud and are far removed from rational discourse.

The Journal - a proposed syslog replacement

Posted Nov 21, 2011 0:09 UTC (Mon) by Tara_Li (guest, #26706) [Link]

Here's the thing - it's not really an improvement.

It's like a friend of mine who had problem with her Kindle. She could not get it to connect to her wireless network. She couldn't, however, manage it - and the Kindle's only advice was to check the Amazon website. *ALL* of the Kindle's help files are stored on the Amazon servers - but if you need help because you can't access those servers - you are out of luck. Here lies the problem with much of the current path of "progress". Can't find the file you had stored in the cloud? Access the cloud help information - stored in the cloud! Can't get online? The help you need is stored - online. Can't get your computer to boot? The help you need - is on that CD you need the *BLEEEEEEEEEEEEEEEEEEEEPING* computer to access!

Exploring new paths is a good thing - but this isn't a new path. This is a path that's already been explored - and found wanting.

The Journal - a proposed syslog replacement

Posted Nov 18, 2011 21:41 UTC (Fri) by quotemstr (subscriber, #45331) [Link] (3 responses)

> People are fleeing the failed Blue coastal states like CA yet bringing the same failed Blue state ideas with them, turning the inner band of states Blue as the migrate in and repeating the cycle as they flee in ever increasing numbers.

Let's keep politics out of this discussion. Not everyone agrees with your assessment of our recent history, and in any case, the analogy adds more heat than light.

The Journal - a proposed syslog replacement

Posted Nov 18, 2011 22:22 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

> the analogy adds more heat than light

I like this analogy. Mind if I later use this quotemstr? ;)

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 3:43 UTC (Sat) by k8to (guest, #15413) [Link]

That's a classic, though well applied.

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 1:54 UTC (Wed) by Baylink (guest, #755) [Link]

Yes, we have enough political and religious wars in computing about topics having nothing to do with politics and religion; I don't see that we need to discuss the actual items, either.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 0:04 UTC (Sat) by cmccabe (guest, #60281) [Link] (5 responses)

I'm going to have to disagree. Security is a pretty major issue, and anything that improves system security on desktop and server Linux is probably a win.

It's a shame that we will lose the simplicity of the plain-text syslog format. But syslogs are usually compressed using gzip anyway. So essentially for me, all this means is that I use <magic-lennart-tool> instead of gzcat as the first part of my shell command.

The big issue that I see is that a lot of system administrators will treat this as magic security dust, and not realize that they need to periodically save those hashes to a remote (and secure!) system in order to get any security benefit.

I also hope Lennart and co. realize the absolute necessity of backwards compatibility for the on-disk format. It would really embitter a lot of system administrators if their old logs became unreadable after upgrading to the shiniest new version. But assuming this is managed well, I don't see any reason why this couldn't be a good idea.

P.S. Providing a FUSE filesystem that can read these files is a good idea!

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 16:52 UTC (Sat) by jmorris42 (guest, #2203) [Link] (4 responses)

> I'm going to have to disagree. Security is a pretty major issue, and
> anything that improves system security on desktop and server Linux is
> probably a win.

Except it isn't. It greatly complicates the system for little or no net gain in security. First rule: if the host might be compromised you must not trust it. They admit the existence of this rule and admit the only solution is to put some of the checkpoints on a different system. They fail to see the obvious, put them ALL on a different system. The people who designed syslog understood security better than Lennart does.

That is The UNIX Way. We learn what works and keep it.

"The access is granted by a shared library and a command line tool."

Haven't we seen this story enough times to see the pattern? The tool will be minimal, just for quick use by UNIX diehards to shut them up, while all 'serious' use will be expected to be through the API/library.

For a 'real programmer' it doesn't matter, but UNIX philosophy envisions a spectrum of skills, not a binary user/developer divide. Windows philosophy on the other hand is different. Remember that a key Microsoft design goal for the registry was to fit into their overarching project to end 'power users.' Professional developers write code, users click on widgets and nothing in the middle.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 23:07 UTC (Sat) by cmccabe (guest, #60281) [Link] (3 responses)

The problem with putting all your syslogs on a remote system is that it can eat quite a bit of bandwidth. I honestly haven't seen any systems actually configured for remote syslog in my time in the industry. In contrast, sending a single hash to a central single-purpose server every hour doesn't raise any bandwidth issues.

This whole project was motivated by the administrators' inability to trust the kernel.org syslogs after the breakin. It's always a good sign when a project solves a real problem that some smart people encountered.

> Haven't we seen this story enough times to see the pattern? The tool will
> be minimal, just for quick use by UNIX diehards to shut them up, while all
> 'serious' use will be expected to be through the API/library.

I agree that there is a lot of DBUS-itis and shared-library-itis out there, especially from the GNOME devs. (The GNOME devs have done a lot of good work as well, but I have to call it like I see it.) But I don't think this falls into that category.

As long as 'logger' continues to work, and there is some way to cat the syslog to a file, it will be as easy to use syslog on the command line as it ever was. I mean, syslogd has been a daemon and syslog(2) has been an API since the seventies; you can hardly complain that Lennart is adding yet more daemons and APIs to the system. If you add a FUSE interface that exposes a writable file named /var/log/syslog that has all the log entries, it's as scriptable as it ever was.

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 21:14 UTC (Wed) by jeremiah (subscriber, #1221) [Link] (2 responses)

> I honestly haven't seen any systems actually configured for remote syslog in my time in the industry.

um... We do this on all of our servers. I love being able to tail one single file and get everything going on. Now, we don't do this outside of our local network. But a gig network seems to be able to handle the bandwidth just fine. I'd love a nice tool that would allow us to securely archive log files. And to me it seems that Journal is trying to do both, Archive, and transport. I don't think these two processes should effect each other. We also find that Splunk works pretty for the archiving and searching. As well as getting a hold of log files that don't output to syslog ie. request.log.

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 23:36 UTC (Wed) by dlang (guest, #313) [Link] (1 responses)

splunk is great for searching logs (I have a large cluster of machines for doing exactly this), but in terms of gathering and transporting logs, it's far from the best.

take a look at syslog-ng and rsyslog and the options they have to gather data from log files written by other apps.

The Journal - a proposed syslog replacement

Posted Nov 25, 2011 15:02 UTC (Fri) by jeremiah (subscriber, #1221) [Link]

those are on my list. It's mostly an issue of upgrading servers at this point to version that support them.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 2:01 UTC (Sat) by HelloWorld (guest, #56129) [Link] (12 responses)

You know what? Nobody in the real world cares about the so-called "Unix philosophy". Rob Pike said it best: "Not only is UNIX dead, it's starting to smell really bad."
What people do care about is problems being solved, like the problems that Lennart documented in his proposal. And the thing that the self-proclaimed guardians of the "Unix philosophy" all have in common is that they don't solve problems but deny them, point to 20-year-old would-be solutions that are known to suck and blather about things such as "cultural heritage". That's why Lennart's projects keep being adopted by pretty much all major distros, despite "Unix philosophers" whining and bitching about it.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 2:16 UTC (Sat) by jubal (subscriber, #67202) [Link] (5 responses)

Mr World, excuse me, are you a sysadmin?

The quoted journald document promises:
breaking compatibility with existing standard and protocols, used not only by Linux systems, but by almost everything else,
coupling journald with systemd, thus making it non-portable, thus locking out non-Linux systems, all Linux systems not using systemd as an init replacement and most appliances,
using *undocumented* binary format for storing compressed and possibly encrypted and/or signed data.

Frankly, these three issues nullify all other described advantages.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 2:55 UTC (Sat) by HelloWorld (guest, #56129) [Link] (3 responses)

> breaking compatibility with existing standard and protocols, used not only by Linux systems, but by almost everything else,
Yeah, except that it doesn't. From said document: "Data can be generated from a variety of sources: [...], userspace messages generated with syslog(3)"

> coupling journald with systemd, thus making it non-portable, thus locking out non-Linux systems, all Linux systems not using systemd as an init replacement and most appliances,
If this is a problem for you, then use something else. Just don't expect Lennart to make his life harder by supporting non-Linux and non-systemd boxes.

> using *undocumented* binary format for storing compressed and possibly encrypted and/or signed data.
Why would one document a file format that isn't stable and may be changed in the future? Especially when you can just use the library to read it...

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 3:40 UTC (Sat) by dskoll (subscriber, #1630) [Link] (2 responses)

Why would one document a file format that isn't stable and may be changed in the future? Especially when you can just use the library to read it...

We're talking about logs here. They could contain important information needed a long time after it is generated, possibly even for legal reasons. If you are given a court order to produce two-year-old logs and can no longer read your old logs because the file format has changed in the meantime... you will not be very happy (and neither will the court.)

Indexing log files for performance is not a bad idea. Making them tamper-evident is a good idea too. But both of those can be done by adding to existing plain-text log file formats. You don't need to throw that out in favour of a binary format.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 20:59 UTC (Sat) by HelloWorld (guest, #56129) [Link] (1 responses)

> We're talking about logs here. They could contain important information needed a long time after it is generated, possibly even for legal reasons. If you are given a court order to produce two-year-old logs and can no longer read your old logs because the file format has changed in the meantime...
I trust Lennart to develop a solution to this, should this case actually arise. There are two obvious solutions. One is to implement a conversion tool after a format change that will be maintained indefinitely (a tool that simply reads a log file and outputs it again in a new format is unlikely to require a lot of maintenance). Another one is to simply keep the code for reading pre-format-change log files in the library, so that everything just works even with newer tools as long as they use the library.
Also, one should keep in mind that journald is a very young project. I'd guess they'll document the file format eventually when they're sure it does what they need it to do.

The Journal - a proposed syslog replacement

Posted Nov 21, 2011 14:40 UTC (Mon) by regala (guest, #15745) [Link]

> journald is a very young project. I'd guess they'll document the file format eventually when they're sure it does what they need it to do

given we're talking about Lennart and Kay, it will never do what they need it to do long enough to call it "stable format". ;)

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 7:43 UTC (Sat) by slashdot (guest, #22014) [Link]

Being open source makes these problems all easily fixable:
- Anyone can add support for whatever "standard and protocol" he wants
- Anyone can decouple it from systemd
- Anyone can read the source and document the format

But Red Hat ought to hire a PR spokesman for Poettering, to stop him from making these unnecessarily divisive statements, since he could just say that they "haven't yet had the time" to do those things, rather than outright saying "HAHA NO!" to reasonable points like those.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 2:38 UTC (Sat) by nybble41 (subscriber, #55106) [Link]

> What people do care about is problems being solved, like the problems that Lennart documented in his proposal.

We also care that these solutions don't reintroduce other problems which we've spent quite a bit of effort over the years resolving. That's what the "UNIX philosophy" is really about--learning from our past mistakes.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 2:58 UTC (Sat) by cmccabe (guest, #60281) [Link]

You are really taking that quote out of context.

Rob Pike created Plan9, which in tried to take UNIX to its logical conclusion. Plan9 tries to really make everything a file, including sockets. It uses the filesystem to control access to resources. And so on. The reason why UNIX "smell[ed] really bad" to Rob Pike is because Plan9 never caught on, because people didn't want to break compatibility. It really is a shame because Plan9 was the superior OS.

That being said, I am in favor of the proposal here-- at least in the context of server systems.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 12:25 UTC (Sat) by rilder (guest, #59804) [Link] (3 responses)

It is a philosophy, not an implementation. Read that again. It is like a specification. You can write a new tool tomorrow conforming to unix philosophy (and many have), and FYI Plan9 has adopted unix philosophy (actually its both way) and Rob Pike, who you are so fancifully quoting is a one of the main developers of Plan9. Go language from Google, is inspired by Plan9 semantics.

"That's why Lennart's projects keep being adopted by pretty much all major distros, despite "Unix philosophers" whining and bitching about it."

Yes, he is doing good work but don't jump the gun. Fedora (the home distro of systemd) is pretty shaky about fully adopting it. RHEL (where the money lies) is not even close. SUSE has delayed it. Ubuntu is going with upstart right ?

If you want to PR work, don't use LWN, use digg -- you will find plenty there who will buy your stuff.

The Journal - a proposed syslog replacement

Posted Nov 19, 2011 14:37 UTC (Sat) by vonbrand (guest, #4458) [Link]

I see no "shakiness" in Fedora's uptake of systemd; quite the contrary, the conversion of legacy sysvinit scipts is comming along nicely. That a very conservative distribution like RHEL hasn't yet adopted systemd is no surprise, AFAIU RHEL 6 was forked off around Fedora 14, way before systemd was solid enough. Debian prefers playing with non-Linux kernels, so systemd is out for them. Ubuntu is home to upstart, so it in little surprise it will take time to win them over.

Again: you are trying to hard...

Posted Nov 19, 2011 14:49 UTC (Sat) by khim (subscriber, #9252) [Link] (1 responses)

"That's why Lennart's projects keep being adopted by pretty much all major distros, despite "Unix philosophers" whining and bitching about it."

Yes, he is doing good work but don't jump the gun. Fedora (the home distro of systemd) is pretty shaky about fully adopting it. RHEL (where the money lies) is not even close. SUSE has delayed it. Ubuntu is going with upstart right ?

This is funny. Systemd is not the first large Lennart's project. More like third (not counting minor ones). First two (avahi and pulseaudio) were met by similar comments as systemd yet now it's hard to find distribution which have not adopted them.

And systemd adoption is in pretty good shape if you'll recall that it's pretty young project still.

Again: you are trying to hard...

Posted Nov 21, 2011 7:01 UTC (Mon) by yoe (guest, #25743) [Link]

Pulseaudio is crap. It hides the devices that ALSA exports for little purpose, and IME introduces more problems than it tries to solve.

Avahi is interesting in theory; but when I try to say "ping celtic.local" on my home network, there's about a 50-50 chance that it'll find what the right IP address is.

I'm not using systemd, and won't change to it, either. Enabling or disabling a sysv init scripts requires moving about a few symlinks. What more does one need?

The Journal - a proposed syslog replacement

Posted Nov 20, 2011 6:51 UTC (Sun) by skissane (subscriber, #38675) [Link] (10 responses)

I disagree completely. I think the UNIX emphasis on plain text, while it had its value in its original historical context of trying to get something working quickly, it holds us back to keep it like it was some kind of unquestionable religious dogma.

What is bad about binary formats? Nothing inherently wrong with them. Sure, if they are poorly designed, poorly documented, lack good tools, etc., then no doubt they can cause a lot of grief, but those are problems with poorly designed and poorly implemented binary formats, not with binary formats per se.

What I would like to see, is a simple, flexible, self-describing, binary format. ASN.1 or binary XML are both good places to start, although I think both suffer from useless extra complexity. Maybe something like Google protocol buffers would be a good choice?

If you really want text, you can easily write a tool to dump the binary out to text. Hey, you could have a format with two official serialisations, a text-based and a binary one.

But the problem I see with most Unix-style plain text formats, is every one is different. There is a lack of consistency/standardisation, especially when it comes to how to escape special characters, etc.

The Journal - a proposed syslog replacement

Posted Nov 20, 2011 7:11 UTC (Sun) by dlang (guest, #313) [Link] (5 responses)

What makes you think that binary formats would be any more standardized than the existing text formats?

There are ways to do self-describing text formats, but developers don't do it.

With text formats it's a lot easier to examine the file and reverse engineer the format than it is from a binary format.

Damaged/lost files are also a place where text files are easier to recover than binary files.

In theory none of this should ever be needed and binary files are just fine. But this is where the quote "in theory, theory and practice are the same, but in practice they are not" applies

The Journal - a proposed syslog replacement

Posted Nov 20, 2011 7:55 UTC (Sun) by skissane (subscriber, #38675) [Link] (2 responses)

I think, if you want to stick to text, it would be much better if tools output in some standardised text format, e.g. XML, JSON, YAML, etc.

But then, once you have a standardised text format, why not save some space and processing time with an efficient binary serialization of XML/JSON/YAML/what-have-you?

And then you can have a tool, e.g. bin2text, which reads the binary format on standard input and writes the text format on standard output, and vice versa. With such a tool, reverse-engineering/examination should be no harder than with a plain text format.

I think this would be better than both (1) the rather poorly-defined text formats used at present by many tools and (2) binary is more efficient than text.

The point you make about trying to recover from corrupted files being easier when they are in text is true, but how often do you have to deal with that? If there were provided some good quality libraries (say C with bindings to other common languages such as C++, Java, Perl, Python, etc.), the odds of a corrupt file due to programmer error should be low, outside of some mid-transaction failure scenario. And if we had transaction support in the library or the underlying filesystem, we could avoid that problem too.

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 22:12 UTC (Wed) by cas (guest, #52554) [Link] (1 responses)

But then, once you have a standardised text format, why not save some space and processing time with an efficient binary serialization of XML/JSON/YAML/what-have-you?

space is irrelevant these days. multi-terabyte disks are cheap, readily available consumer products
in my experience, XML etc *greatly* complicates most jobs, increasing processing time, difficulty of programming, difficulty of understanding WTF is going on. it turns what should be a quick and simple one liner to extract information into a multi-hour programming effort reading API docs, parsing the data in whatever obscured format it's in (and possibly parsing other things like the DTD).
it's completely missing the point of XML, JSON, YAML etc - they're data *transfer* protocols, not data *storage* methods. their purpose is to unambiguosly transfer data from one system to another, not to store data in yet another obscure special purpose file format
it violates the KISS principle. but, then, everything Lennart is involved in does that.

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 23:38 UTC (Wed) by dlang (guest, #313) [Link]

even multi-terabyte disks are expensive if you need a lot of them.

I store my logs at 10:1 compression (or better) and I still have 10's of TB of logs to deal with.

The Journal - a proposed syslog replacement

Posted Nov 20, 2011 8:12 UTC (Sun) by drag (guest, #31333) [Link] (1 responses)

Well they tried. It ended up being XML. :(

The Journal - a proposed syslog replacement

Posted Nov 20, 2011 19:27 UTC (Sun) by skissane (subscriber, #38675) [Link]

The problem with XML is:
1) a syntax originally designed for marking up documents got reused for
data, with the result that XML provides distinctions which are
unnecessary for data purposes (e.g. element vs. attribute distinction)
2) historical baggage, e.g. DTDs
Certainly you can define new syntaxes which avoid those two problems that
XML has. On the other hand, whatever its warts, XML is an industry standard,
and practical considerations often imply choosing the imperfect industry
standard over some technically superior but rarely used alternative.

But, JSON is quite common now, and addresses some of the issues above. (But
I think it has its own deficiencies too)

The Journal - a proposed syslog replacement

Posted Nov 21, 2011 14:50 UTC (Mon) by sorpigal (subscriber, #36106) [Link] (1 responses)

> I think the UNIX emphasis on plain text, while it had its value in its original historical context of trying to get something working quickly, it holds us back to keep it like it was some kind of unquestionable religious dogma.

It seems logical to say that a standard binary format is just as good as a standard text format, which is why this is carefully documented as one bullet point in the Unix philosophy: use text. The extra overhead was a lot worse back when this idea was developed, yet they stuck with it anyway. If you don't meditate on "Why" then you will invent a non-text system that's "as good or better" than text and suffer as a result. You can either accept received wisdom and "just do it," ignore this sage advice at your own peril or embrace the idea wholeheartedly.

It's not true...

Posted Nov 21, 2011 18:41 UTC (Mon) by khim (subscriber, #9252) [Link]

It seems logical to say that a standard binary format is just as good as a standard text format

This is not true. To pull useful data from corrupted text file you need a human being. To pull it from binary format with embedded CRC checks you only need to rigorpously use one very fast function. Sure, this only protects the data from accidental changes (zero-out pages in the middle, bitflips, that kind of things) but the funny thing that when people describe how they heroically recover data from corrupt disk or filesystem it's almost always from accidental corruption.

The extra overhead was a lot worse back when this idea was developed, yet they stuck with it anyway.

Actually it's much worse today. When you had hundreds of kilobytes or may be few megabytes of logs - human as "recovery system" works. When there are gigabytes, terabytes and petabytes of logs - it's hopeless.

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 22:03 UTC (Wed) by cas (guest, #52554) [Link] (1 responses)

what is wrong with binary formats is that I have to use a *different* tool for every different binary format. i'll never use any of them often enough to truly master them and, worse, anything i learn about their usage is trapped within that single usage context.

with plain text formats I can use the *same* collection of tools for everything, and every new thing i learn about sh or sed or grep or awk or perl or whatever is automatically useful in hundreds of other contexts, not just the context in which i originally learnt it.

binary formats suck.

The Journal - a proposed syslog replacement

Posted Nov 27, 2011 2:23 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> with plain text formats I can use the *same* collection of tools for everything, and every new thing i learn about sh or sed or grep or awk or perl or whatever is automatically useful in hundreds of other contexts, not just the context in which i originally learnt it.
I found that in most cases, it is easier to learn a new tool that is specialized for the job at hand than to try to get "standard unix" tools to do what you want, especially if you want to do it in a robust and maintainable way. For example, people keep asking again and again how to handle some XML format with things like sed or awk, which is just a bad idea given the existence of specialized tools like xmlstarlet.

The Journal - a proposed syslog replacement

Posted Nov 20, 2011 14:41 UTC (Sun) by jamesh (guest, #1159) [Link] (2 responses)

If you're talking about the traditional method of sending log messages as simple UDP packets, it might stop an attacker from altering historic logs on the system that generated the logs, but it has its own problems.

Log messages can get lost and since there is no authentication, log messages can be forged. And if the attacker manages to break into the log aggregation server, then you've got the same problems as before.

The Journal - a proposed syslog replacement

Posted Nov 20, 2011 20:48 UTC (Sun) by dlang (guest, #313) [Link] (1 responses)

as soon as you allow machines to send messages over the network it is going to be possible for messages to be forged. the receiving machine has no way of knowing what is happening inside the sending machine and if the data it is getting is correct or not.

This is not entirely true...

Posted Nov 20, 2011 20:52 UTC (Sun) by khim (subscriber, #9252) [Link]

Actually it's exactly the same as with local logging: of correct authentification scheme is used (i.e.: not syslog's UDP) then they can only be forged after takeover. The messages right before takeover are the most valuable. Sure, you must understand that some messages are are probably forged and some are not, but this is always the case when forensic analisys is done.

The Journal - a proposed syslog replacement

Posted Nov 23, 2011 5:12 UTC (Wed) by chuckles (guest, #41964) [Link]

Please keep your politics out of this.

Not interested in hearing them.

thank you.