| |||||||||||||
The Journal - a proposed syslog replacementThe Journal - a proposed syslog replacementPosted Nov 18, 2011 17:26 UTC (Fri) by JoeBuck (subscriber, #2330)In reply to: The Journal - a proposed syslog replacement by willnewton Parent article: The Journal - a proposed syslog replacement Why must the format be undocumented? It could probably be done in a simple enough way that a few lines of Perl or python would dump it in a human-readable form, since all you need is a clean record definition. Each record would have a timestamp, hash, and message.
(Log in to post comments)
The Journal - a proposed syslog replacement Posted Nov 18, 2011 17:51 UTC (Fri) by quotemstr (subscriber, #45331) [Link] From the FAQ:Will the journal file format be standardized? Where can I find an explanation of the on-disk data structures? At this point we have no intention to standardize the format and we take the liberty to alter it as we see fit. We might document the on-disk format eventually, but at this point we dont want any other software to read, write or manipulate our journal files directly. The access is granted by a shared library and a command line tool. (But then again, its Free Software, so you can always read the source code!)No, thanks. Text is universally forward- and backward-compatible. This monstrosity won't be.
The Journal - a proposed syslog replacement Posted Nov 18, 2011 18:17 UTC (Fri) by JoeBuck (subscriber, #2330) [Link] This seems ill-advised, because logs might need to be analyzed some point in the future. They are critical data, and you never want to have a problem reading your critical data.I don't care if they leave things loose during the initial design and evaluation, but you don't want to deploy this thing on a server you care about unless they tighten it up.
The Journal - a proposed syslog replacement Posted Nov 18, 2011 20:14 UTC (Fri) by niner (subscriber, #26151) [Link]
I also keep critical data in a PostgreSQL database which can only be read by a single program - PostgreSQL. It would probably be nice to be able to read it with a text editor, but then I'd probably lose all the benefits PostgreSQL gives me, so I can live with it. As I will with having to use some command like logread on my wireless router to get at the logs.
The Journal - a proposed syslog replacement Posted Nov 18, 2011 18:29 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]
Well, it makes sense to use undocumented format at first, so others won't treat it as ABI. But it's not going to be complex so it's definitely going to be replicated in other tools.
As for backwards compatibility, I'm personally going to write a FUSE file system which exposes journal records as regular log files. After all, I already have one that does that for Windows...
The Journal - a proposed syslog replacement Posted Nov 18, 2011 19:59 UTC (Fri) by drag (subscriber, #31333) [Link]
I don't mind piping out binary files to grep for things.
I do it for syslogs at work anyways. All the files are stored in gzip format and I have to use 'gunzip -c' to read them.
However, Log files are definitely a weak point for Linux. A huge pain in the ass. If it's a mostly-text file with something like null terminated fields then it would make things a lot easier and more efficient to parse.
The Journal - a proposed syslog replacement Posted Nov 18, 2011 20:18 UTC (Fri) by mathstuf (subscriber, #69389) [Link]
Ever heard of zgrep?
The Journal - a proposed syslog replacement Posted Nov 20, 2011 7:49 UTC (Sun) by drag (subscriber, #31333) [Link]
... sure. What about it?
The Journal - a proposed syslog replacement Posted Nov 21, 2011 23:56 UTC (Mon) by kmself (subscriber, #11565) [Link]
I believe the point is that you can access / search / manage / view compressed files with readily available shell tools.
zgrep will search them.
lesspipe + less will allow you to view them with less, as you would any other file (lesspipe will also render numerous other file formats as straight text, which is particularly useful).
Woe unto you if you're on a system without these tools, but that's another story. Writing your own shell wrappers (scripts/functions) is trivial.
The Journal - a proposed syslog replacement Posted Nov 22, 2011 0:55 UTC (Tue) by mathstuf (subscriber, #69389) [Link]
I was more pointing out that tools exist that do the gunzip -c internally. I almost never use gunzip except via the z* tools and tar xf. I always appreciate shotcuts for things like that (at least those commonly available on random machines, e.g. my vim bindings don't modify core behavior that is easy to get a bad habit with when sshing around, like nnoremap jj <Esc>).
The Journal - a proposed syslog replacement Posted Nov 20, 2011 2:54 UTC (Sun) by THe_ZiPMaN (subscriber, #27460) [Link]
Who did say you cannot use text format?
You could even use standard syslog with only one minor change; it's enough to add at the end of the line another piece of text with the hash computed concatenating the syslog line with the hash of the previous line.
line 1: Timestamp1 Message1 Hash(line1 + salt)
3 years ago, in Italy, a new discussed law required to guarantee the integrity of the logs, and at that time I started to write a patch for rsyslog to implement exactly the above schema. The law wasn't approved in that way so I abandoned the develop of such thing, but it's not too complicated to write it.
The Journal - a proposed syslog replacement Posted Nov 20, 2011 6:51 UTC (Sun) by dlang (✭ supporter ✭, #313) [Link]
one interesting issue with using a hash like this (both your proposal and the journal proposal) is that it will cause problems when logs get sent over the network.
the logs may end up arriving in a different order than you sent them for a number of reasons (UDP messages can pass each other on a network, you coudl send some logs to a relay box that fails before it re-sends them and the backup relay box sends newer logs on first, etc)
also if you are combining logs from many systems on one box, the you now have many different hash chains to track (and things get even more fun if you have more than one server sending with the same ID, something that happens in real life, even though it's a really bad idea)
The Journal - a proposed syslog replacement Posted Nov 20, 2011 14:04 UTC (Sun) by THe_ZiPMaN (subscriber, #27460) [Link]
No, that's not the case because you are mixing 2 different problems.
Syslog already allows for secure, autenticated and reliable network communications, particulary using the RELP protocol (rfc 3195), so ther's no need to add an overhead to that phase.
The hash is then added only when the message is saved on the disk, non before sending it over the network. It's scope is only to guarantee that the log has not been modified after being written, not to guarantee it's immutable during network paths.
The solution is not complete as a journal, but it's really really simple to add to existing daemons (rsyslog and syslog-ng), does not require any invasive operation on the syslog infrastructure, allows a complete backward compatibility and satisfy a big part of the problem addressed by journal in a simple.
The Journal - a proposed syslog dropout? Posted Dec 1, 2011 17:06 UTC (Thu) by gvy (guest, #11981) [Link] It's weird that neither msyslogd or the preceding ssyslogd are even mentioned -- it's much better to reinvent the wheel at least knowing of prior art, /and/ to criticize the process thereof either. :)Thanks Solar Designer, here's the link (in Russian).
|
|||||||||||||