Author responds [LWN.net]

Author responds

Posted Mar 26, 2009 0:08 UTC (Thu) by dave_malcolm (subscriber, #15013)
Parent article: Malcolm: SQL for the command line: "show"

Fame at last!

Follow-up posting here: http://dmalcolm.livejournal.com/2009/03/24/ with info on git repo etc.

Thanks for the feedback. As some have noted, this isn't intended just for log files. Note that in the original post it already had "rpm" and "proc" backends, for querying the local rpm database and /proc respectively.

It can now parse many of the files in /etc, using the Augeas library (http://augeas.net)

I also just added tcpdump support, so you can look at e.g. a wireshark dump and run something like this:
$ show "count(*)", "total(length)", src_mac, dst_mac from test.pcap group by src_mac, dst_mac
(though this is merely a messy proof-of-concept hack at this stage)

Ideas for other backends most welcome.

(Log in to post comments)

Author responds

Posted Apr 1, 2009 18:53 UTC (Wed) by ortalo (subscriber, #4654) [Link]

idea/need: ntsyslog backend (for parsing Windows event logs archived via NTsyslog to a Unix machine).

But the actual reason for my comment was another suggestion. Have you considered implementing the same kind of backends inside a full fledged database? It seems to me at least PostgreSQL should offer enough extensibility to allow this. It could free you from dealing with the intricacies/limitations of an "SQL-like" parser and may open the door to more complex treatments (dunno if writing would be feasible).
Or maybe you would find this too overweight for your intended usage? (I routinely have to consider >30Go of compressed log files so, even a full-fledged database engine does not seem overkill sometimes.)
Gonna look at your tool anyway. Thanks for the contribution.

Author responds

Posted Apr 1, 2009 22:38 UTC (Wed) by nix (subscriber, #2304) [Link]

What I've wondered about doing is hacking syslog() in libc and the syslog
protocol to pass the format string and arguments separately (as well as as
a formatted whole), so that syslog-ng can use its existing facilities to
dump the lot in a database. Then we can *really* do log analysis, with
variable and fixed parts spliced out. (The problem is the break of the
syslog protocol, though. I considered analyzing log messages to attempt to
retrospectively determine which parts are format string and which are
arguments, but that rapidly gets into a pattern-matching tarpit.)