idea/need: ntsyslog backend (for parsing Windows event logs archived via NTsyslog to a Unix machine).
But the actual reason for my comment was another suggestion. Have you considered implementing the same kind of backends inside a full fledged database? It seems to me at least PostgreSQL should offer enough extensibility to allow this. It could free you from dealing with the intricacies/limitations of an "SQL-like" parser and may open the door to more complex treatments (dunno if writing would be feasible).
Or maybe you would find this too overweight for your intended usage? (I routinely have to consider >30Go of compressed log files so, even a full-fledged database engine does not seem overkill sometimes.)
Gonna look at your tool anyway. Thanks for the contribution.
Posted Apr 1, 2009 22:38 UTC (Wed) by nix (subscriber, #2304)
[Link]
What I've wondered about doing is hacking syslog() in libc and the syslog
protocol to pass the format string and arguments separately (as well as as
a formatted whole), so that syslog-ng can use its existing facilities to
dump the lot in a database. Then we can *really* do log analysis, with
variable and fixed parts spliced out. (The problem is the break of the
syslog protocol, though. I considered analyzing log messages to attempt to
retrospectively determine which parts are format string and which are
arguments, but that rapidly gets into a pattern-matching tarpit.)