log compression

That is a useful option to have for log messages, but it's also available today. you can store your log messages in a database (of many different kinds, including nosql variations) and have the data indexed umpteen ways.

At the risk of being dismissed as an old fogy, the power of unix is based on the tradition of having many simple tools work together rather than having one tool that tries to do everything for everyone.

Nowhere is this more the case than in logging.

How important the logs are to you will vary drastically (do you want the application to stall if the log can't be written, do you want to spool to disk and run the risk of filling your disk, or do you want to throw away the log message)

how you store the logs will vary drastically, and in many cases you may want to store them in multiple ways.

how you examine the logs for 'interesting' things will vary.

At my office we have all of the following in place

recording to flat files combining all the logs togeather

recording to flat files of specific types of log messages

recording to a nosql database cluster across a large farm of machines with everything indexed

opensouce tools to watch for 'interesting' events and notify us when they happen

custom tools to watch for 'interesting' events and notify us when they happen

commercial closed source tools to watch for 'interesting' events and notify us when they happen

with existing syslog, all of these things can work togeather and I can add other log processing as well