LWN.net Logo

log compression

log compression

Posted Nov 20, 2011 19:47 UTC (Sun) by quotemstr (subscriber, #45331)
In reply to: log compression by slashdot
Parent article: The Journal - a proposed syslog replacement

So build a separate auxiliary file that sits *alongside* conventional log files. This auxiliary file can contain all the journal metadata - PIDs, precise timestamps, message GUIDs - and index some of them. The conventional syslog file would happily exist alongside the auxiliary file, unless disable by an administrator.

Programs that want to log to the journal could use a library that look like this:

struct journal_log_attribute
{
    enum journal_attribute_type type;
    union {
        journal_attribute_guid guid;
        journal_attribute_keyval keyval;
        journal_attribute_module module;
        /* etc */
    };
};

void
journal_vsyslog(
    int priority, 
    const char* msg, 
    va_list args, 
    struct journal_log_attribute* attributes[] /* NULL-terminated */)
{
    if (journald_is_active()) {
        journal_internal_vsyslog(priority, msg, args, attributes);
    }

    vsyslog(priority, msg, args); /* Ignore attributes */
}


(Log in to post comments)

Have you read the document?

Posted Nov 20, 2011 19:58 UTC (Sun) by khim (subscriber, #9252) [Link]

So build a separate auxiliary file that sits *alongside* conventional log files.

Have you read the design doc? This is exactly what journald does.

And if your idea is not to keep separate log with indexing and all additional goodies but to try to attach it to the existing textual file then this is stupudity beyond comprehension: referring to this scheme with "duct tape", "bailing wire", or "chewing gum" does a disservice to all three of those fine building materials.

Have you read the document?

Posted Nov 20, 2011 20:02 UTC (Sun) by quotemstr (subscriber, #45331) [Link]

> Have you read the design doc? This is exactly what journald does.

Except that, AIUI, the goal is to eventually have journal-only logging for some facilities. I don't want that to ever come to pass.

> attach it to the existing textual file

Why not? The index could point into a particular offset in a textual log, or just duplicate the contents of the textual log. The point is to ensure that all logs can be queried with plain-text tools and to _optionally_ provide richer information for these logs. The nightmare scenario is for some messages to appear only in the journal and for other messages to appear only in syslog files.

> stupudity beyond comprehension

Can we try to maintain *some* level of decorum here?

Have you read the document?

Posted Nov 20, 2011 20:13 UTC (Sun) by khim (subscriber, #9252) [Link]

> attach it to the existing textual file

Why not?

Because these files are already processed by quite a few different programs stitched together in non-obvious ways. To hope that you can keep all that synchronized... I'll wish you luck.

The nightmare scenario is for some messages to appear only in the journal and for other messages to appear only in syslog files.

If I understand correctly all messages pass the journald but plain old syslogd messages go to syslogd too. This means journald keeps everything no matter what.

Copyright © 2013, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds