That newfangled Journal thing

Posted Nov 21, 2011 17:51 UTC (Mon) by SEJeff (guest, #51588)
In reply to: That newfangled Journal thing by lothar
Parent article: That newfangled Journal thing

It seems like designing something around google's protobuffers[1] for ondisk and network format would be a sane compromise between the two. I can't possibly fathom why the developers of the Journal didn't do this. It is *really* good and I don't see how they couldn't simply layer an api ontop of protobuffers and call it a day. They also get the "feature" of a bunch of tools to read-write their format and bindings in all major languages for free.

[1] http://code.google.com/apis/protocolbuffers/

to post comments

That newfangled Journal thing

Posted Nov 21, 2011 18:23 UTC (Mon) by hanwen (subscriber, #4329) [Link] (4 responses)

I second the suggestion for protocol buffers. They were made for serializing structured data in a compact yet backward compatible manner, and are standarized so people won't have to worry about loosing the ability to read old log files.

Protobuffers are not ideal for this, sadly...

Posted Nov 21, 2011 18:58 UTC (Mon) by khim (subscriber, #9252) [Link] (3 responses)

Actually Google itself uses protobuffers for logging, but it'll not work outside of Google.

The problem here is that protobuffers separate description of structure and the data. This means that you can not add new type of log without changing the protobuf definition. Works fine if all log types are kept in central repo (log reader just includes all possible log protobuffers from all possible programs), but will probably not work for wide world where you must handle logs from different, independently written programs.

Protobuffers are not ideal for this, sadly...

Posted Nov 22, 2011 0:28 UTC (Tue) by dashesy (guest, #74652) [Link] (2 responses)

Protobuffers seem very interesting! with APIs for C it should fit well with kernel logging as well.

"but will probably not work for wide world where you must handle logs from different, independently written programs"

It seems you know better, but skimming through the docs (for my own validation) it seems there are extensions, so the log reader can have access to a generic definition (that comes with kernel) and independent programs can extend it. Sysadmins can install and (keep updated possibly through repos) the definitions for programs they care most.

Another extension that would be interesting is a reverse .proto generator, where I can supply in my protocol structures header file and .proto is generated/updated based on the last release.

Right, you can do it that way...

Posted Nov 22, 2011 8:25 UTC (Tue) by khim (subscriber, #9252) [Link] (1 responses)

It seems you know better, but skimming through the docs (for my own validation) it seems there are extensions, so the log reader can have access to a generic definition (that comes with kernel) and independent programs can extend it. Sysadmins can install and (keep updated possibly through repos) the definitions for programs they care most.

Well, it should be possible, I'll grant you that. It'll require some scheme to keep all that in sync. Google itself does not have anything like that because they don't need it (centralized repo have it's advantages).

Another extension that would be interesting is a reverse .proto generator, where I can supply in my protocol structures header file and .proto is generated/updated based on the last release.

Not sure what you meant by that. Why will you want to reverse-engineer .proto file? ProtoBuffers are intentionally limited, you can not just shove arbitrary structure in them...

Right, you can do it that way...

Posted Nov 22, 2011 17:02 UTC (Tue) by dashesy (guest, #74652) [Link]

I guess many will object to the idea of ProtoBuffers (specially in kernel) because of its reliance non-mainstream compiler/binutils. Anything that requires more than gcc will probably not get a lot of support, specially if/when it is used only for the logging backend.

Maybe I should have rephrased my wish this way:
A gcc plugin that compiles .proto files directly will be very interesting.
Having Qt in mind, C lovers do not like to see not-exactly-C syntax in their code. Therefore a combination of limited (yes non-arbitrary) C structures and compiler pargmas (for example to specify that a field is optional, and what is the default) instead of .proto syntax, along with the above mentioned plugin will be even more attractive, and likely to be taken into kernel. Here I assumed kernel hackers love C, and I should confess I do not know how powerful gcc plugins are.

That newfangled Journal thing

Posted Nov 21, 2011 20:24 UTC (Mon) by nteon (subscriber, #53899) [Link]

protocol buffers are designed for inter-message compaction, while it seems like their on-disk format is designed for intra-message compaction (string de-duplication, etc). Although protocol buffers + gzip might work just fine.