|
|
Log in / Subscribe / Register

That newfangled Journal thing

By Jonathan Corbet
November 20, 2011
Certain topics, it seems, are just meant to stress-test the LWN comment system; after the unveiling of "the Journal" on November 18, "the works of Lennart Poettering" will have to be added to that list. Lennart (along with Kay Sievers) has clearly hit a nerve with his latest proposal. There is probably enough in this concept to upset just about anybody, but the Journal deserves serious consideration, even if it represents a break from how logging has been handled in the past.

Start with this 2010 quote from Andrew Morton:

The kernel's whole approach to messaging is pretty haphazard and lame and sad. There have been various proposals to improve the usefulness and to rationally categorise things in way which are more useful to operators, but nothing seems to ever get over the line.

The point Lennart and Kay are trying to make is that "haphazard and lame and sad" doesn't just apply to kernel messaging; it describes the approach taken by Linux in general. Messages are human-readable strings chosen by developers, often without a whole lot of thought. These messages may have information useful to automated system monitoring tools, but their free-form nature makes them hard to find and the information contained therein hard to extract. A quick look at what goes on inside a tool like logwatch makes it clear that, whatever their benefits may be, free-form messages are not clean or easy for tools to deal with. The fact that developers can (and do) change messages at any time does not help matters either. Messages are stored in flat files that makes them hard to access, and they are subject to manipulation and editing by attackers. There must be a better way.

The Journal may or may not be a better way, but it does incorporate a number of interesting ideas - though some of them may be "interesting" in the Chinese curse sense. It is tied to systemd, which condemns it in some people's eyes even before the rest of the proposal can be considered. But hooking logging into systemd makes it easy to capture all messages from managed services and get them to their final resting place. It enables tricks like printing the last few messages from a service when querying its status. It also makes it possible for the Journal to verify the source of many messages and to add a bunch of additional metadata to those messages. From this point of view, at least, it looks like a more robust way of dealing with system logging.

When this subject was discussed at the 2011 Kernel Summit, one of the most controversial points was the request that developers attach 128-bit UUID values to most or all messages. Associating a unique identifier with each type of message does make some sense. Users (or their tools) can look up specific messages in a help database to learn what the message really means and how they should respond. Translation mechanisms can use the identifiers to localize messages. System monitoring utilities can immediately recognize relevant events and respond to them. But, while the value of the identifier is clear, there is still a lot of disagreement over its practicability. It is not clear that developers will bother to generate unique identifiers and to use them in a disciplined way.

All complaints about UUIDs were quickly overshadowed by another issue once the full proposal was posted: one might charitably say that there is not, yet, a consensus around the proposed new logfile format. In a sense, that is unsurprising, since that format is deliberately undocumented and explicitly subject to change at any time. But it is clearly a binary format, and a reasonably complex one at that; each field of a log message is stored separately, and there is a mechanism for compressing out duplicated strings, for example. Binary-blob data is allowed as a part of a log message. The Journal's format is clearly a significant departure from the classic, all-text flat-file format used by Unix-like systems since before many of us were born.

There is value in a new log format. Current log files quickly become large and unwieldy; extracting useful information from them can be a slow and painful task. The new format, beyond holding much more information, promises to be space-efficient and very quick to answer queries like "please give me all the disk error messages from the last day." System administrators could probably find a way to get used to the new features of this format in a hurry.

That said, the idea of using an undocumented binary file format for crucial system logging data sets off alarms in the head of just about anybody who has been managing systems for any period of time. The format isn't "secret," of course, but we still should not have to reverse-engineer the format of our logfiles from the Journal source. Unless it is very carefully done, this kind of format could lose a lot of logging data as the result of a relatively small corruption. If the file format really does shift over time, recovery of information from older logfiles could become difficult. It is not surprising that this proposal scares a lot of people; there may have to be some significant changes before the Journal becomes acceptable to a lot of system administrators.

In truth, it may make sense to separate systemd's role in the collection, verification, and annotation of log data from the role of a backend dedicated to distribution and storage of that data. The two tasks are different, and not all sites have the same needs in this area. Mixing the two risks limiting the usefulness of the Journal and muddying the debate about the concept as a whole.

Tradition

This proposal is a request for comments; it has garnered no end of those. What is interesting to see, though, is that a lot of these comments take the form of "syslog has been good for Unix for 40 years, this proposal is trying to change one of our grandest traditions." There is a lot to value in the Unix tradition, but we do need to be careful about sticking to it just because That's How Things Have Always Been Done. Even longstanding mechanisms are subject to improvement. In the case of logging, it seems fairly clear that what we have now is not a thing of timeless beauty. This article will take no position on whether the Journal is the right way to improve on Linux logging, but it will claim that the fact that people are looking at this problem and that they are willing to make significant changes to improve things is good for Linux as a whole.

For much of the history of the Linux system, the unwritten goal was probably something like "let's make something that looks sort of like SunOS 4.x, but which is free, works on current hardware and has the rough edges smoothed a bit." That goal - implement something firmly within the Unix tradition - served to unify Linux development for many years; in our own special way, we were all pulling in the same direction. But that goal has long since been met; there is no longer an obvious target for us to be chasing at this point.

That means that we are heading ever more deeply into new and uncharted territory; Linux is increasingly doing things that no Unix-like system has done before. In the absence of an obvious model, developers doing new things will do them in new ways. The results include Android, KDE4, GNOME3, and, yes, the collected works of Lennart Poettering. If the work is genuinely new, it will look different from what we are used to. Some of this work will certainly turn out to be a bad idea, while some of it will endure for a very long time. But none of it will exist unless developers are willing to take risks and rethink how things have always been done.

Unless a shinier SunOS 4.x is really all that we are after, we as a community are desperately in need of people who are willing to stir things up and try new things to solve problems. To restrict ourselves to what Unix did is an almost certain way to share Unix's fate. We can do better than that - indeed, we have done better than that - but, to do so, we need to create an environment where developers are willing to take some technical risks. Bad ideas need to be exposed as such whenever possible, but ideas should not be shot down just because they do not preserve the way our grandparents did things. The Journal may or may not turn into a sensible option, but the process that creates things like the Journal is one of the most important things we have.

to post comments

That newfangled Journal thing

Posted Nov 20, 2011 18:48 UTC (Sun) by ejr (subscriber, #51652) [Link] (14 responses)

"Changing our grandest traditions" in one person's terminology is "breaking all our existing mechanisms" in another's. Some of the goals within "the Journal" could be met by producing formatted, textual syslog messages. Current systems wouldn't break, and not everyone need jump to a new system all at once. It has been tried and has failed repeatedly.

IME, it often has failed because having to decide on formats, etc. ahead of time places more work on the developer. Logging / diagnostics aren't supposed to be user-visible features, so they're treated like cost centers. Unformatted messages can be parsed and understood later, pushing the cost out to the future. I don't see how yet another interface changes that relationship.

Also, the developers' history of replacing everything wholesale and suffering various levels breakage during the initial time period has followed them into an area where various levels of breakage is not well tolerated. Some of those developers appear unwilling to listen to others' feedback on their methods regardless of differences in goals.

Thank you...

Posted Nov 20, 2011 19:05 UTC (Sun) by khim (subscriber, #9252) [Link] (13 responses)

Wow. You just explaind in a few sentences why Journal is needed (but now why some particular design decisions are needed, of course).

Some of the goals within "the Journal" could be met by producing formatted, textual syslog messages. Current systems wouldn't break, and not everyone need jump to a new system all at once. It has been tried and has failed repeatedly.

Well, this is the best-possible pro-journald argument. If "let's all be friends" approach does not work then it's time for "it's my way or the highway" approach. If developers can not be convinced produce formatted syslog messages then perhaps they can be forced to produce formatted journald messages?

You are right when you say that developers don't think much about logging - but this is when they are left alone. If something is a requirement for admission to some group (repository, certification mark, etc) and if work is not too onerous then they will grumble but do that. If work becomes too hard then they may rethink the decision.

We'll see if journald will manage the right balance, but the fact that voluntary approaches failed in pro-journald argument, not counter-journald one!

Thank you...

Posted Nov 20, 2011 21:15 UTC (Sun) by zlynx (guest, #2285) [Link] (5 responses)

It doesn't work for Windows.

Sure, developers are forced to report their event log messages in the event log binary format. They are also supposed to produce DLLs that contain text identifiers for the log messages.

Guess how many of them do this?

I'm as guilty as anyone. I just used one identifier and stuffed a free-form text message into the binary data field. I wasn't about to go through every debug log print and make a unique ID for it. Especially since C++ iostreams don't lend themselves to that in any way.

Thank you...

Posted Nov 21, 2011 2:07 UTC (Mon) by alankila (guest, #47141) [Link]

I read that the uuid for journal message is optional. The recommendation, probably, is to add it, but of course if you do then you promise not to change the circumstances and meaning of the message, as you are basically declaring it as an API if you do.

Thank you...

Posted Nov 21, 2011 19:00 UTC (Mon) by faramir (subscriber, #2327) [Link] (3 responses)

Microsoft can't stop you from shipping your program if you don't follow the spirit of the rules.

Ubuntu/Fedora/Debian/etc. can effectively do this by refusing to include your program their core distribution.

That's not to say that the distributions would actually do this or that they wouldn't get push back from the developers of "important" packages. Still the differences between the Microsoft and Free Software worlds may make enforcement of good practices easier in the Free Software world.

Thank you...

Posted Nov 23, 2011 13:12 UTC (Wed) by sorpigal (subscriber, #36106) [Link]

Distributions don't have any more power than Microsoft to control what gets used on their OS. Distro packages are not the entire world.

Thank you...

Posted Dec 7, 2011 5:36 UTC (Wed) by Klavs (guest, #10563) [Link] (1 responses)

I will bet that a program, starting with logging everything with free form text to the same ID would, if used by sysadmins, quickly be improved upon, in regards to id logging. As a sysadmin I often have to look through source code to find the real reason for an error message (that's one big reason I prefer Open Source) - and if the program already used something with support for proper logging, like journald idea - I would easily whip up a patch for upstream to improve logging for the instance - helping myself in the future as well.

Thank you...

Posted Dec 7, 2011 10:35 UTC (Wed) by dlang (guest, #313) [Link]

what is it about the journal 'proper logging' that makes it any easier for you to write a patch to fix bad logging than with normal syslog?

normal syslog can deal with structured logging already via RFC 5324 structured logging, and the CEE specs standardize this further http://cee.mitre.org/

why are these not as good as the journal proposal?

Thank you...

Posted Nov 21, 2011 1:42 UTC (Mon) by jamesh (guest, #1159) [Link] (5 responses)

I doubt journald will be able to force users to produce structured messages.

There are many applications and services that produce messages through the syslog() C library call. Either journald will have to accept those messages and deal with some amount of unstructured messages, or you'll need to keep a traditional syslog daemon around and deal with two system logs.

I would be quite surprised if they decided to ignore messages generated by syslog() calls, so there will probably be a lot of unstructured messages for the foreseeable future.

That doesn't reduce the usefulness of converting a few key messages over to a new API though.

Thank you...

Posted Nov 21, 2011 6:19 UTC (Mon) by pilif (subscriber, #3857) [Link] (3 responses)

Reading the proposal, I would say that the structure doesn't come from the application that's doing the logging, but from the Journal itself. This is things like process id, user, date and everything else that can be inferred from an external standpoint (these are the _ prefixed fields in the spec)

What journal allows in addition to that is that the application can provide additional arbitrary key/value pairs (without the beginning _) in addition to the traditional free form message.

So a traditional syslog() sent message would still gain some structure, but if you usr the journal-native API, you even get to log application-specific values.

That said, I'm really torn on this proposal. I DO see many advantages, but on the other hand, NOTHING beats the accessibility of cat/grep and tail -f

Needing a special tool, just to look at logs is a huge increase in complexity. Especially if we are using a binary format. What if that file gets corrupted? You can still read a corrupted textfile, but can those future tools still read a corrupt binary file? Can the developers predict any possible type of corruption?

As I said: I'm torn.

I am looking forward however to see how this is going to play out.

Philip

Thank you...

Posted Nov 21, 2011 8:59 UTC (Mon) by niner (guest, #26151) [Link] (2 responses)

Do you really store your logfiles in plain text for all time? No logrotated with enabled compression?

Speaking for myself, my logfiles are already stored in a binary format. Even more, this format has changed over time! It used to be gzip, has been bzip2 for quite a while and nowadays is xz. Oh and yes, if these files are corrupt in some way, there's just no way to recover. But this has never been a problem. If random files are corrupted, I have a far greater problem than logfiles.

So I even nowadays need a special program to access the binary formatted logfiles: xzcat. On my wireless router it's called logread and I have no idea where logfiles even are stored...

Thank you...

Posted Nov 21, 2011 19:23 UTC (Mon) by cmccabe (guest, #60281) [Link] (1 responses)

A lot of people have compared using the journald binary format to using gzip to compress your logfiles. However, this comparison may be misleading.

gzip is a block compressor. You don't have to have access to the entire file to recover some of the data. You can actually seek to the beginning of a gzip block and start reading without having read all the data that came before.

A lot of media formats have this property as well-- mp3 for one. If the beginning of the mp3 is lost, you can still access the last part. Video formats often have 'key frames' that can be used to recover synchronization if the decoding got screwed up.

Being able to recover something, even if part of the file is messed up, is a very important property for log files. Disks can and do fail and start returning garbage-- I just had one turn this week. I hope that the journald designers think about issues like this, at least in later versions of the format.

Framing with FEC.

Posted Nov 21, 2011 19:43 UTC (Mon) by ejr (subscriber, #51652) [Link]

Indeed. A simple framing including some forward error correction code around some well-known, extensible encoding (maybe text, maybe protocol buffers, dunno) could prove even more resilient to failures than the plain-text logs. Same can be said for the text files used by dpkg, etc. VMS FTW?

Thank you...

Posted Nov 24, 2011 7:30 UTC (Thu) by kleptog (subscriber, #1183) [Link]

I'm really looking forward to what comes out of this project. The problem with structured data in your syslog log now is that every program does it differently and is therefore useless. We now have the opportunity you make a standard structure that can be supported everywhere.

A seriously, many many applications don't use syslog directly but some kind of wrapper, which allows you to say:

logfunction(severity, fmtstring, args...)

Look! Structure! Imagine being able to search log messages like: "all message of level NOTICE with 'sda' as an argument". Use the MD5 of the format string as ID and you've got something that people using translated programs can do something with. You could even translate your logs to another language after the fact.

Or, show the IP fields of all records with "Traceback" in the message.

Hurrah!

Thank you...

Posted Dec 21, 2011 7:15 UTC (Wed) by topher (guest, #2223) [Link]

If developers can not be convinced produce formatted syslog messages then perhaps they can be forced to produce formatted journald messages?

**buzz**

Thanks for playing, better luck next time.

This will not work. You can't force developers into something like that. Instead, they'll just dump their log data to a local file and ignore "the journal". Now you're left in a much worse situation than you were before, because you don't even have the logs hitting syslog.

That newfangled Journal thing

Posted Nov 20, 2011 18:48 UTC (Sun) by cuboci (subscriber, #9641) [Link]

Just the kind of article that reminds me that paying for LWN is really worth it. Thanks a lot.

That newfangled Journal thing

Posted Nov 20, 2011 18:53 UTC (Sun) by drag (guest, #31333) [Link]

Nice article. It was a good read. Hard to find things to argue about in it, so you should be happy. :)

That newfangled Journal thing

Posted Nov 20, 2011 18:56 UTC (Sun) by yoush (guest, #38940) [Link] (9 responses)

Trying new things is good.
Rethinking old traditions is good.

But forgetting of what was good in old traditions is bad.

It would be nice if people who propose to do things differently, will first write explicit lists of what was good in old way of doing things, and explain how these items will be done in the new way.

That's why we have a reviewers :-)

Posted Nov 20, 2011 19:15 UTC (Sun) by khim (subscriber, #9252) [Link] (2 responses)

It would be nice if people who propose to do things differently, will first write explicit lists of what was good in old way of doing things, and explain how these items will be done in the new way.

Well, this is where critics come in the picture. Lennart is suprisingly good when you present him a real-world task and is always ready to change his creations to support them. He notoriously bad when the real-world task is well, I have a script here, I'll not explain what it does and why I need it, but you'll break it so it's your resposibility to fix it, yes, but when you show willingless to think about things - he shows the willingless to help you.

One of the first decisions in the case of journald is side-by-side installability of syslog and journald. This means that the questions like "how should it work with remove loggers and such" have a natural answer "the same way as before". This also means that pressure to fix programs to be journald-firendly is not as big, but apparently Lennart understands that he can not force everyone to use journald right away.

That's why we have a reviewers :-)

Posted Dec 21, 2011 7:22 UTC (Wed) by topher (guest, #2223) [Link] (1 responses)

One of the first decisions in the case of journald is side-by-side installability of syslog and journald.

Better yet, instead of comparing least common denominator syslog, how about comparing it to the research and efforts that are already at work in solving these same problems, and already have solid workable solutions (without many of the negative aspects of "the journal").

If Lennart wants to come up with a new syslog replacement, doing better than basic syslog isn't good enough. Not *nearly* good enough. It needs to be better than syslog *and* all of the recent research *and* better than anything that could be layered on top of existing syslog (which is where it currently fails badly, IMO).

That's why we have a reviewers :-)

Posted Dec 21, 2011 16:58 UTC (Wed) by nix (subscriber, #2304) [Link]

If Lennart wants to come up with a new syslog replacement, doing better than basic syslog isn't good enough.
It is if you can force people using systemd to use it.

(This just provides a reason not to try out systemd: do I really want an increasingly large chunk of the system sucked into it, as now seems likely, disrupting my existing procedures? It's taking over from cron, and now syslogd... thanks but no thanks.)

That newfangled Journal thing

Posted Nov 20, 2011 19:25 UTC (Sun) by smurf (subscriber, #17840) [Link] (4 responses)

The problem is that the old traditions do not work any more.

The number of logging messages has increased significantly. You don't need most of them except when something goes wrong, at which time you realize that you didn't log enough. Text files may be easily greppable but they're a security nightmare. So is remote syslogging via UDP. And so on.

That newfangled Journal thing

Posted Nov 20, 2011 21:05 UTC (Sun) by dlang (guest, #313) [Link]

Remember that there are many different ways to send logs with syslog, not just UDP. the options include encryption and delivery confirmation. Rsyslog (the default syslog on almost every distro now), supports all of this (including delivering well formatted messages that conform to the latest syslog RFC) out of the box.

That newfangled Journal thing

Posted Nov 21, 2011 5:48 UTC (Mon) by gmaxwell (guest, #30048) [Link] (2 responses)

"Something must be done!" "This is _something_" "Then it must be done!"

That newfangled Journal thing

Posted Nov 21, 2011 20:50 UTC (Mon) by jcm (subscriber, #18262) [Link] (1 responses)

Exactly what is going on here.

That newfangled Journal thing

Posted Nov 23, 2011 21:14 UTC (Wed) by jonabbey (guest, #2736) [Link]

How do you figure? Do you imagine that this proposal is going to be adopted without regard to its virtues and deficits?

That newfangled Journal thing

Posted Nov 23, 2011 16:47 UTC (Wed) by sam-williams (guest, #57470) [Link]

yoush makes a particularly valuable point about implementing new technology. If you don't define what you have currently you are just changing things for change sake and that is a hit and miss proposition. Understand your needs, what your system currently provides, and then see if a significant change is still warranted.

Its important that once implemented, the Journal be accessible to the lowest common-denominator interface which means the commandline using the standard set of commandline utilities to extract desired information. If we can't produce this we've lost the value of this function.

Interesting idea, but...

Posted Nov 20, 2011 19:27 UTC (Sun) by tshow (subscriber, #6411) [Link] (16 responses)

From what I've seen of the journal proposal, I have only two real problems with it, and one of them isn't really a reasonable problem. The other problem is a deal breaker, though; it's a fundamental failure.

The lesser (and perhaps unreasonable) problem I have with the journal scheme is that I'm not convinced the security is going to work in most cases. Most people are going to keep the signing keys in a stupid, obvious place (like /root, or ~), so script-kiddying a complete regeneration of the log hashes isn't going to be hard. The people who take proper precautions will be fine, but that's true now; if you run syslog out a network port or a serial line (or to a line printer...) it's nearly impossible to tamper with it after the fact.

On the other hand, extra security is extra security. Maybe sometimes it will make things just that little bit harder.

The major problem I have is with the undocumented binary format. I have no problem with "binary format", it's "undocumented" that isn't doing it for me. There should be a well documented "dump log to raw text" tool, and a reasonable commitment to format stability. I should be able to read the dump utility source, break out the programming language of my choice, and write my own log data mining tools without having to go through the magic log translator.

There is nothing useful to be gained by obscuring the format; it's the worst form of improper incentives. Attackers have lots of incentive to figure the format out in detail, and can take the time to do so. Defenders just want the damned thing to work, and don't want to have to waste time on understanding the fiddly details until they've been compromised and suddenly have no choice.

Furthermore, unless it's a very simple format with no internal referencing, there's always the chance that an attacker can find some way to make the log file skip parts of itself or lie. Or, for that matter, buffer overflow the log reading tool and do yet more damage. If you could get a buffer overflow into the log data that invoked a "they're on to us, clean everything up" script...

Fundamentally, though, this misses the worst part of the design. Here's the deal breaker: We now have a single point of failure; if the attacker can compromise the log reader program, they have you. Everything goes through the log reader program. The attacker doesn't even have to tamper with the logs any more, they just have to replace the log reader program with something that lies.

With text-based logging, the attacker would have to replace nearly every binary in the system to prevent you from reading the real log (though they could probably guess and just hit less, vi and tail...), so it's easier to rewrite the log. With binary logging, your selection of available tools for reading the log becomes much smaller, and the log reading tools become a potential target rather than the log itself. When there's only one tool (or a small suite of standard tools), the tools themselves become an extremely attractive target.

If I were attacking a system with this kind of logging, what I would be trying to do is (1) replace the log reader with something that lies, and (2) try to get an overflow attack into the actual log data so that if someone tries uncorrupted tools I can corrupt them from within.

If the format were documented and I could write my own tools, the attacker can no longer assume that they can compromise anything that could hit the log, and are back to having to tamper directly with the log.

So, to me, hash-chain log messages? Sure, sounds good, should be worth 10% more security. Data mining messages? Also good. Binary format? No problem, as long as it's simple, stable, well documented, heavily audited for potential malicious data attacks, and has a tool that will dump it. Undocumented log format? Biiiiig red flag.

That undocumented format takes this from a fairly good idea to a very bad one.

Interesting idea, but...

Posted Nov 20, 2011 19:44 UTC (Sun) by vrfy (guest, #13362) [Link] (12 responses)

Stabilizing the format is just not on the table right now. We need to get the overall model sorted out first, and that will take its time. As soon as we reach a point where stuff works as expected, there is no problem to make that format versioned, formally defined, and provide backwards read compatibility.

For now, the journal files are just an indexed ring buffer that follow store/forward logic. Anybody who depends on any data/log archival or classic tools to run, should run the syslog daemon at the same time, which gets exactly the same data and writes the same files as before.

The journal data is a zero maintenance buffer, that just takes the maximum configure disk space an no other ressources, it does not change any aspect of classic syslog. Local tools like systemd, or other agents will rely on the indexed data from the journal during runtime, and can not work with syslog files.

Journal files are not a log archive in the same sense as syslog files are for some time. That's just impossible to promise from where we are right now, hence the warning about it.

Interesting idea, but...

Posted Nov 20, 2011 19:59 UTC (Sun) by quotemstr (subscriber, #45331) [Link] (1 responses)

> Stabilizing the format is just not on the table right now.

Having a fluid format during development is acceptable. Shipping a critical component that writes with an unstable format is not.

> the journal files are just an indexed ring buffer that follow store/forward logic

Have you guys looked at ETW, the Windows event logging facility? The designers encountered and solved many of the issues that journal will run into. I still prefer plain-text logs to binary ring-buffer abominations, but if you're going to go down the latter route, you might as well learn from the past.

One convenient feature of ETW is a configurable and flexible approach to log management. ETW lets you decide on a per-log basis whether you want ring or serial logging, whether you want new log files to be created as old ones fill up, how often you want logs flushed to disk, and so on. ETW also has a "real-time consumer" option that basically pipes log messages to a process instead of writing them to disk, allowing programs to perform actions when certain system events happen. (You can even start a service when a certain log message appears.) The kernel also uses ETW to log its messages, allowing you to configure all the logging in one place.

Now, the details of ETW are disgusting: there are too many overlapping log options (you can do log rotation two different ways), too many log-message formats, too much metadata for each message (do we really need separate "keywords" and "categories" and such?), and too baroque an API. It's too damned hard to get simple plain-text output from the Windows event-logging system.

But beneath all the grime, there are some interesting ideas.

Interesting idea, but...

Posted Nov 23, 2011 14:41 UTC (Wed) by skorgu (subscriber, #39558) [Link]

Having recently written some C# code I took one look at the windows event log API, uttered some curse words and used log4net to write to a normal file instead.

The idea of highly structured output appeals to me but it seems more like tracing than logging. I'd rather have a kickass perf and a kickass syslog than try to combine the two.

Interesting idea, but...

Posted Nov 20, 2011 20:03 UTC (Sun) by tshow (subscriber, #6411) [Link] (1 responses)

Hmm. What's been put forth so far has strongly implied a replacement for syslog, but you're describing it as a supplement.

At any rate, if the plan is to have a properly documented format for the log data when the project hits 1.0, I withdraw my objection; my concern was that it appeared that undocumented was the overall plan, not simply a consequence of the early development phase.

Interesting idea, but...

Posted Nov 20, 2011 20:19 UTC (Sun) by vrfy (guest, #13362) [Link]

The format might be stabilized when it's reasonable stable, there is no version number, or any specific plan to tell now. We will get there over time. Until that, nothing should expect to be able to read/write the raw files directly. They need to get dumped with journalctl, or read with the library API, that's all what we can promise now.

The classic syslog model is not touched at all. If anything relies on any of the syslog features, the format, the remote logging, the files, it should run syslog like it always did.

This is something that runs on the local machine, and serves as the base for tools that need to make decisions or provide the 'history' of services. If syslog is the model to look at, journald is just a proxy.

The journald design is network-aware, but in no way network-transparent. All that can be done pretty efficiently by additional tools, but these tools will probably not be part of the core installation.

Interesting idea, but...

Posted Nov 20, 2011 22:19 UTC (Sun) by Cyberax (✭ supporter ✭, #52523) [Link] (7 responses)

Have you considered using reverse domain names for message ids?
UUID are just too unwieldy. Imagine that I'm a developer writing: 
class SomeClass
{
    void someMethod()
    {
         while(true)
         FILE *fl = fopen(...);
         if (!fl)
             log(DEBUG, "File with data is not found, trying to create it");
    ...

At that moment you realize that you need a UUID for this message. Ok, easy enough: 
class SomeClass
{
    void someMethod()
    {
         while(true)
         FILE *fl = fopen(...);
         if (!fl)
             log(DEBUG, "d92bc8ba-7a98-4e49-8384-8ee013e2f773", "File with data is not found, trying to create it");
    ...
But damn! Now my string wraps around 80 character limit! And if I'm unlucky it can wrap around inside the UUID itself, so I need to reformat my code.

Alternatively, I need to create a file with #defines of UUIDs and write things like this: "log(DEBUG, FILE_DATA_NOT_FOUND_UUID, "File with data is not found, trying to create it");" which leads to duplications. So most lazy programmers (such as myself) would just define a couple of UUIDs and use them with free-form text fields for verbose messages.

So please, consider making a good API for developers to be your first priority. And using UUIDs in API is definitely doubleplus ungood.

Actually I think it's good idea to have two types of logs

Posted Nov 20, 2011 23:43 UTC (Sun) by khim (subscriber, #9252) [Link] (6 responses)

I think Google does it right. They have two separate types of logs:
1) Binary structured logs which are moved to centralized storage, kept for a long time and are used for a lot of things.
2) Debug free-text logs which are only kept around till server is restarted.

Sometimes messages generated in logs of second type are deemed valuable enough and are "promoted": at this point unique message type should be added, message should be documented, etc. Adding unique UUID is trivial in comparison :-)

I doubt it makes much sense to force structure on all log messages which program can ever generate. If log record is supposed to be analyzed exclusively by humans then it's Ok to use free-form text for it. If it can be used by other software then it's time to organize and document it.

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 1:22 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

Well, if it can be made easy to use structured logging everywhere, then why not do it?

I don't really see how reverse domain names are worse, except of handwavy 'it's faster to use UUIDs'.

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 1:31 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

what is making it hard to use structured logging with syslog?

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 1:49 UTC (Mon) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

Well, the lack of structure, for one thing. There are even tools (like http://www.splunk.com/ ) that exist to parse at least some of the logs.

Actually I think it's good idea to have two types of logs

Posted Nov 21, 2011 10:31 UTC (Mon) by dlang (guest, #313) [Link]

I am very familiar with splunk, it doesn't parse the logs, it just indexes every word in the logs.

but back to my point. you can send structured logs via syslog today, there is even a standard to do so. People choose not to do this today, but that's not the fault of the syslog mechanism, that's the fault of the programmers.

As has been noted elsewhere, you can still make unstructured logs through this new mechanism, so the new mechanism doesn't give you structured logs any more than syslog does

Actually I think it's good idea to have two types of logs

Posted Nov 23, 2011 17:01 UTC (Wed) by sam-williams (guest, #57470) [Link]

Splunk is really more about using the data in the logs for enterprise systems management. Suggesting its purpose is solely for reformatting syslog data is inaccurate.

Structure would improve things a bit, but no self-respecting systems administrator would suggest they can't do their job without a bit of binary hand-holding. The binary fileformat could cause more problems then it cures. Care should be used in providing an ability to access this information with simply tools.

As I've already said...

Posted Nov 21, 2011 8:08 UTC (Mon) by khim (subscriber, #9252) [Link]

Any conclusion made from false premise will be true so you rant is certainly valid.
Well, if it can be made easy to use structured logging everywhere, then why not do it?

The only way known to humanity is to put fixed value in all fields except textual "details" field and then write free-form description there. A lot of peoples tried to make "structured logging everywhere" work, yet none succeeded. This means it's time to stop trying to push "structured logging everywhere" idea and think about different question "do we really need structured logging?" and the answer is "probably not". A lot of logs only make sense for someone who has detailed knowledge of the program. If you don't think long and hard about what your log is trying to convey and to whom then then no amount of structure applied will help. And not all log messages deserve such attention. At least this is what developers usually think - and if you system will not be accepted by developers then it may as well not exist.

Just a small note...

Posted Nov 20, 2011 19:51 UTC (Sun) by khim (subscriber, #9252) [Link] (1 responses)

With text-based logging, the attacker would have to replace nearly every binary in the system to prevent you from reading the real log (though they could probably guess and just hit less, vi and tail...), so it's easier to rewrite the log.

Actually attacker only needs to inject small kernel module and that's it. There are plenty of material for script kiddied on the internet.

If I were attacking a system with this kind of logging, what I would be trying to do is (1) replace the log reader with something that lies, and (2) try to get an overflow attack into the actual log data so that if someone tries uncorrupted tools I can corrupt them from within.

Well, overflow in log reader tool is valid concern at least. I guess feasibility will depend on size of these tools, but yes, this is pretty scary thing. Of course there are plenty of sandbox options for you to use (seccomp, nacl, etc), but still, this is valid concern.

Just a small note...

Posted Nov 20, 2011 19:57 UTC (Sun) by tshow (subscriber, #6411) [Link]

> Well, overflow in log reader tool is valid concern at least.

It can be (mostly) mitigated by ensuring that the log format is simple and the log reader is paranoid, but it's definitely worth paying attention to. The more self-referential the log format is, the more nightmarish it gets to ensure the reader is bulletproof.

Interesting idea, but...

Posted Nov 20, 2011 19:53 UTC (Sun) by butlerm (subscriber, #13312) [Link]

I agree, an undocumented format for critical data is insane. It is like bringing everything wrong with black-box-proprietary-binary-goo, fall-over-and-die-on-the-slightest-change software to Linux for no good reason.

Who cares if the file format changes? A good format ought to be extensible so that existing readers do not break. And if tools do process the format directly instead of relying on a library as they should, then naturally it is their responsibility to keep up with it.

There are tools that can and do process raw filesystem data, and we don't use that as an excuse for trying to hide the format in a black hole, or to never change it in any substantial respect either.

What if someone wants to write a different library that accesses the same data, possibly in a different language, after the fact, many years later, on a different operating system? Is the log data just supposed to be some sort of impenetrable format that is good for nothing without a compiled version of the original tools? Suggesting that people _reverse engineer_ it is laughably absurd. All the extra cost, repeated multiple times over, without the benefit of actual compatibility - all the bugs, weaknesses, and oversights of the original source base, if that.

That newfangled Journal thing

Posted Nov 20, 2011 19:55 UTC (Sun) by mordae (guest, #54701) [Link] (14 responses)

I really like Journal idea, but where ho got me is the absence of logging over network. You see, we have a bunch of machines with a small rootfs on a flash drive and tmpfs /var. The rsyslog sends all messages to a central machine using RELP.

Now, if I implement Journal, I can either create small logs in the /var and rsync frequently (bad if the machine dies, I don't have the latest logs) or continue using this and gather logs in Journal on the central server. But then I won't have correct metadata, so why even bother?

So... logging to a network file system? Seriously?

That newfangled Journal thing

Posted Nov 20, 2011 20:23 UTC (Sun) by intgr (subscriber, #39733) [Link] (13 responses)

FTFA: "In the initial version journald’s network support will be very simple"

"In a later version we plan to extend the journal minimally to support live remote logging, in both PUSH and PULL modes always using a local journal as buffer for a store-and-forward logic"

The sky isn't falling. In fact, such a store-and-forward model sounds like a much more robust method than syslog over TCP or UDP.

That newfangled Journal thing

Posted Nov 20, 2011 21:13 UTC (Sun) by dlang (guest, #313) [Link] (11 responses)

If you use rsyslog (default on most linux distros), you already have store and forwarding delivery available to you, complete with delivery confirmation, encryption, etc.

Lennart could get almost everything he is lookin for out of rsyslog today.

The hashes would be trivial to add (and for that matter, you could use a database store and have it do the normalization and hashing as part of the insert today)

The ability to easily combine or split files of logs is a very important feature, changing to a binary format that compresses early, but prevents the functionality of simple tools like split and cat is a huge step backwards.

The only think that this proposal actually gains is knowing what pid generated the message, and that only works on a local machine (as soon as you send the log remotely, the remote machine can no longer trust the pid)

Yet again: there are no difference...

Posted Nov 20, 2011 21:23 UTC (Sun) by khim (subscriber, #9252) [Link] (3 responses)

As soon as you send the log remotely, the remote machine can no longer trust the pid

Again: there are no difference between local and remote case. You can trust pid till the "intrusion moment" even if you log remotely and after that moment even local log is suspect.

I still fail to see where you get this weird ideas about local/remote dichotomy.

Yet again: there are no difference...

Posted Nov 20, 2011 21:35 UTC (Sun) by dlang (guest, #313) [Link] (2 responses)

I am not saying that you can trust it locally, I am just pointing out that the remote machine has no way of knowing if what is sent to it is valid or not. The only way to have the remote machine be able to trust the data it's sent is to have the full TPM lockdown in place (and trust that there is never a flaw that allows it to be broken)

As soon as you go through a second daemon on a local system, you have to trust that that daemon hasn't been broken.

As soon as you read a message from disk you have to trust that the file hasn't been tampered with (and if you hash the file or the messages to try and prevent this, you now have to trust that your store of valid hashes hasn't been tampered with)

I don't know what you wanted to say, but I DO know what you said...

Posted Nov 20, 2011 22:14 UTC (Sun) by khim (subscriber, #9252) [Link] (1 responses)

I am not saying that you can trust it locally,

Rilly? Perhaps my English is failing me, but I thought and that only works on a local machine was quite unambigous...

I am just pointing out that the remote machine has no way of knowing if what is sent to it is valid or not.

That's fair. But as I've noted there a little difference between local and remote case: if you know daemon and kernel are Ok you can trust the logs, if you don't know if they are Ok then you don't. Since the usual way to see if something is broken is to analyze logs, again, and they are available on both local and remote system... no, I don't get your point.

What makes logging over network so special and why can you trust info about pid in local case but not in remote case?

I don't know what you wanted to say, but I DO know what you said...

Posted Nov 20, 2011 22:47 UTC (Sun) by dlang (guest, #313) [Link]

Ok, I should have worded it as "and that only has a chance of working on a local machine"

once you go to another machine, you no longer 'know' anything about what is really generating the message (unless you have crytographic authentication to the sending program, and event that only proves that the sender has access to the key)

That newfangled Journal thing

Posted Nov 30, 2011 7:34 UTC (Wed) by alison (subscriber, #63752) [Link] (6 responses)

dlang offers:
The hashes would be trivial to add (and for that matter, you could use a database store and have it do the normalization and hashing as part of the insert today)

Why not in fact use git (or similar) to frequently snapshot a flat ASCII log file, storing the deltas in a repo over which hashes are generated in the usual manner? The UUIDs need not be present in the ASCII log but can be git tags that are stored as part of the commit data. Then git can be used in its usual fashion to propagate the log to networked machines if desired. Furthermore, the generation of hashes and remote propagation provide the usual level of verifiability we associate with git.

In other words, why not keep the flat file, but generate a verifiable log in a standard machine-readable format from it on the fly? Is this not how we are already using DVCS to generate trees from programmer-generated source code? Why wouldn't git work just as well with programmer-formatted log messages?

That newfangled Journal thing

Posted Nov 30, 2011 8:49 UTC (Wed) by dlang (guest, #313) [Link] (5 responses)

git is horrible overkill and inefficient if all you are doing is dealing with one log file.

tripwire, ossec, and other similar tools already can track a log file and detect the difference between the file being extended and the file being modified.

Also, if you have another machine you can send data to, just send the logs using the standard syslog mechanisms. unless you are generating gigantic amounts (hundreds of thousands of logs per second) of logs or would be logging over a wan, the bandwidth needed for the logs is just not going to be significant

That newfangled Journal thing

Posted Nov 30, 2011 22:20 UTC (Wed) by elanthis (guest, #6227) [Link] (4 responses)

I love it. Bigotry and emotions run rampant in technical discussions more and more these days.

Argument: journald is more complex than syslogd, and complexity is evil!

Proposal: use the simpler syslogd and then add complex and error-prone log parsing toolkits to get 70% of the features of journald at 200% the complexity cost.

Rationale: Pulseaudio was buggy a couple years ago.

That newfangled Journal thing

Posted Nov 30, 2011 23:26 UTC (Wed) by dlang (guest, #313) [Link] (3 responses)

for most people, the new features of journald won't matter, they either don't have any worm device to store the hashes to to make things secure, ot they don't care about such features because they send all of the logs to a remote system.

he's 'solving' a problem that isn't really there, and he doesn't actually solve the stated problem.

Also, please point out anywhere that I have said anything about pulseaudio.

That newfangled Journal thing

Posted Nov 30, 2011 23:34 UTC (Wed) by anselm (subscriber, #2796) [Link] (2 responses)

Read the <expletive> proposal. Journald isn't just about the hashes.

That newfangled Journal thing

Posted Nov 30, 2011 23:44 UTC (Wed) by dlang (guest, #313) [Link] (1 responses)

I have read the proposal, and I still think that overall he is solving problems that don't exist in ways that don't really solve the stated problem

if you want to ignore the hashes part of things, we can talk about the structured log part of things. logs are only as structured as the programmer creating them makes them, if you have a super-detailed log structure available and the programmer creates a field "details" type "string" and puts everything into that field it is going to be just as unstructured as syslog traditionally has been.

He ignores or is ignorant of recent standards in syslog (some of which go back quite a few years)

he has a few new ideas buried in the proposal, but they are so overshadowed by misstatements and 'solutions' to problems that already have documented, standardized solutions (that are not compatible with the proposed solution other than running logging services in parallel) that it undermines the entire proposal

That newfangled Journal thing

Posted Dec 1, 2011 17:03 UTC (Thu) by kh (guest, #19413) [Link]

Thank you for being a voice of reason.

That newfangled Journal thing

Posted Nov 21, 2011 8:00 UTC (Mon) by mordae (guest, #54701) [Link]

Damn, I've missed that. I don't know how I've gotten the impression that they ar not going to bother with network logging.

Ad security: The security is not the only reason for logging! It's also about overall serviceability of the infrastructure. And in that case... who cares about trusting remote hosts. If libvirt crashes, I need to know why and in that moment I really don't speculate about incursion.

That newfangled Journal thing

Posted Nov 20, 2011 21:27 UTC (Sun) by dlang (guest, #313) [Link] (14 responses)

> This proposal is a request for comments; it has garnered no end of those. What is interesting to see, though, is that a lot of these comments take the form of "syslog has been good for Unix for 40 years, this proposal is trying to change one of our grandest traditions." There is a lot to value in the Unix tradition, but we do need to be careful about sticking to it just because That's How Things Have Always Been Done. Even longstanding mechanisms are subject to improvement.

Yes there is room to improve log handling, but there is no need to throw away everything that is good about the existing systems and force everything to re-write it's logging in the process.

If you look at rsyslog and syslog-ng you will see that there has been a lot of improvement in 'traditional' logging over the last several years. Most of this improvement is invisible to desktop users because they just don't care about it, but it is very visible to people who run server farms and actually use their logs on a regular basis

There have been many people who have advocated forcing all applications (and the kernel) to change their logs to make them easier to identify and parse. There are even standards on how to do this (the latest syslog RFC defines key-value pair formatting of log messages and CEE defines standards for what to put in the log messages), but application and kernel programmers do not choose to make the effort to standardize their output. I don't see any way that yet another logging mechanism would make it any more likely that the messages would be well standardized.

One thing that systemd is in a unique position to do, is to define a custom /dev/log for each service that it starts, so that it could insert a tag into each message generated by the program claiming that systemd has validated what service it came from, this could be done without any need to introduce any new logging mechanism, just create a simple pass-though that tweaks the message before handing it off the the existing syslog daemon.

If the messages are well formatted, it is very easy for modern syslog tools to do very powerful things with them. logwatch and simple event correlator (SEC) have very complex rules most of the time because they are dealing with poorly formatted log messages, but if you can feed them well formatted log messages they can have rather simple configurations.

That newfangled Journal thing

Posted Nov 20, 2011 23:19 UTC (Sun) by HelloWorld (guest, #56129) [Link] (13 responses)

> Yes there is room to improve log handling, but there is no need to throw away everything that is good about the existing systems and force everything to re-write it's logging in the process.
The proposal mentions that one can use syslog(3) to generate log messages for journald.

That newfangled Journal thing

Posted Nov 20, 2011 23:42 UTC (Sun) by dlang (guest, #313) [Link] (12 responses)

and if someone does so, what is going to make the logs any more structured than the logs created by syslog(3) today?

That newfangled Journal thing

Posted Nov 21, 2011 0:04 UTC (Mon) by elanthis (guest, #6227) [Link] (8 responses)

Why does it have to be all or nothing?

The inability or unwillingness of most LWN users to think like a damn programmer always confuses me.

That newfangled Journal thing

Posted Nov 21, 2011 0:16 UTC (Mon) by dlang (guest, #313) [Link]

well, there's nothing preventing any programmer from writing good, well structured logs today (there are even RFC standards for how to do so)

but the claim is that this new tool will 'solve' the problem of poorly structured logs. If it doesn't actually prevent the programmer from writing poorly structured logs, then that needs to be called out as that claim is false.

That newfangled Journal thing

Posted Nov 21, 2011 0:40 UTC (Mon) by jubal (guest, #67202) [Link] (1 responses)

Ah, I always tried to to think like a saved programmer, you see. So you're saying that damnation is the key to understand a developer?

That newfangled Journal thing

Posted Nov 21, 2011 1:51 UTC (Mon) by elanthis (guest, #6227) [Link]

:) The games industry is basically all Windows, so I'm only really familiar with damned programmers these days. :)

That newfangled Journal thing

Posted Nov 21, 2011 9:05 UTC (Mon) by tpo (subscriber, #25713) [Link] (4 responses)

> Why does it have to be all or nothing?

Indeed. The comments to this article seem to be arguing:

  • I want syslog
  • I want structured logging
And there's a few that argue:
  • we can run syslog in parallel

However, the synthesis can go further than that:

  • Journal could also be accepting unstructured log messages and just stuff them into
       UUID: crap
   Structure: none at all
   Here's-the-content: bla bla
    entries. In API language that would be:
       Journal.log_no_questions_asked("just the message here").
    All other params would be reset to default (or "untidy crap").
  • on the other hand syslog could also receive messages directly from Journal. Of course journal could have a nice configuration facility where you can pick what gets output from Journal so that you only get what is of interest to you in your syslog log.
That way we have the best of both worlds (with additional HD space cost):
  • If needed one can do highly structured messages, which are well machine processible.
  • And we get at the same time an easily human readable/grep-able text.
And both contain all the information that are expected. With the bonus, that I can go and grep my syslog, and if I find out that I indeed need razor sharp data, then I can additionaly go and query the Journal.

I can't see why it's not possible to have this and why such a solution would be problematic for any of the "camps" above?

That newfangled Journal thing

Posted Nov 21, 2011 10:37 UTC (Mon) by dlang (guest, #313) [Link] (3 responses)

you can have structured logging with syslog today, all you have to do is to structure the messages you write from your application (or from the kernel). Syslog even defines the format in the most recent RFC, so no changes are needed to do structured logging via syslog.

no need to run two logging systems in parallel, no need to figure out how to deal with logs that only get written to one of the two systems, just define the format of your logs are you are good to go.

That newfangled Journal thing

Posted Nov 21, 2011 12:46 UTC (Mon) by tpo (subscriber, #25713) [Link] (2 responses)

Are you refering to RFC5424 [1] when you are affirming that syslog defines the format of structured log entries? Or are there more comprehensive syslog standarts wrt structuring?

AFAI understand Journal identifies and solves the following additional problems:

*
there's no standartised API for reading or writing log entries (so I'd guess that each application that wants to do structured logging has to reinvent how to implement that (escaping, ordering of entries, sub-entries dependencies etc.))

*
there's no standartization on what fields (key/values) are mandatory and what the individual fields mean

Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

Is my view of affairs correct?

Also you write:

> no need to figure out how to deal with logs that only get written to one of the two

As far as I can see, if both systems were set up to feed into each other, as suggested, there would be no such thing as "logs that only get written to one of the two"? So I think this is not a base for critique?

I can see the additional cost of running both systems. However they would also yield a gain from standartized interfaces and semantics and unambiguous usage, as offered by the Journal?

[1] http://datatracker.ietf.org/doc/rfc5424/?include_text=1

That newfangled Journal thing

Posted Nov 23, 2011 23:06 UTC (Wed) by dlang (guest, #313) [Link]

RFC5424 defines a standard syntax, but not the semantics

look at the CEE standards process for standardisation of what the individual fields are named and what they should mean.

the sort of thing the CEE is doing only helps in a small subset of the cases, by definition you can only standardise what people have already thought of, so if you need to say something different (because your system works differently, so the meaning is a bit different), then you can't use the standardised semantics, but you can still use the standardised syntax

That newfangled Journal thing

Posted Nov 23, 2011 23:10 UTC (Wed) by dlang (guest, #313) [Link]

> Which suggests that even if modern syslogs were used, parsing tools such as logwatch would still need to do a lot of (guess-)work (account for log writing bugs of the apps that produce the structured logs (due to varying implementations), changes in meaning of fields (as application implementor chooses), guessing meanings of fields of unknown apps etc.).

this is going to be the case no matter what structured format you use. there will always be programs that generate data that is almost, but not quite correct. In many cases such systems will get widely deployed before people start noticing (or caring ) that they are doing the wrong thing. In many cases the result is going to be learning to tolerate the wrong thing instead of refusing to interoperate with it.

This isn't just at the hardware level. Look at the things that Cisco has done 'differently' than any other vendor and you will see that the other vendors have had to adapt to Cisco or they got shut out entirely.

That newfangled Journal thing

Posted Nov 21, 2011 0:41 UTC (Mon) by HelloWorld (guest, #56129) [Link] (2 responses)

> and if someone does so, what is going to make the logs any more structured than the logs created by syslog(3) today?
You can't make something out of nothing.

That newfangled Journal thing

Posted Nov 21, 2011 1:17 UTC (Mon) by dlang (guest, #313) [Link] (1 responses)

I'm not understanding your comment. are you agreeing that the journal won't help enforce formatting? or are you claiming that syslog messages cannot ever be well formatted? (or something else)

That newfangled Journal thing

Posted Nov 21, 2011 3:23 UTC (Mon) by HelloWorld (guest, #56129) [Link]

The point I was trying to make is that you can't expect the journal to generate structured logging messages from programs which weren't written with structured logging in mind.

That newfangled Journal thing

Posted Nov 20, 2011 21:49 UTC (Sun) by PaulWay (guest, #45600) [Link]

I have to add my vote for this article - Jon, you clear an insightful path through the thicket of comments on this issue. Once again I am glad I pay for LWN. Keep up the great work.

Have fun,

Paul

Change... to what?

Posted Nov 20, 2011 22:32 UTC (Sun) by jmorris42 (guest, #2203) [Link] (47 responses)

As with everything it seems, we seem to have two schools of thought here.

On the one hand we have the UNIX people. I'd like to speak up for them here in the face of this "We must change or die" mantra developing.

We appear to be fairly conservative in thinking that after four decades we have found a lot of ideas that have stood the test of time and thus any proposed changes should be viewed with a skeptical eye. This is not a bad thing. What works should not be tossed aside lightly for new and shiny. And when the the proposals aren't just an incremental improvement but a total upending of not only current practice but of what we have accepted as core Truths we are especially skeptical. When they are put forth by a group of people who are on record as thinking that pretty much every one of the core UNIX ideas are rubbish... well our flamethrowers self ignite. And they should unless we have lost the belief our culture should survive. Because it should be clear that if we we lack the confidence to defend our traditions we won't keep them in the face of a determined assault.

We have seen what happens when the whole operating system merges into a couple of hopelessly tangled binary chunks that talk among themselves in obscure ways. Just how much of the OS is currently either in or being sucked into the black hole of systemd? Does ANYONE actually understand all of udev/dbus/*kit? Now anyone who didn't have to refer to the source code to figure parts of it out?

We believe in small individually replacable parts that do one task well, that are configured and communicate in human readable formats.

We believe we have seen the future Pottering & Co. are trying to drag us kicking and screaming to.. It is called Windows. Windows 7 doesn't even crash all the time any more, perhaps he and his followers should migrate back. If they believe in the virtues of Free Software, ReactOS could put their undisputed skills to use.

Personally I believe we should be open to change, just not to bad ideas we have already seen fail everywhere else they have been tried. But Plan9 should be our lodestar, not Windows, not OS X, not iOS, not Android. Plan9 might be carrying a few things beyond where current science (hardware and software) are practical, but when looking for ideas it is the inheritor of the UNIX tradition. OpenStep is probably still ripe territory to mine as well.

This Journal does not solve any actual problems, makes some things more difficult and moves us down the road to a towering mass of fail long term. None of the proposed benefits require a rip and replace, none require a binary log. They are offered as justifications for a preexisting goal, that of slowly centralizing all of the 'OS' part of userspace into systemd.

Change... to what?

Posted Nov 20, 2011 22:55 UTC (Sun) by dlang (guest, #313) [Link]

A key thing here is "incremental improvement", the improvement should also be optional (and not in the "use a different distribution" way of being optional)

This is good question...

Posted Nov 20, 2011 23:18 UTC (Sun) by khim (subscriber, #9252) [Link] (11 responses)

We believe in small individually replacable parts that do one task well, that are configured and communicate in human readable formats.

This is good belief, but sadly it failed "test of time".

But Plan9 should be our lodestar, not Windows, not OS X, not iOS, not Android.

This depends on your goals, I guess. Unix failed everywhere except on server. Plan9 just plain failed. Windows, OS X, iOS, Android... these things survived and currently thrive.

This Journal does not solve any actual problems, makes some things more difficult and moves us down the road to a towering mass of fail long term.

If your definition of fail is "something actually usable by mere mortals" then yes.

The problem with small individually replacable parts that do one task well, that are configured and communicate in human readable formats is that they are unusable without sysadmin. They work fine when there are competent sysadmin which can fix things when they break. And text formats are great boon for said sysadmin: s/he can tweak them in text editor, sed, awk, perl and eventually cobble up some kind of solution even in the unusual case.

When there are no such sysadmin text formats don't actually help anyone and are actually hurtful. Because in this case your only hope is self-healing system. And binary files and protocols can include as many crcs, signatures and built-in checks as you want. Sure, you can add them to text file as well, but what good it'll do? You still will need to use specialized tool to modify these files.

This is good question...

Posted Nov 20, 2011 23:41 UTC (Sun) by dlang (guest, #313) [Link] (9 responses)

>> We believe in small individually replacable parts that do one task well, that are configured and communicate in human readable formats

> This is good belief, but sadly it failed "test of time"

and here is the fundamental disagreement. Many of us think that such systems are the ones that work best, not only in an operating system, but in an network based application as well.

Software as a Service is this same mechanism across multiple companies

people keep deciding that making everything monolithic is the only way to do anything, but then as they need to scale or interact with new things, they get trounced by solutions that are more loosely coupled.

the loosely coupled componenets allow for things to be done in ways that were never imagined by the creators of the software.

Yes, this does take a system administrator to figure out initially, but that doesn't mean that the common things can't be preconfigured or autoconfigured.

Ubuntu became as large as it is because it implemented automatic configuration of the loosely coupled pieces, it didn't require them to scrap existing tools to make it easy for people to use without separate sysadmin support

Hmm... Interesting. Where is the list?

Posted Nov 20, 2011 23:55 UTC (Sun) by khim (subscriber, #9252) [Link] (8 responses)

people keep deciding that making everything monolithic is the only way to do anything, but then as they need to scale or interact with new things, they get trounced by solutions that are more loosely coupled.

Interesting. What user-facing computer system was "trounced" by a solution which is more loosely coupled?

Sure when components are coupled too tightly it can be a liability, but so far we face the opposite problem: way too many moving parts to keep track of.

Ubuntu became as large as it is because it implemented automatic configuration of the loosely coupled pieces, it didn't require them to scrap existing tools to make it easy for people to use without separate sysadmin support

Ubuntu is still small when compared with Windows or MacOS X and even before Unity they already had quite a lot of changes in the various paclkages to better integrate them. For example Ubuntu also invented replacement for sysv init :-)

Hmm... Interesting. Where is the list?

Posted Nov 21, 2011 0:12 UTC (Mon) by dlang (guest, #313) [Link] (7 responses)

> Interesting. What user-facing computer system was "trounced" by a solution which is more loosely coupled?

I think the ultimate example of this is how the web trounced all the tightly coupled communities that existed before it. (AOL, Compuserv, Protogy, etc)

This is great example...

Posted Nov 21, 2011 7:48 UTC (Mon) by khim (subscriber, #9252) [Link] (6 responses)

Well, Web shows both sides of the coin.

1. Web essentially killed the closed alternatives (AOL, Compuserv, MSN, etc).

2. Few years after that happened web itself was overrun by packages with binary-only configs or GUI-only configs (forums, blogs, social networks, etc).

If you'll think about it then you'l see it's natural succession: geeks started the web and in the early phase the ability to mix and match was the key to success. Later, "normal" people have come and they don't need text config and protocols so they were moved to backstage (where sysadmins can still tweak them but most of the population can not).

If Linux wants to conquer user-facing devices it needs to become more robust and thus less flexible (flexibility begets uncertainty and uncertainty is primary enemy of robustness). If it wants to keep server side then it needs to keep flexibility. This is interesting dilemma, but so far it looks like only a handful developers care about robustness thus I think it's premature to start campaign for "true UNIX way". There are enough distributions which (like Gentoo or Slackware) keep UNIX traditions alive...

This is great example...

Posted Nov 21, 2011 9:48 UTC (Mon) by aleXXX (subscriber, #2742) [Link] (5 responses)

AFAIK, even those GUI-config web tools still mostly write into a config.php file or something, which is still plain text.

Alex

Nope...

Posted Nov 21, 2011 12:19 UTC (Mon) by khim (subscriber, #9252) [Link] (4 responses)

Usually you have something like setup.php which writes textual config.php with username, password, etc. But it's just a few lines of remnants of "UNIX heritage". Usually you are supposed to delete setup.php after initial configuration and never touch config.php at all.

Other configs, all the logic, etc - everything goes to the insane mess of SQL tables or (if it's modern, AppEngine-based app) is stored as a blob in Datastore.

Nope...

Posted Nov 21, 2011 21:08 UTC (Mon) by raven667 (subscriber, #5198) [Link] (3 responses)

Well, there the reason that the webapp config is in a DB rather than text files is that this is easier and safer to read/write esp. when there may be multiple users changing settings at any given time. Sure you _could_ try and deal with multiple writers modifying config files while the app is continuously parsing and rereading the config to pick up changes but implementing all the custom parsing and locking is going to be a lot more difficult than writing a few simple select/insert/update/delete transactions.

Nope...

Posted Nov 22, 2011 0:45 UTC (Tue) by dashesy (guest, #74652) [Link] (2 responses)

Very true, even for a single user.
It remembers me of trying to synchronize history across multiple open gnome terminals in a way that they do not step on each others feet!

Nope...

Posted Nov 22, 2011 16:58 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (1 responses)

Share history among *running* terminals or make sure that all history gets to the histfile? I have zsh set up to do the latter (the shell's history gets appended to the global history; it doesn't overwrite the histfile with old history plus its history (which bash does)). It's just "setopt appendhistory".

Nope...

Posted Dec 1, 2011 12:54 UTC (Thu) by nye (subscriber, #51576) [Link]

>it doesn't overwrite the histfile with old history plus its history (which bash does)).

$ shopt -s histappend

This is good question...

Posted Nov 20, 2011 23:49 UTC (Sun) by butlerm (subscriber, #13312) [Link]

>When there are no such sysadmin text formats don't actually help anyone and are actually hurtful.

No doubt there is a market for Linux servers that hold your hand and do not require any particular experience to administer. That niche is already almost completely filled by Windows server operating systems.

But if you did that to most existing Linux server setups, what you would end up would be something that is every bit as crippled as Windows server setups typically are, as in the simple things are simple, and the hard things are impossible. Can you imagine the user interface required to completely configure everything Apache can do, for example? Or Bind? or Exim? or practically anything else that does anything serious?

Without hundreds of millions of dollars in GUI configuration tool investment and automation interfaces to work around them, a text configuration file free Linux server distribution would either be a toy, or something that targets a relatively narrow market. For desktops though, it is probably great. Maybe for SOHOs as well. But not for practically everywhere where UNIX type servers dominate today.

Change... to what?

Posted Nov 20, 2011 23:55 UTC (Sun) by HelloWorld (guest, #56129) [Link] (32 responses)

We appear to be fairly conservative in thinking that after four decades we have found a lot of ideas that have stood the test of time and thus any proposed changes should be viewed with a skeptical eye. This is not a bad thing.
Actually, looking sceptically at things just because they're new is a very bad thing. It's about as resistant to change as it gets.
We have seen what happens when the whole operating system merges into a couple of hopelessly tangled binary chunks that talk among themselves in obscure ways. Just how much of the OS is currently either in or being sucked into the black hole of systemd? Does ANYONE actually understand all of udev/dbus/*kit? Now anyone who didn't have to refer to the source code to figure parts of it out?
People nowadays expect stuff to be integrated; it's not like the components of a modern Linux system talk to each other just for the fun of it. And what is obscure about DBus? One can monitor DBus IPC just fine with tools such as dbus-monitor, one can explore the IPC interfaces offered by the running programs using qdbusviewer. For the first time in the history of Unix, there's an IPC mechanism that works on a higher level of abstraction than Unix sockets and is actually used by significant number of people, and that is one of the reasons why proper integration of the various OS components has even become feasible. Yet, you have nothing else to do that whine about how "obscure" and "tangled" this is.
We believe in small individually replacable parts that do one task well, that are configured and communicate in human readable formats.
Why would I care about the format my programs use to communicate? Those are implementation details. What's would be the point in making DBus IPC unnecessarily inefficient just to make it human-readable, when you'll want to use proper tools to interact with it anyway?

Change... to what?

Posted Nov 21, 2011 0:07 UTC (Mon) by dlang (guest, #313) [Link] (30 responses)

> Why would I care about the format my programs use to communicate?

you don't care, until you need to either debug things or add your own component somewhere in the middle, at those points how the components communicate is critical.

Change... to what?

Posted Nov 21, 2011 0:23 UTC (Mon) by HelloWorld (guest, #56129) [Link] (29 responses)

> you don't care, until you need to either debug things or add your own component somewhere in the middle, at those points how the components communicate is critical.
Those are pretty lame excuses. If I want to add my own component I'll use a library anyway, and to debug things one will want to use specialised tools like dbus-monitor anyway.

Change... to what?

Posted Nov 21, 2011 2:42 UTC (Mon) by jmorris42 (guest, #2203) [Link] (28 responses)

> If I want to add my own component I'll use a library anyway, and
> to debug things one will want to use specialised tools like
> dbus-monitor anyway.

In other words, you don't value human readable or everything is a file. An API is all you care about. So what are doing using a system designed around everything you think is stupid? 90% of the world is using systems build around that idea, yet you are trying to bring them to the final few percent of holdouts.

Change... to what?

Posted Nov 21, 2011 4:31 UTC (Mon) by HelloWorld (guest, #56129) [Link] (25 responses)

> In other words, you don't value human readable
In the context of an IPC system, I simply don't care. It doesn't offer any advantages and only makes things slower, so why bother with a human-readable protocol? Can you name even *one* sensible reason?

As for "everything is a file", I didn't comment on that one.

> So what are doing using a system designed around everything you think is stupid?
There's no better alternative in the free software world.

Change... to what?

Posted Nov 21, 2011 4:55 UTC (Mon) by wahern (subscriber, #37304) [Link]

If human readable protocols didn't have value than ASN.1 would have ruled the world. Instead we have a plethora of far more clumsy human readable alternatives.

Change... to what?

Posted Nov 21, 2011 13:35 UTC (Mon) by sorpigal (subscriber, #36106) [Link]

I think this thread is extremely important because it starkly shows just what it is that we're discussing and what's at stake. Some people who don't like Unix and don't believe it's good to be Unix are trying to incrementally make Linux into something very different.

If you advertise up front that you want to use the Linux kernel to build a a system that isn't Unix or Unix-like then perhaps proposals that move things in that direction would not be met with such debate. Instead your entire goal could be examined and we could split into the camp which wants that and the camp which doesn't and everyone could move on.

Not advertising up front what the real goal is makes all discussion after that meaningless.

Change... to what?

Posted Nov 21, 2011 18:38 UTC (Mon) by jmorris42 (guest, #2203) [Link] (22 responses)

> There's no better alternative in the free software world.

So why aren't you trying to change that? GNU has the goal of remaking a Free UNIX. ReactOS has the goal of remaking a Free Windows. You could also be making a designed from the ground up OS with your ideas, you could even base it on the Linux kernel to leverage its wide hardware support. See Android for just how different a userspace can be and run on the Linux kernel.

Seems to me that if you disagree with pretty much all of the underlying design decisions of UNIX that beginning with a base more to your liking would be more productive that trying to remake Linux/GNU/X into some alien abortion.

It is about time for a fork; to drive the heretics from the temple. To drive a stake in the ground and say here we take a stand; NO MORE. The Windows refugees who want 'a Windows that doesn't suck.' can go forth and do that. Elsewhere. If that means we never get 'the year of Linux on the Desktop.' we have to live with that consequence. We have made it twenty years waiting for that miracle and managed to thrive, if it never comes it isn't a terrible thing. Maybe UNIX just isn't for everyone, but for those who DO love it we would kinda like to keep it something we can recognize.

And people call Lannart arrogant???

Posted Nov 21, 2011 19:21 UTC (Mon) by khim (subscriber, #9252) [Link] (11 responses)

Wow. Just... wow. You really, seriously, think that you have the right to drive the heretics from the temple?

This is free world and free software. If you don't want to use Lennart's creations and you don't want to pay for his work - you are free to do so (albeit I doubt you pay RedHat for anything anyway).

But to say that someone should stop doing what they are doing just because they crossed some imaginable line... wow. This is arrogance beyond belief.

If you want to keep UNIX culture and UNIX-like distribution (or distributions) alive - you are equally free to do so. But please stop proclaiming that you somehow have right to send people elsewhere. RedHat will deicide if it wants to continue to support Lennart's work. People will deicide if they want to support RedHat which supports Lennart. But if you want to drive a stake in the ground then you should find a place where people will allow you to do that.

Slackware, Debian, whatever. Find the place where people will be happy to live in the past and will be Ok with supporting "the old ways of doing things". If the froth in this thread is any indication you should be easily able to find supporters and live "happily ever after". But why people who don't share you obsession with "Unix way" should suddenly "move away" id they are Ok with RedHat, Lennart, etc?

Actually I suspect I know why all the froth happens: you understand that in reality "Unix way lovers" is very small (if vocal) group and without people who are doing the real work you'll soon have a system which is only usable on a small subset of hardware and with pretty limited choice of software. Well... this is the price you should pay if you really want to fight for your ideals.

And people call Lannart arrogant???

Posted Nov 21, 2011 22:40 UTC (Mon) by nix (subscriber, #2304) [Link]

You really, seriously, think that you have the right to drive the heretics from the temple?
Nah, he can just lock 'em out with SELinux :)

And people call Lannart arrogant???

Posted Nov 22, 2011 1:19 UTC (Tue) by jmorris42 (guest, #2203) [Link] (2 responses)

> Slackware, Debian, whatever.

Yes. That is exactly what I'm asking. At this point though it is very hard to figure out which, if any, of the major distributions plan on remaining recognizable as UNIX descendants. Which makes it kinda hard to pick one. What I'm saying is it is time for distributions to be honest with us poor users and tell us what the hell they are doing. Are they chasing The Year of Linux on the Desktop to the exclusion of the users who have been and currently ARE using their system? Have they even thought about it or are they just caught up in the daily grind of packaging up the same tarballs the other distos are so they show the same version number on distrowatch?

Ubuntu has pretty much told us they are. And we will probably know for sure about RedHat if they actually ship the controversial bits of Fedora as RHEL 7.0. (And if the paying customers eat the dog food, guess it was a good business decision and best 'o luck to em.) But that leaves questions. What of Debian? Do they just pull whatever upline throws them and follow along or what? Does this go unopposed until even Slackware is unrecognizable? It is clear the Windows refugees know exactly what they want and have the numbers to impose their will upon us, so if we don't speak up, and pretty quickly, I suspect there won't be many options other than *BSD, and that their taunt will have come true. That BSD is for people who like UNIX and Linux is for those who don't like Windows.

Maybe we end up having to port BSD to the Linux kernel... Sort of the reverse of Debian/FreeBSD.

And people call Lannart arrogant???

Posted Nov 22, 2011 2:40 UTC (Tue) by mpr22 (subscriber, #60784) [Link]

As far as I know, Slackware has explicitly said "no" to systemd.

And people call Lannart arrogant???

Posted Nov 22, 2011 6:06 UTC (Tue) by jrn (subscriber, #64214) [Link]

> What of Debian? Do they just pull whatever upline throws them and follow along or what?

You can see an approximate preview of the next Debian release by looking at what "unstable" does today (just like you can see a preview of the next Fedora release by running rawhide). If you don't like what you see, you can even participate to improve it by sending clear bug reports and patches.

Suppose you disagree at a fundamental level with an operating system developer --- for example, let's say you love KDE 3 and are upset that the KDE maintainers moved on to KDE 4. Even then, as long as you are doing the work, you can add packages that work the way you want in parallel (for example, you could package Trinity). If your use case requires changing someone else's package in a way that the maintainer does not want, then the technical committee[1] may listen to your arguments, weigh them (which involves, implicitly, weighing the impact on users and the amount of work you have done to bring your proposal about), and come up with a way to make it happen without sacrificing too much.

I imagine most distros work somewhat like this.

[1] http://www.debian.org/devel/tech-ctte

And people call Lannart arrogant???

Posted Nov 23, 2011 23:12 UTC (Wed) by dlang (guest, #313) [Link] (6 responses)

> Wow. Just... wow. You really, seriously, think that you have the right to drive the heretics from the temple?

why not? Lennert thinks that he does, the only difference is the definition of 'heretic'

And people call Lannart arrogant???

Posted Nov 24, 2011 8:44 UTC (Thu) by anselm (subscriber, #2796) [Link] (1 responses)

The difference is that Lennart's (I think you should take the trouble to spell his name correctly, the guy deserves at least that much respect) approach is backwards-compatible with syslog – it is fairly trivial to gate syslog messages into »the Journal« – while the Unix traditionalist approach is basically to deny innovation in the syslog space.

It is apparently acceptable to tweak the actual implementation of the syslog daemon, but it would be just as well to acknowledge that the underlying protocol sucks and could use improvement.

And people call Lannart arrogant???

Posted Dec 1, 2011 13:02 UTC (Thu) by nye (subscriber, #51576) [Link]

>the Unix traditionalist approach is basically to deny innovation in the syslog space

That really isn't true. The people I've seen in this thread who are unhappy with journald haven't been saying that everything should stay as it is; they've been saying that evolutionary changes would be better than revolutionary changes, and providing several reasons for that which, on the face of it, seem sane.

Saying "we'd like to improve this carefully rather than starting from scratch" is a far cry from 'denying innovation'.

Well, there are big difference...

Posted Nov 24, 2011 10:32 UTC (Thu) by khim (subscriber, #9252) [Link] (3 responses)

why not? Lennert thinks that he does, the only difference is the definition of 'heretic'

No. The main difference is that Lennart has the mandate to introduce this (and other) changes. Fedora has well-defined procedure where such changes are discussed and accepted and rejected - and so far it was pretty sympathetic to the Lennert's creations. They were considered on their technical merits and accepted or pushed away till they'll be ready. They also have the option to reject proposed changes completely (even it was never implemented so far).

This is normal technical process: we all want to make things better but of course different people differ in the definition of "better". What jmorris42 offers is totally different: we should create some kind of sacred tome which will define out religion and reject everything which does not conform to it. No matter if it's good or bad - if it's sacred tome it's wrong and not allowed.

If you want to worship someone or something - it's your choice, but please do it in your sect, in your temple, don't try to push this on other people.

I understand why jmorris42 does not like to do it this way: world is changing. Just because you refuse to change with it does not mean it'll comply with your wishes and will keep the option to use old code open: *BSD fell in this trap - they are mostly unusable today not because they have some inborn weaknesses but because companies decided they don't care and so it's hard to find hardware to run them on!

It's interesting that Linux itself is in danger of going the same way in some cases: people from Palm complain that when they contact subcontractors and ask about Linux support they are redirected to people who ask which version of Android they plan to use!

But if you fear that this will be the result then you have only two choices actually:
1. Accept changes and "go with the flow".
2. Roll up your sleeves and try to keep your "UNIX-style system" alive (like *BSD people are doing).

Screams "heretics, heretics" are not really a good answer because Linux world has no bishops which can hear you and if someone will try to instill such institution I doubt a lot of people will accept that.

Well, there are big difference...

Posted Nov 24, 2011 11:26 UTC (Thu) by dlang (guest, #313) [Link] (2 responses)

you define heretic as anyone unwilling to accept the latest proposal

others define heretic as people eager to change the system in ways that are not incramental changes, but are instead "throw out what's there and write a replacement, the replacement doesn't need to do everything the old system did because the people who used those features aren't important"

changes that are proposed that are really optional (and not just "pick a different distro if you disagree" optional), and can be ignored if the admin wants to get relatively little opposition (they may get some grumbling if it takes work to figure out how to turn off the new features, but nowhere near the opposition that these re-writes from scratch get)

Well, there are big difference...

Posted Nov 24, 2011 11:34 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

by the way, a note on the number of users and the "most machines don't need this feature argument"

If you count by the number of people at a keyboard of a machine you may be right.

but when you start considering server farms where the admin to system ratio is commonly well over 100 servers per admin, you would get very different numbers.

embedded systems are in many ways far more like servers than desktops. They have skilled admins managing the software and all the non-trivial configurations on the devices (either at the factory, or through 'firmware updates'

'winning the desktop' while loosing servers and embedded devices would not be a overall win.

Well, there are big difference...

Posted Nov 24, 2011 14:23 UTC (Thu) by khim (subscriber, #9252) [Link]

but when you start considering server farms where the admin to system ratio is commonly well over 100 servers per admin, you would get very different numbers.

Sure. For example you can get... zero. We have totally different systems on desktop and on servers here (they only share few common packages like glibc and libstdc++ to simplify testing). It does not matter at all what the servers are using because desktops are using totally different Linux. I'm pretty sure we are not alone.

embedded systems are in many ways far more like servers than desktops. They have skilled admins managing the software and all the non-trivial configurations on the devices (either at the factory, or through 'firmware updates'

Again: there are many different types of embedded systems and they use different packages. What they usually don't use are desktop-based distributions.

'winning the desktop' while loosing servers and embedded devices would not be a overall win.

Why not? Apple ceased to produce their rack-mountable servers and they use quite different set of programs on their iPad and Macbook - and still is fine AFAICS.

Sure, we should not forget servers, routers and other Linux-based machinery, but I fail to see why the desktop-oriented distribution should be driven by the needs of server or an embedded system.

I think this one thing which Ubuntu did right (and Fedora followed): distribution's work is selection first and foremost. It must work fine in the default install. If you want to serve a different niche - you can create a spin-off. Journald does not propose to remove support of syslogd for the time being and I'm pretty sure such support will only be considered in the future if Journald will be good enough for most uses, not just for desktop.

As for "who's the most numerous"... it's silly question: RedHat's accounting department knows the numbers, I'm pretty sure needs of the most important customers will be taken into account when the fate of journald will be determined.

Change... to what?

Posted Nov 21, 2011 22:58 UTC (Mon) by HelloWorld (guest, #56129) [Link]

> So why aren't you trying to change that?
Well, Lennart seems to be doing a good job at creating a better alternative to current systems, so why bother?

> GNU has the goal of remaking a Free UNIX.
That doesn't mean everything in Unix is perfect and must never be changed. In fact, GNU has made several design decisions that are different from SysV or BSD, for example they chose a microkernel design for the Hurd.

> Seems to me that if you disagree with pretty much all of the underlying design decisions of UNIX
I don't. I didn't comment on the idea of "everything is a file" at all, and I don't oppose the idea of a textual format in general. I'm just not making a religion out of it like you do. Now, I'm asking you again: what advantage would a text-based IPC/RPC protocol actually buy you, so that making it more inefficient than it needs to be is worth it?
And it's the same with the journal. I'm pretty sure it'll offer tools to dump logs in a textual format, so that you can use all the Unix tools you like with it. So, what's wrong with using a binary format when it has the potential to be more compact and much more efficient to query than a textual format?

> It is about time for a fork; to drive the heretics from the temple. To drive a stake in the ground and say here we take a stand; NO MORE. The Windows refugees who want 'a Windows that doesn't suck.' can go forth and do that. Elsewhere. If that means we never get 'the year of Linux on the Desktop.' we have to live with that consequence. We have made it twenty years waiting for that miracle and managed to thrive, if it never comes it isn't a terrible thing. Maybe UNIX just isn't for everyone, but for those who DO love it we would kinda like to keep it something we can recognize.
Dude, you have serious issues. Unix is a technology, it's not a religion. Get help.

Change... to what?

Posted Nov 21, 2011 23:05 UTC (Mon) by anselm (subscriber, #2796) [Link] (8 responses)

Who went away and put you in charge of deciding what is Linux and what isn't? There is no reason why we must stick forever with the arbitrary design decisions of the 1970s and 1980s. Lots of aspects of Linux are worth re-thinking. This does not mean that the first alternative proposal for »whatever« must be accepted by everyone, but also that there are no holy cows. (And that you are free to maintain Holy Cow Linux, with none of the newfangled improvements, for however long you care to do so.)

If somebody (not necessarily Lennart Poettering) comes up with a proposal to improve a notoriously badly-designed part of the system such as the logging infrastructure, then that person should at least be heard out. It may just be that they have something good going. That only Lennart and Kay seem to have the cojones to actually tackle some of these things should really give us pause.

In other words: Change is not necessarily for the better but it is also not necessarily for the worse. Biology has a word for things that never change. They're called »dead«.

Change... to what?

Posted Nov 23, 2011 23:16 UTC (Wed) by dlang (guest, #313) [Link] (7 responses)

when you say that syslog is so horribly designed and so limited it makes me think that you haven't used a modern syslog (rsyslog or syslog-ng). Those are both good examples of projects that have added a lot of capabilities to logging, but have done so in ways that are compatible with the existing system while providing the additional power to those who want/need it

this journal proposal sounds like the dozens of logging systems I've seen proposed by junior programmers for their applications. They assume that they know all the needs and so can design the perfect monolithic logging infrastructure.

to paraphrase a common quote

"syslog is the worst logging design ever, except for everything else"

Change... to what?

Posted Nov 25, 2011 4:33 UTC (Fri) by HelloWorld (guest, #56129) [Link] (6 responses)

> when you say that syslog is so horribly designed and so limited it makes me think that you haven't used a modern syslog (rsyslog or syslog-ng). Those are both good examples of projects that have added a lot of capabilities to logging, but have done so in ways that are compatible with the existing system while providing the additional power to those who want/need it
The Journal _is_ compatible with the syslog(3) api. And while it doesn't implement the syslog protocol, it is easily possible to write a gateway tool that implements the protocol and forwards messages to the Journal in order to be stored; in fact this possibility is mentioned in the proposal.
It would be really helpful if people would actually _read_ such proposals before whining about them.

Change... to what?

Posted Nov 25, 2011 5:34 UTC (Fri) by dlang (guest, #313) [Link] (5 responses)

I have read the proposal

It's "compatible" with syslog in that if you are willing to throw out it's advantages of being structured and trusted you can encapsulate syslog data in journal messages.

but if you have done this, what exactly is the advantage of this new tool?

and if you can go the other way and have the journal entries put into nicely formatted structured syslog messages, why do you need the journal to do this again instead of just writing the nicely formatted messages to syslog in the first place?

Especially if you like writing your log entries via a library, there is absolutely nothing preventing you from using a library that writes structured messages to syslog.

Change... to what?

Posted Nov 25, 2011 9:18 UTC (Fri) by anselm (subscriber, #2796) [Link] (3 responses)

Especially if you like writing your log entries via a library, there is absolutely nothing preventing you from using a library that writes structured messages to syslog.

Except that clients will still be able to lie to the syslog service about their identity, etc. Structured messages are one thing that The Journal is about, but Lennart and Kay do address a fair number of other concerns. Some of these might be amenable to changes to rsyslogd, but others are more difficult to treat within that framework while Lennart and Kay do seem to have a handle on them. I think their proposal at least bears thinking about.

Change... to what?

Posted Nov 25, 2011 9:25 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

>> Especially if you like writing your log entries via a library, there is absolutely nothing preventing you from using a library that writes structured messages to syslog.

> Except that clients will still be able to lie to the syslog service about their identity, etc. Structured messages are one thing that The Journal is about, but Lennart and Kay do address a fair number of other concerns. Some of these might be amenable to changes to rsyslogd, but others are more difficult to treat within that framework while Lennart and Kay do seem to have a handle on them. I think their proposal at least bears thinking about.

actually, journald is only able to avoid the application lieing to it for programs that are started by systemd and do all their logging through systemd.

This could be done with normal syslog by creating a unique /dev/log replacement for each application group that's started and then tagging to indicate which one the log arrives on (modern syslog daemons already allow you to have multiple sources and to know which source the log came from)

If anything else generates logs to journald in any way that bypasses the library, that thing could also lie about who it is. good programs won't do this, but good programs won't lie about who they are in the first place (unless they have a good reason to do so, which is very possible)

Change... to what?

Posted Nov 25, 2011 9:58 UTC (Fri) by michich (guest, #17902) [Link] (1 responses)

It does not matter what library you use to talk to journald. It is not possible to lie about who you are. Search for SCM_CREDENTIALS in the proposal. See also SCM_CGROUPS, SCM_COMM in http://0pointer.de/blog/projects/plumbers-wishlist.html

rsyslog now has 'trusted properties'

Posted Dec 1, 2011 3:43 UTC (Thu) by dlang (guest, #313) [Link]

it turns out that rsyslog already used SCM_CREDENTIALS to let you log the pid of the program, it's now added the ability to log some additional values

_UID The userid under which the logging process is being executed.
_GID The group id under which the logging process is being executed.
_PID The PID of the logging process. Note that this PID, if configured, is also put into the syslog tag.
_EXE Path to the binary that is logging
_COMM The name (as visible by top) of the logging process.
_EXE The full command line of the logging process. Note that this string can contain spaces, thus it is always provided in quoted form.

http://blog.gerhards.net/2011/11/trusted-properties-in-rs...

http://www.rsyslog.com/what-are-trusted-properties/

now that didn't take scrapping all existing logging practice to get did it?

Change... to what?

Posted Nov 25, 2011 9:20 UTC (Fri) by michich (guest, #17902) [Link]

Even with the syslog(3) api you still get some of the benefits: addition of the underscore-prefixed fields, indexing by these fields, per-user read access to their own logs.

Change... to what?

Posted Nov 21, 2011 4:35 UTC (Mon) by HelloWorld (guest, #56129) [Link] (1 responses)

Oh, by the way: you may want to check out something from ESR's book about the Unix philosophy:
http://catb.org/~esr/writings/taoup/html/ch01s06.html#id2...

"Rule of Diversity: Distrust all claims for 'one true way'."

I guess that applies to text formats and protocols as well.

Change... to what?

Posted Nov 26, 2011 16:39 UTC (Sat) by cdmiller (guest, #2813) [Link]

Indeed, it also applies to all the recent new and shiny proposals being touted as the 'one true way'...

Change... to what?

Posted Nov 21, 2011 6:23 UTC (Mon) by dvdeug (subscriber, #10998) [Link]

Looking sceptically at things because they're new is hardly as resistant to change as it gets. You're looking at things, you're open to change, you're just not going to jump on just anything. Considering that esd worked for me for almost ten years, and Pulseaudio isn't working nearly as well, I've got to say I like that idea.

Change... to what?

Posted Nov 21, 2011 0:57 UTC (Mon) by bferrell (subscriber, #624) [Link]

What he said!!!

That newfangled Journal thing

Posted Nov 20, 2011 23:51 UTC (Sun) by dlazaro (guest, #38702) [Link] (1 responses)

You don't seem to understand. Syslog is a critical piece of infrastructure. If Red Hat wants to go with that, fine. They can integrate it in the system. But, please, do not force it as the only solution. Just follow a standards track to mess nicely with the rest of the networking gear.

Plus other Lennart inventions have resulted in lots of headaches for me so I mistrust him completely.

That newfangled Journal thing

Posted Nov 21, 2011 0:38 UTC (Mon) by HelloWorld (guest, #56129) [Link]

> Plus other Lennart inventions have resulted in lots of headaches for me
Could you elaborate this further?

That newfangled Journal thing

Posted Nov 21, 2011 0:03 UTC (Mon) by hjh (guest, #4352) [Link] (5 responses)

The biggest problem I see from the comments and the proposals and what is understandably irritating people seems to be the unnecessary monolitic design, with all the resulting problems of:

* no way to generate pure text files
* no network service
* undocumented binary files
* no way to interface, but api calls
* not UNIX style

One obvious separation into a frontend, for receiving, annotating and generating log records, integrated into systemd, and a configurable back end would make good on the biggest concerns mentioned over and over.

The proposed storage design could still be the default backend and might still be integrated in systemd. But by additionally providing a pipe with a general text based standard record format to other external backends, the whole idea would become much more traditional UNIX style and less irritating, because it wouldn't force unnecessary policy on users.

One can envision: live monitoring, network transport, database, null, filter, live monitoring, alarm trigger, statistic and many other backends.

I can not belief that this would be much of a burden for the developers.

The second, often mentioned concern is missing speaking message ids. I can clearly see the benefits of UUIDs as well as those of speaking ids like reverse domain ids. But I think this could be solved, by allowing speaking ids in addition to required UUIDs as labels.

The third concern, security design, probably can be improved by some additional administrative measures, but the basic idea seem sane and is much better than no security at all. Security is a very tricky area and needs experts to comment, which I am not.

Despite those solvable problems, the journal proposal looks very very promising and has many valuable advantages which shouldn't be overlooked and the the argument we managed without is plain ..
..emmm... unconvincing.

That newfangled Journal thing

Posted Nov 21, 2011 9:04 UTC (Mon) by nim-nim (subscriber, #34454) [Link] (1 responses)

> One obvious separation into a frontend, for receiving, annotating and
> generating log records, integrated into systemd, and a configurable back
> end would make good on the biggest concerns mentioned over and over.

This has been tried with gconf and didn't work too well.

The developpers wrote(and rewrote) the frontend so it was fast and easy to integrate in apps (their pet peeve) and the user-friendly backend that anyone could use without reading the source code never materialised.

Developpers don't want to worry about the backend, if you let them sell you an unfinished backend (we'll do better later, promise) you'll never see the replacement before the app dies.

That newfangled Journal thing

Posted Nov 21, 2011 11:56 UTC (Mon) by hjh (guest, #4352) [Link]

> The developers wrote(and rewrote) the frontend so it was fast and easy
> to integrate in apps (their pet peeve) and the user-friendly backend that
> anyone could use without reading the source code never materialised.

not if there is a simple documented streaming interface, to the backends that could even just as well stream into a file. Somthing with log messages like:

#time: <date> #id: org.kernel.subsys.facility.xyz #...
#uuid:<UUID> #other: <other> #future1: <future1>
_ mesg_line_1
_ mesg_lin_e2
_ ...
_ mesg_line_n

#time:<date> # .....
...

such a format should be possible to commit to for developers as it's expandable with new records elements in the future, and as it does not require to commit to more complicated backend storage design. Binaries like core dump shouldn't go to the log anyway, only as references to a file location. Admins concerned can opt out of the provided backend and
use files or a database instead to store the messages with some more
storage space required. I even think that using a database for indexing
purpose is the better solution, than yet another special database for logging only to better with compression.

The best would be, to separate the backend into a second process, which supports the compression, hashing and indexing stuff. The second process could then easily be located on a different host, and gets the records synchronously or in batch mode.

That newfangled Journal thing

Posted Jan 10, 2012 11:13 UTC (Tue) by spongy (guest, #59953) [Link]

you forgot a big bullet: no documentation.

That newfangled Journal thing

Posted Dec 6, 2013 8:49 UTC (Fri) by daniel (guest, #3181) [Link] (1 responses)

Let me tag onto your list of deficiences with a big, read, block letters, flashing, bold warning: might prove irresistible for developers to use log messages for messaging instead of their true purpose of error logging and audit. Which would be a fine recipe for a perfrect mess.

That newfangled Journal thing

Posted Dec 6, 2013 13:42 UTC (Fri) by mathstuf (subscriber, #69389) [Link]

With all the filtering power journalctl has, wouldn't finding the errors be pretty much the same with and without lots of logging? Hell, you even get colors so you can just look for red and yellow messages. Sometimes sub-INFO-level debugging is useful; I don't see why supporting or encouraging it would be a downside.

Not too worried, as long as there is "fallback" mode

Posted Nov 21, 2011 0:21 UTC (Mon) by bojan (subscriber, #14302) [Link]

The FAQ of the journal doohickey says that one can continue running syslog right next to the binary stuff. Given that, I'm personally not overly worried. Disk is cheap.

Thus far systemd has mostly worked for me and as a concept it makes sense. Maybe I'm being naive here, but that's my limited experience with it.

PS. What I don't want is a Linux system where components are not interchangeable. I think this is precisely why Linux desktop environments are doing it wrong (in some areas). Too much integration, not enough communication. Sure, integration without communication is great until the time you want that one thing ripped out and replaced with something else. Then it totally sucks.

That newfangled Journal thing

Posted Nov 21, 2011 0:41 UTC (Mon) by bferrell (subscriber, #624) [Link] (36 responses)

ummm... I'm unclear corbet, what *exactly* makes a proposal that, at the outset, states it *will* break things worthy of serious consideration?

As another poster mentioned, to date, this individual hasn't made one bit of code that makes for a significant improvement in system operation and HAS created significant amounts of chaos.

As to ' ...haphazard and lame and sad doesn't just apply to kernel messaging; it describes the approach taken by Linux in general', I most respectfully suggest you have a look at error messages in:

Windows (any version)
Mac OS X (can someone please find me documentation for the security controller on OS X Server)
or another bit of code written by more than one person intended to be used by another.

I suspect that until God and/or the gods themselves come down from on high and decree under pain of death that some string format be used, there's going to be a certain amount of "differing" ideas about what is correct. (no, I'm not threatening anyone)

The absolute arrogance of these two (this time) is, in my opinion, rivaled only by Hans Reiser.

systemd, being a whole other topic, I'm still trying to fix. It's usefulness has been overshadowed but a certain binary, all or nothing installation style in my favorite distro. That style has certainly broken quite a bit and forced me to engage in a rather time consuming re-installation. Is that a fault of systemd? No, not really, but it does leave a very bad taste in my mouth and enhances my dislike for code proposed and written by Mr Poettering (and company).

Oftimes we hear of OSS developers wha say that they work on what they like it we should to it ourselves, to paraphrase. The fact of the matter is Lennart is paid to do this work and go to things like the kernel summit and his employers seem to like what he does else they would not... Subsidize is wrong, but employ would seem to imply direction by the entity who pays him, but I digress. The fact remains this is NOT hobby work that Lennart is doing and we, as a community need to recognize the chaos his work/idea/thinking is causing, has caused in the past and is proposing now when we say it deserves serious consideration.

All of the forgoing said, I recognize from long experience that I can't sell ice cubes in the desert and I firmly suspect that all I have written here WILL fall on deaf ears. But I did say it and I STILL prefer my much shorter, MUCH more sarcastic comment from before for which I will NOT apologize.

I'm done now

That newfangled Journal thing

Posted Nov 21, 2011 1:56 UTC (Mon) by corbet (editor, #1) [Link] (8 responses)

I'm curious: what will be broken? Syslog will run as it always has.

The fact that other operating systems don't do well with messaging doesn't mean we shouldn't try to do better.

I still have not advocated for the Journal's adoption. But I remain glad that people are willing to take a stab at improving things in this area.

That newfangled Journal thing

Posted Nov 21, 2011 2:35 UTC (Mon) by ebiederm (subscriber, #35028) [Link]

As described journald will be a mandatory code wart on the side of systemd.

As for what will be broken. I expect a lot will be broken in the strongly suggested patch to change all of the user space programs to ad uuids into their log messages. The change is to huge and too vast to prevent tupos from slipping through.

Furthermore their is no indication that such a change will benefit anyone in practice.

In their faq they have already deprecated syslog(3).

Better log messages are mostly a matter of getting the human factors right. I do not see journald doing anything but adding yet another ABI to user space that will have to reasonably be supported forever but that adds nothing of substance, and confuses the issue about how to do a good job of logging messages.

The design appears to have a very high cost to even experiment with, and he design is clearly not thought all of the way through at this point. I see any depolyment of journald as described as achieving exactly the opposite of what it hopes to achieve. That is I see deploying journald as creating an even bigger mess in the logging space then what already exists.

That newfangled Journal thing

Posted Nov 21, 2011 2:37 UTC (Mon) by bferrell (subscriber, #624) [Link] (6 responses)

... Wait, did you REALLY just say, in effect, "what could possibly go wrong?"

Others my disagree with me, but by presenting this not once, but twice and in the second instance presenting with a special note of the ideas worthiness, you are in fact advocating for it.

I'm always happy to see discussion of ideas, but the *stuff* that has been coming out of late is sorely lacking in wide discussion. I kind of like the "build it, and they will come" model. Build it, let me play with it and see if it fills a need. Don't write a position paper and tell me I don't know what I'm doing and you have a better idea.

As another poster put it, I'm insulted. Add that to a certain, less than stellar track record...

My question is why is this getting so much "air play"?

That newfangled Journal thing

Posted Nov 21, 2011 3:57 UTC (Mon) by vonbrand (subscriber, #4458) [Link] (3 responses)

Because the way of discussing groundbreaking new designs is talking it over in a small(ish) group, look over what the rest of the world has done, implement a prototype and write up the design rationale for the rest of the world to comment on and check out the prototype?

As for Poettering's previous "crimes," they are almost universally used in Linux distributions today, so his crowd must be doing something right, even while you believe it is utter crap... particularly systemd has made my Fedora machines boot up noticeably faster. Sure, radical changes have a large cost, and early versions will probably have severe drawbacks until the design and implementations's wrinkles are ironed out.

That newfangled Journal thing

Posted Nov 21, 2011 5:16 UTC (Mon) by bferrell (subscriber, #624) [Link] (2 responses)

At one time, the whole of the population knew the world was flat and that was accepted... Because it was the majority and widely believed.

At other times heinous things have been done because everyone "knew" it was right for various reasons.

As mama said, "if everyone else jumps off a bridge are you going to do it too?"

On my system (XPS1730, quad core and 4G ram), there is no noticeble improvement in my Suse system boot time. I have no idea what effect system spec have on it. But I mention them for completeness.

I, and many people I know, automatically disable pulse... It's there, unused and more trouble that it's worth. Avahi, asi I said, interesting but adds little value that I can see except to make Linux more Mac Like.

Tell you what, next week, we're going to outlaw gasoline and you have to use this new fuel. Eventually they'll work out the bugs, but in the mean time, sometimes car engines are going to blow up and have to be re-built. But when it's done right, it'll be really cool. How do you feel about that?

Look, I've already invested way to much time in this discussion. I recognize that I have little no influence in this. You do and as a wise man once said, never pick a fight with someone who buys ink by the barrell. You have the barrells and an agenda.

I'm out.

Flat Earth Myth

Posted Nov 21, 2011 14:20 UTC (Mon) by tialaramex (subscriber, #21167) [Link] (1 responses)

“At one time, the whole of the population knew the world was flat and that was accepted... Because it was the majority and widely believed.”

Yeah, no. Ancient societies (say thousands of years ago) do seem to have mostly presumed the world was flat, to the extent we have any records. Mostly though they just didn't care, why should they? But the Ancient Greeks figured out that a Flat Earth doesn't work, observations from their large and growing empire conflicted with the idea of a flat world.

The Flat Earth Myth (that Europeans didn't realise the world was round until Columbus) is just that, a myth. Crazy people insisting an ancient religious work trumps empirical observation existed in Europe at that time, as they still do today, but they weren't the majority and their ideas had little influence. Educated Europeans from e.g. 1400 would recognise a modern globe as a plausible map of the world although only those with the best knowledge of geography would fail to be surprised by how small Europe is in context.

Flat Earth Myth

Posted Nov 21, 2011 21:54 UTC (Mon) by tshow (subscriber, #6411) [Link]

http://en.wikipedia.org/wiki/Eratosthenes#Eratosthenes.27...

That newfangled Journal thing

Posted Nov 21, 2011 14:43 UTC (Mon) by corbet (editor, #1) [Link] (1 responses)

Hmm, I've reread my comment for "what could possibly go wrong?"-type statements, but I'm not finding any. Maybe I've not had enough coffee yet?

Clearly there's plenty that can go wrong; I outlined one or two of them in the article. But, then, it's a rare software project that can't go wrong somewhere.

The SCO Group got a lot of airtime; shall I consider myself on record as having advocated for them too?

In fact, a common LWN pattern is to post a brief item when something comes out, followed by a more detailed look. That is what was done here. It also fits into a fairly long series of articles about messaging, which is something I've seen as a problem for some years now.

Still lots of discussion

Posted Nov 24, 2011 23:23 UTC (Thu) by man_ls (guest, #15091) [Link]

The LWN pattern I was naïvely expecting is: a short note with a link gets a lot of comments, then our sage editor writes a wise piece which calms the waters and gets few comments. At this point however the original article has 239 comments, but our editor's take has an astounding 169.

I would say that there is still room for discussion, but (not having read all comments yet) most are still going round and round: Unix good vs Unix bad, PulseAudio good vs PulseAudio bad, Poettering good vs Poettering bad. Very few people are advocating to wait and see what the authors come out with, offering some suggestions in the process -- which is what our beloved editor did in his piece, or at least what i read in it. I am not sure if it is just that people need to vent steam, or some previous sins of Poettering; probably a combination of both. But there is a definite need for more articles on this fundamental issue; most of us will hopefully understand that writing about something critically but with caution is very different from an enthusiastic endorsement. Level-headed requirements, improvements and suggestions are especially interesting since Poettering (aka mezcalero) often reads and comments on LWN. Not on this op-ed though, which I read as a sign of Poettering's maturity.

That newfangled Journal thing

Posted Nov 21, 2011 12:45 UTC (Mon) by niner (guest, #26151) [Link] (4 responses)

"As another poster mentioned, to date, this individual hasn't made one bit of code that makes for a significant improvement in system operation and HAS created significant amounts of chaos."

Pulseaudio finally ended 15 years of Linux audio pain for me. It lowered power usage drastically, so I can finally listen to music while on battery without giving it much thought. It finally lets me control different applications' volume seperately. It lets me use a bluetooth headset without the problems this included before. Systemd will finally end the whole damn init script mess and maybe journald will bring an end to hours of wasting time with grep and perl to parse log files.

On the other hand, I can't think of any contribution at all of yours. Could you please remind me of why we have to thank you?

That newfangled Journal thing

Posted Nov 21, 2011 16:46 UTC (Mon) by bferrell (subscriber, #624) [Link]

Really? OK, My first installation on Linux was my very own back in '93. My next Linux install was in the western network management center for MCI shortly after that. Yes, I know MCI is reviled. Oh well. Anyway ever since then, for nearly 20 years, advocating and supporting the use of Linux for my friends, in business dealings, in every place where it fits and makes things better for people. I suspect your contributions may be similar... Or do you not recognize those as contributions? Are you perhaps implying that unless someone writes well known programs or does high profile work that aren't contributing? At least you're a subscriber.

Just asking

That newfangled Journal thing

Posted Nov 21, 2011 17:41 UTC (Mon) by ThinkRob (guest, #64513) [Link] (2 responses)

> Pulseaudio finally ended 15 years of Linux audio pain for me. It lowered power usage drastically, so I can finally listen to music while on battery without giving it much thought.

That was exactly my experience with OSSv4.

The only difference is that it worked the first time I tried it, and it didn't take a couple years of polishing by the distros before it worked "out of the box". And it supports output at all of my card's sampling rates and depths (ALSA couldn't do better than 48@16).

Then again, as a mere user I haven't written any Linux code with worldwide use, so my opinions and observations don't matter.

> On the other hand, I can't think of any contribution at all of yours. Could you please remind me of why we have to thank you?

On a slightly less smart-ass, more serious note, this sort of attitude is a serious (growing?) problem in the Linux community. With Unity, GNOME 3, systemd, pulseaudio, etc. we've seen several recent examples of a fairly large group of users/sysadmins/non-coders saying that they don't like some proposed change or that they think that a new solution is inferior to that which it replaces. In response, a small but very vocal portion of the community turns around and basically belittles them, mocking their views since they haven't written large amounts of popular code.

This is not how we win new users.

This is how we piss off the existing users until they switch platforms.

Microsoft and Apple don't always listen to user suggestions, but at least they don't mock their users when they offer some.

That newfangled Journal thing

Posted Nov 21, 2011 18:36 UTC (Mon) by niner (guest, #26151) [Link] (1 responses)

"a fairly large group of users/sysadmins/non-coders saying that they don't like some proposed change or that they think that a new solution is inferior to that which it replaces. In response, a small but very vocal portion of the community turns around and basically belittles them, mocking their views since they haven't written large amounts of popular code."

My post was not a repsonse to someone "not liking a proposed change" or "thinking that a new solution is inferior". It was a response to "this individual hasn't made one bit of code that makes for a significant improvement in system operation and HAS created significant amounts of chaos."

If someone claims that Lennart has not contributed any improvement but only created chaos, he really should be prepared to answer a question about his own contributions, which the original poster even did.

That newfangled Journal thing

Posted Nov 22, 2011 23:29 UTC (Tue) by da4089 (subscriber, #1195) [Link]

It's perfectly reasonable to have an informed and valuable opinion without being an author of a widely used piece of software or any other form of contribution.

Attempting to suppress discussion by imposing this sort of constraint suggests only that you lack compelling logical arguments.

Question the "chaos" purportedly created, the relative numbers pro and con, etc. There's plenty of real discussion to be had.

That newfangled Journal thing

Posted Nov 21, 2011 15:42 UTC (Mon) by HelloWorld (guest, #56129) [Link] (21 responses)

> As another poster mentioned, to date, this individual hasn't made one bit of code that makes for a significant improvement in system operation and HAS created significant amounts of chaos.
PulseAudio brings various significant improvements: one can control the volume separately for multiple applications, one can move audio streams from one device to the other (extremely handy for USB headsets) and it's network transparent. Yes, some people might have had problems with it (though most of those problems were actually caused by driver bugs that PulseAudio merely exposed (which is a good thing in my book)), but I think it was worth it.

Also, what "significant amounts of chaos" are you talking about regarding systemd and/or avahi (these being the other two significant Poettering projects)? They've been working just fine for me, and systemd provides features no other Linux init system has before (i. e. reliably killing a service, a sane configuration syntax, unmatched parallelization of the startup process, socket activation and a lot more).

> The absolute arrogance of these two (this time) is, in my opinion, rivaled only by Hans Reiser.
You know what's arrogant? Calling somebody else's code is useless just because YOU don't need it.

> No, not really, but it does leave a very bad taste in my mouth and enhances my dislike for code proposed and written by Mr Poettering (and company).
Oh, brilliant. So you're fully aware that systemd and thus Mr Poettering have *nothing* to do with your problem and you *still* whine about it? Sorry dude, that's just pathetic.

That newfangled Journal thing

Posted Nov 21, 2011 17:47 UTC (Mon) by anselm (subscriber, #2796) [Link] (20 responses)

Lennart Poettering could implement a program that would cause any code to run 10 times faster and require only a quarter of the memory and people would still dismiss it out of hand simply because it comes from Lennart and they couldn't get PulseAudio to work when it was very new.

I think it is a good idea for somebody to take a close look at logging. The major selling point of the current system is its ubiquity, not its compelling design, which was debatable even when it was first proposed (Hard-coded list of categories? Great idea!). It is silly to claim that syslog has no issues whatsoever, and that it should stay the way it is for eternity. There are lots of things about syslog that are worth reevaluating. This does not mean that »the journal« is automatically the best possible approach to a replacement, but it is definitely a start.

It may be unfortunate that Lennart is apparently the only person who is independent-minded enough to actually tackle this sort of thing, simply because so many people seem to hate him that much. The same people who complain about systemd today would at the time have been violently opposed to the introduction of SysV init because it gratuitously overcomplicates something – the /etc/rc file – that worked perfectly fine and had done so for nearly 20 years, and who needs this newfangled idea of runlevels, anyway?

That newfangled Journal thing

Posted Nov 21, 2011 19:25 UTC (Mon) by juliank (guest, #45896) [Link] (4 responses)

> the /etc/rc file – that worked perfectly fine and had done
> so for nearly 20 years, and who needs this newfangled idea
> of runlevels, anyway?

Well, you practically only need two run-levels: Single user (rescue) and Multi User (normal). I certainly don't see a need for more than two runlevels.

That newfangled Journal thing

Posted Nov 21, 2011 19:30 UTC (Mon) by juliank (guest, #45896) [Link]

You actually only need:

a) an rc.d directory with scripts
b) a list of scripts that will always be run (single-user mode)
c) a list of scripts that will be run in normal situations (multi-user)

Without (b), you get the init system used in NetBSD and FreeBSD.

That newfangled Journal thing

Posted Nov 21, 2011 22:51 UTC (Mon) by anselm (subscriber, #2796) [Link] (1 responses)

I certainly don't see a need for more than two runlevels.

Speak for yourself. I used to have a setup where the difference between runlevel 2 and runlevel 3 was that runlevel 3 enabled the fax-modem getty on the serial interface that my modem was connected to, in order to handle incoming calls. There are obviously lots of other ways to do this, but since a big part of SysV init's business is dealing with serial interfaces, anyway, it was the most convenient method.

It is important to note that the customary arrangement of runlevels in use with most Linux distributions is exactly that – a convention. Since few people now operate standalone systems with serial ttys, runlevel 2 in the customary arrangement is rarely used these days, and most people will pick either runlevel 3 or 5 as their standard. However, this does not mean that the runlevel system cannot be employed profitably for other things, or that the runlevels must always mean exactly what they mean in the customary arrangement. (Which is essentially why Debian does not subscribe to the customary arrangement – it mostly doesn't make a great deal of sense, but that doesn't mean runlevels as such are useless.)

That newfangled Journal thing

Posted Nov 23, 2011 12:52 UTC (Wed) by sorpigal (subscriber, #36106) [Link]

I love runlevels so much I just wish I had more of them. Unlimited numbers of named "runlevels" that let me boot to or switch to a certain service target is what I want. That way I can have a "maintenance mode" target, for example; it would be like raising the job fence. Even on my workstation I could have a "gaming" target which stops everything I don't need for games and then a "web development" target which runs my apache and postgres. Funnily, systemd allows me to do this... it's one of the things I like about systemd.

That newfangled Journal thing

Posted Nov 21, 2011 23:42 UTC (Mon) by dashesy (guest, #74652) [Link]

On an embedded device I can use runlevel 2 (the default) to to only run the user-mode instrument controller program (no gettys).
I may go to runlevel 3 to have a GUI to look at some graphs.

That newfangled Journal thing

Posted Nov 25, 2011 4:43 UTC (Fri) by cas (guest, #52554) [Link] (14 responses)

because it comes from Lennart and they couldn't get PulseAudio to work when it was very new.

no, it's because it comes from LP and he quite clearly hates Unix and loves Mac & Windows. Not the kind of person we want designing core unix infrastructure.

Every time I hear of yet another one of Poettering's fads, I can't help but remember the Henry Spencer quote "Those who don't understand UNIX are condemned to reinvent it, poorly.". Poettering is a near-perfect exemplar of that.

BTW, I hated systemd before i had any idea that it was written by the author of Pulse Audio - even aside from its technical problems, it suffers from the Gnome Disease of radical revolutionary toss-out-the-old-bring-in-the-new (and repeat every couple of years), rather than the far more sane and productive method of gradual, incremental improvements - and actually *fixing* bugs rather than just abandoning the code and starting from scratch because maintenance is boring while complete rewrites are exciting.

It may be unfortunate that Lennart is apparently the only person who is independent-minded enough to actually tackle this sort of thing

but he's not doing that. he's just cloning bad ideas borrowed from Mac (launchd) and Windows (win event logging). and worst of all, they're not actually solving any real problems except his perception that unix sucks and should be remade into some bastard hybrid of windows and mac

That newfangled Journal thing

Posted Nov 25, 2011 8:49 UTC (Fri) by anselm (subscriber, #2796) [Link] (12 responses)

[systemd] suffers from the Gnome Disease of radical revolutionary toss-out-the-old-bring-in-the-new (and repeat every couple of years), rather than the far more sane and productive method of gradual, incremental improvements

This is quite obviously untrue. It has been pointed out in detail here and elsewhere that systemd can use, among other things, init scripts for existing services with no change. How is that a »radical revolutionary toss-out-the-old-bring-in-the-new«?

they're not actually solving any real problems except his perception that unix sucks and should be remade into some bastard hybrid of windows and mac

This is not discussion, it is propaganda.

Many people now seem to regard things like SysV init and syslogd as graven-in-stone pieces of infrastructure that were there in the very beginning of Unix, possibly on the original PDP-7 that Thompson and Ritchie hacked on, and which can only be criticised at the peril of sacrilege. It is just as well to remember that SysV init, and to a lesser extent syslogd, started out in the 1980s as exactly the same kind of »radical revolutionary toss-out-the-old-bring-in-the-new« that people are now railing against in the form of systemd and »The Journal«.

We have gone a long way in the intervening years towards discovering better techniques of doing almost anything in system administration, and I'm puzzled as to why topics like the init system must be forever exempt from being changed. Surely we can examine even these areas every 20 years or so, to see if we can do better? (I don't remember a similar sort of spite-fest around Upstart, which attempted exactly the same thing that systemd is trying to do, namely creatively revamp the init subsystem. Is that because it didn't go as far as systemd, or because some people hate Lennart Poettering with more of a vengeance than they do Scott James Remnant?)

That newfangled Journal thing

Posted Nov 25, 2011 9:15 UTC (Fri) by dlang (guest, #313) [Link] (9 responses)

> I don't remember a similar sort of spite-fest around Upstart, which attempted exactly the same thing that systemd is trying to do, namely creatively revamp the init subsystem. Is that because it didn't go as far as systemd, or because some people hate Lennart Poettering with more of a vengeance than they do Scott James Remnant?

one key thing is that the developers of upstart are not nearly as condescending towards people who disagree with them (they don't get dismissed as "people opposed to all change" instantly)

there's also the matter of the degree of change.

BSD init to SysV init are both variations of shell scripts. the difference is just in how the scripts are organized.

I haven't done much with upstart, but from what little I've seen of it, it's also based on shell scripts.

systemd strongly encourages (if not outright requires) that the logic be implemented in C code

this difference, and the attitude towards sysadmins (who generally know scripting of various kinds, but are not as comfortable with C) goes a long way towards alienating the people who are providing the support for their less technical friends and relatives.

Yes, there is a huge advantage to owning the desktop/home user

Microsoft's assault on the datacenter is based on it's ownership of the destop.

RedHat became the linux distro of choice in business mostly because it was what linux folks were running at home.

However, now that Linux is _so_ common in the datacenter, those people are the core of your linux expertise. Even if they are not involved with the initial deployment of Linux to people's desktops, they get pulled in whenever something goes wrong. making it harder for these people to track down the problem and fix it really hurts reputations..

And yes, after generating this sort of reputation for a few projects, it will affect how new proposals from the same people are received.

That newfangled Journal thing

Posted Nov 25, 2011 10:00 UTC (Fri) by anselm (subscriber, #2796) [Link] (7 responses)

one key thing is that the developers of upstart are not nearly as condescending towards people who disagree with them (they don't get dismissed as "people opposed to all change" instantly)

Maybe this is because at least some of them do seem to give the impression of being »opposed to all change«? It is interesting (or even amusing) to watch the contortions some people are going through in this ongoing discussion just because, according to them, the syslog mechanism may apparently never be changed.

Lennart Poettering has gone to the trouble of posting a large number of blog articles to inform people about systemd in a very factual manner. This does not particularly strike me as something a person as condescending as you claim Lennart is would bother doing.

BSD init to SysV init are both variations of shell scripts. the difference is just in how the scripts are organized.

Well, SysV init did add the whole business of run levels, a jungle of symbolic links, etc., when before there was basically one script that was run during boot. A trivial difference, to be sure :^) At the time SysV init was about as revolutionary compared to the prior approach as systemd is today compared to SysV init. Today's systemd haters would probably have railed against it with the same vituperation.

I haven't done much with upstart, but from what little I've seen of it, it's also based on shell scripts.

With that logic, it would make sense to claim that an air mattress is similar in use and performance to an aircraft carrier since they are both based on the physical principle of flotation.

systemd strongly encourages (if not outright requires) that the logic be implemented in C code

It is true that many of the boot-up tasks that systemd performs, which were traditionally implemented as shell scripts, are provided in the shape of C-based executables. I don't think this is an actual requirement; I seem to recall that the basic idea is to speed up the proceedings. As far as I know there would be no problem with adding things that are implemented in shell code, or replacing some of the C-based tools with shell-based implementations if that proved necessary.

As far as service management is concerned, systemd does implement most of the required logic itself and only relies on non-executable configuration files for the details. This is actually a good thing in my book since, among other benefits, it potentially leads to increased standardisation, where right now the shell-based init scripts are different from one distribution to the next. Again, systemd makes it possible to use traditional SysV-init style init scripts if the administrator so desires. I don't see it as a big problem.

I'm sorry but I don't really see what the rest of your comment has to do with anything.

this is exactly what drives people up the wall...

Posted Nov 25, 2011 10:32 UTC (Fri) by khim (subscriber, #9252) [Link] (6 responses)

As far as service management is concerned, systemd does implement most of the required logic itself and only relies on non-executable configuration files for the details.

And this exactly what driver most people up the wall. You see, sysadmins always relied on the power to alter system configuration in twisted, non-standard way. For some it's requirement to fit in twisted, non-standard pre-existing configuration, for some just a source of pride, but they were content in their knowledge that they have power to do anything by changing these shell scripts.

Systemd throws all that away: you can not just "add tiny nodge here and small delay there". You need to either contact upstream (to add configuration knob) or turn the automatic off and write complex and often convoluted script from scratch. This is what irritates people the most in systemd design.

This is actually a good thing in my book since, among other benefits, it potentially leads to increased standardisation, where right now the shell-based init scripts are different from one distribution to the next.

Right. And application developers will probably welcome that. But system administrators will not: they've lost some of their power and are forced to think about real solutions instead of trying to apply chewing gum on the mix of bailing wire. Of course this irritates them.

And, as usual, people who benefit don't have too much energy to fight for systemd (they are there: you can find them in all flamefests about Lennart's creations), but people who are burned fan them hoping against hope that they will be able to keep chaotic nature of Linux where their ability to skillfully manipulate duct tape will be appreciated forever more.

Well, Linux today certainly fulfills the second part of the mantra ("complex things should be possible") but what about first ("easy things should be easy")? The counteroffer (let's add layers upon layers of bandaids on the existing solution) does not help much here so Lennart's way looks like the only viable alternative. Yes, it means that some complex things will not be possible anymore. If they will still be actually needed they can be reintroduced later...

this is exactly what drives people up the wall...

Posted Nov 25, 2011 11:25 UTC (Fri) by anselm (subscriber, #2796) [Link]

Systemd throws all that away: you can not just "add tiny nodge here and small delay there".

Guess what – systemd service definition files do allow you to specify individual shell commands that will be invoked at various points of a service's life cycle. Those system administrators who today need to tweak convoluted, distribution-specific SysV init files will likely find it a lot easier to deal with this interface.

Also, SysV init gives you no real way to override a distribution-provided init script with one that you modified, such that your version will take precedence over the one in the distribution and your configuration will not be overwritten once the next version of the distribution package comes around. Systemd does. (And this is after this issue has had 25 years to be straightened out in SysV init. Give systemd 25 years to mature and see what it looks like then.)

All of these things should be painfully obvious to anyone who actually takes the trouble to read the systemd documentation and descriptive blog posts. But of course it is much easier and more convenient to misrepresent the package (and the intentions of its authors) out of a position of ignorance.

this is exactly what drives people up the wall...

Posted Nov 25, 2011 11:41 UTC (Fri) by michich (guest, #17902) [Link] (2 responses)

Can you give any example where systemd forces you to write a complex and convoluted script from scratch? I've seen cases where instead of running a program directly from the systemd unit file it made some sense to call a shell wrapper, but why would it have to be complex and convoluted?

Because you talk about legacy code?

Posted Nov 25, 2011 12:58 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

One recent example: connection to Corbina.Net (I've helped my friend with that). You need start igmpproxy first (because otherwise it'll crash when L2TP disconnects), then you need to check L2TP server (some of them don't work and then you need to try again), then you connect to L2TP server and wait till everything stabilizes - and only after that you can start IPSec to connect to your real VPN network.

With SysV init you can just order all services in proper (tested and working) order while with systemd you either need to properly describe dependencies or run the whole thing as one service.

In first case you need much, much deeper understanding of the problem then "if I run igmpproxy before L2TP client then everything works, if I do it in opposite order then igmpproxy crashes when L2TP client auto-reconnects" in second case you need to write the whole script from scratch.

Because you talk about legacy code?

Posted Nov 25, 2011 13:26 UTC (Fri) by anselm (subscriber, #2796) [Link]

Yes, but which SysV init are you talking about – the traditional one where you get to set all the symbolic links yourself, or the one most distributions are using these days, where the symbolic links are set automatically from dependencies in the LSB headers of the init scripts?

In the former case, any ordering you can explicitly impose on SysV init scripts through numbers in symbolic links can trivially be replicated through explicit »Before=« and »After=« dependencies in systemd. This most emphatically does not require a deeper understanding of the problem than you would need to come up with a suitable ordering for SysV init in the first place.

In the latter case, you need to specify all the dependencies in the LSB script headers for the benefit of your SysV init, and the LSB script header doesn't let you do anything that systemd dependencies don't let you do too. If anything, the systemd dependencies are a lot more expressive than the LSB script-header ones, so your job should be that much easier.

Finally, if you need to wait for something to »stabilise« after you have started a service in systemd, you can write a shell wrapper around that service which does the check, and that shell wrapper will be vastly simpler than a full SysV init script.

this is exactly what drives people up the wall...

Posted Nov 25, 2011 12:24 UTC (Fri) by dgm (subscriber, #49227) [Link] (1 responses)

> Yes, it means that some complex things will not be possible anymore. If they will still be actually needed they can be reintroduced later...

So they will still be possible after all...
Honest question: is systemd really so inflexible that I _cannot_ do "interesting" stuff? I thought that ability to run current init scripts preserved that flexibility for when it's needed...

this is exactly what drives people up the wall...

Posted Nov 25, 2011 12:38 UTC (Fri) by anselm (subscriber, #2796) [Link]

No.

(Check the systemd page to find out for sure.)

That newfangled Journal thing

Posted Nov 27, 2011 2:44 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> systemd strongly encourages (if not outright requires) that the logic be implemented in C code
I don't understand why people keep spreading this stupid lie. systemd requires no such thing; if you actually want to write a shell script to start your service, you're free to do so.
That said, I agree with Lennarts assessment that bourne-style shell scripts are a nightmare for robustness and maintainability and are best avoided. This is where systemd comes in: it offers a configuration file format that allows most of what is commonly done in init scripts (i. e. setting environment variables, rlimits and all that), yet is *much* simpler, more maintainable and generally easier to work with.

I think it's matter of presentation and the actual implementation

Posted Nov 25, 2011 9:25 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

I don't remember a similar sort of spite-fest around Upstart, which attempted exactly the same thing that systemd is trying to do, namely creatively revamp the init subsystem. Is that because it didn't go as far as systemd, or because some people hate Lennart Poettering with more of a vengeance than they do Scott James Remnant?

I think it's matter of presentation and scale.

Scott's message:
Here is new version if SysV init which does exactly the same thing as the old one, but it can do few new tricks as well if you'll use new "native services". You can switch to them at you leisure. Indeed: upstart was added to Edgy Eft (Ubuntu 6.10) but even the next LTS (Hardy Heron AKA Ubuntu 8.04) used it as "modern SysV replacement" - native Upstart bootup was only introduced in Karmic Koala (Ubuntu 9.10).

Lennart's message:
Here is replacement for SysV init and upstart. It introduces totally new world and it's design does not take in the consideration the existing practice at all. Sure, we understand that compatibility is important and we'll provide compatibility mode till everyone will switch to systemd. We want to see this switch sooner rather then later - so we will patch programs and make them systemd-aware ASAP.

The first mode leads to years to transition but generates few ill feelings (people feel that they are in charge and can "jump ship" at any time) while second mode introduces abrupt change which naturally leads to spite-fests. Especially when another critical piece of the infrastructure is introduced which is intrinsically tied to the "new and unproven" systemd.

I think it's matter of presentation and the actual implementation

Posted Nov 25, 2011 10:24 UTC (Fri) by anselm (subscriber, #2796) [Link]

It introduces totally new world and it's design does not take in the consideration the existing practice at all.

I don't think that is actually entirely fair.

For one, what systemd does is only a »totally new world« if you disregard the experience gained with Apple's launchd, which pioneered many of the ideas that we now find in systemd and which, on the whole, do make sense from a technical point of view. To some people this seems to come across as an evil conspiracy to make Linux »more like MacOS«, but I believe this is more due to the fact that the hackers at Apple appear to be on to something good here.

Also, it is worth bearing in mind that many of the things systemd does are traditionally provided by different pieces of infrastructure (SysV init, inetd, cron, …), not because these were designed that way from the start but because they arose at various points in the history of Unix and were introduced by different groups of people. It does seem worthwhile to me to try and construct a general framework for these tasks in the interest of simplicity, better configurability, and standardisation.

I'm not so sure...

Posted Nov 25, 2011 8:59 UTC (Fri) by khim (subscriber, #9252) [Link]

no, it's because it comes from LP and he quite clearly hates Unix and loves Mac & Windows. Not the kind of person we want designing core unix infrastructure.

Actually I think it's time to bring people who care about things being done more and less about sacred "UNIX way". It does not mean I'll accept journald unconditionally, but I will not reject it before try like most people here are doing.

Every time I hear of yet another one of Poettering's fads, I can't help but remember the Henry Spencer quote "Those who don't understand UNIX are condemned to reinvent it, poorly.". Poettering is a near-perfect exemplar of that.

You know, I think people repeat this mantra so often they start to believe it. Even if it's not true at all.

Exhibit 1: command-line utilities. Solaris included real hardcore UNIX command-line utilities. But to make it even somewhat usable you needed to bring GNU packages and use that. "true UNIX lovers" said the same thing about "overbloated and heavy" GNU tools they say today about systemd and journald yet today these "good old GNU tools" are perceived as "UNIX tradition" because otherwise you must admit that UNIX has mostly dead.

Exhibit 2: remote access. 20 years ago remote access in Windows didn't exist while UNIX boasted "network-native X protocol". 10 years ago Windows got RDP and remote access become easy for Windows people. Yes, I know, Microsoft did that by twisting Citrix arms, but the end result is still the same: easy to use system where you don't need to configure anything in advance - you can just turn knob "on" when you are late and need to go home, then connect from home and continue your work. It happened 10 years ago so we should have similar capabilities on Linux today? Nope: today it's still huge problem with a Linux world: there are bazillion tools for that, but they don't employ "network-native X protocol" (because it lacks the knobs: you can not switch clients from one X server to another "on the fly") and in general it's hard to use and slow. The one big achievement in this regard was introduction of PulseAudio: at least now we can reliably forward not just GUI, but the accompanying sound as well.

Rather than the far more sane and productive method of gradual, incremental improvements - and actually *fixing* bugs rather than just abandoning the code and starting from scratch because maintenance is boring while complete rewrites are exciting.

It does not work, sorry: search for some of his arguments in regard to syslog are technically wrong, but can be considered true if one looks at current practice.

People are vocal social creatures, true - and this calls for "gradual, incremental improvements". But that's lie. People are also lazy creatures and that works for Lennart's "let's redo everything" approach and against "more sane and productive method". Think about it: how much foam was on the web WRT KDE 4. You'd think that outraged people dropped everything and rushed to support KDE 3? Well... four of them did...

Opt-in vs opt-out makes huge difference. Opt-ins are usually ignored while opt-outs lead to large flamefests but then quite fast adoption. And you will have flamefest anyway then why not redo everything to cleanup all the cruft?

If the result will be usable - it'll be adopted anyway, if it'll not be usable - it'll be thrown away no matter what. And as you've correctly noted it's often simpler to rewrite something from scratch rather then to try to fix bugs in existing solution.

The journald design is horrible to the point of useless

Posted Nov 21, 2011 1:45 UTC (Mon) by ebiederm (subscriber, #35028) [Link] (35 responses)

I read through the design requirements ad they sounded like very desirable. If a bit over impossible on a few fronts.

Then I looked at the proposed design, and thought things looked shaky.

Then I read the faq and was seriously insulted.

A logging facility with no network support?
A logging facility without a stable ABI on disk and no plans to create one?

An attitude that says the classic syslog interface is depercated and the native journal interface is preferred.

Not considering the model that has worked very well for translation of linux userspace messages into other languages and insisting we all use unreadable UUID?

This seems to be a design that has lofty goals but horrible human factors. And the authors seem to be deliberately ignoring design choices that are no more difficult for them are much easier for other developers.

An attitude that says every log message in every server must change, to something bizarre because the developer refuses to look at other successful user space models of doing things?

The desisgn is junk and should never be consider seriously. Unfortunately there seems to be enough propellant behind it that the pig might fly.

But in my opinion no one should seriosly consider journald for real world use. The design is irreponsible crap.

The journald design is horrible to the point of useless

Posted Nov 21, 2011 2:23 UTC (Mon) by josh (subscriber, #17465) [Link] (29 responses)

> A logging facility with no network support?

Yet. Seems like a reasonable thing to defer for a first prototype.

> A logging facility without a stable ABI on disk and no plans to create one?

ABIs *suck*. Why should journald need to nail down a binary disk format rather than creating an API? Why should people recreate parsers for a disk format, usually badly, rather than all using the same parser via a library? Versioned formats don't help here, because too many people will write tools around a format, and at best add an assert() on the version number, or at worst not even bother with that because "It's just a quick Perl script"; when the new version comes out, widespread breakage will occur anyway, and people will complain and demand the return of the existing format. And most importantly, why should journald commit to nailing down a disk format *now* that it can never change, rather than allowing for potential changes in the future to meet needs such as improved performance, indexing, and so on.

For that matter, journald could support multiple formats for different use cases, or even provide an implementation of the API that just serves up syslog messages as though they'd gotten sent to journald in the first place; how can they do that if tools ignore the API and poke the disk formats on their own?

As long as journald handles backward compatibility with older versions of the format (once it gets out of development), I don't otherwise care about any particular storage format.

> An attitude that says the classic syslog interface is depercated and the native journal interface is preferred.

I can't imagine having any other attitude; if you come up with something you consider better, why *wouldn't* you advocate using it, even though you provide backward-compatibility for the existing approach?

> Not considering the model that has worked very well for translation of linux userspace messages into other languages and insisting we all use unreadable UUID?

That one I'll grant you, but consider that this is a strawman proposal designed to get feedback; that may well change in the future. And on the other hand, have you considered their complaints about the use of format strings as ABI? The concept of some kind of sensible identifier associated with a log message type rather than a particular wording of that message makes sense to me.

The journald design is horrible to the point of useless

Posted Nov 21, 2011 23:50 UTC (Mon) by alankila (guest, #47141) [Link] (1 responses)

Observing this discussion, there seems to be a faint pattern.

There seems to be a tremendous dislike for using shared libraries. Perhaps it too is part of unix heritage, as at one point these things did not exist and code that was meant to be used from multiple programs was written in the kernel.

Perhaps the "unix way" is to make sure that whatever you do, "cat" can be used to read the data, and if it takes a kernel module to construct that file on the fly for cat to read, that is more acceptable than to have a binary file and some iterator functions in a library...?

The journald design is horrible to the point of useless

Posted Nov 23, 2011 14:45 UTC (Wed) by nix (subscriber, #2304) [Link]

These days, you don't need a kernel module, you can use FUSE. If the FUSE filesystem works by using some shared library, that's great: it means you don't have to duplicate code. But filesystem interfaces are still as important, or more so, than library interfaces in the Unix world.

The journald design is horrible to the point of useless

Posted Nov 24, 2011 8:39 UTC (Thu) by rschroev (subscriber, #4164) [Link] (26 responses)

> Why should journald need to nail down a binary disk format rather than creating an API?

Because, as I see it, it is very important that system administrators can copy log files to another system and still view the data. That other system can be a newer or older version of Linux, or can be a totally different operating system.

With text files that is no problem at all. There's a multitude of tools available on all operating systems for doing that.

With a well defined binary disk format, it can be a pain because the required tools are not necessarily available, but at least you can create your own, be it a quick and dirty perl or python script or a full fledged log analyzer.

With a binary format that is not well defined, you have a big problem if you don't have a system with a compatible version of the tools available.

The journald design is horrible to the point of useless

Posted Nov 24, 2011 10:01 UTC (Thu) by josh (subscriber, #17465) [Link] (25 responses)

As long as "compatible version" means "any version of journald at least as new as the one used on that system", that doesn't seem like a problem to me.

The journald design is horrible to the point of useless

Posted Nov 24, 2011 10:09 UTC (Thu) by rschroev (subscriber, #4164) [Link] (24 responses)

It is still a problem. There's not necessarily another journald-equipped system available. It's perfectly possible that an organization has, for example, a Linux server, a Windows server and a Solaris server, and on the desktop only has Windows and Mac OS X machines.

Again, absolutely no problem if your log files are text files. Show-stopper if your log files are binary, and especially if the binary format is not well defined.

The journald design is horrible to the point of useless

Posted Nov 24, 2011 15:21 UTC (Thu) by anselm (subscriber, #2796) [Link] (23 responses)

Two thoughts:

  • Presumably a well-defined binary format will emerge in due course as the package matures.
  • A basic binary-log-to-text program in Perl would probably be a few dozen lines long and portable to wherever Perl runs, i.e., anywhere it matters. (The same would apply to most similar languages.) Doesn't sound like a big problem to me.

The journald design is horrible to the point of useless

Posted Nov 24, 2011 15:26 UTC (Thu) by rschroev (subscriber, #4164) [Link] (2 responses)

Presumably, yes.

Given the choice between

1) a binary format that presumably in the future will be well-defined, and which can be analyzed using a perl program that will run almost anywhere
2) a text format that can be read anywhere without if's and but's

then my choice for something as essential as log files will always be 2. Especially because all perceived or real shortcomings of the current solution can be implemented in a text format just as well.

The journald design is horrible to the point of useless

Posted Nov 24, 2011 15:56 UTC (Thu) by anselm (subscriber, #2796) [Link] (1 responses)

Especially because all perceived or real shortcomings of the current solution can be implemented in a text format just as well.

Now that's a premise that I would by no means consider adequately established.

That's quite trivial...

Posted Nov 24, 2011 17:03 UTC (Thu) by khim (subscriber, #9252) [Link]

You can convert any binary format to text format using hexdump.

Not sure it'll satisfy the Unix purists, though.

The journald design is horrible to the point of useless

Posted Nov 25, 2011 4:53 UTC (Fri) by cas (guest, #52554) [Link] (19 responses)

Presumably a well-defined binary format will emerge in due course as the package matures.

well, if we're going to base our decisions on presumptions and vaporware, then i prefer the vaporware based around the fact that better versions of the few minor useful tweaks (the hash checksums from previous log entries is the only one that i can think of right now, but there may be others) that are salvageable from this The Journal idea can trivially be added to an existing syslog daemon like rsyslogd or syslog-ng without any need for discarding years/decades of existing practice and command line tools and log processing scripts

The journald design is horrible to the point of useless

Posted Nov 25, 2011 9:09 UTC (Fri) by anselm (subscriber, #2796) [Link] (18 responses)

At the beginning of the 1990s there was that thing called Linux that presumably might be able to mature into a usable operating system given the right kind of attention. The fact that it would have been possible to add the few minor useful tweaks that were salvageable from that Linux idea to an existing operating system like BSD without any need for reinventing years/decades of existing practice does not seem to have discouraged people from providing exactly that sort of attention, and look where we are now.

Many people (including apparently you) seem to confuse the »Journal« proposal with something that has emerged 100% finished, to be adopted immediately with no further changes or improvements. I prefer to see it as a starting point for discussion and thoughts on what the weaknesses are in the current logging infrastructure, and how best to address them. Whether what we will be using 5 or 10 years from now will bear a closer resemblance to 1980s syslog or »The Journal« remains to be seen, but I think it is best to keep an open mind.

Incidentally, nobody talks about »discarding years/decades of existing practice«. If you would actually care to read the Journal proposal, it mentions the fact that it is possible to run both it and a traditional syslogd in parallel. Also, the proposal does raise important points that IMHO can't be solved simply by tweaking rsyslogd; I'd be interested in how you intend to add these »trivially«.

The journald design is horrible to the point of useless

Posted Nov 25, 2011 23:14 UTC (Fri) by jubal (guest, #67202) [Link] (17 responses)

http://blog.gerhards.net/2011/11/what-i-dont-like-about-journald.html

The journald design is horrible to the point of useless

Posted Nov 26, 2011 0:15 UTC (Sat) by anselm (subscriber, #2796) [Link] (16 responses)

As I said, discussion – exactly what we want to encourage. It is well worth noting that while Rainer Gerhards appears to have some legitimate issues with Lennart and Kay's proposal, according to his postings he does not think the proposal as such is a bad thing. In particular he does not seem to think it is »horrible to the point of useless« – unlike many people here who are probably less knowledgeable about logging infrastructures than he is. Lennart and Kay will do well to address his concerns in due course, and this will hopefully lead to a better proposal on their part.

It is also worth noting that, being the author of rsyslogd, Rainer Gerhards has his own horse in this race, so his comments, while interesting and to the point, probably should be read with the appropriate amount of scepticism.

The journald design is horrible to the point of useless

Posted Dec 1, 2011 3:50 UTC (Thu) by dlang (guest, #313) [Link] (15 responses)

Rainer has provided a more detailed response and analysis

http://blog.gerhards.net/2011/11/serious-syslog-problems....

The journald design is horrible to the point of useless

Posted Dec 1, 2011 8:26 UTC (Thu) by anselm (subscriber, #2796) [Link] (14 responses)

rsyslogd is a great piece of software, no doubt. However the fact that there is one outstanding implementation of a syslog receiver (or two, if you want to count syslog-ng, too) doesn't really save syslog as such. What would be needed is a thorough refurbishment of the client-side syslog API, and then to get more of what rsyslogd does into other syslogd implementations (server-side and client-side).

If nothing else, the journald proposal helps us identify how a next-generation syslog API should look like. It seems that Rainer Gerhards has already adopted some journald ideas for rsyslogd, and if journald does indeed lead to improvements in syslog output storage, too (preferably standardised and well-documented ones), then it has already been useful.

Like dlang, I can only recommend Rainer's piece to anybody who is interested in an informed discussion. As he says himself, he is biased in one direction while the journald proposal is biased in another, but he is clearly very well-informed about the issues at hand and it shows. It would be interesting to hear Lennart and Kay comment on his article.

The journald design is horrible to the point of useless

Posted Dec 1, 2011 12:04 UTC (Thu) by dlang (guest, #313) [Link] (13 responses)

If you exclude rsyslog and syslog-ng, how many syslog implementations are in use nowdays?

On linux, sysklog has been poorly maintained for a long time, and has been replaced by rsyslog on just about every distro. older distro releases won't have it, but they also won't have systemd or the journal either.

In terms of a log store, what is it that you are looking for? As I see it, the needs of the log store vary drastically, not just from person to person, but also from use to use.

Yes, sometimes you want to do a search for a particular type of log event, but a lot of the time you want to see what happened around a particular time.

syslog does not dictate how the data is stored, and you can even store the same data in many different ways (on different machines, and even shard it across farms of machines)

Yes, the syslog() call could use improving, but since the journal proposal says that it's going to remain compatible with it, it doesn't fix it.

Also, I see it less as Rainer adopted some journald ideas as the journald people aren't aware of what was already available (the trusted properties being one exception)

The journald design is horrible to the point of useless

Posted Dec 1, 2011 13:25 UTC (Thu) by anselm (subscriber, #2796) [Link] (12 responses)

If you exclude rsyslog and syslog-ng, how many syslog implementations are in use nowdays?

The BSDs apparently have their own implementation. MacOS X probably has, too. There are likely syslogd implementations in all sorts of appliances.

This doesn't matter as much for the purposes of journald, which tries to reboot the franchise to a large extent, but as far as tweaking the existing syslog mechanisms are concerned, it is just as well to be aware that relying on one single implementation for all the improvements isn't all that different from coming up with something completely new and different.

Yes, sometimes you want to do a search for a particular type of log event, but a lot of the time you want to see what happened around a particular time.

I maintain mail servers (among other things). I would be pleased to see a way of grouping all log entries that pertain to one particular mail message, or one particular sender or recipient. Right now this means grepping /var/log/mail.log and friends, which sucks, particularly when log files get rotated.

Improving this would mean getting the developers of a diverse bunch of software packages (Postfix, Amavisd, policyd-weight, …) to adopt a common style of structuring log messages (RFC 5424 is a definite start but really what we need to standardise here is semantics). It would also mean using a log store that actually allows querying for fields such as »message ID«, »queue ID«, or »sender address« in a reasonably efficient fashion, which the current simple-minded method of dumping everything into a text file certainly doesn't. Even dumping everything into a database doesn't unless you actually normalise the individual fields (or an interesting subset of them). I'm not holding my breath.

Given that it involves applications rather than infrastructure, the first goal is of course independent of the »rsyslogd vs. journald« issue but it must be said that journald's binary-format normalised log store, in principle, goes a way towards realising the second goal that, so far, rsyslogd doesn't. Of course nobody says that the current journald proposal is the be-all and end-all of logging, so if all that results from it is that Rainer gets prodded into implementing a more useful log store for structured messages then something important will be gained.

Yes, the syslog() call could use improving, but since the journal proposal says that it's going to remain compatible with it, it doesn't fix it.

The journald proposal doesn't actually say that. It says:

[T]he syslog API syslog(3) is supported as first-class interface to write log messages, and continues to be the primary API for all simple text logging. However, as soon as meta data (especially binary meta data) shall be attached to an entry the native journal API should be used instead.
From Lennart's and Kay's point of view this makes perfect sense, because it allows them to put improvements into the native API, considering that current syslog() users are unlikely to change their code (if they would do that just for journald, they could just as well upgrade it to use the native API).

If we want to hang on to the current syslog scheme, the first thing that it would make sense to do would be to implement a client-side API for RFC 5424 structured-data logging that would guarantee well-formed log messages, and to add this to glibc. The next thing would be to agree on a scheme for identifying facilities that is more flexible and fine-grained than the 24 arbitrary and hard-coded facility codes, many of which do not apply to modern systems. Whether this happens by allowing more numbers or by standardising a structured-data element would be subject to debate.

Also, I see it less as Rainer adopted some journald ideas as the journald people aren't aware of what was already available (the trusted properties being one exception)

Trusted properties, an efficient log store, generality, performance, … there are various things in the journald proposal that rsyslogd has yet to address. As I said, if all that comes out of the journald idea is a better syslogd then we have gained something already. Finally, it would still be fair to allow Lennart and Kay some credit for the cojones to tackle yet another difficult problem, and for getting things moving. The way I see this discussion going, many people (Rainer notably not included) seem to call them stupid for even trying, with no other justification than »We don't like PulseAudio and systemd, and rsyslogd does everything already, nyah, nyah, nyah«, which is (a) untrue and (b) disingenuous.

The journald design is horrible to the point of useless

Posted Dec 1, 2011 13:47 UTC (Thu) by dlang (guest, #313) [Link] (11 responses)

>> If you exclude rsyslog and syslog-ng, how many syslog implementations are in use nowdays?

> The BSDs apparently have their own implementation. MacOS X probably has, too. There are likely syslogd implementations in all sorts of appliances.

> This doesn't matter as much for the purposes of journald, which tries to reboot the franchise to a large extent, but as far as tweaking the existing syslog mechanisms are concerned, it is just as well to be aware that relying on one single implementation for all the improvements isn't all that different from coming up with something completely new and different.

syslog-ng and rsyslog both work across different flavors of Unix. Since LP says that we should only develop for Linux and all other flavors should go hang, this is definantly not an argument in favor of the journal.

there are two independent implementations (not providing the exact same features, but competing and adding features)

but in both of these implementations are doing incremental improvements, not "throw out everything that's worked up until now because I know better"

getting applications to structure their logs is completely independent of what logging mechanism they use, look at the work that CEE is doing to try and define standards (and then watch to see if anyone bothers to use these standards when writing logs) I don't believe that a binary format is going to help this any.

that being said, rsyslog does include the ability to define log parsers that create fields that can then be manipulated by rsyslog (either for decisioning by rsyslog, or for writing to a log store) see http://www.rsyslog.com/doc/mmnormalize.html for a hint on how to invoke this in rsyslog (it then creates $!<name> properties that can be used)

I would strongly suggest that if the default log store you are dealing with does not meet your needs, that you should look at reconfiguring the systems that you manage to use a log store that better fits your needs.

then you say

> Trusted properties, an efficient log store, generality, performance, … there are various things in the journald proposal that rsyslogd has yet to address.

addressing your wish list

rsyslog now has trusted properties (it already had trusted PID)

it supports many different log stores, you need to define what you mean by 'an efficient log store' more precisely before anyone can tell you which one is best for you.

generality seems to be a clear win for rsyslog compared to the journal proposal and it's lock-in to systemd and Linux

as far as performance goes, rsyslog can process a million logs a second (of the right log type, on the right hardware), I seriously doubt that the journal will do nearly as well (especially in it's first implementation), so you must mean something different in terms of performance. what is it that you are looking for in the logging daemon that makes you think that the journal (which doesn't exist yet) will outperform existing, tuned and tested code?

The journald design is horrible to the point of useless

Posted Dec 1, 2011 14:51 UTC (Thu) by anselm (subscriber, #2796) [Link] (10 responses)

syslog-ng and rsyslog both work across different flavors of Unix. Since LP says that we should only develop for Linux and all other flavors should go hang, this is definantly not an argument in favor of the journal.

One difference between you and me is that you seem to be out to prove Lennart Poettering is an idiot while I'm interested in better logging in general, so I'm willing in principle to see some good in the journald proposal. I don't believe that journald as per the proposal is 100% pure gold, but that some of its properties deserve being looked at rather than thrown out outright. (I'm pleased to note that Rainer's approach is apparently fairly similar to mine.)

In particular, I don't think Lennart and Kay are at their most brilliant when considering only Linux-based logging, where logging (to a much larger extent than, say, service management à la systemd) often goes across platform boundaries, and so I'm personally in favour of approaches that do work on various systems. This is one issue that I have with journald – not an insurmountable one but an important one.

On the other hand, while it is obviously possible to install rsyslogd on other flavours of Unix, I personally, before doing anything like that, would want to convince myself that those other flavours of Unix have not made their own tweaks to their own syslogds that rsyslogd doesn't support. With a platform that is as ill-defined and fragmented as syslog this may be an issue. Right now we have one standard for syslog that is wildly obsolete and a newer, better one that people do not seem to be exactly falling over themselves to implement, which is not a good situation either.

addressing your wish list

First of all, this isn't »my wish list«, just a few points I quoted from the proposal.

The performance issue, as far as I'm concerned, pertains to dealing with the actual logs rather than accepting messages. It is all well and good to say that I could in theory implement a log store for rsyslogd that does what I want. Journald apparently has implemented such a log store already, and IMHO actual code beats code that might eventually be written at some point. (And incidentally I did outline what I'd like to see in a log store for e-mail logs; if you'd care to point me at existing rsyslogd-based code that implements something along these lines – preferably as a package for Debian Squeeze –, then I'd be very happy.)

Right now the answer to various points in Lennart's and Kay's proposal is »yes, we could do that too, if we wanted«. Not all of these points appear to be desperately required, but once again I'm not here to prove that journald is better than rsyslogd, I'm interested in what we can learn from the journald proposal in order to improve logging in general. If the journald proposal leads to a better rsyslogd (which at least concerning trusted properties it already has) and nothing else then we have already gained something.

The journald design is horrible to the point of useless

Posted Dec 2, 2011 4:36 UTC (Fri) by dlang (guest, #313) [Link] (9 responses)

I have never said that LP is an idiot, If he was I wouldn't care what he said as such people don't influence Linux much.

I do believe that he is wrong, and I give him the benefit of the doubt and assume that he failed to research the current state of the art around syslog
(and the fact that Rainer says that LP never talked to him about the journal reinforces this), because otherwise he is being deliberately misleading in his statements.

The fact that anyone would propose such a drastic change without learning the state of the art significantly undermines the credibility of their paper as far as I am concerned.

syslog-ng and rsyslog are both commonly used on *BSD and solaris, so while you may want to do research to see if there are missing tweaks, others have done so already.

there are many different ways of storing logs, different ways have different benefits and drawbacks when accessing the logs

the traditional text files rolled periodically are great for looking at what happened around a particular time (including especially, what is happening now, or has happened just recently)

However, they are horrible at doing searches for previously unexpected information over a large volume of logs.

If you know what you are expecting to be looking for, you can tailor your log store to make it easy to find that, but assuming that you don't want to do so, a couple 'off the shelf' approaches to rapidly finding things in large log volumes are:

1. log to a postrgesql database and enable the full-text indexing (I believe it's called gist indexing), this will let you search rapidly for any text

2. log to hadoop and use it's search capabilities (including clustering) to search the logs.

in both cases this is a little more than just 'apt-get magic-tool', because you need to configure the database and then configure rsyslog to write to the database, but all the pieces are there, it really is just a matter of configuring them.

this is more than just "yes we could do that if we wanted", it's "we already created the tools to do that, and people are using the tools to do that, you can do the same"

The journald design is horrible to the point of useless

Posted Dec 2, 2011 9:04 UTC (Fri) by anselm (subscriber, #2796) [Link] (8 responses)

syslog-ng and rsyslog are both commonly used on *BSD and solaris, so while you may want to do research to see if there are missing tweaks, others have done so already.

Yes, but these are the open-source Unices. I'd be more worried about the likes of MacOS X, AIX, or HP/UX.

the traditional text files rolled periodically are great for looking at what happened around a particular time (including especially, what is happening now, or has happened just recently)

Great in that the messages in question do end up near one another. Not so great in that you need to scan a file (or a whole bunch of files) sequentially in order to find where that »particular time« is in the log. (One could probably do per-file binary search based on the log time stamps but if there is a tool to do that it is definitely not in widespread use.) Also it turns out that, like in the e-mail case, related log messages do not always happen to occur around a particular time.

I still like the systemd/journald idea of including the last few lines of log output produced by a system service in a »service status« output. This is wildly impractical under the current scheme since there is no way to tell in which file(s) a service's output even ends up (short of parsing the syslogd configuration), and even then you need to scan the file(s) sequentially to find what you're interested in. You end up having to tweak all your init scripts and/or reinvent large parts of your logging infrastructure. Systemd and journald suddenly start looking pretty sweet by comparison. (And yes, the stock answer to that is »We didn't need this for the last 30 years, it is unnecessary, nobody should bother, systemd sucks.«)

in both cases this is a little more than just 'apt-get magic-tool', because you need to configure the database and then configure rsyslog to write to the database, but all the pieces are there, it really is just a matter of configuring them.

And if the pieces weren't there, it would really just be a matter of putting them in? Seriously, I think you're making it a bit too easy on yourself here. Especially since the real problem is not logging stuff to a database (which rsyslogd can do if you ask it nicely), it is getting at the stuff afterwards – which again is something that Lennart and Kay do acknowledge in their proposal. Getting rid of old stuff would also be something one would have to consider at some point. Full-text search in databases is nice but it would be even better to be able to exploit the structure that some types of log do have, that RFC 5424 suggests, and that journald pretty much enforces. Right now this requires considerable manual work, and »we created the tools to do that« is over-optimistic – »we created the tools to enable people to build the tools to do that on their own« would be closer to the truth (and is already not a bad thing, but a lot less than what you claim).

The journald design is horrible to the point of useless

Posted Dec 2, 2011 14:18 UTC (Fri) by dlang (guest, #313) [Link] (7 responses)

When I talk about rsyslog running on Solaris, I'm not talking opensolaris. as for the other flavors of Unix, if any users of them care enough about rsyslog or syslog-ng running on that platform it will happen (either from contributed code or from paying for support). But in any case the existing syslog daemons do just fine receiving messages from those systems (and various other proprietary systems like Cisco) and then handling those logs just like all the other logs. This means that you can combine them in whatever log store you choose to use. You won't get things like trusted attributes from such systems, but that's because those systems don't offer that capability, and journald sure isn't going to solve this since it won't run on those systems (and can't make such systems provide info that isn't there in the first place)

finding which file in a series of time-rotated files a particular time is in is pretty trivial, as is finding the time in the log file once you have opened it (at least if you use a normal test editor), you can either do the binary search yourself by jumping to various line numbers, or just search for the timestamp you are looking for. You are really overstating the difficulty here. Yes this can be done wrong (rolling the logs daily, even if you have gigs of logs in a day is usually not a wise thing to do for example)

the idea that you can get the log messages when you ask for the status of a program only works in the trivial case where all the logs are written by the one pid that was started by the program you are asking about the status. If that pid started other processes that then wrote logs, systemd (or equivalent) isn't going to have a way of knowing for sure which 'service' those log messages are for.

I don't believe that the Journal is going to enforce programs writing well structured logs any more than the windows event log does (see Rainer's comment that in spite of the 'structure' being enforced by windows event log, the problem of analyzing logs on windows is the same mess that it is on *nix)

It sounds as if you have decided that anything that LP writes is the Pony that you want and any criticism of it just means the person doing the criticizing is against all progress.

The journald design is horrible to the point of useless

Posted Dec 2, 2011 17:07 UTC (Fri) by raven667 (subscriber, #5198) [Link] (1 responses)

If that pid started other processes that then wrote logs, systemd (or equivalent) isn't going to have a way of knowing for sure which 'service' those log messages are for.

I think this particular statement is probably not true. systemd uses the cgroups feature to associate all processes as part of a generic "service" or "session" which is not fooled by daemonization so it could reliably associate all logs across multiple PIDs with a particular "service".

The journald design is horrible to the point of useless

Posted Dec 2, 2011 19:05 UTC (Fri) by dlang (guest, #313) [Link]

this is a point I didn't consider, but identifying what cgroup something is in is going to be another racy lookup isn't it?

in any case, I've suggested to Rainer that he add this lookup to the trusted properties that rsyslog provides. Once that is there then it will be trivial to split the logs by service (and therefor trivial to do a tail of the most recent logs from any service)

The journald design is horrible to the point of useless

Posted Dec 2, 2011 17:16 UTC (Fri) by anselm (subscriber, #2796) [Link]

the idea that you can get the log messages when you ask for the status of a program only works in the trivial case where all the logs are written by the one pid that was started by the program you are asking about the status.

AFAIK this is wrong. Systemd puts each service into its own control group. Processes that the service generates stay in the same control group. The system status command can then consider all messages from the same control group, or the same systemd service, in order to catch everything from the service process and its children. This is actually in the proposal.

That this is a lot more difficult to do with the current infrastructure (SysV init and (r)syslogd) doesn't mean that it is, in fact, impossible in general. Systemd is not a complete exercise in futility.

It sounds as if you have decided that anything that LP writes is the Pony that you want and any criticism of it just means the person doing the criticizing is against all progress.

No. As far as I am concerned it is absolutely OK to criticise Lennart's and Kay's proposal, just as it is absolutely OK to criticise the existing syslog infrastructure and protocol. Progress results not from hanging on to the existing stuff at all costs, but from critical evaluation of both the old and any newly proposed stuff, and from putting the best ideas of both together to create something that is better than what we had before. Whether that is, in the end, a beefed-up syslogd or something entirely different is irrelevant as long as it solves the problems at hand and there is a reasonable compatibility path to the existing infrastructure. You will note that this also seems to be Rainer's attitude, although he is (for understandable reasons) somewhat biased towards the »beefed-up syslogd« end of the spectrum of possible solutions. (Which is of course OK.)

I've been using various syslogd implementations for a very long time and have usually been able to get them to do what I need, but that doesn't mean I'm blind to possible improvements that may come in from elsewhere. This IMHO is a better approach than dismissing all of a number of possible improvements outright just because one does not like the people who came up with them. No one (not even Lennart or Kay) believes that the journald proposal solves all possible problems with logging, but pretending that there are no problems with logging at all, or no serious problems, or that if there are in fact problems then the journald proposal doesn't solve them either, doesn't actually lead to progress.

The journald design is horrible to the point of useless

Posted Dec 2, 2011 17:28 UTC (Fri) by raven667 (subscriber, #5198) [Link] (3 responses)

It sounds as if you have decided that anything that LP writes is the Pony that you want and any criticism of it just means the person doing the criticizing is against all progress.

I can't speak for the OP but from from my perspective, as someone who is not personally invested in the outcome, about 80% of the comments seem to be baseless negative personal animosity against LP or an appeal to tradition against progress. Half of the other 20% are making actual technical arguments pointing out flaws in the proposal and the other half are defending LP, pointing out positives about the proposal or just advocating keeping an open mind. Those that are defending tend to be responding more the 80% than to the actual constructive criticism of the 10%

Some of the constructive criticism is very pursuasive and I'm not at all sure that the journal is the right way to go, unlike with systemd which was so obviously the right, UNIX, way to do things that I wish it was written decades ago. I can't help of think about daemontools supervise and multilog, which I used very successfully for many years, and see LP as this generation's DJB.

The journald design is horrible to the point of useless

Posted Dec 2, 2011 19:06 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

> and see LP as this generation's DJB

I think that may people would agree with you, including a LOT of the people who are being critical of LP and this proposal.

there are good reasons (and not just personal animosity or his license choice) that DJB's software did not take over the world.

LP: the new DJB?

Posted Dec 2, 2011 22:50 UTC (Fri) by anselm (subscriber, #2796) [Link] (1 responses)

there are good reasons (and not just personal animosity or his license choice) that DJB's software did not take over the world.

However, some of DJB's ideas did become popular on technical merit. Maildir comes to mind.

I don't think Lennart and Kay are doing badly in comparison at all. Systemd in particular is likely to go a lot farther than Qmail or DJBDNS ever did, simply because, whatever its detractors may say, it does have all kinds of advantages compared to SysV init (the situation is a lot more obvious than with, say, Qmail vs. other MTAs) and the two don't go for DJB-style rest-of-the-world-be-damned my-way-or-the-highway backwards incompatibility in quite the same way. After all, systemd still interfaces with SysV init scripts and traditional syslogd but in a manner that introduces interesting and useful new features.

As far as journald is concerned, we'll have to see; maybe the future will just be a closer association between systemd and rsyslogd, which wouldn't be a bad thing either.

LP: the new DJB?

Posted Dec 3, 2011 3:39 UTC (Sat) by raven667 (subscriber, #5198) [Link]

Actually when I read some of the first design docs for systems I was greatly reminded of daemontools which I have used in all my big, 24/7 systems over the last decade. I chose daemontools because of its technical merits, because of obvious deficiencies in sysv init and despite the fact the software comes from Mars with its own filesystem layout and other needless complications. We even tried using svscan for pid 1. Systemd has all the technical merits and none of the deficiencies.

I hope this journal thing leads to some improvements but I'm less certain that the journal as described is the be all and end all of logging. Let's see where it goes.

The journald design is horrible to the point of useless

Posted Nov 21, 2011 8:07 UTC (Mon) by michich (guest, #17902) [Link] (4 responses)

An attitude that says the classic syslog interface is depercated and the native journal interface is preferred.
Where in the proposal did you read about deprecating the syslog interface? I see the contrary: the syslog API syslog(3) is supported as first-class interface to write log messages, and continues to be the primary API for all simple text logging.
insisting we all use unreadable UUID
Again, from the proposal: if you don’t like them, then you don’t have to attach them to your messages

The journald design is horrible to the point of useless

Posted Nov 21, 2011 8:29 UTC (Mon) by aleXXX (subscriber, #2742) [Link] (3 responses)

If you replace "insist on using unreadable UUIDs" with "recommend on using unreadable UUIDs" this doesn't really make it better, the part with the unreadable is still there.

Alex

The journald design is horrible to the point of useless

Posted Nov 21, 2011 8:52 UTC (Mon) by michich (guest, #17902) [Link] (2 responses)

The UUID does not replace the human-readable text of the message. It is an optional addition to it.

The journald design is horrible to the point of useless

Posted Nov 21, 2011 11:47 UTC (Mon) by vrfy (guest, #13362) [Link] (1 responses)

Right, the UUIDs are just attached to general useful messages which are meant to be recognized by machines. In these cases the UUID + the key/values replace the unreliable regex log parsing.

The human readable part of the message is still passed to syslog the same way as is always was.

The text says: "Most projects will probably never generate more than 30 of these during their entire development time", which should illustrate that this is in no way *every* message.

But hey, why reading the text when you can match on a few keywords, mix some random things from your head into it, and start talking?

The journald design is horrible to the point of useless

Posted Nov 23, 2011 23:01 UTC (Wed) by dlang (guest, #313) [Link]

so why create an entirely new way to log instead of just putting your UUID in your 'regular' log message?

That newfangled Journal thing

Posted Nov 21, 2011 6:54 UTC (Mon) by k8to (guest, #15413) [Link] (6 responses)

Structured logging is a failure.

There's been so many attempts to build systems with structured logging (sometimes called 'messaging' or 'events') and always the system proves unable to anticipate the needs of real applications, and builds an inflexible overcomplicated system that no one wants.

That's why we still have text-stream-logging to this day.

Text stream logging offers some other benefits, such as simplicity, and possibility to track the stream without tight coupling.

Moreover most applications want to log to the filesystem, not to a special service. This means that most logging will be in the style of the text stream. Therefore most tools around logging will work with that, and this special journald binary format will just be in the way.

It's the wrong answer to a set of problems that don't really exist.

That newfangled Journal thing - some thinking required

Posted Nov 21, 2011 19:16 UTC (Mon) by ndye (guest, #9947) [Link] (1 responses)

Structured logging is a failure.

There's been so many attempts to build systems with structured logging (sometimes called 'messaging' or 'events') and always the system proves unable to anticipate the needs of real applications, and builds an inflexible overcomplicated system that no one wants.

That's why we still have text-stream-logging to this day.

can be, and has been, said many times:

XYZ has never worked, so it never will

Often over the "40 years" of UNIX, rethinking the fundamentals took place in a proprietary cathedral. The artificial limitation of market share limited the pool of engineers thinking about the problem and its interactions.   The non-engineering folks required to think of "market" share so the engineers could eat were also driving anti-engineering efforts to limit the implementation to a segment of brain power, by which I mean at least these two forms of intellect segmentation:

  • people who, having learned one product, can't cross the chasm of artificial differences to adapt to a new environment (I'm looking at you, AIX)
  • people who must keep unnecessary differences in their head as they rove between vendors

Because we're in the shared ecospace of Linux, the preponderance of designers and users have opportunity to influence, beginning at design rather than late beta.

Not that I'm advocating this particular reinvention -- I find lots of this distasteful:   UUID's, binary encoding, the coupling with systemd (solved?), overloading the "journal" name.

As someone else suggested, maybe Lennart should stop publishing his own thoughts and let someone more diplomatic front his ideas, vetting them for obvious mistakes like the name overloading.

Just in the last month I experienced an outage measured in days instead of hours precisely because of name overloading:   It *matters*.

Here's hoping this topic stops reminding me of a maze of twisty passages, nearly alike but not inconsequentially different, strewn with pits, tar-filled and otherwise . . . .

That newfangled Journal thing - some thinking required

Posted Nov 24, 2011 8:19 UTC (Thu) by k8to (guest, #15413) [Link]

Yeah, except sometimes it's like poo-pooing people who try to reinvent bicycle wheels. Or people who try to put a CVT on a bicycle, or eliptical gear-wheels. None of them are better at all. In fact the bicycle wheel is one of the strongest per weight human structures ever designed. And it's cheap and easy to fix! But yet some people keep trying to add on carbon struts and such and sell it as better...

In this case, using a structured format means creating code that knows about the structured format. Now you've either got to do something gross like add all the journald interfaces to the kernel, or you have to do something else gross like add a big library between the emitting code and the fwrite() call.

You want to get the logged data out of the process space immediately, and the reason you want to do that is that in the crash scenario, you want to know what was happening at the time of the crash, not 2 seconds earlier when the buffering got last pushed.

Meanwhile, you want other systems to be able to tell what's logged, so you want to be able to watch the datastream as it's landing on disk. With a binary format all of this gets fairly complicated. Everyone has to link in this clunky library, or (worse) go through another daemon so that you have another place to lose buffers when the expected failure happens, to handle which is the first driver for adding logging.

So you have to have a clunky library and it has to be race condition free with its binary format, and it has to be carefully vetted to handle all unexpected binary structures correctly, or else tail -f becomes a security attack surface -- well it does regardless but at least if you carefully vet it the probability of a problem is decreased.

And the benefit of all of this is.. what exactly?

That newfangled Journal thing

Posted Nov 22, 2011 0:09 UTC (Tue) by alankila (guest, #47141) [Link] (3 responses)

Just because not every place needs structured logging doesn't mean that nobody needs it. And those places that do need it do benefit from journal format that can handle it and provide search facilities on top of it.

The *potential* to generate and find structured messages easily is valuable. There are immediate use cases, from usage statistics on services to automatically showing latest messages from log if daemon fails to start---it's not just more work for programmers for no good reason.

We wrote something like this at work because we needed structured log stream: events our users did, and variable messagetype-specific metadata attached to every log message. It made life simpler for programmers, helpdesk and the beancounters. From this, I would predict: the people who want structured messages are probably more than willing to peek into code to add them, because the alternative is continuing the current regex parsing regime, which is inelegant and error prone.

That newfangled Journal thing

Posted Nov 22, 2011 6:53 UTC (Tue) by k8to (guest, #15413) [Link] (2 responses)

You're talking about the case of the logging application producing messages with some kind of internal structure.

I'm talking about structure imposed by the logging interface.

It's a crucial difference. With text stream logging you can easily produce structured messages as needed by the application, whether it's simple and easily readable key-value pairs, or more elaborate and designed-for-program consumption formats like JSON or (ick) XML. Text stream logging lets you do this, it doesn't impose the limitation.

With a structured interface, you're locked into what that interface provides, and cannot grow beyond it or handle things it did not anticipate.

It's a classic case of trying to apply structure from the wrong side.

That newfangled Journal thing

Posted Nov 24, 2011 16:28 UTC (Thu) by alankila (guest, #47141) [Link] (1 responses)

I disagree & find the entire argument bizarre.

The proposed format is key-value storage. Any value can be your "text stream", thereby showing that the new approach is a superset of the old one.

That newfangled Journal thing

Posted Nov 24, 2011 19:45 UTC (Thu) by k8to (guest, #15413) [Link]

And if you need to create nested enclosures that open and close?

A single blob of text inside an enclosure is not a stream.

That newfangled Journal thing

Posted Nov 21, 2011 16:38 UTC (Mon) by lothar (guest, #14052) [Link] (7 responses)

A lot of comments about the proposed new logging system seems to be complaints about the binary format. IMHO it would be a reasonable compromise to look into JSON formatting. It's text based enough to make the camp of "cat and tail -f" happy and structured enough to make it feasible to automatically process it. Another plus is that it's almost no overhead in writing (only a small amount of characters need to be escaped) and reasonably fast to parse in a log viewer/query application.

That newfangled Journal thing

Posted Nov 21, 2011 17:51 UTC (Mon) by SEJeff (guest, #51588) [Link] (6 responses)

It seems like designing something around google's protobuffers[1] for ondisk and network format would be a sane compromise between the two. I can't possibly fathom why the developers of the Journal didn't do this. It is *really* good and I don't see how they couldn't simply layer an api ontop of protobuffers and call it a day. They also get the "feature" of a bunch of tools to read-write their format and bindings in all major languages for free.

[1] http://code.google.com/apis/protocolbuffers/

That newfangled Journal thing

Posted Nov 21, 2011 18:23 UTC (Mon) by hanwen (subscriber, #4329) [Link] (4 responses)

I second the suggestion for protocol buffers. They were made for serializing structured data in a compact yet backward compatible manner, and are standarized so people won't have to worry about loosing the ability to read old log files.

Protobuffers are not ideal for this, sadly...

Posted Nov 21, 2011 18:58 UTC (Mon) by khim (subscriber, #9252) [Link] (3 responses)

Actually Google itself uses protobuffers for logging, but it'll not work outside of Google.

The problem here is that protobuffers separate description of structure and the data. This means that you can not add new type of log without changing the protobuf definition. Works fine if all log types are kept in central repo (log reader just includes all possible log protobuffers from all possible programs), but will probably not work for wide world where you must handle logs from different, independently written programs.

Protobuffers are not ideal for this, sadly...

Posted Nov 22, 2011 0:28 UTC (Tue) by dashesy (guest, #74652) [Link] (2 responses)

Protobuffers seem very interesting! with APIs for C it should fit well with kernel logging as well.
"but will probably not work for wide world where you must handle logs from different, independently written programs"
It seems you know better, but skimming through the docs (for my own validation) it seems there are extensions, so the log reader can have access to a generic definition (that comes with kernel) and independent programs can extend it. Sysadmins can install and (keep updated possibly through repos) the definitions for programs they care most.

Another extension that would be interesting is a reverse .proto generator, where I can supply in my protocol structures header file and .proto is generated/updated based on the last release.

Right, you can do it that way...

Posted Nov 22, 2011 8:25 UTC (Tue) by khim (subscriber, #9252) [Link] (1 responses)

It seems you know better, but skimming through the docs (for my own validation) it seems there are extensions, so the log reader can have access to a generic definition (that comes with kernel) and independent programs can extend it. Sysadmins can install and (keep updated possibly through repos) the definitions for programs they care most.

Well, it should be possible, I'll grant you that. It'll require some scheme to keep all that in sync. Google itself does not have anything like that because they don't need it (centralized repo have it's advantages).

Another extension that would be interesting is a reverse .proto generator, where I can supply in my protocol structures header file and .proto is generated/updated based on the last release.

Not sure what you meant by that. Why will you want to reverse-engineer .proto file? ProtoBuffers are intentionally limited, you can not just shove arbitrary structure in them...

Right, you can do it that way...

Posted Nov 22, 2011 17:02 UTC (Tue) by dashesy (guest, #74652) [Link]

I guess many will object to the idea of ProtoBuffers (specially in kernel) because of its reliance non-mainstream compiler/binutils. Anything that requires more than gcc will probably not get a lot of support, specially if/when it is used only for the logging backend.

Maybe I should have rephrased my wish this way:
A gcc plugin that compiles .proto files directly will be very interesting.
Having Qt in mind, C lovers do not like to see not-exactly-C syntax in their code. Therefore a combination of limited (yes non-arbitrary) C structures and compiler pargmas (for example to specify that a field is optional, and what is the default) instead of .proto syntax, along with the above mentioned plugin will be even more attractive, and likely to be taken into kernel. Here I assumed kernel hackers love C, and I should confess I do not know how powerful gcc plugins are.

That newfangled Journal thing

Posted Nov 21, 2011 20:24 UTC (Mon) by nteon (subscriber, #53899) [Link]

protocol buffers are designed for inter-message compaction, while it seems like their on-disk format is designed for intra-message compaction (string de-duplication, etc). Although protocol buffers + gzip might work just fine.

Worst. SEO. Ever.

Posted Nov 21, 2011 20:05 UTC (Mon) by dmarti (subscriber, #11625) [Link] (3 responses)

I'm not going to burn my Lennart Poettering fanboy card or start running Get Off My Lawn Linux (featuring fvwm and dmix!), but won't somebody think of the web searchers of the future? "Linux Journal" and discussions of filesystem journals have massive Google juice by now, so unless this thing gets a less common name it might be invisible.

Lennartlog?

Worst. SEO. Ever.

Posted Nov 24, 2011 11:30 UTC (Thu) by dag- (guest, #30207) [Link] (1 responses)

Haha :-)

Pulselog ?

Worst. SEO. Ever.

Posted Nov 25, 2011 4:58 UTC (Fri) by cas (guest, #52554) [Link]

great big steaming log ?

Right on

Posted Nov 25, 2011 16:45 UTC (Fri) by man_ls (guest, #15091) [Link]

Finally, a sensible comment! You put the finger on the worst problem of "The Journal", as if the article "The" was going to save it. Guess in what position of the 105 million current Google hits does Poettering's proposal appear? Not in the first three pages for me. Journald has only 10 million but the results are quite eclectic, not a good choice either. "Linux Journal" has only 1 million but meaning something else.

I am not making suggestions to avoid NIH syndrome triggering, but please, please come up with something else.

I don't like the approach

Posted Nov 21, 2011 20:32 UTC (Mon) by dskoll (subscriber, #1630) [Link]

I think something has to be done to organize logging, but I do not like a binary log format. It's a non-starter for me.

We index our log files and build structured data describing them, but all the indexes and meta-data are stored in separate files. If you have the separate files lying around, you can use tools to quickly search the logs, extract date ranges, see structured data, etc. But if these binary index files are missing, you still have the raw log data available to standard tools like grep and friends.

I don't see why the structured logging problem can't be solved by adding to the existing log files instead of replacing them with a new binary format. You can implement the same API as you would with the binary-only log format, but you get the benefit of plain-text log files for emergency situations.

Dependency creep

Posted Nov 22, 2011 6:39 UTC (Tue) by grg (guest, #76756) [Link]

* Journald can't be untangled from systemd (from the FAQ).
* Applications _will_ start using this, thereby depending on systemd.
* Systemd already depends on things I may not want/need installed on my system (http://cgit.freedesktop.org/systemd/tree/README).

That newfangled Journal thing

Posted Nov 22, 2011 21:01 UTC (Tue) by cmccabe (guest, #60281) [Link] (1 responses)

Forget syslog. Clearly, what we need is a new commenting system for LWN.

Luckily we can apply the same design principles here:

1. In the new system, comments will have to be structured. For now, the categories will be GNOME3 flames, RPM vs. DEB flames, strident messages about how X violates the spirit of UNIX, and "other." More categories will be added later as needed.

2. The comments will be stored in an undocumented binary format. In the short term, this will prevent anyone from reading them. However, the security will be much higher!

3. You'll have to run systemd to make a comment. Luckily, access will be provided through a shared library and a D-BUS interface.

(Lennart, if you're reading this, don't take it too serious)

That newfangled Journal thing

Posted Nov 29, 2011 21:12 UTC (Tue) by BenHutchings (subscriber, #37955) [Link]

You left out KDE4, you DE bigot!

That newfangled Journal thing

Posted Nov 24, 2011 5:58 UTC (Thu) by russell (guest, #10458) [Link] (4 responses)

If it's lasted 40 years, it must be doing a lot of things right. Whoever had the vision to build it should be proud of inventing a system that lasted that long.

That newfangled Journal thing

Posted Nov 24, 2011 8:57 UTC (Thu) by anselm (subscriber, #2796) [Link] (3 responses)

The syslog protocol is similar to, say, SMTP. It was invented a long time ago when issues like security and extensibility weren't as important as they are today. It is very simple, something somebody might write down on a paper napkin. It does a reasonable approximation of what it was supposed to do. There is widespread consensus that it sucks in important respects, but it is so entrenched that it is essentially impossible to replace with something radically different and better. »It's been around for so long, it's got to be good« is not something I would want to say about either.

(At least in the case of SMTP, we have fixed some of the more glaring shortcomings, but there are still lots of problems with the current incarnation that are probably going to stick around for a very long time. With syslog, nothing of the sort is in evidence – we still have the hard-coded facility list we used to have, and there is no provision for authentication, to name but two existing issues.)

That newfangled Journal thing

Posted Nov 24, 2011 11:20 UTC (Thu) by dlang (guest, #313) [Link] (1 responses)

you are correct that the original syslog protocol, RFCs, and implementations were seriously lacking in security, reliability and performance.

However, syslog-ng and rsyslog have added many options that address these concerns, all in ways that are optional extensions from the base, and require no significant changes if you don't use them (the syntax of the config file is the biggest change, and rsyslog lets you ignore most of that if you want to)

yes, there is still the hard coded facility and severity lists, however I don't think that anyone who is doing real serious work with syslog really uses those, instead the far more flexible filtering of the modern syslog daemons is used instead.

go take a serious look at rsyslog (the default on just about every distro, even if some of them have old versions). It has a lot of capabilities that you would not have thought of a few years ago.

That newfangled Journal thing

Posted Nov 24, 2011 15:15 UTC (Thu) by anselm (subscriber, #2796) [Link]

I'm familiar with rsyslogd, thank you very much. However, I think any long-overdue improvements to the logging infrastructure should not depend on running specific software packages at either end. In particular, rsyslogd is still being fed free-format, unauthenticated syslog-style messages – it may be quite a bit more clever in dealing with them than plain syslogd, but there are still opportunities for improvement that rsyslogd alone does not address.

I'm not saying that Lennart's and Kay's proposal should immediately be adopted by everyone. I don't even think Lennart and Kay expect that. We should really consider it as a starting point for tests and a reasonable discussion, rather than dismiss it outright as »heresy« or some of the other things that have been mentioned here.

That newfangled Journal thing

Posted Dec 1, 2011 23:29 UTC (Thu) by jimi (guest, #6655) [Link]

> There is widespread consensus that it [syslog] sucks in important respects

There is?

That newfangled Journal thing

Posted Nov 24, 2011 19:46 UTC (Thu) by jcm (subscriber, #18262) [Link]

I'm looking forward to seeing all of those working on such technology attending Lisa in Boston shortly. That will of course be a great opportunity to get feedback from end-users of the technology.

blog entry on journal by the primary author of rsyslog

Posted Nov 25, 2011 4:05 UTC (Fri) by dlang (guest, #313) [Link] (6 responses)

http://blog.gerhards.net/2011/11/journald-and-rsyslog.html

blog entry on journal by the primary author of rsyslog

Posted Nov 25, 2011 9:23 UTC (Fri) by anselm (subscriber, #2796) [Link] (5 responses)

Sensible man. I think many of the advocates of »syslog is the One True Way and may never change« would do well to read that item.

blog entry on journal by the primary author of rsyslog

Posted Nov 25, 2011 9:30 UTC (Fri) by dlang (guest, #313) [Link] (2 responses)

I am not saying that syslog should never change. (I have been party to changing syslog, both through writing code and through getting my company to pay for code to be written), the fundamental problem isn't that there is change, it's the lack of compatibility of the change.

and no, saying that you can run two logging systems doesn't make it compatible. that makes it possible to co-exist (for now), which is a very different thing.

and the reason I say "for now" is that the advocates of the new system are extremely dismissive of everything on the old system, and of anyone who likes it, so why will they keep going to any effort to keep this co-existance possible? See Gnome for a perfect example of this. they told people that if they didn't like the new Shell they could just use fallback mode so don't complain, now they are saying that fallback mode is strictly temporary.

blog entry on journal by the primary author of rsyslog

Posted Nov 26, 2011 18:32 UTC (Sat) by rahulsundaram (subscriber, #21946) [Link] (1 responses)

You are wrong about GNOME. GNOME developers never have claimed that the fallback mode is anything other than a temporary thing. You are confusing random posters with GNOME developers. This is misleading.

blog entry on journal by the primary author of rsyslog

Posted Dec 1, 2011 8:43 UTC (Thu) by ovitters (guest, #27950) [Link]

It was given the name fallback to make clear it is not the standard mode. Intention from the start was to keep it for the entire GNOME 3.x. However, amount of maintenance it needs was severely underestimated. This coupled with low usage (though impossible to measure) is the reason the maintenance might be given up earlier. Still not decided though.

And if you want it more official:
http://mail.gnome.org/archives/release-team/2011-November...

Please note the lack of reply on that :P

blog entry on journal by the primary author of rsyslog

Posted Nov 26, 2011 17:41 UTC (Sat) by cdmiller (guest, #2813) [Link] (1 responses)

And those advocates of >>anything Lennart says regarding systemd/journal is the One True Way and no one else could possibly know better<< would also do well to read that item and the follow up postings.

blog entry on journal by the primary author of rsyslog

Posted Nov 27, 2011 4:52 UTC (Sun) by raven667 (subscriber, #5198) [Link]

I can't say I've seen any actual posts like that in the LWN discussions of Lennart's proposals. That's a pretty transparent straw-man used to justify your feelings on the matter but not representative of reality.

What I have seen is some people who think we should read and discuss the proposal, the pros and cons, in a serious and adult manner and others who think the proposal should be dismissed out of hand, either because it's not "traditional" or out of animosity to anything proposed by Lennart. Which camp are you in?

That newfangled Journal thing

Posted Nov 25, 2011 9:04 UTC (Fri) by zougloub (guest, #46163) [Link]

The performance+security advantages of structured binary data are undeniable.
Yes, you can get performance by re-reading syslogs and generating indexes...

People get scared because they use cat/grep, but most setups include the *ugly* logrotate, so you *zcat anyway.
In the end, for people who like text, a "journalc log --pretty=xxx" will probably be available...

What bothers me is:
- If journald wants to become a defacto standard, then the file format should be discussed, others may have some insight on this. Other logging/tracing/generic file formats exist, and people would appreciate if they could be referenced so that we know they have been reviewed (and the wheel is not reinvented) when designing a new file format.
Where's the mailing list, and the wiki for the design info ?
Right now it seems pretty obscure.
The file format should consider other software implementations
Nobody wants it to change in 1 year, but it should be future-proof.
- Despite what the release document says, journald could be separated from systemd, as a separate library
- Finally, and this has nothing to do with journald, software that don't provide syslog logging and write their own logs without any good reason !

Lennart, please apply for a job with AAPL.

Posted Nov 25, 2011 11:40 UTC (Fri) by wileyc (guest, #62284) [Link] (1 responses)

Lennart ... if you're reading this ...

... please contact your local Apple HR recruiter and apply for a position in the Core OS team.

You're very clearly intent on copying Apple UNIX innovations into Linux. After a few weeks in Core OS, you might rethink the appropriateness of your Linux projects. Or perhaps not, but you'll be too busy to screw around with interfaces that worked just fine before you arrived on the scene.

Seriously, join Apple. You'll be able to pervert UNIX best practices to your heart's content, all in the name of "progress". You'll be safe and happy in that little bubble of groupthink.

Lennart, please apply for a job with AAPL.

Posted Nov 25, 2011 12:06 UTC (Fri) by anselm (subscriber, #2796) [Link]

Seriously, if you're that adamantly opposed to systemd etc., there are Linux distributions that do not seem to subscribe to Lennart's flavour of Kool-Aid now, and have no apparent plans to do so in the future. (Slackware comes to mind.) Why don't you simply use one of those instead of telling other people where they should direct their creativity?

In the worst case, if at some point even Patrick Volkerding should decide to defect from the faithful, you and any other Lennart-haters can always band together and maintain your own Linux distribution that is true to the spirit of Unix (as defined by you) and free of any external contamination. It will undoubtedly prove very popular indeed. Lennart Poettering has no power over you in the end.

Anyway, systemd and »The Journal« may be good ideas in the long run or they may not. Some people do think they look promising. Others don't. Why not treat them like other innovations in Linux and let them sink or swim on their own merits? If they really suck that hard, people will come off them again soon enough. On the other hand, if they do improve things then more power to us.

That newfangled Journal thing

Posted Nov 25, 2011 15:41 UTC (Fri) by SEJeff (guest, #51588) [Link] (1 responses)

Interesting take by the rsyslog author:

http://blog.gerhards.net/2011/11/journald-log-hash-chaini...

That newfangled Journal thing

Posted Nov 29, 2011 18:02 UTC (Tue) by cmccabe (guest, #60281) [Link]

I usually agree with what Rainer posts, but I think he has it wrong this time. You periodically save the latest hash, not the first hash. Saving the first hash would provide no security whatsoever.

Of course, there are still complicated questions surrounding how often you save this latest hash, and whether the hashing function is strong enough and fast enough, etc.

interoperation with syslog, only on systemd's terms

Posted Dec 3, 2011 19:21 UTC (Sat) by dlang (guest, #313) [Link]

I found this discussion interesting.

Rainer is asking for information on how to interoperate with the journal and the answers (or lack of them) is interesting, especially the way that the journal refuses to provide information that it has to syslog (and in the process, prevents syslog from gathering the information on it's own)

they say that there would be no reason to create the new mechanism if syslog supports structured data, but they seem to then ignore information saying how to do so through established standards.

http://lists.freedesktop.org/archives/systemd-devel/2011-...

tamper-resistant log store available outside of the journal

Posted Dec 12, 2011 21:39 UTC (Mon) by dlang (guest, #313) [Link]

Rainer (the primary author of rsyslog) has released some scripts to do chained hashing of log entries. they work with any stdout/stdin type tools and leave the logs still grepable.

http://logtools.org/2011/12/12/logtools-0-1-0-released/

That newfangled Journal thing

Posted Dec 22, 2011 11:59 UTC (Thu) by Fowl (subscriber, #65667) [Link]

The way I see it, The Journal is more like a local preparsed/indexed cache than a syslog replacement.

At least at this point.

That newfangled Journal thing

Posted Dec 31, 2011 6:44 UTC (Sat) by technosaurus (guest, #82040) [Link] (1 responses)

you can have a flat file _AND_ be easily indexable

UUID=1234-dsa-fweqwe-rwqwer MSG="DEBUG: module xxx loaded" USELESSVAR="I need this for my init hack" BLOBSIZE=12345678 \n

bzcat $FILE | while read LINE ...
case "$LINE" in
*MSG=*)eval $LINE;;
#do more stuff here
*)#treat as binary data or non conformant message
esac
done |bzip -9c >$FILE~ && mv -f $FILE~ $FILE

replace bzip with your favorite compression (bz seems to work better than xz on these type of files though)

you can read the data stream from after the \n for BLOBSIZE bytes
tricky, but doable ... it really should be in a separate file so that the logfile is viewable in a standard text editor

The binary file log could work like this (but I think it is stupid unless you want to tack it on to an avi file as a type of poor man's steganography - use a directory with a unique file name that is identified in the message for crying out loud)
echo '
UUID=1234-dsa-fweqwe-rwqwer MSG="DEBUG: module xxx loaded" USELESSVAR="I need this for my init hack" BLOBSIZE='`stat -c %s myblob` >> /tmp/bloblog
cat myblob >>/tmp/bloblog
(or a similar C equivalent)

then to access the blobs you can use sed (for removing text lines with the messag) + head/tail (using -c $BLOBSIZE to get the blob) ... not the fastest way in the world
... but why not just use a damn directory and a unique file name .../sys/blobs/myblob (it even has a time stamp that could be cross referenced - imagine that)

That newfangled Journal thing

Posted Dec 31, 2011 7:05 UTC (Sat) by technosaurus (guest, #82040) [Link]

forgot to mention that it lends itself to being easily capable of storing in any database format with a simple scripted daemon


Copyright © 2011, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds