From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
Posted Jan 30, 2014 23:49 UTC (Thu) by zuki (subscriber, #41808)In reply to: From anti-systemd to pro-systemd in the shortest time by cdmiller
Parent article: This week in "As the Technical Committee Turns"
2. You run your script. ssh config file has a fatal error. Result the same as above.
3. Let's say that you want to connect to the ssh daemon from a second script. You start the script, only to learn that sshd hasn't actually started yet. You stick a 'sleep 1' in the other script. Everything works, on the faster machines, fails on the slower ones.
4. sshd is unexpectedly killed by oom killer. You run 'service sshd start; service sshd stop' in quick succesion. Ooops, the start script returned before sshd manager to write its pidfile. The stop script kills a random program which got the same pid meanwhile.
5. sshd has hung. You run the stop script. The script send TERM, but the program is seriously stuck and will only react to KILL. Nevertheless, you are notified about success.
Yay for reliability!
Posted Jan 31, 2014 0:56 UTC (Fri)
by hummassa (subscriber, #307)
[Link] (72 responses)
Posted Jan 31, 2014 1:16 UTC (Fri)
by dlang (guest, #313)
[Link] (71 responses)
Posted Jan 31, 2014 1:23 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (70 responses)
Posted Jan 31, 2014 1:36 UTC (Fri)
by dlang (guest, #313)
[Link] (69 responses)
systemd is not the be-all end-all of software, it does have limits. When you hit those limits, or you have a different model of how you want your system to work than the systemd developers imagined, you now have to fight systemd
I know of one case (that I won't go into here) where people are looking at having to do LD_PRELOAD's on everything that's run to work around systemd
Posted Jan 31, 2014 1:38 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (67 responses)
You can always fall back to SysV-init scripts for something too perverted to be started by systemd.
Posted Jan 31, 2014 1:48 UTC (Fri)
by dlang (guest, #313)
[Link] (66 responses)
Posted Jan 31, 2014 2:05 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (65 responses)
The only unique feature that is non-optional on systemd are cgroups. And right now you can construct a parallel cgroups hierarchy just fine. The future plans for The One World Order^H Cgroups tree would stop this, but it's not a direct fault of systemd.
Posted Jan 31, 2014 2:12 UTC (Fri)
by dlang (guest, #313)
[Link] (64 responses)
you are in effect saying that if you want to do something that systemd doesn't want you to do, you don't really want to do it.
we should limit our future activities to only what the systemd developers envision is the one true way to do things.
the issue I know about is different, but it is performance related similar to the cgroups example I gave above. I've been asked to keep quiet about it for a bit while they try to work out options with redhat, but from what I'm hearing, the workarounds are not pretty (the LD_PRELOAD seems like the 'clean' way to do things)
But even if this gets worked out and systemd gets changed to make this particular thing easy to do, the attitude that if systemd doesn't support it you should just not do it is one of the big reasons for people to be opposed to systemd.
Posted Jan 31, 2014 2:16 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (18 responses)
>you are in effect saying that if you want to do something that systemd doesn't want you to do, you don't really want to do it.
> the issue I know about is different, but it is performance related similar to the cgroups example I gave above.
> I've been asked to keep quiet about it for a bit while they try to work out options with redhat, but from what I'm hearing, the workarounds are not pretty (the LD_PRELOAD seems like the 'clean' way to do things)
Posted Jan 31, 2014 2:32 UTC (Fri)
by dlang (guest, #313)
[Link] (17 responses)
Posted Jan 31, 2014 2:36 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Jan 31, 2014 2:37 UTC (Fri)
by zuki (subscriber, #41808)
[Link] (6 responses)
Posted Jan 31, 2014 2:46 UTC (Fri)
by dlang (guest, #313)
[Link] (5 responses)
so yes, it is a systemd related problem
it's the systemd developer's fault because they mandate that if you use systemd, systemd will do this.
This isn't a kernel issue, this is sticking strictly with userspace components.
Posted Jan 31, 2014 3:18 UTC (Fri)
by dlang (guest, #313)
[Link] (4 responses)
It's also something that a desktop user is almost never going to run into, and it is a result of physics, not sloppy programming or something like that. The only way to avoid the problem is to sidestep the mandatory systemd component.
Posted Jan 31, 2014 3:24 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link]
Posted Jan 31, 2014 5:37 UTC (Fri)
by raven667 (subscriber, #5198)
[Link]
Posted Jan 31, 2014 19:07 UTC (Fri)
by jspaleta (subscriber, #50639)
[Link] (1 responses)
-jef
Posted Jan 31, 2014 23:41 UTC (Fri)
by dlang (guest, #313)
[Link]
Posted Jan 31, 2014 9:18 UTC (Fri)
by nzjrs (guest, #35911)
[Link] (8 responses)
Posted Jan 31, 2014 21:42 UTC (Fri)
by dlang (guest, #313)
[Link] (7 responses)
because he's not arrogant enough to believe that he has the answer to all the worlds problems and so he wouldn't say that he needs to take over all logging to begin with, but if he did create a way to intercept the logs and offer the capabilities of the journal, he wouldn't declare it mandatory and make it so hard for long-term successful usage patterns to continue.
The initial journald manifesto showed a drastic lack of knowledge about the current state-of-the-art and best practices in logging and the capabilities of syslog tools. Just about everything that was stated was true for the old sysklogd that used to be the default, but syslog-ng has been around a long time, and around 2006/2007 Rsyslog became the default for just about all distros, and the capabilities of Rsyslog and syslog-ng are far greater than the systemd developers imagined.
Posted Jan 31, 2014 22:29 UTC (Fri)
by raven667 (subscriber, #5198)
[Link] (1 responses)
Yeah, I kind of thought that too, the journal is the weakest part of the systemd tools, much of its functionality can and was added to rsyslogd. I'm not sure that trying to add all the features of rsyslogd or syslog-ng into journald makes sense, although keeping journald as a small systems logger that can run in early boot (PID 2) and make sure that nothing is lost is a good thing I think, it should just be able to move out of the way if you need high performance. Or maybe a new high-throughput protocol between journald and your syslog daemon using shared memory or kdbus or something, something to reduce context switch overhead for those highest of high throughput systems.
Posted Jan 31, 2014 23:21 UTC (Fri)
by dlang (guest, #313)
[Link]
But once you move to a system that's providing services to others, you _very_ quickly get to where the advantages over journald dominate.
As for a high speed interface directly to rsyslog, this will help, avoiding the trip to disk and the need to read and parse the binary format would be a win (there's already been a 'cause rsyslog to fill your disk' bug as the result of journald errors where corruption of the binary format causes the 'next message' pointer to point backwards, and the hack of ignoring it if it points backwards is just that, a hack)
But you still have the situation where the log is written to the socket, then journald reads from the socket into memory, does lookups for the trusted properties, and then will need to format these two sets of data to be sent to rsyslog, send them to rsyslog, and then rsyslog parse the trusted properties out.
that's a lot more steps and copies of data than application writing to a socket, rsyslog reading from the socket and looking up the trusted properties.
Posted Jan 31, 2014 22:42 UTC (Fri)
by cortana (subscriber, #24596)
[Link] (3 responses)
One of the reasons I like journald is because it captures a lot of useful metadata along with each message. Does rsyslog do something similar?
FYI, the fields starting with an underscore are obtained by journald from the kernel, so can be considered unforgeable. The other fields are supplied by the process logging the message, so can't be trusted.
I certainly think that journald has room to improve, but I don't understand the attitude that it's some pointless NIH project that gets in the way of real logging. At the end of the day you can tell journald to set Storage=none and it mostly gets out of the way.
Posted Jan 31, 2014 23:29 UTC (Fri)
by dlang (guest, #313)
[Link] (2 responses)
Rsyslog will get the UID, GID, PID, and path to binary, and name as visible to top (which I think is what you list as command line)
http://www.rsyslog.com/what-are-trusted-properties/
If there are other things that need to be pulled, they can be added (as it notes in that link), just identify what you are interested in and give some sort of explanation of why. Pulling every possible hunk of data takes time and space, so if there's a desire for a lot, it probably will need to be configurable.
mot of the other things that you list are things available to rsyslog already.
This had been discussed between Rainer (lead rsyslog developer) and Lennart prior to the announcement of journald, but Rainer had not added it to Rsyslog yet as of the time of the journald announcement.
> I don't understand the attitude that it's some pointless NIH project that gets in the way of real logging
the fact that there was so much FUD and misinformation in the initial announcement and follow-ups is the cause of a lot of this.
Posted Feb 1, 2014 1:00 UTC (Sat)
by cortana (subscriber, #24596)
[Link] (1 responses)
Posted Feb 1, 2014 2:38 UTC (Sat)
by dlang (guest, #313)
[Link]
Posted Jan 31, 2014 23:53 UTC (Fri)
by anselm (subscriber, #2796)
[Link]
The systemd/journald combination does have its good sides, including things that the traditional logging infrastructure simply does not do (like forwarding stderr output from services to the journal, or making recent log entries available with service status output). It is difficult to foresee every single use case for a piece of software like systemd that has so many. We know that the systemd people aren't interested in portability patches, but that doesn't imply that they generally aren't interested in suggestions for features they didn't happen to think of by themselves.
Also, possibly there is a solution for the problem in question that doesn't involve LD_PRELOAD. The filesystem-namespace idea, for example, sounds reasonably like something systemd could support if it doesn't do so already. I'm pretty sure that, given a little time, this will be figured out.
Posted Jan 31, 2014 2:23 UTC (Fri)
by zuki (subscriber, #41808)
[Link]
Native systemd initialization protocol is pretty flexible too. For the more complicated case, you can have a #!/bin/bash script somewhere that does whatever magic you want, taking as much time as necessary, at some point notifies systemd about the master PID and exits.
But you seem to need even more than that... I guess my imagination is lacking.
Posted Jan 31, 2014 11:21 UTC (Fri)
by nelljerram (subscriber, #12005)
[Link] (43 responses)
So, if I've understood correctly,
- you're aware of a program whose needs aren't currently met by systemd
- discussions are underway with the systemd folk on how to enhance systemd to meet those needs.
Where's the problem, then?
> But even if this gets worked out and systemd gets changed to make this particular thing easy to do, the attitude that if systemd doesn't support it you should just not do it is one of the big reasons for people to be opposed to systemd.
But the actual attitude appears to be different from your last sentence: they've recognized a new need and are working on meeting it.
Posted Jan 31, 2014 21:31 UTC (Fri)
by dlang (guest, #313)
[Link] (42 responses)
journald is considered a mandatory component, you are not allowed to eliminate and bypass it.
But if you are actually going to use your logs, especially in an enterprise environment, you need to get your logs to a central server (or at least to other software)
but routing the logs through journald involves a lot of overhead and for a high volume service writing logs (i.e. a production service), this can be crippling.
without journald, the process of getting your log to syslog is:
syslog() write socket, context switch , syslogd read socket
with systemd the minimum process conceptually is something like:
syslog write socket, context switch, journald read socket, journald persists it's own files (at least one switch to kernel land), syslogd write mimiced socket, context switch, syslogd read socket
in practice, I've heard reports that it can take as many as 11 writes from journald to persist the data to the files.
the fundamental problem is that routing through journald will involve a lot of copies of the data and context switches as the data moves around. no amount of optimization can eliminate this. The current real-life is even worse.
The fix that appears to be what is being settled on is to create a library to LD_PRELOAD that redirects the syslog call from /dev/log that journald takes over to some other socket so that the logs can bypass journald.
This is a concrete example, and the example of ssh/cgroups is another good example, but the point I was trying to make when I made the initial comment was that systemd is making statements about what the right way to do this is, and I don't care who makes such statements, there will always be a time and place where the best thing to do is to not do things that way. This is almost always under conditions that the person making the statement never imagined.
The best systems not only acknowledge this, but they make it easy to escape their limitations (ideally without having to throw out your entire system in the process)
the outraged response that I got on this thread that there may possibly be some reason to do things in a way other than the systemd way is a very bad sign as far as I'm concerned
Posted Jan 31, 2014 21:54 UTC (Fri)
by Cyberax (✭ supporter ✭, #52523)
[Link] (18 responses)
Additionally, anything that spams journald by the sheer amount of context switches is waaaay too broken.
Posted Jan 31, 2014 22:21 UTC (Fri)
by dlang (guest, #313)
[Link] (17 responses)
For many production systems, logs represent activity that needs to be archived, correlated, etc.
It's not just the context switches, it's the need to go through disk I/O
and while you say that journald can be disabled, other folks (ones in direct content with RedHat on this issue) are saying differently, that disabling journald is not a supported configuration to run systemd in.
Posted Feb 1, 2014 1:23 UTC (Sat)
by lsl (subscriber, #86508)
[Link] (1 responses)
Still, I really wonder how much log activity are we talking about here if that leads to performance issues.
Posted Feb 1, 2014 2:36 UTC (Sat)
by dlang (guest, #313)
[Link]
Rsyslog sometimes handles very high log volumes, hundreds of thousands of logs/sec (properly configured you will saturate gig-E before Rsyslog runs out of capacity)
the added overhead here is rather substantial, and on a active server it can be a problem. The problem would be that the system just wouldn't be able to handle as much load, but I don't know at what point the journald processes would bog down. Is journald multithreaded? if not, I can easily see it maxing out a single core long before the rest of the server is saturated.
Posted Feb 1, 2014 12:04 UTC (Sat)
by khim (subscriber, #9252)
[Link] (14 responses)
No. I'm assuming messages sent to syslog represent events which affect the whole system: some process have started or dies, someone did dangerous operation (started suid binary, e.g.), etc. They are meant to be consumed by human and thus should not overwhelm human. If they can owerwhelm and journald then human has no chance. The rule of thumb is simple: if you fear that your message may disappear (because someone will crash your process, or you are in inconsistent state, etc) then you send it via syslog(3), if it's informational message then syslog(3) is just wrong interface. We, too, are generating many megabytes of logs. Of course they are not sent via syslog(3)! They are first collected in-process (similar messages are coalesced—this is feasible because they are not just text but have some internal structure), then they are collected by specialized process and sent away in batches via network. This is sane approach for extremely high amount of logs to be processed by some automated scripts. To push all that via syslog will be just crazy. XKCD comes to mind, really: when you've noticed that rsyslog calls gettimeofday() you first though should have been not “why is it doing that?” but “why noone else cares about that?”.
Posted Feb 1, 2014 12:17 UTC (Sat)
by zdzichu (subscriber, #17118)
[Link] (13 responses)
Sounds like you are describing job for audit subsystem, not syslog.
Posted Feb 1, 2014 12:43 UTC (Sat)
by khim (subscriber, #9252)
[Link] (12 responses)
Posted Feb 1, 2014 17:13 UTC (Sat)
by dlang (guest, #313)
[Link] (11 responses)
syslog can be an extremely capable logging system, and log sent to syslog are not the limited, intended only for humans, thing that you think they are.
you are thinking of syslog as it existed 15-20 years ago, not what Rsyslog, syslog-ng, nxlog, and logstash have all been doing for years.
I've run Rsyslog at gig-E wire speeds (~400,000 logs per second) and other people with faster networks have run it at over 1,000,000 logs per second. This will keep up with logging information about every RPC processed by your daemons.
And people do just that on a routine basis.
Posted Feb 1, 2014 18:42 UTC (Sat)
by raven667 (subscriber, #5198)
[Link] (2 responses)
There is definitely a niche where new development was needed, to reliably capture data from the earliest part of boot, to record _all_ of the available meta-data the kernel can provide and to make the data at rest more easily searchable and tamper resistant. How much of this could work within the framework of the existing competitive logging utilities or needed new development is a value judgement and matter of opinion but while I think that the new journal implementation is fine for small systems, for large systems I'd rather extend the existing dominant log daemons to be able to handle the cases that journald picks up or have an early hand off.
Syslog has clearly defined inputs and outputs and is more amenable to multiple implementations being drop in replacements for one another than some of the other parts of init and systemd.
Posted Feb 1, 2014 19:49 UTC (Sat)
by khim (subscriber, #9252)
[Link] (1 responses)
Syslog's “clearly defined inputs and outputs” include specialized marks for “ftp daemon” and “USENET news subsystem” but have no marks for HTTP server. Nuff said. I can understand why one will want to plug syslogd interface into these modern solutions (these legacy systems needs to be supported, too), but to actually keep that joke of the interface around and build everything around it using larger and larger pile of hacks… that's just sad.
Posted Feb 2, 2014 22:31 UTC (Sun)
by dlang (guest, #313)
[Link]
Posted Feb 1, 2014 19:41 UTC (Sat)
by khim (subscriber, #9252)
[Link] (7 responses)
True. I do understand logging at the Amazon/Bing/Google/Yandex level, though. I don't doubt that for a second. This is just an application of first part of the third fundamental networking truth, though (with sufficient thrust, pigs fly just fine). From my experience “enterprise” guys tend to forget about second part (However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead).
When scalability is concened the best approach is Jeff Dean's rule: design for ~10X growth, but plan to rewrite before ~100X. Syslog design was not intended to handle such use-cases and while it it's possible to stretch it to cover that use case it's really, really, REALLY bad idea. Nope. I'm looking on all that activity in wonder and think: are they just stupid or terminally insane? Syslog was not designed for such usecases. It passes data around via sockets, and sockets are just not designed to handle such usecase. Also if you are generating so much similar data (don't tell me all these 400'000 records are totally dissimilar, I'll not believe it for a second…) there are wast coalescing opportunities. And it's important to use them because when you need to send such massive amount of data around you want to send it in batches. Not just context switches are constly at these rates, but transfers between different CPUs in today's SMP systems are constly, too! May be. But how much overhead will it incur on typical Google server with 4x10GbE (perhaps 4x100GbE by now, I'm not sure)? Is it wise to spend this much computing power on logging?
Posted Feb 1, 2014 20:30 UTC (Sat)
by raven667 (subscriber, #5198)
[Link] (6 responses)
Posted Feb 1, 2014 21:26 UTC (Sat)
by khim (subscriber, #9252)
[Link] (5 responses)
This is part of the answer but it's not the whole answer. “Big data” firms quite often are using “software from multiple vendors” too. And quite often it needs complex massaging to make sure it behaves as it should in the appropriate context. I never worked in the “enterprise” but I have quite a few friends and collegues who did and from their explanation the real problem is related to in-the-enterprise-politics: you can not just go and say “our use of X is wrong, we need to replace it with Y or Z”. Because this will make the one who mandated X angry and may undermine it's authority. If there are a way to keep X and still achieve objectives this is the way that'll be chosen. Quite often this means that more resources and more money will be spent in the end but nobody will be forced to admit that s/he's wrong. That's fine, they could do whatever they want but it also means that it's their responsibility to unclog their mess: if they are using screwdrivers to hammer in nails and new screwdriver is not usable in that role then it's their resposility to invent something to live with it. In this thread there were quite a few explanations for how they could continue to use rsyslogd with systemd (LD_PRELOAD, namespaces, etc). It's doable. Yes, it'll be ugly and perhaps somewhat fragile but so what? The whole sheme was fragile to begin with.
Posted Feb 1, 2014 22:21 UTC (Sat)
by raven667 (subscriber, #5198)
[Link] (4 responses)
Posted Feb 2, 2014 10:48 UTC (Sun)
by khim (subscriber, #9252)
[Link] (3 responses)
There are no need to use one solution for everything. /dev/log and syslog(3) work fine for low-volume logging. They will work even if your program is almost completely broken and this is valuable property to have. As for “we can not drop or change X because it's too expensive”—I've seen that, too and more often then not it ends in a disaster: if it's “it's too expensive” to change X now then what gives you confidence that it'll be cheaper later? This typical enterprise myopia which puts this quarter results above long-term survival. If you don't control something in your company then costs spent on this piece tend to grow till the whole house of card collapses. It's true that sometimes it's cheaper to use off-the-shell solutions because the thing you are doing are not that important, but if you are starting to care about one vs four context switches then either this part is in your core competence (and thus must be under your controls) or you are large enough to dedicate resources for the scaleable solutions even if it's not your core competence.
Posted Feb 2, 2014 16:47 UTC (Sun)
by raven667 (subscriber, #5198)
[Link]
Posted Feb 2, 2014 22:30 UTC (Sun)
by dlang (guest, #313)
[Link] (1 responses)
This includes being able to select between the different syslog implementations, and it also means being able to select between the huge number of tools that exist that deal with syslog messages today (including many that support very large volumes of logs)
No, this is not suitable for Google or Amazon levels of messages without consolidation, but it is suitable for just about any company below those levels.
Companies may choose to write customized logging mechanisms for their custom software, but every company (including google and amazon) runs a lot of software that they did not develop from scratch (think routers and switches for example), and so whatever system they use, it's going to have to support syslog anyway.
Posted Feb 3, 2014 0:18 UTC (Mon)
by khim (subscriber, #9252)
[Link]
Well, yeah, this is good example, LOL. But the fact that Google actually has it's own software on routers and switches is just a funny coincidence. No, that fact is not the main difference. Difference between Amazon/Bing/Google/Yandex and “enterprise” lies not with the fact that “big data companies” deal with larger amount of traffic but with priorities: Amazon/Bing/Google/Yandex know that all their solutions may become deficient in the future and thus have contingency plan which are enrolled long before scalability limits of the existing architecture are reached. “Enterprises” tend to exploit whatever they have till they reach 1000% of the intended scalability level where any minor change can collapse the whole house of card then start running around like headless chickens when, inevitably, such change is actually introduced. Think Danger. I'm not sure why: certainly outages which may ruin the whole company are as important for Blackberry or Verizon as they are important for Google so why such a big difference in attitude?
Posted Jan 31, 2014 21:59 UTC (Fri)
by foom (subscriber, #14868)
[Link]
You seem to believe that journald fundamentally cannot disable writing to disk, which doesn't appear to be true.
http://www.freedesktop.org/software/systemd/man/journald....
Posted Jan 31, 2014 22:18 UTC (Fri)
by raven667 (subscriber, #5198)
[Link] (1 responses)
That's an interesting problem for a high-volume logging. I'm going to be very interested to see how this is solved by the systemd team. It'd be nice if you could replace journald with rsyslogd for high-volume local logging, rsyslogd 7.4 even has journal input and output support, maybe building a way to pass the /dev/log open file descriptor back to your preferred syslog daemon when it starts up later in the boot process, as journald starts very very early to capture everything and I don't think we want to lose that functionality.
systemd is going to have to find a way to handle this workload efficiently
> the outraged response that I got on this thread that there may possibly be some reason to do things in a way other than the systemd way is a very bad sign as far as I'm concerned
Well this whole thing has become so politicized and full of misinformation that it becomes easy to presume malicious intent when there is a lack of information. You've provided one of the few real concrete examples where the systemd tools fall down, and your technical criticism has merit.
Posted Jan 31, 2014 23:14 UTC (Fri)
by dlang (guest, #313)
[Link]
My understanding (as part of rsyslog, but not in direct contact with redhat) is the introduction of liblogging
Redhat was already talking about a liblogging to override the glib syslog writes so that they could add metadata and structure to the logging (see project lumberjack), and they are adding the ability to redirect the logs somewhere other than /dev/log, overriding what the program does.
By doing this and linking it in directly to things they compile from source or LD_PRELOADing it for things that come from elsewhere, they gain the ability to bypass journald.
This is not something I'm happy about (it's far too hackish, and the fact that it's overriding what the program is compiled to do is the right thing in most cases, but not all, see my rant earlier about not deciding the 'one true way' :-)
If systemd did gain the ability to pass /dev/log directly to programs, I'm sure that we would add the ability to accept the filehandle very quickly (one question though, how many filehandles are we talking about? is it a single filehandle globally, or one per service? Since journald talks about being able to aggregate all logs related to a service, no matter how they are written it's now a question)
Posted Jan 31, 2014 22:31 UTC (Fri)
by cortana (subscriber, #24596)
[Link] (2 responses)
Posted Jan 31, 2014 23:22 UTC (Fri)
by dlang (guest, #313)
[Link] (1 responses)
Posted Feb 1, 2014 0:52 UTC (Sat)
by cortana (subscriber, #24596)
[Link]
I'd like to see systemd get an option to configure a private mount namespace, then it could become a bit less ugly:
[Service]
Where the name of the namespace is used to create a file, /run/mountns/foo, which can be used by processes from other namespaces to obtain a file descriptor that they can pass to setns() in order to enter the namespace. That way, several services can share a common mount namespace. 'ip netns' provides a similar facility for managing network namespaces.
Posted Feb 1, 2014 6:32 UTC (Sat)
by fandingo (guest, #67019)
[Link] (16 responses)
I don't claim any insight into the direction of journald development, but it seems quite incomplete. There is the systemd-journald-gateway (package name on Fedora at least). It's such a half-assed network implementation though. First, it's pull only, and second, not that useful at any sort of scale.
I guess what I find is most surprising is that http://lists.freedesktop.org/archives/systemd-devel/2012-... doesn't seem to have gone anywhere that I can find. Ultimately, the journal needs some way to *send* log messages in realtime to via some sort of journal network protocol (which this patch provided in 2012). Then, you can at least get a centralized journal instance. After that, Logstash, Splunk, and the other log parsing vendors can implement a journald receiver, and everything should return to harmony.
Does anyone have any idea why this never got merged or refined further? The journal is so useless without it.
Posted Feb 1, 2014 6:45 UTC (Sat)
by dlang (guest, #313)
[Link] (12 responses)
As it is, you can very easily use rsyslog to get the logs from the journal and send them, as long as you aren't doing that large a volume.
Posted Feb 1, 2014 8:30 UTC (Sat)
by fandingo (guest, #67019)
[Link] (11 responses)
If a journald network sender were implemented, your complaints about scale would almost certainly disappear completely. Obviously, there would be no memory copies to send a message from journald to an external sender (like rsyslog). Additionally, it's already trivial to avoid *any* storage by specifying Storage=none.
Posted Feb 1, 2014 8:57 UTC (Sat)
by dlang (guest, #313)
[Link] (10 responses)
why is it that journald refuses to pass the metadata on to a syslog daemon? that would be a lot simpler.
and even if the journal could send the logs over the network, there is still a lot that syslog daemons can do in terms of filtering that journald can't.
why is it NIH if the syslog daemons try to work around journald to keep doing their job, but it's not NIH for journald to throw away decades of experience and code to do things from scratch?
Posted Feb 1, 2014 9:15 UTC (Sat)
by mchapman (subscriber, #66589)
[Link] (9 responses)
Isn't that precisely what Rsyslog's imjournal module is for?
If you're question is "why can't journald pass on the metadata as part of the syslog message themselves?" then you're going to have a whole lot more complexity in deciding how such metadata should be formatted, and that's probably a lot of config that has no rightful place in journald itself.
Remember, the journal can contain pretty much *arbitrary* metadata, whatever applications might decide to store there, so there's really no standard way of formatting it all. This is the 3/4 fandingo was talking about. As an example, I am personally interested in writing a mod_log_journal for Apache that provides all the kinds of stuff you can get out of mod_log_config, all nicely indexed into different journald fields -- how on Earth is journald expected to be able to provide that to Rsyslog?
Posted Feb 1, 2014 17:07 UTC (Sat)
by dlang (guest, #313)
[Link] (6 responses)
the applications are sending their data to the journal via /dev/log and syslog calls, just exactly the same way that they would send it to syslog if the journal didn't exist (plus capturing stdout and stderr which are usually unformatted garbage)
Posted Feb 2, 2014 0:29 UTC (Sun)
by mchapman (subscriber, #66589)
[Link] (5 responses)
I'm not sure what you mean by "passthrough mode" here. The metadata was never inline with the messages in the first place, even before it got into journald.
Posted Feb 2, 2014 22:35 UTC (Sun)
by dlang (guest, #313)
[Link] (4 responses)
Posted Feb 3, 2014 0:00 UTC (Mon)
by mchapman (subscriber, #66589)
[Link] (3 responses)
I think we might be talking about different things.
I am interested in providing structured messages -- multiple fields containing all sorts of useful metadata -- from applications themselves. I am not too concerned about the "automatic" metadata journald adds; it's clear that if journald can add that, then so can any other logging system.
I note that you've linked to RFC 5424 elsewhere. The structured data extension in that does look interesting; I wonder if journald can support it (or if anybody has asked the developers or proposed it to them).
Posted Feb 3, 2014 4:54 UTC (Mon)
by dlang (guest, #313)
[Link] (2 responses)
The RFC5424 structured data really isn't used very much, and even rsyslog (who's primary developer is the author of RFC5424) didn't have the ability to actually _use_ this data until a few months ago.
in spite of the official standard, the de-facto standard is looking like it's going to end up being JSON structured messages. All of the syslog daemons can parse JSON structured logs and act on the different variables in them (and modify them for that matter)
Posted Feb 3, 2014 18:38 UTC (Mon)
by rodgerd (guest, #58896)
[Link] (1 responses)
Why it seems to have taken systemd to light a fire under the arses of syslog developers is another question entirely...
Posted Feb 3, 2014 18:55 UTC (Mon)
by dlang (guest, #313)
[Link]
It didn't, the syslog developers had already been working on these things (and many others)
RFC5424 and the CEE standardization projects came out several years before the systemd journal.
The only features that the systemd journal had (or at least claimed to have) that syslog didn't have at release were
capturing STDOUT and STDERR (which I'm not sure is really that good an idea, as sloppy as many developers are with these)
trusted properties (some of which are racy to gather, so not as reliable as you may think), which had been discussed and was on the TODO list, but had not made it to the top
log 'security', which as many noted, the journald security did not protect you against malicious root on your system unless you sent data elsewhere, and sending data elsewhere is usually the best defence anyway. Syslog developers have implemented better anti-tampering features that specifically document what external connectivity is needed to make things secure.
It doesn't help that distros tend to ship such old versions of the syslog daemons. It is hard to see progress when the version that's included is several years old.
Posted Feb 1, 2014 18:15 UTC (Sat)
by raven667 (subscriber, #5198)
[Link] (1 responses)
Posted Feb 2, 2014 22:20 UTC (Sun)
by dlang (guest, #313)
[Link]
Posted Feb 1, 2014 13:09 UTC (Sat)
by zuki (subscriber, #41808)
[Link] (2 responses)
It got put on a back-burner :) The reason was simply that due to lack of time, I had to give other things higher priority. Actually voices requesting such remote logging functionality have been almost non-existent, so I simply thought that nobody really cares about that. I've started working on this again, so hopefully there'll be an update soon.
Posted Feb 1, 2014 17:53 UTC (Sat)
by fandingo (guest, #67019)
[Link] (1 responses)
I certainly understand that.
> Actually voices requesting such remote logging functionality have been almost non-existent
Honestly, this surprises me since some of the systemd developers work at Red Hat and even your use-case at GMU (apologies for stalking you on the Fedora wiki). In your experience, how are people using the journal on multiple systems? Are they using Journal->Rsyslog for aggregation, not aggregating their logs, or using a totally separate aggregator like Logstash/Splunk? I hope that you get the time to work on it because it's a really exciting feature.
Posted Feb 2, 2014 3:35 UTC (Sun)
by zuki (subscriber, #41808)
[Link]
Posted Jan 31, 2014 11:00 UTC (Fri)
by anselm (subscriber, #2796)
[Link]
OTOH, if you hit the limits of SysV init (which does tend to happen rather earlier than with systemd) you get to fight SysV init, usually by writing large swathes of shell code that may or may not work reliably on the machine in question and are almost guaranteed not to work at all on the next machine over that uses a different Linux distribution.
The systemd developers are opposed to porting systemd to other kernels but so far they don't seem categorically opposed to suggestions about reasonable use cases on Linux that systemd does not yet handle well, or to bug fixes. Contrast this to Upstart, which has all sorts of serious bugs that nobody seems to be eager to fix, or to SysV init, which is buggy by design.
Posted Feb 1, 2014 23:35 UTC (Sat)
by cdmiller (guest, #2813)
[Link] (7 responses)
Others use Puppet or similar making sure the package is always installed, configured properly, and running, else copious error logs and notifications ensue.
2. You run your script. ssh config file has a fatal error. Result the same as above.
Others use Puppet to manage the configs under version control making sure the latest tested working copy is always in place.
3. Let's say that you want to connect to the ssh daemon from a second script. You start the script, only to learn that sshd hasn't actually started yet. You stick a 'sleep 1' in the other script. Everything works, on the faster machines, fails on the slower ones.
"You" may do something dumb to try and solve the problem, others hopefully do it better.
4. sshd is unexpectedly killed by oom killer. You run 'service sshd start; service sshd stop' in quick succesion. Ooops, the start script returned before sshd manager to write its pidfile. The stop script kills a random program which got the same pid meanwhile.
Sucks to be "you". Others increase the RAM for their VM and troubleshoot what's triggering the OOM killer.
5. sshd has hung. You run the stop script. The script send TERM, but the program is seriously stuck and will only react to KILL. Nevertheless, you are notified about success.
Apparently troubleshooting this script's behaviour is beyond "you". Others track down and fix the problem causing sshd to hang.
Yay for competence, for which there is no software substitute.
Posted Feb 2, 2014 1:10 UTC (Sun)
by mathstuf (subscriber, #69389)
[Link]
Posted Feb 2, 2014 1:50 UTC (Sun)
by vonbrand (subscriber, #4458)
[Link]
I for one much prefer not to get into those situations to be "fixed" with yet another thick layer of software to learn, set up, and generally manage, all to work around the legacy mess. Next claim is that all this is "simple, streamlined," while systemd is "bloated."
Posted Feb 2, 2014 3:51 UTC (Sun)
by zuki (subscriber, #41808)
[Link]
Posted Feb 3, 2014 10:40 UTC (Mon)
by HelloWorld (guest, #56129)
[Link] (1 responses)
Posted Feb 3, 2014 18:35 UTC (Mon)
by rodgerd (guest, #58896)
[Link]
Posted Feb 3, 2014 18:34 UTC (Mon)
by rodgerd (guest, #58896)
[Link]
I quite agree. I'm not sure you've thought what I imagine you think is a devastating riposte through carefully enouhg, because if you think the right answer to your broken hacky scripts is Puppet, or the answer to the OOM killer is always memory, I can tell you it's not the parent poster who looks bad.
Posted Feb 3, 2014 18:50 UTC (Mon)
by cortana (subscriber, #24596)
[Link]
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
Sure.
And why not? At least until they are shown to be wrong.
Which example? What performance issues?
I don't understand the problem - can it be worked around by disabling extra cgroups controllers (perhaps completely?) and/or by using plain old SysV init?
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
Which one? Automount? Does it have something to do with network latency?
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
$ journalctl -o verbose | grep $'^\t' | cut -d = -f 1 | sort | uniq -c | sort -n
1 EXIT_CODE
1 EXIT_STATUS
4 SEAT_ID
5 KERNEL_USEC
5 USERSPACE_USEC
13 ERRNO
13 EXECUTABLE
61 _SYSTEMD_USER_UNIT
187 _KERNEL_DEVICE
187 _KERNEL_SUBSYSTEM
187 _UDEV_SYSNAME
273 LEADER
273 SESSION_ID
273 USER_ID
451 RESULT
795 UNIT
1137 CODE_FILE
1137 CODE_FUNCTION
1137 CODE_LINE
1138 MESSAGE_ID
1409 _SOURCE_MONOTONIC_TIMESTAMP
1824 _SYSTEMD_OWNER_UID
1824 _SYSTEMD_SESSION
52403 SYSLOG_PID
65664 _SYSTEMD_UNIT
67418 _SOURCE_REALTIME_TIMESTAMP
67928 _CMDLINE
67930 _EXE
68473 _SYSTEMD_CGROUP
68474 _COMM
68986 _GID
68986 _PID
68986 _UID
69095 SYSLOG_FACILITY
70330 SYSLOG_IDENTIFIER
70395 _BOOT_ID
70395 _HOSTNAME
70395 _MACHINE_ID
70395 MESSAGE
70395 PRIORITY
70395 _TRANSPORT
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
he wouldn't declare it mandatory and make it so hard for long-term successful usage patterns to continue.
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
You can disable it, if you wish. It's not straightforward, but possible.
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
you are assuming that all logs represent errors.
From anti-systemd to pro-systemd in the shortest time
Well, kinda. Syslog is just simplest and weakest audit subsystem in existence, but yes, it's an audit subsystem. If you'll try to store information about every RPC processed by all your daemons it'll choke, journald or rsyslogd.
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
Syslog has clearly defined inputs and outputs and is more amenable to multiple implementations being drop in replacements for one another than some of the other parts of init and systemd.
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
This shows that you do not understand logging at the enterprise level.
syslog can be an extremely capable logging system, and log sent to syslog are not the limited, intended only for humans, thing that you think they are.
you are thinking of syslog as it existed 15-20 years ago, not what Rsyslog, syslog-ng, nxlog, and logstash have all been doing for years.
I've run Rsyslog at gig-E wire speeds (~400,000 logs per second) and other people with faster networks have run it at over 1,000,000 logs per second. This will keep up with logging information about every RPC processed by your daemons.
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
think routers and switches for example
From anti-systemd to pro-systemd in the shortest time
documents "Storage=none" in journald.conf to disable all writing of log data.
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
PrivateMountNamespace=foo
ExecStartPre=/bin/mount --bind /runs/system/journal/syslog /dev/log
ExecStart=myprogram ...
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
> doesn't seem to have gone anywhere that I can find.
> Does anyone have any idea why this never got merged or refined further?
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
> Are they using Journal->Rsyslog for aggregation, not aggregating their
> logs, or using a totally separate aggregator like Logstash/Splunk?
I know that some people do use journald+rsync or journald-on-nfs. Many use rsyslog to forward messages. Journald could definitely use some care. In addition to the missing push-over-the-network functionality, it is currently fairly slow in collecting its metadata, has other performance issues. kdbus should solve the issues with metadata collections, and probably some smart profiling will be needed to find the reason for other slowness.
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time
From anti-systemd to pro-systemd in the shortest time