That newfangled Journal thing

Posted Nov 21, 2011 17:47 UTC (Mon) by anselm (subscriber, #2796)
In reply to: That newfangled Journal thing by HelloWorld
Parent article: That newfangled Journal thing

Lennart Poettering could implement a program that would cause any code to run 10 times faster and require only a quarter of the memory and people would still dismiss it out of hand simply because it comes from Lennart and they couldn't get PulseAudio to work when it was very new.

I think it is a good idea for somebody to take a close look at logging. The major selling point of the current system is its ubiquity, not its compelling design, which was debatable even when it was first proposed (Hard-coded list of categories? Great idea!). It is silly to claim that syslog has no issues whatsoever, and that it should stay the way it is for eternity. There are lots of things about syslog that are worth reevaluating. This does not mean that »the journal« is automatically the best possible approach to a replacement, but it is definitely a start.

It may be unfortunate that Lennart is apparently the only person who is independent-minded enough to actually tackle this sort of thing, simply because so many people seem to hate him that much. The same people who complain about systemd today would at the time have been violently opposed to the introduction of SysV init because it gratuitously overcomplicates something the /etc/rc file that worked perfectly fine and had done so for nearly 20 years, and who needs this newfangled idea of runlevels, anyway?

to post comments

That newfangled Journal thing

Posted Nov 21, 2011 19:25 UTC (Mon) by juliank (guest, #45896) [Link] (4 responses)

> the /etc/rc file that worked perfectly fine and had done
> so for nearly 20 years, and who needs this newfangled idea
> of runlevels, anyway?

Well, you practically only need two run-levels: Single user (rescue) and Multi User (normal). I certainly don't see a need for more than two runlevels.

That newfangled Journal thing

Posted Nov 21, 2011 19:30 UTC (Mon) by juliank (guest, #45896) [Link]

You actually only need:

a) an rc.d directory with scripts
b) a list of scripts that will always be run (single-user mode)
c) a list of scripts that will be run in normal situations (multi-user)

Without (b), you get the init system used in NetBSD and FreeBSD.

That newfangled Journal thing

Posted Nov 21, 2011 22:51 UTC (Mon) by anselm (subscriber, #2796) [Link] (1 responses)

I certainly don't see a need for more than two runlevels.

Speak for yourself. I used to have a setup where the difference between runlevel 2 and runlevel 3 was that runlevel 3 enabled the fax-modem getty on the serial interface that my modem was connected to, in order to handle incoming calls. There are obviously lots of other ways to do this, but since a big part of SysV init's business is dealing with serial interfaces, anyway, it was the most convenient method.

It is important to note that the customary arrangement of runlevels in use with most Linux distributions is exactly that a convention. Since few people now operate standalone systems with serial ttys, runlevel 2 in the customary arrangement is rarely used these days, and most people will pick either runlevel 3 or 5 as their standard. However, this does not mean that the runlevel system cannot be employed profitably for other things, or that the runlevels must always mean exactly what they mean in the customary arrangement. (Which is essentially why Debian does not subscribe to the customary arrangement it mostly doesn't make a great deal of sense, but that doesn't mean runlevels as such are useless.)

That newfangled Journal thing

Posted Nov 23, 2011 12:52 UTC (Wed) by sorpigal (subscriber, #36106) [Link]

I love runlevels so much I just wish I had more of them. Unlimited numbers of named "runlevels" that let me boot to or switch to a certain service target is what I want. That way I can have a "maintenance mode" target, for example; it would be like raising the job fence. Even on my workstation I could have a "gaming" target which stops everything I don't need for games and then a "web development" target which runs my apache and postgres. Funnily, systemd allows me to do this... it's one of the things I like about systemd.

That newfangled Journal thing

Posted Nov 21, 2011 23:42 UTC (Mon) by dashesy (guest, #74652) [Link]

On an embedded device I can use runlevel 2 (the default) to to only run the user-mode instrument controller program (no gettys).
I may go to runlevel 3 to have a GUI to look at some graphs.

That newfangled Journal thing

Posted Nov 25, 2011 4:43 UTC (Fri) by cas (guest, #52554) [Link] (14 responses)

because it comes from Lennart and they couldn't get PulseAudio to work when it was very new.

no, it's because it comes from LP and he quite clearly hates Unix and loves Mac & Windows. Not the kind of person we want designing core unix infrastructure.

Every time I hear of yet another one of Poettering's fads, I can't help but remember the Henry Spencer quote "Those who don't understand UNIX are condemned to reinvent it, poorly.". Poettering is a near-perfect exemplar of that.

BTW, I hated systemd before i had any idea that it was written by the author of Pulse Audio - even aside from its technical problems, it suffers from the Gnome Disease of radical revolutionary toss-out-the-old-bring-in-the-new (and repeat every couple of years), rather than the far more sane and productive method of gradual, incremental improvements - and actually *fixing* bugs rather than just abandoning the code and starting from scratch because maintenance is boring while complete rewrites are exciting.

It may be unfortunate that Lennart is apparently the only person who is independent-minded enough to actually tackle this sort of thing

but he's not doing that. he's just cloning bad ideas borrowed from Mac (launchd) and Windows (win event logging). and worst of all, they're not actually solving any real problems except his perception that unix sucks and should be remade into some bastard hybrid of windows and mac

That newfangled Journal thing

Posted Nov 25, 2011 8:49 UTC (Fri) by anselm (subscriber, #2796) [Link] (12 responses)

[systemd] suffers from the Gnome Disease of radical revolutionary toss-out-the-old-bring-in-the-new (and repeat every couple of years), rather than the far more sane and productive method of gradual, incremental improvements

This is quite obviously untrue. It has been pointed out in detail here and elsewhere that systemd can use, among other things, init scripts for existing services with no change. How is that a »radical revolutionary toss-out-the-old-bring-in-the-new«?

they're not actually solving any real problems except his perception that unix sucks and should be remade into some bastard hybrid of windows and mac

This is not discussion, it is propaganda.

Many people now seem to regard things like SysV init and syslogd as graven-in-stone pieces of infrastructure that were there in the very beginning of Unix, possibly on the original PDP-7 that Thompson and Ritchie hacked on, and which can only be criticised at the peril of sacrilege. It is just as well to remember that SysV init, and to a lesser extent syslogd, started out in the 1980s as exactly the same kind of »radical revolutionary toss-out-the-old-bring-in-the-new« that people are now railing against in the form of systemd and »The Journal«.

We have gone a long way in the intervening years towards discovering better techniques of doing almost anything in system administration, and I'm puzzled as to why topics like the init system must be forever exempt from being changed. Surely we can examine even these areas every 20 years or so, to see if we can do better? (I don't remember a similar sort of spite-fest around Upstart, which attempted exactly the same thing that systemd is trying to do, namely creatively revamp the init subsystem. Is that because it didn't go as far as systemd, or because some people hate Lennart Poettering with more of a vengeance than they do Scott James Remnant?)

That newfangled Journal thing

Posted Nov 25, 2011 9:15 UTC (Fri) by dlang (guest, #313) [Link] (9 responses)

> I don't remember a similar sort of spite-fest around Upstart, which attempted exactly the same thing that systemd is trying to do, namely creatively revamp the init subsystem. Is that because it didn't go as far as systemd, or because some people hate Lennart Poettering with more of a vengeance than they do Scott James Remnant?

one key thing is that the developers of upstart are not nearly as condescending towards people who disagree with them (they don't get dismissed as "people opposed to all change" instantly)

there's also the matter of the degree of change.

BSD init to SysV init are both variations of shell scripts. the difference is just in how the scripts are organized.

I haven't done much with upstart, but from what little I've seen of it, it's also based on shell scripts.

systemd strongly encourages (if not outright requires) that the logic be implemented in C code

this difference, and the attitude towards sysadmins (who generally know scripting of various kinds, but are not as comfortable with C) goes a long way towards alienating the people who are providing the support for their less technical friends and relatives.

Yes, there is a huge advantage to owning the desktop/home user

Microsoft's assault on the datacenter is based on it's ownership of the destop.

RedHat became the linux distro of choice in business mostly because it was what linux folks were running at home.

However, now that Linux is _so_ common in the datacenter, those people are the core of your linux expertise. Even if they are not involved with the initial deployment of Linux to people's desktops, they get pulled in whenever something goes wrong. making it harder for these people to track down the problem and fix it really hurts reputations..

And yes, after generating this sort of reputation for a few projects, it will affect how new proposals from the same people are received.

That newfangled Journal thing

Posted Nov 25, 2011 10:00 UTC (Fri) by anselm (subscriber, #2796) [Link] (7 responses)

one key thing is that the developers of upstart are not nearly as condescending towards people who disagree with them (they don't get dismissed as "people opposed to all change" instantly)

Maybe this is because at least some of them do seem to give the impression of being »opposed to all change«? It is interesting (or even amusing) to watch the contortions some people are going through in this ongoing discussion just because, according to them, the syslog mechanism may apparently never be changed.

Lennart Poettering has gone to the trouble of posting a large number of blog articles to inform people about systemd in a very factual manner. This does not particularly strike me as something a person as condescending as you claim Lennart is would bother doing.

BSD init to SysV init are both variations of shell scripts. the difference is just in how the scripts are organized.

Well, SysV init did add the whole business of run levels, a jungle of symbolic links, etc., when before there was basically one script that was run during boot. A trivial difference, to be sure :^) At the time SysV init was about as revolutionary compared to the prior approach as systemd is today compared to SysV init. Today's systemd haters would probably have railed against it with the same vituperation.

I haven't done much with upstart, but from what little I've seen of it, it's also based on shell scripts.

With that logic, it would make sense to claim that an air mattress is similar in use and performance to an aircraft carrier since they are both based on the physical principle of flotation.

systemd strongly encourages (if not outright requires) that the logic be implemented in C code

It is true that many of the boot-up tasks that systemd performs, which were traditionally implemented as shell scripts, are provided in the shape of C-based executables. I don't think this is an actual requirement; I seem to recall that the basic idea is to speed up the proceedings. As far as I know there would be no problem with adding things that are implemented in shell code, or replacing some of the C-based tools with shell-based implementations if that proved necessary.

As far as service management is concerned, systemd does implement most of the required logic itself and only relies on non-executable configuration files for the details. This is actually a good thing in my book since, among other benefits, it potentially leads to increased standardisation, where right now the shell-based init scripts are different from one distribution to the next. Again, systemd makes it possible to use traditional SysV-init style init scripts if the administrator so desires. I don't see it as a big problem.

I'm sorry but I don't really see what the rest of your comment has to do with anything.

this is exactly what drives people up the wall...

Posted Nov 25, 2011 10:32 UTC (Fri) by khim (subscriber, #9252) [Link] (6 responses)

As far as service management is concerned, systemd does implement most of the required logic itself and only relies on non-executable configuration files for the details.

And this exactly what driver most people up the wall. You see, sysadmins always relied on the power to alter system configuration in twisted, non-standard way. For some it's requirement to fit in twisted, non-standard pre-existing configuration, for some just a source of pride, but they were content in their knowledge that they have power to do anything by changing these shell scripts.

Systemd throws all that away: you can not just "add tiny nodge here and small delay there". You need to either contact upstream (to add configuration knob) or turn the automatic off and write complex and often convoluted script from scratch. This is what irritates people the most in systemd design.

This is actually a good thing in my book since, among other benefits, it potentially leads to increased standardisation, where right now the shell-based init scripts are different from one distribution to the next.

Right. And application developers will probably welcome that. But system administrators will not: they've lost some of their power and are forced to think about real solutions instead of trying to apply chewing gum on the mix of bailing wire. Of course this irritates them.

And, as usual, people who benefit don't have too much energy to fight for systemd (they are there: you can find them in all flamefests about Lennart's creations), but people who are burned fan them hoping against hope that they will be able to keep chaotic nature of Linux where their ability to skillfully manipulate duct tape will be appreciated forever more.

Well, Linux today certainly fulfills the second part of the mantra ("complex things should be possible") but what about first ("easy things should be easy")? The counteroffer (let's add layers upon layers of bandaids on the existing solution) does not help much here so Lennart's way looks like the only viable alternative. Yes, it means that some complex things will not be possible anymore. If they will still be actually needed they can be reintroduced later...

this is exactly what drives people up the wall...

Posted Nov 25, 2011 11:25 UTC (Fri) by anselm (subscriber, #2796) [Link]

Systemd throws all that away: you can not just "add tiny nodge here and small delay there".

Guess what systemd service definition files do allow you to specify individual shell commands that will be invoked at various points of a service's life cycle. Those system administrators who today need to tweak convoluted, distribution-specific SysV init files will likely find it a lot easier to deal with this interface.

Also, SysV init gives you no real way to override a distribution-provided init script with one that you modified, such that your version will take precedence over the one in the distribution and your configuration will not be overwritten once the next version of the distribution package comes around. Systemd does. (And this is after this issue has had 25 years to be straightened out in SysV init. Give systemd 25 years to mature and see what it looks like then.)

All of these things should be painfully obvious to anyone who actually takes the trouble to read the systemd documentation and descriptive blog posts. But of course it is much easier and more convenient to misrepresent the package (and the intentions of its authors) out of a position of ignorance.

this is exactly what drives people up the wall...

Posted Nov 25, 2011 11:41 UTC (Fri) by michich (guest, #17902) [Link] (2 responses)

Can you give any example where systemd forces you to write a complex and convoluted script from scratch? I've seen cases where instead of running a program directly from the systemd unit file it made some sense to call a shell wrapper, but why would it have to be complex and convoluted?

Because you talk about legacy code?

Posted Nov 25, 2011 12:58 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

One recent example: connection to Corbina.Net (I've helped my friend with that). You need start igmpproxy first (because otherwise it'll crash when L2TP disconnects), then you need to check L2TP server (some of them don't work and then you need to try again), then you connect to L2TP server and wait till everything stabilizes - and only after that you can start IPSec to connect to your real VPN network.

With SysV init you can just order all services in proper (tested and working) order while with systemd you either need to properly describe dependencies or run the whole thing as one service.

In first case you need much, much deeper understanding of the problem then "if I run igmpproxy before L2TP client then everything works, if I do it in opposite order then igmpproxy crashes when L2TP client auto-reconnects" in second case you need to write the whole script from scratch.

Because you talk about legacy code?

Posted Nov 25, 2011 13:26 UTC (Fri) by anselm (subscriber, #2796) [Link]

Yes, but which SysV init are you talking about the traditional one where you get to set all the symbolic links yourself, or the one most distributions are using these days, where the symbolic links are set automatically from dependencies in the LSB headers of the init scripts?

In the former case, any ordering you can explicitly impose on SysV init scripts through numbers in symbolic links can trivially be replicated through explicit »Before=« and »After=« dependencies in systemd. This most emphatically does not require a deeper understanding of the problem than you would need to come up with a suitable ordering for SysV init in the first place.

In the latter case, you need to specify all the dependencies in the LSB script headers for the benefit of your SysV init, and the LSB script header doesn't let you do anything that systemd dependencies don't let you do too. If anything, the systemd dependencies are a lot more expressive than the LSB script-header ones, so your job should be that much easier.

Finally, if you need to wait for something to »stabilise« after you have started a service in systemd, you can write a shell wrapper around that service which does the check, and that shell wrapper will be vastly simpler than a full SysV init script.

this is exactly what drives people up the wall...

Posted Nov 25, 2011 12:24 UTC (Fri) by dgm (subscriber, #49227) [Link] (1 responses)

> Yes, it means that some complex things will not be possible anymore. If they will still be actually needed they can be reintroduced later...

So they will still be possible after all...
Honest question: is systemd really so inflexible that I _cannot_ do "interesting" stuff? I thought that ability to run current init scripts preserved that flexibility for when it's needed...

this is exactly what drives people up the wall...

Posted Nov 25, 2011 12:38 UTC (Fri) by anselm (subscriber, #2796) [Link]

No.

(Check the systemd page to find out for sure.)

That newfangled Journal thing

Posted Nov 27, 2011 2:44 UTC (Sun) by HelloWorld (guest, #56129) [Link]

> systemd strongly encourages (if not outright requires) that the logic be implemented in C code
I don't understand why people keep spreading this stupid lie. systemd requires no such thing; if you actually want to write a shell script to start your service, you're free to do so.
That said, I agree with Lennarts assessment that bourne-style shell scripts are a nightmare for robustness and maintainability and are best avoided. This is where systemd comes in: it offers a configuration file format that allows most of what is commonly done in init scripts (i. e. setting environment variables, rlimits and all that), yet is *much* simpler, more maintainable and generally easier to work with.

I think it's matter of presentation and the actual implementation

Posted Nov 25, 2011 9:25 UTC (Fri) by khim (subscriber, #9252) [Link] (1 responses)

I don't remember a similar sort of spite-fest around Upstart, which attempted exactly the same thing that systemd is trying to do, namely creatively revamp the init subsystem. Is that because it didn't go as far as systemd, or because some people hate Lennart Poettering with more of a vengeance than they do Scott James Remnant?

I think it's matter of presentation and scale.

Scott's message:
Here is new version if SysV init which does exactly the same thing as the old one, but it can do few new tricks as well if you'll use new "native services". You can switch to them at you leisure. Indeed: upstart was added to Edgy Eft (Ubuntu 6.10) but even the next LTS (Hardy Heron AKA Ubuntu 8.04) used it as "modern SysV replacement" - native Upstart bootup was only introduced in Karmic Koala (Ubuntu 9.10).

Lennart's message:
Here is replacement for SysV init and upstart. It introduces totally new world and it's design does not take in the consideration the existing practice at all. Sure, we understand that compatibility is important and we'll provide compatibility mode till everyone will switch to systemd. We want to see this switch sooner rather then later - so we will patch programs and make them systemd-aware ASAP.

The first mode leads to years to transition but generates few ill feelings (people feel that they are in charge and can "jump ship" at any time) while second mode introduces abrupt change which naturally leads to spite-fests. Especially when another critical piece of the infrastructure is introduced which is intrinsically tied to the "new and unproven" systemd.

I think it's matter of presentation and the actual implementation

Posted Nov 25, 2011 10:24 UTC (Fri) by anselm (subscriber, #2796) [Link]

It introduces totally new world and it's design does not take in the consideration the existing practice at all.

I don't think that is actually entirely fair.

For one, what systemd does is only a »totally new world« if you disregard the experience gained with Apple's launchd, which pioneered many of the ideas that we now find in systemd and which, on the whole, do make sense from a technical point of view. To some people this seems to come across as an evil conspiracy to make Linux »more like MacOS«, but I believe this is more due to the fact that the hackers at Apple appear to be on to something good here.

Also, it is worth bearing in mind that many of the things systemd does are traditionally provided by different pieces of infrastructure (SysV init, inetd, cron, ), not because these were designed that way from the start but because they arose at various points in the history of Unix and were introduced by different groups of people. It does seem worthwhile to me to try and construct a general framework for these tasks in the interest of simplicity, better configurability, and standardisation.

I'm not so sure...

Posted Nov 25, 2011 8:59 UTC (Fri) by khim (subscriber, #9252) [Link]

no, it's because it comes from LP and he quite clearly hates Unix and loves Mac & Windows. Not the kind of person we want designing core unix infrastructure.

Actually I think it's time to bring people who care about things being done more and less about sacred "UNIX way". It does not mean I'll accept journald unconditionally, but I will not reject it before try like most people here are doing.

Every time I hear of yet another one of Poettering's fads, I can't help but remember the Henry Spencer quote "Those who don't understand UNIX are condemned to reinvent it, poorly.". Poettering is a near-perfect exemplar of that.

You know, I think people repeat this mantra so often they start to believe it. Even if it's not true at all.

Exhibit 1: command-line utilities. Solaris included real hardcore UNIX command-line utilities. But to make it even somewhat usable you needed to bring GNU packages and use that. "true UNIX lovers" said the same thing about "overbloated and heavy" GNU tools they say today about systemd and journald yet today these "good old GNU tools" are perceived as "UNIX tradition" because otherwise you must admit that UNIX has mostly dead.

Exhibit 2: remote access. 20 years ago remote access in Windows didn't exist while UNIX boasted "network-native X protocol". 10 years ago Windows got RDP and remote access become easy for Windows people. Yes, I know, Microsoft did that by twisting Citrix arms, but the end result is still the same: easy to use system where you don't need to configure anything in advance - you can just turn knob "on" when you are late and need to go home, then connect from home and continue your work. It happened 10 years ago so we should have similar capabilities on Linux today? Nope: today it's still huge problem with a Linux world: there are bazillion tools for that, but they don't employ "network-native X protocol" (because it lacks the knobs: you can not switch clients from one X server to another "on the fly") and in general it's hard to use and slow. The one big achievement in this regard was introduction of PulseAudio: at least now we can reliably forward not just GUI, but the accompanying sound as well.

Rather than the far more sane and productive method of gradual, incremental improvements - and actually *fixing* bugs rather than just abandoning the code and starting from scratch because maintenance is boring while complete rewrites are exciting.

It does not work, sorry: search for some of his arguments in regard to syslog are technically wrong, but can be considered true if one looks at current practice.

People are vocal social creatures, true - and this calls for "gradual, incremental improvements". But that's lie. People are also lazy creatures and that works for Lennart's "let's redo everything" approach and against "more sane and productive method". Think about it: how much foam was on the web WRT KDE 4. You'd think that outraged people dropped everything and rushed to support KDE 3? Well... four of them did...

Opt-in vs opt-out makes huge difference. Opt-ins are usually ignored while opt-outs lead to large flamefests but then quite fast adoption. And you will have flamefest anyway then why not redo everything to cleanup all the cruft?

If the result will be usable - it'll be adopted anyway, if it'll not be usable - it'll be thrown away no matter what. And as you've correctly noted it's often simpler to rewrite something from scratch rather then to try to fix bugs in existing solution.