LWN: Comments on "Quotes of the week"

Systemd incompatible with...

raven667 — Fri, 12 Oct 2012 15:54:38 +0000

> (Do laptops use PCIe yet? Presumably they do: it's got more power-saving features in it than PCI has.)

Oh I'm sure that all the internal busses are PCIe and if there is an external ExpressCard slot, that's hot-plug PCIe. When it comes to power management I imagine that some devices drop out and show up on the bus in a hot-plug fashion when they are powered off/on.

> moved udevd [...] without leaving behind a compatibility symlink, breaking booting ...

That's pretty screwed up in any event. I find the logic and evidence compelling for collapsing / into /usr but that is no excuse for not putting in appropriate compatibility measures, measures which should be totally straightforward and transparent.

> The udev maintainers' motto appears to be "backward compatibility is for other people"

That's an unfortunate motto of much (though thankfully not all!) of userspace, it's one area where having source available has lead people to believe that rebuilding and recompiling the world is acceptable so API/ABI compatibility don't matter.

Systemd incompatible with...

nix — Fri, 12 Oct 2012 15:31:38 +0000

I suppose that's true, if it is common to have PCIe devices that take so long to come up that they haven't started by the time POST has finished. That's not true of any system I own, but since all my systems are desktops and desktops are famously slow to POST...

(Do laptops use PCIe yet? Presumably they do: it's got more power-saving features in it than PCI has.)

I don't actually object to the hotplug-everywhere world, as long as it doesn't break anything much it seems like it might add generality without cost. Unfortunately over the last forty or fifty udev releases (as measured by version numbers) this has not generally been true: every upgrade has, as mentioned elsewhere, been frightening and well above 50% would have broken booting if I hadn't done other things to prevent it. (Those breakages that were due to outright bugs in udev did get fixed fast, but most of them were due to explicit incompatibilities that required sysadmin action. And every one of those was a cost without a benefit. Classic example: udev 174, without advance warning or stated rationale of any kind, moved udevd from /sbin into /lib/udev without leaving behind a compatibility symlink, breaking booting uless you fixed your boot scripts. Because, you know, so many Unix daemons live in subdirectories of /lib or /usr/lib. Oh, wait, none whatsoever do, except for udev. Then udev 175 moved it again (into /usr/lib), and moved udevadm as well, breaking the boot *again* unless you compensated for it. The udev maintainers' motto appears to be "backward compatibility is for other people".)

Systemd incompatible with...

raven667 — Thu, 11 Oct 2012 19:55:28 +0000

That was kind of a crappy comment. Maybe it would be better to say that "in practice" the hardware treats power-on enumeration like a series of hot-plug events which is why it makes sense for the kernel to handle it that way too. "in theory" maybe you could close your eyes and treat the system as static but there would be cases where that would cause you to get wedged that hot-plug wouldn't.

Systemd incompatible with...

raven667 — Thu, 11 Oct 2012 19:42:16 +0000

Maybe I just don't understand but doesn't the fact that these busses are possible to hot-plug and the fact that when it comes to enumeration and probing they behave like hot-plug in that they show up on the bus whenever they become ready mean that the driver systems that handle these devices have to handle the hot-plug case. The infrastructure has to be there to support it so I don't see how you can get rid of the complexity of dealing with these devices as hot-pluggable. It seems a simplification to just handle everything as a hot-plug case, rather than trying to maintain separate static and hot device initialization. infrastructure

Systemd incompatible with...

nix — Thu, 11 Oct 2012 19:03:44 +0000

See, there's a major difference between 'technically' and 'in practice' here. Maybe my PCIe and SATA buses are theoretically hotplugged, but not once in the entire history of any of my systems with any of these buses has any device ever appeared while the OS was running, nor has one ever gone away except on one occasion when a disk failed (so hardly a hotplug event). Nor have I ever heard of systems other than really really high-end boxes (that I could not possibly afford) that can hotplug PCIe (though I have seen machines with external SATA connectors -- always unoccupied in my experience, but possibly useful in emergencies or to connect external drive enclosures, I suppose).

So any recomplication of the system to allow for those possibilities is wasted, from my perspective; and any administrative overhead imposed on me as a system administrator due to changes required by such recomplication is a cost sans benefit, that I would rather not pay.

Systemd incompatible with...

Jonno — Thu, 11 Oct 2012 14:20:45 +0000

Technically all PCIe devices are "hot plugged", at some indeterminate period after it gets power, which it get simultaneously with the main system. If the main OS assumed that all PCIe devices was available at start, that would lead to all kinds of race condition.

The same goes for USB (of course) and SATA, and likely a host other systems...

As an anecdote, I had an oldish SATA device that suddenly stopped being bootable, but worked perfectly once booted from CD.

It turned out that the drive firmware had a boot time of O(n²) where n was the number of bad block relocations. Thus as the drive got old, it began to turn up after BIOS handed over to GRUB, and GRUB didn't have hot plug support :(.

I "solved" the problem by configuring BIOS to always do a full memory check at boot, slowing it down by 5-10 seconds, making booting work 9 cases out of 10. I have since replaced the disc, but it goes to showing that even devices you think are static aren't really...

Systemd incompatible with...

nix — Thu, 11 Oct 2012 11:00:17 +0000

Really? People keep saying this but I really don't think it's true. 90% of the devices on every system I own, including all the 'difficult' ones like the ones that are part of RAID arrays and the networking and the framebuffers and the like, are non-hot-pluggable or hotplugged so rarely that I don't care if I have to do something special to make the system pick them up (much less than once a year).

Just about the only hotplugged stuff is USB devices and the block devices that generally hang off them... and udev has worked perfectly well in *that* case for many years. So the cause of the continuing disruptive churn is completely opaque to me. It certainly doesn't benefit me one bit.

Systemd incompatible with...

dlang — Sat, 06 Oct 2012 01:08:58 +0000

many people never stopped using static device nodes. For systems that don't have the hardware changing on them, a static /dev works very well.

not everything is a laptop with random USB device of the day getting hot-plugged into it.

Systemd incompatible with...

jackb — Sat, 06 Oct 2012 01:03:33 +0000

I remember the transition from static device nodes to devfs because there were problems with static devices, then shortly afterwards we moved to udev because devfs was fatally, unfixably flawed, then devtmpfs + udev for some reason which isn't particularly clear, now there's a mention in LKML of moving udev into the kernel source tree.

Now wondering if we're ever going back to static device nodes, just to really come back full circle.

/etc/mtab as a file not supported?

rleigh — Fri, 05 Oct 2012 12:34:24 +0000

It should be under /run/mount along with utab, surely? We moved all use of dotfiles in /dev to /run; is this an exception which needs addressing?

Systemd incompatible with...

Cyberax — Thu, 04 Oct 2012 15:39:56 +0000

No, that's a sane requirement. Requesting firmware in module_init() can lead to deadlocks, if userspace can't process the request. It happened to work before because most of time userspace usually just reads firmware from a disk/initramfs.

So it makes sense to do something about it, however definitely not in the way udev developers do it.

Systemd incompatible with...

Eckhart — Thu, 04 Oct 2012 09:46:33 +0000

> I just want something that manages a hotplug /dev for simple cases like usb mass storage plugged in after boot

… except that it does not make sense to differentiate between "before boot" and "after boot" at all

Systemd incompatible with...

nix — Thu, 04 Oct 2012 01:14:18 +0000

So it appears.

I'm damn glad that Linus and Al appear to agree with the rest of us who value stability in re the ongoing udev trainwreck. I don't *want* a new init system, I don't want to have to make invasive changes to my system on every udev upgrade, I just want something that manages a hotplug /dev for simple cases like usb mass storage plugged in after boot, lets me customize it by adding symlinks, tweaking permissions and the like, and otherwise gets out of the way.

udev used to do this when gregkh maintained it and for a while after, but currently is so far from this that is just not true. The last udev release that added a feature that I actually valued was about 130, but since then almost every release has been terrifying and introduced heaps of incompatible changes for no reason other than that it can, many releases have required invasive changes to the way my system is booted and break backward compatibility so that if I try to downgrade after a successful upgrade *that* will fail to boot, it's halfway to mandating systemd usage... no no no, bring back a udev with maintainership that understands how boot-critical software should be maintained, please. (That is to say, carefully and with great attention paid to backward compatibility and break-free upgrades without the poor distributors and/or upgraders having to whip themselves into a frenzy to do it.)

(Yes, I know I can just run an old udev. That's what I'm doing. But running unmaintained older versions of critical software required for boot because you can't trust the maintainers not to break your boot on every release is not good. Not good at all. Among other things, eventually programs will presumably start requiring a newer libudev...)

Systemd incompatible with...

bronson — Thu, 04 Oct 2012 00:39:38 +0000

Oof, I'm hoping Kay didn't actually release somewhat broken code without an idea of how he would fix it, but that seems to be what he's saying...?

This looks like an interesting story.

Systemd incompatible with...

nix — Thu, 04 Oct 2012 00:33:51 +0000

Earlier link: <http://www.spinics.net/lists/netdev/msg185742.html>.

I boggle. Lots and lots of modules need firmware when they initialize. They always have, they always will. Decreeing that 'they should not' doesn't change the reality that they *do*.

Systemd incompatible with...

gvy — Wed, 03 Oct 2012 09:20:19 +0000

Seems that the mantra has changed, and now Linux kernel is barely compatible with udev: https://lkml.org/lkml/2012/10/2/303

Lennart, would you please admit that being deaf to users is what hurted KDE4/GNOME3 efforts *big* time and stop advocating Kay no matter what? You're trying to break system layer which isn't easy enough to work around (like switching to XFCE or staying with WindowMaker).

I do understand that Red Hat might have some vision on future init but pushing the result down the throat won't help down the road.

Thank you for all the effort and good code, once upon a time many of your project were my favourite upstreams due to concise info and clean maintenance.

Systemd incompatible with mounted /usr

jrn — Tue, 19 Apr 2011 01:06:16 +0000

I thought the drive to make a /usr -> / symlink possible came from GNU Hurd developers in the hope of making PATH=/bin a viable configuration (and thus simplifying the concept of path resolution for users)?

Systemd incompatible with mounted /usr

nix — Mon, 18 Apr 2011 20:24:47 +0000

Some rules (a very few, but perhaps more in time) want the PCI and USB ID databases, as well. Of course this should be fixed by eliminating /usr and annoying everyone who has to reinstall their entire systems, rather than by, I dunno, introducing /share, moving the ID databases there, and leaving a symlink where they came from.

As you know, "do things the hugely inconvenient way" is the new Linux slogan.

Groundless Hype?

Pierre@G-WAN — Wed, 06 Apr 2011 11:53:43 +0000

And it's not like if Cherokee resolved the problem: it STILL uses opaque configuration files.

By contrast, G-WAN is much faster (without *any* configuration file):

http://gwan.ch/imgs/gwan_nginx_cherokee.png

Cherokee might rock... but maybe in another area:

http://forum.gwan.com/index.php?p=/discussion/182/cheroke...

Oh, and before we are (again) censored by Cherokee fan boys (like on Wikipedia*), here are the "Cherokee, the fastest Web server" facts checked by an independent source:

http://nbonvin.wordpress.com/2011/03/14/apache-vs-nginx-v...

Pierre.

(*): http://forum.gwan.com/index.php?p=/discussion/comment/150...

r/o root and /etc/udev/rules.d/persistentrules

gvy — Fri, 01 Apr 2011 11:22:15 +0000

> If you want to do that to mount it read-only, then you are simply aiming
> too low, and you should rather work on making / read-only.
(diggin' within /lib/udev) BTW it's interesting how you're going to do with persistent rules which are essentially the cache (thus belonging to /var, not /etc) but may be crucial to bringing up the stuff to mount said /var, be it a network interface or FC-connected storage (a tape library over here used to perform a streamer dance until fixated that way).

Not a nitpick, it's quite interesting, actually.

smells Richard Hughes, btw

gvy — Wed, 30 Mar 2011 21:00:16 +0000

> If you want to fix the corner cases, you are welcome to.
Last summer it was a filesystem deadlock. Hope you won't face it now :)

> Your favor distro has it too.
At least not the "udev under /usr" part. The thing is,
- I actually run it with separate /usr and see no problems;
- nothing bothers me about a non-issue for no reason.

> As has been pointed out, systemd is merely the messenger.
It's the messenger which was instructed to, well, lie.

And the instructor seems to have a stance that dumping that all on a user is more sane than going after those who are ignorant enough to introduce that crap in the first place (remember RH#534047?).

Now that's weird. I wouldn't waste time on Mr. Hughes but Lennart is, or at least was for many years (judging as a packager), much different.

Alvaro Ortega : Now that we all are running fancy desktops full of powerful applications

gvy — Mon, 21 Mar 2011 17:41:03 +0000

Reliable you say... while posting a comment at the linked blog, got:

500 Internal Server Error
Cherokee web server 1.2, Port 80

I'd expect a bit better of ex-"Technical Lead en Sun Microsystems, High Performance / High Availability Engineer en Espanix", frankly. And FWIW the comment was:

---
"Id say that none, none at all" -- if that would be the case, then typing all those letterses into scripts making up those very webapps would make no sense either. Folks might as well just do everything by hand in those pesky desktop applications not suited for any sane automation... but you were just trolling cheap, right? ;-)

BTW there is already apsstandard.org -- hope it's not news to folks over there.
---

Systemd incompatible with mounted /usr

gvy — Mon, 21 Mar 2011 17:29:57 +0000

Here in ALT Linux, there's "early udev" in initramfs which is supplied with limited rules enough to bring up rootfs, and it's only used to populate initial /dev then stopped.

"Real" init can start proper udev with full blown rules later, and I can testify that *not* starting it -- or, again, starting to further populate /dev and then stopping it -- did make it possible for ALTSP5 thin client to fit into 16M RAM on a terminal (thus being useful).

Sure /usr is integral to / on a diskless thin client but hope that a use case like that might help to understand that there is awful lot more of uses for init(8) and udev(8) out there than can be imagined in a hurry.

As for initrd machinery, one is invited to have a look at http://en.altlinux.org/Make-initrd (original page in Russian at http://www.altlinux.org/Make-initrd).

PS: thanks for ifplugd :)

separate /usr isn't just about r/o or NFS

rahulsundaram — Mon, 21 Mar 2011 16:32:51 +0000

If you want to fix the corner cases, you are welcome to. The current issues are no Fedora specific in any way. Your favor distro has it too. As has been pointed out, systemd is merely the messenger.

Systemd incompatible with mounted /usr

gvy — Mon, 21 Mar 2011 16:22:30 +0000

# find /usr -name '*udev*'
[.../usr/share/{vim,doc,man}...]
/usr/libexec/ConsoleKit/run-seat.d/udev-acl.ck

And the rest is under /{etc,lib}/udev here on ALT Linux.

Could you please elaborate on "the udev rules are the ones needed /usr", or perhaps just file a bug proper?

> Hence you would want to mount /usr both before and after starting udev.
> Which logic shows is not possible.
Heh. There's a theory for inventor's problems, you might enjoy reading up on that, as well as on formal logic; see at least http://en.wikipedia.org/wiki/TRIZ -- in this particular case, the "visible" controversy is solved by "doing beforehand", that is, moving udev rules from /usr into root filesystem. No rocket science at all.

Systemd incompatible with mounted /usr

gvy — Mon, 21 Mar 2011 16:11:36 +0000

> a small fast drive and a larger but slower one.
Did you just describe modern choices like SSD/HDD, SLC/MLC, especially in embedded/portable hardware, as prehistoric? ;-)

separate /usr isn't just about r/o or NFS

gvy — Mon, 21 Mar 2011 16:08:55 +0000

Ever got filesystem corruption (doesn't really need rw, a lucky bad block might suffice)?

My recommendation: try to understand why smart people have been doing things that way before, and think about this for a while. And then try to fix the broken attitude of declaring working things useful for others broken only because you are working hard on solving entirely different thing.

The better folks I know do work on corner cases. "Mainstream" like McDonald's usually just don't bother. If Fedora is all about GNOME desktop only (as discussed already) then this attitude fits pretty well, of course.

systemd

nix — Thu, 10 Mar 2011 21:46:27 +0000

I was particularly impressed by this comment:

Separating system directories (like /usr and /var) is what has made Unix great.

Not even slightly. "Everything is a file", anyone?

Quotes of the week

welinder — Thu, 10 Mar 2011 17:59:23 +0000

> I never publicly called anybody an idiot.

That statement is not true. It's true for this thread, but not
true as written. Try googling yourself a bit.

For example:

http://www.mail-archive.com/pulseaudio-discuss@mail.0poin...

systemd

foom — Thu, 10 Mar 2011 17:02:45 +0000

I think maybe you should've maybe read the rest of the comments before posting your own rambling uninformed commentary.

systemd

m.galabant — Thu, 10 Mar 2011 07:28:31 +0000

I've been living with perfectly functioning Linux installations without all that udev-systemd-messagebus-whattheyarecalledtoday -- crap comfortably.
My /dev is fully populated and I don't care about a few inodes and MB of filesystem used by that (good filesystem don't have an inode limit anyway). My machines do always boot even if random stuff is missing or corrupt (and they won't with udev etc.). I've converted all CentOS/RH machines back to the good old style /dev system and it's been a relief: no more initrd, just boot the damn kernel via whatever you like. The startup is faster, too. All in all the system is simply robust.

Separating system directories (like /usr and /var) is what has made Unix great. It even helps performance (because of lesser locking) and fragmentation (yes, this is a problem on a write-intensive FS), let alone robustness (one FS can corrupt easily, but not 3 at the same time).
Being told by a freaking little program and their developers (who don't "believe" in separation) that I cannot do that is just ridiculous.
They've had their head too long in that "always on" mentality; they've not grasped proper System V init; they've forgotten about system init (not yet all mounts and maybe R/O) vs. system running (all's there) phases, which is a base paradigma of all technical systems.

I express my shock that those people are allowed to change the future of how we will be running our Linuxes; they maybe should get back to working on Ubuntu, where the "en vogue" seems to trump robust working concepts, and leave the rest of the world alone, just alongside with the Gnome people for removing important window manager buttons from their console.
We don't like to play with our systems, we like to run them in whatever hell breaks loose upon us.

Alvaro Ortega : Now that we all are running fancy desktops full of powerful applications

kleptog — Sat, 05 Mar 2011 20:36:01 +0000

The part I find best about editing files is the permanence. When you edit a configuration is a file you know it'll be there next boot.

On the other hand, most GUI tools don't even try to explain. This setting you're changing, is it just for this session or forever? When I change the volume on my computer it seems to remember this across boots. There's no distinction between defaults after booting and what you want the settings to be now.

That said, I'm not sure if there's a nice solution for this. Which brings you right back to: when you edit a file, you knows it's forever.

Quotes of the week

nix — Fri, 04 Mar 2011 19:32:32 +0000

No, systemd is executing a program in /lib/udev. The problem is that a few programs in /lib/udev depend on glib (which is often in /usr/lib) and a few depend on the USB or PCI databases under /usr/share. Without /usr, those databases are inaccessible, so everything which needs them will fail during coldplugging.

Systemd incompatible with mounted /usr

nix — Fri, 04 Mar 2011 16:06:56 +0000

Yep, that's exactly what I thought. All the applications which might fail because usr/ is not around are various udev extras, principally usb_db and pci_db. The default rules use these for something to do with Dell Bluetooth devices and to dig vendor descriptions out of the USB/PCI database. In fact, looking at the udev 164 source code, it looks like the stuff derived from the USB/PCI database lands in device properties then is never used for anything...

... ah. I see. PulseAudio uses it, nice example of an unadvertised coupling between programs, that. What does it use it for?

... to populate ZeroConf / Bonjour data, and to set vendor name properties on sources and sinks. Is that *all*? You're emitting a scary warning and triggering flamewars over *that*? Small wonder I never noticed it wasn't working. I hope you're planning to use it for something else in future, or this has really been a tempest in a teapot.

(I note that mapping USB/PCI IDs to vendors is something perfectly doable at runtime, when the system is likely to be up and /usr will be mounted: there's no need to do it at coldplug time. It's not like the USB or PCI databases are root-readable-only or anything like that, and the underlying IDs are stuck in the udev database regardless. However, it might require enhancements to libudev to do that, which is more coupling than the existing 'run usb_id/pci_id' approach.)

Quotes of the week

nix — Fri, 04 Mar 2011 15:51:05 +0000

No, that was me!

Moral: never post after four hours of commuting. (Of course, if I'd followed that advice for the last four years I'd never have posted at all.)

Quotes of the week

Wummel — Fri, 04 Mar 2011 15:28:26 +0000

And I agree. You explained the warning message and the misunderstanding is cleared up.
Let's move on to fixing the technical problems that lead to the warning message.

Quotes of the week

Wummel — Fri, 04 Mar 2011 15:22:35 +0000

Yes, the Wiki link clarifies the warning message. The misleading warning text was the reason for the misunderstanding that most people (including me) had.

So the real problem occurs when
a) systemd queries udev information which executes a script in /usr and
b) /usr is not yet mounted.

One suggested solution is to move scripts from /usr/bin to /bin, another solution is to wait until /usr is mounted.

Hopefully, there will soon be concrete bug reports with the offending udev rules so that those problems can be fixed.

Systemd incompatible with mounted /usr

mezcalero — Fri, 04 Mar 2011 15:02:59 +0000

Well, I am not sure I can be more explicit than in the wiki page. But here's another try to explain this to you in simple words:

1) applications install udev rules

2) these are executed at early boot when the hw is detected

3) these rules fail, since /usr is not around

4) later on the apps won't see the devices properly, or can't see some of their features since the rules didn't get executed, and the udev devices lack the properties that should normally have been set had the rules been executed properly.

Quotes of the week

mezcalero — Fri, 04 Mar 2011 14:38:58 +0000

I never publicly called anybody an idiot.

Quotes of the week

mezcalero — Fri, 04 Mar 2011 14:30:14 +0000

BTW, I modified the warning in systemd git now to point users to http://freedesktop.org/wiki/Software/systemd/separate-usr... where we try to explain the situation in more detail.

LWN: Comments on "Quotes of the week"

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

/etc/mtab as a file not supported?

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with...

Systemd incompatible with mounted /usr

Systemd incompatible with mounted /usr

Groundless Hype?

r/o root and /etc/udev/rules.d/*persistent*rules

smells Richard Hughes, btw

Alvaro Ortega : Now that we all are running fancy desktops full of powerful applications

Systemd incompatible with mounted /usr

separate /usr isn't just about r/o or NFS

Systemd incompatible with mounted /usr

Systemd incompatible with mounted /usr

separate /usr isn't just about r/o or NFS

systemd

Quotes of the week

systemd

systemd

Alvaro Ortega : Now that we all are running fancy desktops full of powerful applications

Quotes of the week

Systemd incompatible with mounted /usr

Quotes of the week

Quotes of the week

Quotes of the week

Systemd incompatible with mounted /usr

Quotes of the week

Quotes of the week

r/o root and /etc/udev/rules.d/persistentrules