LWN: Comments on "Systemd catches up with bind events"

udev in the wrong basket?

oldtomas — Fri, 04 Dec 2020 13:43:16 +0000

Watching this I get the impression that udev should be part of the kernel project.

Systemd catches up with bind events

joey — Wed, 02 Dec 2020 16:29:30 +0000

The horribleness of the udev rule file format surely has a lot to do with this. I mean, this is a config file that uses goto for control flow! Take a udev rules file and imagine it implemented in your programming language of choice, and consider how this problem might have been avoided or at least made much more tractable to fix across the code base.

Systemd catches up with bind events - works for me

Wol — Sat, 28 Nov 2020 19:29:38 +0000

How you say things can be so important ...

"Can't reproduce" implies you have tried to replicate the error, you've put in a bit of effort to help the person with the problem.

"Works for me", on the other hand, *could* mean the same thing. It could also mean "I don't suffer that problem, so I can't be bothered to look for it".

And then there's the language problem. I'm probably known for being a bit prickly about language and how, even when you may think you're speaking the "same" language, the identical word may mean different things based on the speaker's background.

Cheers,
Wol

Systemd catches up with bind events - works for me

jezuch — Sat, 28 Nov 2020 09:21:12 +0000

I guess there's a subtle difference between "works for me" and "can't reproduce". I, as the bug report responder, would never say the former as it souds kind of dismissive. Like in, "i reproduced your exact environment and it works for me". The latter admits that you're probably missing some context that makes the reproduction impossible.

But other people will feel differently about this.

Systemd catches up with bind events

AdamW — Wed, 25 Nov 2020 09:12:52 +0000

I'm not sure that *is* what was meant, to be honest - I'm not convinced the "fix" really is a fix.

What the condition is "trying to mean" is basically: "skip this whole script if we're not in some sort of scenario where a device mapper device has appeared or changed". "dm_end" is literally the end of the file: `GOTO="dm_end"` means "just don't do anything else at all".

This is how the comments look on my version of the file:

# Device created, major and minor number assigned - "add" event generated.
# Table loaded - no event generated.
# Device resumed (or renamed) - "change" event generated.
# Device removed - "remove" event generated.
#
# The dm-X nodes are always created, even on "add" event, we can't suppress
# that (the node is created even earlier with devtmpfs). All the symlinks
# (e.g. /dev/mapper) are created in right time after a device has its table
# loaded and is properly resumed. For this reason, direct use of dm-X nodes
# is not recommended.
ACTION!="add|change", GOTO="dm_end"

this makes it pretty clear that what we're really trying to do here is "do stuff if a device is being added or changed, don't do anything if it isn't". So I don't think changing the condition to `ACTION=="remove"` is necessarily a correct fix at all. After all, one thing that means is that we'll go ahead with the script if the action is "unbind", the counterpart to "bind". Is that what we want? Are we sure it isn't going to do anything wrong? I'm pretty sure the script doesn't expect it, though hopefully it'll wind up bailing on a later check and not do anything disruptive...

Systemd catches up with bind events

gswoods — Mon, 23 Nov 2020 21:11:48 +0000

As one who had a 35 year career as a system administrator but who knows only a little about kernel and systemd internals, all of this looks a lot like US politics, where the Republicrats and Demopublicans are more interested in blaming the other side for all our problems than they are in solving them.

Systemd catches up with bind events

anselm — Mon, 23 Nov 2020 16:18:15 +0000

Most of the interesting people seem to be using BSD these days.

It used to be that using Linux was the way to stand out from the crowd, to be nerdy and interesting and metaphorically show the finger to those stodgy Windows and Mac users.

Now Linux has been mainstream for a while and is no longer good for nerd cred. People's elderly relations can (and do) use it. This means that the people who were using Linux 20+ years ago, when it meant not being able to do certain things (that when pointed out, one would adamantly insist weren't worth doing, anyway), spending three days to get a new video card/monitor working, etc., are being forced into BSD if they still want to impress their peers. But that's not because of BSD's versatility, wide compatibility with popular hardware and peripherals, and technical excellence – it's because few other people want to use it. It's the IT equivalent of an Indian fakir's bed of nails; very comfortable and just the thing if you're a fakir, but an item of morbid fascination for others.

Systemd catches up with bind events

flussence — Mon, 23 Nov 2020 13:29:58 +0000

> Please remember why we do this stuff.

It's hard to tell corporate types to remember FOSS had a human element when they came from outside that culture entirely, their salary depends on gentrifying it out of existence, and in their off-time their hobby is talking over everyone to proclaim “Well Actually everyone uses our software because our software is great because everyone uses it”.

Most of the interesting people seem to be using BSD these days.

Systemd catches up with bind events - works for me

giraffedata — Sat, 21 Nov 2020 19:14:11 +0000

This thread is either about someone's serious misinterpretation of the "works for me" response as, "this is your problem; go away" or a misnaming of that actual response.

"Works for me" is a request for more information or diagnostic work.

But I've also been the recipient of the response, "What you're doing is too unusual for me to care about. Do what I do, and it will work." Many times. I'm creative. I suppose someone might characterize that as "works for me."

Systemd catches up with bind events

rahulsundaram — Sat, 21 Nov 2020 12:32:11 +0000

> Only for the desktop distro's. It's too heavyweight for the smaller ones like Apline, OpenWRT or Android, containers like Docker don't use an init system at all.

The vast majority of Linux distros including RHEL, SLES, Debian etc (not just the desktop ones) use systemd by default. Docker containers are not comparable to distros but some of them do run a init system and it is popular enough that several distros include a systemd-container package specifically for this purpose and systemd is not limited to a init system, so other parts gets routinely used in containers as well.

Systemd catches up with bind events

ras — Sat, 21 Nov 2020 06:59:04 +0000

> systemd has become the de-facto Linux init system or PID1 or whatever the hell you want to call it.

Only for the desktop distro's. It's too heavyweight for the smaller ones like Apline, OpenWRT or Android, containers like Docker don't use an init system at all. And that could well cover the bulk of deployed Linux instances.

Systemd catches up with bind events

pebolle — Fri, 20 Nov 2020 23:09:02 +0000

> With respect to Poe's original comment and the way that Wikipedia explains Poe's Law, you were already looking to parody my comment, a rhetorical tactic to dismiss as ridiculous something you can't dismiss as untrue.

You might never read this but I only quoted your hyperbole verbatim. How is that parody?

Systemd catches up with bind events

mathstuf — Fri, 20 Nov 2020 20:08:49 +0000

The key is to coax out the difference in the setup from the reporter. I find it hard to get relevant details from reporters sometimes. I know what I'm looking for, but reporters will sometimes trim output to what they think is important, missing the actual details that are relevant to diagnosing the problem. Screenshots instead of copy/pasted text are also a thing.

It's about communication. I certainly have more to learn on this front, but part of it is realizing the differences in knowledge and expectations on either side of the wire.

Systemd catches up with bind events

rahulsundaram — Fri, 20 Nov 2020 15:45:36 +0000

> It wasn't *exactly* what was said but it was the spirit of what was said.

I don't agree but even assuming that, works for me is a fine thing to say if you don't stop at that point. There was a query for more information. It's up to the reporter to pursue that further

Systemd catches up with bind events

k3ninho — Fri, 20 Nov 2020 15:45:05 +0000

I've blocked your comments. I considered the consequences of my words in this forum and audience and I think what I said was a cogent, reasonable and meaningful contribution. With respect to Poe's original comment and the way that Wikipedia explains Poe's Law, you were already looking to parody my comment, a rhetorical tactic to dismiss as ridiculous something you can't dismiss as untrue.

Beyond that, I don't have to care how you respond.

K3n.

Systemd catches up with bind events

k3ninho — Fri, 20 Nov 2020 15:16:52 +0000

>> Just saying "works for me" is being a horrible maintainer.

>That wasn't what was said however.

It wasn't *exactly* what was said but it was the spirit of what was said.

K3n.

Systemd catches up with bind events

nix — Thu, 19 Nov 2020 23:40:36 +0000

It updates in bursts. Frankly I'm fairly happy this sort of breaks-boot-if-it-goes-wrong stuff is mature enough that it doesn't *need* constant updates...

Systemd catches up with bind events

kpfleming — Thu, 19 Nov 2020 14:49:19 +0000

So much this. A kernel config option which causes random, undocumented, events to appear, both for existing devices and for 'nonsense' devices. Correctly formatted messages (e.g. not fuzzing), but with types that are never used for real devices.

Chaos engineering can be quite useful.

Systemd catches up with bind events

pebolle — Wed, 18 Nov 2020 19:32:32 +0000

> Life's going to miserable for everyone if we presume bad faith.

The point here is that the statement I quoted is entirely over the top but it's still impossible to be sure whether it was made sincerely or not.

Look: developers that say 'works for me' are simply stating a mundane fact. If things didn't work for them they could start working on a fix. (If they have the time and the motivation to do that, of course.) But as long as it's unclear what triggers the bug that's been reported to them they are about as clueless as any random person using their software. I'd guess that all of this should be obvious to the kind of people reading lwn.net.

So if I read a comment containing little treasures like "a further issue at the level of our civilisation and society" and "[something] gives people with power [...] a habit of denying the lived experience of users and the struggles that users have with our software" and "life-long denial of the lived experience and struggle of other human people" (human people!) then, yes, Poe's law kicks in one again.

Systemd catches up with bind events

cladisch — Wed, 18 Nov 2020 15:45:59 +0000

When you want to apply the same filter to multiple rules, the simplest way to write this is the equivalent of "if ACTION != remove goto past_the_rules". Writing the filter non-inverted would require another goto to jump over the second goto.

Systemd catches up with bind events

k3ninho — Wed, 18 Nov 2020 14:37:37 +0000

>>> There's a further issue at the level of our civilisation and society where 'works for me' gives people with power -- to fix bugs raised by users -- a habit of denying the lived experience of users and the struggles that users have with our software, which can become a life-long denial of the lived experience and struggle of other human people.

>Poe's law works both ways: one is never sure whether someone is sarcastic or sincere on the internet.

You have to live your life, I can't make this statement a positive for you if you've taken it on bad faith. Plus, I hope that you can overcome whatever made it difficult to trust words from a random internet person. Maybe the world also needs to change to allow you this.

Life's going to miserable for everyone if we presume bad faith.

K3n.

Systemd catches up with bind events

Cyberax — Wed, 18 Nov 2020 09:30:55 +0000

Be VERY evil and send nonsensical events with random names?

This is actually a thing in TLS. At least several TLS implementations send deliberately non-existing cipher suite names during negotiation to make sure that middleboxes don't encode stuff like "use the first cipher".

Systemd catches up with bind events

zurdo — Wed, 18 Nov 2020 08:52:02 +0000

I suspect I'm grossly misunderstanding something, or missing an elementary computing class here.

Given n possibilities, wouldn't the correct expression have looked more like `ACTION==remove` in the first place? If that's what it meant back when that was written, surely it was an option to specify the ACTION you want to run on instead of every ACTION you don't want to act on?

Systemd catches up with bind events

zuki — Tue, 17 Nov 2020 20:43:30 +0000

It's very hard to figure out which rules would need this legacy mode... I looked at various rules installed in Fedora when working on this and for many I couldn't say what the effect of the change will be.

If a file mentions BIND events, than it's pretty clear that it has been adapted for the new events. But for other files, it's hard to say anything without knowing if the kernel drivers for that type of hardware ever emit BIND|UNBIND events. If they don't, a rule that only seems to care about ADD|CHANGE|REMOVE and hasn't been modified in 10 years might still be fully adequate. In other cases the driver might emit BIND|UNBIND events, but the rule just doesn't need to do anything for them, and translating BIND to ADD would actively break things.

Overall, I don't think a mode like this would be extremely brittle.

Systemd catches up with bind events

pebolle — Mon, 16 Nov 2020 21:14:09 +0000

> There's a further issue at the level of our civilisation and society where 'works for me' gives people with power -- to fix bugs raised by users -- a habit of denying the lived experience of users and the struggles that users have with our software, which can become a life-long denial of the lived experience and struggle of other human people.

Poe's law works both ways: one is never sure whether someone is sarcastic or sincere on the internet.

Systemd catches up with bind events

mbiebl — Mon, 16 Nov 2020 18:33:19 +0000

In case this wasn't meant sarcasticly:

https://github.com/gentoo/eudev/commits/master looks like this project is pretty much dormant.
I see no signs that eudev intends to address this issue.

Systemd catches up with bind events

rahulsundaram — Mon, 16 Nov 2020 18:30:54 +0000

> Just saying "works for me" is being a horrible maintainer.

That wasn't what was said however. There was a question back on what makes the hardware different which seems to have gone unanswered. Given the wide variations in hardware, this is a reasonable question.

Systemd catches up with bind events

jezuch — Mon, 16 Nov 2020 17:59:18 +0000

You create a unit/regression test which mocks hardware to behave the way described by the bug reporter? Consult the spec to confirm that this is a valid use case? Just saying "works for me" is being a horrible maintainer.

Systemd catches up with bind events

Fowl — Mon, 16 Nov 2020 14:14:42 +0000

Couldn't some sort of "compatibility mode" be implemented for individual rules, where udev pretends the new events don't exist?

Systemd catches up with bind events

hkario — Mon, 16 Nov 2020 12:21:45 +0000

put yourself in the developers boots for a minute:

you get a bug report, you look at the experienced behaviour, you haven't encountered it before; you try it with your hardware, it's not reproducible; you look at code that *may* be related, it doesn't seem possible to trigger this kind of behaviour

now, what on earth can you do more than to ask for more information?

Developers aren't omniscient and omnipotent entities that exist beyond confines of space and time, entities that fix bugs based only on a fickle. They're human, and they need to understand the bug before they can fix it.

Systemd catches up with bind events

magfr — Mon, 16 Nov 2020 12:15:13 +0000

This is interesting and further enforces my half formed thought that matching of inverse patterns are bad since it assumed that the set of values är fixed and unchanged.

The example in the article was matching
~add|change when what is needed is remove.

For this poster the problem sounds like the reverse, he needs to match ~add|change and someone have "optimzed" that to remove.

This proves that one need to know what one is doing and that crap can be written in any language, in this case the udev config rule language.

One way to fix this is to document an ERROR event.
Any rule that mentions an ERROR event is broken.
Any action that happens when ERROR is issued is a bug.

Systemd catches up with bind events

k3ninho — Mon, 16 Nov 2020 10:56:17 +0000

>That is not trolling, that is an entirely fair response to "you don't see it because you don't have the right hardware."
Many years of past experience have given this a name: 'works for me'. Developer doesn't experience the problem and can't conceive of the imagined version of the code in their head not running as they think it will.

This is dull, unhelpful pushback -- what would be better if not actually helpful is to call no-op ACTION=="remove", GOTO='end_stanza' an antipattern making you think that add/change/remove are the only legitimate udev action types and core to this 4.12-changed-userspace issue.

(There's a further issue at the level of our civilisation and society where 'works for me' gives people with power -- to fix bugs raised by users -- a habit of denying the lived experience of users and the struggles that users have with our software, which can become a life-long denial of the lived experience and struggle of other human people. I get that, in software, unanticipated complexity means that fixes have to not also break other things and that makes an apparently-simple change expensive and unpredictable, easier to push back and not make changes. Here's the question from this rhetoric: Are we not the wizards and masters of these systems that we should be able to change them to work more correctly for more people?)

K3n.

Systemd catches up with bind events

pbonzini — Mon, 16 Nov 2020 07:55:18 +0000

> oh there's a bug in systemd that causes it to spam the buffer, please upgrade to a fixed version

Technically the upstream people couldn't have known, since the bug was introduced by an incorrect distro backport. And if a buggy systemd, one that spews assertion failures all the time, will slow boot down to a crawl, the systemd people might even consider that to be a feature. It can and will happen for kernel WARNs as well, and a buggy PID 1 is not much better than a buggy kernel. But these are details, and in general I think we agree.

What this shows to me, is that Linux is sorely lacking postmortems. Whenever Linus screams at me, I try to figure out what went wrong in my workflow and how I can improve it to avoid being screamed at in the future. On the other hand, if 5 years later people still believe that "debug" is a sacred part of the kernel command line (and not the more nuanced explanation that you gave), something went wrong on the kernel side in figuring out what happened.

LPC

nevets — Mon, 16 Nov 2020 00:38:53 +0000

I didn't like the direction LPC was heading and joined the committee to fix that. I happened to join *after* santa fe (as I had issues with that one). My biggest focus was to get away from being kernel centric and I believe I (with help from others on the committee) was successful.

That's why I asked you to come back and give it another try ;-)

-- Steve

Systemd catches up with bind events

nevets — Mon, 16 Nov 2020 00:20:47 +0000

It wasn't really the command line usage that was the problem. The real problem was that the command line triggered systemd to spam the kernel printk buffer so much that we lost all debug messages from the kernel. When reporting this as a problem, instead of saying there was a bug in systemd, we were told it was not a bug and systemd is perfectly fine using the debug command line to trigger writing debug messages in the kernel printk buffer. I thought the real solution was to separate kernel writes into printk from userspace writes, preventing userspace from overwriting what the kernel produces.

This would not have escalated the way it did if we were told from the beginning, "oh there's a bug in systemd that causes it to spam the buffer, please upgrade to a fixed version". But instead told to bugger off. Yes, it really is a lack of communication and good faith between the two communities and I hope we can work better in the future.

Systemd catches up with bind events

NYKevin — Sun, 15 Nov 2020 20:41:28 +0000

Eh, I'm unconvinced. IPv4 did that with Class E addresses, back when classful networking was still a thing. Nowadays, classes are otherwise dead, we're running short of addresses, and Class E (now known as 240.0.0.0/4) is still "reserved for future expansion" - because everyone hard-coded it as invalid, sometimes even at the hardware level. So you can't use 240/4 without losing backwards compatibility. But if you're going to do a compatibility break anyway, you're far better off switching to IPv6 altogether, as it gives you far more addresses to play with and is generally less of a hassle to administer (for example, it lacks NAT). So those ~268 million IPv4 addresses will likely never be publicly routable.

Perhaps the RFCs of the time could have been written with greater care, but the trouble is that at the time (see RFC 988), they were in the process of designating class D (224.0.0.0/4) as multicast. They didn't know what class E would be used for, so they couldn't just say "treat class E as if it were unicast, unless a later standard says otherwise." For all they knew at the time, they would later want to use class E for some even weirder thing, and unicast processing would have been inappropriate or even harmful. So they just left it as "reserved," and the people who had to actually make the silicon and software decided that "reserved" meant "invalid." IMHO, they didn't really have much of a choice.

In short: From userspace's perspective, "reserved for future expansion" means "I don't know what this value represents, so if the kernel hands it to me, the only not-wrong thing I can possibly do is crash." In some contexts, ignoring the value *might* be not-wrong, but it's hard for userspace to predict that in advance. Regardless, the kernel cannot rely on userspace taking any particular interpretation, because as Linus has previously said, they don't break userspace, even where userspace is wrong.

Systemd catches up with bind events

jthill — Sun, 15 Nov 2020 20:09:05 +0000

If the add event had been for fully-ready devices, shouldn't BIND have been more delivered *first*, and possibly spelled "CONNECT", for devices that aren't fully ready? Or --oh, I guess the check for whether the device needs prep done in userspace, in response to the ADD? Still, seems to me there's now no "device ready" event, because you can't tell what an ADD means, and that just feels wrong.

Systemd catches up with bind events

NYKevin — Sun, 15 Nov 2020 18:47:29 +0000

That is not trolling, that is an entirely fair response to "you don't see it because you don't have the right hardware."

If you don't tell them about the hardware that reproduces the bug, they cannot reproduce it. If they are unable to reproduce a bug, how are they supposed to evaluate a patch for that bug?

Systemd catches up with bind events

NYKevin — Sun, 15 Nov 2020 18:44:00 +0000

> which is the whole problem with the unixy philosophy of being liberal with what you accept, and strict in what you emit.

That's not Unix, that's Postel's Law, which IIRC originates from TCP/IP (where it is *also* an unholy mess, but of a different kind).

Systemd catches up with bind events

pbonzini — Sun, 15 Nov 2020 15:19:46 +0000

> Perhaps it is reasonable to consider systemd exempted from the kernel's ABI/API stability promise, because it is sometimes almost the only user of certain interfaces?

That's complicated. With more and more people using containers—including OS containers running a full-blown init system—it's not that rare to see very new userspace on old kernels or vice versa. This also means that it will be much harder to remove features in distro kernels: for example, even if your distro ships with an nftables-based iptables(8), there could be containers using the older iptables API.