Udev rules and the management of the plumbing layer

For a little while, it seemed like that something might be devfs, but that story did not end well. The real solution to the /dev mess turned out to be a tool called "udev," originally written by Greg Kroah-Hartman. Udev would respond to device addition and removal events from the kernel, creating and removing special files in /dev. Over time, udev gained more powerful features, such as the ability to run external programs which would help to create persistent names for transient devices. Udev is now a key component in almost all Linux systems. It's like the plumbing in a house; most people never notice it until it breaks. Then they realize how important a component it really is.

Udev is configured via a set of rules, found under /etc/udev/rules.d on most systems. These rules specify how devices should be named, what their ownership and permissions should be, which kernel modules should be loaded, which programs should be run, and so on. The udev rule set also allows distributors and system administrators to tweak the system's device-related behavior to match local needs and taste.

Or maybe not. Udev maintainer Kay Sievers has recently let it be known that he would like all distributors to be using the set of udev rules shipped with the program itself. Says Kay:

This request was surprising to some. A Linux system is full of utilities with configuration files under /etc; there is not normally a push for all distributions to use the same ones. So why should all distributors use the same udev rules? The reasoning here would appear to come down to these points:

• The udev rules files are not really configuration files - they are, instead, code written in a domain-specific language. For a distributor to change those files is akin to patching the underlying C code; far from unheard of, but generally seen as being undesirable. As a way of underscoring this point, the udev developers are moving the udev rules out of /etc and into /lib.

• There is little reason for distributors to differentiate themselves based on their device naming schemes, and every reason to have all Linux systems use the same device names. For the situations where reasonable distributions may still differ - which group should own a device, for example - there is a mechanism to add distributor-specific rules.

• Increasingly, other packages will depend on a specific udev setup for the underlying system. Distributors which use their own rules will have a harder time making these new tools work right.

That last point refers, in particular, to DeviceKit, a set of tools designed to make the management of devices easier. Between them, udev and DeviceKit are being positioned to replace most of the functionality in the much-maligned hal utility. See this posting from David Zeuthen for lots more information on DeviceKit and the migration away from hal in general.

The only problem is that some distributors aren't playing along. Marco d'Itri, the Debian udev maintainer, responded that a common set of udev rules is "not going to happen." The default rules, he says, do not meet Debian's need to support older kernels, and, besides, "I consider my rules much more readable and elegant than yours". Ubuntu maintainer Scott James Remnant is also reluctant to use the default rules.

Scott appears to be willing to consider a change to the default rules if it can be made to work right; Marco, instead, seems determined to hold out. When encouraged to send patches to improve the default rules (and make them more elegant), he responded:

It appears likely that most of the distributors will come to see the udev rules as code which is to be maintained upstream; even Debian may come along eventually. As this happens, the layer of "plumbing" which sits just on top of the kernel should be worked into better shape. Kernel developers may find themselves involved in this process; David has posted a proposal that all new kernel subsystems, before being merged, must be provided with a set of udev rules. That would help the udev developers get a set of default rules into shape before the distributors feel the need to step in to make things work.

Increasingly, the operation of the kernel is being tied to a set of low-level user-space applications; there is not much which can be done with a bare kernel. How all of this low-level plumbing should work, and how it should interoperate with the kernel, is still being worked out. The management of udev policies is just one of the outstanding issues. So the upcoming Linux Plumbers Conference would seem to be well timed; there's a lot to talk about.

I think you're missing the aim of this proposal.  Think of it as parallel to the Linux
Standard Base - a set of rules that is reasonably comprehensive and comprehensible between
distros.  That way a Debian sysadmin can look at a SuSE machine and still know their way
around the udev rules, without throwing up their hands and saying "I don't know, Debian does
it differently to everyone else".  Additions, deletions and modifications to that set of rules
won't surprise anyone if they're done in a way that's consistent across all distributions.

In other words, no-one's saying "you must have rules for the Foomeister 3000 video card on
your system", they're saying "rules for video cards will look like this and be stored here".

Frankly, a Debian developer saying "no, the entire rest of the community must conform to my
ways" is a bit disappointing but doesn't surprise me either.  You'd think the OpenSSH debacle
would have taught them a bit of hubris...

Or, to correct myself, humility.  I think they probably have enough hubris... :-)

> Frankly, a Debian developer saying "no, the entire rest of the community must conform to my
ways" is a bit disappointing but doesn't surprise me either.  You'd think the OpenSSH debacle
would have taught them a bit of hubris...

I think that's a bit too sweeping generalization. Marco is only speaking as himself, you
should not make the assumption that that's the way the whole Debian community thinks/acts.
Since the openssl debacle there is now a large group in Debian that believes any divergence
from upstream is _bad_. But since we are talking about a very large group pf people, suprise
suprise, there is a large range of opinions and attitudes. In future please don't label a huge
group on one members actions. Or atleast reserve that for /. - lets keep lwn.net at a bit
higher quality discussion.

At the macro scale, I'm afraid the situations are quite a bit more similar than you realize.
Both involve Debian developers believing that their way is more "elegant" (an awfully
subjective term) and patching upstream with little regard for the chance that upstream
understands the code quite a bit better than they do, and little motivation to push those
changes back upstream.

If it was for additional functionality, then I could understand.  But elegance??  What an
awful reason to diverge!

> Having a Linux user be told that it's "my way or the highway" kind of
> defies the allure of running Linux to begin with.

a) The Linux user is not being told to do squat, it was a call for support
from distributions.

b) As mentioned in the article there is already a mechanism in udev for
local changes.  What's being requested is that the default provided by the
different distributions are consistent.

It is easier for the user to do what he wants when he needs to make the
same set of changes between distros he uses instead of having to have
distribution-specific changes to achieve his goal.

If it works, I don't care. If it doesn't, and hardware integration is one
area where it sometimes doesn't, I want something easy to hack.

I suppose if everyone uses the same setup, bugs will be fixed, and eventually
I won't have any need to know where the configuration files are.

I hope.

Derek

Out of curiosity, have you ever *really* spent time optimizing your udev rules?

Doesn't everybody? On any computer with more than one network interface, I use custom udev
rules to give each one a explanatory name ("inside" and "outside" for firewalls, "wireless"
and "wired" for my laptop, etc; the default rules make the names persistent, but you still
have to remember which is which). I've also used rules to do different things with different
USB devices that would otherwise all be undifferentiated CDC-ACM devices.

I tried to create a working automatic /dev/tape symlink on debian etch. For an hour or so. I
was unsuccessful although I followed the manual. After reading this article, I suspect the
optimized debian rules are the cause which spots that the rules may be more elegant, but the
documentation for them was not findable.

Not optimising, but I have had to fix the occasional bug when it failed to detect a filesystem
or something.

Change your udev rules all you want.  Nobody cares.

The proposal had abso-fucking-lutely nothing to do with taking away "choice" (ugh, that word
is like poison now thanks to people like you).  It's about distributions shipping a standard
set of rules that comes with the udev package so that you, the user, can exercise your
"choice" and install third-party packages that rely on udev without yourself having to
manually integrate said package's rules into some arbitrary set of unknown udev rules shipped
by your distribution.

This proposal makes it EASIER for a user like you to customize your distribution by making it
easier to install and change software.

Actually one of the things mentioned is moving the files to /lib/ because you shouldn't tweak
them, just "edit them as a whole"....

Remind me again... didn't a certain CEO of Canonical go out of his way to make a public call
for cross distro collaboration in the last couple of months?  And now, one of his companies
employees is reluctant to change how they are doing things...to aid in cross-distro
collaboration with regard to udev? Perhaps Shuttleworth should send a few more internal
company memos and get his employees on board before his next public challenge to the open
source development community.  I would have thought a Canonical employee would be jumping at
the chance to publically prove that Canonical was really as interested in bringing down the
barriers between distros given Shuttleworth's passion about it previously...even if there are
technical concerns on how to do it right.  

-jef"Walk the walk..if you are going to talk the talk"spaleta  

The Ubuntu developer seems to be genuinely interested in an improved set of rules for _all_.

The Debian dev is the only one that doesn't seem to be interested in harmony at all, unless
everyone else does it his way...

Marco is known to be difficult to deal with, reminding me of Debians Branden Robinson in his
early days (before he calmed down a little bit :-)

Hopefully Marcos contributions to udev will be as fruitful as Brandens efforts with the X
Window System. I'd really like to see some patches exchanged between Debian and upstream
authors.

Well, as is made clear in that thread, Remnant has various technical concerns about using the
upstream udev rules. They discuss the problems, but it doesn't seem like they are anywhere
close to a a mutual understanding of the actual issues. So, at this point, I would expect
caution and not immediate enthusiasm for change.

Stone's Law: In any discussion mentioning Canonical or Ubuntu, when any issue tangentially
related to freedom, contributions or similar is raised, the odds of a non-technical Fedora
person slagging Canonical and Ubuntu off rapidly approaches one.

I'm not saying you're wrong, just that I'm unbelievably bored of it.  Do you genuinely have
nothing better to do?

This is a bizarre comment...  jspaleta has a very good point: Shuttleworth and Remnant
definitely appear to be at odds here.  I don't understand the point you're trying to make.  If
you're so bored of it (what? jspaleta making a point about inconsistency?), you could just
close the window and go do something else yourself.  Why the reply?

Mark saying 'we need to be serious about cross-distribution collaboration and grow the common
ground' is a very good thing that I can only hope comes to fruition.

Scott stepping forward with technical concerns doesn't mean he's pulling a Marco d'Itri and
attempting to deny the rest of the world.

Why hasn't Fedora adopted dpkg? Debian's procps? etc.

Just because you have valid technical concerns about one proposal for further
cross-distribution collaboration (and a proposal on how to fix it -- see his blog), doesn't
mean you're completely anti-collaboration.

I posted the comment because I'm absolutely sick of it.  If people want to lambast Canonical
and Ubuntu, there are absolutely valid reasons to do so (cf. gregkh's analysis of their kernel
contributions, or lack thereof), but most of the time I see the Fedora leaders (mainly Jef and
Max, really) speaking about them, it's yet more uninformed and petty bullshit.

Any package is not in Fedora either because it doesn't meet the legal requirements or somebody
hasn't volunteered to do it. Fedora does have dpkg in the review queue already and procps
(from http://procps.sf.net) is in in Fedora already although I don't have any idea of Debian
has a different fork. Packaging formats are rather uninteresting. What you want to look at is
PackageKit and several other projects developed within Fedora that aid distributions in a
level that makes sense for everybody.  

Btw, syncing on distribution release schedule is different from upstream collaboration and
upstream collaboration has been advocated by Fedora for a very long time

http://fedoraproject.org/wiki/Objectives
http://fedoraproject.org/wiki/PackageMaintainers/WhyUpstream

Perhaps you can also learn to differentiate between personal opinions and project view points
too. 

Sigh.

Yes, I know that.  I've seen (particularly within X.Org) that Fedora is a fantastic free
software citzen, and its upstream work has been nothing short of amazing.  I also admire and
respect the dedication to software freedom and improving the entire free software ecosystem
that no-one else embodies.  (Honestly, I'd be using it if it actually, y'know, worked.  But
that's another matter.)

It was rhetoric to make a point: just because Fedora have not adopted dpkg as their base
packaging format (not just provided dpkg), does not mean that they're not interested in
cross-distribution collaboration.  Of course, it would be nice if RPM simply ceased to exist,
but that's not going to happen, and that wasn't my point.

My point was that just because you hold back for whatever reason on doing one thing which
would increase cross-distribution collaboration, that you're not completely against
cross-distribution collaboration.  Hopefully that's a reasonable enough statement that
everyone agrees.

I don't see what your beef with RPM is. 

In a similar way, if someone makes a opinion, there is no need to paint a broad brush just
because the person also happens to be a Fedora contributor. Again, you need to differentiate
between opinions representative of a project and personal ones. 

He has identified a pattern.  He has requested that the pattern stop.  

This is not an overbroad brush.

It might be wrong, but it is not on its face an overreach.

If you want to see why there are differences of opinion on the virtues of dpkg and RPM you can
just do some simple web searches.  There have been endless threads on the topic.  I don't find
the discussion personally interesting but there are definitely technical differences that
matter in some cases.

You need citations in order to demonstrate a pattern. I don't see that happening here. I don't
see why someone trying to demonstrate that there is unwanted hostility would want to destroy
his own argument by wanting RPM to completely ceast to exist either. Sure, there are
differences and in some cases there are advantages to each of the solutions too. Remember,
Intel switching from Ubuntu to Fedora claiming RPM to be a reason?

You continue to miss the point by such an astounding margin that I think it's better if we
walk away and just forget this ever happened.

If you don't have a point worth understanding, feel free to walk away. I thought you had one
while engaging in this conversation. Good luck.

That makes sense.  Remnant's misgivings seem easily solvable...  afict, he just needs upstream
to make groups work for initramfs, and upstream sounds amenable to those changes.  That would
be most excellent.

I'll try not to let Marco d'Itri's "elegance" windmill-tilting skew my interpretation of SJR's
comments.  :)

Canonical employees aren't whipped into speaking the party line, y'know.. =)

There's healthy room for discussion around an idea - Scott's spent a huge amount of time
tweaking the startup and shutdown times of Ubuntu systems.  It wouldn't surprise me if Ubuntu
now relies on some udev tweaks that aren't in upstream.  I'm sure that it will, like other
distros, eventually settle into the mold. But Ubuntu usually doesn't do that too quickly. =/

 - Jeff "former Canonical employee" Bailey

I think you misunderstand the comment. It has nothing to do with desktop or server software.
Rather, important functionality is increasingly required from user-space (not linked into the
kernel) daemons that run at the system level.

> It has nothing to do with desktop or server software.
 But it has to do with desktop/server deployments. Embedded systems with 
a fixed set of hardware can often still be built with all the drivers 
built-in, a fixed /dev containing the correct nodes, and no need for 
early userspace or udev or probably most of the daemons you had in mind.     
There is plenty that can be done with a bare kernel and a minimal 
userpsace to push config. into it, just not in the desktop space.

Well at somepoint it's going to be cheaper to just run udev than mapping everything on your
own, even in embedded.

On my embedded systems, which are busybox-based, I use mdev instead.  Same basic idea though.

Isn't it fair to assume the desktop space by default and make it explicit if you're talking
about embedded Linux?

I think that this poster is referring to the embedded problem. for embedded systems udev is
not an admantage, it's something to remove. currently they can get along just fine with an
old-stule /dev/directlry (populated only with the devices that actually exist)

as long as these things are kept optional I don't see this as a big problem (although the
authors of these tools do tend to forget that not every system is going to use them)

In my mind the bigger problem is that these tools are a middle ground between the kernel and
userspace, sometimes they are tied to one and sometimes to the other. this can cause surprises
when upgrading, and it's all to easy to get into a bind where you can't upgrade one piece
without having to upgrade a lot more, or where once you upgrade you can't go back.

Indeed, the Openmoko guys found tat removing udev and prepopulating /dev saved them over 5s of
boot time - something very desirable in a cellphone!

Doesn't OpenMoko take about 1.5 minutes to boot up?

Whiskey Tango Foxtrot?

I recently hit a small issue with this...

My HTPC system runs debian etch. Suspend is broken on this system, patches are available for
some kernel versions. So, I upgraded my kernel to 2.6.26 plus some ACPI patches, and now
suspend works... But, debian's udev won't create the usual devices for my SATA DVD drive after
I upgrade the kernel to 2.6.26. /dev/scd0 is created normally, but /dev/dvd and /dev/cdrom
links are not showing up.

I can't really complain to debian, since they don't support that kernel version. But, I can't
easily do much about this either, since updating my udev to a newer upstream version would be
extremely likely to break more things than it fixes (not bashing upstream udev here, just that
things would end up not set up the way debian expects them).

lenny is expected to ship with 2.6.26. udev really ought to be working with it.

Yes, at the moment 2.6.25 is in testing but we're working on getting 2.6.26 in.

It may well be that the udev version in etch is too old to work with recent kernels. Udev and
the kernel should always be upgraded to somewhat matching versions, together.

You should definately file a bug about this anyway--if etch's udev does not work with lenny's
kernel then we will need to add instructions on how to upgrade both together to the lenny
release notes (as we did for the sarge -> etch upgrade).

Turned out enabling CONFIG_SYSFS_DEPRECATED_V2 in my kernel fixed it. Marco pointed that to me
after I filed the bug. So, assuming lenny's kernel will be built with this, there should be no
issues.

I want init=/bin/bash to just work, among other things. I like things simple.
So I've a static /dev/ with only the device files that are needed (and a 
few more), and use udev for hotplug stuff. I let it manage /udev/. No
HAL or dbus on my system either. Everything works just fine. Then again,
I fail to see the excitement about running KDE or Gnome, so I'm probably 
strange and missing something.

If you never tried KDE or Gnome, then believe me, you _are_ missing something. I use both, and
though I prefer KDE, I do find Gnome awesome.

Of course I tried both. Thing is, I'm used to do things from a terminal, 
and then desktop environments like KDE and Gnome add very little value,
or so it seems. Maybe they have some great programs that are bundled up 
with them, but can use those without running Gnome or KDE too I'd hope. 
Can you explain what all the excitement is about?

I'm maintaining dev-minimal package in ALT Linux which is sufficient for rescue jobs (several
dozen entries just in case there's no udev to overlay these).  Do the same?

And if you're using initramfs, there might be a stripped-down udev there which would take
care...

Thanks for the backgrounder.

I discovered recently that my Debian boxes will assign /dev/sda1 to a usb drive, rather than
to the internal scsi drive that Debian booted from. This is a gaping security hole to say the
least.

So can I change this behavior by modifying Marco's lovely ruleset? Guess I'll find out.

I know, I should use disk labels or uuids... but then why doesn't the Debian Installer use
disk labels or uuids? 

How is that a security flaw? If it really is so, please please *please* file a bug so it can
be fixed!

If it happens in initramfs then plugging an USB stick in while booting will 
give you control over the machine. Which is only relevant if the machine
is protected in other ways, or else they could boot from something else
anyway.

Not sure how it can be abused after bootup though, except by causing
confusion.

Well okay, you need physical access to the box, which is pretty much game over in terms of
security anyway, so this is more of an annoyance than a security issue.

The system boots from internal scsi raid, but after kernel loads scsi and usb drivers, it
remounts the filesystems according to /etc/fstab. Problem is, the usb drive is seen first so
gets /dev/sda and the boot drive gets /dev/sdb. 

Bootup craps out because the usb drive doesn't have /sbin/init. I was thinking that if the usb
drive had a full, working Linux system on it, an attacker would have control of the system.
But lets face it, if someone can get to the box, plug in a usb drive, and reboot, you have
bigger problems.

I'll file a bug, but I remember seeing an earlier one that said that device assignments aren't
guaranteed, so use labels or uuids. The bug in that case should be filed on the installer.

Not that I would see a security flaw, rather an annoyance, I think there were problems with
Grub and UUID booting that were resolved recently (at least in Debian)...

BTW, it's the kernel itself that decides what is sda and what is sdb. Udev just uses the
kernel names for such devices.

Udev also sets up symlinks in /dev/by-{id,path,uuid,label} that will always point at the
device containing the correct filesystem. These, or the UUID= or LABEL= syntax should be used
when referring to your filesystems.

Maybe it's just that the installed still uses the 'legacy' device paths in the /etc/fstab file
that it generates. That should really be changed, but it may well be too late in the release
cycle to start on it now.

Too bad you can't put udev under FHS and tell everybody to STFU.

From the experience (admittedly very little) with writing rules for the Wacom tablet PC, I
prefer the Ubuntu rules. The Wacom digitizer found in tablet PCs has 2 sub-units, 1 for the
pen input and 1 for the touch screen. The standard rules choose the touch screen as
\dev\input\wacom, which is not really very helpful. I found that on Ubuntu enough information
is provided in the hierarchy of udev nodes to differentiate the 2 sub-units, but this is not
present in the standard set of rules (as seen in Gentoo). At the time I was very surprised
(and annoyed) that the rules weren't standard. 

Get out of here.  No one is interested in your throughtful, erudite comments based on
real-world experience.  We prefer long-winded rants based on juvenile fantasies of a world of
hot, nerdy women customizing udev while feeding us grapes and whispering dirty python jokes in
our ear.

Unlike the other commenter I have a more thoughtful response to your experience.

I see this as exactly the reason why distributions should converge on udev rules. You are
seeing the consequences of divergence, you are expecting the same results on all
distributions, which is why the Ubuntu rules should be sent upstream so that all may benefit
from them.

That Ubuntu has not done so, or upstream has decided not to merge them, is concerning.

uhh, that was a joke

While I did know that, I found it out of place, so I found it more productive to give a good
answer instead of making jokes.

I do not mind jokes, but I just felt his post deserved a better reply than yours.

The thing I find strange is that there is any kind of stand-off at all.

Greg K-H and Kay Sievers can preempt Marco's comments by going here:
http://packages.debian.org/sid/udev

Download the diff.gz, applies it to the orig (or upstream tar ball) and see what Marco's rules
are and if they are indeed better.

If we look at it from Marco's (or more generally any Debian maintainer's) point-of-view, he's
already posted his patch in one of the most accessible ways possible: to the web for upstream
to take notice of. Many upstreams do take notice and merge stuff back.

I can already see Greg / Kay's point-of-view from the article, but seriously, I don't
understand the reluctance to examine and, if appropriate, rebut Marco's comments based on
examining his rulesets.

afaik, the debian rules were directly available in the upstream git repository but deleted a
few days ago:
http://git.kernel.org/?p=linux/hotplug/udev.git;a=commit;...

It would be interesting to go back and look at the various initramfs' in the distros at some
point.  When I wrote initramfs-tools in Ubuntu (which was adopted and later largely taken over
by Debian Developers), there were two camps: One saying that anything for early userspace
startup (up to and including partition mapping) should be done in userspace, and some saying
that it should always be in the kernel.

I'd guess that we're far enough along do either finish putting things into klibc to make this
work or give up on the notion.  I can't guess, though, which it might be.