Weekly Edition
Return to the Kernel page
| |||||||||||||
Char devices for network interfaces
One of the longstanding quirks of BSD-inspired networking is that network
interfaces are a strange sort of device. They live in their own namespace,
do not appear in /dev, and generally fail to live up to the
"everything is a file" idea that drives much of the POSIX API. That said,
the Unix way of networking has functioned well for nearly 30 years. It is
likely that few people were expecting a serious patch which tries to change
things.
This patch from "Narendra K" at Dell does exactly that, though, and in surprising ways. With this patch in place, every network interface gets an associated char device. It is a singularly useless device: any attempt to open it just returns ENOSYS. The only real reason for this device's existence, it turns out, is to generate events for udev which, in turn, can generate alternative names for the interface. Why this change? System vendors and administrators are getting tired of their network interfaces changing name at each boot. Non-deterministic interface naming is the result of a few factors, including weird BIOS behavior and the way current kernels enumerate devices via a parallel hot-plug approach. When interfaces change names, configuration scripts can get confused; the end result is rarely a working network. Some of this confusion can be avoided by carefully configuring interfaces based on their MAC address, but that, too, can fail in the face of "swap out the entire server" approach to fast failure recovery. Vendors have tried to work around some of these difficulties at the hardware level. Dell machines are designed to enumerate network interfaces in the same order as often as possible. HP blade servers can configure interface MAC addresses at power-on time. But there are limits to how many hardware hacks the vendors are willing to add to deal with this problem. This message from Matt Domsch is recommended reading for anybody wanting a full understanding of the difficulties here. Thus the patch, which allows udev to create pseudo-names for each interface based on criteria like the PCI slot number, chassis label, or anything else that seems to make sense. The patch is tied to the libnetdevname library, which maps these pseudo-names into the real interface name, which can then be used with the socket system calls. The patch has gotten a bit of a rough reception; it looks to some like a hack for problems which can be solved in other ways. The discussion has made it clear that a real problem exists, though, so some sort of solution will likely be applied in the end. Udev seems like the place for this solution to happen - that is how naming has been handled for every other device in the system, after all. So expect something to get in eventually, though the current may evolve somewhat before it is considered ready for merging. (Log in to post comments)
Char devices for network interfaces Posted Oct 15, 2009 2:03 UTC (Thu) by marineam (subscriber, #28387) [Link]
udev has handled persistent names (even automatically!) based on MAC address for ages... I even had to submit a patch to disable the behavior on Xen because the ethX name was getting incremented by 1 every time the VM rebooted and got a new randomly assigned MAC address. :-P
Is everyone using distros that like to disable this feature or something?
Char devices for network interfaces Posted Oct 15, 2009 4:14 UTC (Thu) by gdt (guest, #6284) [Link] You can't assume the MAC address remains the same before and after a repair. Sysadmins curse every time a faulty mainboard is replaced and then the operating system fails to come up properly. Read Matt's mail, it's a good summary of the current suckiness of network interface name enumeration. Although even it misses some classics, such as some Intel motherboards being marked NIC1 and NIC2 and getting eth1 and eth0 respectively assigned to those interfaces, despite DMI being correct.
Char devices for network interfaces Posted Oct 15, 2009 6:28 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
I am currently removing udev from my server systems entirely (in large part due to this issue)
prior to this I had manually removed the entry in the udev config that had it deal with network devices.
while I understand the theory (you don't want a device to change names and send traffic out the wrong interface), this feature has always caused me significant amounts of trouble, and I've never once had it save me (I manage several hundred firewalls, so in theory I would be the prime beneficiary of this feature)
Char devices for network interfaces Posted Oct 15, 2009 12:47 UTC (Thu) by nix (subscriber, #2304) [Link]
Er, you can tell udev to keep its hands off your network interfaces by simply removing the network generator rules. There's no need to ditch the whole thing.
Char devices for network interfaces Posted Oct 15, 2009 17:46 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
yes, and I did that for a while, but then an update to udev added it back in.
Char devices for network interfaces Posted Oct 15, 2009 18:54 UTC (Thu) by nix (subscriber, #2304) [Link]
This of course is the downside of having this sort of thing in a set
of 'standard rules': it's hard to eliminate them without hacking udev before installation :/
Char devices for network interfaces Posted Oct 15, 2009 19:07 UTC (Thu) by joey (subscriber, #328) [Link]
Removing rules files would work fine if a) using dpkg and b) udev had left
its rules files as conffiles under /etc. This sort of situation is why dpkg does not add back deleted conffiles on upgrade. But since udev's rules files have moved to /lib/udev, deletion of rules files will no longer persist across upgrades.
Now it seems you'd have to create an empty version of the net generator rule
Rather a mess.
Char devices for network interfaces Posted Oct 15, 2009 20:09 UTC (Thu) by nix (subscriber, #2304) [Link]
That was exactly my moan :)
I'm not sure that creating an empty version of the net generator rule file
> It does not matter in which directory a rule file lives, all files are
which could go either way.
Char devices for network interfaces Posted Oct 16, 2009 3:02 UTC (Fri) by njs (guest, #40338) [Link]
dpkg-divert --add --local /lib/udev/rules.d/75-persistent-net-generator.rules --divert /lib/udev/die-die-die --rename ?
The use of /lib here does seem bizarre though.
Char devices for network interfaces Posted Oct 16, 2009 16:15 UTC (Fri) by nix (subscriber, #2304) [Link]
Well, /lib/udev was already in use for other udev stuff which has to be runnable before /usr is mounted (e.g. /lib/udev/vol_id).
But, yes, putting what damn well should be conffiles (as witness the fact that people want to change them, even if udev upstream is resistant) in /lib is demented.
Char devices for network interfaces Posted Oct 22, 2009 18:18 UTC (Thu) by jengelh (subscriber, #33263) [Link]
You are supposed to truncate the file, not remove it. It's the only way the SHA check will work as intended, presuming that the udev files are also marked as %config(noreplace) in rpm.
Char devices for network interfaces Posted Oct 16, 2009 3:21 UTC (Fri) by pjones (subscriber, #31722) [Link]
That doesn't really address the problem - when you boot the installer the first time, udev is basically randomly picking which one is which. This means that any setup an IT department needs to do elsewhere in the network relating to, for example, which port is which, can't be done until after installation - which may be too late.
Char devices for network interfaces Posted Oct 15, 2009 2:54 UTC (Thu) by BenHutchings (subscriber, #37955) [Link]
Actually, the char devices don't even emit uevents. The idea is that the minor number of the device node identifies the net device by its ifindex (an immutable identifier, unlike its name). The char driver only exists to reserve a major number because I (and possibly others) objected to creating device nodes without reserving their numbers.
I have some unfinished code to make the char devices usable as an alternate means of submitting ioctls to net devices. The current method is to call ioctl() on a socket, embedding the device name in the request structure, which can race with device renaming. However, this is probably unnecessary as most or all device control operations can be done through netlink-based APIs which already use ifindex.
Char devices for network interfaces Posted Oct 16, 2009 18:59 UTC (Fri) by giraffedata (subscriber, #1954) [Link] So this is just about keeping a directory of ifindexes? Character device special files seem like a pretty crazy implementation for that. How about a regular file that contains the ifindex? Or a single file that contains the whole directory? And given that the comments indicate that udev is already capable of setting the name of the network interface, why doesn't the network interface name provide the desired mapping?
Char devices for network interfaces Posted Oct 15, 2009 15:24 UTC (Thu) by Banis (guest, #59011) [Link]
This is a real problem that needs solved. When I switched from RHEL4 to RHEL5 every single one of my boxes with multiple network cards had the order of network interfaces randomized. For me that's about 100 boxes that used to have one port as eth0 with RHEL4 and with RHEL5 it was a different one. It was more than a little annoying. When I put a multi-nic card in a box it's a 30 minute exercise to figure out which new ethX device corresponds to which port on the new card. The order never makes sense and never maps to the numbered ports on the card.
Char devices for network interfaces Posted Oct 15, 2009 17:46 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
ethtool -p ethX is your friend, for many cards it will blink a light on the card so you can identify which interface it is.
Char devices for network interfaces Posted Oct 15, 2009 19:47 UTC (Thu) by Cato (subscriber, #7643) [Link]
That's nice but it doesn't really work for a large number of servers.
Char devices for network interfaces Posted Oct 15, 2009 19:59 UTC (Thu) by dlang (✭ supporter ✭, #313) [Link]
which is why my current builds remove udev ;-)
Char devices for network interfaces Posted Oct 27, 2009 18:54 UTC (Tue) by mdomsch (subscriber, #5920) [Link]
Thanks everyone for your comments here. Network device naming has been a recurring problem for several years, and I appreciate seeing that it's still a big problem for people. That helps provide the incentive to getting it fixed.
Char devices for network interfaces Posted Oct 27, 2009 19:18 UTC (Tue) by dlang (✭ supporter ✭, #313) [Link]
unfortunately I see most of this work as being a solution in search of a problem.
there are a handful of cases (USB devices) where simple detection order is non-determinant, but the vast majority of devices fall into two categories
1. a single interface
2. simple detection order produces reliable device numbering
yes, when you change a card it can change the detection order, but that is true for every operating system, and is seldom a real big problem (because people are used to dealing with it)
the attempts to solve this problem end up causing more grief for many people, so much so that I doubt if the solution is worth it.
when you have a fairly rare problem to start with, your solution must be _very_ reliable for it to be better than doing nothing.
if you have a problem that affect 0.1% of the population, and your solution has an error rate of 1%, you have made life worse for 0.9% of the population.
this assumes that your solution is something that is deployed everywhere. creating a solution like this and having it option, used only in the case where people are experiencing problems to start with, you have a very different situation, then you are making life better for 0.099% of the population and not making life worse for anyone.
Char devices for network interfaces Posted Oct 28, 2009 13:26 UTC (Wed) by mdomsch (subscriber, #5920) [Link]
If I didn't think it was really a problem, I wouldn't have spent so much time trying to find ways to solve it cleanly. :-)
By not changing how the kernel chooses to name devices, I can't break things for anyone. If they like the non-deterministic nature of device naming, and it works for them, great.
But that 0.1% you refer to, is around 20% of Dell's server sales, likewise for other server companies. Not insignificant to that userbase.
I'm not suggesting making this optional. I'm suggesting we add it into the base udev rules, where, if a platform provides additional information to help is provide "better" names, in addition to the normal kernel-provided names, that thsoe "better" names can be used too. For various definitions of "better", all simultaneously.
Yes, other OSs have this same problem too. I get to discuss the problem and approaches at solutions with them also.
|
|||||||||||||