LWN.net News from the source
ARP, the Address Resolution Protocol, is the means by which IP addresses are translated to physical layer MAC (usually ethernet) addresses. ARP makes local area networks work by enabling systems to find each other. When one system has a packet to transmit to another on the local network, it broadcasts an ARP request packet seeking a MAC address for a given IP address. Some machine (usually the intended recipient) hopefully responds with the corresponding MAC address, and the packet gets sent.
If a Linux system (with a default configuration) receives an ARP request on one of its interfaces, and that request is looking for an IP address assigned to any of the systems interfaces, the system will respond to the ARP request through the interface that received it. This response happens even if the interface involved is not the one to which the requested address has been assigned. Mr. Bloemsaat's problems came about because his system has two interfaces plugged into the same network. Both interfaces receive - and respond to - ARP requests sent on that network. Depending on the order in which the responses are sent, traffic could be directed to the wrong interface.
Mr. Bloemsaat included a patch which restricts ARP responses to the interface actually implementing the requested address. But, over almost a month of discussion, the networking hackers have made it clear that they do not intend to change the way Linux behaves. Their reasoning follows, more or less, these lines:
A lot of the confusion, it seems, comes down to a subtle difference in how systems handle IP addresses. Many (perhaps most) networking implementations treat addresses as "belonging" to the interfaces they are assigned to. With that view of the world, no network interface has any business responding to an ARP request for an address which is assigned elsewhere. Linux, instead, sees IP addresses as a property of the system as a whole. So it makes sense for an interface to respond to a request for one of the system's addresses, even if that address is normally associated with a different interface.
The networking RFCs make it clear that either view of IP addresses is
legitimate. Armed with that, and their sense of how things should work,
the networking hackers are determined to keep Linux's ARP behavior as it
is.
Posted Aug 21, 2003 5:50 UTC (Thu) by dlang (guest, #313) [Link]
In some postings on Aug 20 David Miller said he would be willing to accept a change that supported a flag to change this behaviour, as long as it produced code similar to what he posted to actually control the change.discussion on this topic has greatly slowed today, but it's sounding more like good programming/design give and take and less like people beating their heads against a brick wall
Posted Aug 21, 2003 11:41 UTC (Thu) by RobDavies (guest, #9930) [Link]
The current behaviour can be exploited so that multi-homed hosts can have
Posted Aug 21, 2003 20:08 UTC (Thu) by iabervon (subscriber, #722) [Link]
I am somewhat mystified by people's insistance that a Linux associates IP addresses with machines, not interfaces. When I do /sbin/ifconfig, I get:eth0 ... inet addr:192.168.0.1
eth1 ... inet addr:18.85.44.195
lo ... inet addr:127.0.0.1
It can clearly report to me what addresses each interface has. So the Linux position must really be that an interface should claim an IP address if it may receive packets destined for the specified IP address. I would assume that iptables behavior ought to be included in this; eth1 should only respond to someone looking for 192.168.0.1 if it will pay attention to packets destined for 192.168.0.1 which arrive on eth1. That is, if eth1 "has" the address 192.168.0.1, because it <i>can but will not</i> receive packets for the host with that address, it should not advertize it.
If I'm reading the thread correctly, there are actually two discussions about ARP in the same thread, only one of which is reported in this article. The other is involved with Linux sending requests out of an interface using a source address not assigned to that interface (but only if that address is assigned to the host and isn't localhost), and caching failures.
That is, if you ping {remote-on-network1} -I {local-on-network2}, you'll ARP on network1 with a source address on network2. The remote machine may not believe that it can reach any machine with address {local-on-network2} without routing, despite the fact that the machine requesting managed to put the ARP packet on this network, and may therefore not respond. Linux will then cache this failure, and not try again with the other source address. This means that a non-root user on a Linux box on a network with a Cisco gateway can break networking if they manage to run ping when the gateway's MAC address is not cached, if the machine has a second network card with an address which doesn't fit the router's internal netmask.
Assuming that Linux is right to treat IP addresses as belonging to the host, shouldn't an ARP failure be cached only if all of the source addresses the host has have failed to elicit a response from the remote machine? If the reason for sending responses for all addresses is to maximize the chance that a packet destined for the host get there, shouldn't it send requests with all of its addresses as the source, in order to maximize the chance that the reply will get back? I can certainly imagine a situation where a switch doesn't forward the response to the sender, because it doesn't know the sender's address is on that segment, since it has a different address for the machine on that segment.
Out of curiousity, will a Linux machine accept an ARP response on a different interface than the one that sent the request? That is, if the network causes the replay addressed to the other interface to actually arrive at the other interface, will the machine realize that it actually asked for this?
Posted Aug 22, 2003 17:44 UTC (Fri) by giraffedata (subscriber, #1954) [Link]
>I am somewhat mystified by people's insistance that a Linux associates IP >addresses with machines, not interfaces. When I do /sbin/ifconfig, I get:That was certainly my view before today, but analyzing the wording of some of the documentation, I can see that the philosophy these people are espousing, and have built into Linux, is this:
An address by definition identifies something. What an IP address identifies is a host, not an interface on a host. Unlike most addresses, IP addresses aren't unique for a host -- many IP addresses can identify the same host.
Nonetheless, the statements above that an interface "has" an IP address and Linux "associates an IP address with" an interface are true. The IP address associated with an interface is routing information -- it and the netmask determine which nodes Linux will try to reach through that interface. There are other controls that refine this routing more.
I'm pretty sure the reason this is set up in such a nonobvious way is that the original model really was IP address identifies interface. Otherwise, ifconfig wouldn't set any IP address and you would have to use a separate routing command to declare which IP addresses are to be reached through which interfaces.
Posted Aug 21, 2003 22:03 UTC (Thu) by gswoods (subscriber, #37) [Link]
I only understand part of how all this works, but one thing we have seen here is, if you have a dual-homed Solaris machine and a dual-homed Linux machine on the same two different networks, after a while, the Solaris machine will not be able to communicate with the Linux server any more.
Posted Aug 22, 2003 0:33 UTC (Fri) by tmattox (subscriber, #4169) [Link]
Yes it is related. The issue seems to be that Linux 2.4.x (and probabbly previous versions) selects the Source IP address used in its own ARP Request packets in a "different" way, but possibly still RFC compliant way. The result is that an ARP Request from a Linux box can have a Source IP address field that some devices and OS's consider to be invalid, and thus don't respond to the ARP Request. Again, it is unclear to me if Linux is really in the wrong on this, but it looks like at least two people are working on a patch to add a sysctl that would allow you to force the Source IP address in a Linux ARP Request packet to be set to an IP address of the outgoing NIC.
Posted Aug 22, 2003 0:16 UTC (Fri) by tmattox (subscriber, #4169) [Link]
I researched this Linux ARP problem/feature over 3 years ago when building our KLAT2 cluster supercomputer. It has 4 NICs (each with unique IP addresses) in each node, and ARP was just messing up all over the place. Luckily, the sysctl arp_filter, though tersely documented, seemed to resolve most of the problems at the time. Ultimately, I just preloaded the ARP caches with the proper entries and left it at that.
Copyright © 2003, Eklektix, Inc.
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds