Harping on ARP
ARP, the Address Resolution Protocol, is the means by which IP addresses are translated to physical layer MAC (usually ethernet) addresses. ARP makes local area networks work by enabling systems to find each other. When one system has a packet to transmit to another on the local network, it broadcasts an ARP request packet seeking a MAC address for a given IP address. Some machine (usually the intended recipient) hopefully responds with the corresponding MAC address, and the packet gets sent.
If a Linux system (with a default configuration) receives an ARP request on one of its interfaces, and that request is looking for an IP address assigned to any of the systems interfaces, the system will respond to the ARP request through the interface that received it. This response happens even if the interface involved is not the one to which the requested address has been assigned. Mr. Bloemsaat's problems came about because his system has two interfaces plugged into the same network. Both interfaces receive - and respond to - ARP requests sent on that network. Depending on the order in which the responses are sent, traffic could be directed to the wrong interface.
Mr. Bloemsaat included a patch which restricts ARP responses to the interface actually implementing the requested address. But, over almost a month of discussion, the networking hackers have made it clear that they do not intend to change the way Linux behaves. Their reasoning follows, more or less, these lines:
- Blocking ARP responses in this way is putting filtering decisions
at the wrong layer of the networking code. This sort of action
belongs at the netfilter level, rather than down at the device level.
- Linux's approach to ARP responses is fully compliant with all
applicable RFCs.
- In some situations, responding out of all interfaces is the only way
to successfully get communication established.
- For situations where the default ARP behavior causes problems, the arp_filter sysctl knob can be used to change things. This knob is described in networking/ip-sysctl.txt in the kernel documentation directory. For those who do not want to do this sort of tweaking directly, the ebtables package presents an easier interface.
A lot of the confusion, it seems, comes down to a subtle difference in how systems handle IP addresses. Many (perhaps most) networking implementations treat addresses as "belonging" to the interfaces they are assigned to. With that view of the world, no network interface has any business responding to an ARP request for an address which is assigned elsewhere. Linux, instead, sees IP addresses as a property of the system as a whole. So it makes sense for an interface to respond to a request for one of the system's addresses, even if that address is normally associated with a different interface.
The networking RFCs make it clear that either view of IP addresses is
legitimate. Armed with that, and their sense of how things should work,
the networking hackers are determined to keep Linux's ARP behavior as it
is.
(Log in to post comments)
there may be some give on this yet
Posted Aug 21, 2003 5:50 UTC (Thu) by dlang (guest, #313) [Link]
In some postings on Aug 20 David Miller said he would be willing to accept a change that supported a flag to change this behaviour, as long as it produced code similar to what he posted to actually control the change.discussion on this topic has greatly slowed today, but it's sounding more like good programming/design give and take and less like people beating their heads against a brick wall
Harping on ARP
Posted Aug 21, 2003 11:41 UTC (Thu) by RobDavies (guest, #9930) [Link]
The current behaviour can be exploited so that multi-homed hosts can havethe far side addressed directly using the near interface, without going
through a router and through the far-side network. Clients set up without
netmasks, and proxy arp enabled on router.
Until advent of switching hubs, 2 interfaces on one network could not make
any sense, it would fail under SunOS4 & Solaris2 for example.
Harping on ARP
Posted Aug 21, 2003 20:08 UTC (Thu) by iabervon (subscriber, #722) [Link]
I am somewhat mystified by people's insistance that a Linux associates IP addresses with machines, not interfaces. When I do /sbin/ifconfig, I get:eth0 ... inet addr:192.168.0.1
eth1 ... inet addr:18.85.44.195
lo ... inet addr:127.0.0.1
It can clearly report to me what addresses each interface has. So the Linux position must really be that an interface should claim an IP address if it may receive packets destined for the specified IP address. I would assume that iptables behavior ought to be included in this; eth1 should only respond to someone looking for 192.168.0.1 if it will pay attention to packets destined for 192.168.0.1 which arrive on eth1. That is, if eth1 "has" the address 192.168.0.1, because it <i>can but will not</i> receive packets for the host with that address, it should not advertize it.
If I'm reading the thread correctly, there are actually two discussions about ARP in the same thread, only one of which is reported in this article. The other is involved with Linux sending requests out of an interface using a source address not assigned to that interface (but only if that address is assigned to the host and isn't localhost), and caching failures.
That is, if you ping {remote-on-network1} -I {local-on-network2}, you'll ARP on network1 with a source address on network2. The remote machine may not believe that it can reach any machine with address {local-on-network2} without routing, despite the fact that the machine requesting managed to put the ARP packet on this network, and may therefore not respond. Linux will then cache this failure, and not try again with the other source address. This means that a non-root user on a Linux box on a network with a Cisco gateway can break networking if they manage to run ping when the gateway's MAC address is not cached, if the machine has a second network card with an address which doesn't fit the router's internal netmask.
Assuming that Linux is right to treat IP addresses as belonging to the host, shouldn't an ARP failure be cached only if all of the source addresses the host has have failed to elicit a response from the remote machine? If the reason for sending responses for all addresses is to maximize the chance that a packet destined for the host get there, shouldn't it send requests with all of its addresses as the source, in order to maximize the chance that the reply will get back? I can certainly imagine a situation where a switch doesn't forward the response to the sender, because it doesn't know the sender's address is on that segment, since it has a different address for the machine on that segment.
Out of curiousity, will a Linux machine accept an ARP response on a different interface than the one that sent the request? That is, if the network causes the replay addressed to the other interface to actually arrive at the other interface, will the machine realize that it actually asked for this?
machines, not interfaces
Posted Aug 22, 2003 17:44 UTC (Fri) by giraffedata (guest, #1954) [Link]
>I am somewhat mystified by people's insistance that a Linux associates IP >addresses with machines, not interfaces. When I do /sbin/ifconfig, I get:>
>eth0 ... inet addr:192.168.0.1
>eth1 ... inet addr:18.85.44.195
>lo ... inet addr:127.0.0.1
>
>It can clearly report to me what addresses each interface has.
That was certainly my view before today, but analyzing the wording of some of the documentation, I can see that the philosophy these people are espousing, and have built into Linux, is this:
An address by definition identifies something. What an IP address identifies is a host, not an interface on a host. Unlike most addresses, IP addresses aren't unique for a host -- many IP addresses can identify the same host.
Nonetheless, the statements above that an interface "has" an IP address and Linux "associates an IP address with" an interface are true. The IP address associated with an interface is routing information -- it and the netmask determine which nodes Linux will try to reach through that interface. There are other controls that refine this routing more.
I'm pretty sure the reason this is set up in such a nonobvious way is that the original model really was IP address identifies interface. Otherwise, ifconfig wouldn't set any IP address and you would have to use a separate routing command to declare which IP addresses are to be reached through which interfaces.
Harping on ARP
Posted Aug 21, 2003 22:03 UTC (Thu) by gswoods (subscriber, #37) [Link]
I only understand part of how all this works, but one thing we have seen here is, if you have a dual-homed Solaris machine and a dual-homed Linux machine on the same two different networks, after a while, the Solaris machine will not be able to communicate with the Linux server any more.We have traced this down to the different way that Linux does ARP, and the admin of the Solaris machine has flooded me with RFC references showing that Linux does this "wrong". I'm not a great interpreter of RFCs either so I don't know if his evidence is valid. But in the end, the only way to fix the problem was to set the MAC address of all the interfaces on the Linux machine to be the same, so that if the Linux machine sends an ARP response that is associated with the "wrong" interface, it still has the right MAC address. Does this sound like it's related to the same issue?
Harping on ARP
Posted Aug 22, 2003 0:33 UTC (Fri) by tmattox (subscriber, #4169) [Link]
Yes it is related. The issue seems to be that Linux 2.4.x (and probabbly previous versions) selects the Source IP address used in its own ARP Request packets in a "different" way, but possibly still RFC compliant way. The result is that an ARP Request from a Linux box can have a Source IP address field that some devices and OS's consider to be invalid, and thus don't respond to the ARP Request. Again, it is unclear to me if Linux is really in the wrong on this, but it looks like at least two people are working on a patch to add a sysctl that would allow you to force the Source IP address in a Linux ARP Request packet to be set to an IP address of the outgoing NIC.For now the best workaround seem to be to set a prefered source IP address for each route, using the "ip route ... src ..." command, and to echo "1" into the arp_filter sysctl in /proc for each NIC. I've not tested this myself, I just finished reading the LONG discussion on the linux-netdev mailing list.
Harping on ARP
Posted Aug 22, 2003 0:16 UTC (Fri) by tmattox (subscriber, #4169) [Link]
I researched this Linux ARP problem/feature over 3 years ago when building our KLAT2 cluster supercomputer. It has 4 NICs (each with unique IP addresses) in each node, and ARP was just messing up all over the place. Luckily, the sysctl arp_filter, though tersely documented, seemed to resolve most of the problems at the time. Ultimately, I just preloaded the ARP caches with the proper entries and left it at that.I do hope the current discussion on the mailing lists will find a solution amenable to everyone. I advocate a sysctl switch for "Set the source IP address of an ARP request to that of the sending NIC", but of course I'd have more influence if I posted patches to accomplish that... too bad the neighbor code (ARP) in Linux is larger than I care to rummage through today.
-- Tim