Multiple Network Interface Gotcha in Linux

Ok here’s a recipe to try:

  • Get a development board or PC that has two or more network interfaces,
  • Assign each of them with a unique IP address,
  • Connect the interfaces to a common network,
  • Finally, ping one of the IP addresses from another machine.

Now, depending on which IP address you chose to ping, you may find that your pings will suddenly fail to respond and timeout when you disconnect the other interface (i.e. the interface you are not pinging). Bizarre, isn’t it?

However, as my colleague and I recently discovered whilst debugging a new Ethernet driver, this gotcha is actually correct behaviour for Linux – and in fact it is correct behaviour as defined by the relevant RFC’s. I thought I’d use this post to discover what is going on and why this is OK.

In order to reproduce this behaviour I set up a virtual machine and assigned it with a number of NAT’d Ethernet devices (4 in fact). I also set up Wireshark (what used to be Ethereal) so that I could monitor any traffic. Here is cut down version of the output from ifconfig.

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0c:29:12:0b:bd
          inet addr:192.168.27.132  Bcast:192.168.27.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr 00:0c:29:12:0b:c7
          inet addr:192.168.27.133  Bcast:192.168.27.255  Mask:255.255.255.0

eth2      Link encap:Ethernet  HWaddr 00:0c:29:12:0b:d1
          inet addr:192.168.27.135  Bcast:192.168.27.255  Mask:255.255.255.0

eth3      Link encap:Ethernet  HWaddr 00:0c:29:12:0b:db
          inet addr:192.168.27.134  Bcast:192.168.27.255  Mask:255.255.255.

I cleared the ARP cache on my Windows machine by using “arp -d” and then pinged 192.168.27.132. The packet exchange captured by Wireshark proved to be quite interesting. Let’s take a look.

No.  Source                Destination        Protocol   Info
1  Vmware_c0:00:08       Broadcast             ARP      Who has 192.168.27.132?  Tell 192.168.27.1
2  Vmware_12:0b:db       Vmware_c0:00:08       ARP      192.168.27.132 is at 00:0c:29:12:0b:db
3  192.168.27.1          192.168.27.132        ICMP     Echo (ping) request
4  Vmware_12:0b:d1       Vmware_c0:00:08       ARP      192.168.27.132 is at 00:0c:29:12:0b:d1
5  Vmware_12:0b:c7       Vmware_c0:00:08       ARP      192.168.27.132 is at 00:0c:29:12:0b:c7
6  Vmware_12:0b:bd       Vmware_c0:00:08       ARP      192.168.27.132 is at 00:0c:29:12:0b:bd
7  192.168.27.132        192.168.27.1          ICMP     Echo (ping) reply
8  192.168.27.1          192.168.27.132        ICMP     Echo (ping) request
9  192.168.27.132        192.168.27.1          ICMP     Echo (ping) reply
10 192.168.27.1          192.168.27.132        ICMP     Echo (ping) request
11 192.168.27.132        192.168.27.1          ICMP     Echo (ping) reply

After I invoke the ping command, my machine issues an ARP broadcast, asking for the MAC address currently associated with 192.168.27.132. However all of the network interfaces of my virtual machine respond – resulting in 4 ARP replies. When this happens Windows (and other OS’s) will ignore all but the first response, with the assumption that the first reply must have come from the quickest route.

In this example the quickest ARP reply came from the MAC address associated with eth3. Therefore whenever we communicate with 192.168.27.132, as we have done via Ping, the traffic will be sent to eth3. As a result, if we now down interface eth3 with “ifconfig eth3 down”, our pings will fail. This behaviour can be confusing as why should eth3 going down affect traffic that is directed to 192.168.27.132 which we believed to be associated with eth1?

Despite the impression ifconfig gives, Linux associates IP addresses with the host as opposed to individual interfaces of the host. With that in mind, the behaviour we’ve seen doesn’t seem so bizarre. When a network interface receives an ARP request for an IP address which it owns, then in effect a valid network route been made between the requestor and the requested. This route could potentially be the only route and as it is likely that the two will communicate with each other, it makes sense to reply to the ARP request. And this is what happens – the network interface that received the ARP request will now act as a proxy for the requested IP address.

This behaviour is actually quite convenient. In our example, even though our pings began to fail once we disconnected a route to the host – as soon as the Windows ARP cache times out (after 10 minutes) another ARP request will be broadcast. Like before, any interface that can provide a route to the host will respond, and so connectivity will be restored. If Linux wasn’t designed in this way and each interface truly owned an IP address, then if that link went down connectivity would never be restored to that address – even though there are other physical connections to the machine that has that IP address!

The other point of interest here, at least with this contrived networking configuration, is that reliability is favoured over performance. The reason is that where multiple interfaces exist on a machine, it’s quite likely that a priority ordering will exist between them. And so if, as in this case, eth3 replies the quickest then it is likely that it will always be the quickest. As a result, it is also likely to respond to all the ARP requests first and so all traffic for the 4 IP addresses will arrive on a single interface. We can demonstrate this. After pinging all of the IP addresses assigned to my virtual machine we can examine the ARP cache of Windows.

arp -a

Interface: 192.168.27.1 --- 0x2
  Internet Address      Physical Address      Type
  192.168.27.132        00-0c-29-12-0b-bd     dynamic
  192.168.27.133        00-0c-29-12-0b-bd     dynamic
  192.168.27.134        00-0c-29-12-0b-bd     dynamic
  192.168.27.135        00-0c-29-12-0b-bd     dynamic

As you can see all the IP addresses correspond to the same interface, i.e. eth3. Thus all the traffic will go over a single 10/100Mbit link instead of 4 links.

Fortunately, where this behaviour isn’t ideal, the proc interface provides a means to modify it. Of particular interest are the arp_filter and rp_filter sysctl knobs which can be found in the proc interface. I’ve not really managed to make complete sense of these yet and may well write another post on these in the future. Though for the behaviour described above it was necessary for me to invoke “echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter”, I found without this I would only ever get two ARP replies instead of 4 – I’m not entirely sure why this is… suggestions anyone?

And finally, for those that wish to read more, I recommend an article on LWM which provides some more background information. [© 2011 embedded-bits.co.uk]

, , , , , , , , , , , , ,

About Andrew Murray

Andrew is an experienced commercial Linux developer with a first class degree in Software Engineering and is the founder of Embedded Bits Limited. His day-to-day role fulfils his passion for learning and provides him with plenty of embedded Linux experience including kernel and embedded applications development on a wide variety of platforms. He loves to talk about boot time reduction and has performed a number of presentations on the topic at technical conferences - he has also been successful in achieving sub-second cold boot on Linux based products. Feel free to drop him an email at amurray@embedded-bits.co.uk

7 Responses to “Multiple Network Interface Gotcha in Linux”

  1. Andy November 11, 2008 at 2:27 pm # Reply

    For those that have come across this as multi-homed devices don’t work in the way expected (i.e. doesn’t utilise the avaliable bandwidth) there is plenty of software to assist. A popular example for load sharing is TEQL, the following link http://lartc.org/howto/lartc.loadshare.html provides information about how this can be used.

  2. Den deveembeply December 18, 2008 at 11:29 am # Reply

    First of all congratulation for such a great site. I learned a lot reading article here today. I will make sure i visit this site once a day so i can learn more.

  3. Timekeeper January 2, 2009 at 5:37 pm # Reply

    I really very liked this post. Can I copy it to my site?
    Thanks in advance.

    Sincerely, Timur.

  4. James August 1, 2011 at 8:41 am # Reply

    Hi,
    the issue is I believe a standard network issue your first network interface has the default route associated with it and also you could class as the general network interface, the other 3 are in the same sub net so if you look more deeply into the tcp/udp/etc traffic you will see your request go in on the ip address/port you requested, but the response will be from the main interface on that sub net.
    If they were each in their own subnet then requests on that subnet would be from the interface on that port as long as those making the request are in that sub net, if they are not it would reply again from the primary interface with the gateway setup.
    Eg. interface0 192.168.1.7 netmask 255.255.255.0 gateway 192.168.1.1
    interface1 172.168.1.7 netmask 255.255.255.0
    if ipaddress 192.168.1.20 connects to the 172 address the response will still be from the 192 conneted interface.
    I ipaddress 172.168.1.20 connects to 172 interface/address then the response will be from the 172 interface.
    Just a quick one things like SSH/firewalls etc can have a real big problem if a connection to one interface is responded to by another.
    As some one mentioned earlier, if you used something like bonding and put all ip adrresses in the same subnet across all 4 interfaces that would be a solution, also giving you redundacy and increased through put.

    Regards,

    James

  5. Richard Hoser August 1, 2011 at 5:30 pm # Reply

    Thanks, you got it right here

Trackbacks/Pingbacks

  1. lolekblog » Blog Archive » Embedded Bits » Blog Archive » Multiple Network Interface Gotcha … - November 10, 2008

    [...] post by unknown Filed in Generic Novell considers embedded Linux – ZDNet UK [...]

  2. Varias interfaces en la misma red con linux « El Hombre que Reventó de Información - August 12, 2011

    [...] http://www.embedded-bits.co.uk/2008/multiple-network-gotcha/ GA_googleAddAttr("AdOpt", "1"); GA_googleAddAttr("Origin", "other"); GA_googleAddAttr("theme_bg", [...]

Leave a Reply