Load Balancing using 2 NICS (same network) with (iproute2) multipath route

From: Christopher Markov (me_at_home.now)
Date: 03/04/04 

Date: 4 Mar 2004 07:30:13 -0800

Greetings,

I´m trying to set up load balancing on a Linux (Trustix) box, which is acting as
a firewall + HTTP/FTP proxy (Squid) + NAT gateway for an internal network.

It has 3 network interfaces: two that connect to the external world (Internet)
and one that connects to the internal network. The external connection is
currently going through two Cisco routers, each with a 1Mbps link. The external
network interfaces are on the same network, together with the two Ciscos.

I´m currently balancing outgoing traffic using iproute2´s ip util and a
multipath route with equal weights. However, I lack the confidence that this is
indeed the proper way of doing such thing, specially because since this
configuration has been adopted, the internal network users have been
experiencing some odd network behavior. Some regular web sites just became hard
to access (usually returning 500 Internal Server Errors) and, in particular,
on-line banking sites started failing miserably.

We concluded that, perhaps, since the banking sites use secure connections (SSL)
and some kind of application validation, it might get confused if some packets
arrive with a source address of the first external NIC and others with a source
address of the second external NIC. We added static routes to those sites, but
we still get trouble. Still, I thought routes were cached and thus, all packets
flowing to a specific site would (at least for some time) flow out using the
same external NIC.

Does iproute2´s multipath route balance each packet as they come or does it work
based on connections? Additionaly, is this the proper way of doing such
balancing? Are there other alternatives, perhaps channel bonding or such?

I´m including relevant config information below (IPs slightly changed to protect
the innocent, netmasks preserved). I´d appreciate any help, suggestion or report
on the subject. 

--
eth0 (external 1) -> 209.X.X.19 / 255.255.255.240
eth1 (external 2) -> 209.X.X.22 / 255.255.255.240
eth2 (internal) -> 192.168.1.1 / 255.255.255.0
Cisco 1 (1Mbps) -> 209.X.X.17 / 255.255.255.240
Cisco 2 (1Mbps) -> 209.X.X.20 / 255.255.255.240
--
root@hal ~# ip route show
209.X.X.16/28 dev eth0  scope link
209.X.X.16/28 dev eth1  proto kernel  scope link  src 209.X.X.22
192.168.1.1/24 dev eth2  scope link
127.0.0.0/8 dev lo  scope link
default
        nexthop via 200.X.X.17  dev eth0 weight 1
        nexthop via 200.X.X.20  dev eth1 weight 1
root@hal ~#
--
root@hal ~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
209.X.X.192 	0.0.0.0         255.255.255.240 U     0      0        0 eth0
209.X.X.192 	0.0.0.0         255.255.255.240 U     0      0        0 eth1
192.168.1.1     0.0.0.0         255.255.255.0   U     0      0        0 eth2
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         209.X.X.19      0.0.0.0         UG    0      0        0 eth0
root@hal ~#
--
Here´s how I´m adding the multipath route:
# to add a second route, to the same network, through a different interface.
otherwise we get two routes using eth1.
ip route replace 209.X.X.16/28 dev eth0
#
ip route add 209.X.X.16/28 dev eth0 src 209.X.X.19 table firstlink
ip route add default via 209.X.X.17 table firstlink
ip route add 209.X.X.16/28 dev eth1 src 209.X.X.22 table secondlink
ip route add default via 209.X.X.20 table secondlink
ip rule add from 209.X.X.19 table firstlink
ip rule add from 209.X.X.22 table secondlink
ip route add default scope global nexthop via 200.X.X.17 dev eth0 weight 1
nexthop via 200.X.X.20 dev eth1 weight 1
--
And here are some relevant iptables rules:
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j SNAT --to-source
209.X.X.19
iptables -t nat -A POSTROUTING -o eth1 -s 192.168.1.0/24 -j SNAT --to-source
209.X.X.22
--
Regards,
Christopher Markov

Relevant Pages

  • Re: Bridging network adapters in Linux
    ... ip addr add $address/$netbits dev br0 ... Either a default route ... two network devices assuming, ... This is done using the iptables mechanism. ...
    (comp.os.linux.networking)
  • Generate traffic with only one machine - whats wrong with this routing?
    ... ip route add 10.0.1.11 table ixchariot dev eth1 src 10.0.1.1 proto ... local 10.0.1.11 dev eth2 proto kernel scope host ...
    (comp.os.linux.networking)
  • Re: iptables zwei provider auf zwei interfaces
    ... ip route add $GATE1NET dev $IF_GATE1 src $GATE1 table telekom ... 11.11.11.192/29 dev eth1 proto kernel scope link src 11.11.11.194 ...
    (de.comp.security.firewall)
  • Re: CTCSS tone generator
    ... If, at the dev. ... No one is suggesting this a preferred route but the original poster wanted ... horrible results on some repeaters. ... In fact even just a slightly higher level of CTCSS tone than normal can ...
    (uk.radio.amateur)
  • Re: all traffic goes via eth0.. why?
    ... networks (data network, private network, corporate network, etc), We ... You can use iproute2 with a number of rules to route the traffic from ... ip route add default dev eth0 via 192.168.70.200 table 1 ...
    (alt.os.linux.suse)