Multihomed Masquerading, routing and iptables

From: Gordan Bobic (gordan_at_SPAMCONTROLbobich.net)
Date: 12/31/03 

Date: Wed, 31 Dec 2003 11:32:16 +0000

Hi.

I have a networking problem that is driving me nuts at the moment. I
have a multi homed network: Cable + DSL.

The problem I have is that although I am 99% sure that I have the
routing table rules set up correctly, for some reason
masqueraded/NATed traffic doesn't go out of the correct interface.
i.e. I am getting traffic leaving eth2 with the source IP header set
to eth3 and vice versa.

There are 3 network interfaces:

eth0 (internal)
eth2 (DSL)
eth3 (Cable)

(eth1 is unused at present)

Here is my iptables setup (/etc/sysconfig/iptables):
################################
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003
*nat
:PREROUTING ACCEPT [0:0]
# Port forwarding to an internal machine
-A PREROUTING -i eth2 -d 217.79.103.2 -p tcp -m tcp --dport 18001 -j
DNAT --to-destination 192.168.0.10:18001
-A PREROUTING -i eth3 -d 62.252.21.17 -p tcp -m tcp --dport 18001 -j
DNAT --to-destination 192.168.0.10:18001
# SSH Port Forwarding
-A PREROUTING -i eth2 -d 217.79.103.3 -p tcp -m tcp --dport 22 -j DNAT
--to-destination 192.168.0.10:22
:POSTROUTING ACCEPT [0:0]
# IP Masquerading Traffic From eth2 and eth3
-A POSTROUTING -o eth2 -j MASQUERADE
-A POSTROUTING -o eth3 -j MASQUERADE
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sat Dec 27 10:47:54 2003
# Generated by iptables-save v1.2.7a on Sat Dec 27 10:47:54 2003
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
-A FORWARD -i eth0 -o eth2 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state
--state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth0 -o eth3 -s 192.168.0.0/16 -d 0.0.0.0/0 -m state
--state NEW,ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth2 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state
--state ESTABLISHED,RELATED -j ACCEPT
-A FORWARD -i eth3 -o eth0 -s 0.0.0.0/0 -d 192.168.0.0/16 -m state
--state ESTABLISHED,RELATED -j ACCEPT
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Sat Dec 27 10:47:54 2003
###################################

Additionally, here is the script I use to set up the multi homed
routing:

####################################
# Add ip rules for routing
ip rule add from 217.79.103.0/29 table Griffin
ip rule add from 62.252.21.17 table NTL

# Add routing rules for specific interfaces to insure connectivity
ip route add to default via 217.79.103.1 dev eth2 table Griffin
ip route add to default via 62.252.21.254 dev eth3 table NTL

ip route add to 217.79.103.0/29 dev eth2 table Griffin
ip route add to 62.252.21.0/24 dev eth3 table NTL

# Default route is multi homed
ip route add to default \
        nexthop via 217.79.103.1 dev eth2 weight 1 \
        nexthop via 62.252.21.254 dev eth3 weight 1

# Commit routing changes
ip route flush cache
#############################

However, looking at tcpdump output from eth2:
11:19:27.153771 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 >
217.81.134.183.57626: R 0:0(0) ack 2502579442 win 0 (DF)
11:19:30.212427 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 >
217.81.134.183.57626: R 0:0(0) ack 1 win 0 (DF)
11:20:23.928900 cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com.18001 >
217.81.134.183.58367: R 0:0(0) ack 2551899092 win 0 (DF)

This is wrong because cpc4-cbly1-3-0-cust17.glfd.cable.ntl.com is
62.252.21.17, which is the IP address of eth3.

Similarly, tcpdump from eth3 says things like:
11:18:32.787404 217.79.103.2.adsl.griffin.net.uk.18001 >
p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 4066315873 win 0 (DF)
11:18:35.683228 217.79.103.2.adsl.griffin.net.uk.18001 >
p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF)
11:18:41.744790 217.79.103.2.adsl.griffin.net.uk.18001 >
p50811062.dip.t-dialin.net.33062: R 0:0(0) ack 1 win 0 (DF)

This is again wrong, because 217.79.103.2.adsl.griffin.net.uk is the
IP address of eth2.

I am pretty sure the IP rules I set up should work. They assign all
packets with source IP of a particular interface to a routing table
that is routed out via the correct gateway. However, some packets
(from what I have been able to tell, only the masqueraded packets,
but the test was not exhaustive) get sent out of the wrong interface.

Can anybody see a problem with this setup?

TIA.

Gordan

Relevant Pages

  • Linux router w/3 nics - need help with routing
    ... I boot up the machine the routing table shows entries for eth2. ... While I can easily install the route I ... # This script will be executed *after* all the other init scripts. ...
    (comp.os.linux.networking)
  • ping send to problem, what am i missing
    ... I'm trying to get some routing working from my freebsd 6.1 machine that i thought should be extremely easy, ... When trying to ping the destination address i get: "ping: sendto: Invalid argument" ... I manually added the 192.168.5 route using the following command (please correct me if i did it incorrectly ...
    (freebsd-questions)
  • Re: IP forwarding
    ... The routing table seems to have allocated 'lo' as the device for ... instead of eth2. ... as we see from the output of 'route' command. ...
    (comp.os.linux.networking)
  • Re: Valid Routing Query
    ... The route you describe for a journey from Woking to Slough route ... Exeter, Bristol or Swindon would be much higher than the cost of your ... however based on the online routing ... other maps to allow a complete tracing. ...
    (uk.railway)
  • RE: Fax routing
    ... I understand you could not route your ... Open Server Management ... E-mail incoming routing method" ...
    (microsoft.public.windows.server.sbs)