Re: 2 NICs in same system routing question

From: Floyd L. Davidson (floyd_at_barrow.com)
Date: 05/06/05 

Date: Thu, 05 May 2005 16:36:39 -0800

Michael W Cocke <cocke@catherders.com> wrote:
>I'm certain that I'm missing something, because this seems too
>straightforward... 8-)>
>
>I'm using SuSE 9.3. I have two file servers, each with 2 NICs (one
>Netgear GA111 and one Intel 1000/TM. I want the intel NIC in each
>system to be connected to the other system via a crossover cable to
>the other Intel NIC. The Netgears go to a switch and should carry all
>the traffic except the connection to the other server.
>
>Eth0 are the intels and eth1 are the netgears, server A is
>192.168.1.100, and server B is 192.168.1.101

Hence eth0 on each server is the crossover to the other server,
and eth1 is the inteface for the default route. Given what you
then say below, can I assume that the above IP addresses are
assigned to the eth0 interfaces?

You don't mention the IP addresses assigned to the eth1
interfaces, so I'm going to arbitrarily use 192.168.0.10 for the
A server and 192.168.0.20 for the B server. That puts them on a
separate subnet. Note that you could also put them on the same
subnet, as we'll see below.

>Here's where I'm missing something - I think all I need to do is:

First, you need to consider what happens when the interface is
initially configured using ifconfig. That command will also add
a route for the subnet, hence when you do

  ifconfig eth0 192.168.1.100

you also get a route that is the same as if you had run this command,

  route add -net 192.168.1.100 netmask 255.255.255.0 eth0

>On server A
>route add 192.168.1.101 eth0

  route add -host 192.168.1.101 eth0

Except you already have that route because of the ifconfig command...

>route add default eth1

First you need to configure eth1 with an IP address,

  ifconfig eth1 192.168.0.10

That also gives it a route, the same as noted above except this time
it routes the 192.168.0.n subnet.

Once there is an existing route to the gateway address, you can
add that as the default, with this command,

  route add default gw 192.168.0.20 eth1

>and on server B
>route add 192.168.1.100 eth0
>route add default eth1

The B server is of course done exactly the same, using different
IP addresses as needed.

>Try not to laugh too hard and please tell me what I've forgotten.

Ha! You were only slightly confused when you asked. I suspect
that the first time you read this you'll be truely confused, and
it may take serious study before you get past that. Hence, let
me show you a bit more! (If you are going to be discombobulated
and laughed at, you might as well go whole hog and get what you
can out of it.)

You don't have to use separate subnets. All of that can be on
one subnet, which will actually more likely be what you want.
That is because it will force *all* addresses through the
server. (The above will route all 192.168.0.n address to the
LAN for a direct connection, not gatewayed through Server B.)

You don't have to allow ifconfig to add a subnet route, as
giving it a netmask argument of 255.255.255.255 will prevent
that.

So, on Server A, you might to use 192.168.1.10 as the IP address
for eth1, and you could do this:

    ifconfig eth0 192.168.1.100 netmask 255.255.255.255
    ifconfig eth1 192.168.1.10 netmask 255.255.255.255

That configures the two network interfaces, but adds no routes for
either of them.

    route add -host 192.168.1.101 eth0
    route add -host 192.168.1.20 eth1
    route add default gw 192.168.1.20 eth1

That adds in turn, a route to 192.168.1.101 via eth0, a route to
192.168.1.20 via eth1, and defaults *everything* else to be gatewayed to
192.168.1.20 via the eth1 interface.

Of course you might actually want something like this,

    route add -host 192.168.1.101 eth0
    route add -net 192.168.1.20 netmask 255.255.255.0 eth1
    route add default gw 192.168.1.20 eth1

The difference is that now if you have another host on the LAN (accessable
through the switch, for example, at any 192.168.1.n address, it will be
access directly instead of through the server gateway. Obviously if you
narrow that netmask down to 255.255.255.255 for a host route, no host
other than Server B would be accessed directly, and if you broaden it
up with a netmask like 255.255.0.0 it would then send any 192.168.n.n
address directly instead of via the gateway. Take your pick to suit
your needs.

Look at it all in terms of the physical layout:

      Server A Server B
 =================== ===================

 192.168.1.100 eth0 <=====X=====> eth0 192.168.1.101

 192.168.1.10 eth1 <==+ +==> eth1 192.168.1.20
                         | |
                    +----+-----+----+
                    | Switch |
                    +-------+-------+
                            |
                    +-------+-------+
                    | Host C |
                    | 192.168.1.30 |
                    +---------------+

The bottom block, Host C with an IP address of 192.168.1.30 is
representative of everything else on your LAN. If you use the
narrow routing, to host addresses rather than to entire subnets,
the only way for Server A to access Host C will be via Server B,
which of course may or may not filter and forward packets,
depending on what you have it doing. If you use a subnet mask
for routing from Server A though, an IP given to Host C that is
within the routed subnet will get traffic directly, via eth1,
rather than through the gateway functions of Server B. 

-- 
Floyd L. Davidson           <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska)                         floyd@barrow.com