Re: Problems with IP forwarding

From: prg (rdgentry1_at_cablelynx.com)
Date: 02/10/05 

Date: 9 Feb 2005 18:32:56 -0800

trevorelbourne@gmail.com wrote:
> Hi,
>
> Fisrtly, thank you very much for taking the time to look at this.
Much
> appreciated!
>
> Here's our setup (note: prepend 203.1.78 to all IPs, so [.3] is
really
> [203.1.78.3])
>
>
> Router[.195]<------>[.225]bunsen[.66]<------->[.65]beaker[.3]<----->
> LAN
>
> I can ping, say, www.google.com from beaker, so I know there's no
> firewall problems on bunsen, or connectivity problems from beaker to
> outside world. I have configured IP tables on beaker to accept
> everything, and log it. IP tables on bunsen (firewall) is
restrictive,
> but allows pings out.
>
> If I do a ping from a LAN machine to, again say www.google.com, I can
> see entries in the IP tables log saying that beaker accepted an ICMP
> from the LAN machine, destined for 64.233.187.99 (google). So I know
> packets from the LAN are getting to beaker, and accepted, but they
then
> seemingly disappear. I can also successfully ping beaker from any LAN
> machine.

Pinging from beaker is not conclusive. The packets _from_ beaker
travel a different netfilter path than packets received on an
interface. Can your lan boxes ping bunsen? Can they ping beaker's
203.1.78.65 address (though that too is inconclusive of anything)?
What does a traceroute to bunsen's GW provide? Where does it stop?

> I am in the process of recompiling my 2.6.10 kernel to turn on kprobe
> support, and use the netpktlog module to trace packets through the
> kernel to see what happens.

Unless the kernel you're using now is a home grown compile, I seriously
doubt that a new effort will provide a fix. RH/FC have included all
the routing capabilities in their "stock distro" kernels for years now.
 And googling turned up nothing suspicious about routing code.

> Here's some config info:
>
> BUNSEN (Firewall)
> -----------------
>
> [root@bunsen root]# route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref
Use
> Iface
> 203.1.78.3 203.1.78.65 255.255.255.255 UGH 0 0
0
> eth0

This is the only config that looks off. Is this an artifact of NATing
on beaker? Otherwise, you should have a network route here rather than
a host route. Ie., 203.1.78.0 (255.255.255.192) rather than 203.1.78.3
(255.255.255.255). That way any packet destined for 203.1.78.0/26 will
have a route via beaker.
# route add -net 203.1.78.0 netmask 255.255.255.192 gw 203.1.78.65
added after 203.1.78.64 I usually add all my GW routes at the end
after all the networks have been entered.

> 203.1.78.224 0.0.0.0 255.255.255.224 U 0 0
0
> eth1
> 203.1.78.64 0.0.0.0 255.255.255.192 U 0 0
0
> eth0
> 127.0.0.0 0.0.0.0 255.0.0.0 U 0 0
0
> lo
> 0.0.0.0 203.1.78.226 0.0.0.0 UG 0 0
0
> eth1

It is tedious but you really need to ping and note responses (or none)
in the sequence I posted. The idea is to find which box is the
culprit, then what about that box is set up wrong. You confirm the
culprit by pinging from the other end of the path, ie., from bunsen to
a lan box.

With your network setup, it may not be possible/reasonable to turn off
the firewall/filtering rules, but you will save much effort if you can
confirm that routing works correctly when they are disengaged. Then
you know for sure where the problem is. Otherwise, only very close
scrutiny and study will likely reveal the key.

If the machines are new to you, you really ought to document how they
are set up, including route tables, IPs, and firewall/filter rules of
both bunsen and beaker. Also check that no one has engaged policy
routing:
$ ip route show ??? < you will look for additional tables
http://linux-ip.net/html/routing-tables.html

$ ip rule show
looking for policy rules

These last two are not likely, but you never know. I would say they
are more likely in a network set up such as yours than most others.
And if you don't know they are there, they will eat your lunch ;)

After double checking all the route tables, interface configs, and
firewall/filter rules, if no solution presents itself, it is time to
sniff the wire. The only other alternatives are hunches and
duplicating the path from bunsen<->beaker<->lan on 3 test boxes. Ugly
...

Logs just can't capture enough data of the right sort to be definitive
as seeing the decoded wire packets/frames. They _may_ reveal an
internal problem that a sniffer can't see, but you usually need a
sniffer to give you a clue where/what to look for in the logs.

Here are some links you may find handy to have -- turn around on google
and the chances of failed/erased delivery from my end are too great
(earlier post was eaten alive by google).

RH9 Reference Guide for all the networking files and their locations:
http://www.redhat.com/docs/manuals/linux/

Routing/filtering on Linux:
http://open-source.arkoon.net/kernel/kernel_net.png
http://linux-ip.net/html/routing-selection.html
http://www.netfilter.org/documentation/HOWTO//packet-filtering-HOWTO-6.html
http://linux-ip.net/html/routing-tables.html
http://linux-ip.net/html/routing-rpdb.html
http://linux-ip.net/html/tools-ip-route.html

BTW, how _are_ you using your network if no packets can get out of the
lan?

hth,
prg
email above disabled