Problem with IP Masquerade and MTUs

From: Shawn Willden (scubashawn_at_willden.org)
Date: 08/07/03


Date: Thu, 07 Aug 2003 13:08:29 -0600

A month or so ago I started having a problem with TCP connections that just
seemed to hang, when connecting to hosts on the other side of my VPN
connection. I think I've tracked the problem down, but I don't know what
to do about it. First let me describe the network:

I have a Linux box (Debian woody, running a 2.4.18 kernel) connected to my
DSL, acting as a masquerading router/firewall (iptables 1.2.6) for my home
network (which is a 10.x.x.x net). This box has two real network
interfaces (eth0, which connects to my LAN, and eth1, which connects to my
DSL) and a virtual interface (ipsec0) that is connected to my company's
network via IPSEC (Free S/WAN 1.96, patched to support AT&T's heartbeat
packets). There are a variety of systems (mostly Linux 2.4.x, one Win2K)
on the network. The router is configured to SNAT/MASQUERADE the local
hosts to both eth1 and ipsec0.

What happens when I make connections across the VPN from the router is that
everything works beautifully. When I connect from any of the other hosts,
however, large packets don't go through. For example, if I request a small
web page across the VPN from a host on my network, it works fine. When I
request a large page, the SYN and ACK packets get through, and the request
goes out (and the ACK comes back), but the packet containing the response
never shows up.

By looking at the packets on both the external interface (eth1) and the
IPSEC interface, I can verify (by looking at timestamps) that the encrypted
packet containing the response data never arrives at the external
interface, so it's not being dropped by Free S/WAN on my side.

Watching from the sending side, it appears there are some clear indications
that it's a path MTU discovery issue. When I make a request from the
router to a server on the other side of the VPN, and run tcpdump on the
server side, I see that the server sends no more than 1443 bytes in each
packet, which is precisely the MTU for my IPSEC interface. OTOH, when I
send the request from one of the NATed machines the server sends 1500 bytes
per packet.

Even clearer, if I manually set the MTU on the NATed host to 1443, responses
come back just fine.

Any ideas why MTU discovery isn't working? What's really baffling is that
this setup has worked for a year and then stopped working without any
change AFAIK. The change may have happened on the other side, I suppose.

More to the point, does anyone have any suggestions as to what I can do,
other than manually lowering my MTUs to match the IPSEC MTU?

Thanks,

-- 
Shawn
Procrastination:  Hard work sometimes pays off later,
but laziness always pays off now.


Relevant Pages

  • [REVS] Sinit P2P Trojan Analysis
    ... A common tactic among Trojan writers is the multi-stage install. ... intermediary layer of 20 hosts that would point it to the real download ... Sinit, there is no central server that can be shut down. ... The packets Sinit uses in its discovery protocol were detected quickly by ...
    (Securiteam)
  • TCP Connections, Bluesocket, and Mac OS X
    ... concerning OSX systems and Bluesocket wireless technology. ... due to too many open network connections. ... You can see how many sessions your ... 18908 data packets ...
    (alt.internet.wireless)
  • Re: Improving FreeBSD NFS performance (esp. directory updates)
    ... >> I don't think the network is at fault, nor is the server really going ... 155645171 data packets ... discarded for bad header offset fields ... 790 connections established ...
    (freebsd-questions)
  • Re: Only some websites will open - Ubuntu
    ... incoming packets discarded ... 236 active connections openings ... 184 delayed acks sent ... TCPAbortOnSyn: 0 ...
    (comp.os.linux.misc)
  • Re: FreeBSD 7.1 tcp problem (syncache)?
    ... Completed 200 requests ... 31728 data packets ... 9740 connections closed ... segment rexmit in SACK recovery episodes ...
    (freebsd-net)