How reliable is netstat?

From: Michael Sims (see_at_sig.for.email.address)
Date: 03/25/04 

Date: Thu, 25 Mar 2004 10:44:00 -0600

Hi,

I've searched Google Groups to find an answer to this, but have been
unsuccessful. Please forgive me if this has been discussed before.

I have a box on my network running Red Hat Enterprise Linux 2.1 that
acts as a web server. My dynamic pages all use an error handler that
emails me diagnostic information if an error occurs. Included with
this diagnostic information is the output from "netstat -an". I was
looking at one of these netstat outputs and I saw suspicious network
activity. There were several entries with the source address of my
web server, on a high, unprivileged port, and the destination address
of an adsl customer on the "tiscali.it" network, on port 1433
(MS-SQL). These entries were in the SYN_SENT state, indicating that
my firewall was correctly blocking these connections.

Fearing my box had been hacked, I put together a small perl script
which calls netstat in an endless loop and logs any unusual
connections. (I'm not an expert sysadmin, so I realize there are most
likely much better tools for this, but that is beside the point of
this particular post). If I leave this script running for about an
hour, I'll get about 10-15 alerts where netstat reported a connection
from my web server's address on an unprivileged port (not a port that
any services are using, such as 80, 25, 22, etc.), to some unknown IP
on a similarly random unprivileged port.

None of these remote IP addresses appear in any of my server logs.
Netstat claims that most of them are caused by "httpd" processes, and
there is nothing in my Apache logs to correlate with these
connections. Some of them are caused by "sshd", according to netstat,
but there is nothing in /var/log/secure or /var/log/messages
indicating any SSH connections other than the ones I can account for.
Also, my firewall logs don't show any record of connections from these
IP addresses.

Yesterday I installed ethereal (a network sniffer) locally on the web
server, and I configured it to capture ALL network traffic for one
hour. At the same time, I ran my perl script that uses netstat and
saved all of the "odd" connections. NONE of these odd entries have
any corresponding packets in my ethereal capture logs.

Can anyone make any sense out of this? The man page for netstat says
"Occasionally strange information may appear if a socket changes as it
is viewed. This is unlikely to occur." Of course, the authors
probably didn't intend for someone to run it in a continuous loop
either. Is it possible that these odd connections are bogus? If not,
and I've been hacked, why aren't these packets being caught by my
firewall, or ethereal? Any help would be greatly appreciated.

BTW, here are my server's vitals:
Red Hat kernel 2.4.9-e.27smp
net-tools 1.60
netstat 1.42 (2001-04-15)
I'd be glad to provide any other information that might be helpful.

Michael Sims
michaels at crye-leike dot com