syslog server, RH ES 4, large amounts of UDP loss. please help

I am having trouble pinpointing why the system is dropping so many UDP
packets, I could really use some help here as I was not expecting to
see such a high number of packet errors (so far we have 3 systems
sending remote syslog to this system which is equating to about
1MB/minute. I have hundreds of systems to eventually configure and can
build more syslog servers but this is going to be hard to justify if 1
system is having trouble with the load from 3
servers).

If there is any other information I can provide to help answer this I
will do my best.

thanks,

netstat -su
Udp:
65321 packets received
21 packets to unknown port received.
28401 packet receive errors
21829 packets sent

System: Dell Precision 650
3190.755 MHz processor
Memory: 1032492k/1048020k available
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex

Install was minimal. Then I turned everything off via chkconfig
excluding syslog (reconfigured for remote syslog accepting) and sshd.
iptables (used fwbuilder) is configured to block all in/outbound
requests by default. The holes are poked in for ssh and syslog,
outgoing for dns queries, and ntp.

my sysctl.conf has the following mods which was put together after
looking at these and a few other docs
http://www.29west.com/docs/THPM/udp-buffer-sizing.html
http://dsd.lbl.gov/TCP-tuning/linux.html

net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.ipv4.tcp_rmem = 4096 87380 33554432
net.ipv4.tcp_wmem = 4096 65536 33554432
net.ipv4.tcp_mem = 33554432 33554432 33554432


netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR
TX-DRP TX-OVR Flg
eth0 1500 0 74924 0 0 0 18204 0
0 0 BMRU
lo 16436 0 0 0 0 0 0 0
0 0 LRU


vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 1 0 885752 39428 74192 0 0 37 841 1215 366 0
1 43 56
0 1 0 885696 39428 74192 0 0 0 1156 1249 442 0
0 25 75
0 1 0 885640 39436 74184 0 0 0 1464 1295 545 1
1 8 90
0 0 0 885576 39436 74184 0 0 0 252 1046 104 0
0 84 16
0 1 0 885548 39436 74444 0 0 0 920 1331 422 0
1 43 56
0 0 0 885492 39436 74444 0 0 0 1232 1292 543 0
0 22 78
0 1 0 885464 39436 74444 0 0 0 900 1219 351 0
0 41 59
0 0 0 885436 39436 74444 0 0 0 1140 1205 434 0
1 25 74
0 0 0 885436 39436 74444 0 0 0 0 1022 25 0
0 100 0
1 1 0 885408 39444 74436 0 0 0 1012 1316 436 0
0 38 62
0 1 0 885352 39444 74436 0 0 0 1092 1273 495 1
1 32 66
0 1 0 885324 39444 74436 0 0 0 940 1211 359 0
0 40 60
0 0 0 885296 39444 74436 0 0 0 1088 1216 434 0
1 27 72
0 0 0 885296 39444 74436 0 0 0 0 1005 7 0
0 100 0
0 1 0 885268 39452 74688 0 0 0 972 1364 430 0
0 36 64
0 0 0 885212 39452 74688 0 0 0 1448 1376 624 0
1 14 85
0 1 0 885184 39452 74688 0 0 0 1028 1238 392 0
1 37 62
0 1 0 885128 39452 74688 0 0 0 1424 1301 509 0
1 6 93
0 0 0 885100 39452 74688 0 0 0 636 1116 260 0
0 51 49
0 1 0 885072 39452 74688 0 0 0 796 1208 344 0
0 44 56

sar -n EDEV 2 10
Linux 2.6.9-34.0.2.ELsmp (systemname) 08/02/2006

03:13:39 PM IFACE rxerr/s txerr/s coll/s rxdrop/s txdrop/s
txcarr/s rxfram/s rxfifo/s txfifo/s
03:13:41 PM lo 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
03:13:41 PM eth0 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
03:13:41 PM sit0 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00

[note all the other 9 entries were the same as above for the sar output
command]

Some netstat -na --inet | grep ':514' output (removed any with null
result return)
udp 30576 0 0.0.0.0:514
udp 39984 0 0.0.0.0:514
udp 7056 0 0.0.0.0:514 0.0.0.0:*
udp 63504 0 0.0.0.0:514 0.0.0.0:*

.

Relevant Pages

  • Re: Fw: Remote logging
    ... > There are also issues involving forging packets from a third host. ... > Syslog uses UDP after all, so the source information can be forged from ... which isn't available in the default syslog. ... UDP socket programming. ...
    (FreeBSD-Security)
  • Re: syslog server, RH ES 4, large amounts of UDP loss. please help
    ... 26 packets to unknown port received. ... all inbound udp except syslog that tools like netstat would not record ... the higher levels on the receiving system stack are tripping over themselves. ... Looks like syslog may have a memory ...
    (comp.os.linux.networking)
  • Re: FTP and Packet Filtering
    ... there are a number of free syslog clients available... ... However I'm capturing packets ... I switched to a third party FTP ... I hope your packet filtering has logging as well. ...
    (microsoft.public.inetserver.iis.security)
  • (long) high traffic syslog server.
    ... I have some questions about what needs tuned on a high traffic syslog box. ... I seem to be dropping quite a few syslog packets. ... This is a syslog server for a high usage Firewall btw. ... pci0: <PCI bus> on pcib0 ...
    (freebsd-stable)
  • Re: syslog server, RH ES 4, large amounts of UDP loss. please help
    ... As for storage, I ... Thus the syslog daemon was having to make two ... significant difference in udp errors when running netstat -su. ... 26 packets to unknown port received. ...
    (comp.os.linux.networking)