Re: bad networking related lag in v2.6.22-rc2

On Monday 21 May 2007 13:42:01 Ingo Molnar wrote:
* Ingo Molnar <mingo@xxxxxxx> wrote:
ouch! a nearly 1 second delay got observed by the scheduler - something
is really killing your system!

ah, you got the latency tracer from Thomas, as part of the -hrt patchset
- that makes it quite a bit easier to debug. [...]

and ... you already did a trace for Thomas, for the softirq problem:

http://cybertek.info/taitai/trace.txt.bz2

this trace shows really bad networking related kernel activities!

gkrellm-5977 does this at timestamp 0:

gkrellm-5977 0..s. 0us : cond_resched_softirq (established_get_next)

2 milliseconds later it's still in established_get_next() (!):

gkrellm-5977 0..s. 2001us : cond_resched_softirq (established_get_next)

and the whole thing takes ... 455 msecs:

gkrellm-5977 0..s. 455443us+: cond_resched_softirq (established_get_next)

i think this suggests that you have tons of open sockets. What does
"netstat -ts" say on your box?
On 2.6.21.1 doing normal work while seeding few torrents produces this
with "netstat -ts". I will send you same information for 2.6.22-rc2 after a
reboot.

Regards
Ananitya


Ingo



--
Out of many thousands, one may endeavor for perfection, and of
those who have achieved perfection, hardly one knows Me in truth.
-- Gita Sutra Of Mysticism
Tcp:
1233 active connections openings
845 passive connection openings
9 failed connection attempts
164 connection resets received
5 connections established
44995 segments received
43171 segments send out
183 segments retransmited
0 bad segments received.
192 resets sent
UdpLite:
TcpExt:
696 TCP sockets finished time wait in fast timer
3273 delayed acks sent
12 delayed acks further delayed because of locked socket
Quick ack mode was activated 38 times
4867 packets directly queued to recvmsg prequeue.
31660 packets directly received from backlog
6759887 packets directly received from prequeue
12038 packets header predicted
3228 packets header predicted and directly queued to user
4795 acknowledgments not containing data received
12680 predicted acknowledgments
15 times recovered from packet loss due to SACK data
4 congestion windows recovered after partial ack
12 TCP data loss events
5 timeouts after SACK recovery
8 timeouts in loss state
20 fast retransmits
25 retransmits in slow start
65 other TCP timeouts
5 sack retransmits failed
3 times receiver scheduled too late for direct processing
16 DSACKs sent for old packets
152 connections reset due to unexpected data
5 connections reset due to early user close

Relevant Pages

  • interpreting netstat -s
    ... 529092 duplicate acks ... what does the tcp output "embryonic connections ... 11772048 total packets received ... IP Multicast packets dropped due to no receiver ...
    (comp.unix.aix)
  • TCP Connections, Bluesocket, and Mac OS X
    ... concerning OSX systems and Bluesocket wireless technology. ... due to too many open network connections. ... You can see how many sessions your ... 18908 data packets ...
    (alt.internet.wireless)
  • Re: Improving FreeBSD NFS performance (esp. directory updates)
    ... >> I don't think the network is at fault, nor is the server really going ... 155645171 data packets ... discarded for bad header offset fields ... 790 connections established ...
    (freebsd-questions)
  • Re: Only some websites will open - Ubuntu
    ... incoming packets discarded ... 236 active connections openings ... 184 delayed acks sent ... TCPAbortOnSyn: 0 ...
    (comp.os.linux.misc)
  • Re: Deaf CAsyncSocket on Windows Service.
    ... To read them out I need a buffer to ... Short packets at the sender are coalesced into longer packets. ... THe receiver says "I have this much buffer space" and the sender is free to send ... You get an OnReceive notification if there is input data to be read. ...
    (microsoft.public.vc.mfc)