high iowait, high load average when copying file

Hi,

I have a Dell Poweredge 2950, 8GB memory, 5x73GB disks RAID5, 2-way
3.00 GHz processor running RHEL ES3 (update 8), kernel
2.4.21-50.ELsmp, 2GB swap.

The customer is reporting almost complete system halts when copying
files.

I attempted to copy a 1GB file to another name and had to abort after
5 minutes due to 100% iowait, 0% idle and a load avaerage of 26.

The snapshot of "top" looks like:

16:00:48 up 17 days, 9:34, 34 users, load average: 15.03, 6.58,
2.65
404 processes: 402 sleeping, 1 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait
idle
total 0.0% 0.0% 2.2% 0.0% 0.0% 97.6%
0.0%
cpu00 0.0% 0.0% 15.5% 0.0% 0.0% 84.4%
0.0%
cpu01 0.1% 0.0% 0.5% 0.0% 0.0% 99.2%
0.0%
cpu02 0.1% 0.0% 0.9% 0.0% 0.0% 98.8%
0.0%
cpu03 0.0% 0.0% 0.1% 0.0% 0.0% 99.8%
0.0%
cpu04 0.1% 0.0% 0.5% 0.0% 0.0% 99.2%
0.0%
cpu05 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
0.0%
cpu06 0.0% 0.0% 0.3% 0.0% 0.0% 99.6%
0.0%
cpu07 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
0.0%
Mem: 8199452k av, 7883568k used, 315884k free, 0k shrd,
114100k buff
2217892k actv, 4201540k in_d, 141500k in_c
Swap: 2096440k av, 1916k used, 2094524k free
7008716k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU
COMMAND
5697 root 39 19 0 0 0 SWN 2.0 0.0 286:11 0
kipmi0
1 root 15 0 496 496 436 S 0.0 0.0 0:17 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0
migration/0
3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1
migration/1
4 root RT 0 0 0 0 SW 0.0 0.0 0:00 2
migration/2
5 root RT 0 0 0 0 SW 0.0 0.0 0:00 3
migration/3
6 root RT 0 0 0 0 SW 0.0 0.0 0:00 4
migration/4
7 root RT 0 0 0 0 SW 0.0 0.0 0:00 5
migration/5
8 root RT 0 0 0 0 SW 0.0 0.0 0:00 6
migration/6
9 root RT 0 0 0 0 SW 0.0 0.0 0:00 7
migration/7
10 root 15 0 0 0 0 SW 0.0 0.0 0:00 6
keventd
11 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0
ksoftirqd/0
12 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1
ksoftirqd/1
13 root 34 19 0 0 0 SWN 0.0 0.0 0:00 2
ksoftirqd/2
14 root 34 19 0 0 0 SWN 0.0 0.0 0:00 3
ksoftirqd/3
15 root 34 19 0 0 0 SWN 0.0 0.0 0:00 4
ksoftirqd/4
16 root 34 19 0 0 0 SWN 0.0 0.0 0:00 5
ksoftirqd/5
17 root 34 19 0 0 0 SWN 0.0 0.0 0:00 6
ksoftirqd/6
18 root 34 19 0 0 0 SWN 0.0 0.0 0:00 7
ksoftirqd/7
21 root 15 0 0 0 0 SW 0.0 0.0 0:08 2
bdflush
19 root 15 0 0 0 0 SW 0.0 0.0 9:03 2
kswapd
20 root 15 0 0 0 0 SW 0.0 0.0 2:36 2
kscand
22 root 15 0 0 0 0 DW 0.0 0.0 3:13 1
kupdated
23 root 25 0 0 0 0 SW 0.0 0.0 0:00 0
mdrecoveryd
30 root 15 0 0 0 0 SW 0.0 0.0 0:00 3
ahc_dv_0
31 root 15 0 0 0 0 SW 0.0 0.0 0:00 2
ahc_dv_1
32 root 25 0 0 0 0 SW 0.0 0.0 0:00 3
scsi_eh_0
33 root 25 0 0 0 0 SW 0.0 0.0 0:00 3
scsi_eh_1
35 root 21 0 0 0 0 SW 0.0 0.0 0:00 7
scsi_eh_2

I have other very similar servers running the same O/S on almost
identical hardware. I have tried downgrading and upgrading the kernel
with slight differences but all basically not performing.

I can see many people with similar problems, but no real fix.

Any ideas?

Thanks in advance.

.

Relevant Pages