Machine with load when CPU 100% idle

Hello,
recently we received 15 sun x4100 with 2 dual core amd processors and after installing centos 4 on them we noticed that there was a load on some of the machines and not others when the system is completely idle.

This load stays on for some time then goes off. Machines that didn't have the load eventually start having it and eventually it goes away. I looked at the processes and there's nothing there out of the ordinary.

We have lots of other machines that we installed with the same configuration and same patch level and they don't have that behavior. This seems to happen only on the 2 dual cores machines that we have.

top - 09:55:13 up 31 min, 1 user, load average: 1.04, 0.74, 0.47
Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 0.1% sy, 0.0% ni, 99.6% id, 0.2% wa, 0.1% hi, 0.0% si
Mem: 8046460k total, 115932k used, 7930528k free, 12136k buffers
Swap: 12586888k total, 0k used, 12586888k free, 47160k cached

These machines are running at level 3 without X running.
I ran a vmstat on a machine that had this problem at the same time as on one that didn't and both showed the same numbers.

There are probably a couple more services i could turn off but i tried disabling: gpm xfs acpid, iiim and ntpd and it didn't make any difference.

Here's a proccess list:

F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 S root 1 0 0 76 0 - 1188 - 09:24 ? 00:00:00 init [3]
1 S root 2 1 0 -40 - - 0 migrat 09:24 ? 00:00:00 [migration/0]
1 S root 3 1 0 94 19 - 0 ksofti 09:24 ? 00:00:00 [ksoftirqd/0]
1 S root 4 1 0 -40 - - 0 migrat 09:24 ? 00:00:00 [migration/1]
1 S root 5 1 0 94 19 - 0 ksofti 09:24 ? 00:00:00 [ksoftirqd/1]
1 S root 6 1 0 -40 - - 0 migrat 09:24 ? 00:00:00 [migration/2]
1 S root 7 1 0 94 19 - 0 ksofti 09:24 ? 00:00:00 [ksoftirqd/2]
1 S root 8 1 0 -40 - - 0 migrat 09:24 ? 00:00:00 [migration/3]
1 S root 9 1 0 94 19 - 0 ksofti 09:24 ? 00:00:00 [ksoftirqd/3]
5 S root 10 1 0 65 -10 - 0 worker 09:24 ? 00:00:00 [events/0]
1 S root 11 1 0 65 -10 - 0 worker 09:24 ? 00:00:00 [events/1]
1 S root 12 1 0 65 -10 - 0 worker 09:24 ? 00:00:00 [events/2]
1 S root 13 1 0 65 -10 - 0 worker 09:24 ? 00:00:00 [events/3]
1 S root 14 10 0 68 -10 - 0 worker 09:24 ? 00:00:00 [khelper]
1 S root 15 10 0 75 -10 - 0 worker 09:24 ? 00:00:00 [kacpid]
1 S root 52 10 0 65 -10 - 0 worker 09:24 ? 00:00:00 [kblockd/0]
1 S root 53 10 0 65 -10 - 0 worker 09:24 ? 00:00:00 [kblockd/1]
1 S root 54 10 0 65 -10 - 0 worker 09:24 ? 00:00:00 [kblockd/2]
1 S root 55 10 0 65 -10 - 0 worker 09:24 ? 00:00:00 [kblockd/3]
1 S root 56 1 0 75 0 - 0 hub_th 09:24 ? 00:00:00 [khubd]
1 S root 70 10 0 80 0 - 0 pdflus 09:24 ? 00:00:00 [pdflush]
1 S root 71 10 0 75 0 - 0 pdflus 09:24 ? 00:00:00 [pdflush]
1 S root 74 10 0 74 -10 - 0 worker 09:24 ? 00:00:00 [aio/0]
1 S root 72 1 0 85 0 - 0 kswapd 09:24 ? 00:00:00 [kswapd1]
1 S root 73 1 0 85 0 - 0 kswapd 09:24 ? 00:00:00 [kswapd0]
1 S root 75 10 0 65 -10 - 0 worker 09:24 ? 00:00:00 [aio/1]
1 S root 76 10 0 65 -10 - 0 worker 09:24 ? 00:00:00 [aio/2]
1 S root 77 10 0 65 -10 - 0 worker 09:24 ? 00:00:00 [aio/3]
1 S root 150 1 0 84 0 - 0 serio_ 09:24 ? 00:00:00 [kseriod]
1 S root 247 1 0 76 0 - 0 160455 09:24 ? 00:00:00 [scsi_eh_0]
1 S root 273 1 0 75 0 - 0 kjourn 09:24 ? 00:00:00 [kjournald]
4 S root 1292 1 0 66 -10 - 901 - 09:24 ? 00:00:00 udevd
1 S root 1801 1 0 80 0 - 0 160455 09:24 ? 00:00:00 [scsi_eh_1]
1 S root 1802 1 0 75 0 - 0 - 09:24 ? 00:00:00 [usb-storage]
1 S root 1837 1 0 76 0 - 0 160455 09:24 ? 00:00:00 [scsi_eh_2]
1 D root 1838 1 0 75 0 - 0 usb_st 09:24 ? 00:00:00 [usb-storage]
1 S root 1890 12 0 66 -10 - 0 kaudit 09:24 ? 00:00:00 [kauditd]
1 S root 2017 13 0 65 -10 - 0 worker 09:24 ? 00:00:00 [kmpathd/0]
1 S root 2018 13 0 65 -10 - 0 worker 09:24 ? 00:00:00 [kmpathd/1]
1 S root 2019 13 0 65 -10 - 0 worker 09:24 ? 00:00:00 [kmpathd/2]
1 S root 2020 13 0 68 -10 - 0 worker 09:24 ? 00:00:00 [kmpathd/3]
1 S root 2044 13 0 68 -10 - 0 worker 09:24 ? 00:00:00 [kmirrord]
1 S root 2045 13 0 68 -10 - 0 worker 09:24 ? 00:00:00 [kmir_mon]
5 S root 2726 1 0 76 0 - 1652 - 09:25 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhcp/dhclient-eth0.leases -pf /var/run/dhclient-5 S root 2784 1 0 76 0 - 907 - 09:25 ? 00:00:00 syslogd -m 0
5 S root 2788 1 0 75 0 - 634 syslog 09:25 ? 00:00:00 klogd -x
5 S root 2799 1 0 76 0 - 637 - 09:25 ? 00:00:00 irqbalance
5 S rpc 2809 1 0 76 0 - 1187 - 09:25 ? 00:00:00 portmap
5 S rpcuser 2829 1 0 78 0 - 1716 - 09:25 ? 00:00:00 rpc.statd
5 S root 2878 1 0 76 0 - 18156 - 09:25 ? 00:00:00 ypbind
5 S root 2989 1 0 76 0 - 2499 - 09:25 ? 00:00:00 /usr/sbin/automount --timeout=60 /home yp auto.home
5 S root 3003 1 0 79 0 - 635 - 09:25 ? 00:00:00 /usr/sbin/acpid
5 S root 3050 1 0 76 0 - 5490 - 09:25 ? 00:00:00 /usr/sbin/sshd
5 S root 3065 1 0 75 0 - 2178 - 09:25 ? 00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
5 S ntp 3078 1 0 76 0 - 4924 - 09:25 ? 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
5 S root 3088 1 0 76 0 - 904 - 09:25 ? 00:00:00 rpc.rstatd
1 S nobody 3100 1 0 81 0 - 907 - 09:25 ? 00:00:00 rpc.rusersd
4 S root 3155 1 0 76 0 - 6289 - 09:25 ? 00:00:00 /usr/libexec/postfix/master
5 S root 3165 1 0 75 0 - 1044 - 09:25 ? 00:00:00 gpm -m /dev/input/mice -t imps2
4 S postfix 3166 3155 0 76 0 - 6571 - 09:25 ? 00:00:00 pickup -l -t fifo -u
4 S postfix 3167 3155 0 76 0 - 6584 - 09:25 ? 00:00:00 qmgr -l -t fifo -u
1 S htt 3196 1 0 78 0 - 887 wait 09:25 ? 00:00:00 /usr/sbin/htt -retryonerror 0
0 S htt 3197 3196 0 75 0 - 10229 - 09:25 ? 00:00:00 htt_server -nodaemon
5 S root 3207 1 0 76 0 - 14822 - 09:25 ? 00:00:00 crond
5 S xfs 3245 1 0 76 0 - 3691 - 09:25 ? 00:00:00 xfs -droppriv -daemon
5 S root 3264 1 0 76 0 - 2238 - 09:25 ? 00:00:00 /usr/sbin/atd
5 S dbus 3280 1 0 76 0 - 2687 - 09:25 ? 00:00:00 dbus-daemon-1 --system
5 S root 3290 1 0 76 0 - 2248 - 09:25 ? 00:00:00 cups-config-daemon
5 D root 3301 1 0 75 0 - 5061 scsi_w 09:25 ? 00:00:00 hald
1 S root 3336 1 0 75 0 - 0 rpciod 09:25 ? 00:00:00 [rpciod]
1 S root 3337 1 0 79 0 - 0 - 09:25 ? 00:00:00 [lockd]
5 S sungrid 3496 1 0 76 0 - 14557 - 09:25 ? 00:00:00 /home/sungrid/sge-6.0u3/bin/lx24-amd64/sge_execd
4 S root 3499 1 0 77 0 - 631 - 09:25 tty1 00:00:00 /sbin/mingetty tty1
4 S root 3500 1 0 78 0 - 631 - 09:25 tty2 00:00:00 /sbin/mingetty tty2
4 S root 3501 1 0 78 0 - 631 - 09:25 tty3 00:00:00 /sbin/mingetty tty3
4 S root 3502 1 0 78 0 - 631 - 09:25 tty4 00:00:00 /sbin/mingetty tty4
4 S root 3503 1 0 78 0 - 631 - 09:25 tty5 00:00:00 /sbin/mingetty tty5
4 S root 3505 1 0 78 0 - 631 - 09:25 tty6 00:00:00 /sbin/mingetty tty6
4 S root 4022 3050 0 76 0 - 9315 - 09:32 ? 00:00:00 sshd: root@pts/0
4 S root 4024 4022 0 76 0 - 13752 wait 09:32 pts/0 00:00:00 -bash
4 R root 4232 4024 0 78 0 - 1906 - 09:56 pts/0 00:00:00 ps -elf

List of services:

microcode_ctl 0:off 1:off 2:on 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
yum 0:off 1:off 2:off 3:off 4:off 5:off 6:off
ripd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
cups-config-daemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
arptables_jf 0:off 1:off 2:on 3:on 4:on 5:on 6:off
amd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
hpoj 0:off 1:off 2:off 3:off 4:off 5:off 6:off
spamassassin 0:off 1:off 2:off 3:off 4:off 5:off 6:off
dhcp6s 0:off 1:off 2:off 3:off 4:off 5:off 6:off
mysqld 0:off 1:off 2:off 3:off 4:off 5:off 6:off
smartd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
autofs 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rstatd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
readahead_early 0:off 1:off 2:off 3:off 4:off 5:on 6:off
snmpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
postgresql 0:off 1:off 2:off 3:off 4:off 5:off 6:off
named 0:off 1:off 2:off 3:off 4:off 5:off 6:off
lm_sensors 0:off 1:off 2:on 3:on 4:on 5:on 6:off
vncserver 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rpcidmapd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
ospfd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
portmap 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rawdevices 0:off 1:off 2:off 3:on 4:on 5:on 6:off
rpcgssd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
ospf6d 0:off 1:off 2:off 3:off 4:off 5:off 6:off
dhcrelay 0:off 1:off 2:off 3:off 4:off 5:off 6:off
netplugd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rhnsd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
bluetooth 0:off 1:off 2:off 3:off 4:off 5:off 6:off
netdump 0:off 1:off 2:off 3:off 4:off 5:off 6:off
anacron 0:off 1:off 2:on 3:on 4:on 5:on 6:off
ypbind 0:off 1:off 2:off 3:on 4:on 5:on 6:off
xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:off 3:on 4:on 5:on 6:off
syslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
mdmonitor 0:off 1:off 2:off 3:off 4:off 5:off 6:off
irda 0:off 1:off 2:off 3:off 4:off 5:off 6:off
ldap 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rusersd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
radvd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
pcmcia 0:off 1:off 2:off 3:off 4:off 5:off 6:off
readahead 0:off 1:off 2:off 3:off 4:off 5:on 6:off
ip6tables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ypxfrd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
winbind 0:off 1:off 2:off 3:off 4:off 5:off 6:off
firstboot 0:off 1:off 2:off 3:on 4:off 5:on 6:off
innd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
dhcpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
xfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off
httpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
FreeWnn 0:off 1:off 2:off 3:off 4:off 5:off 6:off
mailman 0:off 1:off 2:off 3:off 4:off 5:off 6:off
iscsi 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rpcsvcgssd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
gpm 0:off 1:off 2:on 3:on 4:on 5:on 6:off
zebra 0:off 1:off 2:off 3:off 4:off 5:off 6:off
psacct 0:off 1:off 2:off 3:off 4:off 5:off 6:off
mdmpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
diskdump 0:off 1:off 2:off 3:off 4:off 5:off 6:off
canna 0:off 1:off 2:off 3:off 4:off 5:off 6:off
iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
krb524 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rwhod 0:off 1:off 2:off 3:off 4:off 5:off 6:off
nfslock 0:off 1:off 2:off 3:on 4:on 5:on 6:off
arpwatch 0:off 1:off 2:off 3:off 4:off 5:off 6:off
ntpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
dc_server 0:off 1:off 2:off 3:off 4:off 5:off 6:off
vsftpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
auditd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
dc_client 0:off 1:off 2:off 3:off 4:off 5:off 6:off
radiusd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
NetworkManager 0:off 1:off 2:off 3:off 4:off 5:off 6:off
acpid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
smb 0:off 1:off 2:off 3:off 4:off 5:off 6:off
iiim 0:off 1:off 2:on 3:on 4:on 5:on 6:off
krb5kdc 0:off 1:off 2:off 3:off 4:off 5:off 6:off
squid 0:off 1:off 2:off 3:off 4:off 5:off 6:off
cups 0:off 1:off 2:off 3:off 4:off 5:off 6:off
bootparamd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sgeexecd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
bgpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
dovecot 0:off 1:off 2:off 3:off 4:off 5:off 6:off
isdn 0:off 1:off 2:on 3:on 4:on 5:on 6:off
multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
lisa 0:off 1:off 2:off 3:off 4:off 5:off 6:off
snmptrapd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
cyrus-imapd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
nscd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
kprop 0:off 1:off 2:off 3:off 4:off 5:off 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ypserv 0:off 1:off 2:off 3:off 4:off 5:off 6:off
yppasswdd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off
netdump-server 0:off 1:off 2:off 3:off 4:off 5:off 6:off
tux 0:off 1:off 2:off 3:off 4:off 5:off 6:off
ipmi 0:off 1:off 2:off 3:off 4:off 5:off 6:off
kadmin 0:off 1:off 2:off 3:off 4:off 5:off 6:off
kudzu 0:off 1:off 2:off 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ripngd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
saslauthd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
xinetd based services:
telnet: off
time-udp: off
swat: off
krb5-telnet: off
ktalk: off
rsync: off
amanda: off
amidxtape: off
finger: off
rexec: off
talk: off
eklogin: off
auth: on
daytime: off
chargen: off
gssftp: off
klogin: off
kshell: off
dbskkd-cdb: off
rsh: on
amandaidx: off
daytime-udp: off
echo: off
time: off
tftp: off
rlogin: off
echo-udp: off
ntalk: off
chargen-udp: off
cups-lpd: off


I see nothing in the log files that indicate any sort of problems.
Any idea on how to fix this?
.

Relevant Pages

  • Re: do_raw_spin_lock using a lot of the system cpu time
    ... machines spiked in load. ... I'm also seeing this on 3 different kernels: ... Provide a free area cache for the vmalloc virtual address allocator, ...
    (Linux-Kernel)
  • VM in 2.6 doing a worse job of caching than 2.4?
    ... day by a proprietary database system. ... I recently started evaluating the 2.6 kernel for these machines. ... and see's CPU idle of 30-35%. ... Theoretically, they both should receive similar traffic, though the load ...
    (comp.os.linux.development.system)
  • Re: Very high load on P4 machines with 2.4.28
    ... The machines have normal load averages hovering not higher than ... Booted back in the old kernel, ... other box with the similar configuration to the virtuals (also a virtual ...
    (Linux-Kernel)
  • Re: Virtualization options for Sparc?
    ... since I am not expecting a whole lot of ... load, but obviously all-out cpu emulation is not an option. ... UltraSparc II based machines. ... non-virtualized server has got only 512 MB and is doing ...
    (comp.sys.sun.admin)
  • Re: Power supply protection networks
    ... load the carousel for N boards when we only want to stuff N/3, ... But that carousel load is still done from a picked kit, ... In my area of work the typical method is to always load whole reels and then return them to the storage area when no longer needed. ... those machines must be pretty ancient. ...
    (sci.electronics.design)