VM in 2.6 doing a worse job of caching than 2.4?
From: Clint Byrum (cbyrum_at_spamaps.org)
Date: 05/28/04
- Next message: Clint Byrum: "Re: New FS with search properties"
- Previous message: Tauno Voipio: "Re: why I told dd command to write but it reads in first?"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Thu, 27 May 2004 15:44:31 -0700
**** Post for FREE via your newsreader at post.usenet.com ****
Ok, so I run a load balanced (using mod_backhand) web/search cluster of
a bunch of Dual P4 Xeon 2.8Ghz (512k Cache) machines with 1GB RAM
running 2.4.23 - 2.4.26 (upgrades are slow..). These servers each have a
text database, about 3GB large that is searched millions of times per
day by a proprietary database system. This text database is stored on a
software RAID1 volume made up of 2x 8MB Cache 40GB 7200rpm WDC ATA/100
drives, plugged into an ICH3 onboard controller. No swap is configured
(whenever it has been configured, less than 2MB of it gets used, as
these machines run almost nothing that can be swapped).
No, these disks aren't supposed to go fast. In fact, the cheapest disks
that we thought could survive a server workload were used. The reason
is, while there is 3GB of data to search, really only the indexes get
hit a lot, and they only take up about 600 - 700MB of virtual memory.
I recently started evaluating the 2.6 kernel for these machines. Our
readings on 2.6 indicated that it would improve scheduler performance,
reduce kernel memory footprint, and improve disk and VM performance.
This got us excited, as it had the potential to hopefully give us a boost.
This, unfortunately, does not appear to be the case. All of these
machines have identical hardware, and on 2.4.23 - 2.4.26, show identical
results (less than 1% difference). Generally at peak times, one of these
machines will be doing 8 requests per second, and have CPU idle of about
15 - 30%. Running vmstat will show that they're reading in about 5 - 10
blocks per second.
The host that we upgraded to 2.6.6, with an identical config (except the
new options like 4KSTACKS, which is on, and elevator=deadline), gets to
about 7 requests per second, and see's CPU idle of 30-35%. The kicker
is, in 2.6, the box will be reading 25 - 50 blocks per second, and
sometimes spike up to 250 blocks per second. This chokes off the data
flowing to the CPU pretty badly.
Some command output is below. These commands were run simultaneously on
two identical machines, whose only difference is kernel version.
Theoretically, they both should receive similar traffic, though the load
balancing is based on the load average of the system, which can
fluctuate some for more complex searches. After 2 weeks of running
similar tests, this behavior seems consistent though.
I'm eager to answer any questions or try out experiments. I personally
think this is some sort of fluke, and I'd love to get to the bottom of
it. Thanks
------ Command Output ------
2.4.23:
$ free -m ; uptime ; vmstat 5 5
total used free shared buffers cached
Mem: 1007 983 24 0 12 812
-/+ buffers/cache: 157 850
Swap: 0 0 0
14:51:06 up 113 days, 22:22, 1 user, load average: 0.70, 0.59, 0.64
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 0 0 25284 13284 832500 0 0 1 1 1 1 12
2 86 0
1 0 0 20748 13312 832688 0 0 37 39 139 92 3
0 97 0
3 0 0 16720 13348 833020 0 0 63 80 189 222 12
2 86 0
0 0 0 23672 13376 833184 0 0 31 57 166 142 7
1 92 0
1 0 0 16572 13412 833288 0 0 20 51 155 137 5
1 93 0
2.6.6:
$ free -m ; uptime ; vmstat 5 5
total used free shared buffers cached
Mem: 1010 990 20 0 27 867
-/+ buffers/cache: 95 915
Swap: 0 0 0
14:51:05 up 7 days, 1:17, 1 user, load average: 0.59, 0.66, 0.76
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 1 0 20732 28220 888556 0 0 11 12 11 14 9
2 88 2
0 0 0 27552 28260 883416 0 0 223 107 198 232 19
3 76 2
0 0 0 26452 28276 884420 0 0 226 33 192 217 6
1 87 6
0 0 0 26388 28308 884660 0 0 36 56 154 136 4
0 95 0
0 0 0 25536 28344 885236 0 0 114 62 173 186 8
2 89 1
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
*** Usenet.com - The #1 Usenet Newsgroup Service on The Planet! ***
http://www.usenet.com
Unlimited Download - 19 Seperate Servers - 90,000 groups - Uncensored
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
- Next message: Clint Byrum: "Re: New FS with search properties"
- Previous message: Tauno Voipio: "Re: why I told dd command to write but it reads in first?"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
