RAID 10 jfs: surprisingly different bonnie++ file creation times

I'm benchmarking two different disk subsystems on the same server. One
is a 4-disk RAID 10 with 10k drives, the other is a 12-disk RAID 10
with 15k drives. Both have JFS and use MegaRAID controllers for
hardware RAID.

I'm using bonnie++ as part of the benchmarking suite. I'm seeing
results that I expect for file I/O tests. The 12-disk RAID 10 is much
faster for sequential output, as I would expect.

The strange part is that for file creation, the numbers for the 4-disk
RAID 10 are high enough compared to the 12-disk RAID 10 that it's cause
for concern.

At first, I thought it might have something to do with the default
bonnie++ setting of testing 0-byte files for file creation, but I was
able to duplicate the discrepancy with larger files specified.

Here's the 4-disk RAID 10 with -n 128:20000:16:512:

Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
foo 16G 46631 94 66317 21 43116 8 43286 82 129767 12
533.6 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
128:20000:16/512 4216 21 108367 100 32620 76 4682 27 95906 100
1169 4

Here's the 12-disk RAID 10 with the same arguments:

Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
foo 16G 49728 97 132648 26 74033 13 41362 77 101426 7
596.9 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
128:20000:16/512 923 4 109247 100 621 1 615 3 94779 99
577 1

One striking thing is the very low CPU usage during the file creation
and deletion on the 12-disk RAID.

Does anyone know anything I should be looking at to help shed some
light on this aspect of the benchmark?

.

Relevant Pages

  • md: raid5 vs raid10 (f2,n2,o2) benchmarks [w/10 raptors]
    ... There has been a lot of discussion on the mailing list regarding the various raid10 replicas so I benchmarked them compared to RAID 5 all with no optimizations and then re-ran the tests with optimizations, RAID 5 still generally turns out the best speed for sequential writes but raid10_f2 seems to overtake raid5 for reads. ... Run the following bonnie++ benchmark 3 times and take the average. ... bonnie++ csv line below: ...
    (Linux-Kernel)
  • Re: Stress testing a 2.7 TB filesystem?
    ... I have also used Bonnie++ to test RAID systems larger than 1TB both ... including Sun x86 systems, IBM x86 systems and HP x86 systems when ... single CPU V210 and slightly more on the dual CPU V240.. ...
    (comp.sys.sun.admin)
  • Inconsistent bonnie++ Output Across Various RAID Configurations
    ... a 4-disk JFS RAID 10, ... I tried the tests repeatedly, also throwing in -n 128:20000:16:512 thinking that 0-byte files might have something to do with it, but the file creation performance profiles stayed roughly the same. ... To UNSUBSCRIBE, email to debian-user-REQUEST@xxxxxxxxxxxxxxxx with a subject of "unsubscribe". ...
    (Debian-User)