followup: ext3 10x slowdown on rhel4 writing large files

(below I describe a suspicious data point, when discontinuity
wraps to lower block numbers - this is a new discovery - and
journalling options did not affect the results)

I have been analyzing an anomaly using the ext3 filesystem,
under redhat enterprise 4 (rhel4) but not under version 3
(rhel3).

I?ve reported in another thread that I?ve seen an order of
magnitude slowdown that occurs in the middle of writing to a
file. It will write at ~40+meg/sec until it reaches a certain
point, and then only ~4mb/sec. I am using direct I/O in a
small test program, writing 100meg per write. I have created a
dual boot system where I can be certain the hardware is
identical. I write to a file, then delete it. When I write the
same file, I get the same file extents under both rhel3 and
rhel4.

The slowdown seems to be occurring at a spot where there is
some discontinuity, and quite suspiciously, there is a
wraparound of blocks. I can?t be certain it hits exactly here,
but it?s real close. I kill the program as soon as it begins
to slow down.

I had to write a rather large file to force this to happen, as
it seems to depend on the current state of the free blocks on
the partition. However, I?ve seen it on smaller files, but it
has been difficult to reproduce consistently. So, I chose to
write a file until it slows down, in this case at about 17
gigs. There are some 87 gigs free before I start. It repeats
very consistently.


Here is what filefrag tells me?

Checking zzz
Filesystem type is: ef53
Filesystem cylinder groups is approximately 65944
Blocksize of file zzz is 4096
File size of zzz is 18329600000 (4475000 blocks)
Discontinuity: Block 7 is at 23210817 (was 23182791)
Discontinuity: Block 13 is at 23211265 (was 23210823)
Discontinuity: Block 20 is at 23254785 (was 23211271)

snip...

Discontinuity: Block 4290572 is at 34229256 (was 34229248)
Discontinuity: Block 4303864 is at 34243080 (was 34242559)
Discontinuity: Block 4331410 is at 9761 (was 34270652)
Discontinuity: Block 4340072 is at 18679 (was 18431)
Discontinuity: Block 4350299 is at 28917 (was 28915)
Discontinuity: Block 4354146 is at 34316 (was 32767)
Discontinuity: Block 4385336 is at 66051 (was 65535)
Discontinuity: Block 4417557 is at 99852 (was 98303)
Discontinuity: Block 4448747 is at 131586 (was 131071)
Discontinuity: Block 4469914 is at 152775 (was 152773)
Discontinuity: Block 4472786 is at 155650 (was 155648)
Discontinuity: Block 4474831 is at 157698 (was 157696)
zzz: 571 extents found, perfection would be 137 extents

Note the spot at 9761, where it wraps around to the front of
the disk. Could this be triggering some sort of arithmetic
sign change bug? If I kill the program, start it over and have
it overwrite all the same blocks, when it gets to the current
end and continues extending the file, the slowdown is NOT
there. It can occur later in the file (but not in this
particular case ? however I have seen this on other smaller
files).

One other item of interest: Under rhel3, when I overwrite the
file, I get an average write rate of 45 mb/sec, while under
rhel4, it is consistently faster at 50 mb/sec. So, I suspect
some optimization work has been done here.

The device is:

/dev/sdb5 129G 68G 56G 55% /home


Rhel3 is:
cat /proc/version
Linux version 2.4.21-20.ELhugemem
(bhcompile@xxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.2.3
20030502 (Red Hat Linux 3.2.3-42)) #1 SMP Wed Aug 18 20:32:11
EDT 2004

Rhel4 is:

cat /proc/version
Linux version 2.6.9-11.ELhugemem
(bhcompile@xxxxxxxxxxxxxxxxxxxxxxxxxx) (gcc version 3.4.3
20050227 (Red Hat 3.4.3-22)) #1 SMP Fri May 20
18:34:54 EDT 2005

I have not been able to repeat this anomaly on rhel3, although
it creates the same file with the same extents.

I?ve also tried this with 3 levels of journaling, defaults,
data=ordered, and data=writeback. All results are the same.

I first began seeing a problem when writing large files NOT
using direct I/O.

In this case, I suspect that whatever software flushes buffers
to disk, must be doing the equivalent of direct I/O, and if
that software runs into this anomaly, it could well explain an
apparent hang of my ?normal? writes.

I need to change this ext3 partition to ext2 and redo the test,
as before, going to ext2 did eliminate this problem. However,
I'm afraid my users will not accept this as a solution since it
could mean a long startup time if there is a crash.






.