Re: need fastest way to write 2gig array to disk file

From: John Fusco (fusco_john_at_yahoo.com)
Date: 09/10/05 

Date: Sat, 10 Sep 2005 02:11:34 GMT

Eric Taylor wrote:
> I have a two gigabtye array. From a nothing special C program
> what is the fastest way to write this to a disk file.
>
> I'm currently using fwrite with 25meg shots at a time so I can
> provide a % progress output. I am seeing very different
> results.
>
> On one system, a 12 gig 2.6 kernel (redhat el4 hugemem) all the
> writes seem to go directly to the cache and so a 2 gig output
> takes under 30 seconds before the program thinks it's done.
>
> On another similar system, with scsi disks, once the program
> has written about 800 meg, it slows down to about 4 meg/sec
> as reported by iostat. (when writing to the cache, it says about
> 70meg/sec).
>
> Top says almost all the memory is free, except what I'm using.
> If once it is done with writing one file (2 gigs) I start it writing a
> a second file, it goes slow throughout. If while it is
> writing, I delete the first file, then it takes off again, presumably
> writing to the cache which has now been freed up.
>
> Are there tuning parameters to set the maximum amount of cache,
> or some way to base it on the amount of free memory left?
>
> And why would a write from memory to a scsi disk go at only
> 4 mb/sec. That seems awfully slow.
>
> any help would be most appreciated.
>
> eric
>
>
Try write() instead of fwrite(). ANSI FILE handles keep user-space
buffers on top of the disk cache, which means an extra memcpy.

You might also try using the O_DIRECT flag when you open the file, which
will bypass the filesystem cache and make your write synchronous (if the
filesystem you are writing to supports it). Generally, bypassing the
cache should be faster, but I've seen different behavior on different
filesystems.

See also open(2), setvbuf(3)

John

Relevant Pages

  • Re: need fastest way to write 2gig array to disk file
    ... Actually, I think what I need is to be able to write entirely to the cache, so ... made as large as, say, 75% of all free memory? ... The file I am writing is a saved state of a large simulation. ... >> On another similar system, with scsi disks, once the program ...
    (comp.os.linux.development.apps)
  • RE: Disk vs Tape scenario
    ... With the storage available for me to test it is hard to eliminate cache ... Theoretically a volume using RAID-10 on 8x10K RPM disks can handle ... A write SSCH rate of 475/sec would push the RAID-10 parity group to 950 ... I just tested with two IEBDG jobs writing 20 million 80 byte records ...
    (bit.listserv.ibm-main)
  • Re: Cache questions
    ... > sets and the MMX and XMM registers where possible. ... > I'm writing a routine which will need to read from a source bitmap ... especially when it causes cache misses. ... > some stack space, and I could write one of my local variables with, ...
    (comp.lang.asm.x86)
  • Re: [opensuse] Hard disc questions Slight OT
    ... speed than outer radius. ... but they pack more data onto the outer tracks than ... Since you are writing to a file, you don't have anyway to say what ... Kernel cache can be even bigger. ...
    (SuSE)
  • Cache questions
    ... sets and the MMX and XMM registers where possible. ... I'm writing a routine which will need to read from a source bitmap ... But actually my knowledge of how the L1/L2 cache works is a little ... my local variables to memory, what's the best way to do that? ...
    (comp.lang.asm.x86)