Re: need fastest way ... found a problem on the disk

From: Eric Taylor (et1_at_rocketship1.com)
Date: 09/12/05

  • Next message: John Fusco: "Re: need fastest way to write 2gig array to disk file" 
    Date: Sun, 11 Sep 2005 18:34:26 -0700

    Well, it seems I've got an anomoly with either my disk, or the file
    system is having trouble. I now find that by writing my file to a
    different directory on this system all goes very fast. Into the
    cache in 10 seconds, no delays. And in top I can see it
    is added to the cache. May be time to rebuild this disk's
    file system.

    Sorry for the trouble, this is not an issue of how the program
    has been written.

    Eric Taylor wrote:

    > I have a two gigabtye array. From a nothing special C program
    > what is the fastest way to write this to a disk file.
    >
    > I'm currently using fwrite with 25meg shots at a time so I can
    > provide a % progress output. I am seeing very different
    > results.
    >
    > On one system, a 12 gig 2.6 kernel (redhat el4 hugemem) all the
    > writes seem to go directly to the cache and so a 2 gig output
    > takes under 30 seconds before the program thinks it's done.
    >
    > On another similar system, with scsi disks, once the program
    > has written about 800 meg, it slows down to about 4 meg/sec
    > as reported by iostat. (when writing to the cache, it says about
    > 70meg/sec).
    >
    > Top says almost all the memory is free, except what I'm using.
    > If once it is done with writing one file (2 gigs) I start it writing a
    > a second file, it goes slow throughout. If while it is
    > writing, I delete the first file, then it takes off again, presumably
    > writing to the cache which has now been freed up.
    >
    > Are there tuning parameters to set the maximum amount of cache,
    > or some way to base it on the amount of free memory left?
    >
    > And why would a write from memory to a scsi disk go at only
    > 4 mb/sec. That seems awfully slow.
    >
    > any help would be most appreciated.
    >
    > eric

  • Next message: John Fusco: "Re: need fastest way to write 2gig array to disk file"

    Relevant Pages

    • Re: Continuous disk write performance in ported real-time application
      ... good use of RAM to cache the file before actually writing it to the CF ... see that Linux is writing out the buffered data to disk in spurts. ... Because the cache is useful for buffering and smoothing out the ...
      (comp.os.linux.embedded)
    • need fastest way to write 2gig array to disk file
      ... writes seem to go directly to the cache and so a 2 gig output ... (when writing to the cache, ... Top says almost all the memory is free, ... And why would a write from memory to a scsi disk go at only ...
      (comp.os.linux.development.apps)
    • Re: Continuous disk write performance in ported real-time application
      ... good use of RAM to cache the file before actually writing it to the CF ... see that Linux is writing out the buffered data to disk in spurts. ... Because the cache is useful for buffering and smoothing out the ...
      (comp.os.linux.embedded)
    • Re: need fastest way to write 2gig array to disk file
      ... >what is the fastest way to write this to a disk file. ... >writes seem to go directly to the cache and so a 2 gig output ... (when writing to the cache, ... >any help would be most ap-t perform device read timings ...
      (comp.os.linux.development.apps)
    • Temporary lock-up under heavy write, MegaRAID RAID-5
      ... LSI Logic MegaRAID 320-1, 64MB cache ... when writing very large files. ... disk write access, actually). ... The RAID controller is currently set to "write-through". ...
      (Debian-User)