Re: Continuous disk write performance in ported real-time application

On Thu, 30 Jul 2009, Jonathan wrote:
On Jul 30, 6:49 am, Vitus Jensen <vi...@xxxxxxxxxxxxxxxx> wrote:
On Wed, 29 Jul 2009, Jonathan wrote:

[...300KB images at 10 Hz...]

What we are seeing is that in most cases, it takes 1-3 ms to "write"
each image file.  Since this is obviously much faster than it is
actually possible to write 300 kB of data, it is clear it is making
good use of RAM to cache the file before actually writing it to the CF
disk.  However, sometimes it will take much longer to complete the
image write call, from a few hundred ms to a few seconds.  Using
things like vmstat, iostat, and /proc/meminfo, we have been able to
see that Linux is writing out the buffered data to disk in spurts.
When it starts to get "behind" it will block the application and wait
for some data to be written out to disk before continuing.

What we would like to do is to even out those spurts of actual disk
writing to be more consistent.  The application takes a relatively
small portion of the 100 ms per image to process, so it should have
quite a bit of time to write out an image to disk.  But it seems to
just sit there at times, content to let the disk cache build up.  Does
anyone know where we might look to tune something or change some
setting that would get us more consistent disk writing?  Thanks!

As you are writing much more than the cache on your embedded device can
hold why use the cache at all?  I it were real small chunks you would
win something because of write combining but in your case I don't see any
benefit at all.

So I would use open(,O_SYNC), create another thread in your application,
pass images to it via some sort of queue and let it do the writing.  You
need the queue because the CF disk itself could sometimes stop to do some
housekeeping but otherwise you get a constant write rate.

Because the cache is useful for buffering and smoothing out the
writing. Like you said, the CF may not always allow a constant write
rate, so there needs to be buffering somewhere. Letting the OS do it
is "free" in terms of development. We have 1 GB of RAM, of which our
application uses very little, so while the cache cannot hold a hour
worth of video, it can hold quite a bit. We have tried using O_SYNC
and it takes longer than 100 ms to write the files every time (even
though 100 ms seems like plenty of time to write 300 kB out to disk).

I see. So O_SYNC might write every fs allocation unit directly to disk or similar. BTW: which filesystem are you using?

We have not experimented with changing the architecture of the program
yet (like adding in-application buffering). There already is a
separate thread that is doing the writing to disk. Thanks for the
reply.

Now you did confuse me, i thought it was a

for(;;) {
get_image();
write();
}

loop?

Anyway, you asked for tuning of the cache behaviour. Look at /proc/sys/vm, documented in Documentation/sysctl/vm.txt. Decreasing "dirty" limits might help you.

I still think this is not the best approach: dependend on kernel version behaviour and if you hit some limit during write() the vm might decide to do the flush during your write() and lengthen the syscall. A seperate thread and some local queue, fsync()s to not compete against the cache about memory (as O_SYNC doesn't work).

Vitus


PS: 1GB of RAM and "embedded" :-O Where did those 128K devices go?

--
Vitus Jensen, Hannover, Germany, Earth, Universe (current)
.

Relevant Pages

  • Temporary lock-up under heavy write, MegaRAID RAID-5
    ... LSI Logic MegaRAID 320-1, 64MB cache ... when writing very large files. ... disk write access, actually). ... The RAID controller is currently set to "write-through". ...
    (Debian-User)
  • need fastest way to write 2gig array to disk file
    ... writes seem to go directly to the cache and so a 2 gig output ... (when writing to the cache, ... Top says almost all the memory is free, ... And why would a write from memory to a scsi disk go at only ...
    (comp.os.linux.development.apps)
  • Re: need fastest way to write 2gig array to disk file
    ... >what is the fastest way to write this to a disk file. ... >writes seem to go directly to the cache and so a 2 gig output ... (when writing to the cache, ... >any help would be most ap-t perform device read timings ...
    (comp.os.linux.development.apps)
  • Re: Invalidating the file cache
    ... will force a cache flush to ensure that the data is to disk. ... I write to a file (with buffering). ... I start reading from the same file with no buffering. ...
    (microsoft.public.win32.programmer.kernel)
  • Re: Continuous disk write performance in ported real-time application
    ... good use of RAM to cache the file before actually writing it to the CF ... see that Linux is writing out the buffered data to disk in spurts. ... Because the cache is useful for buffering and smoothing out the ...
    (comp.os.linux.embedded)