Re: Throughput question with CF/DiskOnChip

From: Chris Isbell (chris217_NOSPAAM_at_voila.fr)
Date: 01/02/05 

Date: Sun, 02 Jan 2005 10:24:26 +0000

On 1 Jan 2005 14:39:01 -0800, "jro" <john@jrobot.net> wrote:

>In my application, I need to store streaming data to disk (i.e. CF or
>DOC), at a speed of approximately 176 Kbit/sec. This is typically in
>the form of 16 separate files that have been opened (with 11 Kbits/sec
>of data going to each file). I originally found that streaming the
>data to DOC caused a large "blip" in the CPU usage at various times
>(could be after 1 min or 10 mintes or 45 minutes). This "blip" was
>detected by running "top" on the system, and the DOC driver would end
>up using 99% or the CPU cycles during this blip which lasted about 20
>seconds or so. Since the DOC driver runs at a higher priority (-20)
>than my application (0), it essentially locked up the rest of the
>system and which caused an eventual failure in the system due to data
>buffers overflowing.
>

Three points:

1. Flash media is generally very slow, especially for writing data,
and especially if it has to erase previously written data before
writing.

2. The DOC is probably not ideal for what you are trying to achieve;
there is simply too much going on outside of your direct control.

3. Using magnetic disk filing systems, especially journalling ones,
with flash media is generally bad news.

When designing a system recording data onto flash media - MultiMedia
Card (MMC) in my case - we took a very different approach. Rather than
using files, we simply divided the MMC address space into a number of
different areas and treated those to which data was being streamed as
ring buffers. The application firmware writes directly to the media
and does not go through a filing system.

We also wrote a static FAT-16 file system structure onto the media to
make it easier for other application to read the data. This is not
used by our firmware.

A further performance enhancement is to pre-erase a number of blocks
ahead of the data to be written. It is also a good idea to have the
largest possible RAM memory buffer between the data source and the
media.

All of this may not be necessary in your case. (We were working on a
16-bit 20MHz microcontroller, communicating with the MMC using a slow
SPI interface.) However, you could, for example, pre-allocate the
space for your files in advance, to avoid multiple file extensions and
associated file structure updates. I suspect that this activity may be
triggering wear-leveling in the DOC driver to avoid wearing-out the
flash memory blocks holding filing system data. 

-- 
Chris Isbell
Southampton, UK