Re: need fastest way to write 2gig array to disk file

From: Basile Starynkevitch [news] (basile-news_at_starynkevitch.net)
Date: 09/10/05 

Date: Sat, 10 Sep 2005 06:54:52 +0000 (UTC)

On 2005-09-09, Eric Taylor <et1@rocketship1.com> wrote:
> I have a two gigabtye array. From a nothing special C program
> what is the fastest way to write this to a disk file.

You might consider using mmap on this array. Basically, instead of
allocating the array with malloc, you open a file for writing, then
mmap it, then msync; something like (untested code, and you really
should add all the error checks)

// the array size in byte has to be a multiple of the page size
#define BIGSIZE 16384*1024 /* number of doubles in your array */
int fd = open("your_big_file", O_RDWR|O_CREAT, 0640);
if (fd<0) errored("failed to open");
size_t arraysize = sizeof(double)*BIGSIZE;
// grow the file if needed by truncating it
ftruncate(fd, arraysize);
// map the array in memory
void* arrayaddr = mmap((void*)0, arraysize, PROT_READ|PROT_WRITE,
                MAP_SHARED, fd, (off_t)0);
if (arrayaddr==MAP_FAILED) errored("failed to mmap");
double* array = arrayaddr;
// compute & fill the array
compute_your_big_array(array, BIGSIZE);
// sync the array to disk
msync(ad, arraysize, MS_ASYNC);
// unmap memory and close file
munmap(arrayaddr, arraysize);
close(fd);

Read a good book on Linux or Posix system programming, and the man pages for
open(2), mmap(2), msync(2), munmap(2), close(2) system calls

Regards 

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/ 
email: basile(at)starynkevitch(dot)net 
8, rue de la Faïencerie, 92340 Bourg La Reine, France

Relevant Pages

  • Re: Marshalling Array In/Out C++ COM Control
    ... using Marshal.AllocCoTaskMem to allocate memory for the array, ... VARIANT_BOOL DetectCode(long* codeArray, short* arraySize) ... "Mattias Sjögren" wrote: ...
    (microsoft.public.dotnet.framework.interop)
  • Dynamic array
    ... i wanted that it automatically calcule the size of the array, ... Dim anotherIteration As Boolean ... Dim arraySize As Integer ...
    (microsoft.public.excel.programming)
  • Re: puzzle
    ... > the array appear exactly two times except one. ... Consider that if you choose a random number or the midpoint from the ... delete, delete// Changes arraySize ... Select the midpoint of the middle partition, or the last entry of the ...
    (comp.programming)
  • Re: Marshalling Array In/Out C++ COM Control
    ... I am having this exact problem. ... > using Marshal.AllocCoTaskMem to allocate memory for the array, ... > When the method is called, codeArray is intialized, and contains all 0's. ... > arraySize is the number of elements in codeArray. ...
    (microsoft.public.dotnet.framework.interop)