Re: need fastest way to write 2gig array to disk file

From: Basile Starynkevitch [news] (basile-news_at_starynkevitch.net)
Date: 09/10/05 

Date: Sat, 10 Sep 2005 06:54:52 +0000 (UTC)

On 2005-09-09, Eric Taylor <et1@rocketship1.com> wrote:
> I have a two gigabtye array. From a nothing special C program
> what is the fastest way to write this to a disk file.

You might consider using mmap on this array. Basically, instead of
allocating the array with malloc, you open a file for writing, then
mmap it, then msync; something like (untested code, and you really
should add all the error checks)

// the array size in byte has to be a multiple of the page size
#define BIGSIZE 16384*1024 /* number of doubles in your array */
int fd = open("your_big_file", O_RDWR|O_CREAT, 0640);
if (fd<0) errored("failed to open");
size_t arraysize = sizeof(double)*BIGSIZE;
// grow the file if needed by truncating it
ftruncate(fd, arraysize);
// map the array in memory
void* arrayaddr = mmap((void*)0, arraysize, PROT_READ|PROT_WRITE,
                MAP_SHARED, fd, (off_t)0);
if (arrayaddr==MAP_FAILED) errored("failed to mmap");
double* array = arrayaddr;
// compute & fill the array
compute_your_big_array(array, BIGSIZE);
// sync the array to disk
msync(ad, arraysize, MS_ASYNC);
// unmap memory and close file
munmap(arrayaddr, arraysize);
close(fd);

Read a good book on Linux or Posix system programming, and the man pages for
open(2), mmap(2), msync(2), munmap(2), close(2) system calls

Regards 

-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/ 
email: basile(at)starynkevitch(dot)net 
8, rue de la Faïencerie, 92340 Bourg La Reine, France

Relevant Pages

  • Re: How to write Property Let for the single elements of an array
    ... the array) as well as the array size, an increment, and load it all ... Private pYValues() As Double ... Private ArrayIndex As Long ... Private ArraySize As Long ...
    (microsoft.public.excel.programming)
  • Re: Marshalling Array In/Out C++ COM Control
    ... using Marshal.AllocCoTaskMem to allocate memory for the array, ... VARIANT_BOOL DetectCode(long* codeArray, short* arraySize) ... "Mattias Sjögren" wrote: ...
    (microsoft.public.dotnet.framework.interop)
  • Dynamic array
    ... i wanted that it automatically calcule the size of the array, ... Dim anotherIteration As Boolean ... Dim arraySize As Integer ...
    (microsoft.public.excel.programming)
  • Re: Sorting a string array containing combinations
    ... simultaneously create another array which does not contain occurrences of x1 in it. ... Dim TempArrayAs Long ... Dim WithX(1 To Rows, ArraySize) As Long ... Dim CountWith As Long, CountWithout As Long ...
    (microsoft.public.excel.programming)
  • Re: puzzle
    ... > the array appear exactly two times except one. ... Consider that if you choose a random number or the midpoint from the ... delete, delete// Changes arraySize ... Select the midpoint of the middle partition, or the last entry of the ...
    (comp.programming)