Writing ia32 programs with data larger than 4GB

From: John Fusco (fusco_john_at_yahoo.com)
Date: 06/20/04 

Date: 20 Jun 2004 07:32:22 -0700

I have a 32-bit x86 Linux app that is running out of room for its
data. Currently the data resides in memory and we are running into
the 3GB user space limit. Actually, with dynamic memory this turns
out to be more like 2.5 GB of usable memory.

I've been asked to come up with a solution, and I am wondering if
there isn't a better solution, or perhaps the same solution available
as open source.

The data must reside in memory because swapping latency cannot be
tolerated. We have enough RAM to throw at the problem, we just need a
decent way to get at at.

My solution is to mmap and munmap the data as needed with a device
(such as /dev/shm). The data set, which can exceed 4GB, will be
persistent when it is not mapped by the application.

I have created a library so that the application programmers can use
this memory like ordinary dynamic memory. The difference is that they
must provide a 64-bit offset with each allocation since the data is
persistent after it is free'd.

Any thoughts are appreciated.

Thanks,
John

Relevant Pages

  • linker script
    ... What is this linker script? ... I have gone through some .lds files in which the entire memory ... information or data resides in those sections? ...
    (comp.sys.arm)
  • linker script
    ... What is this linker script? ... I have gone through some .lds files in which the entire memory ... information or data resides in those sections? ...
    (comp.sys.arm)
  • Re: Dynamic Memory Settings Unavailable
    ... rather than the GUI, so I don't look at the GUI all often and I thought I ... But even if there was a 'dynamic memory' section on the dialog, ... Kalen Delaney, SQL Server MVP ...
    (microsoft.public.sqlserver.setup)
  • Re: Memory problem
    ... Then there is a dynamic memory allocation failure (the LIB ... I'd look to recode that and to check the error handling, as an allocation error should not have gotten as far as the free. ... Now I tend to assume it is my own code that harbors a bug, until and unless I can prove the bug exists elsewhere. ...
    (comp.os.vms)
  • Nokia N96 16GB -GSM- Cell Phone (Unlocked) Quad Band
    ... 16GB internal flash memory, plus microSD memory card slot (hot ... Approximate dynamic memory capacity indication with 16GB ... WCDMA HSDPA 900/2100 MHz with simultaneous voice and packet data ... slot class 11, max speed DL/UL: ...
    (talk.atheism)