Writing ia32 programs with data larger than 4GB
From: John Fusco (fusco_john_at_yahoo.com)
Date: 20 Jun 2004 07:32:22 -0700
I have a 32-bit x86 Linux app that is running out of room for its
data. Currently the data resides in memory and we are running into
the 3GB user space limit. Actually, with dynamic memory this turns
out to be more like 2.5 GB of usable memory.
I've been asked to come up with a solution, and I am wondering if
there isn't a better solution, or perhaps the same solution available
as open source.
The data must reside in memory because swapping latency cannot be
tolerated. We have enough RAM to throw at the problem, we just need a
decent way to get at at.
My solution is to mmap and munmap the data as needed with a device
(such as /dev/shm). The data set, which can exceed 4GB, will be
persistent when it is not mapped by the application.
I have created a library so that the application programmers can use
this memory like ordinary dynamic memory. The difference is that they
must provide a 64-bit offset with each allocation since the data is
persistent after it is free'd.
Any thoughts are appreciated.