How to guarantee that an embedded (RealTime) application never causes a page fault

From: Alex van der Wal (nomailplease_at_hotmail.com)
Date: 05/09/05 

Date: Mon, 09 May 2005 14:11:05 +0200

Question
========
What solutions exist for locking an application in RAM that uses dynamic memory
allocation in such a way that no page fault occurs within the application after
real-time deadlines become active (e.g. after initialisation)?

Abstract
========
Our software system will consist of a number of executables. One (possibly two)
of these has real-time deadlines (< 5 milisecond latencies) and in order to meet
these deadlines, the RT application is not allowed to page memory to/from disk.

The other executables are allowed to page and will do so due to the limited
amount of physical memory (performance/cost trade off).

The RT application has the following characteristics:
- The application runs in user space.
- Memory usage has a practical ceiling, and it will consume no more than 50% of
the available physical memory.
- The application is multi-threaded, but all threads are started during
initialisation (e.g. before deadlines become real-time).
- The application is in C++ and makes extensive use of the STL template library.
As a result, quite a lot of dynamic memory allocation and deallocation takes
place during normal operation.

Initial investigations
======================
I've looked into possible solutions myself and will present my results first.
Note that all of this is theoretical. I have not yet had the time to verify any
of this.

Linux provides a method for de-activating paging for individual applications.
Simply run as root and add a call to mlockall(MCL_CURRENT | MCL_FUTURE); to the
application. However, this is not a complete solution since:
- Any threads started after the call receive a stack frame that may not yet be
resident in RAM resulting in one-time paging as the stack grows.
- Any memory that is allocated afterwards may not yet be resident in RAM.

The first issue can be resolved by calling mlockall() after all threads have
started, but the second is not that simple.

My first idea to solve the second issue was to implement some form of private
heap that is first made resident in RAM. This involved malloc()ing a heap and
then implementing private malloc() and free() operations that perform their work
within this private heap. I have not found any example of this and frankly the
idea seems too far fetched. Specially after I found out what sort of cleverness
malloc() uses internally.

After studying the glibc documentation, I've come up with a theoretical scenario
that handles the second problem. This is just an idea, so don't take my word for
it just yet.

Perform the following steps:

#define MB * 1024 * 1024
s_MyHeapSize = 30 MB; // Define my maximum memory usage
f_StartAllThreads(); // Start all threads

//
// Allocate my heap.
// Internally this should result in a single mmap() call
// Don't use new[] since we want to do a realloc later.
//
char *lp_MyHeap = (char*)malloc(s_MyHeapSize);

//
// Lock all pages and ensure that all is paged in RAM before the call returns
//
mlockall(MCL_CURRENT | MCL_FUTURE);

//
// Tweak the allocation subsystem so it does not internally call sbrk() to
// return freed heap memory to the kernel when free() or realloc() are called.
// Note that mallopt is undocumented in linux, but not in glibc.
//
mallopt(M_TRIM_THRESHOLD, s_MyHeapSize);

//
// Tweak the allocation subsystem to it uses one mmap() region only. This may
// decrease malloc() efficiency in a multi-threaded environment but may help
// to ensure that no new mmap() call is made whatever the circumstances.
// This is extremely preliminary, since I have no idea that the effects are.
//
mallopt(M_MMAP_MAX, 1);

//
// Free my heap memory, except for one byte. Hopefully this prevents the
// allocation subsystem from calling munmap() and/or sbrk()
//
char *lp_MyHeap2 = (char*)realloc((void*)lp_MyHeap, 1);

//
// Ensure that realloc reduced my initial heap in size and did
// not allocate a new block.
//
ASSERT(lp_MyHeap == lpMyHeap2);

At this point, any subsequent malloc() calls should receive a section from the
same mmap()ed block as the initial mp_MyHeap allocation. However, this is just a
threory. I don't fully understand the dynamics of the allocation subsystem (it's
not trivial) in the glibC library, so I might be dead wrong. Besides this trick
makes assumptions about the behavior of the allocation subsystem so there is no
way of telling that it will work forever.

Can anyone comment on my work so far and/or provide a good solution?
Thanks very much in advance for your time.

Best regards,

Alex

Relevant Pages

  • Re: Whats the difference between the heap and the freestore?
    ... >> In general, when discussing dynamically allocated memory, I hear people ... As of a couple of days ago, I though the heap ... > When you use the term heap in the context of memory allocation it annoys ... > without saying that they have to be allocated on a stack or anything else. ...
    (comp.lang.cpp)
  • Memory problems - WinDbg and SOS: Who recognizes this pattern?
    ... Last night I managed to get a memory dump using ADPlus and I analyzed it ... ephemeral segment allocation context: none ... Large object heap starts at 0x0a0d0030 ...
    (microsoft.public.dotnet.framework.performance)
  • Re: Inter-process communication
    ... of available RAM. ... As far as "tuning" more generally goes, you'll note that part of the tuning that goes on is IIS ensuring that it doesn't starve other processes of memory. ... more memory even while physical RAM has 1GB or more available. ... The first part of any memory allocation involves allocating a chunk of virtual address space, whether or not there is sufficient physical RAM to satisfy that allocation. ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: Tip Required on Dynamic Memory Allocation in Server Applications ...
    ... a memory allocation failure after 300 usages of the string ... Suggestions where given to pre-reserve the size of the heap, ... My program works untill 200-300 requests are ...
    (microsoft.public.win32.programmer.kernel)
  • Re: Competing criteria
    ... solution to "not heap allocation" is to define a reasonable maximum count ... memory allocation will not fail due to memory fragmentation, ... If you have some memory left over, you might want to switch to making a heap request when your preallocated buffers are exhausted, in this case since you are now allowed to fail, you are ok. ... To the degree that you can't preallocate as efficiently as you think you might be able to use the heap, consider that the cost for getting the guarantee. ...
    (comp.lang.c)