Re: help with huge memory consumption??

On Sun, 4 Mar 2007 17:58:54 +0000 (UTC) Michel Talon <talon@xxxxxxxxxxxxxxxx> wrote:

| Robert Redelmeier <redelm@xxxxxxxxxxxxxxx> wrote:
|> Rainer Weikusat <rainer.weikusat@xxxxxxxxx> wrote in part:
|> > By definition, threads run in the same address space and each
|> > thread as access to the memory all other threads have access
|> > to, too (assuming it has a pointer pointing into first).
|>
|> If all the memory is marked as shared (with the exception of
|> the stacks) this puts a lot of memory pressure on the heap.
|> A special thread-safe heap-only malloc() may be needed.
|
| But precisely, modern mallocs are programmed to be thread friendly, for
| example by having several arenas for allocating memory, so that several
| threads can allocate concurrently without locking. I think this is the
| case for Linux, Solaris and FreeBSD.

One problem with threads is they are more likely to fragment the memory.
One reason is the concurrency where multiple threads are entangling in
time and interleaving their allocations (if in the same arena). If some
threads go away but others stay, then you have a jumbled collection of
allocated and free spaces in the heap. Compare that to a simple process
that extinguishes the whole virtual memory when it is done. And programs
designed with threads tend to be those that are long lived (a major desk
application) as opposed to really short ones (simple web CGI).

For that and other reasons, I avoid using threads as much as possible.
If it can be done reasonably well without threads, that's how I do it.

--
|---------------------------------------/----------------------------------|
| Phil Howard KA9WGN (ka9wgn.ham.org) / Do not send to the address below |
| first name lower case at ipal.net / spamtrap-2007-03-04-1404@xxxxxxxx |
|------------------------------------/-------------------------------------|
.

Relevant Pages

  • Re: [Announce]: Target_Core_Mod/ConfigFS and LIO-Target v3.0 work
    ... allocations would help the performance of a storage target. ... A single codepath memory allocating *AND* mapping for: ... Allocating multiple contigious struct page from the memory allocator ... I never claimed that RDMA is only possible from user space -- that was ...
    (Linux-Kernel)
  • Re: Forcing a Large Object Heap allocation.
    ... you should be passing the recommended 60% memory limit. ... that point and you should be experiencing an unstable app IMO. ... > compacts the heap. ... > objects in the regular heap causes poorer performance than allocating many ...
    (microsoft.public.dotnet.framework.performance)
  • Re: [00/41] Large Blocksize Support V7 (adds memmap support)
    ... some memory will be wasted. ... blocks with slab pages can be targetted and cleared if necessary. ... Allocating userpages from slab in 4k chunks with a 64k PAGE_SIZE is ... it remains a possibility to make the kernel more generic. ...
    (Linux-Kernel)
  • Re: ORACLE 9i - memory usage
    ... Oracle allocates 13GB (which is the total size of your SGA) at startup ... within physical memory. ... Can I guess it is actually allocating 13Gb of memory? ...
    (comp.databases.oracle.server)
  • Re: abnormal OOM killer message
    ... But I think it seems not to relate with stale data problem in compcache. ... My question was why last chance to allocate memory was failed. ... I think that last allocating memory should have succeeded. ...
    (Linux-Kernel)