Re: How to set up a Linux machine that occupies the minimum memory footprint ?

From: Jean-David Beyer (jdbeyer_at_exit109.com)
Date: 11/29/04 

Date: Mon, 29 Nov 2004 08:00:35 -0500

Penang wrote:
> peterwn@parazzdise.net.nz (Peter) wrote in message news:<41aa6e35.4838763@news.paradise.net.nz>...
>
>
>>>In comp.os.linux.misc Peter <peterwn@parazzdise.net.nz> wrote:
>>>
>>>>Anyway, is there not some limiting device in Linux to put a cap on the
>>>>mount of memory each user is allowed to use?
>>>
>>>Sure. But how do you know how much he is using? Plenty of it will be
>>>shared - is he using it or only 1/n'th of it? And then what about
>>>memory he has been promised but will never ask for? Do you count memory
>>>that is copy-on-write but has not been written to?
>>>
>>>And so on.
>>>
>>
>>Agreed in general. However there must be some limitation of a 'user'
>>program that obtains then uses all available memory until the swap
>>partition is full. sort of imagine a program that calls gobs of
>>Mallocs, writes them with zeroes and keeps going until the kernel is
>>virtually paralysed.
>>
>>If the user in question runs his rendering software in 'user' mode
>>with say a limit of 3.8G out of 4G RAM, then it can optimise its
>>functions around this rather than fondly imagining it has (say) 8G
>>memory and hence causes the kernel to 'thrash' thus slowing down the
>>application.
>
>
>
>
> I read the above with great interest, because it addresses the
> potential traps about the very task that I am about to commit.
> However, I must apologize for my own stupidity, because I do not fully
> comprehend what is stated above.
>
> Therefore, if there's anyone who can explain it to me - in simpler
> English, por favor, svp, please - I would be extremely grateful.

I think the poster is saying that if you have a process that asks for 8
GBytes of RAM on a machine with 4 GBytes, it will think it has got 8
GBytes of RAM, but in fact it will get at most 4, and the rest will be
paged. Now if the working set at any time is smaller than 4 GB, you can be
OK, but otherwise, the paging will probably kill you in terms of performance.

But if your program is written to request only a total of 4GB (including
its instructions and data), by itself it not need page much. The amount of
RAM needed to avoid paging problems is not the static size of the program,
but its working set, which is (rougly) the amount of memory it needs at
any one time. For example, if the program needs 3GB to initialize itself,
3GB to run, and 3GB to finish off and exit, it does not have a 9GB working
set (which is what matters), but a 3GB working set. But unless your
program is really simple, it is difficult to determine its working set
other than by running it and watching it (with top, for example).
>
> The software in question is a beast - originally run under Solaris -
> and although we've managed to ported it to run under Linux, we haven't
> have the chance to throw everything on it, yet.
>
> True, we have successfully get it to render spnippets of
> not-so-very-detailed test runs, and because we haven't really tapped
> out it's full potential, I can't really tell you how hungry the beast
> can be, if we are to render something like what we are about to do.
>
> The 4GB figure is just a rough calculation - it could be more ! - and
> that's why I am worried about whether the entire thing could be
> achieved via the PC platform.

I am afraid there is no alternative to measuring it when actually running
with typical tasks and data.
>
> About optimization - we thought about it, but unfortunately we neither
> have the talent nor the time, budget, etc to achieve anything
> meaningful at that front, so right now what I am doing is to bite the
> bullet and set up a working system and start throw everything at it.
>
> After reading the above quoted message, my main worry is that if I set
> up a system with 8GB, would it hinder the rendering job _MORE_ than a
> mere 4GB configuration ?

Probably not enough to notice. Not compared to paging on a 4GB machine.
>
> Or is there any special switch that I need to know about - either
> compilation, or configuration, or both - that can minimize the
> headache ?
>
> Please help.
>
> Thank you ! 

-- 
   .~.  Jean-David Beyer           Registered Linux User 85642.
   /V\                             Registered Machine   241939.
  /( )\ Shrewsbury, New Jersey     http://counter.li.org
  ^^-^^ 07:50:00 up 37 days, 10:45, 4 users, load average: 4.28, 4.26, 4.19

Relevant Pages

  • Re: How to set up a Linux machine that occupies the minimum memory footprint ?
    ... Now if the working set at any time is smaller than 4 GB, ... > OK, but otherwise, the paging will probably kill you in terms of performance. ... > RAM needed to avoid paging problems is not the static size of the program, ... crashing in the middle of a rendering job. ...
    (comp.os.linux.setup)
  • Re: How to set up a Linux machine that occupies the minimum memory footprint ?
    ... Now if the working set at any time is smaller than 4 GB, ... > OK, but otherwise, the paging will probably kill you in terms of performance. ... > RAM needed to avoid paging problems is not the static size of the program, ... crashing in the middle of a rendering job. ...
    (comp.os.linux.misc)
  • Re: Lock memory ?
    ... paging kernel. ... the working set model is ... >> There is far less memory than hard disk space. ... >> You can write a device driver that does it. ...
    (microsoft.public.vc.mfc)
  • Re: why does minimizing my app cut mem usage by 90%?
    ... Trimming working set could make sense with smaller RAM sizes, ... It makes Windows paging even more sluggish. ... a limit, by expense of application memory, starving the applications? ...
    (microsoft.public.vc.mfc)
  • Re: Xp related to RAM?
    ... monitoring what gets written in the pagefile has hardly any indication ... of the amount of memory that's being used, ... when a working set is increased, ... in addition, if there is enough RAM, when there is a pagefault, the ...
    (microsoft.public.windowsxp.general)