Re: How to set up a Linux machine that occupies the minimum memory footprint ?
From: General Schvantzkoph (schvantzkoph_at_yahoo.com)
Date: 11/30/04
- Next message: Mark A. Odell: "Re: How to determine the size of cylinder?"
- Previous message: General Schvantzkoph: "Re: How to set up a Linux machine that occupies the minimum memory footprint ?"
- In reply to: Penang: "Re: How to set up a Linux machine that occupies the minimum memory footprint ?"
- Next in thread: ray: "Re: How to set up a Linux machine that occupies the minimum memory footprint ?"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Tue, 30 Nov 2004 08:31:24 -0500
On Mon, 29 Nov 2004 23:01:06 -0800, Penang wrote:
> Thanks for your reply.
>
> Jean-David Beyer <jdbeyer@exit109.com> wrote in message news:<10qm7bl82jrqk0f@corp.supernews.com>...
>
>> >>>>Anyway, is there not some limiting device in Linux to put a cap on the
>> >>>>mount of memory each user is allowed to use?
>
>> >>>Sure. But how do you know how much he is using? Plenty of it will be
>> >>>shared - is he using it or only 1/n'th of it? And then what about
>> >>>memory he has been promised but will never ask for? Do you count memory
>> >>>that is copy-on-write but has not been written to?
>
>> >>Agreed in general. However there must be some limitation of a 'user'
>> >>program that obtains then uses all available memory until the swap
>> >>partition is full. sort of imagine a program that calls gobs of
>> >>Mallocs, writes them with zeroes and keeps going until the kernel is
>> >>virtually paralysed.
>
>> >>If the user in question runs his rendering software in 'user' mode
>> >>with say a limit of 3.8G out of 4G RAM, then it can optimise its
>> >>functions around this rather than fondly imagining it has (say) 8G
>> >>memory and hence causes the kernel to 'thrash' thus slowing down the
>> >>application.
>
>> > I read the above with great interest, because it addresses the
>> > potential traps about the very task that I am about to commit.
>> > However, I must apologize for my own stupidity, because I do not fully
>> > comprehend what is stated above.
>
>> > Therefore, if there's anyone who can explain it to me - in simpler
>> > English, por favor, svp, please - I would be extremely grateful.
>
>> I think the poster is saying that if you have a process that asks for 8
>> GBytes of RAM on a machine with 4 GBytes, it will think it has got 8
>> GBytes of RAM, but in fact it will get at most 4, and the rest will be
>> paged. Now if the working set at any time is smaller than 4 GB, you can be
>> OK, but otherwise, the paging will probably kill you in terms of performance.
>
>> But if your program is written to request only a total of 4GB (including
>> its instructions and data), by itself it not need page much. The amount of
>> RAM needed to avoid paging problems is not the static size of the program,
>> but its working set, which is (rougly) the amount of memory it needs at
>> any one time. For example, if the program needs 3GB to initialize itself,
>> 3GB to run, and 3GB to finish off and exit, it does not have a 9GB working
>> set (which is what matters), but a 3GB working set. But unless your
>> program is really simple, it is difficult to determine its working set
>> other than by running it and watching it (with top, for example).
>
> The one thing that worries me the most is the potential of application
> crashing in the middle of a rendering job. A normal render job may
> take from hours to days, and it would kill me if it crashes on the nth
> day - and everything that it has done is gone forever.
>
> If there's a graceful way for the program to "exit" without a sudden
> crash - while have everything saved - then the delay wouldn't be too
> severe. I am looking for ways to achieve that, and in the meantime,
> I'm concentrate on looking for a uni-proc mobo that can handle at
> least 8GB of RAM, as insurance.
>
>
>> > The software in question is a beast - originally run under Solaris -
>> > and although we've managed to ported it to run under Linux, we haven't
>> > have the chance to throw everything on it, yet.
>
>> > True, we have successfully get it to render spnippets of
>> > not-so-very-detailed test runs, and because we haven't really tapped
>> > out it's full potential, I can't really tell you how hungry the beast
>> > can be, if we are to render something like what we are about to do.
>
>> > The 4GB figure is just a rough calculation - it could be more ! - and
>> > that's why I am worried about whether the entire thing could be
>> > achieved via the PC platform.
>
>> I am afraid there is no alternative to measuring it when actually running
>> with typical tasks and data.
>
> That's what I am facing. Because the rendering going to HUGE, I really
> have absolutely no idea what I am talking about, in terms of memory
> usage.
>
>
>> > About optimization - we thought about it, but unfortunately we neither
>> > have the talent nor the time, budget, etc to achieve anything
>> > meaningful at that front, so right now what I am doing is to bite the
>> > bullet and set up a working system and start throw everything at it.
>
>> > After reading the above quoted message, my main worry is that if I set
>> > up a system with 8GB, would it hinder the rendering job _MORE_ than a
>> > mere 4GB configuration ?
>
>> Probably not enough to notice. Not compared to paging on a 4GB machine.
>
> Just one idea: Anyone ever heard of a "hard-drive" made up of DRAM
> MODULES ? If the cost is reasonable, maybe I'd even try to put one
> 16GB-size "HD" made up of DRAM into the test machine - paging delay
> shouldn't be a huge problem if that's possible.
You can write your application to check point itself periodically.
- Next message: Mark A. Odell: "Re: How to determine the size of cylinder?"
- Previous message: General Schvantzkoph: "Re: How to set up a Linux machine that occupies the minimum memory footprint ?"
- In reply to: Penang: "Re: How to set up a Linux machine that occupies the minimum memory footprint ?"
- Next in thread: ray: "Re: How to set up a Linux machine that occupies the minimum memory footprint ?"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
