Re: 'out of memory' errors
- From: tom arnall <kloro@xxxxxxx>
- Date: Sun, 30 Apr 2006 14:26:34 -0700
On Saturday 29 April 2006 04:22 pm, Linas ?virblis wrote:
tom arnall wrote:
I am trying to us a perl application that handles large files (.5GB) in
scalar variables. When I try to 'slurp' one of these files into a
variable (e.g., '$_ = `cat filename`) I get an out of memory error.
You should consider alternative solutions, like processing files line by
line for example. It highly depends on what you are trying to achieve,
but loading half a gig into memory is not something you should do
without having a very good reason for doing so.
Understood, but I am doing a regex on text units which span more than one
line. Trying to process this text w' line-by-line io would be possible but,
as far as I can see, very awkward.
is there not some way to get linux to use swap space when physical memory
is exceeded?
It does that by default, assuming you do have swap partitions or files
available. Note that this can be slow, and I assume this is what you
described as a "freeze".
Looking at 'top' and the 'MEM%' column, it seems to me that the system is
finally swapping, but only after allowing the perl applicationg to hog all
available memory, which then of course has everyone else swapping and
therefore grinding to a virtual halt. What I want is for system to be
swapping mostly for just the perl application.
There are many tunable Linux virtual memory parameters you can set via
sysctl interface. You can get a list of them with "sysctl -a | grep vm".
The most useful one is vm.swappiness. You can set it to a number from 0
to 100. 0 means "try not to use swap at all" and 100 - "swap as much as
possible" with all other values in between. You can set it (and other
sysctl options) like this:
# sysctl -w vm.swappiness=60
I tried setting it to 100, but i get an 'out of memory' error anyway. But
shouldn't 'limits.conf' also be tuned to deal with the situation? My current
settings:
# /etc/security/limits.conf
kloro hard rss 256000
kloro hard memlock 256000
kloro hard as 768000
The 'as' parameter, as set above, causes an 'out of memory' error when the
perl appl' runs. If I make the 'as' parameter to be very big (e.g., 2G), the
system allows the perl appl' to run but also to hog memory. To my way of
thinking, there should be a way to put a limit on the physical memory an
appl' is allowed to use, and to cause swap space to be used when that limit
is reached, i.e., a way to put a limit on the physical memory which the
individual application is allowed to use.
Thanks,
Tom Arnall
north spit, ca
--
To UNSUBSCRIBE, email to debian-user-REQUEST@xxxxxxxxxxxxxxxx
with a subject of "unsubscribe". Trouble? Contact listmaster@xxxxxxxxxxxxxxxx
- References:
- 'out of memory' errors
- From: tom arnall
- Re: 'out of memory' errors
- From: Linas Žvirblis
- 'out of memory' errors
- Prev by Date: howto start something 5 mins after boot
- Next by Date: DPI setting
- Previous by thread: Re: 'out of memory' errors
- Next by thread: bootsplash kernel patch
- Index(es):
Relevant Pages
|
