Re: [RFC] Parallelize IO for e2fsck



On Thu, Jan 24, 2008 at 06:32:15PM +0100, Bodo Eggert wrote:
Alan Cox <alan@xxxxxxxxxxxxxxxxxxx> wrote:

I'd tried to advocate SIGDANGER some years ago as well, but none of
the kernel maintainers were interested. It definitely makes sense
to have some sort of mechanism like this. At the time I first brought
it up it was in conjunction with Netscape using too much cache on some
system, but it would be just as useful for all kinds of other memory-
hungry applications.

There is an early thread for a /proc file which you can add to your
poll() set and it will wake people when memory is low. Very elegant and
if async support is added it will also give you the signal variant for
free.

IMO you'll need a userspace daemon. The kernel does only know about the
amount of memory available / recommended for a system (or container),
while the user knows which program's cache is most precious today.

(Off cause the userspace daemon will in turn need the /proc file.)

I think a single, system-wide signal is the second-to worst solution: All
applications (or the wrong one, if you select one) would free their caches
and start to crawl, and either stay in this state or slowly increase their
caches again until they get signaled again. And the signal would either
come too early or too late. The userspace daemon could collect the weighted
demand of memory from all applications and tell them how much to use.

I don't think that's something that would require finetuning on a
per-application basis - the kernel should tell all applications once to
reduce memory consumption and write a fat warning to the logs (which
will on well-maintained systems be mailed to the admin).

Your "and tell them how much to use" wouldn't work for most applications
- e.g. I've worked the last weeks with a computer with 512 MB RAM and no
Swap, which means usually only 200 MB of free RAM. I've gotten quite
used to git aborting with "fatal: Out of memory, malloc failed" when
200 MB weren't enough for git, and I don't think there is any reasonable
way for git to reduce the memory usage while continuing to run.

In practice, there is a small number of programs that are both the
common memory hogs and should be able to reduce their memory consumption
by 10% or 20% without big problems when requested (e.g. Java VMs,
Firefox and databases come into my mind).

And from a performance point of view letting applications voluntarily
free some memory is better even than starting to swap.

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/



Relevant Pages

  • Re: Cached memory never gets released
    ... Stock linux 2.4.26 kernel. ... Due to flash bug 3M of memory gets lost due to font memory getting lost ... The output of "free" cache number steadily grows. ... longer to exhaust all of system memory with the cache. ...
    (Linux-Kernel)
  • Re: xmalloc string functions
    ... require memory allocations depending on the way the system works. ... If the toolkit being used is not one of those, then it is irrelevant that some provide a means to do so, particularly if the "some" are not available for the platform being targeted. ... Not enough context for most real-world applications to recover at this point. ... At this point g_malloccalling abortbecomes a moot point, particularly if your auto-save code is robust against memory allocation errors. ...
    (comp.lang.c)
  • Re: RFC: CONFIG_PAGE_SHIFT (aka software PAGE_SIZE)
    ... but now we waste more memory on small files.... ... allowing us to increase the fundamental block size of the filesystem. ... _whole_ kernel faster. ... If I want a inode cache, it always needs to be 8k based. ...
    (Linux-Kernel)
  • Re: Applications "buying" resources
    ... L4's approach for virtual memory is similar to yours: ... tasks ask their pager for memory, which the pager gives them, but he has ... into the kernel without involving the task ... the applications ...
    (comp.arch.embedded)
  • Re: IA64 Linux VM performance woes
    ... the server with over 8GByte memory. ... cache memory pages. ... Linux would have kernel parameter tuning interface. ...
    (Linux-Kernel)