Re: large files unnecessary trashing filesystem cache?

From: Badari Pulavarty (pbadari_at_gmail.com)
Date: 10/18/05

  • Next message: Rudolf Polzer: "Re: kernel allows loadkeys to be used by any user, allowing for local root compromise" 
    To: Guido Fiala <gfiala@s.netic.de>
Date:	Tue, 18 Oct 2005 13:48:04 -0700

    On Tue, 2005-10-18 at 22:01 +0200, Guido Fiala wrote:
    > (please note, i'am not subscribed to the list, please CC me on reply)
    >
    > Story:
    > Once in while we have a discussion at the vdr (video disk recorder) mailing
    > list about very large files trashing the filesystems memory cache leading to
    > unnecessary delays accessing directory contents no longer cached.
    >
    > This program and certainly all applications that deal with very large files
    > only read once (much larger than usual memory) - it happens that all other
    > cached blocks of the filessystem are removed from memory solely to keep as
    > much as possible of that file in memory, which seems to be a bad strategy in
    > most situations.
    >
    > Of course one could always implement f_advise-calls in all applications, but i
    > suggest a discussion if a maximum (configurable) in-memory-cache on a
    > per-file base should be implemented in linux/mm or where this belongs.
    >
    > My guess was, it has something to do with mm/readahead.c, a test limiting the
    > result of the function "max_sane_readahead(...) to 8 MBytes as a quick and
    > dirty test did not solve the issue, but i might have done something wrong.
    >
    > I've searched the archive but could not find a previous discussion - is this a
    > new idea?
    >
    > It would be interesting to discuss if and when this proposed feature could
    > lead to better performance or has any unwanted side effects.
    >
    > Thanks for ideas on that issue.

    Is there a reason why those applications couldn't use O_DIRECT ?

    Thanks,
    Badari

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  • Next message: Rudolf Polzer: "Re: kernel allows loadkeys to be used by any user, allowing for local root compromise"

    Relevant Pages

    • Re: readdir loses renamed files
      ... > slower, penalizing all other applications on your system, just because ... > memory, while the application calls readdir(). ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • large files unnecessary trashing filesystem cache?
      ... list about very large files trashing the filesystems memory cache leading to ... Of course one could always implement f_advise-calls in all applications, ... My guess was, it has something to do with mm/readahead.c, a test limiting the ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: Zeroed pages returned for heap
      ... >>also an application) to assume that the pages returned by the OS for heap ... > memory would seem to be making very dangerous assumptions. ... > pieces of memory to applications without erasing the old contents of the ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: Testing optimize-for-size suitability?
      ... >> Is there a benchmark or set of benchmarks that would allow me to test ... I haven't noticed any other improvement, but I guess that more memory ... I will also try to compile other applications, ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: xmalloc string functions
      ... require memory allocations depending on the way the system works. ... If the toolkit being used is not one of those, then it is irrelevant that some provide a means to do so, particularly if the "some" are not available for the platform being targeted. ... Not enough context for most real-world applications to recover at this point. ... At this point g_malloccalling abortbecomes a moot point, particularly if your auto-save code is robust against memory allocation errors. ...
      (comp.lang.c)