large files unnecessary trashing filesystem cache?

From: Guido Fiala (gfiala_at_s.netic.de)
Date: 10/18/05

  • Next message: Greg KH: "Re: 2.6.14-rc4-mm1 - drivers/serial/" 
    To: linux-kernel@vger.kernel.org
Date:	Tue, 18 Oct 2005 22:01:10 +0200

    (please note, i'am not subscribed to the list, please CC me on reply)

    Story:
    Once in while we have a discussion at the vdr (video disk recorder) mailing
    list about very large files trashing the filesystems memory cache leading to
    unnecessary delays accessing directory contents no longer cached.

    This program and certainly all applications that deal with very large files
    only read once (much larger than usual memory) - it happens that all other
    cached blocks of the filessystem are removed from memory solely to keep as
    much as possible of that file in memory, which seems to be a bad strategy in
    most situations.

    Of course one could always implement f_advise-calls in all applications, but i
    suggest a discussion if a maximum (configurable) in-memory-cache on a
    per-file base should be implemented in linux/mm or where this belongs.

    My guess was, it has something to do with mm/readahead.c, a test limiting the
    result of the function "max_sane_readahead(...) to 8 MBytes as a quick and
    dirty test did not solve the issue, but i might have done something wrong.

    I've searched the archive but could not find a previous discussion - is this a
    new idea?

    It would be interesting to discuss if and when this proposed feature could
    lead to better performance or has any unwanted side effects.

    Thanks for ideas on that issue.
    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  • Next message: Greg KH: "Re: 2.6.14-rc4-mm1 - drivers/serial/"

    Relevant Pages

    • Re: readdir loses renamed files
      ... > slower, penalizing all other applications on your system, just because ... > memory, while the application calls readdir(). ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: large files unnecessary trashing filesystem cache?
      ... > list about very large files trashing the filesystems memory cache leading to ... > My guess was, it has something to do with mm/readahead.c, a test limiting the ... Is there a reason why those applications couldn't use O_DIRECT? ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: Zeroed pages returned for heap
      ... >>also an application) to assume that the pages returned by the OS for heap ... > memory would seem to be making very dangerous assumptions. ... > pieces of memory to applications without erasing the old contents of the ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: Testing optimize-for-size suitability?
      ... >> Is there a benchmark or set of benchmarks that would allow me to test ... I haven't noticed any other improvement, but I guess that more memory ... I will also try to compile other applications, ... send the line "unsubscribe linux-kernel" in ...
      (Linux-Kernel)
    • Re: xmalloc string functions
      ... require memory allocations depending on the way the system works. ... If the toolkit being used is not one of those, then it is irrelevant that some provide a means to do so, particularly if the "some" are not available for the platform being targeted. ... Not enough context for most real-world applications to recover at this point. ... At this point g_malloccalling abortbecomes a moot point, particularly if your auto-save code is robust against memory allocation errors. ...
      (comp.lang.c)