Re: Max Files Per Directory

From: Tony Nelson (tonynelson_at_georgeanelson.com)
Date: 07/16/05

  • Next message: Timothy Murphy: "Re: WARNING:DO NOT UPGRADE TO CORE 4" 
    Date: Sat, 16 Jul 2005 14:22:51 -0400
To: For users of Fedora Core releases <fedora-list@redhat.com>

    At 8:53 AM -0400 7/16/05, Matthew Miller wrote:
    >On Sat, Jul 16, 2005 at 08:04:22AM -0400, fredex wrote:
    >> Now, for performance reasons, it is often not a good idea ot have
    >> many thousands of files in a single directory. As the number of files
    >> grows large the time it takes to access a file grows larger. I haven't
    >> looked into this on any linux file system, but on other unixes I've
    >> observed delays reaching up into the whole-second region when many thousands
    >> of files are in a single directory.
    >
    >Shouldn't be a problem on a new install of ext3 on modern Linux. See
    ><http://lwn.net/Articles/11481/>.

    I suppose the simple test is to try it. Time (microseconds) per file to run
    a script that creates files by touching them, to ls a few near the end, and
    to remove all the files; see script at end of transcript.

    #files touch ls rm
    ------ ----- ----- -----
      1000 309 016 054
     10000 331 008 051
    100000 326 007 357
    200000 330 007 1065

    I didn't want to try a million files. Creation and listing seem to be
    linear with the number of files, but rm seems quadratic. I think this
    indicates that in my current 2.6.12 FC3 kernel using Ext3 the directory
    data structure is still a list and not a tree.

    I expect that some sort of directory hierarchy to limit the number of files
    per directory would still be a win.

    Transcript follows. Max line width 114 chars.

    ---------- cut here ----------
    [tonyn@localhost ~]$ time ./manyfilestest 1000

    real 0m0.309s
    user 0m0.066s
    sys 0m0.237s
    [tonyn@localhost ~]$ time ls manyfiles/99*
    manyfiles/99 manyfiles/991 manyfiles/993 manyfiles/995 manyfiles/997 manyfiles/999
    manyfiles/990 manyfiles/992 manyfiles/994 manyfiles/996 manyfiles/998

    real 0m0.016s
    user 0m0.007s
    sys 0m0.004s
    [tonyn@localhost ~]$ time rm -rf manyfiles

    real 0m0.054s
    user 0m0.000s
    sys 0m0.049s
    [tonyn@localhost ~]$ time ./manyfilestest 10000

    real 0m3.312s
    user 0m0.728s
    sys 0m2.518s
    [tonyn@localhost ~]$ time ls manyfiles/999*
    manyfiles/999 manyfiles/9991 manyfiles/9993 manyfiles/9995 manyfiles/9997 manyfiles/9999
    manyfiles/9990 manyfiles/9992 manyfiles/9994 manyfiles/9996 manyfiles/9998

    real 0m0.075s
    user 0m0.051s
    sys 0m0.021s
    [tonyn@localhost ~]$ time rm -rf manyfiles

    real 0m0.519s
    user 0m0.006s
    sys 0m0.501s
    [tonyn@localhost ~]$ time ./manyfilestest 100000

    real 0m32.561s
    user 0m7.494s
    sys 0m24.285s
    [tonyn@localhost ~]$ time ls manyfiles/9999*
    manyfiles/9999 manyfiles/99991 manyfiles/99993 manyfiles/99995 manyfiles/99997 manyfiles/99999
    manyfiles/99990 manyfiles/99992 manyfiles/99994 manyfiles/99996 manyfiles/99998

    real 0m0.686s
    user 0m0.513s
    sys 0m0.162s
    [tonyn@localhost ~]$ time rm -rf manyfiles

    real 0m35.653s
    user 0m0.082s
    sys 0m5.561s
    [tonyn@localhost ~]$ time ./manyfilestest 200000

    real 1m6.031s
    user 0m15.243s
    sys 0m47.962s
    [tonyn@localhost ~]$ time ls manyfiles/9999*
    manyfiles/9999 manyfiles/99991 manyfiles/99993 manyfiles/99995 manyfiles/99997 manyfiles/99999
    manyfiles/99990 manyfiles/99992 manyfiles/99994 manyfiles/99996 manyfiles/99998

    real 0m1.459s
    user 0m1.113s
    sys 0m0.269s
    [tonyn@localhost ~]$ time rm -rf manyfiles

    real 3m32.851s
    user 0m0.159s
    sys 0m12.205s
    [tonyn@localhost ~]$ cat manyfilestest
    #!/bin/bash

    # Make lots of files in a directory.

    TESTDIR="manyfiles"
    MAXFILES=${1:-100000}

    mkdir $TESTDIR
    cd $TESTDIR
    for (( i=0; i*10<$MAXFILES; i=i+1 )) ; do touch ${i}0 ${i}1 ${i}2 ${i}3 ${i}4 ${i}5 ${i}6 ${i}7 ${i}8 ${i}9 ; done
    [tonyn@localhost ~]$
    ---------- cut here ----------
    ____________________________________________________________________
    TonyN.:' <mailto:tonynelson@georgeanelson.com>
          ' <http://www.georgeanelson.com/> 

    -- 
fedora-list mailing list
fedora-list@redhat.com
To unsubscribe: http://www.redhat.com/mailman/listinfo/fedora-list
  • Next message: Timothy Murphy: "Re: WARNING:DO NOT UPGRADE TO CORE 4"