Re: Large number of files in single directory

From: Chris (redhat-list_at_dotcomdesigners.com)
Date: 05/25/05

  • Next message: Stephen Carville: "Re: Large number of files in single directory" 
    To: "General Red Hat Linux discussion list" <redhat-list@redhat.com>
Date: Wed, 25 May 2005 10:41:03 -0700

    Tom - forget the "by character" method... my message below has a one-liner
    you can use to do the same thing on very large directories. I'm trying to
    do something similar with tarring large directories, without having to
    resort to splitting things up into multiple files based on filenames (not
    always practical, especially if you try to automate things and don't know
    what the filenaming convention is... files may as well all start with the
    same character for all I know)

    Anyone?

    Chris

    ----- Original Message -----
    From: "Burke, Thomas G." <tg.burke@ngc.com>
    Sent: Wednesday, May 25, 2005 10:28 AM
    Subject: RE: Large number of files in single directory

    >I delete them by character... e.g. rm -rf *1.tmp, rm -rf *2.tmp, and so
    >on. Don't know of any other way to do it. - although I wrote a little C
    >program once to handle it for me.
    >
    > -Tom
    >
    > -----Original Message-----
    > From: redhat-list-bounces@redhat.com
    > Sent: Wednesday, May 25, 2005 1:19 PM
    > Subject: Large number of files in single directory
    >
    >
    >
    > There seems to be a filesystem limitation on most flavors of Linux I've
    > worked on, in terms of a max number of files in a single directory -
    > before
    > tools like tar, gzip, rm, mv, cp and others stop working properly. For
    > example, I have some users that have 2000+ files in a single directory
    > (some
    > as many as 10,000 files) and trying to tar these directories is always
    > coming up with "argument list too long."
    >
    > Is there a way for tar and these other tools to "see" all these files and
    > process them as normal? I recall once I had to resort to something like
    > "find . -print | xargs rm -fr" to remove thousands of files from a single
    > directory. Is doing something similar but replacing "rm" with "tar" the
    > only way to make this work, or does tar have some sort of command line
    > switch (I couldn't find one) to work with extremely long argument lists?
    >
    > Chris 

    -- 
redhat-list mailing list
unsubscribe mailto:redhat-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list
  • Next message: Stephen Carville: "Re: Large number of files in single directory"

    Relevant Pages

    • Re: Untarring / German umlaut characters
      ... create UTF-8 file names. ... did not explicitely use the format option. ... command line editing keys don't work as expected, the shells or terminal programs ignoring or misinterpreting character keys up to tools completely running berserk if they encounter a completely valid character. ... While tar tvf tarfile runs fine under certain settings, tar xvf tarfile finally files using the same settings. ...
      (comp.sys.mac.misc)
    • RE: Large number of files in single directory
      ... in terms of a max number of files in a single directory - ... > tools like tar, gzip, rm, mv, cp and others stop working properly. ... Do You Yahoo!? ... Mail has the best spam protection around ...
      (RedHat)
    • Re: [newbie] cp -r across volumes - making it carry on if it fails..
      ... And WHAT command line? ... character by character, do an ls, and pass the output thrugh od -a. ... > little trick using tar etc.. ... tar and rsync will do a cp foo bar1; ...
      (comp.os.linux.development.system)
    • Tar UTF-8 conversion error
      ... We're occasionally seeing the following errors when using tar ... under Solaris 9 and Solaris 10: ... UTF-8 conversion failed. ... and when it encounters a character in the filename ...
      (SunManagers)
    • Re: Versioning file system
      ... uses legal path character, but doesn't break anything. ... userspace tools will have to be taught that: ... by never actually showing .snapshot in directory listings, ... You never risked having tar ...
      (Linux-Kernel)