Re: Huge number of files in a directory

Thanks Andrew for the reply

Actually, the 'ls' in the directory isn't work, takes too much time and blow the load up high. The number of files is a projection based in the size of the directory, with the avg size of the files, that dont vary too much.

I have 2 SAS disk in RAID 1. Before this, all the data was in a single disk without RAID. When we make the migration to a powerfull server, with the new disks, it took 4 days to finish all the transfer, and it was performed about 1 year ago.

I didnt have a througput problem yet, not network, or even write or read. But I'm concerned about when this will start to botter me. More time I let to start to apply a solution, harder it will be to be acomplished.

I'll realy make a deep research before start to work and migrate anything. I just wanted t know if anyone has already have some similar problem, to share experiences.

Thanks again.

[]s
Fábio Jr.

Andrew Elliott escreveu:
The limit of the number of files in a directory is variable and set when the
FS is created. It depends on the size of the volume; if you do a 'df -i' it
will tell you the available inodes you have. This will help to determine
the limits of the fs which could help in determining where the bottleneck
is.

Quantity?...I've had problems doing an 'ls -al' on a single directory with
45000 files in it (EXT3 on external scsi array)...so I'm surprised that
you're not having problems already...

Suggestions? I've read that XFS and ReiserFS are the best fs types for
working with a large amount of small files, although the article was old
(2006). Reiser will consume more CPU than the others, but that's just from
my personal experience.

If you do find the number of files is a bottleneck, hardware is the easiest
fix. I'd recommend getting the fastest drives and bus that you can
afford...

I would definitely research the issue before doing anything about it...

Andrew.



-------
Hello,

I serve static content with an Apache server, and store the files in a storage server, wich is mounted in the webserver via NFS. 95% of the files that I serve are images, and the format of the file name is {number}.png.

I have these images all together in a single directory, and there was about 4 million files in this folder.

I wanto to change this directory structure to something more secure and dynamic, to permit an easier way to scale and backup these files.

My questions are:

- When the quantity of files will start to become a bottleneck for my filesystem? ( they are stored in partition with ext3)

- When the quantity of files will start to become a bottleneck for my OS?

- Suggestions?

Thanks

[]s
Fábio Jr.


--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@xxxxxxxxxx?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list

Relevant Pages

  • RE: Huge number of files in a directory
    ... I have 2 SAS disk in RAID 1. ... When we make the migration to a powerfull server, ... Quantity?...I've had problems doing an 'ls -al' on a single directory ... If you do find the number of files is a bottleneck, ...
    (RedHat)
  • RE: Huge number of files in a directory
    ... I have 2 SAS disk in RAID 1. ... When we make the migration to a powerfull server, ... If you do find the number of files is a bottleneck, ... I have these images all together in a single directory, ...
    (RedHat)
  • Re: What hardware components bottleneck Linux?
    ... >> disk arms, NIC. ... >I expect the bottleneck is going to be your internet connection. ... user sees if he hits your web server. ... running a server or have other obnoxious fine print. ...
    (comp.os.linux.hardware)
  • Re: How can i cluster sql 2005 for load balancing?
    ... If you have a disk I/O bottleneck, it could be that you're memory starved. ... data will still be managed by a single server. ... medium price hardware for my server. ...
    (microsoft.public.sqlserver.clustering)
  • SUMMARY: Disk contention
    ... I'm still convinced that it's not significant for big blocks (say 1 Mb I/O). ... S> due to the disk part. ... You can either use AT&T'ssfio library, or use Solaris 64-bit ... A significant bottleneck threshold occurs if b (processes ...
    (SunManagers)
Loading