Re: too many files....

On Fri, 21 Jul 2006 15:48:05 -0600, Douglas Mayne staggered into the
Black Sun and said:
On Fri, 21 Jul 2006 14:28:05 -0700, CptDondo wrote:
I have a website with lots of pictures. At the moment, some 5,000+
and growing at the rate of about 300-500/month. The problem is that
access to those files is getting slower and slower as the number of
pictures gets larger.

Dividing the files into groups and directories is not an option;
they are used in a website and all of the picture URLs are stored in
an SQL database. Dividing them now would be a huge nightmare.....

You screwed up. Fix it now, or fix it later; the pain will be less if
you fix it now. It's *never* a good idea to try to pack every file
into 1 directory if the number of files is going to grow and keep
growing.

So... The pictures are stored on an ext3 filesystem. They range from
small (10K) to moderate (1MB) in size. Is there some way to optimize
this for faster access? Can I use a different filesystem?

Directory lookup times are faster for ReiserFS and XFS than they are for
ext3, for various reasons. If you think you have to keep every file in
1 directory, and you don't want to fix your broken database/website,
back up everything on this partition and mkreiserfs (or mkxfs) it. If
using ReiserFS, make sure to use "-h tea" if you'll ever have more than
about 200,000 files in 1 directory.

The hardware is a dual PIII 1Ghz HP server, 1 GB RAM, 5 SCSI 36 GB
drives in a softRAID5 array
The hardware seems fine to me.

SoftRAID 5 requires more CPU resources than softRAID-1 for obvious
reasons. That shouldn't be causing the problem that Dondo's reporting.
One of the work machines has a hardRAID-5 of 72G 7200 RPM SCSI disks,
and on 1 230G ext3 partition, it can open and read 1 100K file in a
directory containing 20,000 files in about 0.1 seconds. (This machine
is just a fileserver, though, not a web or database server.)

[Does] part of the problem of slow access involve some SQL lookup?

I would guess this is part of the problem. Make sure you're not doing
something dumb in the queries being used and the server-side whatever
(PHP? Ruby? CGI-Perl? *yuck* Java?) you're using.

--
Matt G|There is no Darkness in Eternity/But only Light too dim for us to see
Brainbench MVP for Linux Admin / mail: TRAP + SPAN don't belong
http://www.brainbench.com / "He is a rhythmic movement of the
-----------------------------/ penguins, is Tux." --MegaHAL
.

Relevant Pages

  • slow drag and drop to CD
    ... I did not see any fix so I am reposting the problem. ... in the middle of the copying function. ... it is getting slower and ...
    (microsoft.public.windowsxp.basics)
  • [HPADM] Lousy system response times
    ... the response time for CDE gets slower and slower until ... the connection times out. ... To me this sounds like banging the TV to fix the picture. ...
    (HP-UX-Admin)
  • Re: Leopard and Office 2004
    ... well but its my impression that office programs like entourage and word ... That seemed to fix the slow launch problems. ... Now also is normal (although still a slower launch than a native application ...
    (comp.sys.mac.apps)
  • Re: Rib-tickler. Shediquack advice please
    ... seems that there's nothing that can be done to fix a broken rib - apart ... Growing old is mandatory; growing up is optional. ...
    (uk.rec.sheds)
  • Re: Rib-tickler. Shediquack advice please
    ... seems that there's nothing that can be done to fix a broken rib - apart ... Growing old is mandatory; growing up is optional. ... horrid dot squeak snailything zetnet point co period uk ...
    (uk.rec.sheds)