Re: too many files....

On Fri, 21 Jul 2006 14:28:05 -0700, CptDondo wrote:

I seem to have gotten myself into a bit of a bind...

I have a website with lots of pictures. At the moment, some 5,000+ and
growing at the rate of about 300-500/month. Right now they take up 1.7 GB.

The problem is that access to those files is getting slower and slower
as the number of pictures gets larger. The server ain't exactly the
fastest, and the drives, while SCSI RAID5, are old and run at fairly
slow speeds.

Upgrading the hardware is not an option unless someone donates a bunch
of equipment.

Dividing the files into groups and directories is also not an option;
they are used in a website and all of the picture URLs are stored in an
SQL database. Dividing them now would be a huge nightmare.....

So... The pictures are stored on an ext3 filesystem. They range from
small (10K) to moderate (1MB) in size. Is there some way to optimize
this for faster access? Can I use a different filesystem? Set some
options for ext3? Tell linux to keep those pics in cache? Add more
ram? I am searching for things I can do on the cheap; this is my
personal server and money is tight at the moment.

The hardware is a dual PIII 1Ghz HP server, 1 GB RAM, 5 SCSI 36 GB
drives in a softRAID5 array (mostly 10K Seagates; there may be a 15K
Cheetah in there).

Thanks,

--Yan

The hardware seems fine to me. Raid5 could have been part of your
problem if your file set was changing a lot- but it is not. Is part of the
problem of slow access involve some SQL lookup? Is the server heavily
loaded by internet traffic? I'm just grasping at straws because I would
expect your server (hardware) to be fine serving that load.

I don't have a live internet server, but do have a similar hardware
setup (dual PIII 1GHz, with 1G RAM). It serves a much larger (by bytes)
set of Domino databases and its quite snappy on the LAN. Also, I am using
the XFS filesystem.

--
Douglas Mayne



.

Relevant Pages

  • Re: Anyone??? (was Reproducible data corruption on 6.1-Stable)
    ... pictures I have taken and noticed that a large number of pictures where ... marked as changed on the server. ... replaced with the complete new hardware and the problem was gone. ... I was told that it is some kind of known bug in older p4's ...
    (freebsd-stable)
  • Re: Anyone??? (was Reproducible data corruption on 6.1-Stable)
    ... pictures I have taken and noticed that a large number of pictures where ... marked as changed on the server. ... replaced with the complete new hardware and the problem was gone. ... I was told that it is some kind of known bug in older p4's ...
    (freebsd-stable)
  • Sun picture website..?
    ... Once upon a time Sun used to have a website with nothing but pictures of all ... their hardware... ... I used to use them for visio docs.. ...
    (SunManagers)
  • Website to serve a large number of images
    ... I would like to build a website that just serves images. ... i would like to be able to add pictures to a directory on the server ...
    (alt.html)
  • I want to publish all pictures and content of a website to the se.
    ... I want to publish all pictures and content of a website to the server I have. ... I would like to know how to publish my internet site. ...
    (microsoft.public.publisher.webdesign)