Re: too many files....

Dances With Crows wrote:

Dividing the files into groups and directories is not an option;
they are used in a website and all of the picture URLs are stored in
an SQL database. Dividing them now would be a huge nightmare.....

You screwed up. Fix it now, or fix it later; the pain will be less if
you fix it now. It's *never* a good idea to try to pack every file
into 1 directory if the number of files is going to grow and keep
growing.

I'm a victim of my own success.... I never thought there'd be more than about 1K images altogether. It's not until the website started slowing down that I actually did a count....

I'm going to try and create directories for new images, but there's an awful lot of scripts written around the images and realistically it will be a few months (and maybe another thousand or two pics) before I can devote the time to that.

SoftRAID 5 requires more CPU resources than softRAID-1 for obvious
reasons. That shouldn't be causing the problem that Dondo's reporting.
One of the work machines has a hardRAID-5 of 72G 7200 RPM SCSI disks,
and on 1 230G ext3 partition, it can open and read 1 100K file in a
directory containing 20,000 files in about 0.1 seconds. (This machine
is just a fileserver, though, not a web or database server.)

[Does] part of the problem of slow access involve some SQL lookup?

I would guess this is part of the problem. Make sure you're not doing
something dumb in the queries being used and the server-side whatever
(PHP? Ruby? CGI-Perl? *yuck* Java?) you're using.


PHP + SQL. And I know next to nothing about either one.... So yes, most likely it is SQL dragging the whole thing down. The SQL DB is stored on the same physical devices as the images, and that could also be a part of the problem.

I assume that I could do some sort of SQL magic to optimize the indexes to make retrieval faster....

--Yan
.

Relevant Pages

  • Re: Jebus Keerist Im Not Getting Paid Enough
    ... LEAST he's got to do a Misty shot, ... Convert proprietary images from '92-'06. ... Project Management and SQL dbas with clustering experience. ...
    (alt.2600)
  • Re: Storing Images and Files in SQL2000 with a project DB as front end.
    ... Does anyone have a sample code that I could use to achieve this (using VB ... Bitmaps can be significantly larger than .jpg since they don't have any ... Check the format of the ole objects stored in SQL Server from ... When I create SQL Reports I am not able to display the images. ...
    (microsoft.public.access.formscoding)
  • Re: Securing documents - SQL Server or FileSystem
    ... ADODB.Stream to present images that are not in the web folders. ... SQL Server MVP ... > security credentials opens a filestream to the file and writes it into ...
    (microsoft.public.inetserver.asp.db)
  • stored procedure xp_cmdshell
    ... I making an ASP.NET website were users can upload ... I was originally planning to have the images to SQL, ... I thought that inside my stored procedure, ...
    (microsoft.public.sqlserver.programming)
  • RE: Storing Images
    ... One drawback that I know of: These large images may use a lot of your SQL ... limited in the amount of hardware resources for your SQL server, ... How many images will you store? ...
    (microsoft.public.sqlserver.server)