Re: [opensuse] experience with large number of SATA drives ?

On Mon, Feb 1, 2010 at 9:34 AM, Per Jessen <per@xxxxxxxxxxxx> wrote:
Roger Oberholtzer wrote:

On Mon, 2010-02-01 at 13:27 +0100, Per Jessen wrote:
I'm putting together a hardware proposal for a new project - I/O
rates are paramount, so I'm looking at running e.g. 16 SATA-II drives
on a
single box.  I have opted not to use SAS drives due to price and lack
of concurrency in the application.  I'm thinking of using two fairly
inexpensive Intel SATA (for RAID0) controllers, each with 8 SATA
each in a PCIe x8 slot.  Before I go and build the prototype, I was
just wondering if anyone's had any experience with running so many
SATA drives on a single mainboard?

The closest I have is 4 SATA. They are each receiving continuous
JPEG2000 image streams. The file system is XFS. I have had no trouble
at all. So that is at least one data point for you. Sorry I cannot say
anything about 16 disks. I think it must be more a question of what
the 16 disks will be doing more than that all are present.

In general terms, the box will be doing read data, process data, write
data on a small set of very large files striped across the disks. The
application throughput will be completely limited by the I/O rate.  The
disks will (hopefully) be very busy delivering a solid stream of data.

One concern I have is scalability - I easily picture a box with 4 or 6
drives delivering very good I/O rates, but I can't help wondering if
there is some ceiling somewhere that I'm going to hit if I just add
more and more drives.  There is of course a plain hardware/physical
limit, but 16 drives is still achieveable.
In selecting the controller(s), I'm looking at price per SATA port, and
I only need RAID0, but I have also been wondering about throughput,
e.g. is there any reason to think that a 4-port card might perform
better than an 8-port?


Per Jessen, Zürich (-0.8°C)

I'm most interested in the sequential vs. random i/o nature of your
workload. There is a huge difference in throughput. It can be as big
as 10x just on that one question.

Since you say you are working with large files, I'll assume sequential.

The fastest individual sata drives I've seen deliver speeds
approaching 100MB/sec (or 1 Gbit/sec) while doing pure sequential

So you seem to be hoping to achieve 1.6 GB/sec (16 Gbit/sec). That is
really pushing the limits and you are going to have to very carefully
select your components.

That should not be a problem for the "northbridge". Unfortunately
most PCIexpress controllers hang of the "southbridge". I think you
will find few if any southbridges that can run at 1.6 GB/sec. on the
Northbridge / southbridge interconnect.

If you do, then you have to look at the southbridge to PCIexpress bus.
I'd guess you will need multiple of those buses in order to
cumulatively handle 1.6 GB/sec.

I think some very high-end motherboards have multiple independent
PCIexpress buses in order to handle this sort of thing. I believe
that is what you will need.

At the end of the day, if your serious about building something like
this up you are going to have to start reading detailed throughput /
bus specs. And once you do I'd post directly on the kernel ide
mailing list to ask questions. They occasionally host discussions of
very high performance systems, although I don't recall anything
attempting to get 1.6 GB/sec out of the disk subsystem.

Post to "IDE/ATA development list" <linux-ide@xxxxxxxxxxxxxxx>. No
subscription required. Reply all is the standard for all kernel
lists, so you will be kept in copy of replies.

To unsubscribe, e-mail: opensuse+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse+help@xxxxxxxxxxxx