On Mon, Feb 1, 2010 at 9:34 AM, Per Jessen
Roger Oberholtzer wrote:
On Mon, 2010-02-01 at 13:27 +0100, Per Jessen wrote:
I'm putting together a hardware proposal for a new project - I/O rates are paramount, so I'm looking at running e.g. 16 SATA-II drives on a single box. I have opted not to use SAS drives due to price and lack of concurrency in the application. I'm thinking of using two fairly inexpensive Intel SATA (for RAID0) controllers, each with 8 SATA ports, each in a PCIe x8 slot. Before I go and build the prototype, I was just wondering if anyone's had any experience with running so many SATA drives on a single mainboard?
The closest I have is 4 SATA. They are each receiving continuous JPEG2000 image streams. The file system is XFS. I have had no trouble at all. So that is at least one data point for you. Sorry I cannot say anything about 16 disks. I think it must be more a question of what the 16 disks will be doing more than that all are present.
In general terms, the box will be doing read data, process data, write data on a small set of very large files striped across the disks. The application throughput will be completely limited by the I/O rate. The disks will (hopefully) be very busy delivering a solid stream of data.
One concern I have is scalability - I easily picture a box with 4 or 6 drives delivering very good I/O rates, but I can't help wondering if there is some ceiling somewhere that I'm going to hit if I just add more and more drives. There is of course a plain hardware/physical limit, but 16 drives is still achieveable. In selecting the controller(s), I'm looking at price per SATA port, and I only need RAID0, but I have also been wondering about throughput, e.g. is there any reason to think that a 4-port card might perform better than an 8-port?
/Per
-- Per Jessen, Zürich (-0.8°C)
I'm most interested in the sequential vs. random i/o nature of your
workload. There is a huge difference in throughput. It can be as big
as 10x just on that one question.
Since you say you are working with large files, I'll assume sequential.
The fastest individual sata drives I've seen deliver speeds
approaching 100MB/sec (or 1 Gbit/sec) while doing pure sequential
access.
So you seem to be hoping to achieve 1.6 GB/sec (16 Gbit/sec). That is
really pushing the limits and you are going to have to very carefully
select your components.
That should not be a problem for the "northbridge". Unfortunately
most PCIexpress controllers hang of the "southbridge". I think you
will find few if any southbridges that can run at 1.6 GB/sec. on the
Northbridge / southbridge interconnect.
If you do, then you have to look at the southbridge to PCIexpress bus.
I'd guess you will need multiple of those buses in order to
cumulatively handle 1.6 GB/sec.
I think some very high-end motherboards have multiple independent
PCIexpress buses in order to handle this sort of thing. I believe
that is what you will need.
At the end of the day, if your serious about building something like
this up you are going to have to start reading detailed throughput /
bus specs. And once you do I'd post directly on the kernel ide
mailing list to ask questions. They occasionally host discussions of
very high performance systems, although I don't recall anything
attempting to get 1.6 GB/sec out of the disk subsystem.
Post to "IDE/ATA development list"