[opensuse] experience with large number of SATA drives ?
I'm putting together a hardware proposal for a new project - I/O rates are paramount, so I'm looking at running e.g. 16 SATA-II drives on a single box. I have opted not to use SAS drives due to price and lack of concurrency in the application. I'm thinking of using two fairly inexpensive Intel SATA (for RAID0) controllers, each with 8 SATA ports, each in a PCIe x8 slot. Before I go and build the prototype, I was just wondering if anyone's had any experience with running so many SATA drives on a single mainboard? /Per -- Per Jessen, Zürich (-0.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Mon, 2010-02-01 at 13:27 +0100, Per Jessen wrote:
I'm putting together a hardware proposal for a new project - I/O rates are paramount, so I'm looking at running e.g. 16 SATA-II drives on a single box. I have opted not to use SAS drives due to price and lack of concurrency in the application. I'm thinking of using two fairly inexpensive Intel SATA (for RAID0) controllers, each with 8 SATA ports, each in a PCIe x8 slot. Before I go and build the prototype, I was just wondering if anyone's had any experience with running so many SATA drives on a single mainboard?
The closest I have is 4 SATA. They are each receiving continuous JPEG2000 image streams. The file system is XFS. I have had no trouble at all. So that is at least one data point for you. Sorry I cannot say anything about 16 disks. I think it must be more a question of what the 16 disks will be doing more than that all are present. -- Roger Oberholtzer OPQ Systems / Ramböll RST Ramböll Sverige AB Krukmakargatan 21 P.O. Box 17009 SE-104 62 Stockholm, Sweden Office: Int +46 10-615 60 20 Mobile: Int +46 70-815 1696 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Roger Oberholtzer wrote:
On Mon, 2010-02-01 at 13:27 +0100, Per Jessen wrote:
I'm putting together a hardware proposal for a new project - I/O rates are paramount, so I'm looking at running e.g. 16 SATA-II drives on a single box. I have opted not to use SAS drives due to price and lack of concurrency in the application. I'm thinking of using two fairly inexpensive Intel SATA (for RAID0) controllers, each with 8 SATA ports, each in a PCIe x8 slot. Before I go and build the prototype, I was just wondering if anyone's had any experience with running so many SATA drives on a single mainboard?
The closest I have is 4 SATA. They are each receiving continuous JPEG2000 image streams. The file system is XFS. I have had no trouble at all. So that is at least one data point for you. Sorry I cannot say anything about 16 disks. I think it must be more a question of what the 16 disks will be doing more than that all are present.
In general terms, the box will be doing read data, process data, write data on a small set of very large files striped across the disks. The application throughput will be completely limited by the I/O rate. The disks will (hopefully) be very busy delivering a solid stream of data. One concern I have is scalability - I easily picture a box with 4 or 6 drives delivering very good I/O rates, but I can't help wondering if there is some ceiling somewhere that I'm going to hit if I just add more and more drives. There is of course a plain hardware/physical limit, but 16 drives is still achieveable. In selecting the controller(s), I'm looking at price per SATA port, and I only need RAID0, but I have also been wondering about throughput, e.g. is there any reason to think that a 4-port card might perform better than an 8-port? /Per -- Per Jessen, Zürich (-0.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Before I go and build the prototype, I was just wondering if anyone's had any experience with running so many SATA drives on a single mainboard?
One concern I have is scalability - I easily picture a box with 4 or 6 drives delivering very good I/O rates, but I can't help wondering if there is some ceiling somewhere that I'm going to hit if I just add more and more drives. There is of course a plain hardware/physical limit, but 16 drives is still achieveable. In selecting the controller(s), I'm looking at price per SATA port, and I only need RAID0, but I have also been wondering about throughput, e.g. is there any reason to think that a 4-port card might perform better than an 8-port?
The good news is that I do have a server with 16 SATA disks - well 17 actually - and it does what I need. The bad news is that I'm not sure how much that helps you :) The machine is 5 years old so probably doesn't help your design decisions. It has a dual-xeon Tyan motherboard with an on board SATA controller - the 160 GB system disk is connected to that. It also has 2 off 3ware 9600 controllers, each with 8 250 GB disks. It has 2 GB memory and 4 off 1 Gb NICs (two on motherboard, two plugin). It basically just acts as an NFS server. There have been various disk failures over the years and we just had a processor fail. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Mon, Feb 1, 2010 at 9:34 AM, Per Jessen
Roger Oberholtzer wrote:
On Mon, 2010-02-01 at 13:27 +0100, Per Jessen wrote:
I'm putting together a hardware proposal for a new project - I/O rates are paramount, so I'm looking at running e.g. 16 SATA-II drives on a single box. I have opted not to use SAS drives due to price and lack of concurrency in the application. I'm thinking of using two fairly inexpensive Intel SATA (for RAID0) controllers, each with 8 SATA ports, each in a PCIe x8 slot. Before I go and build the prototype, I was just wondering if anyone's had any experience with running so many SATA drives on a single mainboard?
The closest I have is 4 SATA. They are each receiving continuous JPEG2000 image streams. The file system is XFS. I have had no trouble at all. So that is at least one data point for you. Sorry I cannot say anything about 16 disks. I think it must be more a question of what the 16 disks will be doing more than that all are present.
In general terms, the box will be doing read data, process data, write data on a small set of very large files striped across the disks. The application throughput will be completely limited by the I/O rate. The disks will (hopefully) be very busy delivering a solid stream of data.
One concern I have is scalability - I easily picture a box with 4 or 6 drives delivering very good I/O rates, but I can't help wondering if there is some ceiling somewhere that I'm going to hit if I just add more and more drives. There is of course a plain hardware/physical limit, but 16 drives is still achieveable. In selecting the controller(s), I'm looking at price per SATA port, and I only need RAID0, but I have also been wondering about throughput, e.g. is there any reason to think that a 4-port card might perform better than an 8-port?
/Per
-- Per Jessen, Zürich (-0.8°C)
I'm most interested in the sequential vs. random i/o nature of your
workload. There is a huge difference in throughput. It can be as big
as 10x just on that one question.
Since you say you are working with large files, I'll assume sequential.
The fastest individual sata drives I've seen deliver speeds
approaching 100MB/sec (or 1 Gbit/sec) while doing pure sequential
access.
So you seem to be hoping to achieve 1.6 GB/sec (16 Gbit/sec). That is
really pushing the limits and you are going to have to very carefully
select your components.
That should not be a problem for the "northbridge". Unfortunately
most PCIexpress controllers hang of the "southbridge". I think you
will find few if any southbridges that can run at 1.6 GB/sec. on the
Northbridge / southbridge interconnect.
If you do, then you have to look at the southbridge to PCIexpress bus.
I'd guess you will need multiple of those buses in order to
cumulatively handle 1.6 GB/sec.
I think some very high-end motherboards have multiple independent
PCIexpress buses in order to handle this sort of thing. I believe
that is what you will need.
At the end of the day, if your serious about building something like
this up you are going to have to start reading detailed throughput /
bus specs. And once you do I'd post directly on the kernel ide
mailing list to ask questions. They occasionally host discussions of
very high performance systems, although I don't recall anything
attempting to get 1.6 GB/sec out of the disk subsystem.
Post to "IDE/ATA development list"
Greg Freemyer wrote:
I'm most interested in the sequential vs. random i/o nature of your workload. There is a huge difference in throughput. It can be as big as 10x just on that one question.
Since you say you are working with large files, I'll assume sequential.
Yep, very much sequential.
The fastest individual sata drives I've seen deliver speeds approaching 100MB/sec (or 1 Gbit/sec) while doing pure sequential access.
So you seem to be hoping to achieve 1.6 GB/sec (16 Gbit/sec). That is really pushing the limits and you are going to have to very carefully select your components.
Something along those lines, yes. There is a price/performance balance to be struck somewhere of course.
That should not be a problem for the "northbridge". Unfortunately most PCIexpress controllers hang of the "southbridge". I think you will find few if any southbridges that can run at 1.6 GB/sec. on the Northbridge / southbridge interconnect.
If you do, then you have to look at the southbridge to PCIexpress bus. I'd guess you will need multiple of those buses in order to cumulatively handle 1.6 GB/sec.
I think some very high-end motherboards have multiple independent PCIexpress buses in order to handle this sort of thing. I believe that is what you will need.
Thanks, I'll have to check that out. I've not yet looked at any board in detail, although I did see one or two with one PCIe x16 slot that got "split" into two PCIe x8 with two cards plugged in.
At the end of the day, if your serious about building something like this up you are going to have to start reading detailed throughput / bus specs. And once you do I'd post directly on the kernel ide mailing list to ask questions. They occasionally host discussions of very high performance systems, although I don't recall anything attempting to get 1.6 GB/sec out of the disk subsystem.
Ok, thanks. DIdn't even know there was such a list. /Per -- Per Jessen, Zürich (-1.9°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi, Per Jessen wrote:
I'm putting together a hardware proposal for a new project - I/O rates are paramount, so I'm looking at running e.g. 16 SATA-II drives on a single box. I have opted not to use SAS drives due to price and lack of concurrency in the application. I'm thinking of using two fairly inexpensive Intel SATA (for RAID0) controllers, each with 8 SATA ports, each in a PCIe x8 slot. Before I go and build the prototype, I was just wondering if anyone's had any experience with running so many SATA drives on a single mainboard?
Yes, we do. Unfortunately I'm somewhat lacking exact HW references as the machines are down and 4000km away. But we have two types. One uses a RAID card with 16 SATA ports (we have 15 drives connected there), the other one has a mainboard with (iirc) 12 onboard ports. With disk sizes between 400 and 750GB those have RAID sizes (RAID5) between 3.6 and 7.4TB. We use them as backend for our high framerate camera system, they get filled via bonded GB Ethernet (~130MB/s sustained). FS is Reiser4. In all cases we don't use any HW RAID options, but also with the RAID cards only export the disks as JBOD and build a SW RAID on them. No real trouble with the number of disks itself except that the chance of failure is quite high using 'normal' desktop drives (for us the main reason to go SW RAID as then there are more chances to fix things). No idea what else to write - if you have (more) specific questions send me a mail... Cheers, Pit -- Dr. Peter "Pit" Suetterlin http://www.astro.su.se/~pit Institute for Solar Physics Tel.: +34 922 405 590 (Spain) P.Suetterlin@royac.iac.es +46 8 5537 8534 (Sweden) Peter.Suetterlin@astro.su.se -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Pit Suetterlin wrote:
Yes, we do. Unfortunately I'm somewhat lacking exact HW references as the machines are down and 4000km away. But we have two types. One uses a RAID card with 16 SATA ports (we have 15 drives connected there),
I have not yet come across one those. I've only seen 8-port sofar.
In all cases we don't use any HW RAID options, but also with the RAID cards only export the disks as JBOD and build a SW RAID on them.
I have still to make that decision, I think I'll have to benchmark it. Thanks Pit. /Per -- Per Jessen, Zürich (-2.1°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Pit Suetterlin wrote:
Yes, we do. Unfortunately I'm somewhat lacking exact HW references as the machines are down and 4000km away. But we have two types. One uses a RAID card with 16 SATA ports (we have 15 drives connected there),
I have not yet come across one those. I've only seen 8-port sofar.
Okay, I've seen the 16-port cards now. Adaptec and Highpoint. Thanks for the hint. -- Per Jessen, Zürich (-2.2°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Monday 01 February 2010, Per Jessen wrote:
Per Jessen wrote:
Pit Suetterlin wrote:
Yes, we do. Unfortunately I'm somewhat lacking exact HW references as the machines are down and 4000km away. But we have two types. One uses a RAID card with 16 SATA ports (we have 15 drives connected there),
I have not yet come across one those. I've only seen 8-port sofar.
Okay, I've seen the 16-port cards now. Adaptec and Highpoint. Thanks for the hint.
MSI has at least one MB with 4 PCIe16 slots. I looked at the 790FX-GD70. They also have some with just two if that's all you need. Mike -- Powered by SuSE 11.0 Kernel 2.6.25 KDE 3.5 Kmail 1.9 6:40pm up 8 days 5:42, 3 users, load average: 2.32, 2.25, 2.29 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Mike wrote:
On Monday 01 February 2010, Per Jessen wrote:
Per Jessen wrote:
Pit Suetterlin wrote:
Yes, we do. Unfortunately I'm somewhat lacking exact HW references as the machines are down and 4000km away. But we have two types. One uses a RAID card with 16 SATA ports (we have 15 drives connected there),
I have not yet come across one those. I've only seen 8-port sofar.
Okay, I've seen the 16-port cards now. Adaptec and Highpoint. Thanks for the hint.
MSI has at least one MB with 4 PCIe16 slots. I looked at the 790FX-GD70. They also have some with just two if that's all you need.
Mike
I have in fact been looking at exactly those MSI boards. K9A2 I think it is. /Per -- Per Jessen, Zürich (-2.4°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Mon, Feb 1, 2010 at 1:06 PM, Per Jessen
Mike wrote:
On Monday 01 February 2010, Per Jessen wrote:
Per Jessen wrote:
Pit Suetterlin wrote:
Yes, we do. Unfortunately I'm somewhat lacking exact HW references as the machines are down and 4000km away. But we have two types. One uses a RAID card with 16 SATA ports (we have 15 drives connected there),
I have not yet come across one those. I've only seen 8-port sofar.
Okay, I've seen the 16-port cards now. Adaptec and Highpoint. Thanks for the hint.
MSI has at least one MB with 4 PCIe16 slots. I looked at the 790FX-GD70. They also have some with just two if that's all you need.
Mike
I have in fact been looking at exactly those MSI boards. K9A2 I think it is.
/Per
Per, I got curious and looked up the PCIexpress 2.0 spec. http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_2.0 It came out 3 years ago, but it looks like a 16 channel PCI_Express_2.0 can handle 8 GB/sec (the article says a PCIe x16 graphics link (8 GB/s)). So if you can find motherboards that have PCI_express 2.0 support and Sata controllers with similar you may be able to keep up with your 16 drives. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Greg Freemyer wrote:
Per,
I got curious and looked up the PCIexpress 2.0 spec.
http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_2.0
It came out 3 years ago, but it looks like a 16 channel PCI_Express_2.0 can handle 8 GB/sec (the article says a PCIe x16 graphics link (8 GB/s)).
So if you can find motherboards that have PCI_express 2.0 support and Sata controllers with similar you may be able to keep up with your 16 drives.
Hi Greg, thanks for that - the motherboards seem to be widely available, but the controllers not quite so. I saw Highpoint had announced one, but only just September last year. /Per -- Per Jessen, Zürich (-1.1°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (6)
-
Dave Howorth -
Greg Freemyer -
Mike -
Per Jessen -
Pit Suetterlin -
Roger Oberholtzer