| Author |
Message |
Guest
|
 Posted:
Sun Oct 02, 2005 10:17 pm Post subject:
How to build an infiniband cluster |
|
|
I am planning to build my own high performance 1u cluster out of
"commodity" components. I think I can choose all the components
(motherboard, processors, memory, disks, etc.) and order them on the
web.
My main question is about the infiniband components. Any advice on this
would be appreciated. Most importantly, how do I buy these components?
I am thinking of doing this work with a friend in a garage (so to
speak) and
most vendors would only talk to you if you are a large company.
Thanks,
Ed |
|
| Back to top |
|
 |
vincecate
Guest
|
| Posted:
Mon Oct 03, 2005 12:09 am Post subject:
Re: How to build an infiniband cluster |
|
|
I think that the infiniband card and software at pathscale.com is the
way to go for HPC. It works with Opteron boards that have the HTX
connector and any of the infiniband switches. Gives very low latency
times for MPI code. Best way to roll your own supercomputer.
http://pathscale.com/infinipath.php |
|
| Back to top |
|
|
Del Cecchi
Guest
|
| Posted:
Mon Oct 03, 2005 6:59 am Post subject:
Re: How to build an infiniband cluster |
|
|
"vincecate" <vincecate@gmail.com> wrote in message
news:1128280176.578558.103090@z14g2000cwz.googlegroups.com...
| Quote: |
I think that the infiniband card and software at pathscale.com is the
way to go for HPC. It works with Opteron boards that have the HTX
connector and any of the infiniband switches. Gives very low latency
times for MPI code. Best way to roll your own supercomputer.
http://pathscale.com/infinipath.php
That is a good way to go, very attractive arrangement. Do they have the |
full IB compatibility done? Otherwise you have to plug the IB into a
PCI/PCI-X slot. does that work with 1U servers? It will work with the
white box type.
del cecchi |
|
| Back to top |
|
|
Colonel Forbin
Guest
|
| Posted:
Mon Oct 03, 2005 8:15 am Post subject:
Re: How to build an infiniband cluster |
|
|
In article <1128273479.722178.237720@f14g2000cwb.googlegroups.com>,
<long_term_capital@yahoo.com> wrote:
| Quote: |
I am planning to build my own high performance 1u cluster out of
"commodity" components. I think I can choose all the components
(motherboard, processors, memory, disks, etc.) and order them on the
web.
My main question is about the infiniband components. Any advice on this
would be appreciated. Most importantly, how do I buy these components?
I am thinking of doing this work with a friend in a garage (so to
speak) and
most vendors would only talk to you if you are a large company.
|
You may want to look at appro.com. They have highly integrated
high density solutions with chassis-integrated InfiniBand, including
blade servers.
I have no personal experience with them, but they have some pretty
interesting looking products. |
|
| Back to top |
|
|
vincecate
Guest
|
| Posted:
Mon Oct 03, 2005 8:15 am Post subject:
Re: How to build an infiniband cluster |
|
|
Del Cecchi wrote:
| Quote: | That is a good way to go, very attractive arrangement. Do they have the
full IB compatibility done? Otherwise you have to plug the IB into a
PCI/PCI-X slot.
|
They say begining of Q4 (so any day now) they will have "Open IB":
http://pathscale.com/pdf/PathScaleInfiniPath.pdf
To get the low latency their card does not use PCI/PCI-X but the HTX
connector which goes directly to a hypertransport bus on the Opteron.
| Quote: | does that work with 1U servers?
|
A press release says SuperMicro's 4 socket (so 8-core) Opteron board
H8QC8/E is
"Optimized for 1U deployments" and has the HTX connector.
http://pathscale.com/pr_080805.html |
|
| Back to top |
|
|
Greg Lindahl
Guest
|
| Posted:
Mon Oct 03, 2005 10:49 pm Post subject:
Re: How to build an infiniband cluster |
|
|
In article <1128311346.247985.119340@g47g2000cwa.googlegroups.com>,
vincecate <vincecate@gmail.com> wrote:
| Quote: | does that work with 1U servers?
A press release says SuperMicro's 4 socket (so 8-core) Opteron board
H8QC8/E is
"Optimized for 1U deployments" and has the HTX connector.
http://pathscale.com/pr_080805.html
|
In addition to this SuperMicro board, which is due out fairly soon,
there is an already-shipping 2-socket motherboard from Iwill, which
works great in a 1U, plus there are several other motherboards in
the works.
-- greg
(employed by, not speaking for, PathScale) |
|
| Back to top |
|
|
tony.p.lee@gmail.com
Guest
|
| Posted:
Tue Oct 04, 2005 12:15 am Post subject:
Re: How to build an infiniband cluster |
|
|
Greg,
This looks interesting.
What are the advantage of cluster with IB instead of HTX directly?
It seems
like one more protocol layer.
Instead of latency for just HTX, it is latency for IB on top of HTX.
-Tony |
|
| Back to top |
|
|
David Kanter
Guest
|
| Posted:
Tue Oct 04, 2005 12:15 am Post subject:
Re: How to build an infiniband cluster |
|
|
tony.p.lee@gmail.com wrote:
| Quote: | Greg,
This looks interesting.
What are the advantage of cluster with IB instead of HTX directly?
It seems
like one more protocol layer.
Instead of latency for just HTX, it is latency for IB on top of HTX.
-Tony
|
You cannot go out of the box using Hypertransport. You can with
Infiniband. So I guess to answer you question, the main advantage is
that it is actually possible to do.
DK |
|
| Back to top |
|
|
Wes Felter
Guest
|
| Posted:
Tue Oct 04, 2005 7:09 am Post subject:
Re: How to build an infiniband cluster |
|
|
On 2005-10-02 12:17:59 -0500, long_term_capital@yahoo.com said:
| Quote: | I am planning to build my own high performance 1u cluster out of
"commodity" components. I think I can choose all the components
(motherboard, processors, memory, disks, etc.) and order them on the
web.
My main question is about the infiniband components. Any advice on this
would be appreciated. Most importantly, how do I buy these components?
I am thinking of doing this work with a friend in a garage (so to
speak) and
most vendors would only talk to you if you are a large company.
|
I've noticed the same thing. HP seems to be the only company that
treats IB as an off-the-shelf product rather than a custom solution.
Personally, I'm looking forward to buying Linksys IB gear at Fry's one
day. :-)
The Mellanox InfiniHost III Lx DDR HCA looks like it should have the
best price/performance... whenever it finally ships.
--
Wes Felter - wesley@felter.org - http://felter.org/wesley/ |
|
| Back to top |
|
|
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: How to build an infiniband cluster |
|
|
The price listed on their web site is $900 for single port, which is
too much.
You can get dual port IB adapters from other vendors for much less.
I may be comparing apples to oranges, but I am not really interested in
very low latency.
My main interest is in high bandwidth.
Again, I'm not sure I can buy this stuff because they refer me to their
resellers, who
typically sell you a whole cluster. I want to build my own cluster. |
|
| Back to top |
|
|
Guest
|
| Posted:
Tue Oct 04, 2005 4:15 pm Post subject:
Re: How to build an infiniband cluster |
|
|
But I can't buy the components from HP, can I?
The Mellanox InfiniHost III Lx DDR HCA is supposed to be under $100.
I didn't know it is not shipping... it is on their web site. They have
a more ambitious card there too: Mellanox InfiniHost III Ex.
They also refer you to resellers who build clusters. |
|
| Back to top |
|
|
Thomas Womack
Guest
|
| Posted:
Tue Oct 04, 2005 8:28 pm Post subject:
Re: How to build an infiniband cluster |
|
|
In article <3qg2e7Fesjk5U1@individual.net>,
Del Cecchi <cecchinospam@us.ibm.com> wrote:
| Quote: | Greg Lindahl wrote:
Meanwhile, the $900 number quoted for the PathScale InfiniPath HCA on
our website is a _list_ price. Our intention is for clusters built with
our HCA to have better price/performance than the competition. Remember
that our performance is as much as 15X faster* than the competition --
and the HCA is only a small part of the cost of a cluster.
-- greg
(employed by, not speaking for, PathScale.)
(* 128 byte MPI payload, streaming, vs Mellanox.)
Of course if one doesn't care about latency, there is 10g ethernet....
Although I don't know how much cheaper it is.
|
$900 is a _lot_ cheaper than Intel's 10G PXLA8591[SL]R cards, though I
suspect a lot of that saving comes from using copper rather than fibre
as the connection medium; an HP 24-port Infiniband 4x switch is $7000
if I believe froogle, I imagine the cables from the cards to the
switch are probably another few thousand dollars for a 24-node system. I
can't find pricing for 10Gbit Ethernet switches.
Ah. Intel has PXLA8591CX4 which is 10Gbit over 'CX4', and seems on some
dubious European tech sites to be advertised for EUR800 or so.
Tom |
|
| Back to top |
|
|
Greg Lindahl
Guest
|
| Posted:
Tue Oct 04, 2005 11:25 pm Post subject:
Re: How to build an infiniband cluster |
|
|
In article <1128375188.745224.161400@f14g2000cwb.googlegroups.com>,
tony.p.lee@gmail.com <tony.p.lee@gmail.com> wrote:
| Quote: | What are the advantage of cluster with IB instead of HTX directly?
It seems
like one more protocol layer.
Instead of latency for just HTX, it is latency for IB on top of HTX.
|
The normal path is this:
CPU -> NorthBridge -> PCI -> HCA
And for us it's just:
CPU -> HTX -> HCA
So there's one layer _less_. In addition, HTX is more efficient than
PCI express. And also note that we are using 16-bit HTX, while
Mellanox's chips generally use PCI express 8X, which is a bit less
than 1/2 the performance. This makes a big difference for medium-sized
packets.
-- greg |
|
| Back to top |
|
|
Greg Lindahl
Guest
|
| Posted:
Tue Oct 04, 2005 11:37 pm Post subject:
Re: How to build an infiniband cluster |
|
|
In article <1128441275.513441.259960@g49g2000cwa.googlegroups.com>,
<long_term_capital@yahoo.com> wrote:
| Quote: | The Mellanox InfiniHost III Lx DDR HCA is supposed to be under $100.
|
There has been a *lot* of confusion since there was an announcement a
while ago that the Mellanox SDR silicon was only $69. This number was
for just the chip, in quantity 10,000. Street prices for a complete
Mellanox-based HCA board are in the $400-$600 range, depending on
volume. The Mellanox DDR board is supposedly going to sell at a 30%
premium over SDR.
Meanwhile, the $900 number quoted for the PathScale InfiniPath HCA on
our website is a _list_ price. Our intention is for clusters built with
our HCA to have better price/performance than the competition. Remember
that our performance is as much as 15X faster* than the competition --
and the HCA is only a small part of the cost of a cluster.
-- greg
(employed by, not speaking for, PathScale.)
(* 128 byte MPI payload, streaming, vs Mellanox.) |
|
| Back to top |
|
|
Del Cecchi
Guest
|
| Posted:
Wed Oct 05, 2005 12:11 am Post subject:
Re: How to build an infiniband cluster |
|
|
Greg Lindahl wrote:
| Quote: | In article <1128441275.513441.259960@g49g2000cwa.googlegroups.com>,
long_term_capital@yahoo.com> wrote:
The Mellanox InfiniHost III Lx DDR HCA is supposed to be under $100.
There has been a *lot* of confusion since there was an announcement a
while ago that the Mellanox SDR silicon was only $69. This number was
for just the chip, in quantity 10,000. Street prices for a complete
Mellanox-based HCA board are in the $400-$600 range, depending on
volume. The Mellanox DDR board is supposedly going to sell at a 30%
premium over SDR.
Meanwhile, the $900 number quoted for the PathScale InfiniPath HCA on
our website is a _list_ price. Our intention is for clusters built with
our HCA to have better price/performance than the competition. Remember
that our performance is as much as 15X faster* than the competition --
and the HCA is only a small part of the cost of a cluster.
-- greg
(employed by, not speaking for, PathScale.)
(* 128 byte MPI payload, streaming, vs Mellanox.)
|
Of course if one doesn't care about latency, there is 10g ethernet....
Although I don't know how much cheaper it is. I guess you could use
1000baseT if you really wanted cheap....
Like the old saying about buying oats. Top quality costs money. Oats
that have been through the horse once already are cheaper.
--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.” |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in