| Author |
Message |
Thomas Womack
Guest
|
 Posted:
Fri Oct 07, 2005 10:36 pm Post subject:
Re: How to build an infiniband cluster |
|
|
In article <3qo5laFfvtjkU1@individual.net>,
Del Cecchi <cecchinospam@us.ibm.com> wrote:
| Quote: | [Looking at Mellanox's page, is Infiniband DDR a 'real' Infiniband
standard with interoperable implementations of which only Mellanox's
is available today, or Mellanox's own way of running the serdes twice
as fast over the same cabling infrastructure?]
The 1.2 version of the IB spec defines both DDR and QDR, although
whether the doof that wrote chapter 6 did a good enough job for them to
be interoperable is open to speculation. I think a QDR 12X would be a
nice interconnect....
|
Indeed; though my head is hurting thinking how you'd get any
performance at all with small packets on an interconnect significantly
faster than most processor front-side buses. Give it one of the ports
on a very high-bandwidth memory system, I suppose, but I start to
wonder about the practicality of having several processors fill in
bits of a packet that all lie in the same cache line.
Or the single network chip connects on all four non-interconnect HT
ports of a quad Opteron -- but that means your receives and transmits
will end up interleaved across processors. Might work for the right
sort of spatially-decomposable problem, maybe, but I fear the problems
for which it works well are ones for which connecting single Opterons
to individual SDR 12x links would work just as well.
Cell's probably an answer -- now there's I/O and memory bandwidth in
the right kind of ballpark, and full employment for years for a horde
of signals-integrity people working on the system router chip.
Tom |
|
| Back to top |
|
 |
Del Cecchi
Guest
|
| Posted:
Sat Oct 08, 2005 12:15 am Post subject:
Re: How to build an infiniband cluster |
|
|
"Thomas Womack" <twomack@chiark.greenend.org.uk> wrote in message
news:QRt*u4B0q@news.chiark.greenend.org.uk...
| Quote: | In article <3qo5laFfvtjkU1@individual.net>,
Del Cecchi <cecchinospam@us.ibm.com> wrote:
[Looking at Mellanox's page, is Infiniband DDR a 'real' Infiniband
standard with interoperable implementations of which only Mellanox's
is available today, or Mellanox's own way of running the serdes twice
as fast over the same cabling infrastructure?]
The 1.2 version of the IB spec defines both DDR and QDR, although
whether the doof that wrote chapter 6 did a good enough job for them
to
be interoperable is open to speculation. I think a QDR 12X would be a
nice interconnect....
Indeed; though my head is hurting thinking how you'd get any
performance at all with small packets on an interconnect significantly
faster than most processor front-side buses. Give it one of the ports
on a very high-bandwidth memory system, I suppose, but I start to
wonder about the practicality of having several processors fill in
bits of a packet that all lie in the same cache line.
Or the single network chip connects on all four non-interconnect HT
ports of a quad Opteron -- but that means your receives and transmits
will end up interleaved across processors. Might work for the right
sort of spatially-decomposable problem, maybe, but I fear the problems
for which it works well are ones for which connecting single Opterons
to individual SDR 12x links would work just as well.
Cell's probably an answer -- now there's I/O and memory bandwidth in
the right kind of ballpark, and full employment for years for a horde
of signals-integrity people working on the system router chip.
Tom
|
As Tom Lehrer had Wehner Von Braun saying "I shoot them up, who knows
where they come down. That's not my department". :-)
Don't get confused. A 12X QDR is only 12GB/s/direction. HT3 at 5G and
2B wide is in the ballpark. |
|
| Back to top |
|
|
Del Cecchi
Guest
|
| Posted:
Sat Oct 08, 2005 12:15 am Post subject:
Re: How to build an infiniband cluster |
|
|
Thomas Womack wrote:
| Quote: | In article <2005100620570216807%wesley@felterorg>,
Wes Felter <wesley@felter.org> wrote:
Sun just released Infiniband products:
http://www.sun.com/products/networking/infiniband/index.html
It looks like they're mostly targeting 12X.
They've got a nice-looking 9-port 12x switch in 1U for $8500, and a
PCI-X 64bit/133MHz dual 4x HCA for $1000, with Solaris drivers for
Sparc and x86. This means you need 4x->12x cables for everything, a
snip at $150 per two metres if you buy from Sun, but you've got only
one model of switch for all the levels of your interconnect tree.
Does anyone produce 12x HCAs? 30Gbit/sec in each direction is quite a
lot of bandwidth, it doesn't _quite_ saturate PCI-E 16x but I'd be
impressed at the chipset that could get close to peak.
[Looking at Mellanox's page, is Infiniband DDR a 'real' Infiniband
standard with interoperable implementations of which only Mellanox's
is available today, or Mellanox's own way of running the serdes twice
as fast over the same cabling infrastructure?]
Tom
The 1.2 version of the IB spec defines both DDR and QDR, although |
whether the doof that wrote chapter 6 did a good enough job for them to
be interoperable is open to speculation. I think a QDR 12X would be a
nice interconnect....
--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.” |
|
| Back to top |
|
|
Wes Felter
Guest
|
| Posted:
Sat Oct 08, 2005 5:17 am Post subject:
Re: How to build an infiniband cluster |
|
|
On 2005-10-07 12:48:04 -0500, Thomas Womack
<twomack@chiark.greenend.org.uk> said:
| Quote: | Does anyone produce 12x HCAs? 30Gbit/sec in each direction is quite a
lot of bandwidth, it doesn't _quite_ saturate PCI-E 16x but I'd be
impressed at the chipset that could get close to peak.
|
IBM's eHCA (that Del just mentioned) supports 12x. Instead of PCI-E it
uses GX for lower overhead.
You might also guess that Sun is working on a 12x HCA.
--
Wes Felter - wesley@felter.org - http://felter.org/wesley/ |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in