| Author |
Message |
Mark W Brehob
Guest
|
 Posted:
Mon Jan 10, 2005 3:07 pm Post subject:
point-to-point busses |
|
|
Hello all,
At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).
Here is the question:
What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?
Related:
What limits point-to-point bandwidth per pin? It it power, cost, or just
raw physical limits? Something else (transistor switching speed?)
What does the cost per signal pin look like? Put another way, at a
reasonable price, what is the bandwidth trend for point-to-point
communication?
I did a bit of a paper search (~1 hour) and turned up very little. Hot
Interconnects seemed to have a fair bit, but no real trends or predictions I
could find.
Mark |
|
| Back to top |
|
 |
Paul Rubin
Guest
|
| Posted:
Mon Jan 10, 2005 3:13 pm Post subject:
Re: point-to-point busses |
|
|
Mark W Brehob <brehob@wildwood.eecs.umich.edu> writes:
| Quote: | What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase?
|
I thought we already had 10 gbits or so. Is that surprising?
We have 600 gbit long haul optical, right? |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Jan 10, 2005 3:42 pm Post subject:
Re: point-to-point busses |
|
|
In article <NtsEd.73$UN1.65@news.itd.umich.edu>,
Mark W Brehob <brehob@wildwood.eecs.umich.edu> writes:
|>
|> At the University of Michigan we just had Robert Colwell (formally of Intel)
|> come in and give a talk. After the talk one issue came up which I found I
|> was disagreeing with everyone about (generally a sign I'm wrong of course).
Who said what?
|> Here is the question:
|> What is the expected trend in bits/second/wire over a point-to-point
|> connection (off-chip)? Do we expect to see a cap of just over a GHz or
|> will see a steady increase? And if an increase, where do we think the
|> number will be in 5 years?
An increase, but not necessarily a steady one. We are hitting some
physical limits, and don't know how many more can be resolved. No,
I can't explain what they are in detail, being a mediocre physicist.
|> Related:
|> What limits point-to-point bandwidth per pin? It it power, cost, or just
|> raw physical limits? Something else (transistor switching speed?)
Physical limits, but not raw ones.
For copper (electrical): leakage, crosstalk and noise generally.
The 10GE people are heavily into this.
For fibre (optical): making fast, integrated lasers for conversion.
We aren't yet at the fibre limits, though they will hit eventually.
|> What does the cost per signal pin look like? Put another way, at a
|> reasonable price, what is the bandwidth trend for point-to-point
|> communication?
Erratically upwards.
|> I did a bit of a paper search (~1 hour) and turned up very little. Hot
|> Interconnects seemed to have a fair bit, but no real trends or predictions I
|> could find.
For the reasons I give above. Every development needs solutions to
some nasty problems.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
David Wang
Guest
|
| Posted:
Mon Jan 10, 2005 4:40 pm Post subject:
Re: point-to-point busses |
|
|
Mark W Brehob <brehob@wildwood.eecs.umich.edu> wrote:
| Quote: | Hello all,
At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).
Here is the question:
What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?
Related:
What limits point-to-point bandwidth per pin? It it power, cost, or just
raw physical limits? Something else (transistor switching speed?)
What does the cost per signal pin look like? Put another way, at a
reasonable price, what is the bandwidth trend for point-to-point
communication?
I did a bit of a paper search (~1 hour) and turned up very little. Hot
Interconnects seemed to have a fair bit, but no real trends or predictions I
could find.
|
These trends or projections can be found at ITRS's web site.
You have to dig around to find what you need, but here's an example.
http://www.itrs.net/Common/2004Update/2004_000_ORTC.pdf
In table 3b (page 17 of 29), you can find the number of pads expected
for a microprocessor, the number signal I/O expected for a few years
down the road. Scroll down to the next page, you can find the expected
cost ranges per pin for expensive high frequency processors/ASICs as
well as commodity memory packaging parts.
Keep on scrolling and you can find more projected trends for cost
and frequency. So that should be what you're looking for.
--
davewang202(at)yahoo(dot)com |
|
| Back to top |
|
|
George William Herbert
Guest
|
| Posted:
Mon Jan 10, 2005 4:58 pm Post subject:
Re: point-to-point busses |
|
|
Paul Rubin <http://phr.cx@NOSPAM.invalid> wrote:
| Quote: | Mark W Brehob <brehob@wildwood.eecs.umich.edu> writes:
What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase?
I thought we already had 10 gbits or so. Is that surprising?
We have 600 gbit long haul optical, right?
|
Key word there is "optical".
Copper wires are less cooperative.
The current standard "fast" long haul
network is OC-192 at 10 Gbps.
We're in the 40 Gbps range for cutting
edge long-haul networks right now, with
160 Gbps per channel technology in test
but not in common use, and terabit-range
stuff in the labs and a few long distance
demos (using multiple slower channels).
-george william herbert
gherbert@retro.com |
|
| Back to top |
|
|
Pekka Pietikainen
Guest
|
| Posted:
Mon Jan 10, 2005 6:56 pm Post subject:
Re: point-to-point busses |
|
|
In article <41e26dd5$0$71989$c0de7616@dsl.net>, George William Herbert wrote:
| Quote: | I thought we already had 10 gbits or so. Is that surprising?
We have 600 gbit long haul optical, right?
Key word there is "optical".
Copper wires are less cooperative.
The current standard "fast" long haul
network is OC-192 at 10 Gbps.
If I understood things correctly, if you account price the game gets a bit |
more interesting. It wasn't too many years ago that the most cost-effective
way of transmitting lots (== more than Ethernet gives you) of bits short
distance (<100m) was copper (and just adding more pairs to get more
bandwidth), and even with fibre say 4*2.5Gbps (possibly using WDM, but even
with separate fibres) being more cost-effective than say 1*10Gbps. Long-haul
is a completely different game, sure, there you can get away with expensive
optics much more easily.
Not sure about the current economics, though :-) |
|
| Back to top |
|
|
Del Cecchi
Guest
|
| Posted:
Mon Jan 10, 2005 7:58 pm Post subject:
Re: point-to-point busses |
|
|
Mark W Brehob wrote:
| Quote: | Hello all,
At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).
Here is the question:
|
OK, right down my alley. Free consulting. Such a deal.
| Quote: | What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?
|
For moderate length interconnections (up to maybe 5 meters)
1 Gbit/second/pair is old news. Fibre Channel was at this rate a while
back. (1.0625 Gb/s)
2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express, Ethernet
CX4, and whatever they call double speed fibre channel)
5 Gb/second is coming soon. InfiniBand DDR, PCI-Express Gen 2, CEI from
OIF.
10 Gbit is coming not far behind.
| Quote: |
Related:
What limits point-to-point bandwidth per pin? It it power, cost, or just
raw physical limits? Something else (transistor switching speed?)
|
All of the above. Jitter, high frequency attenuation, gain bandwidth,
noise, materials, crosstalk, etc. Which one is the most significant
depends on the objectives, like distance and number of connections.
| Quote: |
What does the cost per signal pin look like? Put another way, at a
reasonable price, what is the bandwidth trend for point-to-point
communication?
|
Just the PHY level on an asic is pretty cheap and widely available now
at 2.5 Gbits (HyperTransport style or Serial) 5 Gb/s is coming
relatively soon. And really won't cost much more. But may be priced
higher. The trend is up. :-)
| Quote: |
I did a bit of a paper search (~1 hour) and turned up very little. Hot
Interconnects seemed to have a fair bit, but no real trends or predictions I
could find.
Mark
|
This is my specialty, more or less.
del cecchi
> |
|
| Back to top |
|
|
Paul Rubin
Guest
|
| Posted:
Mon Jan 10, 2005 8:14 pm Post subject:
Re: point-to-point busses |
|
|
Del Cecchi <cecchinospam@us.ibm.com> writes:
| Quote: | 2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
Ethernet CX4, and whatever they call double speed fibre channel)
|
PCI-express is an interconnect for up to 5 meters? I'd thought it was
for plug-in cards on a backplane. Hmmm. I've been wondering what low
cost alternatives there are for low-latency RDMA between computers in
a cluster. The raw bandwidth isn't as important as speed of small
transfers. Any advice? |
|
| Back to top |
|
|
Kai Harrekilde-Petersen
Guest
|
| Posted:
Mon Jan 10, 2005 8:38 pm Post subject:
Re: point-to-point busses |
|
|
nmm1@cus.cam.ac.uk (Nick Maclaren) writes:
| Quote: | In article <NtsEd.73$UN1.65@news.itd.umich.edu>,
Mark W Brehob <brehob@wildwood.eecs.umich.edu> writes:
|
|> At the University of Michigan we just had Robert Colwell (formally of Intel)
|> come in and give a talk. After the talk one issue came up which I found I
|> was disagreeing with everyone about (generally a sign I'm wrong of course).
Who said what?
|> Here is the question:
|> What is the expected trend in bits/second/wire over a point-to-point
|> connection (off-chip)? Do we expect to see a cap of just over a GHz or
|> will see a steady increase? And if an increase, where do we think the
|> number will be in 5 years?
|
3.125Gbit/s (for XAUI / 10G Ethernet) is already in reasonably wide
deployment and 10.3Gb/s (XFI) is coming along.
| Quote: | An increase, but not necessarily a steady one. We are hitting some
physical limits, and don't know how many more can be resolved. No,
I can't explain what they are in detail, being a mediocre physicist.
|> Related:
|> What limits point-to-point bandwidth per pin? It it power, cost, or just
|> raw physical limits? Something else (transistor switching speed?)
Physical limits, but not raw ones.
For copper (electrical): leakage, crosstalk and noise generally.
The 10GE people are heavily into this.
|
Errh, it's not the copper per see, that is the problem. It's the loss
of the crappy FR-4 PCB material that is the problem. In fact, cables
will quite often have better worst-case parameters than standard FR-4.
You can run 10-11Gbit/s (on a differential pair) for a short distance,
but don't put too many connectors, via transitions etc into that path
please. I'd limit it to less than 10 cm (4 inches).
I walked down the hall and talked to our resident high-speed guru, and
he said that even at 1G the loss-effects can be seen. He wouldn't
says whether the skin-effect loss or the dielectric loss is dominant,
as it depends too much on the actual PCB fabricated. Due to the small
feature sizes these days, a differential strip-line pair can see pure
resin in the horizontal plane and mostly glass in the vertical,
yielding two different propagation delays in the X and Y axis,
resulting in non-TEM waves.
| Quote: | |> What does the cost per signal pin look like? Put another way, at a
|> reasonable price, what is the bandwidth trend for point-to-point
|> communication?
|
Depends on your perspective. Having pre-emphasis, equalizing, SERDES,
CMU, PLLs, encoding (apparently 8B/10B is everyones favorite these
days), and protocol machine can add quite a burden on a chip.
Regards,
Kai
--
Kai Harrekilde-Petersen <khp(at)harrekilde(dot)dk> |
|
| Back to top |
|
|
Tim Clacy
Guest
|
| Posted:
Mon Jan 10, 2005 8:40 pm Post subject:
Re: point-to-point busses |
|
|
Mark W Brehob wrote:
| Quote: | Hello all,
At the University of Michigan we just had Robert Colwell (formally of
Intel) come in and give a talk. After the talk one issue came up
which I found I was disagreeing with everyone about (generally a sign
I'm wrong of course).
Here is the question:
What is the expected trend in bits/second/wire over a
point-to-point connection (off-chip)? Do we expect to see a cap
of just over a GHz or will see a steady increase? And if an
increase, where do we think the number will be in 5 years?
Related:
What limits point-to-point bandwidth per pin? It it power, cost,
or just raw physical limits? Something else (transistor switching
speed?)
What does the cost per signal pin look like? Put another way, at a
reasonable price, what is the bandwidth trend for point-to-point
communication?
|
Here's an interesting paper on RF interconnect that proposes data rates of
100Gbps per line (and, therfore, up to 20Tbps for 200 lines). The paper
focuses on inter-chip and intra-chip interconnect but should work equally
well over much larger distances. |
|
| Back to top |
|
|
Tim Clacy
Guest
|
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Mon Jan 10, 2005 9:18 pm Post subject:
Re: point-to-point busses |
|
|
In article <uy8f11eh0.fsf@harrekilde.dk>,
Kai Harrekilde-Petersen <khp@harrekilde.dk> writes:
|> >
|> > |> Related:
|> > |> What limits point-to-point bandwidth per pin? It it power, cost, or just
|> > |> raw physical limits? Something else (transistor switching speed?)
|> >
|> > Physical limits, but not raw ones.
|> >
|> > For copper (electrical): leakage, crosstalk and noise generally.
|> > The 10GE people are heavily into this.
|>
|> Errh, it's not the copper per see, that is the problem. It's the loss
|> of the crappy FR-4 PCB material that is the problem. In fact, cables
|> will quite often have better worst-case parameters than standard FR-4.
Well, I said that I didn't know the details :-)
The sort of complications you refer to are why I don't believe that
we will see a steady trend. Every time the speed is ramped a notch,
a new set of problems will stick up and have to be hammered down
again. Some will be easy to deal with; others won't.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
Del Cecchi
Guest
|
| Posted:
Tue Jan 11, 2005 12:28 am Post subject:
Re: point-to-point busses |
|
|
Paul Rubin wrote:
| Quote: | Del Cecchi <cecchinospam@us.ibm.com> writes:
2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
Ethernet CX4, and whatever they call double speed fibre channel)
PCI-express is an interconnect for up to 5 meters? I'd thought it was
for plug-in cards on a backplane. Hmmm. I've been wondering what low
cost alternatives there are for low-latency RDMA between computers in
a cluster. The raw bandwidth isn't as important as speed of small
transfers. Any advice?
|
If it isn't 5 meters it soon will be. It isn't the physical, it's the
logical stuff for switches et al.
What does the InfiniBand stuff cost? The switches seem reasonable
unless it is personal money. 4X stuff from one of the three hardware
companies would do the job. Plugs into PCI-X.
Or there is the HTX to IB that plugs into hypertransport. Pathscale,
right Greg? Should be motherboards with HTX connectors out Real Soon
Now. http://www.pathscale.com/pdf/PathScaleInfiniPath.pdf
In fact that seems the most attractive solution if the price is
reasonable. Couldn't get a price on the website.
del cecchi (clearly personal opinions)
del cecchi
del cecchi |
|
| Back to top |
|
|
del cecchi
Guest
|
| Posted:
Tue Jan 11, 2005 7:56 am Post subject:
Re: point-to-point busses |
|
|
"Paul Rubin" <http://phr.cx@NOSPAM.invalid> wrote in message
news:7xk6qlz57p.fsf@ruckus.brouhaha.com...
| Quote: | Del Cecchi <cecchinospam@us.ibm.com> writes:
2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
Ethernet CX4, and whatever they call double speed fibre channel)
PCI-express is an interconnect for up to 5 meters? I'd thought it was
for plug-in cards on a backplane. Hmmm. I've been wondering what low
cost alternatives there are for low-latency RDMA between computers in
a cluster. The raw bandwidth isn't as important as speed of small
transfers. Any advice?
|
Yep, roll your own. Xilinx sells several FPGAs containing high speed
serial I/O physical interfaces. What is it you want to attach to what?
If it is X86, the best current solution that I know would be an opteron
with HTX and InfiniPath from Pathscale, except you can't get it quite
yet. It is still in "early availability" mode according to their
website. And I don't know about the availability of HTX on Motherboards
either.
Otherwise, does anyone make PC with PCI-Express ports, except for
graphics? Or one of the IB vendors as I mentioned before should be
affordable.
del cecchi |
|
| Back to top |
|
|
Stephen Fuld
Guest
|
| Posted:
Tue Jan 11, 2005 7:56 am Post subject:
Re: point-to-point busses |
|
|
"del cecchi" <dcecchi.nojunk@att.net> wrote in message
news:34h03jF49t3vaU1@individual.net...
| Quote: |
"Paul Rubin" <http://phr.cx@NOSPAM.invalid> wrote in message
news:7xk6qlz57p.fsf@ruckus.brouhaha.com...
Del Cecchi <cecchinospam@us.ibm.com> writes:
2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
Ethernet CX4, and whatever they call double speed fibre channel)
PCI-express is an interconnect for up to 5 meters? I'd thought it was
for plug-in cards on a backplane. Hmmm. I've been wondering what low
cost alternatives there are for low-latency RDMA between computers in
a cluster. The raw bandwidth isn't as important as speed of small
transfers. Any advice?
Yep, roll your own. Xilinx sells several FPGAs containing high speed
serial I/O physical interfaces. What is it you want to attach to what?
If it is X86, the best current solution that I know would be an opteron
with HTX and InfiniPath from Pathscale, except you can't get it quite
yet. It is still in "early availability" mode according to their
website. And I don't know about the availability of HTX on Motherboards
either.
Otherwise, does anyone make PC with PCI-Express ports, except for
graphics? Or one of the IB vendors as I mentioned before should be
affordable.
|
Questions out of ignorance. Is a PCI Express port for graphics and
different from any other PCI express port? Since the OP talked about a
database server, he probably doesn't need much graphics so could he use any
old graphics thing and use the PCI Express port for his interconnect???
Also, I thought that you could connect two PCI Express ports together (on
different motherboards) for a high speed interconnect, but to have more than
two, you needed the Advanced Switching extensions, which aren't quite cooked
yet. Is that right?
--
- Stephen Fuld
e-mail address disguised to prevent spam |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in