| Author |
Message |
glen herrmannsfeldt
Guest
|
 Posted:
Tue Jan 11, 2005 7:56 am Post subject:
Re: point-to-point busses |
|
|
Mark W Brehob wrote:
(snip)
| Quote: | Here is the question:
What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?
Related:
What limits point-to-point bandwidth per pin? It it power, cost, or just
raw physical limits? Something else (transistor switching speed?)
|
I don't know the real answer, but pin inductance has been a problem
for a long time. Ground bounce due to the inductance of ground
pins and such.
I would guess that pin inductance would eventually limit
the pin bandwidth. Also driving current decreases as transistors
get faster.
-- glen |
|
| Back to top |
|
 |
Paul Rubin
Guest
|
| Posted:
Tue Jan 11, 2005 7:56 am Post subject:
Re: point-to-point busses |
|
|
"del cecchi" <dcecchi.nojunk@att.net> writes:
| Quote: | a cluster. The raw bandwidth isn't as important as speed of small
transfers. Any advice?
Yep, roll your own. Xilinx sells several FPGAs containing high speed
serial I/O physical interfaces. What is it you want to attach to what?
If it is X86, the best current solution that I know would be an opteron
with HTX and InfiniPath from Pathscale, except you can't get it quite
yet. It is still in "early availability" mode according to their
website. And I don't know about the availability of HTX on Motherboards
either.
|
That sounds promising, yes, I'm interested in the general problem of
clustering with cheap commodity hardware, more for database-like
servers than for number crunching. That does tend to mean x86.
Infiniband adapters seem to start around $1K per node, which is orders
of magnitude more expensive than Firewire or 1G Ethernet. Switches
seem to start around $10K. Myrinet is also around $1K per node.
It's possible that I'm missing something, though.
Maybe I should just think about kernel software hacks to use low-level
Ethernet packets for RDMA, bypassing TCP/IP, using x86-64 paging
hardware to map the memory of a whole cluster into each node's user
space, and handling page faults over the switched ethernet. I've also
heard of such a thing as low latency 1G ethernet cards. Is that some
special type of card with a nonstandard protocol, or just some
particular card that happens to do well on small-packet benchmarks?
Is there a better newsgroup for this kind of question?
And on a more traditional comp.arch subject, the x86-64 gets its huge
address space by having something like five levels of page tables. Is
that workable at all?! I'd never heard of other 64 bit cpu's doing
that.
thanks |
|
| Back to top |
|
|
Paul Rubin
Guest
|
| Posted:
Tue Jan 11, 2005 7:56 am Post subject:
Re: point-to-point busses |
|
|
Paul Rubin <http://phr.cx@NOSPAM.invalid> writes:
| Quote: | That sounds promising, yes, I'm interested in the general problem of
clustering with cheap commodity hardware, more for database-like
servers than for number crunching. That does tend to mean x86.
|
Oh shoot, I should have put this in the firewire thread that I started.
I didn't mean to hijack the point-to-point bus thread. Sorry. |
|
| Back to top |
|
|
Del Cecchi
Guest
|
| Posted:
Tue Jan 11, 2005 7:25 pm Post subject:
Re: point-to-point busses |
|
|
glen herrmannsfeldt wrote:
| Quote: | Mark W Brehob wrote:
(snip)
Here is the question: What is the expected trend in
bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a
GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?
Related:
What limits point-to-point bandwidth per pin? It it power, cost,
or just
raw physical limits? Something else (transistor switching speed?)
I don't know the real answer, but pin inductance has been a problem
for a long time. Ground bounce due to the inductance of ground
pins and such.
I would guess that pin inductance would eventually limit
the pin bandwidth. Also driving current decreases as transistors
get faster.
-- glen
Most if not all high speed interfaces use small signal differential |
signaling so the problem of SSO is lessened. And modern BGA packages
have quite low inductance.
And driving current doesn't decrease as transistors get faster, it goes
up which allows the same current to be driven by smaller transistors.
del cecchi |
|
| Back to top |
|
|
David Kanter
Guest
|
| Posted:
Wed Jan 12, 2005 1:06 am Post subject:
Re: point-to-point busses |
|
|
Stephen Fuld wrote:
| Quote: | "del cecchi" <dcecchi.nojunk@att.net> wrote in message
news:34h03jF49t3vaU1@individual.net...
"Paul Rubin" <http://phr.cx@NOSPAM.invalid> wrote in message
news:7xk6qlz57p.fsf@ruckus.brouhaha.com...
Del Cecchi <cecchinospam@us.ibm.com> writes:
2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express,
Ethernet CX4, and whatever they call double speed fibre channel)
PCI-express is an interconnect for up to 5 meters? I'd thought it
was
for plug-in cards on a backplane. Hmmm. I've been wondering what
low
cost alternatives there are for low-latency RDMA between computers
in
a cluster. The raw bandwidth isn't as important as speed of small
transfers. Any advice?
Yep, roll your own. Xilinx sells several FPGAs containing high
speed
serial I/O physical interfaces. What is it you want to attach to
what?
If it is X86, the best current solution that I know would be an
opteron
with HTX and InfiniPath from Pathscale, except you can't get it
quite
yet. It is still in "early availability" mode according to their
website. And I don't know about the availability of HTX on
Motherboards
either.
Otherwise, does anyone make PC with PCI-Express ports, except for
graphics? Or one of the IB vendors as I mentioned before should be
affordable.
Questions out of ignorance. Is a PCI Express port for graphics and
different from any other PCI express port? Since the OP talked about
a
database server, he probably doesn't need much graphics so could he
use any
old graphics thing and use the PCI Express port for his
interconnect???
Also, I thought that you could connect two PCI Express ports together
(on
different motherboards) for a high speed interconnect, but to have
more than
two, you needed the Advanced Switching extensions, which aren't quite
cooked
yet. Is that right?
|
I believe that the PCIe graphics ports are different. They certainly
have a different physical connector (I have a system with both PCIe and
PCIe graphics).
I would stick with PCs that have PCIe...
David |
|
| Back to top |
|
|
David Kanter
Guest
|
| Posted:
Wed Jan 12, 2005 1:08 am Post subject:
Re: point-to-point busses |
|
|
del and Paul,
There are PCs with PCIe ports. I have an Asus NCT-D, that has both
PCIe graphics and PCIe slots. There are no PCIe motherboards that
support the K8 that I know of. However, if you are happy with a
Prescott, the NCT-D should work, as will many others.
David Kanter |
|
| Back to top |
|
|
FredK
Guest
|
| Posted:
Wed Jan 12, 2005 1:49 am Post subject:
Re: point-to-point busses |
|
|
"David Kanter" <dkanter@gmail.com> wrote in message
news:1105473990.050919.90800@f14g2000cwb.googlegroups.com...
| Quote: |
I believe that the PCIe graphics ports are different. They certainly
have a different physical connector (I have a system with both PCIe and
PCIe graphics).
I would stick with PCs that have PCIe...
|
PCI-express is based on 1 or more "lanes" (up to 32 I think),
where the lane is a bi-directional serial link. At the slowest
simplest would be one lane. Using 2 lanes gets you twice the
bandwidth.
The standard connectors are 4, 8 and 16 lanes. So far only
graphics and some high-end cluster interconnects are using
16-lanes. Cards and slots are supposed to play such that you
can put a card with fewer lane requirements into a slot with
more lanes, but each slot "type" was supposed to provide all
the lanes defined for it. But of course, they also started with
an exception and allow an 8 lane slot to be underfunded with
4 lanes. They are now talking about allowing 16 lane slots to
be underfunded as well. |
|
| Back to top |
|
|
Mark W Brehob
Guest
|
| Posted:
Wed Jan 12, 2005 6:49 am Post subject:
Re: point-to-point busses |
|
|
Tim Clacy <nospamtcl@nospamphaseone.nospamdk> wrote:
Thanks Tim. Sounds perfect.
I'm due to present a paper soon to a reading group. If I do this one, I'll
post the slides or summary from my presentation.
Mark
|
|
| Back to top |
|
|
del cecchi
Guest
|
| Posted:
Thu Jan 13, 2005 7:56 am Post subject:
Re: point-to-point busses |
|
|
"Mark W Brehob" <brehob@wildwood.eecs.umich.edu> wrote in message
news:Um%Ed.145$UN1.107@news.itd.umich.edu...
| Quote: | Tim Clacy <nospamtcl@nospamphaseone.nospamdk> wrote:
Thanks Tim. Sounds perfect.
I'm due to present a paper soon to a reading group. If I do this one,
I'll
post the slides or summary from my presentation.
Mark
...better late than never:
http://www.ee.ucla.edu/faculty/papers/vwani_IEEEproc_apr01.pdf
I would think you might choose something more related to current and |
near future practice. I found it difficult to take that paper
seriously. But that's my old fogey speaking I guess. Actually an
interesting subject and example of wretched excess is the 802.3AN or
10GbaseT effort to transmit 10G ethernet over 50 to 100 meters of CAT6
Unshielded Twisted Pair cable. One could probably teach a semester
course just covering all the topics in that standard (eventually to
be ).
del cecchi |
|
| Back to top |
|
|
Guest
|
| Posted:
Tue Jan 18, 2005 10:11 pm Post subject:
Re: point-to-point busses |
|
|
"What limits point-to-point bandwidth per pin?"
The size/complexity of the pin electronics--given that you engineer the
rest of the signal conduit.
On the output side, consider what it means to assert a signal. In
classical digital logic, the signal must pass certain fixed voltage
thresholds. On the input side, then compare how careful the input
discrimination must be so that the receiver gets the signal that the
transmitter asserted. As these thresholds are made closer to each
other, it is easier to assert a series of signals, but it becomes
correspondingly harder to figure out what the wiggling signals mean at
the receiver. After a while, you transit from digital signaling into
radio signaling where the digital signals are coded onto a carrier
signal with certain properties that makes the receivers job doable. So,
there is a spectrum of signaling techinques that connect the dots (as
it were).
Then, you did not specify what electrical environment these signals
will traverse. There are board issues (FR4 versus teflon; electrical
environmental issues) connector issues, electrical isolation issues,
discontiguous grounding issues. Compaing the data rate achievable by
two chips topologically adjacent and soldered down on a single board is
vastly different than a chip->connector->accross board->connector->coax
cable->connector->different board->connector->other chip. The former is
limited only by chip electronics and the ability to test the devices,
while the later is limited more by connector and electrical issues
rather than pin electronic issues.
If you consider the pin electronics to be a tri-state gate (output) and
an inverter (input) then you are basically limited to the 1.5 GHz limit
for chip-connector-board-connector-chip transmission. And this will
slowly grow.
If you add precompensation to the output electronics, 6 GHz to 10 GHz
will be achieved in the not to distant future.
If you add coding to the output electronics, 10-20 GHz will be
achieved.
If you use a code that looks and smells like a radio carrier frequency,
you can wiggle the pins around 100 GHz data transmission rate
If you add a differential input to the receiver, you can achieve
usefully low error rates up into the 2 GHz range--clock jitter limted
If you add clock recovery to sets of receivers, you can achieve useful
error rates into the 5 GHz range--limited by skew tollerance
If you add clock recovery to each receiver, you can achieve useful
error rates up into the 40 GHz range
if you add extensive signal processing to the input before converting
back to digital, you can achieve useful error rates up into the 100 GHz
range
So, by the time an output pin looks like a complete radio transmitter
and an input pin looks like a complete radio receiver you can indeed
approach 100 GHz of pin to pin bandwidth. However, you will not be able
to get very many of these pins on your part.
Mitch |
|
| Back to top |
|
|
del cecchi
Guest
|
| Posted:
Wed Jan 19, 2005 5:52 am Post subject:
Re: point-to-point busses |
|
|
<MitchAlsup@aol.com> wrote in message
news:1106068278.837190.102020@c13g2000cwb.googlegroups.com...
| Quote: | "What limits point-to-point bandwidth per pin?"
shannon's law.
The size/complexity of the pin electronics--given that you engineer
the
rest of the signal conduit.
On the output side, consider what it means to assert a signal. In
classical digital logic, the signal must pass certain fixed voltage
thresholds. On the input side, then compare how careful the input
discrimination must be so that the receiver gets the signal that the
transmitter asserted. As these thresholds are made closer to each
other, it is easier to assert a series of signals, but it becomes
correspondingly harder to figure out what the wiggling signals mean at
the receiver. After a while, you transit from digital signaling into
radio signaling where the digital signals are coded onto a carrier
signal with certain properties that makes the receivers job doable.
So,
there is a spectrum of signaling techinques that connect the dots (as
it were).
|
Fanciest coding in the world can't beat shannon. And you better have
enough high frequency capacity. Remember, the more you modulate the
signal, the wider the bandwidth. Ahh, bessel functions. I remember
them well.
| Quote: |
Then, you did not specify what electrical environment these signals
will traverse. There are board issues (FR4 versus teflon; electrical
environmental issues) connector issues, electrical isolation issues,
discontiguous grounding issues. Compaing the data rate achievable by
two chips topologically adjacent and soldered down on a single board
is
vastly different than a chip->connector->accross
board->connector->coax
cable->connector->different board->connector->other chip. The former
is
limited only by chip electronics and the ability to test the devices,
while the later is limited more by connector and electrical issues
rather than pin electronic issues.
|
The former is still limited by the bandwidth of the channel and the
signal to noise ratio. Going between chips has noise which limits the
number of bits in a baud, and bandwidth limitations caused by parasitics
and device limitations that limit the bandwidth. Testing has nothing to
do with it.
I agree that if the channel is crap, it makes life difficult.
| Quote: |
If you consider the pin electronics to be a tri-state gate (output)
and
an inverter (input) then you are basically limited to the 1.5 GHz
limit
for chip-connector-board-connector-chip transmission. And this will
slowly grow.
|
You talking about single ended NRZ signaling? Where you get the 1.5 GHz
bidness? you mean 1.5 gbit/sec or 3.0 gbit/sec?
| Quote: | will be achieved in the not to distant future.
If you add coding to the output electronics, 10-20 GHz will be
achieved.
10 Gb/s already, no fancy coding. See CEI or UXPI among others. |
| Quote: | If you use a code that looks and smells like a radio carrier
frequency,
you can wiggle the pins around 100 GHz data transmission rate
|
And the signal will disappear into nothing before it gets an inch or
two.
| Quote: | If you add a differential input to the receiver, you can achieve
usefully low error rates up into the 2 GHz range--clock jitter limted
|
You were doing all that single ended? Now ISI is a real problem.
| Quote: | If you add clock recovery to sets of receivers, you can achieve useful
error rates into the 5 GHz range--limited by skew tollerance
If you add clock recovery to each receiver, you can achieve useful
error rates up into the 40 GHz range
|
Clock recovery or clock forwarded makes no difference. And building
those differential nets to have no mode conversion is a real trick at 40
GHz. Length match to a fraction of a picosecond? Better not turn any
corners.
| Quote: | if you add extensive signal processing to the input before converting
back to digital, you can achieve useful error rates up into the 100
GHz
range
|
And how are you building a signal processor that handles 100 GHz input
bandwidth?
| Quote: |
So, by the time an output pin looks like a complete radio transmitter
and an input pin looks like a complete radio receiver you can indeed
approach 100 GHz of pin to pin bandwidth. However, you will not be
able
to get very many of these pins on your part.
Mitch
|
I don't believe a word of it. Not with copper interconnect you can't.
and OC 768 which is a mere paltry 40Gb/s takes everything folks know how
to do on a stand alone chip.
del cecchi
> |
|
| Back to top |
|
|
Terje Mathisen
Guest
|
| Posted:
Wed Jan 19, 2005 12:21 pm Post subject:
Re: point-to-point busses |
|
|
del cecchi wrote:
| Quote: | Fanciest coding in the world can't beat shannon. And you better have
enough high frequency capacity. Remember, the more you modulate the
signal, the wider the bandwidth. Ahh, bessel functions. I remember
them well.
|
The fanciest coding in the world would currently be a turbo style
infinite impulse response feedback code, right?
This approach can get you arbitrarily close to Shannon's limit, but at
the cost of a coding/decoding latency that also goes towards infinity.
Since you obviously know this, you probably meant something like getting
into the neighborhood of Shannon, right? :-)
Terje
--
- <Terje.Mathisen@hda.hydro.com>
"almost all programming can be viewed as an exercise in caching" |
|
| Back to top |
|
|
Thomas Womack
Guest
|
| Posted:
Wed Jan 19, 2005 8:23 pm Post subject:
Re: point-to-point busses |
|
|
In article <355qq4F4ho84kU1@individual.net>,
del cecchi <dcecchi.nojunk@att.net> wrote:
| Quote: | If you use a code that looks and smells like a radio carrier
frequency,
you can wiggle the pins around 100 GHz data transmission rate
And the signal will disappear into nothing before it gets an inch or
two.
|
What's causing this dissipation? Is it a matter of copper being
"opaque" in the same way it is opaque in the optical, in which case
are there materials which are transparent -- 500THz carrier waves
modulated at 2GHz seem to travel quite happily in quartz?
Is it that copper is dispersive to the point that the upper and lower
side-bands travel at different enough speeds to smear out the signal
entirely?
Or is it some other non-obvious consequence of Maxwell's equations?
As you may have noticed, I am very ignorant of RF engineering; sorry
if these are stupid questions.
Tom |
|
| Back to top |
|
|
Guest
|
| Posted:
Thu Jan 20, 2005 12:54 am Post subject:
Re: point-to-point busses |
|
|
del cecchi wrote:
| Quote: | MitchAlsup@aol.com> wrote in message
news:1106068278.837190.102020@c13g2000cwb.googlegroups.com...
Fanciest coding in the world can't beat shannon.
|
Yes, and the Shannon limit for a 10mW signal at 300K is what? About
10,000Tb/s? |
|
| Back to top |
|
|
Del Cecchi
Guest
|
| Posted:
Thu Jan 20, 2005 1:40 am Post subject:
Re: point-to-point busses |
|
|
Terje Mathisen wrote:
| Quote: | del cecchi wrote:
Fanciest coding in the world can't beat shannon. And you better have
enough high frequency capacity. Remember, the more you modulate the
signal, the wider the bandwidth. Ahh, bessel functions. I remember
them well.
The fanciest coding in the world would currently be a turbo style
infinite impulse response feedback code, right?
This approach can get you arbitrarily close to Shannon's limit, but at
the cost of a coding/decoding latency that also goes towards infinity.
Since you obviously know this, you probably meant something like getting
into the neighborhood of Shannon, right? :-)
Terje
yep, can't beat shannon. In fact you probably can't even tie shannon. |
Of course NRZ is like 20dB worse than Shannon. :-( But it has low
latency :-)
Now about that 100 Gbit/second bandwidth....... That's a hell of a channel.
del cecchi |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in