point-to-point busses

[Del Cecchi]

Thomas Womack wrote:

In article <355qq4F4ho84kU1@individual.net>,
del cecchi <dcecchi.nojunk@att.net> wrote:


If you use a code that looks and smells like a radio carrier

frequency,

you can wiggle the pins around 100 GHz data transmission rate

And the signal will disappear into nothing before it gets an inch or
two.


What's causing this dissipation? Is it a matter of copper being
"opaque" in the same way it is opaque in the optical, in which case
are there materials which are transparent -- 500THz carrier waves
modulated at 2GHz seem to travel quite happily in quartz?

Is it that copper is dispersive to the point that the upper and lower
side-bands travel at different enough speeds to smear out the signal
entirely?

Or is it some other non-obvious consequence of Maxwell's equations?

As you may have noticed, I am very ignorant of RF engineering; sorry
if these are stupid questions.

Tom

Copper has resistance which causes loss. The resistance increases
approximately as the square root of the frequency due to the "skin
effect". So, yes the copper in a sense becomes opaque. The energy in
the signal gets dissippated in the copper and doesn't get to the load.
We in electric land have have nothing with the low loss, wide band
capabilities of a nice graded index fiber.

del

[Del Cecchi]

already5chosen@yahoo.com wrote:

del cecchi wrote:

MitchAlsup@aol.com> wrote in message
news:1106068278.837190.102020@c13g2000cwb.googlegroups.com...

Fanciest coding in the world can't beat shannon.


Yes, and the Shannon limit for a 10mW signal at 300K is what? About
10,000Tb/s?

Way smaller than that kemo sabe. What kind of channel are you assuming?

If it is a copper wire with 100 ohm impedence 10 mw is about 300 mv
RMS if I did my calculation right, and to get 10**16b/sec over a useful
distance one would have to have a signal to noise of what, 10**6 at
least. So I have to have an input stage with 300 nanovolt sensitivity,
and on a chip since we were talking about pin electronics. And it has
to have a bandwidth of 10GHz. Seems way unlikely.

del cecchi

[Guest]

I meant theoretical limit at unlimited bandwidth in the presence of the
thermal noise only.
Shennon says: C = W * log2(1+S/N)
Nyquist says: N = W * 4kT
where W=bandwidth
For very big W: S/N << 1
C = (W * S)/(N*ln2) = (W * S)/(W *4kT*ln2) = S/(4kT*ln2);
For S = 0.01W, T=300K
C = 0.87E18 b/s
In my previous post I was pessimistic by factor of 87.

[del cecchi]

<already5chosen@yahoo.com> wrote in message
news:1106177171.769323.173230@f14g2000cwb.googlegroups.com...

I meant theoretical limit at unlimited bandwidth in the presence of
the
thermal noise only.
Shennon says: C = W * log2(1+S/N)
Nyquist says: N = W * 4kT
where W=bandwidth
For very big W: S/N << 1
C = (W * S)/(N*ln2) = (W * S)/(W *4kT*ln2) = S/(4kT*ln2);
For S = 0.01W, T=300K
C = 0.87E18 b/s
In my previous post I was pessimistic by factor of 87.

Not for any chip I worked on. I think you left out a few noise sources.

Shot noise for one, not counting man made noise sources.

del cecchi

[Terje Mathisen]

Del Cecchi wrote:

Copper has resistance which causes loss. The resistance increases
approximately as the square root of the frequency due to the "skin
effect". So, yes the copper in a sense becomes opaque. The energy in
the signal gets dissippated in the copper and doesn't get to the load.
We in electric land have have nothing with the low loss, wide band
capabilities of a nice graded index fiber.

Ouch Del!

'nice _graded_ index fiber'?

A graded index fiber usually means a big (~50 um?) multimode fiber
employing a near parabolic refractory index profile to minimize
multi-mode dispersion.

However, even doing this leaves you with an order of magnitude less
bandwidth than a conceptually much simpler (step-index) single-mode
fiber afaik. (At least it did 24+ years ago when I wrote my thesis about
this kind of stuff!)

Terje

--
- <Terje.Mathisen@hda.hydro.com>
"almost all programming can be viewed as an exercise in caching"

[Del Cecchi]

Terje Mathisen wrote:

Del Cecchi wrote:

Copper has resistance which causes loss. The resistance increases
approximately as the square root of the frequency due to the "skin
effect". So, yes the copper in a sense becomes opaque. The energy in
the signal gets dissippated in the copper and doesn't get to the load.
We in electric land have have nothing with the low loss, wide band
capabilities of a nice graded index fiber.


Ouch Del!

'nice _graded_ index fiber'?

A graded index fiber usually means a big (~50 um?) multimode fiber
employing a near parabolic refractory index profile to minimize
multi-mode dispersion.

However, even doing this leaves you with an order of magnitude less
bandwidth than a conceptually much simpler (step-index) single-mode
fiber afaik. (At least it did 24+ years ago when I wrote my thesis about
this kind of stuff!)

Terje

You prompted me to go do some review. You are indeed correct, graded
index is used to reduce dispersion in multimode fibres. And single mode
fibers are step index. Well, sort of. Apparently all sorts of stuff is
now being done with the cladding to reduce dispersion type effects in
single mode fibre as well. But indeed I was wrong to refer to graded
index single mode fiber. Thanks for the correction

And something I learned a while back that really surprised me is that
single mode fibre is significantly cheaper per meter than multimode
fiber is.

And both of them have way less loss and ISI than a copper wire. :-(

del cecchi

[Terje Mathisen]

Del Cecchi wrote:

Terje Mathisen wrote:
A graded index fiber usually means a big (~50 um?) multimode fiber
employing a near parabolic refractory index profile to minimize
multi-mode dispersion.

However, even doing this leaves you with an order of magnitude less
bandwidth than a conceptually much simpler (step-index) single-mode
fiber afaik. (At least it did 24+ years ago when I wrote my thesis
about this kind of stuff!)

Terje

You prompted me to go do some review. You are indeed correct, graded
index is used to reduce dispersion in multimode fibres. And single mode
fibers are step index. Well, sort of. Apparently all sorts of stuff is
now being done with the cladding to reduce dispersion type effects in
single mode fibre as well. But indeed I was wrong to refer to graded
index single mode fiber. Thanks for the correction

And something I learned a while back that really surprised me is that
single mode fibre is significantly cheaper per meter than multimode
fiber is.

Not really surprising, since (a) you need much less high-quality raw
material (optical fiber is 3-4 orders of magnitude clearer than the best
glass used in expensive optical lenses), (b) you can make do with a
relatively coarse step between the core and the cladding.

And both of them have way less loss and ISI than a copper wire. :-(

The _only_ real cost of fiber is at the ends.

Terje

--
- <Terje.Mathisen@hda.hydro.com>
"almost all programming can be viewed as an exercise in caching"

[Mark W Brehob]

Sorry for popping in-and-out, really rough semester....

Yeah, after reading the paper, I agree. It is "out there".

Mark

del cecchi <dcecchi.nojunk@att.net> wrote:

"Mark W Brehob" <brehob@wildwood.eecs.umich.edu> wrote in message
news:Um%Ed.145$UN1.107@news.itd.umich.edu...
Tim Clacy <nospamtcl@nospamphaseone.nospamdk> wrote:

Thanks Tim. Sounds perfect.

I'm due to present a paper soon to a reading group. If I do this one,
I'll
post the slides or summary from my presentation.

Mark

...better late than never:

http://www.ee.ucla.edu/faculty/papers/v ... _apr01.pdf


I would think you might choose something more related to current and
near future practice. I found it difficult to take that paper
seriously. But that's my old fogey speaking I guess. Actually an
interesting subject and example of wretched excess is the 802.3AN or
10GbaseT effort to transmit 10G ethernet over 50 to 100 meters of CAT6
Unshielded Twisted Pair cable. One could probably teach a semester
course just covering all the topics in that standard (eventually to
be ).

del cecchi

[Mark W Brehob]

Again, sorry for popping in and out. Just too busy to do things that aren't
purely work or family. But I did have some questions:



Del Cecchi <cecchinospam@us.ibm.com> wrote:

Mark W Brehob wrote:
Hello all,

At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).

Here is the question:

OK, right down my alley. Free consulting. Such a deal.

:-)

What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?

For moderate length interconnections (up to maybe 5 meters)

1 Gbit/second/pair is old news. Fibre Channel was at this rate a while
back. (1.0625 Gb/s)

2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express, Ethernet
CX4, and whatever they call double speed fibre channel)

5 Gb/second is coming soon. InfiniBand DDR, PCI-Express Gen 2, CEI from
OIF.

10 Gbit is coming not far behind.

So I'm thinking about processor to memory bandwidth. I realize pins are
pins, but I'm guessing that trying to run a large number of pairs at 5GHz
off of one chip might not be viable. The ITRS numbers
(http://www.itrs.net/Common/2004Update/2004_000_ORTC.pdf) predict a fairly
rapid increase in off-chip frequency, but only for a "small number of pins.
They don't specify the size of "small" but they are going with 7GHz by 2009
and over 50GHz by 2018. I'm assuming the small is due to interference with
each other as well as power concerns.

As what I'm mainly worried about is total off-chip bandwidth, any idea how
"small" small would be?

I guess the real question is: What off-chip bandwidth can be achieved in a
reasonable processor (so reasonable cost including cooling) if all
communication is point-to-point? When I worked at Intel, bandwidth was
limited by the broadcast nature of the bus (communicating with up to 4
processors and the chipset). I know point-to-point helps, but how much?

<clip>

[Mark W Brehob]

Mark W Brehob <brehob@wildwood.eecs.umich.edu> wrote:

Hello all,

At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).

Reading over everything, I realized that I hadn't explained my modivation
for asking very well. Colwell was mainly discussing multi-core processors.
His comment was that one of the big hurdles for them is memory bandwidth.


I don't believe that most applications saturate the memory bus
when using a single processor (though my experiance is about 7 years
out-of-date). Although I've spoken to a database expert who indicates that
modern databases are becoming memory bound rather than disk bound.
Multi-processor bus-based systems do have (potentially serious) bandwidth
problems.

My feeling is that it really isn't that bad. With a multi-core I'm handling
all of the coherance traffic on-chip. So the coherance misses need not show
up on the memory bus (assuming the other processor(s) still have a copy.)
Further, the interconnect is now a true point-to-point connection. This
should allow significantly higher bandwdith than a broadcast based bus (such
as Intel's). And even better than AMD who has to dedicate pins to multiple
ports for coherance purposes. Obviously I'm assuming we don't have more
than one chip, which I think is reasonable as silicon realestate becomes
more and more viable.

So....

That is the context in which I'm asking about bandwidth.....


Mark

[Nick Maclaren]

In article <rlLJd.654$UN1.292@news.itd.umich.edu>,
Mark W Brehob <brehob@wildwood.eecs.umich.edu> wrote:

At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).

Reading over everything, I realized that I hadn't explained my modivation
for asking very well. Colwell was mainly discussing multi-core processors.
His comment was that one of the big hurdles for them is memory bandwidth.

I don't believe that most applications saturate the memory bus
when using a single processor (though my experiance is about 7 years
out-of-date). Although I've spoken to a database expert who indicates that
modern databases are becoming memory bound rather than disk bound.
Multi-processor bus-based systems do have (potentially serious) bandwidth
problems.

Remember that on modern cache-based systems, the memory bandwidth needed to
support a certain access rate may be many times that delivered to the code.

Writing single bytes at random leads to a bandwidth requirement of twice the
cache line size per bytes written.


Regards,
Nick Maclaren.

[Del Cecchi]

Mark W Brehob wrote:

Again, sorry for popping in and out. Just too busy to do things that aren't
purely work or family. But I did have some questions:



Del Cecchi <cecchinospam@us.ibm.com> wrote:

Mark W Brehob wrote:

Hello all,

At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).

Here is the question:


OK, right down my alley. Free consulting. Such a deal.


:-)


What is the expected trend in bits/second/wire over a point-to-point
connection (off-chip)? Do we expect to see a cap of just over a GHz or
will see a steady increase? And if an increase, where do we think the
number will be in 5 years?


For moderate length interconnections (up to maybe 5 meters)


1 Gbit/second/pair is old news. Fibre Channel was at this rate a while
back. (1.0625 Gb/s)


2.5-3.125 is commonplace today (OC-48? InfiniBand, PCI-Express, Ethernet
CX4, and whatever they call double speed fibre channel)


5 Gb/second is coming soon. InfiniBand DDR, PCI-Express Gen 2, CEI from
OIF.


10 Gbit is coming not far behind.


So I'm thinking about processor to memory bandwidth. I realize pins are
pins, but I'm guessing that trying to run a large number of pairs at 5GHz
off of one chip might not be viable. The ITRS numbers
(http://www.itrs.net/Common/2004Update/2004_000_ORTC.pdf) predict a fairly
rapid increase in off-chip frequency, but only for a "small number of pins.
They don't specify the size of "small" but they are going with 7GHz by 2009
and over 50GHz by 2018. I'm assuming the small is due to interference with
each other as well as power concerns.

As what I'm mainly worried about is total off-chip bandwidth, any idea how
"small" small would be?

I guess the real question is: What off-chip bandwidth can be achieved in a
reasonable processor (so reasonable cost including cooling) if all
communication is point-to-point? When I worked at Intel, bandwidth was
limited by the broadcast nature of the bus (communicating with up to 4
processors and the chipset). I know point-to-point helps, but how much?

clip


If you want to know Intel current thinking, look up "FBDIMM" talks from

last IDF. As I recall they are talking about multiple channels at 4.8
Gb/s*10b out and 14b into the processor.

see for example
http://download.micron.com/pdf/presenta ... n_2004.pdf

I wouldn't be too afraid of signing up for 100 lanes at 5Gb in a 90nm
technology.

It would take some space and suck some power fer sure.

del cecchi

[Del Cecchi]

Mark W Brehob wrote:

Mark W Brehob <brehob@wildwood.eecs.umich.edu> wrote:

Hello all,


At the University of Michigan we just had Robert Colwell (formally of Intel)
come in and give a talk. After the talk one issue came up which I found I
was disagreeing with everyone about (generally a sign I'm wrong of course).


Reading over everything, I realized that I hadn't explained my modivation
for asking very well. Colwell was mainly discussing multi-core processors.
His comment was that one of the big hurdles for them is memory bandwidth.


I don't believe that most applications saturate the memory bus
when using a single processor (though my experiance is about 7 years
out-of-date). Although I've spoken to a database expert who indicates that
modern databases are becoming memory bound rather than disk bound.
Multi-processor bus-based systems do have (potentially serious) bandwidth
problems.

My feeling is that it really isn't that bad. With a multi-core I'm handling
all of the coherance traffic on-chip. So the coherance misses need not show
up on the memory bus (assuming the other processor(s) still have a copy.)
Further, the interconnect is now a true point-to-point connection. This
should allow significantly higher bandwdith than a broadcast based bus (such
as Intel's). And even better than AMD who has to dedicate pins to multiple
ports for coherance purposes. Obviously I'm assuming we don't have more
than one chip, which I think is reasonable as silicon realestate becomes
more and more viable.

So....

That is the context in which I'm asking about bandwidth.....


Mark

For Servers, I think there will be more than a single chip for the
forseeable future.

Secondly, it is not just total bandwidth, but the effect bandwidth has
on latency. IE if one is fetching a cache line as a packet, one has to
get the whole thing and check it before going too far down the road of
using it. So if your bandwidth is 1B/ns and the packet is 128B, that is
128 ns of added time. Of course some of it can be hidden. If
bandwidth is say 8B/ns, then the added time is only 16 ns.

del cecchi

[Tim Clacy]

Mark W Brehob wrote:

Sorry for popping in-and-out, really rough semester....

Yeah, after reading the paper, I agree. It is "out there".

Mark

....but beautiful. The RF ultra-wide-band physical layer is pretty simple (no
security required, simpler error handling) and pretty low power and allows
data rates of tens of Gbps... and allows multiple concurrent Gbps channels
over a single crappy old conductor.

[Mark W Brehob]

Tim Clacy <nospamtcl@nospamphaseone.nospamdk> wrote:

Mark W Brehob wrote:
Sorry for popping in-and-out, really rough semester....

Yeah, after reading the paper, I agree. It is "out there".

Mark


...but beautiful. The RF ultra-wide-band physical layer is pretty simple (no
security required, simpler error handling) and pretty low power and allows
data rates of tens of Gbps... and allows multiple concurrent Gbps channels
over a single crappy old conductor.

I don't know enough to judge it. But it is brave. If it works it opens up
a whole can of worms with security I should think. But it is cool.

Mark