| Author |
Message |
keith
Guest
|
 Posted:
Mon Dec 06, 2004 7:46 am Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On Sun, 05 Dec 2004 19:47:30 +0000, Patrick Schaaf wrote:
| Quote: | Tony Hill <hilla_nospam_20@yahoo.ca> writes:
Not that they know something the rest of the world doesn't, just that
they have access to processors that most of us do not. IBM sells them
as well, but for the time being Intel will ONLY sell them for use in
servers. Why? I really don't know. Maybe it's just a bit too much
crow for them to eat after saying (only a bit over a year ago) that
64-bit wouldn't be useful for the desktop until the end of the year?
How much does Intel stockpile? Could it be that they have warehouses
full of already produced non-64-bit processors, and those want to be
sold at the projected prices, not thrown away?
|
Unsold inventory is a very bad thing indeed. The tax man isn't happy.
Stockholders aren't happy. Executives shiver.
--
Keith |
|
| Back to top |
|
 |
Greg Lindahl
Guest
|
| Posted:
Mon Dec 06, 2004 8:12 am Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
In article <pan.2004.12.06.02.44.34.997332@att.bizzzz>,
keith <krw@att.bizzzz> wrote:
| Quote: | I'd say that because in small systems (less than 8 CPUs), Opterons are
coherent in hardware thus sufficiently tightly coupled to be called UMA,
as far as the user is concerned.
|
However, it's not hard to show with benchmarks that paying attention
to the NUMA nature of the Opteron is a significant win. So you can
call it what you want, but...
Newsgroups trimmed.
-- greg |
|
| Back to top |
|
|
keith
Guest
|
| Posted:
Mon Dec 06, 2004 8:22 am Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On Sun, 05 Dec 2004 21:01:14 +0100, Grumble wrote:
| Quote: | John Dallman wrote:
Intel are probably also shipping the 32-bit Prescotts with their
version of the AMD NX bit (I think they call it "EDP") by now too.
AFAICT, AMD calls it NX, Intel calls it XD, and MS calls DEP its
implementation based on the bit.
|
Oh good. Intel can't admit to being caught pants-less and M$ plays along.
Come on! I think even the Opteron has an AX register. ;-)
--
Keith |
|
| Back to top |
|
|
keith
Guest
|
| Posted:
Mon Dec 06, 2004 8:24 am Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On Sun, 05 Dec 2004 19:12:44 -0800, Greg Lindahl wrote:
| Quote: | In article <pan.2004.12.06.02.44.34.997332@att.bizzzz>,
keith <krw@att.bizzzz> wrote:
I'd say that because in small systems (less than 8 CPUs), Opterons are
coherent in hardware thus sufficiently tightly coupled to be called UMA,
as far as the user is concerned.
However, it's not hard to show with benchmarks that paying attention
to the NUMA nature of the Opteron is a significant win. So you can
call it what you want, but...
|
Point well taken. So we have a desert topping and a floor wax. ;-)
| Quote: | Newsgroups trimmed.
|
..chips added back in.
--
Keith |
|
| Back to top |
|
|
George Macdonald
Guest
|
| Posted:
Mon Dec 06, 2004 12:04 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On Sun, 05 Dec 2004 16:30:15 -0500, Yousuf Khan <bbbl67@ezrs.com> wrote:
| Quote: | George Macdonald wrote:
Hmm and the following quote: "However, the latency difference between local
and remote accesses is actually very small because the memory controller is
integrated into and operates at the core speed of the processor, and
because of the fast interconnect between processors." is relevant to
another discussion here. I wish we could get a firm answer on this one.
Yeah, but that's why I think AMD insists on calling their multiprocessor
connection scheme as SUMO (Sufficiently Uniform Memory Organization),
rather than NUMA. It's not worth headaching over such small differences
in latency, is basically what they're saying.
|
See my reply to Rob Stow.
Rgds, George Macdonald
"Just because they're paranoid doesn't mean you're not psychotic" - Who, me?? |
|
| Back to top |
|
|
George Macdonald
Guest
|
| Posted:
Mon Dec 06, 2004 12:04 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On Sun, 05 Dec 2004 17:37:16 GMT, Rob Stow <rob.stow.nospam@shaw.ca> wrote:
| Quote: | George Macdonald wrote:
On Sun, 05 Dec 2004 01:02:11 -0500, Yousuf Khan <bbbl67@ezrs.com> wrote:
I found this whitepaper from HP to be pretty good, it is surprisingly
candid, considering HP was the coinventor of the Itanium. It does a
pretty good job of explaining and summarizing the similarities and
differences between AMD64 and EM64T, and their comparison to the
Itanium's IA64 instruction set. AMD64 and EM64T are "broadly
compatible", but IA64 is a different animal altogether.
Yousuf Khan
http://h200001.www2.hp.com/bc/docs/support/SupportManual/c00238028/c00238028.pdf
Hmm and the following quote: "However, the latency difference between local
and remote accesses is actually very small because the memory controller is
integrated into and operates at the core speed of the processor, and
because of the fast interconnect between processors." is relevant to
another discussion here. I wish we could get a firm answer on this one.
Not sure if this is exactly what you are looking for in the
way of a "firm answer", but the latencies in a Opteron system are:
0 hops 80 ns uniprocessor (Local access)
100 ns multiprocessor (Local access, with cache snooping on other processors)
1 hop 115 ns
2 hops 150 ns
3 hops 190 ns
I couldn't find my original source for those numbers, and
the two and three hop numbers above are a little higher
than I remembered them as being. This time around I got
them from this thread:
http://www.aceshardware.com/forum?read=80030960
That thread refers to this article:
http://www.digit-life.com/articles2/amd-hammer-family/
which gives slightly different numbers for a 2 GHz Opteron
with DDR333:
Uni-processor system: 45 ns
Dual-processor system: 0-hop - 69 ns, 1-hop - 117 ns.
Four-processor system: 0-hop - 100 ns, 1-hop - 118 ns, 2-hop - 136 ns.
I don't know if any of the numbers above are for cache misses
or if they are averages that include both hits and misses.
|
Thanks for the data but no I guess I should have highlighted better what I
was getting at: "the memory controller is integrated into and operates at
the core speed of the processor", which is what was being
discussed/disputed in another thread.
I haven't been able to find any hard data from AMD on where the clock
domain boundaries are in the Opteron/Athlon64 but if the memory controller
is not operating at "core speed" it's now at the stage of Internet
Folklore.
Rgds, George Macdonald
"Just because they're paranoid doesn't mean you're not psychotic" - Who, me?? |
|
| Back to top |
|
|
Greg Lindahl
Guest
|
| Posted:
Mon Dec 06, 2004 12:29 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
In article <8c97r0hqh2sqf8sh89ut3153lpdmddfs76@4ax.com>,
George Macdonald <fammacd=!SPAM^nothanks@tellurian.com> wrote:
| Quote: | I haven't been able to find any hard data from AMD on where the clock
domain boundaries are in the Opteron/Athlon64 but if the memory controller
is not operating at "core speed" it's now at the stage of Internet
Folklore.
|
Note that the STREAM bandwidth and lmbench latency changes with every
cpuspeedbump. So clearly part of the memory controller is at the cpu
core frequency, or a related frequency, and not at the HT frequency,
or the SDRAM external bus frequency.
Please reduce the cross-post. Followups set to a group I read.
-- greg |
|
| Back to top |
|
|
Per Ekman
Guest
|
| Posted:
Mon Dec 06, 2004 3:17 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
Yousuf Khan <bbbl67@ezrs.com> writes:
| Quote: | George Macdonald wrote:
Hmm and the following quote: "However, the latency difference between local
and remote accesses is actually very small because the memory controller is
integrated into and operates at the core speed of the processor, and
because of the fast interconnect between processors." is relevant to
another discussion here. I wish we could get a firm answer on this one.
Yeah, but that's why I think AMD insists on calling their multiprocessor
connection scheme as SUMO (Sufficiently Uniform Memory Organization),
rather than NUMA. It's not worth headaching over such small differences
in latency, is basically what they're saying.
|
It's a bit of a crap argument isn't it? Even if the latency is small,
the fact that it's a NUMA system impacts performance (potentially by a
lot) as the available memory bandwidth is coupled to where you place
your data.
Classic example is OpenMP parallelized STREAM. Parallelize all the
loops except the data initialization loop on a system with hard memory
affinity (such as Linux), then parallelize _all_ the loops and explain
how the difference is "not worth headaching over".
Bottom line IMO is that pretending that the system isn't NUMA is doing
customers a disservice. They should know that treating the system as a
UMA one is a bad idea.
*p |
|
| Back to top |
|
|
Tony Hill
Guest
|
| Posted:
Mon Dec 06, 2004 4:53 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On 6 Dec 2004 00:30:48 GMT, ammonton@cc.full.stop.helsinki.fi wrote:
| Quote: | In comp.arch Tony Hill <hilla_nospam_20@yahoo.ca> wrote:
Not that they know something the rest of the world doesn't, just that
they have access to processors that most of us do not. IBM sells them
as well, but for the time being Intel will ONLY sell them for use in
servers. Why? I really don't know.
FWIW, Dell are shipping EM64T-equipped non-Xeon P4 workstations (the
Precision 370).
|
Ahh, thanks. When I first wrote the above I had actually included
Dell's name as well, but then removed it when I couldn't find any
EM64T P4 processors in any of their servers (didn't think to check
workstations first). I figured that if anyone was selling 64-bit P4s
it would be Dell!
-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca |
|
| Back to top |
|
|
Tony Hill
Guest
|
| Posted:
Mon Dec 06, 2004 4:53 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On 05 Dec 2004 19:47:30 GMT, mailer-daemon@bof.de (Patrick Schaaf)
wrote:
| Quote: | Tony Hill <hilla_nospam_20@yahoo.ca> writes:
Not that they know something the rest of the world doesn't, just that
they have access to processors that most of us do not. IBM sells them
as well, but for the time being Intel will ONLY sell them for use in
servers. Why? I really don't know. Maybe it's just a bit too much
crow for them to eat after saying (only a bit over a year ago) that
64-bit wouldn't be useful for the desktop until the end of the year?
How much does Intel stockpile? Could it be that they have warehouses
full of already produced non-64-bit processors, and those want to be
sold at the projected prices, not thrown away?
|
ALL of the "Prescott" and "Nocona" cores are 64-bit capable excluding
those that would pass a validation as 32-bit chips but fail as 64-bit
chips, but such chips would be rather few and far between. It could
be that Intel still has a reasonable amount of inventory of their old
"Northwood" P4 chips and they want to clear those out first, but that
certainly doesn't seem to be the case looking at Intel's pricing
structure and what is being sold by the major OEMs (Intel seems to be
pushing Prescott VERY hard here).
Long story short, I'm not quite sure what the actual answer is, but
excessive inventory of 32-bit chips doesn't seem to make sense from
what I've seen.
-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca |
|
| Back to top |
|
|
Tony Hill
Guest
|
| Posted:
Mon Dec 06, 2004 4:53 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
On 06 Dec 2004 11:17:19 +0100, Per Ekman <pek@pdc.kth.se> wrote:
| Quote: | Yousuf Khan <bbbl67@ezrs.com> writes:
Yeah, but that's why I think AMD insists on calling their multiprocessor
connection scheme as SUMO (Sufficiently Uniform Memory Organization),
rather than NUMA. It's not worth headaching over such small differences
in latency, is basically what they're saying.
It's a bit of a crap argument isn't it? Even if the latency is small,
the fact that it's a NUMA system impacts performance (potentially by a
lot) as the available memory bandwidth is coupled to where you place
your data.
|
It does, but the difference is small, usually less than 10% and often
much closer to 0%. When well over 90% of your memory access is coming
from cache anyway and (assuming a totally random distribution in a
strictly UMA setup) 50% of your memory access is going to be local,
most of the performance difference is lost in the noise.
Besides, remember that even in a classic UMA environment (ie a 2P or
4P Xeon server... or even a single-processor system) you STILL have
differences in latency depending on where in memory your data resides
due to open vs. closed pages, TLB misses, etc.
| Quote: | Classic example is OpenMP parallelized STREAM. Parallelize all the
loops except the data initialization loop on a system with hard memory
affinity (such as Linux), then parallelize _all_ the loops and explain
how the difference is "not worth headaching over".
|
Most users don't use their computer to run STREAM though. Even in the
HPC community where memory bandwidth is king, STREAM is still a rather
extreme case.
| Quote: | Bottom line IMO is that pretending that the system isn't NUMA is doing
customers a disservice.
|
I've said it before and I'll say it again: Hardware is cheap,
software is expensive. It would be a true disservice to your
customers to tell them to spend thousands upon thousands of dollars
changing all their software for the small improvement in performance
equal to a few hundred dollars of hardware costs.
| Quote: | They should know that treating the system as a
UMA one is a bad idea.
|
Spending lots of money to make all your software NUMA is a bad idea
when treating it as UMA and throwing a tiny amount of extra hardware
at the job will do the trick. That's all that AMD is getting at.
Besides, they do recognize that it is NUMA, just that they are saying
you don't NEED to worry about that if you don't want to because for
the vast majority of times the performance difference is lost in the
noise.
-------------
Tony Hill
hilla <underscore> 20 <at> yahoo <dot> ca |
|
| Back to top |
|
|
Janne Blomqvist
Guest
|
| Posted:
Mon Dec 06, 2004 5:19 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
In article <gbe8r0pp5ip0dl4d58fvktklbuu35442it@4ax.com>, Tony Hill wrote:
| Quote: | It could
be that Intel still has a reasonable amount of inventory of their old
"Northwood" P4 chips and they want to clear those out first, but that
certainly doesn't seem to be the case looking at Intel's pricing
structure and what is being sold by the major OEMs (Intel seems to be
pushing Prescott VERY hard here).
|
A friend recently (1 month ago IIRC) wanted a Northwood for his DIY
computer, but he found that none of the usual suspects around here had
them in stock. Eventually he called the importer, who said that
they're out of stock and they're not getting anymore either, buy a
Prescott instead.
| Quote: | Long story short, I'm not quite sure what the actual answer is, but
excessive inventory of 32-bit chips doesn't seem to make sense from
what I've seen.
|
Considering the rate chips depreciate I guess manufacturers think
pretty hard about what they can do to minimize inventory.
--
Janne Blomqvist |
|
| Back to top |
|
|
Greg Lindahl
Guest
|
| Posted:
Mon Dec 06, 2004 5:48 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
In article <pqe8r0hsb4vfiqv4uvpk6h2h7cn8gq5q37@4ax.com>,
Tony Hill <hilla_nospam_20@yahoo.ca> wrote:
| Quote: | It does, but the difference is small, usually less than 10% and often
much closer to 0%.
|
No, it's not. The Opteron builds the best 4-cpu SMP system out there
according to the SPECrate2000 cpu benchmark, but in order to get that
best result, you need to pin the individual processes to cpus and
memory using a utility. Without it, the performance is no longer the
best. So people really care about that last bit of performance.
Now I don't have the directly comparison for that, but here's a
comparison on some benchmarks for a recent competitive bid. "Slow" is
a system without the processor binding and with "node interleave"
turned on. "Fast" is with processor binding and node interleave off,
which lets the processor binding have the best benefit. Note that it's
only a trivial amount of work to get this improvement for a serial
code, so this is a common situation, although these benchmarks are, of
course, particular to this scientific-computing customer. In these
results, the comparison is scaling for 4 processes on a 4 cpu machine.
4.0 would be a perfect score.
fast slow difference
benchmark 1 3.71 3.03 + 22 %
benchmark 2 3.76 3.29 + 14 %
benchmark 3 3.78 3.26 + 16 %
benchmark 4 3.79 3.45 + 10 %
benchmark 5 3.92 3.89 + 1 %
benchmark 6 3.88 3.71 + 5 %
These benchmarks were run with the best Opteron compiler, so this
scaling improvement was very good to see. And it's bigger than
"usually less than 10%".
| Quote: | When well over 90% of your memory access is coming
from cache anyway and (assuming a totally random distribution in a
strictly UMA setup) 50% of your memory access is going to be local,
most of the performance difference is lost in the noise.
|
Handwaving is a bad way to evaluate effects like this.
| Quote: | I've said it before and I'll say it again: Hardware is cheap,
software is expensive. It would be a true disservice to your
customers to tell them to spend thousands upon thousands of dollars
changing all their software for the small improvement in performance
equal to a few hundred dollars of hardware costs.
|
Customers know what 10% or 20% more performance means, as do vendors
who are doing competitive bidding. The fact that I care a lot about
this should give you a clue. And in some cases, such as serial codes,
the benefits are easy to achieve. It took only a moderate amount of
work in our OpenMP compiler and runtime to get these benefits for some
parallel programs, too. Well worth it to our customers.
-- greg
speaking for myself, not PathScale |
|
| Back to top |
|
|
Niels Jørgen Kruse
Guest
|
| Posted:
Mon Dec 06, 2004 6:02 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
Greg Lindahl <lindahl@pbm.com> wrote:
| Quote: | In article <8c97r0hqh2sqf8sh89ut3153lpdmddfs76@4ax.com>,
George Macdonald <fammacd=!SPAM^nothanks@tellurian.com> wrote:
I haven't been able to find any hard data from AMD on where the clock
domain boundaries are in the Opteron/Athlon64 but if the memory controller
is not operating at "core speed" it's now at the stage of Internet
Folklore.
Note that the STREAM bandwidth and lmbench latency changes with every
cpuspeedbump. So clearly part of the memory controller is at the cpu
core frequency, or a related frequency, and not at the HT frequency,
or the SDRAM external bus frequency.
|
Memory latency follow CPU clock too for an offchip memory controller, if
the clock multiplier is held constant.
I ran LMbench 2.0 on an iMac G5 at highest performance (1.8 GHz core /
600 MHz FSB) and at lowest performance (900 MHz core / 300 MHz FSB) and
got 184.1 ns and 331.5 ns respectively. The DRAM dependent component is
then 36.7 ns. Scaling to higher FSB speeds result in the predicted
latencies below:
FSB MHz Main memory latency
900 134.97 nanoseconds
1000 125.14 nanoseconds
1100 117.10 nanoseconds
1250 107.45 nanoseconds
1500 95.66 nanoseconds
The designers of the 970 FSB probably expected clock to scale a bit
faster than it has. If the DRAM component stays largely constant in the
future, then the latency difference between onchip and offchip memory
controller would be expected do diminish over time.
--
Mvh./Regards, Niels Jørgen Kruse, Vanløse, Denmark |
|
| Back to top |
|
|
Grumble
Guest
|
| Posted:
Mon Dec 06, 2004 6:05 pm Post subject:
Re: Pretty good explanation of x86-64 by HP |
|
|
Greg Lindahl wrote:
| Quote: | These benchmarks were run with the best Opteron compiler [...]
|
Which compiler would that be? PathScale?
:-) |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in