Yue Luo
Guest
|
 Posted:
Wed Dec 08, 2004 3:30 am Post subject:
accuracy requirement for simulation |
|
|
Hi,
I am a graduate student doing some research in processor simulation.
Modern benchmarks are usually very long, so it is almost impossible to
simulate the complete run with cycle-accurate simulator. People have
proposed techniques to reduce the benchmark. Reducing simulation time
will result in reduced accuracy. Here, "accuracy" does not refer to the
accuracy of the simulator model. It only refers to how well the reduced
benchmark represents the original benchmark. My question is: what
accuracy is usually required in real world? Many people with extensive
industry experience frequent this news group so I figured some of you
may help me.
I can think of two factors that make very high accuracy unnecessary.
First, the orginal benchmark is only a sample from the real world
programs. Take SPECcpu2000 for example. It can be considered a sample
from all cpu-intensive programs. SPECint score 1379 (machine A) and 1385
(machine B) may not make a difference because when running other
cpu-intensive programs machine A is possibly faster than machine B. In
other word, if we view SPECcpu2000 as a sample from all cpu-intensive
programs (i.e. the population), and our goal is to estimate the average
speed of running all possible cpu-intensive programs (i.e. population
mean), then we can only estimate the population mean to a certain range
with a certain confidence level. The difference between machine A and B
may very well be statistically insiginificant.
The other factor is user's perception. Does the user really care
about 2% speed difference?
Any comment based on your real world experience is appreciated.
Yue |
|
John Dallman
Guest
|
| Posted:
Thu Dec 09, 2004 12:28 am Post subject:
Re: accuracy requirement for simulation |
|
|
In article <cp5atq$ggt$2@geraldo.cc.utexas.edu>, nospam@nospam.nospam (Yue
Luo) wrote:
| Quote: | I can think of two factors that make very high accuracy unnecessary.
First, the orginal benchmark is only a sample from the real world
programs.
|
Yes, and is not fully representative - it can't be, given the
simplifications involved in making a standardised, repeatable benchmark. I
keep some track of how SPEC figures relate to the performance of the code
I work on, and the correlations are fairly approximate. SPEC gives you a
general idea, but it definitely isn't something to base a choice of
processor, compiler or coding technique on by itself.
| Quote: | The other factor is user's perception. Does the user really care
about 2% speed difference?
|
I work on a big math library, which is performance-critical in a fair
number of applications. Our attitude to performance depends a lot on if
it's being added or subtracted.
Adding 2% throughput is nice, but there are definite limits to how much
trouble and expense it's worth. It certainly isn't worth having to build
and test an extra platform, for example; that starts becoming worthwhile
at about +25% throughout. We haven't taken up profile-guided optimisation,
because it slows down the compile-link-test cycle so much that you lose
gains that you could make via algorithm tuning.
Taking performance away from customers goes down very badly. For example,
our Windows version used to be built on Visual C++ v6 and tuned for P6
(Pentium Pro/II/III family) processors. When Visual C++ v7.1 came out, and
could tune for P7 (Pentium 4 family) processors, the extra performance on
those was good to have, but if the losses on P6 had been reliably
measurable by customers (say, over 2%) we'd have worried about introducing
it. Some customers will benchmark every patch release, and complain if
anything gets slower. This isn't altogether reasonable, and we argue back
if only one or two functions have slowed down and the rest speeded up, but
they're the ones who are paying. In fact, we didn't have that problem,
because P7 tuning on VC71 produces code that runs faster on a P6 than
P6-tuned code from VC6 - the optimiser is generally rather better - but it
could have been sticky.
So, an inaccurate simulation that says choice A will be 5% faster then
choice B, when it's actually the other way around probably won't cause a
disaster, because anyone sensible will confirm the results before they
commit on anything. But it will mean that they'll only trust the simulator
for a broad idea of what's going on.
---
John Dallman jgd@cix.co.uk
"Any sufficiently advanced technology is indistinguishable from a
well-rigged demo" |
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in