| Author |
Message |
Lee Witten
Guest
|
 Posted:
Sun Jan 30, 2005 12:43 am Post subject:
History of performance counters (was: New Linux Power5 even |
|
|
anton@mips.complang.tuwien.ac.at (Anton Ertl) wrote in
news:2005Jan25.091041@mips.complang.tuwien.ac.at:
| Quote: | An advantage of the PPC74xx over the PPC970 is that information about
the performance monitoring counters is publically available, and
therefore the Linux perfctr patch supports the PPC74xx, whereas IBM
has not provided that information for the PPC970, and therefore the
perfctr patch does not support the PPC970. That's fairly important
for me, and I would expect it to be even more important for Terje.
(For more information on how to use perfctr's perfex on the PPC7450
familiy, see
http://www.complang.tuwien.ac.at/anton/linux-perfex/ppc7450.html>).
|
This posting made me wonder about the history of performance counters in
general, and the history of providing public information of their use.
I was an IBM employee in the late 80s / early 90s. At one point, I was
working on UNIX for the 3090 mainframes, and I saw a guy carrying a
listing several inches thick down the hall. I asked a co-worker who he
was and what he was doing, and he said he was from IBM Research, and the
listing was a dump of performance counters gathered when the kernel was
running, and he was analyzing them. I asked how I could do this, and
was told that was confidential info, and only certain priviledged folks
could learn how.
At a similar point in time, I was logged on to a certain system within
IBM, and I read that the first generation RS/6000s had some sort of
performance counters that could only be read if you had a specially
modified motherboard with special probes hooked up to read them. I
never found out any more about them.
I was at DEC during the early Alpha days (1992 or so), and they had
performance counters on the first Alpha chip and all the follow ons. I
recall we documented how they worked via the usual programming
references i.e. man pages for uprofile and kprofile on OSF/1 and
documents on IPROBE for VMS. But I don't think we particularly
documented how a third party could use them, although I think there was
obscure information on the kprof system call in the OSF/1 man pages.
I recall reading there may have been some profile counter support on the
first pentium (p5) and there definitely were on the second (p6).
So, I'm wondering if there are any earlier implementations of
performance counters, and when they were publically described. Were the
counters generally available to third parties, or where they only really
usable via vendor-provided tools. Who was the first to provide the
'interrupt every N events of interest' style of performance counters?
Are there any relevant papers, patents, etc?
--lw-- |
|
| Back to top |
|
 |
Anne & Lynn Wheeler
Guest
|
| Posted:
Sun Jan 30, 2005 1:19 am Post subject:
Re: History of performance counters |
|
|
Lee Witten <lw99@yahoo.com> writes:
| Quote: | This posting made me wonder about the history of performance counters in
general, and the history of providing public information of their use.
I was an IBM employee in the late 80s / early 90s. At one point, I was
working on UNIX for the 3090 mainframes, and I saw a guy carrying a
listing several inches thick down the hall. I asked a co-worker who he
was and what he was doing, and he said he was from IBM Research, and the
listing was a dump of performance counters gathered when the kernel was
running, and he was analyzing them. I asked how I could do this, and
was told that was confidential info, and only certain priviledged folks
could learn how.
At a similar point in time, I was logged on to a certain system within
IBM, and I read that the first generation RS/6000s had some sort of
performance counters that could only be read if you had a specially
modified motherboard with special probes hooked up to read them. I
never found out any more about them.
I was at DEC during the early Alpha days (1992 or so), and they had
performance counters on the first Alpha chip and all the follow ons. I
recall we documented how they worked via the usual programming
references i.e. man pages for uprofile and kprofile on OSF/1 and
documents on IPROBE for VMS. But I don't think we particularly
documented how a third party could use them, although I think there was
obscure information on the kprof system call in the OSF/1 man pages.
I recall reading there may have been some profile counter support on the
first pentium (p5) and there definitely were on the second (p6).
So, I'm wondering if there are any earlier implementations of
performance counters, and when they were publically described. Were the
counters generally available to third parties, or where they only really
usable via vendor-provided tools. Who was the first to provide the
'interrupt every N events of interest' style of performance counters?
Are there any relevant papers, patents, etc?
|
back in the 360 & 370 days when you could still attach probes to the
computer ... there was a special hardware box that had lots of probes
.... which you could hook up all over the computer ... and reduce for
all sorts of statistics. one of the things you could do was attach
probes to the instruaction address and create a log of sampled
instruction addresses ... in attempt to build up a profile of where
the computer was spending all its time.
note it was also possible to display the current instruction address
in lights ... and i believe it was boeing wichita that trained a video
recorder on the instruction address lights as a "cheap" performance
monitor.
a problem with TCMs and advancing technology ... it was no longer
possible to place probes at will thruout the computer. This sort of
eliminated the old-style performance monitors but also created a
problem for field engineering. For years, field service/sengineering
had a requirement that they could boot-strap diagnose hardware
problems starting with scope probes. As a result, the service
processors were born. Service processors were simpler computing
technology that could be scoped ... and in turn the service
processor(s) had builtin probes at manufactoring for everything else
needing to diagnose hardwrae problems.
Initially, 3081 had a uc.5 service processor. The follow-on 3090
initially was going to a 4331 as a service processor running a highly
customized version of vm/370 release 6. This was upgraded to having
dual 4361s as the 3090 service processors, both running highly
customized version of vm/370 release 6.
--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn/ |
|
| Back to top |
|
|
Lee Witten
Guest
|
| Posted:
Sun Jan 30, 2005 1:50 am Post subject:
Re: History of performance counters |
|
|
Anne & Lynn Wheeler <lynn@garlic.com> wrote in
news:m3651gq98c.fsf@lhwlinux.garlic.com:
| Quote: | a problem with TCMs and advancing technology ... it was no longer
possible to place probes at will thruout the computer. This sort of
eliminated the old-style performance monitors but also created a
problem for field engineering. For years, field service/sengineering
had a requirement that they could boot-strap diagnose hardware
problems starting with scope probes. As a result, the service
processors were born. Service processors were simpler computing
technology that could be scoped ... and in turn the service
processor(s) had builtin probes at manufactoring for everything else
needing to diagnose hardwrae problems.
|
Interesting stuff. I think the listings I mentioned in my other posting
were from the service processor on the 3090. As I said, the details of how
to use it seemed to be closely held.
--lw-- |
|
| Back to top |
|
|
Terje Mathisen
Guest
|
| Posted:
Sun Jan 30, 2005 2:20 am Post subject:
Re: History of performance counters (was: New Linux Power5 e |
|
|
Lee Witten wrote:
| Quote: | I recall reading there may have been some profile counter support on the
first pentium (p5) and there definitely were on the second (p6).
So, I'm wondering if there are any earlier implementations of
performance counters, and when they were publically described. Were the
|
The Pentium counters were undocumented, except by NDA, so it took me a
year from I first heard about it until I had reverse engineered them and
documented their working for a Byte article in 1994.
http://www.byte.com/art/9407/sec12/art3.htm
Terje
--
- <Terje.Mathisen@hda.hydro.com>
"almost all programming can be viewed as an exercise in caching" |
|
| Back to top |
|
|
Anne & Lynn Wheeler
Guest
|
| Posted:
Sun Jan 30, 2005 7:55 am Post subject:
Re: History of performance counters |
|
|
artie <artie.m@gmail.com> writes:
| Quote: | Back in the Days of Yore (mid 70's), CP-V from SDS had a large
number of software performance counters, and tuning parameters to go
with them. Sufficiently privileged users could view and diddle
things. Time quanta for timesharing users, for batch jobs, priority
increments for the scheduler, tracking I/O operations per second,
number ofthings in different states and queues, all sorts of fun
stuff.
|
when i did the original dynamic adaptive stuff back when i was an
undergraduate ... generalized resource allocation policy, attempting
to dynamically determine the bottlenecks, default policy was called
fair share, etc ... it got dropped into cp67. in the morphing from
cp67 to vm370 ... a lot of the stuff got lost ... so i got
an opportunity to re-introduce it with the resource manager
.... recent posting about the resource manager
http://www.garlic.com/~lynn/2005b.html#49 The mid-seventies SHARE survey
..... however, some people said that I couldn't put out a resource
manager that didn't have tuning knobs ... because the most modern
state of the art performance management stuff had lots & lots of
tuning knobs. unfortunately ... i had done a whole lot of stuff with
self monitoring, dynamic adaptive, etc. ... so I put in some turning
knobs to make it look a lot more modern ... give people lots and lots
of manual activity ... instead of just doing the right thing
dynamically adaptive continously as configuration and workload
changed. So there were official product documents describing all the
formulas ... and all the source was readily available (i.e. source
distribution and source maintenance back them).
so there was this little joke ... right in plain site in the code ...
.... in OR relms it is sometimes referred to as degrees of freedom
.... guess whether the manual knobs or the automatic dynamic adaptive
feedback values had the greater degrees of freedom?
--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn/ |
|
| Back to top |
|
|
Lee Witten
Guest
|
| Posted:
Sun Jan 30, 2005 7:55 am Post subject:
Re: History of performance counters |
|
|
Tom Van Vleck <thvv@multicians.org> wrote in
news:thvv-0E3590.23024129012005@comcast.dca.giganews.com:
| Quote: | Your post seems to cover both hardware and software
counters. Hardware counters were done by a device that
hooked into the backplane of a Honeywell 6180, called a
"yellow submarine."
|
Sorry I wasn't clearer. I did mean hardware performance counters. Where I
mentioned software, I meant the software that was used to enable the
counters and analyze the data they generated.
| Quote: | Regarding "interrupt every N" metering, a command called
"snoop" was part of Multics metering, which interrupted
every N seconds and sampled the PC. It was written by
Ross Klinger when he worked for me at MIT IPC, about 1972.
|
So, what did the command do with the PC values? I imagine one could
capture the output then correlate the PC values to the instructions in the
program(s) being run. Was anything else done with them?
--lw-- |
|
| Back to top |
|
|
Lee Witten
Guest
|
| Posted:
Sun Jan 30, 2005 7:55 am Post subject:
Re: History of performance counters (was: New Linux Power5 e |
|
|
Thanks, Artie. Very interesting stuff.
artie <artie.m@gmail.com> wrote in news:290120052136004781%
artie.m@gmail.com:
| Quote: | Rewriting that particular routine resulted in a dramatic improvement in
loader performance -- not a 10%, 20% kind of thing, but 10x, 12x, 15x.
|
I only had one thing like that. A fortran program ran much slower on one
OS versus another on the same hardware. At first it was thought it was the
OS kernel doing something wrong on the slow OS, but I quickly found it was
the runtime library routine to print the decimal point used when printing
numbers in Fortran. Just like you said, I could not figure out how someone
could code that in a more inefficient way. Fixing it resulted in a 9x
performance improvement.
In another case, I got something like an 8x speedup on a machine with 14
cpus. It was due to an unintended sequence that would cause some critical
code to run on one and only one of the 14 cpus.
--lw-- |
|
| Back to top |
|
|
Lee Witten
Guest
|
| Posted:
Sun Jan 30, 2005 7:55 am Post subject:
Re: History of performance counters |
|
|
"del cecchi" <dcecchi.nojunk@att.net> wrote in news:3632gnF4smft6U1
@individual.net:
| Quote: | I think in Poughkeepsie the location of the rest rooms was IBM
Confidential. :-)
|
When I worked there, every terminal and computer had a sticker that said
"This equipment can be used for IBM Mangement Approved Purposes only", or
some such statement. And I got a huge laugh when someone put one of those
stickers on all the urinals in the IBM Mens Room! Such naughtyness was
rare at IBM!
Actually, when I was working on UNIX at IBM, someone covered up the large
outdoors IBM sign with an AT&T logo on one side and a Sun logo on the
other. In fact I knew the joker who did this. And now, 20 or so years
later, I'm wondering if he did the urinal thing too.
--lw-- |
|
| Back to top |
|
|
artie
Guest
|
| Posted:
Sun Jan 30, 2005 7:55 am Post subject:
Re: History of performance counters (was: New Linux Power5 e |
|
|
Two tales, of software performance monitors, and hardware.
Back in the Days of Yore (mid 70's), CP-V from SDS had a large number
of software performance counters, and tuning parameters to go with
them. Sufficiently privileged users could view and diddle things.
Time quanta for timesharing users, for batch jobs, priority increments
for the scheduler, tracking I/O operations per second, number ofthings
in different states and queues, all sorts of fun stuff.
Of all the counters and tuning parameters, we quickly came to realize
that one was more powerful than all the rest.
That was SL:BB
SL:BB was a tuning parameter, with range 0-100, controlling batch bias
in scheduling.
What made it so powerful, and so important? It gave us one of the best
diagnostic tools we had. How?
It wasn't hooked up to anything! The needed system code didn't get
done, but the display/tuning code was there!
A site manager (customer) calls in with performance questions. Have
you run some tests? Sure, the answer was *always* yes. Okay, re-run
them, varying SL:BB and see what happens. Best performance? Worst?
Let us know.
Every so often, one would call back and confess that whatever they did
with SL:BB, it didn't alter performance significantly. Friend!
Comrade! How can we help you?
But more often than not, we'd get calls back that a value of 37, or 81,
or some gibberish produced dramatic changes in results. Yeah, right...
We also had a hardware performance monitoring facility, a clip-on box
called Diamond. Wish I could remember Jim's last name -- a gentleman
and a scholar. Diamond was an associative box, full of
content-addressible memory. Hook it up to a running system, and it
could tell you all sorts of very interesting things. We used it for a
number of things, such as identifying the "unclean 13," the 13 longest
interrupt disable paths in the operating system (CP/V) -- we were
supposed to be multi-purpose, real time, remember? Long disables were
hunted down.
Diamond also helped us with performance tuning the remainder of the
system (UTS and CP-V, and was used on other systems as well).
Thanks to Diamond, one very talented programmer (John) found that a
particular routine in the Loader (linking overlay loader, used to build
big executable images, including the system image), had a key routine,
written in the year zero, that did things in just about the *slowest*
way anyone could figure. I remember a bunch of us sitting around one
afternoon trying to figure out how to get that chunk of code to go
*slower* and we couldn't do it without doing something stupid.
And it wasn't that the code was *wrong* -- it just made bad
assumptions. The code in particular forced worst-case paths (and
times) in the hardware. It also performed checks that hadn't been
needed for many, many generations of the system. Again, nothing wrong,
just needless leftovers.
Rewriting that particular routine resulted in a dramatic improvement in
loader performance -- not a 10%, 20% kind of thing, but 10x, 12x, 15x.
--
Namaste-- |
|
| Back to top |
|
|
Tom Van Vleck
Guest
|
| Posted:
Sun Jan 30, 2005 7:55 am Post subject:
Re: History of performance counters |
|
|
Lee Witten <lw99@yahoo.com> writes:
| Quote: | So, I'm wondering if there are any earlier implementations of
performance counters, and when they were publically described. Were the
counters generally available to third parties, or where they only really
usable via vendor-provided tools. Who was the first to provide the
'interrupt every N events of interest' style of performance counters?
Are there any relevant papers, patents, etc?
|
The Multics hardcore supervisor had tons of performance
counters in ring zero, and the system was provided with
many metering commands that read and formatted them.
Access to these counters was governed by the ACL to
metering_gate_; different sites had different policies
about how to set this.
The initial design for metering was Don Widrig's document
M0062, System Metering, dated 3/17/66.
The Multics metering philosophy and implementation is
described in Saltzer, J. H., and J. W. Gintell, The
instrumentation of Multics, Commun. ACM 13, No. 8, 495-500,
August 1970. The paper is available online as
http://www.multicians.org/InstrumentationPaper.html
"abstract: An array of measuring tools devised to aid in
the implementation of a prototype computer utility is
discussed. These tools include special hardware clocks and
data channels, general purpose programmed probing and
recording tools, and specialized measurement facilities.
Some particular measurements of interest in a system which
combines demand paging with multiprogamming are described
in detail. Where appropriate, insight into effectiveness
(or lack thereof) of individual tools is provided."
Honeywell manual AN52, Multics System Metering, February
1979, describes the metering commands.
Your post seems to cover both hardware and software
counters. Hardware counters were done by a device that
hooked into the backplane of a Honeywell 6180, called a
"yellow submarine."
Regarding "interrupt every N" metering, a command called
"snoop" was part of Multics metering, which interrupted
every N seconds and sampled the PC. It was written by
Ross Klinger when he worked for me at MIT IPC, about 1972. |
|
| Back to top |
|
|
del cecchi
Guest
|
| Posted:
Sun Jan 30, 2005 7:55 am Post subject:
Re: History of performance counters |
|
|
"Lee Witten" <lw99@yahoo.com> wrote in message
news:Xns95EDA12E2FA4Bnn48@199.125.85.9...
| Quote: | Anne & Lynn Wheeler <lynn@garlic.com> wrote in
news:m3651gq98c.fsf@lhwlinux.garlic.com:
a problem with TCMs and advancing technology ... it was no longer
possible to place probes at will thruout the computer. This sort of
eliminated the old-style performance monitors but also created a
problem for field engineering. For years, field service/sengineering
had a requirement that they could boot-strap diagnose hardware
problems starting with scope probes. As a result, the service
processors were born. Service processors were simpler computing
technology that could be scoped ... and in turn the service
processor(s) had builtin probes at manufactoring for everything else
needing to diagnose hardwrae problems.
Interesting stuff. I think the listings I mentioned in my other
posting
were from the service processor on the 3090. As I said, the details
of how
to use it seemed to be closely held.
--lw--
|
I think in Poughkeepsie the location of the rest rooms was IBM
Confidential. :-)
In Rochester, the idea that the CE would use a scope was totally
foreign. We were lucky if we could get them to use a voltmeter. There
was a "logic probe" switchable for MST or SLT that would light a light
to show 1 or 0 on a pin.
del cecchi |
|
| Back to top |
|
|
Tom Van Vleck
Guest
|
| Posted:
Sun Jan 30, 2005 6:45 pm Post subject:
Re: History of performance counters |
|
|
Lee Witten <lw99@yahoo.com> wrote:
| Quote: | Tom Van Vleck <thvv@multicians.org> wrote
Regarding "interrupt every N" metering, a command called
"snoop" was part of Multics metering, which interrupted
every N seconds and sampled the PC. It was written by
Ross Klinger when he worked for me at MIT IPC, about 1972.
So, what did the command do with the PC values? I imagine one could
capture the output then correlate the PC values to the instructions in the
program(s) being run. Was anything else done with them?
|
I forget. snoop saved the PC values in a segment and printed
a histogram at end of run with the locations decoded back
to segment name and offset. I think if symbol information
was available in a program, it would display source line
number: can't remember if it would also display the source
line itself. |
|
| Back to top |
|
|
CBFalconer
Guest
|
| Posted:
Sun Jan 30, 2005 8:09 pm Post subject:
Re: History of performance counters |
|
|
Tom Van Vleck wrote:
| Quote: | Lee Witten <lw99@yahoo.com> writes:
So, I'm wondering if there are any earlier implementations of
performance counters, and when they were publically described.
Were the counters generally available to third parties, or where
they only really usable via vendor-provided tools. Who was the
first to provide the 'interrupt every N events of interest'
style of performance counters? Are there any relevant papers,
patents, etc?
The Multics hardcore supervisor had tons of performance
counters in ring zero, and the system was provided with
many metering commands that read and formatted them.
Access to these counters was governed by the ACL to
metering_gate_; different sites had different policies
about how to set this.
|
In the middle to late '70s I built a comprehensive profiling
facility for both machine code and pcode for my 8080 based systems
at Yale. This was possible because I had built hardware that
included programmable interval counters that could interrupt. The
data collected had to be somewhat preprocessed (one location
counted interrupts in a range of addresses, and that overall range
was limited). The mechanism included a 'startprofile' and a
'stopprofile' call together with 'initprofile(params)' and a
'dumpprofile(file)' operation. Auxiliary software processed the
dumped profile.
Worked quite well, and allowed me to close in on the hot spots. I
believe it is mentioned in the PascalP manual from those days that
is mounted on my pages.
--
"If you want to post a followup via groups.google.com, don't use
the broken "Reply" link at the bottom of the article. Click on
"show options" at the top of the article, then click on the
"Reply" at the bottom of the article headers." - Keith Thompson |
|
| Back to top |
|
|
Jan Vorbrüggen
Guest
|
| Posted:
Mon Jan 31, 2005 1:48 pm Post subject:
Re: History of performance counters (was: New Linux Power5 e |
|
|
I believe the Crays has a set of hardware performance counters, and that
the interface on using them way documented. Anybody remember details?
Jan |
|
| Back to top |
|
|
K Williams
Guest
|
| Posted:
Mon Jan 31, 2005 9:13 pm Post subject:
Re: History of performance counters |
|
|
In article <Xns95EEB4AFBCD5nn48@199.125.85.9>, lw99@yahoo.com says...
| Quote: | "del cecchi" <dcecchi.nojunk@att.net> wrote in news:3632gnF4smft6U1
@individual.net:
I think in Poughkeepsie the location of the rest rooms was IBM
Confidential. :-)
|
And the paper was so marked. ;-)
| Quote: | When I worked there, every terminal and computer had a sticker that said
"This equipment can be used for IBM Mangement Approved Purposes only", or
some such statement. And I got a huge laugh when someone put one of those
stickers on all the urinals in the IBM Mens Room! Such naughtyness was
rare at IBM!
|
Dunno, I've seen "designed for Microsoft Windows" stickers on urinals.
The P'ok swap-n-shop board had some interesting items for sale in the
early 90's; "Bikini, yellow pokadot; itsy-bitsy, teeny-weeny style" and
others rather more crude.
| Quote: | Actually, when I was working on UNIX at IBM, someone covered up the large
outdoors IBM sign with an AT&T logo on one side and a Sun logo on the
other. In fact I knew the joker who did this. And now, 20 or so years
later, I'm wondering if he did the urinal thing too.
|
On black-Tuesday (first layoffs in '93) in E. Fishkill someone came in
in yellow and red tights and cape with a big 'S' on the front. Yes,
"SurplusMan" was surplussed. :-(
--
Keith |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in