CASTalk.com

Brendan · Guest

Hi,

I'm looking for information (datasheets, specifications, standards, etc)
regarding the IBM/Sequent Numa-Q architecture.

Specifically, anything containing details of how to detect the relationships
between CPUs, APICs, memory and I/O buses, details of the OEM MPC structure
(an extension to Intel's MP Specification), or any hardware differences
between Numa-Q and normal SMP (and/or AMD style NUMA) systems.

Failing that, similar information on any large scale server that uses 80x86
CPUs and hierarchical/cluster mode APICs (rather than "flat mode", as is
used in small scale SMP/NUMA) will suffice - Numa-Q, Summit, Summit II,
Horus(?), etc.

Notes: I've already written an 80x86 kernel which supports SMP, AMD style
NUMA, hyper-threading, etc, and intend to rewrite this kernel with
additional support for large scale servers. The idea is that by attempting
to implement support for one of these servers, the design and framework of
the kernel will become adequate to eventually support all of them
(regardless of whether the initial code works on anything in practice or
not - it's about the design of the code, not the code itself).

I have been unable to find anything relevant (marketing material rather than
documentation a developer can use to implement code). Any help
appreciated...

Thanks,

Brendan

Andi Kleen · Guest

"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> writes:

[quote]I'm looking for information (datasheets, specifications, standards, etc)
regarding the IBM/Sequent Numa-Q architecture.
[/quote]
I suspect the only public reference for that is the Linux kernel source
(search for CONFIG_X86_NUMAQ and )

[quote]Specifically, anything containing details of how to detect the relationships
between CPUs, APICs, memory and I/O buses, details of the OEM MPC structure
(an extension to Intel's MP Specification), or any hardware differences
between Numa-Q and normal SMP (and/or AMD style NUMA) systems.
[/quote]
Intel MP is quite outdated and many of the modern systems don't support
it anymore. It's all in ACPI now. ACPI has special tables for all of this.

Implementing ACPI parsing is unfortunatenly quite complicated and
pretty much needs the Intel ACPI reference implementation as base (and
even then it's a daunting task)

[quote]Failing that, similar information on any large scale server that uses 80x86
CPUs and hierarchical/cluster mode APICs (rather than "flat mode", as is
used in small scale SMP/NUMA) will suffice - Numa-Q, Summit, Summit II,
Horus(?), etc.
[/quote]
Many of the modern big systems can actually use flat mode even for
larger setups. Cluster mode is only needed for a few older Intel
systems anymore and is probably getting phased out (it really never
made much sense in the newer XAPIC model)

[quote]Notes: I've already written an 80x86 kernel which supports SMP, AMD style
NUMA, hyper-threading, etc, and intend to rewrite this kernel with
additional support for large scale servers. The idea is that by attempting
to implement support for one of these servers, the design and framework of
the kernel will become adequate to eventually support all of them
(regardless of whether the initial code works on anything in practice or
not - it's about the design of the code, not the code itself).
[/quote]
The NUMAQ is not a good model for that. It is very un PC like and requires
a lot of own special case code. The machine is really very dated now.
Also optimizing for it is not a good idea because it has a really bad
NUMA factor while most other systems are not that bad and behave much better.
It's commonly called Numasaurus because of all that. Best is probably
to let it rest in peace.

Modern big x86 machines tend to be configured in a far more
standard way using ACPI, and follow the Intel standards much closer.

-Andi

Anne & Lynn Wheeler · Guest

"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> writes:
[quote]I'm looking for information (datasheets, specifications, standards,
etc) regarding the IBM/Sequent Numa-Q architecture.
[/quote]
numa-q was one of the sci 64-port memory things ... convex did one
with 64 2-way pa/risc boards (128 pa/risc processor), sequent and DG
did ones with 64 4-way intel processor boards (256 intel processors).

convex did a mach flavor (aka similar mach to what was in NeXT and
apple).

sequent did version of their dynix ... all this well before IBM bought
sequent.

We went to sci meetings ... but since power/pc wasn't out yet ... there
was no processor chip that supported cache consistency ... that was
somewhat why we did ha/cmp ... from cluster standpoint ... since
we could do scale-up w/o requiring cache consistency stuff.
http://www.garlic.com/~lynn/subtopic.html#hacmp

sci web site
http://www.scizzl.com/

some random past sci postings:
http://www.garlic.com/~lynn/2001f.html#11 Climate, US, Japan & supers query
http://www.garlic.com/~lynn/2002j.html#45 M$ SMP and old time IBM's LCMP
http://www.garlic.com/~lynn/2003.html#6 vax6k.openecs.org rebirth
http://www.garlic.com/~lynn/2004.html#1 Saturation Design Point
http://www.garlic.com/~lynn/2004d.html#68 bits, bytes, half-duplex, dual-simplex, etc
http://www.garlic.com/~lynn/2005.html#40 clusters vs shared-memory (was: Re: CAS and LL/SC (was Re: High Level Assembler for MVS & VM & VSE))
http://www.garlic.com/~lynn/2005f.html#18 Is Supercomputing Possible?
http://www.garlic.com/~lynn/2005h.html#13 Today's mainframe--anything to new?
http://www.garlic.com/~lynn/2005n.html#4 54 Processors?
http://www.garlic.com/~lynn/2005n.html#6 Cache coherency protocols: Write-update versus write-invalidate
http://www.garlic.com/~lynn/2005n.html#38 What was new&important in computer architecture 10 years ago ?

--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn/

Del Cecchi · Guest

Brendan wrote:
[quote]Hi,

I'm looking for information (datasheets, specifications, standards, etc)
regarding the IBM/Sequent Numa-Q architecture.

Specifically, anything containing details of how to detect the relationships
between CPUs, APICs, memory and I/O buses, details of the OEM MPC structure
(an extension to Intel's MP Specification), or any hardware differences
between Numa-Q and normal SMP (and/or AMD style NUMA) systems.

Failing that, similar information on any large scale server that uses 80x86
CPUs and hierarchical/cluster mode APICs (rather than "flat mode", as is
used in small scale SMP/NUMA) will suffice - Numa-Q, Summit, Summit II,
Horus(?), etc.

Notes: I've already written an 80x86 kernel which supports SMP, AMD style
NUMA, hyper-threading, etc, and intend to rewrite this kernel with
additional support for large scale servers. The idea is that by attempting
to implement support for one of these servers, the design and framework of
the kernel will become adequate to eventually support all of them
(regardless of whether the initial code works on anything in practice or
not - it's about the design of the code, not the code itself).

I have been unable to find anything relevant (marketing material rather than
documentation a developer can use to implement code). Any help
appreciated...

Thanks,

Brendan

When I searched redbooks for numa-q I came up with SG24-6215-00 that has[/quote]
some info.

Also TIPS0476 or SG24-5287-03 or SG24-6411-00 might be of interest.

--
Del Cecchi
"This post is my own and doesn’t necessarily represent IBM’s positions,
strategies or opinions.”

Del Cecchi · Guest

"Anne & Lynn Wheeler" <lynn@garlic.com> wrote in message
news:m3u0fnr7wa.fsf@lhwlinux.garlic.com...
[quote]"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> writes:
I'm looking for information (datasheets, specifications, standards,
etc) regarding the IBM/Sequent Numa-Q architecture.

numa-q was one of the sci 64-port memory things ... convex did one
with 64 2-way pa/risc boards (128 pa/risc processor), sequent and DG
did ones with 64 4-way intel processor boards (256 intel processors).

convex did a mach flavor (aka similar mach to what was in NeXT and
apple).

sequent did version of their dynix ... all this well before IBM bought
sequent.

We went to sci meetings ... but since power/pc wasn't out yet ... there
was no processor chip that supported cache consistency ... that was
somewhat why we did ha/cmp ... from cluster standpoint ... since
we could do scale-up w/o requiring cache consistency stuff.
http://www.garlic.com/~lynn/subtopic.html#hacmp

sci web site
http://www.scizzl.com/

Back in those days, the processors were groups of chips and Rochester was[/quote]
going to NUMA the AS400 until a certain IBM Fellow convinced the Powers
and Principalities that we couldn't possibly pull it off because POK had
hundreds of folks doing sysplex so our estimates must be incorrect. So
AS400 did a 24 way SMP and the numa thing got killed, but was
reincarnated as the Summit Xseries stuff. This was in the early 90's,
pre powerPC. There was an intermediate effort with Austin for RS but it
died with a change in management.

del

Anne & Lynn Wheeler · Guest

"Del Cecchi" <dcecchi.nospam@att.net> writes:
[quote]Back in those days, the processors were groups of chips and Rochester was
going to NUMA the AS400 until a certain IBM Fellow convinced the Powers
and Principalities that we couldn't possibly pull it off because POK had
hundreds of folks doing sysplex so our estimates must be incorrect. So
AS400 did a 24 way SMP and the numa thing got killed, but was
reincarnated as the Summit Xseries stuff. This was in the early 90's,
pre powerPC. There was an intermediate effort with Austin for RS but it
died with a change in management.
[/quote]
i think we have just the bare tip of the iceberg here

very strong memory consistency was so very strongly ingrained in certain
quarters ... in part the whole 801 non-cache consistency was to take
the opposite extreme (can't have smp operation at all).

this is my sporadic reference to the pok advance tech conference in
1976 ... where we talked about 16-way 370 smp w/o strong memory
consistency ... and it was the first time i saw the 801 presentation.

the 16-way wasn't strong memory consistency ... and eventually some of
mvs proponents realized that it would never be a mvs machine. we had
co-opted some of the 3033 processor engineers to do this as spare time
effort. i think they may have then got counseling about remaining
focused (3033 wasn't shipping yet and didn't even have a working
engineering model). some people got told they were no longer welcome
on pok plantsite.

there was some issue with the concept of a spectrum of memory
consistency ... as opposed all or nothing.

there was some rumors that early 370 smps had some special fixes
for immediate instructions ... and-immediate, or-immediate, etc.
the original 360 smp serialization instruction was test&set.

charlie invented compare&swap at the science center doing some
smp fine-grain locking work on cp67 (had to come up with
"compare&swap" because CAS are charlie's initials)
http://www.garlic.com/~lynn/subtopic.html#sm

which also made it into 370. in any case, none of the immediate
instructions are defined as smp consistent (they do byte operate
fetch, modify, store) ... but the rumor was that their use in certain
operating system kernels was so pervasive and couldn't be corrected
.... that the hardware may have been required to fix kernel memory
consistency issues related to the use of immediate instructions (lot of
flag, bit, and status maintenance).

another rumor was that the caches in 3090s ran ten times faster cycle
than the standard machine cycle ... in order to provide all the stuff
for strong memory consistency. pok was beginning to face needing to
have infinitely fast cache machine cycle for their scale-up (since
there was enormous amounts of cross-cache chatter going on between all
the processor caches). two-way 370 smp machines started off running
base machine cycle ten percent slower than uniprocessor .... just so
the caches could process all the cross-cache chatter from one other
cache. going to six-way smp met that a cache was getting cross-cache
chatter from five other caches ... implying that the machine cycle
possibly has to be cut in half ... or the caches had to run much, much
faster than the rest of the machine. trying to come up with caches
that operate infinitely fast was indeed consuming large amounts of
resources.

--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn

Brendan · Guest

Hi,

From: "Andi Kleen" <freitag@alancoxonachip.com>
[quote]"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> writes:

I'm looking for information (datasheets, specifications, standards, etc)
regarding the IBM/Sequent Numa-Q architecture.

I suspect the only public reference for that is the Linux kernel source
(search for CONFIG_X86_NUMAQ and )
[/quote]
I'm thinking your right...

I'd much rather do things from datasheets, etc though - people make choices
when implementing things, such that the eventual implementation only
contains information relevant to those choices, rather than allowing
different choices to be made. For example, during boot I auto-detect which
"kernel modules" to use, rather than expecting users to compile a version
of the kernel specifically for their system (AFAIK Linux doesn't have code
to
auto-detect Numa-Q). There's many such differences between the (monolithic)
Linux kernel and my own "modular micro-kernel" design.

[quote]Specifically, anything containing details of how to detect the
relationships
between CPUs, APICs, memory and I/O buses, details of the OEM MPC
structure
(an extension to Intel's MP Specification), or any hardware differences
between Numa-Q and normal SMP (and/or AMD style NUMA) systems.

Intel MP is quite outdated and many of the modern systems don't support
it anymore. It's all in ACPI now. ACPI has special tables for all of this.

Implementing ACPI parsing is unfortunatenly quite complicated and
pretty much needs the Intel ACPI reference implementation as base (and
even then it's a daunting task)
[/quote]
Parsing the ACPI tables isn't much harder than parsing Intel's MP tables
(it's the AML stuff that's daunting).

It was a patch for Linux source code that led me to ask about OEM tables
(e.g. http://www.uwsg.iu.edu/hypermail/linux/kernel/0110.3/0440.html).

[quote]Failing that, similar information on any large scale server that uses
80x86
CPUs and hierarchical/cluster mode APICs (rather than "flat mode", as is
used in small scale SMP/NUMA) will suffice - Numa-Q, Summit, Summit II,
Horus(?), etc.

Many of the modern big systems can actually use flat mode even for
larger setups. Cluster mode is only needed for a few older Intel
systems anymore and is probably getting phased out (it really never
made much sense in the newer XAPIC model)
[/quote]
I guess I'm worried that if I do the design and early code without worrying
about cluster mode APICs I'll eventually find that retro-fitting support for
cluster mode APICs will break too much (i.e. the purpose of attempting to
support cluster mode APICs would be to avoid design flaws). This sort of
thinking tends towards either attempting to support cluster mode APICs from
the start or never supporting them (or taking my chances later on).

I do take the point though - never supporting cluster mode APICs may be the
most viable option...

In this case (no cluster mode APIC support ever), what would be the total
loss (other than Numa-Q)?

Cheers,

Brendan

Brig Campbell · Guest

"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> wrote in message
news:PVS2f.15249$U51.5092@news-server.bigpond.net.au...
[quote]
I have been unable to find anything relevant (marketing material rather
than
documentation a developer can use to implement code). Any help
appreciated...

[/quote]
The CTO for www.osdl.org is a "sequent guy" (there's a few of them hanging
around there). you could ping him, his email address is off the homepage in
the management section.

-brig

Andi Kleen · Guest

"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> writes:

[quote]I'd much rather do things from datasheets, etc though - people make choices
[/quote]
The available documentation on all of this is unfortunately quite
poor (on x86 APIC architectures in general). The Intel IA32
manual has the best overview, but it seems to leave out a lot of
detail. Good luck getting docs.

[quote]when implementing things, such that the eventual implementation only
contains information relevant to those choices, rather than allowing
different choices to be made. For example, during boot I auto-detect which
"kernel modules" to use, rather than expecting users to compile a version
of the kernel specifically for their system (AFAIK Linux doesn't have code
to
auto-detect Numa-Q).
[/quote]
It has code to auto detect all machines that still matter ;-)

[quote]I guess I'm worried that if I do the design and early code without worrying
about cluster mode APICs I'll eventually find that retro-fitting support for
cluster mode APICs will break too much (i.e. the purpose of attempting to
support cluster mode APICs would be to avoid design flaws). This sort of
thinking tends towards either attempting to support cluster mode APICs from
the start or never supporting them (or taking my chances later on).

I do take the point though - never supporting cluster mode APICs may be the
most viable option...
[/quote]
Implementing clustered is not very hard, just it's likely not needed.

[quote]In this case (no cluster mode APIC support ever), what would be the total
loss (other than Numa-Q)?
[/quote]
First NUMAQ would need much more support in other areas because it is in many
ways not PC compatible.

In Linux the Unisys ES7000 and IBM Summit1/2/3 based systems use clustered
mode, but I suspect they could all use flat physical mode too.
Everything else modern (P4+, AMD) works in flat physical mode AFAIK.

-Andi

Brendan · Guest

Hi,

"Andi Kleen" <freitag@alancoxonachip.com> wrote:
[quote]"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> writes:

I'd much rather do things from datasheets, etc though - people make
choices

The available documentation on all of this is unfortunately quite
poor (on x86 APIC architectures in general). The Intel IA32
manual has the best overview, but it seems to leave out a lot of
detail. Good luck getting docs.
[/quote]
You're right - from what I can gather, all Linux patches for both Numa-Q
and Summit were done by IBM themselves, and all patches for ES7000
were done by Unisys (with others making generic changes after), and none
of the information I want is available for any large scale 80x86 systems.

That leave me the choice of relying on Linux source, or only supporting
ACPI's SRAT and SLIT tables (with no reason to support cluster mode
for APICs). I'm thinking the latter might be my best option - it'd take me
too long to become familiar with Linux's source code, and by the time
my kernel is ready for real use these systems may have vanished entirely.

Thanks,

Brendan

Petter Gustad · Guest

"Brendan" <NOSPAMbtrotter@gmail.comNOSPAM> writes:

	CASTalk.com Forum Index -> Computer Architecture	All times are GMT
Page 1 of 1