CASTalk.com

In article <4393bb75$0$23327$5a62ac22@per-qv1-newsreader-
01.iinet.net.au>, markm@vl.com.au says...

Keith wrote:

That doesn't change the cacheability. Caches are a processor thing
and *NOT* under control of any PCI device.

Which begs the question, why would the PCI spec refer to something that
has nothing to do with PCI?

Short answer: Snoops from any PCI initiator into PCI cached memory
are in the purview of the spec. ;-)

Longer answer[*]: The SBO# (Snoop Back Off) and SDONE (snoop done)
signals are part of the pre-PCI2.2 spec (actually, I believe in 2.2
it's recommended that they not be used and pulled high). These are
used by the memory bridge to initiate retries to cached memory.
SDONE indicates an access to cached memory is complete. SBO# active
indicates a cached line is being accessed and the access must be
terminated by the initiator and retried later.

[*] I've ever used these things so I'm not real up on the spec
here. The performance is horrible so isn't often implemented and
less often used.

If a PCI memory space is marked as 'pre-fetchable' then it guarantees,
among other things, that the act of pre-fetching memory has no
side-effects. This means nothing more than the fact that it may be a
suitable candidate for caching, if the platform supports it. In this
case, a master may issue MRL (& MRM) commands.

prefetching <> cacheing

OTOH, cache-coherency (which I assume you're hinting at) is a different
problem altogether - especially if you've got multiple bus masters
accessing PCI memory space with their own caches. However, this is a
*system* problem and (IMHO) not really any concern of the PCI bus spec
group to mandate that PCI memory is not 'cacheable' - whatever that
means in each context!

But it *is* part of the (pre 2.2) spec. The bus must guarantee
coherency in this case. The way it does it is with back-offs and
retries. Now think about this with multiple bridges and
initiators. It gets to be a mess.

In fact, there's little discussion what-so-ever in the spec (that I can
see) about 'caches' - which is just what I would expect.

It's there in the older versions of the spec. As I've mentioned,
it's been deprecated in later versions.

BTW I'm quite happy to be shown the error in my reasoning!

--
Keith

On Sat, 03 Dec 2005 09:31:04 -0800, John Larkin
<jjlarkin@highNOTlandTHIStechnologyPART.com> wrote:

Consider a Pentium PC with main memory and a PCI bus. One can plug
memory-type devices into the PCI bus, things like video or ADC
buffers, CPCI cards, and including, I suppose, more program-space
memory.

A couple of questions:

Is there (I guess there must be) a mechanism for a Windows program to
directly map a chunk of PCI-bus memory into its virtual address space?
Anybody know how this works?

With Windows NT (2k/xp/2k3) applications can never access hardware directly.
(*) You will have to write a kernel mode device driver to control your piece
of hardware. The NT kernel mode api is very different from what you know
from the user mode Windows api. You will need the Microsoft Platform SDK and
the DDK and the Microsoft C compiler. The DDK contains samples and
documentation for everything. Try to minimize the user/kernel mode switches.

(*) There are kludges like giveio.sys etc, which allow user applications to
access io ports. Forget about these - it won't be enough for you.


Mit freundlichen Grüßen

Frank-Christian Krügel

John Larkin wrote:

Does anybody know how the BIOS decides what should be cached? There's
nothing in a device's PCI config registers that says "don't cache me"
as far as I can tell.

The BIOS doesn't. The device driver (which knows the PCI card
intimately) asks for non-cached memory when it asks for a virtual
mapping of the physical address range of the card. If that's
appropriate. If there's no reason why it can't be cached, it doesn't.

Steve

--------------------------------------------------------------------------------------------
Steve Schefter phone: +1 705 725 9999 x26
The Software Group Limited fax: +1 705 725 9666
642 Welham Road,
Barrie, Ontario CANADA L4N 9A1 Web: www.wanware.com

On 5 Dec 2005 13:31:00 -0800, steve_schefter@hotmail.com wrote:

John Larkin wrote:
Does anybody know how the BIOS decides what should be cached? There's
nothing in a device's PCI config registers that says "don't cache me"
as far as I can tell.

The BIOS doesn't. The device driver (which knows the PCI card
intimately) asks for non-cached memory when it asks for a virtual
mapping of the physical address range of the card. If that's
appropriate. If there's no reason why it can't be cached, it doesn't.

Steve

--------------------------------------------------------------------------------------------
Steve Schefter phone: +1 705 725 9999 x26
The Software Group Limited fax: +1 705 725 9666
642 Welham Road,
Barrie, Ontario CANADA L4N 9A1 Web: www.wanware.com

But we often use PCI devices under DOS, with no device driver at all.
It appears to me that the BIOS locates devices in PCI config space,
looks at the requested resources (in the PCI overhead reggies) and
assigns memory space, as requested, to the gadgets. Usually these
addresses are really high, past 2G as I recall. So the cached/uncached
situation must be resolved, somehow, before the os boots, although it
can certainly be changed by drivers later.


John

yusufilker@gmail.com wrote:

I guess the answer for linux is hidden between ;


"Chapter 9 page 236 I/O Registers and Conventional Memory"


and


"Chapter 12 page 316 Accessing the I/O and Memory Spaces"

What book? The usual answer for Linux is O'Reilly's Linux Device
Driver, chapters 7, 8 and 13. The free online version can be found at:

http://www.xml.com/ldd/chapter/book/

It's not clear to me why this needs to be resolved by the BIOS.

First, the fact that the card is placed at a high memory address
is unrelated to caching. The BIOS putting things at high
addresses is convient to avoid things at low addresses (such
as the 2G of RAM you might have). But that doesn't make
the area unworthy of caching.

Second, what kind of device would sit on the PCI bus that is
simple enough to not need a device driver and yet requires
caching to be turned off for that area?

The only bits in the PCI configuration space that go with the
request for a range of memory are: prefetchable, type (2 bits
identifying where it can be placed) and a memory versus
I/O flag. That's all the BIOS has to work with.

Why would a device on the PCI bus not want to have its
memory range cached? Because the memory can change
by means other then the system CPU. For example, our
cards have serial chips which have their internal registers
mapped to PCI memory space. If the CPU writes to one
of these, it can't be cached -- it needs to go right through
to the memory/register immediately. Likewise, the CPU
can't refer to its cache to get a value. The registers change
all the time based on what's going on on the serial line.
So any cache would instantaneously be "dirty".

The above card would be useless without a device driver.
What kind of situation are you worried about needing to
have the PCI device's memory range uncached that is
simple enough to not need a device driver?

Steve

--------------------------------------------------------------------------------------------
Steve Schefter phone: +1 705 725 9999 x26
The Software Group Limited fax: +1 705 725 9666
642 Welham Road,
Barrie, Ontario CANADA L4N 9A1 Web: www.wanware.com

On 6 Dec 2005 06:02:42 -0800, steve_schefter@hotmail.com wrote:

The above card would be useless without a device driver.
What kind of situation are you worried about needing to
have the PCI device's memory range uncached that is
simple enough to not need a device driver?

My question was actually an attempt to understand how the BIOS sets up
caching and, in general, what devices on the PCI bus might get cached,
and what controls whether they do. All the answers, so far, is that
nobody knows.

Specifically, we often write test programs to run under DOS so we can
test interfaces at maximum CPU speed, without a stupid OS gobbling up
resources in millisecond chunks, and without having to write drivers
for untested hardware (google "chicken, egg"). Since any pci-compliant
BIOS actually finds our interfaces and assigns memory resources, I was
wondering what the caching situation is. Again, nobody seems to know.

Besides, a device isn't "useless without a device driver" as long as
an application can get at its registers somehow. Could be more useful,
actually.

John

In article <qndbp1llst1qkmtge5s08sc89ovg8vg3kr@4ax.com>,
jjlarkin@highNOTlandTHIStechnologyPART.com says...

On 6 Dec 2005 06:02:42 -0800, steve_schefter@hotmail.com wrote:



The above card would be useless without a device driver.
What kind of situation are you worried about needing to
have the PCI device's memory range uncached that is
simple enough to not need a device driver?


My question was actually an attempt to understand how the BIOS sets up
caching and, in general, what devices on the PCI bus might get cached,
and what controls whether they do. All the answers, so far, is that
nobody knows.

You can ask on comp.sys.ibm.pc.hardware.chips, but you'll get the
same answer I gave you. PCI addresses are *not* cached. The
memory configuration, including cacheability, is set up by BIOS (or
perhaps the OS) using the MTRRs on the processor and north bridge.

Specifically, we often write test programs to run under DOS so we can
test interfaces at maximum CPU speed, without a stupid OS gobbling up
resources in millisecond chunks, and without having to write drivers
for untested hardware (google "chicken, egg"). Since any pci-compliant
BIOS actually finds our interfaces and assigns memory resources, I was
wondering what the caching situation is. Again, nobody seems to know.

If you want to test at the highest speed possible, I'd use a PCI
initiator to do the work. Crossing the processor bus/PCI bridge
takes time which varies between north bridges. Some don't allow
much bandwidth from the CPU and you can't test PCI bursts at all.

Besides, a device isn't "useless without a device driver" as long as
an application can get at its registers somehow. Could be more useful,
actually.

Sure, as long as you accept the security/stability implications
that go along with writing directly to system resources from user
space.

--
Keith

On Tue, 6 Dec 2005 11:45:17 -0500, Keith Williams <krw@att.bizzzz>
wrote:

In article <qndbp1llst1qkmtge5s08sc89ovg8vg3kr@4ax.com>,
jjlarkin@highNOTlandTHIStechnologyPART.com says...
On 6 Dec 2005 06:02:42 -0800, steve_schefter@hotmail.com wrote:



The above card would be useless without a device driver.
What kind of situation are you worried about needing to
have the PCI device's memory range uncached that is
simple enough to not need a device driver?


My question was actually an attempt to understand how the BIOS sets up
caching and, in general, what devices on the PCI bus might get cached,
and what controls whether they do. All the answers, so far, is that
nobody knows.

You can ask on comp.sys.ibm.pc.hardware.chips, but you'll get the
same answer I gave you. PCI addresses are *not* cached. The
memory configuration, including cacheability, is set up by BIOS (or
perhaps the OS) using the MTRRs on the processor and north bridge.

I suspect you're right about that.

Specifically, we often write test programs to run under DOS so we can
test interfaces at maximum CPU speed, without a stupid OS gobbling up
resources in millisecond chunks, and without having to write drivers
for untested hardware (google "chicken, egg"). Since any pci-compliant
BIOS actually finds our interfaces and assigns memory resources, I was
wondering what the caching situation is. Again, nobody seems to know.

If you want to test at the highest speed possible, I'd use a PCI
initiator to do the work. Crossing the processor bus/PCI bridge
takes time which varies between north bridges. Some don't allow
much bandwidth from the CPU and you can't test PCI bursts at all.

Besides, a device isn't "useless without a device driver" as long as
an application can get at its registers somehow. Could be more useful,
actually.

Sure, as long as you accept the security/stability implications
that go along with writing directly to system resources from user
space.

Windows has such a wealth of security and stability holes, we can
hardly make it much worse by accessing our own registers. I find
Microsoft's warnings to be highly ironic in this respect.

John

John, cachability is controlled by the memory controller inside the
processor. There is a bit in each page table entry that describes
whether that 4k page may be cached. These are the same entries that
will map the physical PCI addresses into your logical address space, so
you were going to have to play with them anyway.

What version of Windows are you planning to use?

Find the DDK for that version and start tracking down the page table
manipulation routines. Find a small sample driver that maps pages, and
modify it to meet your needs.

- Tim.

John Larkin wrote:

My question was actually an attempt to understand how the BIOS sets up
caching and, in general, what devices on the PCI bus might get cached,
and what controls whether they do. All the answers, so far, is that
nobody knows.

As a PCI device driver writer, I know. You just don't seem to want to
believe me ;-)

Since any pci-compliant
BIOS actually finds our interfaces and assigns memory resources, I was
wondering what the caching situation is.

Assigning resources (done by the BIOS) has nothing to do with
caching. The device can be placed into I/O or memory according
to the resources it asks for, but this is unrelated to whether the
CPU will use its cache when accessing that memory range.
The latter is under control of the device driver when it maps the
range into virtual memory.

Besides, a device isn't "useless without a device driver" as long as
an application can get at its registers somehow.

Which registers? They only registers that are generic to all
cards are the PCI configuration space registers and they are
generally accessed via configuration space (different from
memory and I/O space). In configuration space caching doesn't
apply. The registers (or other parts of the card) which may be
mapped into memory space are specific to that card and
therefore raise the question of caching. They are different for
every card design and therefore not useful without a device
driver that understands what hardware is involved with the
memory.

You can't decide whether caching can lead to data corruption
(and therefore should be disabled for the range) unless you
understand the hardware (ie, you're the device driver). As I
pointed out earlier from the PCI spec, there are only 4 bits
that go with the memory request and none of them indicate
whether caching should be on or off. Since the device driver
does the virtual mapping and it has to know the hardware
well enough to know whether caching is appropriate or not,
there is no point in putting it in configuration space.

Steve

steve_schefter@hotmail.com wrote:

John Larkin wrote:
Besides, a device isn't "useless without a device driver" as long as
an application can get at its registers somehow.

Which registers? They only registers that are generic to all
cards are the PCI configuration space registers and they are
generally accessed via configuration space (different from
memory and I/O space). In configuration space caching doesn't
apply. The registers (or other parts of the card) which may be
mapped into memory space are specific to that card and
therefore raise the question of caching. They are different for
every card design and therefore not useful without a device
driver that understands what hardware is involved with the
memory.

Steve,

I did a handful of PCI designs (for use on PMC sites in VME SBCs). The
SBC of course ran Linux or VxWorks or Integrity or whatever, but we
also had a low-level monitor/debug environment roughly equivalent (OK,
far superior!) to DOS BIOS.

Most of the designs used PLX chips. PLX puts the registers needed to
configure their chips' peripherals in BAR 0 and BAR 1. I'd generally
put my design's application-specific registers in BAR 2. On-board
memory goes in another BAR. And so forth.

As the board booted, the monitor enumerated the PCI bus and assigned
valid base addresses to all of the PCI devices. From its command line,
you could do the equivalent of PEEK and POKE to the PLX peripheral
control registers or to the board's custom registers. Standard
memory-dump and memory-modify commands were available, too,

So, what John wants to do -- peek and poke registers, etc -- is
reasonable.

As for caching -- that really depends on how the system controller chip
is set up. I don't know to what extent the BIOS firmware sets up a
PC's system controller; presumably, it does enough to find the boot
device for a higher-level OS, to which it passes control. The OS then
may set up the system controller in ways that depend on its needs. One
presumes that the system controller driver honors the cacheable bit and
doesn't cache a memory space declared as non-cacheable!

-a

Mark McDougall wrote:

Keith Williams wrote:

prefetching <> cacheing

OK, but I believe it is right to say that *only* prefetchable memory can
(also) be cacheable?

Yes, but I would word it as "cacheable memory must be prefetchable but
prefetchable memory may or may not be cachable". Just to be clear.

Keith Williams wrote:

prefetching <> cacheing

OK, but I believe it is right to say that *only* prefetchable memory can
(also) be cacheable?

But it *is* part of the (pre 2.2) spec. The bus must guarantee
coherency in this case. The way it does it is with back-offs and
retries. Now think about this with multiple bridges and initiators.
It gets to be a mess.

Which is exactly why I was surprised that they (PCISIG) would even
attempt to handle it! :O Thinking about it, I suppose if the bus doesn't
specify it, then what other scope to do so is there?

It's there in the older versions of the spec. As I've mentioned,
it's been deprecated in later versions.

OK, I stand corrected. Although I have worked on pre 2.2 designs, this
was never an issue so it was duly forgotten.

I've learnt something new today. Can I go home now? ;)

Regards,
Mark

Andy Peters wrote:

As the board booted, the monitor enumerated the PCI bus and assigned
valid base addresses to all of the PCI devices. From its command line,
you could do the equivalent of PEEK and POKE to the PLX peripheral
control registers or to the board's custom registers. Standard
memory-dump and memory-modify commands were available, too,

So, what John wants to do -- peek and poke registers, etc -- is
reasonable.

This was kind of a tangent to the caching discussion. But sure, no
issue with doing this. What you are doing is essentially is putting
the knowledge of the hardware/registers into the peek/poke operator
instead of a device driver.

In order to get at the high addresses that the BIOS will put the PCI
device at, I suspect that some sort of extended memory add-on
will be required (haven't worked in DOS in ages). Whether it
defaults to caching the memory or not will be dependent on that
software. If I were writing it and had to pick just one, I'd disable
caching for that memory range to ensure compatibility, though at
the cost of performance.

As for caching -- that really depends on how the system controller chip
is set up. I don't know to what extent the BIOS firmware sets up a
PC's system controller; presumably, it does enough to find the boot
device for a higher-level OS, to which it passes control. The OS then
may set up the system controller in ways that depend on its needs. One
presumes that the system controller driver honors the cacheable bit and
doesn't cache a memory space declared as non-cacheable!

Reasonable, although it depends on the OS. In the ones I've
worked with, you can't take a default but have to indicate the
characteristics (including cache/non-cache) that you need.

Also, assuming an OS/drivers that left the setup entirely to
the BIOS, the implication would be that the entire PCI range
would be non-cached since the BIOS has no way to know if
caching will cause grief for any particular card.

Steve