Re: [RFC] MMIO accessors & barriers documentation

On Monday, September 11, 2006 2:45 pm, Benjamin Herrenschmidt wrote:
These sound fine. I think PPC64 is the only platform that will need
them?

Ah ? What about the comment in e1000 saying that it needs a wmb()
between descriptor updates in memory and the mmio to kick them ? That
would typically be a memory_to_io_wb(). Or are your MMIOs ordered cs.
your cacheable stores ?

I think that's a separate issue? As Jeff points out, those macros are
intended to provide memory vs. I/O ordering, but isn't PPC the only platform
that will reorder accesses so aggressively and independently? I don't think
ia64 for example will reorder them separately, so a regular memory barrier
*should* be enough to ensure ordering in both domains.

They are, but I was thinking about providing more IO-like examples. I
suppose I could refer to memory-barriers.txt from here and update it
with IO-like examples.

Yeah, either way. Not sure if adding more I/O examples to the existing doc is
better or worse than an I/O specific document.

But isn't this how you'll implement io_to_lock_wb() on PPC anyway? If
so, might be best to name it and document it that way (though keeping the
idea of barriering before unlocking prominent in the documentation).

Well, the whole question is what does the linux semantics guarantee to
driver writers (accross archs), not what PowerPC implements :) I'd
rather not add guarantees that aren't useful to drivers even if all
current implementations happen to provide them. I'm trying to find a
case where ordering MMIO W + memory W is useful and I can't see any
since the MMIO W will take any time to go to the device anyway. The lock
rule seems to be the only useful, thus the only I think I'll guarantee.

Sure, that's fair. If any potential application of the more precise semantics
is just theoretical, we may as well limit our guarantees to locks only.

Well, as far as I'm concerned, the whole point is rule #2 and #3 :)
Those are the ones biting us on PowerPC (we haven't seen the lock
problem but then it can't happen the way our current accessors are
written. However, if we change our accessors to provide rule #2 more
specifically, we'll end up with 2 sync instructions in writel, one for
rule #2 before the store and one for rule #4, thus we go from expensive
to very expensive). It's also my understanding that mmiowb is very
expensive on ia64 and gets worse as the box grows bigger.

Yeah, that's true (I see your point about being more worried about other
things on PPC as well ;).

Hence the question: do we provide -fully- ordered accessors in class 1,
or do we provide -mostly- ordered accessors, ordered in all means except
rule #4 vs locks. ia64 is afaik by far the platform taking the biggest
hit if you have to provide #4, so I'm interesting in your point of view
here.

Either way is fine with me as long as we have a way to get at the fast and
loose stuff (and required barriers of course) in a portable way. And that we
don't regress the existing users of mmiowb().

We don't need counters, just a flag. We did a test implementation, seems
to work. We also clear the flag in spin_lock. That means that MMIOs
issued before a lock aren't ordered vs. the locked section. But because
of rule #1, they should be ordered vs. other MMIOs inside the locked
section and thus implicitely get ordered anyway.

Oh right, a flag would be enough. Is it good enough for -mm yet? Might be
fun to run on an Altix machine with a bunch of supported devices (not that I
work with them anymore...).

For ia64 in particular it doesn't matter, though there was speculation
several years that it might be necessary. No actual examples stepped
forward though, so the current implementation doesn't take an argument.

Ok. My question is wether it would improve the implementation to take
it. If we define a new macro with a new name, we can do it....

Right, but unless there's a real need at this point, we probably shouldn't
bother. Let the poor sucker with the future machine needing the device
argument do the work. :)

Thanks,
Jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • [RFC] MMIO accessors & barriers documentation
    ... "writel/readl" type accessors behave differently based on an ioremap ... 1- MMIO + MMIO: This type of ordering means that two consecutive MMIO ... 2- memory W + MMIO W: This type of ordering means that an store to main ... must be visible to that device before the MMIO store reaches it. ...
    (Linux-Kernel)
  • Re: [RFC] MMIO accessors & barriers documentation
    ... between a memory store and an MMIO store. ... with write accessors of Class 2 and 3. ...
    (Linux-Kernel)
  • Re: [RFC] MMIO accessors & barriers documentation
    ... between descriptor updates in memory and the mmio to kick them? ... followed by MMIO store, that is does ia64 -current- accessors provide ... the barriers allowing to implement those ordering rules when the ...
    (Linux-Kernel)
  • [RFC] MMIO accessors & barriers documentation #2
    ... Definitions of MMIO accessors and IO related barriers semantics. ... Such ordering means that both sets of MMIO stores will not ... unless an explicit barrier is used. ...
    (Linux-Kernel)
  • Re: lib/iomap.c mmio_{in,out}s* vs. __raw_* accessors
    ... I'm tempted to remove those mmio_* things from iomap.c completely. ... tests for whether something is MMIO or PIO can be done _once_, ... So Yes, iomap is a library that can be replaced, Yes, all you said is ... that provide pretty much the complete set of accessors using that macro ...
    (Linux-Kernel)
Quantcast