Re: [PATCH 1/24] make atomic_read() behave consistently on alpha

On Fri, Aug 10, 2007 at 03:49:03PM -0400, Chris Snook wrote:
Paul E. McKenney wrote:
On Thu, Aug 09, 2007 at 03:24:40PM -0400, Chris Snook wrote:
Paul E. McKenney wrote:
On Thu, Aug 09, 2007 at 02:13:52PM -0400, Chris Snook wrote:
Paul E. McKenney wrote:
On Thu, Aug 09, 2007 at 01:14:35PM -0400, Chris Snook wrote:
If you're depending on volatile writes
being visible to other CPUs, you're screwed either way, because the
CPU can hold that data in cache as long as it wants before it writes
it to memory. When this finally does happen, it will happen
atomically, which is all that atomic_set guarantees. If you need to
guarantee that the value is written to memory at a particular time in
your execution sequence, you either have to read it from memory to
force the compiler to store it first (and a volatile cast in
atomic_read will suffice for this) or you have to use LOCK_PREFIX
instructions which will invalidate remote cache lines containing the
same variable. This patch doesn't change either of these cases.
The case that it -can- change is interactions with interrupt handlers.
And NMI/SMI handlers, for that matter.
You have a point here, but only if you can guarantee that the interrupt
handler is running on a processor sharing the cache that has the
not-yet-written volatile value. That implies a strictly non-SMP
architecture. At the moment, none of those have volatile in their
declaration of atomic_t, so this patch can't break any of them.
This can also happen when using per-CPU variables. And there are a
number of per-CPU variables that are either atomic themselves or are
structures containing atomic fields.
Accessing per-CPU variables in this fashion reliably already requires a
suitable smp/non-smp read/write memory barrier. I maintain that if we
break anything with this change, it was really already broken, if less
obviously. Can you give a real or synthetic example of legitimate code
that could break?

My main concern is actually the lack of symmetry -- I would expect
that an atomic_set() would have the same properties as atomic_read().
It is easy and cheap to provide them with similar properties, so why not?
Debugging even a single problem would consume far more time than simply
giving them corresponding semantics.

But you asked for examples. These are synthetic, and of course legitimacy
is in the eye of the beholder.

1. Watchdog variable.

atomic_t watchdog = ATOMIC_INIT(0);

...

int i;
while (!done) {

/* Do so stuff that doesn't take more than a few us. */
/* Could do atomic increment, but throughput penalty. */

i++;
atomic_set(&watchdog, i);
}
do_something_with(&watchdog);


/* Every so often on some other CPU... */

if ((new_watchdog = atomic_read(&watchdog)) == old_watchdog)
die_horribly();
old_watchdog = new_watchdog;


If atomic_set() did not have volatile semantics, the compiler
would be within its rights optimizing it to simply get the
final value of "i" after exit from the loop. This would cause
the watchdog check to fail spuriously. Memory barriers are
not required in this case, because the CPU cannot hang onto
the value for very long -- we don't care about the exact value,
or about exact synchronization, but rather about whether or
not the value is changing.

In this (toy) example, one might replace the atomic_set() with
an atomic increment (though that might be too expensive in some
cases) or with something like:

atomic_set(&watchdog, atomic_read(&watchdog) + 1);

However, other cases might not permit this transformation,
for example, an existing heavily used API might take int rather
than atomic_t.

Some will no doubt argue that this example should use a
macro or an asm similar to the "forget()" asm put forward
elsewhere in this thread.

2. Communicating both with interrupt handler and with other CPUs.
For example, data elements that are built up in a location visible
to interrupts and NMIs, and then added as a unit to a data structure
visible to other CPUs. This more-realistic example is abbreviated
to the point of pointlessness as follows:

struct foo {
atomic_t a;
atomic_t b;
};

DEFINE_PER_CPU(struct foo *, staging) = NULL;

/* Create element in staging area. */

__get_cpu_var(staging) = kzalloc(sizeof(*p), GFP_WHATEVER);
if (__get_cpu_var(staging) == NULL)
die_horribly();
/* allocate an element of some per-CPU array, get the result in "i"
*/
atomic_set(__get_cpu_var(staging).a, i);
/* allocate another element of a per-CPU array, with result in "i" */
atomic_set(__get_cpu_var(staging).b, i);
rcu_assign_pointer(some_global_place, __get_cpu_var(staging));

If atomic_set() didn't have volatile semantics, then an interrupt
or NMI handler could see the atomic_set() to .a and .b out of
order due to compiler optimizations.

Remember, you -did- ask for these!!! ;-)

Ok, I'm convinced. Part of the motivation here is to avoid heisenbugs,
so if people expect volatile atomic_set behavior, I'm inclined to give
it to them. I don't really feel like indulging the compiler bug
paranoiacs, but developer expectations are a legitimate motivation, and
a major part of why I posted this in the first place. I'll resubmit the
patchset with a volatile cast in atomic_set. Before I do, is there
anything *else* that desperately needs such a cast? As far as I can
tell, all the other functions are implemented with __asm__ __volatile__,
or with spinlocks that use that under the hood.

Sounds good!!!

The only other API that I am aware of needing volatile semantics is
rcu_dereference(), but I already sent a patch in for it. So as far
as I know, atomic_read() and atomic_set() should cover it.

Thanx, Paul
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: double-checked locking in C
    ... Of course you may think that the object takes place on cache line boundary and is only partially updated. ... Then if I stick to standards, the C norm is enough and I just have to use volatile. ... # 4.10 Memory Synchronization ...
    (comp.programming.threads)
  • Re: Automatic parallelization - was Re: LISP Object Oriented?
    ... the cache coherence system. ... that package to main memory and the design of that memory system, ... ramifications far beyond the design of the package alone. ... Now, the situation changes as you add CPUs, but the hit each CPU takes ...
    (comp.lang.lisp)
  • Re: Embedded software interview question collection
    ... volatile really does and because of this the new piece of hardware they have released witb an embedded processor does not work if the cache is switched on!!!! ... I now need to got through all my code and try to move every variable into a seperate block of memory so there will be no memory corruption. ... and this could be catagorized as DMA. ...
    (comp.arch.embedded)
  • Re: [patch 0/6] mm: alloc_percpu and bigrefs
    ... David S. Miller a écrit: ... with no per-cpu or per-node additional memory but got no comment. ... It's important to place mostly read parts together, so that a cache lines can ... bus trafic between CPUS. ...
    (Linux-Kernel)
  • RE: Scaling noise
    ... numbers of CPUs. ... It *seems* that the high-end degradation of cache performance as N increases ... and not the code to accommodate them is the cause of the invalidations. ... runtime is the unused memory for the 6 or 62 per-cpu-data structures. ...
    (Linux-Kernel)