Re: [RFC/PATCH] cpuset: cpuset irq affinities



Like Ingo, I like the approach.

But I am concerned it won't work, as stated.

Unfortunately, my blithering ignorance of how one might want to
distribute irq's across a system is making it difficult for me
to say for sure if this works or not.

The thing about /dev/cpuset that I am afraid will get in the way
with this use of cpusets to place irqs is that we can really only
have a single purpose hierarchy below /dev/cpuset.

For example, lets say we have:

/dev/cpuset
boot
big_special_app
a_few_isolated_rt_nodes
batchscheduler
batch job 1
batch job 2
...

I guess, with your "cpuset: cpuset irq affinities" patch, we'd start
off with /dev/cpuset/irqs listing the irqs available, and we could
reasonably decide to move any or all irqs to /dev/cpuset/boot/irqs,
by writing the numbers of those irqs to that file, one irq number
per write(2) system call (as is the cpuset convention.)

Do these irqs have any special hardware affinity? Or are they
just consumers of CPU cycles that can be jammed onto whatever CPU(s)
we're willing to let be interrupted?

If for reason of desired hardware affinity, or perhaps for some other
reason that I'm not aware of, we wanted to have the combined CPUs in
both the 'boot' and 'big_special_app' handle some irq, then we'd be
screwed. We can't easily define, using the cpuset interface and its
conventions, a distinct cpuset overlapping boot and big_special_app,
to hold that irq. Any such combining cpuset would have to be the
common parent of both the combined cpusets, an annoying intrusion on
the expected hierarchy.

If the actual set of CPUs we wanted to handle a particular irq wasn't
even the union of any pre-existing set of cpusets, then we'd be even
more screwed, unable even to force the issue by imposing additional
intermediate combined cpusets to meet the need.

If there is any potential for this to be a problem, then we should
examine the possibility of making irqs their own cgroup, rather than
piggy backing them on cpusets (which are now just one instance of a
cgroup module.)

Could you educate me a little, Peter, on what these irqs are and on
the sorts of ways people might want to place them across CPUs?


+ if (s->len > 0)
+ s->len += scnprintf(s->buf + s->len, s->buflen - s->len, " ");

The other 'vector' type cpuset file, "tasks", uses a newline '\n'
field terminator, not a space ' ' separator. Would '\n' work here,
or is ' ' just too much the expected irq separator in such ascii lists?
My preference is toward using the exact same vector syntax in each
place, so that once someone has code that handles one, they can
repurpose that code for another with minimum breakage.

--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <pj@xxxxxxx> 1.940.382.4214
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/



Relevant Pages

  • Re: boot cgroup questions
    ... In a sense that no mater how we decide to handle irqs we may still want 'boot' group. ... It'd be nice if it was in the other thread (cpuset: irq affinity support) but I'm ok with replying here. ... excessively overloads the cpuset hierarchy, ... subsets of CPUs and Memory Nodes, ...
    (Linux-Kernel)
  • Re: [RFC/PATCH] cpuset: cpuset irq affinities
    ... Forget that idea (allowing the same irq in multiple 'irqs' files.) ... cpuset into the cpuset hierarchy, where that system cpuset is there to ... flag -- which if set True, asks the system to "please leave my CPUs ... set for those who do upgrade kernels without upgrading user-space). ...
    (Linux-Kernel)
  • Re: [RFC/PATCH] cpuset: cpuset irq affinities
    ... So some of these irqs might have very particular node affinity then, ... If my thingamajig board is attached to node 72, ... putting that irq in my boot cpuset that only has nodes ... That way, most irqs can go to most CPUs, depending on their specific ...
    (Linux-Kernel)
  • Re: [PATCH 1/2] cpuset: cpuset irq affinities
    ... I'm treating irqs just like tasks. ... I see the one comment in your Patch 2/2 noting you're unsure ... I'm thinking of irqs as tasks (in the cpuset context). ... individual irq to a list of cpus. ...
    (Linux-Kernel)
  • Re: [RFC/PATCH] cpuset: cpuset irq affinities
    ... So if /cgroup/boot is changed to have less CPUs then the "default" irqs move along with it. ... I've been playing with latest sched-devel tree and while I think we'll endup adding a lot more code, doing it with the cpuset is definitely more flexible. ... This way we can provide more fine grain control of what part of the "system" services are allowed to run on a cpuset. ...
    (Linux-Kernel)