Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

From: Tim Hockin (thockin_at_hockin.org)
Date: 10/03/04

  • Next message: Erich Focht: "Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement" 
    Date:	Sun, 3 Oct 2004 13:10:05 -0700
To: "Martin J. Bligh" <mbligh@aracnet.com>

    On Sun, Oct 03, 2004 at 07:36:46AM -0700, Martin J. Bligh wrote:
    > > This is where I see the need for "CPU sets". I.e. as a
    > > replacement/modification to the CPU affinity mechanism basically adding
    > > an extra level of abstraction to make it easier to use for implementing
    > > the type of isolation that people seem to want. I say this because,
    > > strictly speaking and as you imply, the current affinity mechanism is
    > > sufficient to provide that isolation BUT it would be a huge pain to
    > > implement.
    >
    > The way cpusets uses the current cpus_allowed mechanism is, to me, the most
    > worrying thing about it. Frankly, the cpus_allowed thing is kind of tacked
    > onto the existing scheduler, and not at all integrated into it, and doesn't
    > work well if you use it heavily (eg bind all the processes to a few CPUs,
    > and watch the rest of the system kill itself).

    7 years ago, before cpus_allowed was dreamed up, I proposed a pset patch
    and was shot down hard. Now it's back, and we're trying to find a way to
    cram it in on top.

    Yeah, it does not fit nicely with cpus_allowed.

    I have to ask - do we REALLY need cpusets? I meant, even SGI dropped
    PSET at some point, because (if I recall) NO ONE USED IT.

    What's the problem being solved that *requires* psets?

    I have a customer I work with periodically who was using my pset patch up
    until they moved to RH8, when the O(1) scheduler and cpus_allowed changed
    everything. This was their requirement for pset:

    1. Take a processor out of the general execution pool (call it
    PROC_RESTRICTED). This processor will not schedule general tasks.
    2. Assign a task to the PROC_RESTRICTED cpu. Now that CPU will only
    schedule the assigned task (and it's children).
    3. Repeat for every CPU, with the caveat that one CPU must remain
    PROC_ENABLED.

    I had an array of enum procstate and a new syscall pair:
    sched_{gs}etprocstate(). The scheduler checks the procstate, and if it is
    not ENABLED, it checks that (cpus_allowed == 1<<cpu). Simple, but works.
    Could be baked a bit more, for general use.

    What if I proposed a patch like this, now? It would require cleanup for
    2.6, but I'm game if it's useful.

    Tim

    -
    To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at http://vger.kernel.org/majordomo-info.html
    Please read the FAQ at http://www.tux.org/lkml/

  • Next message: Erich Focht: "Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement"

    Relevant Pages

    • Re: Integrating cpusets and cpu isolation [was Re: [CPUISOL] CPU isolation extensions]
      ... I think you are assuming that I only need to deal with RT scheduler and scheduler ... No. That's confusing cpusets and the scheduler again. ... Hence the proposal is to extend current CPU ... The new definition of the CPU isolation would be: ...
      (Linux-Kernel)
    • One minute delay when booting 2.6.24.1
      ... Switched to high resolution mode on CPU 1 ... io scheduler anticipatory registered ... # Infrared-port device drivers ...
      (Linux-Kernel)
    • [RFC][PATCH] O(1) Entitlement Based Scheduler
      ... This patch is a modification of the Oscheduler that introduces ... _entitlement_ to CPU resources that is determined by the number of _shares_ ... This patch provides both soft and hard CPU usage rate caps per ... one getting the most can be given a better priority, ...
      (Linux-Kernel)
    • Scheduler Economy prototype patch for CFS
      ... X should get more CPU time simply ... the clients, not relative to any client individually. ... today i've implemented a quick prototype of this "Scheduler Economy" ... * Scheduler work account object: it consists of the price, ...
      (Linux-Kernel)
    • Re: Re: OSDL Bug 3770
      ... We're discussing the fact that on SMP machines, ... N> CPU to be starved, while tasks on another CPU with the same low ... L> could had run on another CPU if the scheduler had decided otherwise). ... Does Linux tolerate hard CPU binding? ...
      (Linux-Kernel)