Re: [ckrm-tech] Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

From: Paul Jackson (pj_at_sgi.com)
Date: 10/02/04

  • Next message: Rafael J. Wysocki: "Re: 2.6.9-rc3-mm1 [immediate crash on AMD64]" 
    Date:	Sat, 2 Oct 2004 12:14:30 -0700
To: Hubertus Franke <frankeh@watson.ibm.com>

    Hubertus wrote:
    >
    > A minimal quote from your website :-)

    Ok - now I see what you're saying.

    Let me expound a bit on this line, from a different perspective.

    While big NUMA boxes provide the largest available single system image
    boxes available currently, they have their complications. The bus and
    cache structures and geometry are complex and multilayered.

    For more modest, more homogenous systems, one can benefit from putting
    CKRM controllers (I hope I'm using this term correctly here) on things
    like memory pages, cpu cycles, disk i/o, and network i/o in order to
    provide a fairly rich degree of control over what share of resources
    each application class receives, and obtain both efficient and
    controlled balance of resource usage.

    But for the big NUMA configuration, running some of our customers most
    performance critical applications, one cannot achieve the desired
    performance by trying to control all the layers of cache and bus, in
    complex geometries, with their various interactions.

    So instead one ends up using an orthogonal (thanks, Hubertus) and
    simpler mechanism - physical isolation(*). These nodes, and all their
    associated hardware, are dedicated to the sole use of this critical
    application. There is still sometimes non-trivial work done, for a
    given application, to tune its performance, but by removing (well, at
    least radically reducing) the interactions of other unknown applications
    on the same hardware resources, the tuning of the critical application
    now becomes a practical, solvable task.

    In corporate organizations, this resembles the difference between having
    separate divisions with their own P&L statements, kept at arms length
    for all but a few common corporate services [cpusets], versus the more
    dynamic trade-offs made within a single division, moving limited
    resources back and forth in order to meet changing and sometimes
    conflicting objectives in accordance with the priorities dictated by
    upper management [CKRM].

     (*) Well, not physical isolation in the sense of unplugging the
         interconnect cables. Rather logical isolation of big chunks
         of the physical hardware. And not pure 100% isolation, as
         would come from running separate kernel images, but minimal
         controlled isolation, with the ability to keep out anything
         that causes interference if it doesn't need to be there, on
         those particular CPUs and Memory Nodes.

         And our customers _do_ want to manage these logically isolated
         chunks as named "virtual computers" with system managed permissions
         and integrity (such as the system-wide attribute of "Exclusive"
         ownership of a CPU or Memory by one cpuset, and a robust ability
         to list all tasks currently in a cpuset). This is a genuine user
         requirement to my understanding, apparently contrary to Andrew's.

    The above is not the only use of cpusets - there's also providing
    a base for ports of PBS and LSF workload managers (which if I recall
    correctly arose from earlier HPC environments similar to the one
    I described above), and there's the work being done by Bull and NEC,
    which can better be spoken to by representives of those companies. 

    -- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
  • Next message: Rafael J. Wysocki: "Re: 2.6.9-rc3-mm1 [immediate crash on AMD64]"

    Relevant Pages

    • The cost of just saying no
      ... control their behavior in the face of temptation. ... Once these resources are exhausted, ... another task that requires a significant amount of self-control. ... performed worse on the Stroop task, indicating that they had used up ...
      (uk.philosophy.humanism)
    • Re: GUI INTERFACE
      ... the message passing paradigm is the ... > return its resources to the available pool. ... Just why can't a *CONTROL* program/process be produced ... the hung process and release/retrieve the resources? ...
      (comp.lang.pl1)
    • Re: Trouble with strongly-typed Image resources at design-time
      ... problem using Image resources from the Resource Manager. ... one for the control and one for the test ... and all works fine at runtime and design-time. ... disposing the Image objects incorrectly because it works fine when I ...
      (microsoft.public.dotnet.framework.windowsforms)
    • Re: ATI Catalyst Control Center with .NET v2
      ... the Microsoft Error Report pop-up opened "ATI EXTERNAL EVENT UTILITY.EXE ... > conclusion that it's just not worth the system resources it takes up. ... > I have spent more time trying to get the cat control ctr running right than ...
      (microsoft.public.dotnet.general)
    • Re: Help with MOSFET as switch.
      ... > I'm looking to control some devices from my computer. ... > play around with the parallel port. ... For starters I'm just looking to switch some LED's on and off ... To get the isolation and minimise the risk you might want to buy a kit. ...
      (sci.electronics.design)
    Loading