Re: [PATCH]cpuset: add new API to change cpuset top group's cpus

* Len Brown <lenb@xxxxxxxxxx> [2009-05-27 22:34:38]:

On Tue, 19 May 2009, Vaidyanathan Srinivasan wrote:

We tried similar approaches to create idle time for power savings, but
cpu hotplug interface seem to be a clean choice. There could be
issues with the interface, we should fix it. Is there any other
reason why cpuhotplug is 'ugly' other than its performance (speed)?

I have tried few load balancer hacks to evacuate cores but not a solid
design yet. It has its advantages but still needs more work.

http://lkml.org/lkml/2009/5/13/173

Thanks for the pointer.
I agree with Andi, please avoid the term "throttling", since
it has been used for ages to refer processor clock throttling --
which is actually significantly less effective at saving
energy than what you are trying to do. (not the word "energy"
here, where the word "power" is incorrectly used in the thread above)

Yes, you are right. This throttling is used to refer to hardware
methods to slow down things and it is less effective in saving energy.
It reduces average power but make the work load run much longer and
consume more energy.

"core evacuation" is a better description, I agree, though I wonder
why you don't simply call it "forced idling", since that is what
you are trying to do.

Yes, core evacuation is what I propose, but actually what we are doing
is starving or throttling tasks in software to create idle time, just
to make the description clear.

Furthermore, we should not want anything outside of that, either the cpu
is there available for work, or its not -- halfway measures don't make
sense.

Furthermore, we already have power aware scheduling which tries to
aggregate idle time on cpu/core/packages so as to maximize the idle time
power savings. Use it there.

Power aware scheduling can optimally accumulate idle times. Framework
to create idle time to force idle cores is good and useful for power
savings. Other than the speed of online/offline I do not know of any
other major issue for using cpu hotplug for this purpose.

It sounds like you want to use this technique more often
that I had in mind. You are thinking of a warm rack, which
may stay warm all day long. I am thinking of a rack which
has a theoretical power draw higher than the providioned
electrical supply. As there is a huge difference between
actual and theoretical power draw, this saves many dollars.

Yes, this framework can be used more often to balance average power
consumption in systems. Exploiting the margin between theoretical
limits and practical usage will definitely save money in a data
center. Present generation power capping techniques and related
infrastructure are available to exploit this margin.

Core evacuation can compliment this safety limit mechanism by
providing more fine grain control.

So what you're looking at is more frequent use than we need,
and that is fine -- as long as you exhaust P-states first --
since forcing cores to be idle has a more severe performance
impact than running at a deeper P-state.

Yes, that is the idea. After getting all core to lowest P-State, we
can further cut power by forcing idle. Even when not at the lowest
P-State, forced idle of complete packages may save more power as
compared to running all cores in a large system at lowest P-State.
This is generally not the case, but the framework can be more flexible
and provide more degrees of control.

I didn't see P-states addressed in your thread.

P-States can be flexibly managed using the present cpufreq governors.
Ondemand, conservative or userspace can provide us with the required
level of control from userspace. Idle cores will be at lowest
P-States and C-State in case of ondemand governor. Independent of the
P-States the idle cores will save power from C-State and hence cpufreq
governors does not make an impact.

In the case of busy cores, end users can decide to pick conservative
or userspace governor before invoking core evacuation.

The main motivation for the core evacuation framework is to provide
another degree of control to exploit C-States based power savings
apart from P-State manipulation (for which good framework already
exist).

Besides, a hot removed cpu will do a dead loop halt, which isn't power saving
efficient. To make hot removed cpu enters deep C-state is in whish list for a
long time, but still not available. The acpi_processor_idle is a module, and
cpuidle governor potentially can't handle offline cpu.

Then fix that hot-unplug idle loop. I agree that the hlt thing is silly,
and I've no idea why its still there, seems like a much better candidate
for your efforts than this.

I agree with Peter. We need to make cpu hotplug save power first and
later improve upon its performance.

We do have a patch to fix the offline idle loop to save power.

This will definitely help the objective. I have looked at Venki's
patch. We certainly need that feature even outside of the current
context where we want to hotplug faulty CPUs or setup special system
configurations where all cores in a package is not to be used.

We can use hotplug in the short term until something better comes along.
Yes, it will break cpusets, just like Shaohua's original patch broke them
-- and that will make using it inappropriate for some customers.

It will good to have a solution that does not affect user policy.
Otherwise that will discourage its adoption and usability. But the
cpu-hotplug solution will work in short term.

While I think this mechanism is important, I don't think that a large %
of customers will deploy it. I think the ones that deploy it will do so
to save money on electrical provisioning, not on pushing the limits
of their air conditioner. So I don't expect its performance requirement
to be extremely severe. I don't think it will justify tuning the
performance of cpu-hotplug, which I don't think was ever intended
to be in the performance path.

The motivation to improve cpu-hotplug is that we have begin to find
more uses for the framework and if there are issues, this is a good
time to fix it. Opportunities to improve performance should be
explored because we will have to hotplug multiple CPUs to have an
impact. The number of cores in the system will become quite large and
we will always have to hotplug multiple cpus to isolate a package for
hardware faults or power saving purposes.

On a system with 4096 CPUs, perhaps 128 cores my be a package or
entity that needs to go off in bulk. We will certainly not be dealing
with online/offline of one or two cpus in such a system. Well this is
an extreme case and weired example. Hope you get the idea on why we
should try to improve cpu-hotplug path.

Thanks for the detailed comments and suggestions.

--Vaidy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages