Re: Large Pages - Linux Foundation HPC

On Tue, Apr 21, 2009 at 11:55:55PM +0530, Balbir Singh wrote:
[Fix my email address to balbir@xxxxxxxxxxxxxxxxxx]

* Dave Hansen <dave@xxxxxxxxxxxxxxxxxx> [2009-04-21 09:57:05]:
On Tue, 2009-04-21 at 09:32 -0700, Badari Pulavarty wrote:
Hi Dave,

On the Linux foundation HPC track summary, I saw:

-- Memory and interface to it - mapping memory into apps
- large pages important - current state not good enough

I'm not sure exactly what this means. But, there was continuing concern
about large page interfaces. hugetlbfs is fine, but it still requires
special tools, planning, and requires some modification of the app. We
can modify it with linker tricks or with LD_PRELOAD, but those certainly
don't work everywhere. I was told over and over again that hugetlbfs
isn't a sufficient interface for large pages, no matter how much
userspace we try to stick in front of it.

Some of their apps get a 6-7x speedup from large pages!

Fragmentation also isn't an issue for a big chunk of the users since
they reboot between each job.

Perhaps this policy?

In mlock(), populate huge pages if (1) the mlock range is large enough
to hold some huge pages; (2) there are more than enough free high
order pages.

Based on Dave's descriptions that HPC apps typically
- do mlock(), to pre-populate memory and pin them in memory
- run at fresh boot, with loads of high order pages available

Thanks,
Fengguang

nodes going down due to memory exhaustion

Virtually all the apps in an HPC environment start up try to use all the
memory they can get their hands on. With strict overcommit on, that
probably means brk() or mmap() until they fail. They also usually
mlock() anything they're able to allocate. Swapping is the devil to
them. :)

Basically, what all the apps do is a recipe for stressing the VM and
triggering the OOM killer. Most of the users simply hack the kernel and
replace the OOM killer with one that fits their needs. Some have an
attitude that "the user's app should never die" and others "the user
caused this, so kill their app". Basically, there's no way to make
everyone happy since they have conflicting requirements. But, this is
true of the kernel in general... nothing special here.

OOM killer has been a hot topic. Have you seen Dan Malek's patches at
http://lkml.org/lkml/2009/4/13/276.


The split LRU should help things. It will at least make our memory
scanning more efficient and ensure we're making more efficient reclaim
progress. I'm not sure that anyone there knew about the oom_adjust and
oom_score knobs in /proc. They do now. :)

:-)


One of my suggestions was to use the memory resource controller. They
could give each app 95% (or whatever) of the system. This should let
them keep their current "consume all memory" behavior, but stop them at
sane limits.


Soft limits should help as well, basically we are trying to allow
unrestricted memory access until there is contention. The patches are
still under development.

That leads into another issue, which is the "wedding cake" software
stack. There are a lot of software dependencies both in and out of the
kernel. It is hard to change individual components, especially in the
lower levels. This leads many of the users to use old (think 2.6.9)
kernels. Nobody runs mainline, of course.

Then, there's Lustre. Everybody uses it, it's definitely a big hunk of
the "wedding cake". I haven't seen any LKML postings on it in years and
I really wonder how it interacts with the VM. No idea.

There's a "Hyperion cluster" which is for testing new HPC software on a
decently sized cluster. One suggestion of ours was to try and get
mainline tested on this every so often to look for regressions since
we're not able to glean feedback from 2.6.9 kernel users. We'll see
where that goes.

checkpoint/restart

Many of the MPI implementations have mechanisms in userspace for
checkpointing of user jobs. Most cluster administrators instruct their
users to use these mechanisms. Some do. Most don't.


Good inputs and summary. Thanks!

--
Balbir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Relevant Pages

  • Re: Large Pages - Linux Foundation HPC
    ... -- Memory and interface to it - mapping memory into apps ... isn't a sufficient interface for large pages, ... Some of their apps get a 6-7x speedup from large pages! ... true of the kernel in general... ...
    (Linux-Kernel)
  • Re: Large Pages - Linux Foundation HPC
    ... -- Memory and interface to it - mapping memory into apps ... isn't a sufficient interface for large pages, ... Some of their apps get a 6-7x speedup from large pages! ... true of the kernel in general... ...
    (Linux-Kernel)
  • Re: Tech directions for Delphi?
    ... allocator, ... Most desktop apps build a relatively modest working set (a ... the case for .NET doesn't really rest on automatic memory ... management being cheaper overall than manual memory management. ...
    (borland.public.delphi.non-technical)
  • Re: Id appreciate your opinion(s)
    ... The fact that so much memory is used really doesn't mean you ... When it comes to all of the apps except for ones defined in the ... then I would say that .NET or Java are perfectly fine ... Java and .NET can both do "smart" clients, fat clients, or web-based ...
    (microsoft.public.dotnet.languages.csharp)
  • Re: memory requirements of closed vs quit apps
    ... majority of programs remain open when all their windows are closed. ... they use bugger all memory - and are ... individual apps Real and Virtual memory entries. ... In order to push it out to swap, ...
    (uk.comp.sys.mac)