Re: Allocating kernel memory

From: George Nelson (grn_at_freeuk.com)
Date: 05/11/04 

Date: 11 May 2004 07:34:06 -0700

Kasper Dupont <kasperd@daimi.au.dk> wrote in message news:<409BA6D3.E2DF9577@daimi.au.dk>...
> George Nelson wrote:
> >
> > Since I have, for good reason, not given details of my particular
> > application I do object to your statement that I do not understand my
> > own design and just because you can't see any case where large amounts
>
> That's not what I said. I said you don't understand the
> design of the kernel you are using. Which is quite
> obvious from your suggestion about a dynamic split
> between user and kernel space.
>

Wrong. That may be what you intended but not what you said and I
quote:

       "Your case sounds like you just wrote your code without
understanding
        the design"

> And most of such applications belongs in user mode, not
> in the kernel.

True, but not necessarily all of them.

>
> > I can assure you that I understand my design perfectly.
>
> But that doesn't necesarilly make it a good design.

Also true but often designs are limited by the underlying hardware and
software and you just have to make the best of a bad job. Given a
choice I would use a different variant of Unix than Linux but, at the
moment I do not have this luxury and must make do with the facilities
offered by Linux.

> Neither does it imply, you understand the system you
> are building it on top of.

Should I take this to mean that I am required to know the detailed
implementation of a piece of code before I can use it? If so pardon my
gross presumption in assuming that an understanding of the programming
interfaces and the limitations of those interfaces is not sufficient
to develop code. My code works perfectly well as it is but I could
extract better performance with more memory since I maintain a cache
of data to reduce the amount of real I/O I perform. The fact that I
was unaware that there is an (artificial) limit on the amount of
memory available within the kernel does not imply I do not understand
the system, rather it shows that I did not see the need to know the
design details of a kernel system in order to develop my code.

>
> > I could have adopted an alternative
> > approach of simply reserving a large amount of memory to my driver at
> > boot time and handle the management of it myself completely.
>
> That wouldn't change anything. You could allocate some
> physical memory that way, but it would still not be
> mapped in your address space. Essentially this means
> allocating high memory pages would work at least as
> well as allocating the memory at boot. It is possible
> to allocate high memory pages and map them into kernel
> address space. But there is a limit to the number of
> such pages that can be mapped at a time.

So you are saying that the kernel does not map all of physical memory
into its address space! Without looking at the code details, I find
this hard to believe.

>
> > I
> > considered this was not a good approach and would have made the driver
> > larger and more complex than needed so instead opted to use the memory
> > management facilities available and hit upon this problem.
>
> The memory management present in the kernel has some
> features that would allow you to access more memory.
> Why don't you use them?
>

Such as? I am aware of kmalloc, get_free_pages and vmalloc. Since my
main requirement is buffers for I/O I require contiguous physical
address space which kmalloc does not provide. kmalloc does not provide
large enough areas and that leaves get_free_pages. If there is
something I have missed then I would be grateful to know what it is.

> >
> > I admit my understanding of the x86 architecture is not as good as I
> > would like but, correct me if I'm wrong, I do believe that the kernel
> > address space and the user process address space are mapped through
> > separate segment registers
>
> Yes.
>
> > and thus are totally distinct.
>
> No, not on x86.

Distinct in the sense that they map different virtual address spaces.
Or at least I would sincerely hope they should. The fact that they map
onto the same physical address space is neither here nor there.

>
> > For a user
> > process to access the kernel, I was under the impression that the set
> > of segment registers are changed so I really do not understand your
> > comment about overlap. The two virtual address spaces are separate and
> > all that needs to be mapped into the user address space is the kernel
> > code, once the switch to kernel space is made you enter a totally
> > separate virtual address space.
>
> You don't understand the architecture you are
> working with.
>

I may not understand the architecture in detail but I do know that
each process can have a 4GB virtual address space (I checked this in
the Intel architecture manual). According to the book The Linux Kernel
(admittedly covering 2.2 kernel and the only 2.4 changes mentioned is
support for the PAE to handle upto 64GB of physical memory), 3GB of
the user address space is accessible by user processes or the kernel
while the remaining 1GB is accessible to the kernel only. There may be
some performance benefits from the scheme that is used in Linux
although these escape me for the moment.

> > As I say, I may be wrong in my
> > understanding of the hardware architecture
>
> You are wrong.
>
> > but I do understand OS theory extermely well.
>
> Welcome to the real world, where backward compatibility
> is design criteria number one before all those you know
> from the theory. In one of Tanenbaums books, you will
> find a fairly good overview of the i386 paging and
> segmentation design. IIRC it is Structure Computer
> Organization (otherwise it must be Modern Operating
> Systems).

I am well aware of the real world. I work in it all the time. Having
just replaced the motherboard on my system (the old one was less than
two years old) and had to ditch my processor, memory and soundcard
with the graphics adaptor just escaping the bin, there appears to be
little attempt in the H/W arena to address backward compatibility
issues in the PC arena. As for software, I agree backward compatibilty
is a prime design criteria but the memory managment features needed to
support paging and a 4GB virtual address space in the x86 architecture
have been there at least since the 386. Therefore the initial design
was not hampered by earlier hardware constraints. The only changes
have been the addition of a futher 2 H/W segment registers but the
basic memory management H/W is as it was when introduced.

>
> >
> > I don't claim to be an expert in the Linux kernel, but I have come
> > across a number of areas within the kernel that do not make sense to
> > me when I compare similar functions in other OS's I have worked on.
> > Anyway I would like to thank you for your comments and the time you
> > have taken to respond. It is much appreciated and at least I now know
> > I can modify the kernel split - just have to find out where this is
> > set up.
>
> In 2.2 kernels there was a compile time option to chose
> between a 3:1 split and a 2:2 split. But that might
> have been removed with the introduction of high memory,
> that removed the need for more than 1GB of kernel
> address space.
>
> Otherwise just try changing __PAGE_OFFSET in
> include/asm-i386/page.h from 0xC0000000 to 0x80000000.
> I think you need to change arch/i386/vmlinux.lds as well.

George Nelson

Relevant Pages

  • Re: Allocating kernel memory
    ... read the Intel architecture spec closely enough although I still wish ... surprise at the 1GB kernel limit. ... The memory bandwidth really should ... not know enough about how the TLB works. ...
    (comp.os.linux.development.system)
  • [UNIX] Linux Kernel do_brk() Vulnerablility (Explained)
    ... Get your security news from a reliable source. ... A critical security bug has been found in the Linux kernel 2.4.22 (and ... earlier) memory management subsystem. ... for the code working at the lowest privilege level. ...
    (Securiteam)
  • kernel panic - not syncing: out of memory and no killable processes
    ... Kernel panic - not syncing: Out of memory and no killable processes... ... Re-tune the scheduler latency defaults to decrease worst-case latencies ... # Device Drivers ...
    (Linux-Kernel)
  • Re: Cached memory never gets released
    ... Stock linux 2.4.26 kernel. ... Due to flash bug 3M of memory gets lost due to font memory getting lost ... The output of "free" cache number steadily grows. ... longer to exhaust all of system memory with the cache. ...
    (Linux-Kernel)
  • Re: memory not released after using cdrecord/cdrdao (was: audio cd writing causes massive swap and c
    ... The kernel is most definitely not releasing the memory ... I further would have to say that the kernel isn't being ... >after cdrdao or cdrecord have used it, ... the Linux Kernel maintainers have not been very helpful with Linux ...
    (Linux-Kernel)