Re: Help - Memory Leak Issue with SCSI Stack on Linux

From: John-Paul Stewart (jpstewart_at_binaryfoundry.ca)
Date: 08/30/04 

Date: Mon, 30 Aug 2004 17:49:27 -0400

L wrote:
>
>>Leave the test running in an infinite loop if
>>you can, while periodically checking memory use. If the test eventually
>>uses up all memory, then that *might* indicate a leak. On the other
>>hand, for all I know, the memory used could just be for the test program
>>to track test results!
>
>
> Might I ask what you meant by "the memory used could just be for the test
> program to track test results!"?

Your results show memory is being used. It doesn't show *where* it is
being used.

So consider the case where your test program (iometer, was it?)
allocates some memory for itself to store whatever the programmers want
to store. The size of the data structures within the test program could
grow with time. IOW, after one minute the test program stores X bytes
of data. After two minutes, it stores 2X bytes. After three minutes it
stores 3X bytes, and so on up to some limit.

That scenario is just one of many that fits the data you have shown but
does *not* indicate a memory leak.

>>To be quite frank, I've seen absolutely no evidence of a memory leak,
>>nor even any abnormal behaviour. Might I ask, why are you so concerned
>>about a memory leak?
>
>
> Sure and I really appreciate your help! Here is the whole story why I have
> been trying to trace memory usage on Linux.
>
> We are working on a project that inserts a couple of extra layers including
> a SCSI low-level driver into Linux SCSI stack. People reported that the
> memory been used kept increasing while using our project.
>
> To get some experience on how the memory is used on Linux system, especially
> when running I/Os in SCSI stack, I use iometer to generate such I/Os.
>
> Without using our project (just use original SCSI mid-layer, low-level
> driver, etc. come with Linux), it seems memory been used ('-/+
> buffers/cache' in the output of 'free') increases from 155MB to 211 MB only
> during the first 10 min, then it stays the same. This is good.
>
> With our project (extra layers inserted into Linux SCSI stack), memory been
> used (still '-/+ buffers/cache' in the output of 'free') keeps increasing
> from 150MB to 650MB during the first 100 min, then the system hangs (it can
> not responde to keyboard or mouse any more, and a hard reboot is needed).
> This is bad.
>
> Based on what you told me, it seems our project probably has a sort of
> memory leak some where, agree?

Yes, it certainly seems as though your project has a leak.

> I'm not sure if there are any tools
> available for tracing memory leak in kernel modules. Our project has a few
> kernel modules working together, and I guess tracing memory leaking in
> kernel is not a easy job.

I'm no kernel developer, so I don't know how much help I can offer, but
it certainly seems like you've got a leak in your module. You'll need
to track every allocation of memory somehow. Probably something along
the lines of this will suffice:

pointer = kmalloc(size)
#ifdef DEBUG
   printk("allocated %d bytes of memory at %d\n", size, pointer)
#endif

kfree(pointer)
#ifdef DEBUG
   printk("Freed %d", pointer)
#endif

You'd have to make every call to kmalloc log what it is doing. Then
every call to kfree would have to log its actions. You can then look
through those logs and see which pointers are *not* being freed. You
might want your log messages to also indicate where in the source
they're coming from so you can figure out what is being allocated
without being properly freed.

Debugging a memory leak in the kernel is really no different from
debugging any other memory leak *manually*. OTOH, if you're used to
relying on automated tools for tracking leaks, things could be a little
trickier....

Relevant Pages

  • [NT] Services for UNIX 2.0 Suffer from a Remotely Triggered Memory Leak
    ... Services for UNIX 2.0 Suffer from a Remotely Triggered Memory Leak ... 2000 are not affected by the vulnerability. ... * A vulnerability that could enable an attacker to cause the NFS service ...
    (Securiteam)
  • Re: Annoying kernel memory leak in U6?
    ... This is a lot of memory taken by the kernel, ... not necessarily a memory leak. ... memory allocation in the kernel, ...
    (comp.unix.solaris)
  • RE: Memory increasing even when in debug-break
    ... there are 2 types of memory leak in .Net: ... More interesting is the debugger breaks while memory still increasing ... Microsoft Online Community Support ...
    (microsoft.public.vsnet.debugging)
  • Re: 6.1-RC2: strange kernel panic!
    ... is too heavy for your current settings, or there is a memory leak ... Try to increase VM_KMEM_SIZE_MAX in your kernel, ... the memory goes on this server, and how i can prevent this type of leak. ... our current settings are: ...
    (freebsd-questions)
  • Re: Memory leak in the Driver
    ... will it also work for device driver ??? ... if the following application is used then, there is no memory leak. ...
    (microsoft.public.windowsce.platbuilder)