dmalloc.html

减少内存碎片的malloc分配函数

<P>
<A NAME="IDX42"></A>
<A NAME="IDX43"></A>


<P>
Any program can be divided into 2 logical parts: text and data.  Text is
the actual program code in machine-readable format and data is the
information that the text operates on when it is executing.  The data,
in turn, can be divided into 3 logical parts according to where it is
stored: <EM>static</EM>, <EM>stack</EM>, and <EM>heap</EM>.


<P>
<A NAME="IDX44"></A>


<P>
Static data is the information whose storage space is compiled into the
program.



<PRE>
/* global variables are allocated as static data */
int numbers[10];

main()
{
        ...
}
</PRE>

<P>
<A NAME="IDX45"></A>


<P>
Stack data is data allocated at runtime to hold information used inside
of functions.  This data is managed by the system in the space called
stack space.



<PRE>
void foo()
{
        /* this local variable is stored on the stack */
        float total;
        ...
}

main()
{
        foo();
}
</PRE>

<P>
<A NAME="IDX46"></A>


<P>
Heap data is also allocated at runtime and provides a programmer with
dynamic memory capabilities.



<PRE>
main()
{
        /* the address is stored on the stack */
        char * string;
        ...

        /*
         * Allocate a string of 10 bytes on the heap.  Store the
         * address in string which is on the stack.
         */
        string = (char *)malloc(10);
        ...

        /* de-allocate the heap memory now that we're done with it */
        (void)free(string);
        ...
}
</PRE>

<P>
It is the heap data that is managed by this library.


<P>
Although the above is an example of how to use the malloc and free
commands, it is not a good example of why using the heap for runtime
storage is useful.


<P>
Consider this: You write a program that reads a file into memory,
processes it, and displays results.  You would like to handle files with
arbitrary size (from 10 bytes to 1.2 megabytes and more).  One problem,
however, is that the entire file must be in memory at one time to do the
calculations.  You don't want to have to allocate 1.2 megabytes when you
might only be reading in a 10 byte file because it is wasteful of system
resources.  Also, you are worried that your program might have to handle
files of more than 1.2 megabytes.


<P>
A solution: first check out the file's size and then, using the
heap-allocation routines, get enough storage to read the entire file
into memory.  The program will only be using the system resources
necessary for the job and you will be guaranteed that your program can
handle any sized file.




<H3><A NAME="SEC7" HREF="dmalloc.html#TOC7">2.3.2 Functionality Supported by All Malloc Libraries</A></H3>

<P>
<A NAME="IDX47"></A>


<P>
All malloc libraries support 4 basic memory allocation commands.  These
include <EM>malloc</EM>, <EM>calloc</EM>, <EM>realloc</EM>, and <EM>free</EM>.  For
more information about their capabilities, check your system's manual
pages -- in unix, do a <CODE>man 3 malloc</CODE>.


<P>
<DL>
<DT><U>Function:</U>  <B></B>
<DD><A NAME="IDX48"></A>
void *malloc ( unsigned int <VAR>size</VAR> )
<A NAME="IDX49"></A>


<P>
Usage: <CODE>pnt = (type *)malloc(size)</CODE>


<P>
The malloc routine is the basic memory allocation routine.  It allocates
an area of <CODE>size</CODE> bytes.  It will return a pointer to the space
requested.


</DL>
<P>
<DL>
<DT><U>Function:</U>  <B></B>
<DD><A NAME="IDX50"></A>
void *calloc ( unsigned int <VAR>number</VAR>, unsigned int
<VAR>size</VAR> )
<A NAME="IDX51"></A>
<A NAME="IDX52"></A>
<A NAME="IDX53"></A>


<P>
Usage: <CODE>pnt = (type *)calloc(number, size)</CODE>


<P>
The calloc routine allocates a certain <CODE>number</CODE> of items, each of
<CODE>size</CODE> bytes, and returns a pointer to the space.  It is
appropriate to pass in a <CODE>sizeof(type)</CODE> value as the size argument.


<P>
Also, calloc nulls the space that it returns, assuring that the memory
is all zeros.


</DL>
<P>
<DL>
<DT><U>Function:</U>  <B></B>
<DD><A NAME="IDX54"></A>
void *realloc ( void *<VAR>old_pnt</VAR>, unsigned int
<VAR>new_size</VAR> )
<A NAME="IDX55"></A>


<P>
Usage: <CODE>new_pnt = (type *)realloc(old_pnt, new_size)</CODE>


<P>
The realloc function expands or shrinks the memory allocation in
<CODE>old_pnt</CODE> to <CODE>new_size</CODE> number of bytes.  Realloc copies as
much of the information from <CODE>old_pnt</CODE> as it can into the
<CODE>new_pnt</CODE> space it returns, up to <CODE>new_size</CODE> bytes.  If there
is a problem allocating this memory, 0L will be returned.


<P>
If the <CODE>old_pnt</CODE> is 0L then realloc will do the equivalent of a
<CODE>malloc(new_size)</CODE>.  If <CODE>new_size</CODE> is 0 and <CODE>old_pnt</CODE> is
not 0L, then it will do the equivalent of <CODE>free(old_pnt)</CODE> and will
return 0L.


</DL>
<P>
<DL>
<DT><U>Function:</U>  <B></B>
<DD><A NAME="IDX56"></A>
void free ( void *<VAR>pnt</VAR> )
<A NAME="IDX57"></A>


<P>
Usage: <CODE>free(pnt)</CODE>


<P>
The free routine releases allocation in <CODE>pnt</CODE> which was returned by
malloc, calloc, or realloc back to the heap.  This allows other parts of
the program to re-use memory that is not needed anymore.  It guarantees
that the process does not grow too big and swallow a large portion of
the system resources.


</DL>

<P>
<EM>WARNING</EM>: there is a quite common myth that all of the space that
is returned by malloc libraries has already been cleared.  <EM>Only</EM>
the <CODE>calloc</CODE> routine will zero the memory space it returns.




<H2><A NAME="SEC8" HREF="dmalloc.html#TOC8">2.4 General Features of the Library</A></H2>

<P>
<A NAME="IDX58"></A>


<P>
The debugging features that are available in this debug malloc library
can be divided into a couple basic classifications:


<DL COMPACT>

<DT>file and line number information
<DD>
<A NAME="IDX59"></A>
<A NAME="IDX60"></A>
One of the nice things about a good debugger is its ability to provide
the file and line number of an offending piece of code.  This library
attempts to give this functionality with the help of <EM>cpp</EM>, the C
preprocessor.  See section <A HREF="dmalloc.html#SEC11">3.1 Macros Providing File and Line Information</A>.

<DT>return-address information
<DD>
<A NAME="IDX61"></A>
To debug calls to the library from external sources (i.e. those files
that could not use the allocation macros), some facilities have been
provided to supply the caller's address.  This address, with the help of
a debugger, can help you locate the source of a problem.  See section <A HREF="dmalloc.html#SEC12">3.2 Getting Caller Address Information</A>.

<DT>fence-post (i.e. bounds) checking
<DD>
<A NAME="IDX62"></A>
<A NAME="IDX63"></A>
<A NAME="IDX64"></A>
<EM>Fence-post</EM> memory is the area immediately above or below memory
allocations.  It is all too easy to write code that accesses above or
below an allocation -- especially when dealing with arrays or strings.
The library can write special values in the areas around every
allocation so it will notice when these areas have been overwritten.
See section <A HREF="dmalloc.html#SEC21">3.8.3 Diagnosing Fence-Post Overwritten Memory</A>.

<EM>NOTE</EM>: The library cannot notice when the program <EM>reads</EM>
from these areas, only when it writes values.  Also, fence-post checking
will increase the amount of memory the program allocates.

<DT>heap-constancy verification
<DD>
<A NAME="IDX65"></A>
The administration of the library is reasonably complex.  If any of the
heap-maintenance information is corrupted, the program will either crash
or give unpredictable results.

By enabling heap-consistency checking, the library will run through its
administrative structures to make sure all is in order.  This will mean
that problems will be caught faster and diagnosed better.

The drawback of this is, of course, that the library often takes quite a
long time to do this.  It is suitable to enable this only during
development and debugging sessions.

<EM>NOTE</EM>: the heap checking routines cannot guarantee that the tests
will not cause a segmentation-fault if the heap administration
structures are properly (or improperly if you will) overwritten.  In
other words, the tests will verify that everything is okay but may not
inform the user of problems in a graceful manner.

<DT>logging statistics
<DD>
<A NAME="IDX66"></A>
<A NAME="IDX67"></A>
<A NAME="IDX68"></A>
<A NAME="IDX69"></A>
One of the reasons why the debug malloc library was initially developed
was to track programs' memory usage -- specifically to locate
<EM>memory leaks</EM> which are places where allocated memory is never
getting freed.  See section <A HREF="dmalloc.html#SEC20">3.8.2 Tracking Down Non-Freed Memory</A>.

The library has a number of logging capabilities that can track un-freed
memory pointers as well as runtime memory usage, memory transactions,
administrative actions, and final statistics.

<DT>examining freed memory
<DD>
<A NAME="IDX70"></A>
<A NAME="IDX71"></A>
Another common problem happens when a program frees a memory pointer but
goes on to use it again by mistake.  This can lead to mysterious crashes
and unexplained problems.

To combat this, the library can write special values into a block of
memory after it has been freed.  This serves two purposes: it will make
sure that the program will get garbage data if it trying to access the
area again, and it will allow the library to verify the area later for
signs of overwriting.
</DL>

<P>
If any of the above debugging features detect an error, the library will
try to recover.  If logging is enabled then an error will be logged with
as much information as possible.


<P>
The error messages that the library displays are designed to give the
most information for developers.  If the error message is not
understood, then it is most likely just trying to indicate that a part
of the heap has been corrupted.


<P>
<A NAME="IDX72"></A>
<A NAME="IDX73"></A>
The library can be configured to quit immediately when an error is
detected and to dump a core file or memory-image.  This can be examined
with a debugger to determine the source of the problem.  The library can
either stop after dumping core or continue running.


<P>
<A NAME="IDX74"></A>
<A NAME="IDX75"></A>
<EM>NOTE</EM>: do not be surprised if the library catches problems with
your system's routines.  It took me hours to finally come to the
conclusion that the localtime call, included in SunOS release 4.1,
overwrites one of its fence-post markers.




<H2><A NAME="SEC9" HREF="dmalloc.html#TOC9">2.5 How the Library Checks Your Program</A></H2>

<P>
This is one of the newer sections of the library implying that it is
incomplete.  If you have any questions or issues that you'd like to see
handled here, please let me know.


<P>
The dmalloc library replaces the heap library calls normally found in
your system libraries with its own versions.  When you make a call to
malloc (for example), you are calling dmalloc's version of the memory
allocation function.  When you allocate memory with these functions, the
dmalloc library keeps track of a number of pieces of debugging
information about your pointer including: where it was allocated,
exactly how much memory was requested, when the call was made, etc..
This information can then be verified when the pointer is freed or
reallocated and the details can be logged on any errors.


<P>
Whenever you reallocate or free a memory address, the dmalloc library
always performs a number of checks on the pointer to make sure that it
is valid and has not been corrupted.  You can configure the library to
perform additional checks such as detected fence-post writing.  The
library can also be configured to overwrite memory with non-zeros (only
if calloc is not called) when it is allocated and erase the memory when
the pointers are freed.


<P>
<A NAME="IDX76"></A>
In addition to per-pointer checks, you can configure the library to
perform complete heap checks.  These complete checks verify all internal
heap structures and include walking all of the known allocated pointers
to verify each one in turn.  You need this level of checking to find
random pointers in your program which got corrupted but that won't be
freed for a while.  To turn on these checks, you will need to enable the
'check-heap' debug token.  See section <A HREF="dmalloc.html#SEC30">4.4 Description of the Debugging Tokens</A>.  By default this will
cause the heap to be fully checked each and every time dmalloc is called
whether it is a malloc, free, realloc, or another dmalloc overloaded
function.


<P>
Performing a full heap check can take a good bit of CPU and it may be
that you will want to run it sporadically.  This can be accomplished in
a couple different ways including the '-i' interval argument to the
dmalloc utility.  See section <A HREF="dmalloc.html#SEC26">4. Dmalloc Utility Program</A>.  This will cause the check to
be run every N-th time.  For instance, 'dmalloc -i 3' will cause the
heap to be checked before every 3rd call to a memory function.  Values
of 100 or even 1000 for high memory usage programs are more useful than
smaller ones.


<P>
<A NAME="IDX77"></A>