WinDbg : How To Debug Memory Leaks With The !heap Command

WinDbg : How To Debug Memory Leaks With The !heap Command

Memory and resource leaks are best debugged on a live system. There are several user and kernel mode tools available to help us. But there are times when we get a process/kernel crash dump file, and the reason shown is that the entire virtual memory was consumed! These type of crashes are more likely to be seen on older 32 bit operating systems, since the 64 bit address space is huge and hence hard to overflow, also, the newer operating systems have more efficient heap managers and memory managers. Before we go on, I would recommend that you read this article here about the introductory concepts of heaps. 

All through out this article we will be focusing on a user mode service crash on a 32 bit Windows 2003 OS. 

Commands used: 

1. !heap (introduced for the first time in this article)
2. dds (this command is used in this article here). For more details of the d command, please read this.
3. $
4. ||
5. .expr
6. ??

The dump file used here is a full user mode dump.

0:004> $ Lets start with seeing what kind of dump file we are dealing with...

0:004> ||

.  0 Full memory user mini dump: E:\Public\PID-18800__MYFAULTYSERVICE.EXE__2nd_chance_CPlusPlusEH__full_19ac_2014-09-09_15-54-23-328_4970.dmp

0:004> $ Okay, so it is a full memory user dump. now lets see that state of the heap

0:004> !heap -s

  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 

                    (k)     (k)    (k)     (k) length      blocks cont. heap 

-----------------------------------------------------------------------------

00150000 00000002    4096    644   1244     74     8    10    0      0   L  

00250000 00008000      64     12     12     10     1     1    0      0      

007e0000 00001002      64     52     52      5     1     1    0      0   L  

00800000 00000002    1024     20     20      3     1     1    0      0   L  

00a00000 00001002    4096   2388   2388     13     5     3    0      0   L  

Virtual block: 01040000 - 01040000 (size 00000000)

00b00000 00001002 1960256 1960256 1960256     70     0     0    1      7   L  

00f30000 00001002      64     28     28      7     4     1    0      0   L  

01520000 00001002      64     16     16      4     1     1    0      0   L  

06330000 00001002    7232   4060   5824   1077   226    24    0      0   L  

    External fragmentation  26 % (226 free blocks)

06350000 00001003    1280    220    268     37     7     7    0    N/A   

065c0000 00001003     256      4      4      2     1     1    0    N/A   

06a90000 00001003     256      4      4      2     1     1    0    N/A   

06ad0000 00001003     256      4      4      2     1     1    0    N/A   

06b10000 00001003     256      4      4      2     1     1    0    N/A   

06390000 00001002      64     16     16      3     1     1    0      0   L  

-----------------------------------------------------------------------------

0:004> $ The largest allocations seem to be happening in the heap 00b00000. 

0:004> $ That doesn't mean it is leaking, but it is a good starting point

0:004> $ using the heap's statistics, we can see the types of allocations happening in it

0:004> !heap -stat -h 00b00000

 heap @ 00b00000

group-by: TOTSIZE max-display: 20

    size     #blocks     total     ( %) (percent of total busy bytes)

    70 84a897 - 3a09c210  (53.53)

    30 aa8149 - 1ff83db0  (29.49)

    60 25da12 - e31c6c0  (13.09)

    38 12ecf1 - 423d4b8  (3.82)

    a8980 1 - a8980  (0.04)

    40 97a - 25e80  (0.01)

    200 106 - 20c00  (0.01)

    47 3d9 - 1112f  (0.00)

    18 725 - ab78  (0.00)

    d0 80 - 6800  (0.00)

    90 96 - 5460  (0.00)

    50 fa - 4e20  (0.00)

    20 1fa - 3f40  (0.00)

    8c 4e - 2aa8  (0.00)

    80 54 - 2a00  (0.00)

    24 129 - 29c4  (0.00)

    78 3e - 1d10  (0.00)

    800 3 - 1800  (0.00)

    28 74 - 1220  (0.00)

    1000 1 - 1000  (0.00)

0:004> $ In this case, there seem to be an allocation unit of 0x70 bytes,

0:004> $ which is consuming 53.53 percent of the heap allocations

0:004> $ This allocation is definitely a candidate for suspicion 

0:004> $ Lets try to get all information about this heap (this command can take a really long time)

    

0:004> !heap -a 00b00000

Index   Address  Name      Debugging options enabled

  6:   00b00000 

    Segment at 00b00000 to 00b10000 (00010000 bytes committed)

    Segment at 00b10000 to 00c10000 (00100000 bytes committed)

<Output trimmed to save space>

    Segment at 77980000 to 77b5e000 (001de000 bytes committed)

    Segment at 7c4e0000 to 7c6be000 (001de000 bytes committed)

    Segment at 7dd60000 to 7df3f000 (001df000 bytes committed)

    Flags:                00001002

    ForceFlags:           00000000

    Granularity:          8 bytes

    Segment Reserve:      77b40000

    Segment Commit:       00002000

    DeCommit Block Thres: 00000200

    DeCommit Total Thres: 00002000

    Total Free Size:      00002365

    Max. Allocation Size: 7ffdefff

    Lock Variable at:     00b00608

    Next TagIndex:        0000

    Maximum TagIndex:     0000

    Tag Entries:          00000000

    PsuedoTag Entries:    00000000

    Virtual Alloc List:   00b00050

    Unable to read nt!_HEAP_VIRTUAL_ALLOC_ENTRY structure at 01040000

    UCR FreeList:        010f0d70

    UCRSegment - 010f0000: 00003000 . 00010000

    FreeList Usage:      0000027c 00000000 00000000 00000000

    FreeList[ 02 ] at 00b00188: 7df0e970 . 00b26798  

    Unable to read nt!_HEAP_FREE_ENTRY structure at 00b26798

    FreeList[ 03 ] at 00b00190: 5766a650 . 0189e728  

    Unable to read nt!_HEAP_FREE_ENTRY structure at 0189e728

    FreeList[ 04 ] at 00b00198: 5b228800 . 0178cee0  

    Unable to read nt!_HEAP_FREE_ENTRY structure at 0178cee0

    FreeList[ 05 ] at 00b001a0: 64bb7cc0 . 00b4cc70  

    Unable to read nt!_HEAP_FREE_ENTRY structure at 00b4cc70

    FreeList[ 06 ] at 00b001a8: 016a1418 . 00b286d0  

    Unable to read nt!_HEAP_FREE_ENTRY structure at 00b286d0

    FreeList[ 09 ] at 00b001c0: 7defe050 . 7defe050  

    Unable to read nt!_HEAP_FREE_ENTRY structure at 7defe050

    Segment00 at 00b00640:

        Flags:           00000000

        Base:            00b00000

        First Entry:     00b00680

        Last Entry:      00b10000

        Total Pages:     00000010

        Total UnCommit:  00000000

        Largest UnCommit:00000000

        UnCommitted Ranges: (0)

    Heap entries for Segment00 in Heap 00b00000

         address: psize . size  flags   state (requested size)

        00b00640: 00640 . 00040 [01] - busy (40)

<Output trimmed to save space>

        00b04dc8: 00018 . 00038 [01] - busy (30)

        00b04e00: 00038 . 00078 [01] - busy (70)

        00b04e78: 00078 . 00078 [01] - busy (70)

        00b04ef0: 00078 . 00078 [01] - busy (70)

        00b04f68: 00078 . 00078 [01] - busy (70)

        00b04fe0: 00078 . 00078 [01] - busy (70)

        00b05058: 00078 . 00038 [01] - busy (30)

        00b05090: 00038 . 00038 [01] - busy (30)

        00b050c8: 00038 . 00078 [01] - busy (70)

        00b05140: 00078 . 00028 [01] - busy (20)

        00b05168: 00028 . 00078 [01] - busy (70)

        00b051e0: 00078 . 00038 [01] - busy (30)

        00b05218: 00038 . 00078 [01] - busy (70)

        00b05290: 00078 . 00028 [01] - busy (20)

        00b052b8: 00028 . 00078 [01] - busy (70)

        00b05330: 00078 . 00028 [01] - busy (20)

        00b05358: 00028 . 00078 [01] - busy (70)

<Output truncated to save space>

0:004> $ Since we suspect the 70 byte allocation, lets start dumping the memory of these objects for any clues

0:004> db 00b04e00

00b04e00  0f 00 07 00 15 01 08 00-3c 5a 5b 00 78 03 b0 00  ........<Z[.x...

00b04e10  00 00 00 00 00 00 00 00-80 4e b0 00 00 00 00 00  .........N......

00b04e20  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................

00b04e30  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................

00b04e40  0f 00 00 00 00 00 00 00-41 70 70 20 54 6f 74 61  ........App Tota

00b04e50  6c 00 00 00 00 00 00 00-09 00 00 00 0f 00 00 00  l...............

00b04e60  00 00 00 00 d0 4d b0 00-5c fb d3 e4 e9 c3 02 00  .....M..\.......

00b04e70  01 00 00 00 00 00 00 00-0f 00 0f 00 1a 01 08 00  ................

0:004> $ Looks like an ASCII string in it, perhaps an eye catcher field in a structure or object...

0:004> $ For C++ objects, the first DWORD is the VTABLE pointer, and sometimes these pointers have class names in symbols

0:004> $ Lets try the dds command to see if it can identify any symbol

0:004> dds 00b04e00

00b04e00  0007000f

00b04e04  00080115

00b04e08  005b5a3c MyFaultyService!CTimer::`vftable'

00b04e0c  00b00378

00b04e10  00000000

00b04e14  00000000

00b04e18  00b04e80

00b04e1c  00000000

<Output truncated>

0:004> $ Lets try the dds command on another 70 byte entry to see if it can identify any symbol

0:004> dds 00b04f68

00b04f68  000f000f

00b04f6c  00080138

00b04f70  005b5a3c MyFaultyService!CTimer::`vftable'

00b04f74  00b00210

00b04f78  0038ce92

00b04f7c  00000000

<Output truncated>

Looks to be a problem with the class CTimer inside the binary. At this point, we are in a position to examine the class and it's objects to see if a possible problem can be detected. 

As as additional check, we can try to see whether our timer object indeed takes up 0x70 bytes.

0:004> .expr
Current expression evaluator: MASM - Microsoft Assembler expressions
0:004> .expr /s c++
Current expression evaluator: C++ - C++ source expressions
0:004> ??sizeof(myfaultyservice!Ctimer)

unsigned int 0x70

Which confirms our hunch.

We were lucky, since this was a C++ object and it had a VTABLE. We are also lucky, that it is our code and we had the symbols for it. The case of finding such objects in memory without symbols might be an even daunting task.

Also to be noted is that just because there are too many instances of an object, doesn't mean that it leaks. The debugger is trying to help you here, but you are the best judge of your code.

WinDbg : An Introduction To Windows Heaps

WinDbg : An Introduction To Windows Heaps 

1. dt
2. dd
3. !process
4. .process
5. $ 

Heaps are used by applications which need to allocate and release memory dynamically. Even though the heap is the most common facility to accommodate dynamic memory allocations, it is not the only way. Several other methods are available:

• Memory can be requested through the C run time (CRT).
• The virtual memory manager or some other private memory manager.

Even though these methods are different, they are tightly coupled internally at the memory manager's level.

When a process starts, the heap manager automatically creates a new heap called the 'default process heap'. Although some processes are known to use the default heap, a very very large number rely on the CRT heap (using malloc/free family of APIs, or the default new/delete family of APIs).

Some processes however also create additional heaps (via HeapCreate(...) ). this method has advantage of isolating different components running in the process. Thus memory leaks can be easier to detect and debug, if we know exactly which heap is leaking. It is indeed good programming practice. Kernel programmers are used to using different tags for allocating memory for different objects, this method described above is the usermode solution to the problem.

The Windows heap manager has two sub components. 

• Front End Allocator.
• Back End Allocator.

The Front End Allocator is an optimization layer for the actual Back End Allocator. Thus, by choosing different front end allocators applications with differnt memory requirements can function appropriately. Example, some programs might expect small bursts of allocations, and thus might prefer to use a low fragmentation front end allocator. 

Windows supplies two different front end allocators:

• Low fragmentation (LF) Front End Allocator.
• Look aside List (LAL) Front End Allocator.

Vista and above use the LF allocator by default, where as all older generations use the LAL allocator. The differences between these allocators re beyond the scope of the current discussion.

If the Front End Allocator is unable to satisfy the allocation requests, then the Back End Allocator takes over. It is made up of free lists. Each list has blocks of specific sizes. Such free blocks are sorted in ascending order of sizes. Again, a more elaborate discussion on how this is managed, or of it's algorithms is beyond the scope of this current article.

Enough theory, lets get into some practicals.

The first step for us is to determine which all heaps are active for a process.

kd> $ Lets use the lsass process as an example.

kd> $ First we need to find it's EPROCESS

kd> !process 0 0 lsass.exe

PROCESS 84cdc860  SessionId: 0  Cid: 0208    Peb: 7ffda000  ParentCid: 018c

    DirBase: 1eed30e0  ObjectTable: 95f00d18  HandleCount: 513.

    Image: lsass.exe

kd> $ Next we need to switch contexts to this process (this is because this session is running from Kernel mode)

kd> .process /p /r 84cdc860  

Implicit process is now 84cdc860

Loading User Symbols

............................................................

kd> $ The Process execution block (_PEB) helps us with finding the active heaps, so lets use dt to find it

kd> dt _PEB @$peb

ntdll!_PEB

   +0x000 InheritedAddressSpace : 0 ''

   +0x001 ReadImageFileExecOptions : 0 ''

   +0x002 BeingDebugged    : 0 ''

   +0x003 BitField         : 0x8 ''

   +0x003 ImageUsesLargePages : 0y0

   +0x003 IsProtectedProcess : 0y0

   +0x003 IsLegacyProcess  : 0y0

   +0x003 IsImageDynamicallyRelocated : 0y1

   +0x003 SkipPatchingUser32Forwarders : 0y0

   +0x003 SpareBits        : 0y000

   +0x004 Mutant           : 0xffffffff Void

   +0x008 ImageBaseAddress : 0x00d90000 Void

   +0x00c Ldr              : 0x77c77880 _PEB_LDR_DATA

   +0x010 ProcessParameters : 0x00330f18 _RTL_USER_PROCESS_PARAMETERS

   +0x014 SubSystemData    : (null) 

   +0x018 ProcessHeap      : 0x00330000 Void

   +0x01c FastPebLock      : 0x77c77380 _RTL_CRITICAL_SECTION

   +0x020 AtlThunkSListPtr : (null) 

   +0x024 IFEOKey          : (null) 

   +0x028 CrossProcessFlags : 0

   +0x028 ProcessInJob     : 0y0

   +0x028 ProcessInitializing : 0y0

   +0x028 ProcessUsingVEH  : 0y0

   +0x028 ProcessUsingVCH  : 0y0

   +0x028 ProcessUsingFTH  : 0y0

   +0x028 ReservedBits0    : 0y000000000000000000000000000 (0)

   +0x02c KernelCallbackTable : 0x7790d568 Void

   +0x02c UserSharedInfoPtr : 0x7790d568 Void

   +0x030 SystemReserved   : [1] 0

   +0x034 AtlThunkSListPtr32 : 0

   +0x038 ApiSetMap        : 0x77de0000 Void

   +0x03c TlsExpansionCounter : 0

   +0x040 TlsBitmap        : 0x77c77260 Void

   +0x044 TlsBitmapBits    : [2] 0x1fffff

   +0x04c ReadOnlySharedMemoryBase : 0x7f6f0000 Void

   +0x050 HotpatchInformation : (null) 

   +0x054 ReadOnlyStaticServerData : 0x7f6f0590  -> (null) 

   +0x058 AnsiCodePageData : 0x7ffb0000 Void

   +0x05c OemCodePageData  : 0x7ffc0224 Void

   +0x060 UnicodeCaseTableData : 0x7ffd0648 Void

   +0x064 NumberOfProcessors : 1

   +0x068 NtGlobalFlag     : 0

   +0x070 CriticalSectionTimeout : _LARGE_INTEGER 0xffffe86d`079b8000

   +0x078 HeapSegmentReserve : 0x100000

   +0x07c HeapSegmentCommit : 0x2000

   +0x080 HeapDeCommitTotalFreeThreshold : 0x10000

   +0x084 HeapDeCommitFreeBlockThreshold : 0x1000

   +0x088 NumberOfHeaps    : 4

   +0x08c MaximumNumberOfHeaps : 0x10

   +0x090 ProcessHeaps     : 0x77c77500  -> 0x00330000 Void

   +0x094 GdiSharedHandleTable : 0x004d0000 Void

   +0x098 ProcessStarterHelper : (null) 

   +0x09c GdiDCAttributeList : 0x14

   +0x0a0 LoaderLock       : 0x77c77340 _RTL_CRITICAL_SECTION

   +0x0a4 OSMajorVersion   : 6

   +0x0a8 OSMinorVersion   : 1

   +0x0ac OSBuildNumber    : 0x1db1

   +0x0ae OSCSDVersion     : 0x100

   +0x0b0 OSPlatformId     : 2

   +0x0b4 ImageSubsystem   : 2

   +0x0b8 ImageSubsystemMajorVersion : 6

   +0x0bc ImageSubsystemMinorVersion : 1

   +0x0c0 ActiveProcessAffinityMask : 1

   +0x0c4 GdiHandleBuffer  : [34] 0

   +0x14c PostProcessInitRoutine : (null) 

   +0x150 TlsExpansionBitmap : 0x77c77268 Void

   +0x154 TlsExpansionBitmapBits : [32] 1

   +0x1d4 SessionId        : 0

   +0x1d8 AppCompatFlags   : _ULARGE_INTEGER 0x0

   +0x1e0 AppCompatFlagsUser : _ULARGE_INTEGER 0x0

   +0x1e8 pShimData        : (null) 

   +0x1ec AppCompatInfo    : (null) 

   +0x1f0 CSDVersion       : _UNICODE_STRING "Service Pack 1"

   +0x1f8 ActivationContextData : 0x00040000 _ACTIVATION_CONTEXT_DATA

   +0x1fc ProcessAssemblyStorageMap : (null) 

   +0x200 SystemDefaultActivationContextData : 0x00030000 _ACTIVATION_CONTEXT_DATA

   +0x204 SystemAssemblyStorageMap : (null) 

   +0x208 MinimumStackCommit : 0

   +0x20c FlsCallback      : 0x0033daa0 _FLS_CALLBACK_INFO

   +0x210 FlsListHead      : _LIST_ENTRY [ 0x347828 - 0x3cb680 ]

   +0x218 FlsBitmap        : 0x77c77270 Void

   +0x21c FlsBitmapBits    : [4] 7

   +0x22c FlsHighIndex     : 2

   +0x230 WerRegistrationData : (null) 

   +0x234 WerShipAssertPtr : (null) 

   +0x238 pContextData     : 0x00050000 Void

   +0x23c pImageHeaderHash : (null) 

   +0x240 TracingFlags     : 0

   +0x240 HeapTracingEnabled : 0y0

   +0x240 CritSecTracingEnabled : 0y0

   +0x240 SpareTracingBits : 0y000000000000000000000000000000 (0)

kd> $ Now lets use dd to see what this address contains

kd> dd 0x77c77500  

77c77500  00330000 00010000 00250000 00d10000

77c77510  00000000 00000000 00000000 00000000

77c77520  00000000 00000000 00000000 00000000

77c77530  00000000 00000000 00000000 00000000

77c77540  00000000 77c77340 77c7ab08 77c77220

77c77550  00000000 00000004 00000000 00000000

77c77560  77c77220 003e3198 00000000 00000000

77c77570  00000000 00000000 00000000 00000000

The Default Process heap pointer is always the first one in this list. Since most applications work with the default heap, we will focus our attention on that.

The _HEAP structure in Windows is used to maintain a heap. So lets typecast this address to it.

kd> dt _HEAP 0x00330000

ntdll!_HEAP

   +0x000 Entry            : _HEAP_ENTRY

   +0x008 SegmentSignature : 0xffeeffee

   +0x00c SegmentFlags     : 0

   +0x010 SegmentListEntry : _LIST_ENTRY [ 0x3300a8 - 0x3300a8 ]

   +0x018 Heap             : 0x00330000 _HEAP

   +0x01c BaseAddress      : 0x00330000 Void

   +0x020 NumberOfPages    : 0x100

   +0x024 FirstEntry       : 0x00330588 _HEAP_ENTRY

   +0x028 LastValidEntry   : 0x00430000 _HEAP_ENTRY

   +0x02c NumberOfUnCommittedPages : 0x47

   +0x030 NumberOfUnCommittedRanges : 1

   +0x034 SegmentAllocatorBackTraceIndex : 0

   +0x036 Reserved         : 0

   +0x038 UCRSegmentList   : _LIST_ENTRY [ 0x3e8ff0 - 0x3e8ff0 ]

   +0x040 Flags            : 2

   +0x044 ForceFlags       : 0

   +0x048 CompatibilityFlags : 0

   +0x04c EncodeFlagMask   : 0x100000

   +0x050 Encoding         : _HEAP_ENTRY

   +0x058 PointerKey       : 0x37d910ba

   +0x05c Interceptor      : 0

   +0x060 VirtualMemoryThreshold : 0xfe00

   +0x064 Signature        : 0xeeffeeff

   +0x068 SegmentReserve   : 0x100000

   +0x06c SegmentCommit    : 0x2000

   +0x070 DeCommitFreeBlockThreshold : 0x800

   +0x074 DeCommitTotalFreeThreshold : 0x2000

   +0x078 TotalFreeSize    : 0xac0

   +0x07c MaximumAllocationSize : 0x7ffdefff

   +0x080 ProcessHeapsListIndex : 1

   +0x082 HeaderValidateLength : 0x138

   +0x084 HeaderValidateCopy : (null) 

   +0x088 NextAvailableTagIndex : 0

   +0x08a MaximumTagIndex  : 0

   +0x08c TagEntries       : (null) 

   +0x090 UCRList          : _LIST_ENTRY [ 0x3e8fe8 - 0x3e8fe8 ]

   +0x098 AlignRound       : 0xf

   +0x09c AlignMask        : 0xfffffff8

   +0x0a0 VirtualAllocdBlocks : _LIST_ENTRY [ 0x3300a0 - 0x3300a0 ]

   +0x0a8 SegmentList      : _LIST_ENTRY [ 0x330010 - 0x330010 ]

   +0x0b0 AllocatorBackTraceIndex : 0

   +0x0b4 NonDedicatedListLength : 0

   +0x0b8 BlocksIndex      : 0x00330150 Void

   +0x0bc UCRIndex         : 0x00330590 Void

   +0x0c0 PseudoTagEntries : (null) 

   +0x0c4 FreeLists        : _LIST_ENTRY [ 0x3db6a8 - 0x3de378 ]

   +0x0cc LockVariable     : 0x00330138 _HEAP_LOCK

   +0x0d0 CommitRoutine    : 0x37d910ba     long  +37d910ba

   +0x0d4 FrontEndHeap     : 0x00336548 Void

   +0x0d8 FrontHeapLockCount : 0

   +0x0da FrontEndHeapType : 0x2 ''

   +0x0dc Counters         : _HEAP_COUNTERS

   +0x130 TuningParameters : _HEAP_TUNING_PARAMETERS

Note: The _HEAP structure might be different for different versions of Windows.

A point to note here is that inside the _PEB there are two fields:

kd> dt _PEB ProcessH*

ntdll!_PEB

   +0x018 ProcessHeap : Ptr32 Void

   +0x090 ProcessHeaps : Ptr32 Ptr32 Void

The ProcessHeaps is an array of pointers to the various heaps for the process. The ProcessHeap on the other hand will always store the address of the default heap of the process.

To browse all heaps in this process, we can use the WinDbg .for command, which is described here.

kd> .for (r $t0 = 0; @$t0 < 0x4; r $t0 = @$t0 + 1){dt _HEAP poi(0x77c77500 + ((@$t0)*4)) - y SegmentSignature}

ntdll!_HEAP
   +0x008 SegmentSignature : 0xffeeffee
ntdll!_HEAP
   +0x008 SegmentSignature : 0xffeeffee
ntdll!_HEAP
   +0x008 SegmentSignature : 0xffeeffee
ntdll!_HEAP
   +0x008 SegmentSignature : 0xffeeffee

We will revisit heaps again for further elaborate discussions and even walking the heaps when we explore heap corruption scenarios.