Search This Blog

Thursday 30 October 2014

WinDbg : Debugging A Critical Section Based Dead Lock Scenario

WinDbg : Debugging A Critical Section Based Dead Lock Scenario


Dead locks can be very nasty and sometimes very hard to debug and root cause. This post is an attempt to walk you through one such scenario, where a critical section dead locks a thread. As with most of the previous posts, we will refrain from having source level help, that is, we will try and debug the faulty application without any source code and only symbols. 

Commands Used:

We start by attaching the debugger to he hung application, or we can also start the application inside the debugger if that is an alternative. In this example the name of the application is FaultyApp.Exe.

Once the application hangs, we start by taking a stack dump of all the threads currently running. Since this is an user land debugging scenario, the ~* command will give us the thread wise stack dump.

kd>~*kvn
0:004> ~*kvn

   0  Id: 6a0.5f8 Suspend: 1 Teb: 7efdd000 Unfrozen
 # ChildEBP RetAddr  Args to Child              
00 0016f7b8 770315f7 00000003 0016f808 00000001 ntdll!NtWaitForMultipleObjects+0x15 (FPO: [5,0,0])
01 0016f854 76e719f8 0016f808 0016f87c 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 (FPO: [Non-Fpo])
02 0016f89c 76e74200 00000003 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 (FPO: [Non-Fpo])
03 0016f8b8 00bd11b5 00000003 0016f8d4 00000000 kernel32!WaitForMultipleObjects+0x18 (FPO: [Non-Fpo])
04 0016f8e4 00bd143d 00000001 00699318 00699368 FaultyApp!main+0x95 (FPO: [Non-Fpo]) (CONV: cdecl) 
05 0016f92c 76e7338a 7efde000 0016f978 77d79f72 FaultyApp!__tmainCRTStartup+0xfe (FPO: [Non-Fpo]) (CONV: cdecl) 
06 0016f938 77d79f72 7efde000 735ac1c9 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
07 0016f978 77d79f45 00bd1500 7efde000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
08 0016f990 00000000 00bd1500 7efde000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   1  Id: 6a0.bc4 Suspend: 1 Teb: 7efda000 Unfrozen
 # ChildEBP RetAddr  Args to Child              
00 0089feb4 77d78e44 00000048 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
01 0089ff18 77d78d28 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
02 0089ff40 00bd105c 6ce392e0 00000000 00000000 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
03 0089ff54 76e7338a 00000000 0089ffa0 77d79f72 FaultyApp!FnThread1+0x2c (FPO: [Non-Fpo]) (CONV: cdecl) 
04 0089ff60 77d79f72 00000000 73c5c711 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
05 0089ffa0 77d79f45 00bd1005 00000000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
06 0089ffb8 00000000 00bd1005 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   2  Id: 6a0.12fc Suspend: 1 Teb: 7efaf000 Unfrozen
 # ChildEBP RetAddr  Args to Child              
00 00dffa50 77d78e44 00000048 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
01 00dffab4 77d78d28 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
02 00dffadc 00bd10fc 6ce392e0 00000000 00000000 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
03 00dffaf0 76e7338a 00000000 00dffb3c 77d79f72 FaultyApp!FnThread3+0x2c (FPO: [Non-Fpo]) (CONV: cdecl) 
04 00dffafc 77d79f72 00000000 7393c38d 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
05 00dffb3c 77d79f45 00bd100f 00000000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
06 00dffb54 00000000 00bd100f 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

   3  Id: 6a0.11f0 Suspend: 1 Teb: 7efd7000 Unfrozen
 # ChildEBP RetAddr  Args to Child              
00 00a6fa34 77d78e44 00000048 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15 (FPO: [3,0,0])
01 00a6fa98 77d78d28 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x13e (FPO: [Non-Fpo])
02 00a6fac0 00bd10ac 6ce392e0 00000000 00000000 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])
03 00a6fad4 76e7338a 00000000 00a6fb20 77d79f72 FaultyApp!FnThread2+0x2c (FPO: [Non-Fpo]) (CONV: cdecl) 
04 00a6fae0 77d79f72 00000000 73eac391 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
05 00a6fb20 77d79f45 00bd100a 00000000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
06 00a6fb38 00000000 00bd100a 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

#  4  Id: 6a0.11c8 Suspend: 1 Teb: 7efac000 Unfrozen
 # ChildEBP RetAddr  Args to Child              
00 00f0fd3c 77ddf926 73bcc5dd 00000000 00000000 ntdll!DbgBreakPoint (FPO: [0,0,0])
01 00f0fd6c 76e7338a 00000000 00f0fdb8 77d79f72 ntdll!DbgUiRemoteBreakin+0x3c (FPO: [Non-Fpo])
02 00f0fd78 77d79f72 00000000 73bcc509 00000000 kernel32!BaseThreadInitThunk+0xe (FPO: [Non-Fpo])
03 00f0fdb8 77d79f45 77ddf8ea 00000000 ffffffff ntdll!__RtlUserThreadStart+0x70 (FPO: [Non-Fpo])
04 00f0fdd0 00000000 77ddf8ea 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b (FPO: [Non-Fpo])

The clue here is to look for the Windows API call RtlEnterCriticalSection. There might be  more than one thread in the stack making this function call, we can use any such thread.

Looking up on MSDN we see that the API is defined as follows:

NTSTATUS RtlEnterCriticalSection
 (
  RTL_CRITICAL_SECTION* crit
 )

In the above stack dump, we have several such threads, we take any one and them from the parameters list take the address of the RTL_CRITICAL_SECTION object pointer and type cast it to see the contents.

02 0089ff40 00bd105c 6ce392e0 00000000 00000000 ntdll!RtlEnterCriticalSection+0x150 (FPO: [Non-Fpo])

0:004> dt ntdll!_RTL_CRITICAL_SECTION 6ce392e0
   +0x000 DebugInfo        : 0x0069acf8 _RTL_CRITICAL_SECTION_DEBUG
   +0x004 LockCount        : 0n-14
   +0x008 RecursionCount   : 0n1
   +0x00c OwningThread     : 0x000005f8 Void
   +0x010 LockSemaphore    : 0x00000048 Void
   +0x014 SpinCount        : 0

We see that one of the members of this object gives us the Owning thread ID. 0x000005f8.

Lets try and find out more details about this thread. In user mode, the double tilde (~~) command will do it for us.

0:004> ~~[0x000005f8 ]
   0  Id: 6a0.5f8 Suspend: 1 Teb: 7efdd000 Unfrozen
      Start: FaultyApp!mainCRTStartup (00bd1500)
      Priority: 0  Priority class: 32  Affinity: ff

Ok so the thread offending is the main thread. This is good. Now we know where in the code of FaultyApp we need to start digging.

Lets start with disassembling the Main function. I will add the code in small sections explaining the relevant portions in between.

0:004> uF FaultyApp!main
FaultyApp!main:
00bd1120 55              push    ebp
00bd1121 8bec           mov    ebp,esp 

This is standard function prologue code, that we see in the above two lines.


00bd1123 83ec14        sub    esp,14h                                           

The above line is allocating space for local variables in the stack 

00bd1126 a11070bf00  mov   eax,dword ptr [FaultyApp!__security_cookie (00bf7010)]
00bd112b 33c5            xor     eax,ebp                                           


These two lines above is the code for stack guard.


00bd112d 8945fc          mov     dword ptr [ebp-4],eax
00bd1130 a100b0bf00   mov     eax,dword ptr [FaultyApp!_imp__g_CritSec (00bfb000)]
00bd1135 50                 push    eax
00bd1136 ff1534b0bf00 call    dword ptr [FaultyApp!_imp__InitializeCriticalSection (00bfb034)]    

the four lines above  is the code to initialize the critical section

00bd113c c745ec00000000  mov     dword ptr [ebp-14h],0
00bd1143 eb09                   jmp     FaultyApp!main+0x2e (00bd114e)

FaultyApp!main+0x25:
00bd1145 8b4dec          mov     ecx,dword ptr [ebp-14h]
00bd1148 83c101          add      ecx,1
00bd114b 894dec          mov     dword ptr [ebp-14h],ecx

FaultyApp!main+0x2e:
00bd114e 837dec03        cmp     dword ptr [ebp-14h],3                             

The line just above is comparing a local variable with a constant '3'

00bd1152 7333            jae     FaultyApp!main+0x67 (00bd1187)                  

If the comparison result is equal or greater than 3 then then jump out, else loop                                                    
FaultyApp!main+0x34:
00bd1154 6a00                    push    0
00bd1156 6a00                    push    0
00bd1158 6a00                    push    0
00bd115a 8b55ec                mov     edx,dword ptr [ebp-14h]
00bd115d 8b04950070bf00  mov     eax,dword ptr FaultyApp!pFnarry (00bf7000)[edx*4]   

This above sections seems to be initializing a function pointer array for thread functions.  We know this because of the call to CreateThread API a few lines below....
                                
00bd1164 50              push    eax
00bd1165 6a00            push    0
00bd1167 6a00            push    0
00bd1169 ff154cb0bf00    call    dword ptr [FaultyApp!_imp__CreateThread (00bfb04c)]

(the reader is encouraged to use MSDN to see the signature for CreateThread and all the parameters pushed into it inside this function to decrypt what all was passed)

00bd116f 8b4dec          mov     ecx,dword ptr [ebp-14h]
00bd1172 89448df0       mov     dword ptr [ebp+ecx*4-10h],eax
00bd1176 8b55ec          mov     edx,dword ptr [ebp-14h]
00bd1179 837c95f000    cmp     dword ptr [ebp+edx*4-10h],0
00bd117e 7505              jne     FaultyApp!main+0x65 (00bd1185)

FaultyApp!main+0x60:
00bd1180 83c8ff          or      eax,0FFFFFFFFh
00bd1183 eb32            jmp     FaultyApp!main+0x97 (00bd11b7)

FaultyApp!main+0x65:
00bd1185 ebbe            jmp     FaultyApp!main+0x25 (00bd1145)

FaultyApp!main+0x67:
00bd1187 a1b892bf00      mov     eax,dword ptr [FaultyApp!hStartThreads (00bf92b8)]
00bd118c 50                    push    eax
00bd118d ff1540b0bf00   call    dword ptr [FaultyApp!_imp__SetEvent (00bfb040)]
00bd1193 6a00                push    0
00bd1195 68f4010000      push    1F4h
00bd119a ff1544b0bf00   call    dword ptr [FaultyApp!_imp__SleepEx (00bfb044)]
00bd11a0 e885010000     call    FaultyApp!DllFunction (00bd132a)                      

After setting an event and sleeping for a while, it finally jumps to this function in the above line. DllFunction. We need to disassemble it to see where it leads us. I am not pasting the rest of the disassembly of the Main function as it is irrelevant to the rest of the context.

Lets start with the disassembly of DllFunction.


0:004> u FaultyApp!DllFunction
FaultyApp!DllFunction:
00bd132a ff2504b0bf00    jmp     dword ptr [FaultyApp!_imp__DllFunction (00bfb004)]
FaultyApp!__security_check_cookie:
00bd1330 3b0d1070bf00    cmp     ecx,dword ptr [FaultyApp!__security_cookie (00bf7010)]
00bd1336 7502                  jne     FaultyApp!__security_check_cookie+0xa (00bd133a)
00bd1338 f3c3                  rep ret
00bd133a e908020000       jmp     FaultyApp!__report_gsfailure (00bd1547)
FaultyApp!__tmainCRTStartup:
00bd133f 6a14                  push    14h
00bd1341 68d05dbf00       push    offset FaultyApp!__rtc_tzz+0x104 (00bf5dd0)
00bd1346 e8551a0000       call    FaultyApp!__SEH_prolog4 (00bd2da0)

Whats going on? This doesn't look like a function at all....so look closely at the first line copied again in the line below for your convenience...

00bd132a ff2504b0bf00    jmp     dword ptr [FaultyApp!_imp__DllFunction (00bfb004)]

This is actually a function pointer which is being de-referenced. So if it is a pointer, we need to see what it is pointing at:

0:004> dd 00bfb004
00bfb004  6ce11005 00000000 00000000 00000000
00bfb014  00000000 00000000 00000000 00000000
00bfb024  00000000 00000000 00000000 00000000
00bfb034  77d72c8a 77d622c0 77d62280 76e71691
00bfb044  76e71215 76e7324c 76e73495 76e741e8
00bfb054  76e75169 76e74a25 76e751fd 76e711c0
00bfb064  76e711a9 76e717b8 76e71420 77d8107b
00bfb074  77d79dd5 76e779d8 76e74a37 76e71222

Lets have a look at what is actually at this address.

0:004> u 6ce11005
FaultyDLL!ILT+0(_DllFunction):
6ce11005 e916000000      jmp     FaultyDLL!DllFunction (6ce11020)
FaultyDLL!ILT+5(_DllMain:
6ce1100a e931000000      jmp     FaultyDLL!DllMain (6ce11040)
6ce1100f cc              int     3
6ce11010 cc              int     3
6ce11011 cc              int     3
6ce11012 cc              int     3
6ce11013 cc              int     3
6ce11014 cc              int     3

It shows the name of a different module. FaultyDll. Looks like it is a DLL function being used, through the import address table (I know the name of the module has the word DLL in it, but that is not why i reached this conclusion).

Now lets disassemble the actual function the address of which we have been able to decipher.

0:004> u 6ce11020
FaultyDLL!DllFunction:
6ce11020 55                    push    ebp
6ce11021 8bec                mov     ebp,esp
6ce11023 68e092e36c      push    offset FaultyDLL!g_CritSec (6ce392e0)
6ce11028 ff1500b0e36c   call    dword ptr [FaultyDLL!_imp__EnterCriticalSection (6ce3b000)]
6ce1102e b801000000      mov     eax,1
6ce11033 5d                    pop     ebp
6ce11034 c3                    ret
6ce11035 cc                    int     3

It is a small function and what do we see in it? It enter's the critical section, but never leaves it. This is the bug!!

Main is shown as the owner of the critical section because it calls a faulty function which enters but never leaves the critical section.

Note: This is a very simple test application tailor made to produce the hang, real life scenarios might be much harder to debug and locate. But I do hope that this gives a starting point to folks wanting to chase such bugs!


Trivia : !cs command also gives info about all critical sections. !cs <address of crit> will dump info about that particular critical section. I deliberately avoided using it, as this exercise helps one understand such debugging scenarios in depth, rather than being dependent on a debugger extension.

Wednesday 22 October 2014

WinDbg : Debugging A Stack Corruption Scenario

WinDbg : Debugging A Stack Corruption Scenario


The call stack is one of the most important steps to trace a fault in a program. It tells us the sequence of function calls which led to the fault. However, there are times when the call stack itself is corrupted. This makes the debuggers job very difficult, in tracing the fault. Sometimes, it is still possible to reconstruct the stack and hence trace the flow of the program. In this post, we sill try one such instance of tracking the fault by reconstructing the stack. 

Commands used:
  1. kb
  2. dds  (this command is used in this article here). For more details of the d command, please read this.
  3. uF


At the point of the crash the debugger complains:

(99c.1d48): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000001 ebx=00000000 ecx=00000005 edx=0000002f esi=00000000 edi=00000000
eip=00000100 esp=00b8fd2c ebp=00b8fd4c iopl=0         nv up ei pl nz na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010206
00000100 ??              ???

This gives no hint of a corrupt stack, it rather says Access Violation. The values of the registers is where we get the first clue from. The instruction pointer EIP is set to 0x100. That doesn't seem quite right does it? lets try to see the call stack:

0:002> kb
ChildEBP RetAddr  Args to Child              
WARNING: Frame IP not in any known module. Following frames may be wrong.
00b8fd28 00000000 00000005 0000002f 00000030 0x100

This shows the stack is corrupt. Now we have no clue what sequence of instruction executions lead to this stage. To root cause this problem, we need to start with reconstructing the call stack. the question is, where do we start.

call stack is per thread. A good place to begin would be the thread block. lets see what the TEB for this particular thread can tell us. We will use the !teb debugger extension here.

0:002> !teb
TEB at 7efd7000
    ExceptionList:        00b8fdf0
    StackBase:            00b90000
    StackLimit:           00b8e000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7efd7000
    EnvironmentPointer:   00000000
    ClientId:             0000099c . 00001d48
    RpcHandle:            00000000
    Tls Storage:          7efd702c
    PEB Address:          7efde000
    LastErrorValue:       0
    LastStatusValue:      0
    Count Owned Locks:    0
    HardErrorMode:        0

The TEB tells us where the stack starts and how big it can grow. On x86 we know that the stack grows from high to low (downwards), thus we have to start at the StackLimit value and go till the StackBase. We will try to see if any of the values in these memory location make any sense to us. A word of caution here : the discarded stack frames are not cleaned up by any one, thus not all legitimate looking values are actually trustable. That is where this might get tricky.

We will use dds, so that the output is dword aligned as well as is any of the dwords map to a symbol which can be resolved then it will.

The output will be long and tedious to go through and requires patience.

0:002> dds b8e000 b90000
00b8e000  00000000
00b8e004  00000000
<Output snipped to save space>
00b8ef48  00000000
00b8ef4c  76ba0000 kernel32!_imp__DebugBreak <PERF> (kernel32+0x0)
00b8ef50  00000000
00b8ef54  00b8ef8c
00b8ef58  00000000
00b8ef5c  00000000
00b8ef60  00002045
00b8ef64  00000000
00b8ef68  00000000
00b8ef6c  00000000
00b8ef70  00000000
00b8ef74  0000000f
00b8ef78  00000000
00b8ef7c  00002045
00b8ef80  00b8ef94
00b8ef84  76bb3418 kernel32!GetConsoleLangId+0x15
00b8ef88  76bb33f4 kernel32!SetTEBLangID+0xf
00b8ef8c  00b8ef90
00b8ef90  00b8efac
00b8ef94  775aec62 ntdll!bsearch+0x70
00b8ef98  00b8efd4
00b8ef9c  00b8efb0
00b8efa0  775aeba1 ntdll!ARRAY_FITS+0x2d
00b8efa4  000007a0
00b8efa8  0000174c
00b8efac  00b8efbc
00b8efb0  00b8efec
00b8efb4  775ae971 ntdll!RtlpLocateActivationContextSection+0x1d9
00b8efb8  0000174c
00b8efbc  00001eec
00b8efc0  000007a0
00b8efc4  0006174c
00b8efc8  00b8f070
00b8efcc  00000000
00b8efd0  536cd652
00b8efd4  00b8f024
00b8efd8  775aedff ntdll!RtlpFindUnicodeStringInSection+0x71
00b8efdc  00b8f104
00b8efe0  74957b88*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\SysWOW64\SYSFER.DLL - 
 SYSFER!FirstHookFunc+0x21038
00b8efe4  00000001
00b8efe8  00b8f074
00b8efec  0006174c
00b8eff0  c0150008
00b8eff4  00000000
00b8eff8  000007a0
00b8effc  00000002
00b8f000  00b8f078
00b8f004  c0150008
00b8f008  00000000
00b8f00c  7efde000
00b8f010  7efd7000
00b8f014  00b8f034
00b8f018  775aef7b ntdll!RtlFindNextActivationContextSection+0x46
00b8f01c  00b8f058
00b8f020  00b8f07c
00b8f024  00b8f080
00b8f028  00b8f030
00b8f02c  0006174c
00b8f030  00000000
00b8f034  00b8f088
00b8f038  775aed75 ntdll!RtlFindActivationContextSectionString+0xe1
00b8f03c  00b8f058
00b8f040  00b8f07c
00b8f044  00b8f080
00b8f048  00b8f078
00b8f04c  00b8f178
00b8f050  00b8f1a4
00b8f054  00b8f194
00b8f058  00000018
00b8f05c  00000003
00b8f060  00000000
00b8f064  00000002
00b8f068  00000003
00b8f06c  00000002
00b8f070  00000001
00b8f074  536cd652
00b8f078  00000000
00b8f07c  0006174c
00b8f080  000007a0
00b8f084  00b8f094
00b8f088  00b8f144
00b8f08c  775af3df ntdll!AitFireParentUsageEvent+0x772
00b8f090  00000003
00b8f094  00000000
00b8f098  00000002
00b8f09c  00b8f104
00b8f0a0  00b8f0ac
00b8f0a4  775afaa4 ntdll!LdrpDefaultExtension
00b8f0a8  775af442 ntdll!AitFireParentUsageEvent+0x88d
00b8f0ac  00000040
00b8f0b0  00000000
<Output snipped to save space>
00b8f100  00000000
00b8f104  001a0018
00b8f108  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f10c  00000000
00b8f110  00b8f178
00b8f114  00b8f144
00b8f118  00000000
00b8f11c  00020000
00b8f120  00b8f13c
00b8f124  00b8f13c
00b8f128  00b8f13c
00b8f12c  00000002
00b8f130  00000002
00b8f134  00000000
00b8f138  00000005
00b8f13c  00000000
00b8f140  775afaa4 ntdll!LdrpDefaultExtension
00b8f144  775afaa4 ntdll!LdrpDefaultExtension
00b8f148  00b8f2a0
00b8f14c  775af201 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x31e
00b8f150  00b8f17c
00b8f154  00b8f330
00b8f158  00b8f4f4
00b8f15c  775af26c ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3bc
00b8f160  00000000
00b8f164  00000000
00b8f168  00800000
00b8f16c  00b8f1fc
00b8f170  00000000
00b8f174  775afaa4 ntdll!LdrpDefaultExtension
00b8f178  00000000
00b8f17c  00000000
00b8f180  00000000
00b8f184  00000000
00b8f188  00000000
00b8f18c  00000000
00b8f190  00000000
00b8f194  001a0018
00b8f198  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f19c  00b8f338
00b8f1a0  00000000
<Output snipped to save space>
00b8f1d4  00000000
00b8f1d8  00200000
00b8f1dc  00b8f27c
00b8f1e0  00b8f27c
00b8f1e4  00b8f27c
00b8f1e8  00000020
00b8f1ec  00000020
00b8f1f0  00000000
<Output snipped to save space>
00b8f24c  00000000
00b8f250  00b8f268
00b8f254  004835f0
00b8f258  0048362c
00b8f25c  776848a0 ntdll!LdrpHashTable+0x80
00b8f260  00b8f280
00b8f264  775afa39 ntdll!LdrpFindLoadedDllByName+0x68
00b8f268  74957b88 SYSFER!FirstHookFunc+0x21038
00b8f26c  0048361c
00b8f270  8bb2ba48
00b8f274  00b8f338
00b8f278  776801a0 ntdll!LdrpDefaultPath
00b8f27c  00000000
00b8f280  00b8f2ec
00b8f284  775afeae ntdll!LdrpFindLoadedDll+0xd0
00b8f288  00b8f338
00b8f28c  00000000
00b8f290  00b8f340
00b8f294  775afaa4 ntdll!LdrpDefaultExtension
00b8f298  0000002e
00b8f29c  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f2a0  00b8f2e0
00b8f2a4  775afb16 ntdll!LdrpApplyFileNameRedirection+0x128
00b8f2a8  00000001
00b8f2ac  00b8f4f4
00b8f2b0  775afaa4 ntdll!LdrpDefaultExtension
00b8f2b4  00b8f338
00b8f2b8  00b8f328
00b8f2bc  00b8f344
00b8f2c0  00000000
00b8f2c4  00000000
00b8f2c8  00000000
00b8f2cc  00b8f330
00b8f2d0  00b8f4f4
00b8f2d4  775afaa4 ntdll!LdrpDefaultExtension
00b8f2d8  00000018
00b8f2dc  00000018
00b8f2e0  00b8f468
00b8f2e4  0000002e
00b8f2e8  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f2ec  00b8f468
00b8f2f0  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f2f4  00b8f468
00b8f2f8  775b00b3 ntdll!LdrGetDllHandleEx+0x3cc
00b8f2fc  775ae1b2 ntdll!_SEH_epilog4_GS+0xa
00b8f300  775b0098 ntdll!LdrGetDllHandleEx+0x451
00b8f304  3150d44e
00b8f308  00000001
00b8f30c  775afd37 ntdll!LdrGetDllHandle
00b8f310  00000000
00b8f314  00000000
00b8f318  00000000
00b8f31c  00b8f4bc
00b8f320  00000000
00b8f324  00000000
00b8f328  00000000
00b8f32c  00000000
00b8f330  00000001
00b8f334  776801a0 ntdll!LdrpDefaultPath
00b8f338  001a0018
00b8f33c  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f340  004835f0
00b8f344  00b8f338
00b8f348  01000000
00b8f34c  00000000
<Output snipped to save space>
00b8f444  00000000
00b8f448  00000000
00b8f44c  3150d44e
00b8f450  00b8f304
00b8f454  00000000
00b8f458  00b8f4c8
00b8f45c  775f71f5 ntdll!_except_handler4
00b8f460  46b29a86
00b8f464  fffffffe
00b8f468  775b0098 ntdll!LdrGetDllHandleEx+0x451
00b8f46c  775afd4f ntdll!LdrGetDllHandle+0x18
00b8f470  00000001
00b8f474  00000001
00b8f478  00000000
00b8f47c  00b8f4f4
00b8f480  00b8f4bc
00b8f484  00b8f4d8
00b8f488  77141a43 KERNELBASE!GetModuleHandleForUnicodeString+0x22
00b8f48c  00000001
00b8f490  00000000
00b8f494  00b8f4f4
00b8f498  77141ace KERNELBASE!GetModuleHandleForUnicodeString+0xad
00b8f49c  c6fafa63
00b8f4a0  00000000
00b8f4a4  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f4a8  00000002
00b8f4ac  00000000
00b8f4b0  00000000
00b8f4b4  00000000
00b8f4b8  00000000
00b8f4bc  76ba0000 kernel32!_imp__DebugBreak <PERF> (kernel32+0x0)
00b8f4c0  00b8f49c
00b8f4c4  00000000
00b8f4c8  00b8f940
00b8f4cc  77167000 KERNELBASE!_except_handler4
00b8f4d0  b15487b3
00b8f4d4  fffffffe
00b8f4d8  77141ace KERNELBASE!GetModuleHandleForUnicodeString+0xad
00b8f4dc  771673ae KERNELBASE!_SEH_epilog4_GS+0xa
00b8f4e0  77141d09 KERNELBASE!BasepGetModuleHandleExW+0x233
00b8f4e4  c6faf7eb
00b8f4e8  00000000
00b8f4ec  00440bb0
00b8f4f0  74910000 SYSFER
00b8f4f4  001a0018
00b8f4f8  74957b70 SYSFER!FirstHookFunc+0x21020
00b8f4fc  00b8f970
00b8f500  00000000
00b8f504  00000000
00b8f508  00000000
00b8f50c  02080000
00b8f510  00b8f72c
00b8f514  02080000
00b8f518  00b8f524
00b8f51c  00000000
00b8f520  76ba0000 kernel32!_imp__DebugBreak <PERF> (kernel32+0x0)
00b8f524  00000000
<Output snipped to save space>
00b8f5c4  00000000
00b8f5c8  c6fafadb
00b8f5cc  00b8fa20
00b8f5d0  76bb3342 kernel32!_BaseDllInitialize+0x92
00b8f5d4  00000002
00b8f5d8  00000000
00b8f5dc  00b8fab0
00b8f5e0  00b8fa34
00b8f5e4  76bb3357 kernel32!_BaseDllInitialize+0x2cf
00b8f5e8  00000000
00b8f5ec  00000000
00b8f5f0  00481780
00b8f5f4  00000000
00b8f5f8  76ba0000 kernel32!_imp__DebugBreak <PERF> (kernel32+0x0)
00b8f5fc  00000000
00b8f600  00000000
00b8f604  00000000
00b8f608  0048b340
00b8f60c  00000000
00b8f610  00000005
00b8f614  00b8f9d4
00b8f618  74938973 SYSFER!FirstHookFunc+0x1e23
00b8f61c  00000003
00b8f620  00000005
00b8f624  ec3e8105
00b8f628  00000000
00b8f62c  00000002
00b8f630  74938a4c SYSFER!FirstHookFunc+0x1efc
00b8f634  00000000
00b8f638  00000000
00b8f63c  0000007f
00b8f640  00b8f72c
00b8f644  775b3c5a ntdll!RtlpAllocateHeap+0xb5d
00b8f648  0048b340
00b8f64c  775b2caf ntdll!RtlpAllocateHeap+0xc78
00b8f650  0048b128
00b8f654  00000000
00b8f658  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8f65c  3150d70a
00b8f660  00000208
00b8f664  00000000
00b8f668  00480000
00b8f66c  00480150
00b8f670  00440150
00b8f674  00000000
00b8f678  00480150
00b8f67c  00000eb0
00b8f680  00480180
00b8f684  00440180
00b8f688  00000194
00b8f68c  00000004
00b8f690  00480150
00b8f694  0000007f
00b8f698  0048b128
00b8f69c  00590ad8
00b8f6a0  0000007f
00b8f6a4  0000007f
00b8f6a8  00000044
00b8f6ac  00000046
00b8f6b0  00000000
00b8f6b4  00000000
00b8f6b8  0048b338
00b8f6bc  00000000
00b8f6c0  06040002
00b8f6c4  00484338
00b8f6c8  00580048
00b8f6cc  00000000
00b8f6d0  00000000
00b8f6d4  00000208
00b8f6d8  0000007f
00b8f6dc  0000007f
00b8f6e0  dd0401d8
00b8f6e4  00000eb0
00b8f6e8  00000000
00b8f6ec  00000000
00b8f6f0  502c016b
00b8f6f4  00000018
00b8f6f8  0048b128
00b8f6fc  0048b128
00b8f700  0048b123
00b8f704  00000000
00b8f708  00000000
00b8f70c  00480000
00b8f710  03440000
00b8f714  00b8f65c
00b8f718  00b8f660
00b8f71c  00b8f7ec
00b8f720  775f71f5 ntdll!_except_handler4
00b8f724  46b2e8b6
00b8f728  fffffffe
00b8f72c  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8f730  775b3cfe ntdll!RtlAllocateHeap+0x23a
00b8f734  00000208
00b8f738  00000220
00b8f73c  0048b122
00b8f740  0048b120
00b8f744  00000208
00b8f748  00480000
00b8f74c  00000001
00b8f750  00000001
<Output snipped to save space>
00b8f780  00000000
00b8f784  00b8f7b4
00b8f788  77650806 ntdll!RtlpValidateHeap+0x20
00b8f78c  00480000
00b8f790  00000000
00b8f794  00000208
00b8f798  00480000
00b8f79c  00000001
00b8f7a0  00000004
00b8f7a4  00000000
00b8f7a8  0048b120
00b8f7ac  00000000
00b8f7b0  00480000
00b8f7b4  00b8f7fc
00b8f7b8  7765108a ntdll!RtlDebugAllocateHeap+0x31f
00b8f7bc  00480138
00b8f7c0  7765106e ntdll!RtlDebugAllocateHeap+0x308
00b8f7c4  3150d7da
00b8f7c8  00000000
00b8f7cc  00000000
00b8f7d0  00480000
00b8f7d4  00440000
00b8f7d8  00000000
00b8f7dc  0048b128
00b8f7e0  01440bb0
00b8f7e4  00b8f7c4
00b8f7e8  00b8f7c8
00b8f7ec  00b8f8d0
00b8f7f0  775f71f5 ntdll!_except_handler4
00b8f7f4  46b2f046
00b8f7f8  fffffffe
00b8f7fc  7765106e ntdll!RtlDebugAllocateHeap+0x308
00b8f800  7760af0d ntdll!RtlpAllocateHeap+0xc4
00b8f804  00480000
00b8f808  502c016b
00b8f80c  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8f810  3150d8c6
00b8f814  7efd7000
00b8f818  00000000
00b8f81c  00480000
00b8f820  00440000
00b8f824  00000000
<Output snipped to save space>
00b8f8a0  00000000
00b8f8a4  402c006a
00b8f8a8  4000006a
00b8f8ac  00000000
00b8f8b0  00000000
00b8f8b4  00000000
00b8f8b8  00000000
00b8f8bc  00000000
00b8f8c0  00480000
00b8f8c4  00440000
00b8f8c8  00b8f810
00b8f8cc  00b8f814
00b8f8d0  00b8f9c4
00b8f8d4  775f71f5 ntdll!_except_handler4
00b8f8d8  46b2e8b6
00b8f8dc  fffffffe
00b8f8e0  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8f8e4  775b3cfe ntdll!RtlAllocateHeap+0x23a
00b8f8e8  00000208
00b8f8ec  00000000
00b8f8f0  00000000
00b8f8f4  00b8f954
00b8f8f8  7efd7000
00b8f8fc  7efd7000
00b8f900  7efde000
00b8f904  74910000 SYSFER
00b8f908  00000000
00b8f90c  00000000
00b8f910  00000000
00b8f914  00000000
00b8f918  00000000
00b8f91c  00000000
00b8f920  7768200c ntdll!RtlpFlsLock
00b8f924  0048b128
00b8f928  00000000
00b8f92c  00000000
00b8f930  00000000
00b8f934  c6faf7eb
00b8f938  00b8f4e4
00b8f93c  00000000
00b8f940  00b8f98c
00b8f944  00000001
00b8f948  74962da0 SYSFER!FirstHookFunc+0x2c250
00b8f94c  00000001
00b8f950  74910000 SYSFER
00b8f954  00440bb0
00b8f958  00b8f964
00b8f95c  74930cb2 SYSFER+0x20cb2
00b8f960  74964c20 SYSFER!org_first_hook_func_addr+0x13fc
00b8f964  00b8f99c
00b8f968  7492d466 SYSFER+0x1d466
00b8f96c  0000000c
00b8f970  7492d44f SYSFER+0x1d44f
00b8f974  ec3e814d
00b8f978  00000000
00b8f97c  00440bb0
00b8f980  74910000 SYSFER
00b8f984  00b8f974
00b8f988  00b8f99c
00b8f98c  00b8f9c4
00b8f990  7492da00 SYSFER+0x1da00
00b8f994  9813bff9
00b8f998  fffffffe
00b8f99c  7492d44f SYSFER+0x1d44f
00b8f9a0  749290bb SYSFER+0x190bb
00b8f9a4  00440bb0
00b8f9a8  749290eb SYSFER+0x190eb
00b8f9ac  ec3e8105
00b8f9b0  00000000
00b8f9b4  00000002
00b8f9b8  74910000 SYSFER
00b8f9bc  00b8f9ac
00b8f9c0  ec3e8105
00b8f9c4  00b8f624
00b8f9c8  00b8fa04
00b8f9cc  74953853 SYSFER!FirstHookFunc+0x1cd03
00b8f9d0  ffffffff
00b8f9d4  00b8fa14
00b8f9d8  7492915a SYSFER+0x1915a
00b8f9dc  74910000 SYSFER
00b8f9e0  00000002
00b8f9e4  00000001
00b8f9e8  00b8fab0
00b8f9ec  00484110
00b8f9f0  00b8fa34
00b8f9f4  00000001
00b8f9f8  00000001
00b8f9fc  0048b718
00b8fa00  7760a7a9 ntdll!RtlpCreateSplitBlock+0x4fc
00b8fa04  0048b728
00b8fa08  000008b8
00b8fa0c  feeefeee
00b8fa10  0000007b
00b8fa14  0048b340
00b8fa18  00480000
00b8fa1c  0048bfe0
00b8fa20  00b8fa40
00b8fa24  775b99a0 ntdll!LdrpCallInitRoutine+0x14
00b8fa28  74910000 SYSFER
00b8fa2c  00000000
00b8fa30  0000007f
00b8fa34  00b8fb20
00b8fa38  775b3c5a ntdll!RtlpAllocateHeap+0xb5d
00b8fa3c  0048b718
00b8fa40  775b2caf ntdll!RtlpAllocateHeap+0xc78
00b8fa44  0048b348
00b8fa48  00000000
00b8fa4c  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8fa50  3150db06
00b8fa54  000003bc
00b8fa58  00000000
00b8fa5c  00480000
00b8fa60  00480150
00b8fa64  7768206c ntdll!LdrpProcessInitialized
00b8fa68  00000000
00b8fa6c  00480150
00b8fa70  00000c90
00b8fa74  00480180
00b8fa78  00000000
00b8fa7c  00000119
00b8fa80  00000004
00b8fa84  00480150
00b8fa88  0000007f
00b8fa8c  0048b348
00b8fa90  00000001
00b8fa94  0000007f
00b8fa98  0048469c
00b8fa9c  0000007b
00b8faa0  ffffffff
00b8faa4  ffb7b963
00b8faa8  775ba1c0 ntdll!LdrpInitializeThread+0x131
00b8faac  0048b710
00b8fab0  00000000
00b8fab4  06040002
00b8fab8  00484338
00b8fabc  7768206c ntdll!LdrpProcessInitialized
00b8fac0  0000017e
00b8fac4  77680214 ntdll!PebLdr+0x14
00b8fac8  000003bc
00b8facc  0000007f
00b8fad0  00b8fb1c
00b8fad4  91040194
00b8fad8  00000c90
00b8fadc  00000000
00b8fae0  00b8fb2c
00b8fae4  5000016b
00b8fae8  0000001c
00b8faec  0048b348
00b8faf0  0048b348
00b8faf4  0048b343
00b8faf8  00000000
00b8fafc  00b8fb34
00b8fb00  00480000
00b8fb04  03000000
00b8fb08  00b8fa50
00b8fb0c  00000000
00b8fb10  00b8fbe0
00b8fb14  775f71f5 ntdll!_except_handler4
00b8fb18  46b2e8b6
00b8fb1c  fffffffe
00b8fb20  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8fb24  775b3cfe ntdll!RtlAllocateHeap+0x23a
00b8fb28  000003bc
00b8fb2c  000003d8
00b8fb30  0048b342
00b8fb34  0048b340
00b8fb38  000003bc
00b8fb3c  00480000
00b8fb40  00000001
00b8fb44  00b8fb50
00b8fb48  77580000 ntdll!`string' <PERF> (ntdll+0x0)
00b8fb4c  00000000
00b8fb50  0001002f
00b8fb54  00000000
00b8fb58  00000000
00b8fb5c  00000000
00b8fb60  00000000
00b8fb64  00000000
00b8fb68  00000000
00b8fb6c  0000027f
00b8fb70  00000000
00b8fb74  0000ffff
00b8fb78  00b8fba8
00b8fb7c  77650806 ntdll!RtlpValidateHeap+0x20
00b8fb80  00480000
00b8fb84  00000000
00b8fb88  000003bc
00b8fb8c  00480000
00b8fb90  00000001
00b8fb94  00000004
00b8fb98  00000000
00b8fb9c  0048b340
00b8fba0  00000000
00b8fba4  00480000
00b8fba8  00b8fbf0
00b8fbac  7765108a ntdll!RtlDebugAllocateHeap+0x31f
00b8fbb0  00480138
00b8fbb4  7765106e ntdll!RtlDebugAllocateHeap+0x308
00b8fbb8  3150dbd6
00b8fbbc  00000000
00b8fbc0  00000000
00b8fbc4  00480000
00b8fbc8  00000000
00b8fbcc  00000000
00b8fbd0  0048b348
00b8fbd4  01000000
00b8fbd8  00b8fbb8
00b8fbdc  0000002b
00b8fbe0  00b8fcc4
00b8fbe4  775f71f5 ntdll!_except_handler4
00b8fbe8  46b2f046
00b8fbec  fffffffe
00b8fbf0  7765106e ntdll!RtlDebugAllocateHeap+0x308
00b8fbf4  7760af0d ntdll!RtlpAllocateHeap+0xc4
00b8fbf8  00480000
00b8fbfc  5000016b
00b8fc00  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8fc04  3150dcf2
00b8fc08  00000000
00b8fc0c  00000000
00b8fc10  00480000
00b8fc14  00b8fe1c
00b8fc18  0000002b
00b8fc1c  0000027f
00b8fc20  00000000
00b8fc24  00000000
00b8fc28  00000000
00b8fc2c  00000000
00b8fc30  00000000
00b8fc34  00001f80
00b8fc38  0000ffff
00b8fc3c  00000000
<Output snipped to save space>
00b8fc94  00000000
00b8fc98  4000006a
00b8fc9c  00000000
00b8fca0  00000000
00b8fca4  00000000
00b8fca8  00000000
00b8fcac  00000000
00b8fcb0  00000000
00b8fcb4  00480000
00b8fcb8  00000000
00b8fcbc  00b8fc04
00b8fcc0  00000000
00b8fcc4  00b8fdf0
00b8fcc8  775f71f5 ntdll!_except_handler4
00b8fccc  46b2e8b6
00b8fcd0  fffffffe
00b8fcd4  775b3cd3 ntdll!RtlpAllocateHeap+0xe73
00b8fcd8  775b3cfe ntdll!RtlAllocateHeap+0x23a
00b8fcdc  000003bc
00b8fce0  00000000
00b8fce4  00000000
00b8fce8  00b8fd48
00b8fcec  00000000
00b8fcf0  000003bc
00b8fcf4  00000000
00b8fcf8  00000000
00b8fcfc  00000000
00b8fd00  00b8fd0c
00b8fd04  013a3426*** WARNING: Unable to verify checksum for SimpleApp.exe
 SimpleApp!__crtFlsGetValue+0x15 [f:\dd\vctools\crt\crtw32\misc\winapisupp.c @ 428]
00b8fd08  00000003
00b8fd0c  00b8fd4c
00b8fd10  013a169b SimpleApp!_getptd_noexit+0x6a [f:\dd\vctools\crt\crtw32\startup\tidtable.c @ 312]
00b8fd14  00000000
00b8fd18  00000000
00b8fd1c  00000000
00b8fd20  013a161f SimpleApp!_getptd+0x6 [f:\dd\vctools\crt\crtw32\startup\tidtable.c @ 337]
00b8fd24  00b8fd4c
00b8fd28  00000100
00b8fd2c  0000002f
00b8fd30  00000005
00b8fd34  0000002f
00b8fd38  00000030
00b8fd3c  00000031
00b8fd40  00000032
00b8fd44  00000033
00b8fd48  aab7d537
00b8fd4c  00b8fd88
00b8fd50  013a112e SimpleApp!FunctionThree+0x4e 
00b8fd54  0000002f
00b8fd58  0000000a
00b8fd5c  0000002b
00b8fd60  0000002c
00b8fd64  0000002d
00b8fd68  0000002e
00b8fd6c  0000002f
00b8fd70  00000030
00b8fd74  00000031
00b8fd78  00000032
00b8fd7c  00000033
00b8fd80  00000034
00b8fd84  aab7d5f3
00b8fd88  00b8fda8
00b8fd8c  013a119e SimpleApp!FunctionTwo+0x4e 
00b8fd90  0000002b
00b8fd94  00000003
00b8fd98  00000029
00b8fd9c  0000002a
00b8fda0  0000002b
00b8fda4  aab7d5d3
00b8fda8  00b8fdb4
00b8fdac  013a11dc SimpleApp!FunctionOne+0x1c 
00b8fdb0  00000029
00b8fdb4  00b8fdc0
00b8fdb8  76bb338a kernel32!BaseThreadInitThunk+0xe
00b8fdbc  00000000
00b8fdc0  00b8fe00
00b8fdc4  775b9f72 ntdll!__RtlUserThreadStart+0x70
00b8fdc8  00000000
00b8fdcc  3150de26
00b8fdd0  00000000
00b8fdd4  00000000
00b8fdd8  00000000
00b8fddc  00000000
00b8fde0  00000000
00b8fde4  00000000
00b8fde8  00b8fdcc
00b8fdec  00000000
00b8fdf0  00b8fe10
00b8fdf4  775f71f5 ntdll!_except_handler4
00b8fdf8  46b2e576
00b8fdfc  00000000
00b8fe00  00b8fe18
00b8fe04  775b9f45 ntdll!_RtlUserThreadStart+0x1b
00b8fe08  013a1019 SimpleApp!ILT+20(_FunctionOne)
00b8fe0c  00000000
00b8fe10  ffffffff
00b8fe14  77647428 ntdll!FinalExceptionHandler
00b8fe18  00000000
00b8fe1c  00000000
00b8fe20  013a1019 SimpleApp!ILT+20(_FunctionOne)
00b8fe24  00000000
<Output snipped to save space>
00b8fffc  00000000
00b90000  ????????


Since the stack is not cleaned up after usage, we need to find a legitimate starting point inside this memory dump where we can begin. We know that threads usually start with the function RtlUserThreadStart, so we can search for this function's address in the stack. In the above example, RtlUserThreadStart() is highlighted with this color for your help.

So we have located a legitimate point where we can start looking. Now we start walking the frames to see till which point things make sense and where we go awry.

Note: Function pointers in the stack might mean return addresses, though this is not always true but will be, most of the time. from the above output, it appears that things were going well till FunctionThree was called. Things seems to have gone bad somewhere inside FunctionThree. Thus FunctionThree might be a good place to start further digging. Lets start by disassembling the function.

0:002> uF SimpleApp!FunctionThree
SimpleApp!FunctionThree:
013a10e0 55              push    ebp
013a10e1 8bec            mov     ebp,esp
013a10e3 83ec30          sub     esp,30h
013a10e6 a100703c01      mov     eax,dword ptr [SimpleApp!__security_cookie (013c7000)]
013a10eb 33c5            xor     eax,ebp
013a10ed 8945fc          mov     dword ptr [ebp-4],eax
013a10f0 c745d000000000  mov     dword ptr [ebp-30h],0
013a10f7 eb09            jmp     SimpleApp!FunctionThree+0x22 (013a1102)

SimpleApp!FunctionThree+0x19:
013a10f9 8b45d0          mov     eax,dword ptr [ebp-30h]
013a10fc 83c001          add     eax,1
013a10ff 8945d0          mov     dword ptr [ebp-30h],eax

SimpleApp!FunctionThree+0x22:
013a1102 837dd00a        cmp     dword ptr [ebp-30h],0Ah
013a1106 730f            jae     SimpleApp!FunctionThree+0x37 (013a1117)

SimpleApp!FunctionThree+0x28:
013a1108 8b4d08          mov     ecx,dword ptr [ebp+8]
013a110b 034dd0          add     ecx,dword ptr [ebp-30h]
013a110e 8b55d0          mov     edx,dword ptr [ebp-30h]
013a1111 894c95d4        mov     dword ptr [ebp+edx*4-2Ch],ecx
013a1115 ebe2            jmp     SimpleApp!FunctionThree+0x19 (013a10f9)

SimpleApp!FunctionThree+0x37:
013a1117 e875010000      call    SimpleApp!rand (013a1291)
013a111c 99                    cdq
013a111d b90a000000      mov     ecx,0Ah
013a1122 f7f9                  idiv    eax,ecx
013a1124 8b5495d4          mov     edx,dword ptr [ebp+edx*4-2Ch]
013a1128 52                     push    edx
013a1129 e8e6feffff         call    SimpleApp!ILT+15(_FunctionFour) (013a1014)
013a112e 83c404              add     esp,4
013a1131 b801000000       mov     eax,1
013a1136 8b4dfc               mov     ecx,dword ptr [ebp-4]
013a1139 33cd                  xor     ecx,ebp
013a113b e882010000       call    SimpleApp!__security_check_cookie (013a12c2)
013a1140 8be5                  mov     esp,ebp
013a1142 5d                     pop     ebp
013a1143 c3                     ret

Looks like there is some kind of a loop inside this function. Look at the following sequence of instructions:

013a111d b90a000000      mov     ecx,0Ah
013a1122 f7f9                  idiv    eax,ecx
013a1124 8b5495d4          mov     edx,dword ptr [ebp+edx*4-2Ch]
013a1128 52                     push    edx

The register ECX is being populated to 16 (0Ah), making the loop iterate 16 times. 
013a111d b90a000000      mov     ecx,0Ah

This statement below looks like a dereference of a pointer. But what else can we find from this?
013a1124 8b5495d4          mov     edx,dword ptr [ebp+edx*4-2Ch]

Assume the following C code:
int i   = 10;
int *p = &i;

here we are assigning a pointer to a variable. But since it is a normal variable and not a structure, the pointer can be directly pointed to the location. But in the case of a structure (or class) we would need to move the base pointer to somewhere inside the structure to get the address of the member (very similar to how CONTAINING_RECORD macro works).

struct *pS;
pS->MemberPointer;

In the code we are debugging, it seems to be the case of the later. Hinting that it is a complex data type being used.

Also the multiplication before the dereference hints that it is probably an array whose indices are being populated.
013a1122 f7f9                  idiv    eax,ecx

We also see that FunctionFour is being called from inside the loop. So lets see what FunctionFour is doing.

0:002> uF SimpleApp!FunctionFour
SimpleApp!FunctionFour:
013a1070 55                 push    ebp
013a1071 8bec              mov     ebp,esp
013a1073 83ec1c          sub     esp,1Ch                                               // Creating stack space for locals
013a1076 a100703c01   mov     eax,dword ptr [SimpleApp!__security_cookie (013c7000)]
013a107b 33c5             xor     eax,ebp
013a107d 8945fc          mov     dword ptr [ebp-4],eax
013a1080 c745e400000000  mov     dword ptr [ebp-1Ch],0
013a1087 eb09            jmp     SimpleApp!FunctionFour+0x22 (013a1092)

SimpleApp!FunctionFour+0x19:
013a1089 8b45e4          mov     eax,dword ptr [ebp-1Ch]
013a108c 83c001          add     eax,1
013a108f 8945e4          mov     dword ptr [ebp-1Ch],eax

SimpleApp!FunctionFour+0x22:
013a1092 837de405    cmp   dword ptr [ebp-1Ch],5             // loop exit condition, loop executes 5 times
013a1096 730f            jae     SimpleApp!FunctionFour+0x37 (013a10a7)

SimpleApp!FunctionFour+0x28:
013a1098 8b4d08     mov     ecx,dword ptr [ebp+8]           // 8 is pointer size so we are incrementing by pointer size
013a109b 034de4     add     ecx,dword ptr [ebp-1Ch]
013a109e 8b55e4     mov     edx,dword ptr [ebp-1Ch]
013a10a1 894c95e8  mov     dword ptr [ebp+edx*4-18h],ecx // this looks like the operation to index into an array and dereference into the base object.
013a10a5 ebe2         jmp     SimpleApp!FunctionFour+0x19 (013a1089)   // Unconditional jump to the beginning of loop

SimpleApp!FunctionFour+0x37:
013a10a7 e8e5010000  call    SimpleApp!rand (013a1291)
013a10ac 99                cdq
013a10ad b905000000  mov   ecx,5
013a10b2 f7f9             idiv    eax,ecx
013a10b4 8b5495e8     mov   edx,dword ptr [ebp+edx*4-18h]
013a10b8 52                push  edx
013a10b9 e851ffffff     call    SimpleApp!ILT+10(_FunctionFive) (013a100f)
013a10be 83c404         add    esp,4
013a10c1 b801000000   mov   eax,1
013a10c6 8b4dfc          mov   ecx,dword ptr [ebp-4]
013a10c9 33cd             xor     ecx,ebp
013a10cb e8f2010000   call    SimpleApp!__security_check_cookie (013a12c2)
013a10d0 8be5             mov   esp,ebp
013a10d2 5d                pop    ebp
013a10d3 c3                ret

This function seems to call FunctionFive inside the loop. Lets see what FunctionFive has to say:

SimpleApp!FunctionFive:
013a1050 55                       push    ebp
013a1051 8bec                   mov     ebp,esp
013a1053 c7450400010000  mov     dword ptr [ebp+4],100h
013a105a b801000000         mov     eax,1
013a105f 5d                       pop     ebp
013a1060 c3                       ret

Whoa!!! we are putting 100h into EBP + 4. But EBP+4 is the return address. This means that this function will return to an incorrect location. Which is exactly what is happening in our program. If we recall, then code at location 100h was being executed by the CPU (EIP = 100h).

Thus we see that even when the stack is corrupt, it can sometimes be reconstructed from the hints, and then the original fault be located.