How to debug application crash/hang in production environment?

A common tough scenario is the application crash/hang in production environment. Production environment is different from development environment that it normally does not have any powerful tools to obtain information regarding application failure. In this blog entry, I will talk about 2 wonderful tools to help collect information in production environment: Windbg and Process Explorer.

 

Application Crash:

Crash is the most common scenario of application failure. It is actually caused by an unhanded exception generated in the application. There are 2 types of information that are common critical to resolve application crash:

1.       Unhandled exception type (it is called exception code in unmanaged world) and error message

2.       The full stack trace for the unhandled exception

 

These 2 types of information are trivial to obtain in development environment, because the application runs under the VS debugger, the debugger will tell us this information while unhandled exception is caught.

 

In production environment, it may not be practical to install Visual Studio IDE for debugging, because: 1. Visual Studio is not free. 2. VS has a large memory footprint. 3. The problem can even disappear after installing the Visual Studio.

 

Windbg and Process Explorer are 2 free download tools with much less impact on end user.

 

Using Windbg to obtain Application Crash information:

Windbg is included in the Microsoft “Debugging tools for Windows”. You may download the “Debugging tools for Windows” from the link below:

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

 

After installing windbg, the first thing is setting up the symbol path for it. The symbol path is critical during debugging, because windbg will retrieve the Windows system dlls internal symbols from the symbol path to display the meaningful name for functions/variables. Without the correct symbol loaded, the debugger can only display the numeric memory address for the functions/variables. For example, in a windbg without symbol path, if we use “k” command to obtain the current call stack, we will see following output(note the red numeric):

 

0:000> k

*** ERROR: Module load completed but symbols could not be loaded for notepad.exe

ChildEBP RetAddr 

WARNING: Stack unwind information not available. Following frames may be wrong.

0007f6ec 01001fe4 USER32!MessageBoxW

0007f70c 010027e7 notepad+0x1fe4

0007f944 010035b9 notepad+0x27e7

0007f960 7739c3b7 notepad+0x35b9

0007f98c 7739c484 USER32!EnableMenuItem+0x4cd4

0007fa04 7739ca68 USER32!EnableMenuItem+0x4da1

0007fa60 7739ce7a USER32!TranslateMessageEx+0xd5

0007fa88 7c82ec9e USER32!MsgWaitForMultipleObjectsEx+0x126

0007fb08 773966f0 ntdll!KiUserCallbackDispatcher+0x2e

0007fb24 7739668a USER32!DefWindowProcW+0xc7

0007fb6c 010038e2 USER32!DefWindowProcW+0x61

0007fb94 7739c3b7 notepad+0x38e2

0007fbc0 7739c484 USER32!EnableMenuItem+0x4cd4

0007fc38 7739ca68 USER32!EnableMenuItem+0x4da1

0007fc94 7739ce7a USER32!TranslateMessageEx+0xd5

0007fcbc 7c82ec9e USER32!MsgWaitForMultipleObjectsEx+0x126

0007fd3c 773966f0 ntdll!KiUserCallbackDispatcher+0x2e

0007fd58 7739668a USER32!DefWindowProcW+0xc7

0007fda0 010037fc USER32!DefWindowProcW+0x61

0007fdc8 7739c3b7 notepad+0x37fc

 

Yes, the windbg can only recognize the export function names of the dll. Since we do not have the symbol for notepad.exe, the windbg will only display a *magic* number for the function memory address which is not meaningful to us.

 

To set the symbol path, you may set the following path in the windbg menu item File->”Symbol File Path…”:

“srv*DownstreamStore*http://msdl.microsoft.com/download/symbols”

 

 

Change “DownstreamStore” to any local folder to cache the downloaded symbol files on your machine. For example, I normally use “srv*c:\LocalSymbols* http://msdl.microsoft.com/download/symbols”.

 

Now, you may open the windbg and click File->”Open Executable…” and select your application for launching. Windbg will launch your application and break in the initial loader breakpoint. You can check the red rectangle in the figure to ensure the symbol path is correct:

 

You may press “F5” or input “g” command to tell windbg to ignore the initial loader breakpoint. Now, you may manipulate your application to reproduce the crash. When the application crashes with exception, windbg will automatically break-in your application. Then you may input “k” command to get the full stack trace of this exception/failure.

 

Note: the “k” command may require some time to load the symbols from Microsoft symbol server for the system dlls the first time. So the windbg will report “busy…” for sometime. You may wait to have a coffee for the symbol loading and output.

 

Use Process Explorer to obtain application hangs information:

Process Explorer can be free downloaded in the

http://www.microsoft.com/technet/sysinternals/utilities/ProcessExplorer.mspx

 

Process Explorer is especially suitable for troubleshooting hang problem in application, because it has a very good UI for monitor threads activity.

 

After downloading the Process Explorer, the first thing to do is still setting the symbol server path as we do in windbg:

 

 

Note: in the “Dbghelp.dll path” textbox above, I used to Dbghelp.dll in “Debugging Tools for Windows” package, because this Dbghelp.dll has a higher version than the one in “%windir%/System32/”. If you do not have “Debugging Tools for Windows” installed on end user machine, you can live with using the one in “%windir%/System32/” directory.

 

Let’s demonstrate the Process Explorer usage with a sample. In a .Net Winform application, we write the following code in an existing VC application to simulate the hang in application:

 

Sleep(30000);

 

While the application is hang, we can find the application in the Process Explorer and double click it to launch its property dialog. Then, we should switch to the “Threads” tabpage to examine the thread status. This is what I got in the sample:

 

 

As you can see there is only one thread in the test application. Then, we can select the thread and click “Stack” button in the dialog, we got:

 

 

Aha, by examining this stack trace, we can see that “VMQueryHelper” function calls Sleep() function, which causes the hang.

 

Use Windbg to obtain application hangs information:

Windbg can also be used to troubleshoot the application hang. I assume you have downloaded the windbg and set up the symbol path the same way as said in first section.

 

Now, another .Net application is hanging in customer’s machine. We can use windbg to attach the hang application with the menu item below:

 

 

In the dialog, we can choose the hang application and click Ok button:

 

 

In the windbg, the first thing we should do is checking how many threads are there in the hang application with “~*” command, please see the figure below.

 

As you can see in the figure, thread #0 is executing _CorExeMain method, which is .Net Winform application main GUI thread. The thread #5 has a “.” in the front which means the current examine thread. Because the application hang occurs in the main GUI thread, we should switch the current thread to the #0 thread by inputting “~0s”, see figure.

 

Now, the context is switched to the main GUI hang thread. We can input the “K” command to get the full stack trace of the hang thread to see what it is waiting for.

 

All these 3 commands and output are listed in the figure below.

 

 

As you can see it is the Button.Click() method in .Net Winform calls the System.Threading.Thread.Sleep(30000) that caused the hang.

http://blogs.msdn.com/b/msdnts/archive/2006/11/24/how-to-debug-application-crash-hang-in-production-environment.aspx