21 September 2006

Debugging Production Applications using ADPlus

ADPlus is used to take snapshots of processes in environments where development-time debugging tools are not available e.g. Production server farms, end-users workstations. ADPlus is a Visual Basic script provided as part of the Debugging Tools for Windows that wraps the CDB debugger.

Snapshots of processes taken with ADPlus are analysed using the Windows Debugger (WinDbg). Managed processes in particular are analysed using SOS.dll, which contains extensions to WinDbg for debugging managed processes.

Practical applications include taking snapshots of WinForms and ASP.NET processes to interrogate unhandled exceptions, memory utilisation, resource contention etc.

Installing Debugging Tools for Windows in Target Environment

Download debugging tools;
XCOPY deploy to target environment;
Create a folder to store snapshots;
Set CScript as the default scripting engine (required by ADPlus, also requires Windows Script Host 5.6 or higher, interrogate version using CSCRIPT /? command)

CSCRIPT /H:CScript

Taking a Snapshop using ADPlus

The following examples demonstrate taking a snapshot of a WinForms process:

Via process id
ADPlus.vbs -c ADPlus_Config.xml -p 968
Via process name
ADPlus.vbs -c ADPlus_Config.xml -pn HelloWorldGUI.exe

Note that ADPlus will warn that the _NT_SYMBOL_PATH environment variable is not set. This is ok as the environment variable is not required for taking snapshots, however, it is required for analysing the snapshot later.

The following example ADPlus_Config.xml file configures ADPlus for Hang mode (hang mode instructs ADPlus to take the snapshot immediately, crash mode is another option that attaches the debugger to the process, then waits for process to exit before generating the snapshot):

<ADPlus>
    <Settings>
        <RunMode>HANG</RunMode>
        <Option>Quiet</Option>
      <OutputDir> C:\Snapshots </OutputDir>
    </Settings>
    
    <PreCommands>
        <Cmd> .load C:\Program Files\Debugging Tools for Windows\clr10\sos.dll </Cmd>
    </PreCommands>
    <Exceptions>
        
        
        <Config>
            <Code>
                 AllExceptions
            </Code>
            <Actions1>
                 FullDump;
            </Actions1>
            <ReturnAction1>
                 GN
            </ReturnAction1>
        </Config>
    </Exceptions>
</ADPlus>

Using WinDBG and SOS.dll

Install debugging tools;
Add installation location to PATH environment variable;
Create a folder to store symbols;
Add new environment variable, referencing the symbols folder:

Name: _NT_SYMBOL_PATH
Value: SRV*c:\temp\symbols* http://msdl.microsoft.com/download/symbols

This setting is used by WinDbg to dynamically download symbols (pdb files) on demand.

Launch WinDbg.exe;
Open the snapshot using File --> Open Crash Dump;
Load SOS.dll by entering the following command:

.load C:\Program Files\Debugging Tools for Windows\clr10\sos.dll

Useful WinDbg/SOS.dll commands are:

!help (all available commands)
!help <functionname> (help on provided function)
!clrstack (shows managed stack)
!dae (dumps all managed exceptions, noting that instances of ExecutionEngineException, StackOverflowException and OutOfMemoryException are normal)
!EEStack (shows full stack trace, including managed and unmanaged calls)
!do (dump object, including fields)

Further Information

How to use ADPlus to troubleshoot "hangs" and "crashes"

Production Debugging for .NET Framework Applications

Mini Dump Snapshots and the New SOS

Tips on Debugging Managed Exceptions with ADPlus and WinDbg