Recently we’ve been experiencing a number of software failures company-wide in our custom-built line-of-business app. The application connects via Web Services to the Sabre GDS to obtain hotel room pricing and availability at multiple hotels simultaneously.
The application is multi-threaded, consuming upwards of 300 or more threads during periods of heavy use. Even thought the system is multi-threaded occasionally the GUI becomes unresponsive since the software is using almost all available CPU cycles. When this happens, the user sometimes decides to “end-task” or terminate the program prior to completion of the cpu-intensive operation. When the application terminates unexpectedly or the user ends the program early, we are left with server-tier connections being tied up until they time out approximately 20 minutes later.
When a crash occurs Windows Error Reporting or Microsoft Error Reporting fires up and asks the user to send an error report to Microsoft to enable the WinQual website to provide diagnostic information. We cannot send our data off to a third party and so needed a simply way to disable the Windows Error Reporting / Microsoft Error Reporting tool, while providing our tech-support team a way to measure and respond to software failures. Microsoft has a tool called the Corporate Error Reporting Tool which allows a company to send these error reports to the server of their choice, however Microsoft only provides their Corporate Error Reporting Tool to customers of the Software Assurance for Volume Licensing program.
I decided to create a reporting tool that we could use in house on our Windows XP machines that was light, fast, and completely configurable in house by us. I knew from years of experience with Windows NT that there was a program built into Windows that did post-mortem crash analysis called Dr. Watson, so I did some research on Google looking for information about how to replace Dr. Watson (drwtsn32.exe) with the program of my choosing. This seemed straight-forward enough. Replace the registry entry in HKEY_LOCAL_MACHINE/Software/Microsoft/Windows NT/CurrentVersion/AEDebug with one referring to our crash analysis tool. No Go. It appears the AEDebug entry is not active when Windows Error Reporting or Microsoft Error Reporting is installed. So I did a little more poking around in the registry, and what did I find? A wonderful little registry key titled “Image File Execution Options”. Looked interesting. According to Microsoft’s documentation this registry key enables us to instruct Windows to start the particular debugger of our choice whenever a certain program is started by the user or by the operating system. This means whenever the DWWIN.EXE process is started by the Windows Error Reporting Tool (WER), we can instead open the program of our choice. DWWIN.EXE is the process which gathers the hex-memory-dump file and other information collected by the WER, packages it in a .CAB file and sends it to http://watson.microsoft.com. I wrote a small utility that gathers some information from the WER manifest, along with some pertinent hardware and OS specifics such as the computer name, the user name, and available physical and virtual memory stats, and emails our tech-support team alerting them of the problem as it happens. This allows us to use our Spector 360 software to see exactly what the user was doing at the time of the crash.
The following is an excerpt from Microsoft’s MSDN website on how to configure the appropriate Registry Key:
(USE THIS AT YOUR OWN RISK – I CANNOT BE HELD RESPONSIBLE IF YOU CORRUPT YOUR REGISTRY)
- Start the Registry Editor (regedit).
- In the Registry Editor, open the HKEY_LOCAL_MACHINE folder.
- Navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\currentversion\image file execution options.
- In the Image File Execution Options folder, locate the name of the application you want to debug, such as myapp.exe. If you cannot find the application you want to debug:
- Right-click the Image File Execution Options folder, and on the shortcut menu, click New Key.
- Right-click the new key, and on the shortcut menu, click Rename.
- Edit the key name to the name of your application; myapp.exe, in this example.
- Right-click the myapp.exe folder, and on the shortcut menu, click New String Value.
- Right-click the new string value, and on the shortcut menu, click Rename.
- Change the name to debugger.
- Right-click the new string value, and on the shortcut menu, click Modify.
The Edit String dialog box appears.
- In the Value data box, type vsjitdebugger.exe.
- Click OK.
- From the Registry menu, click Exit.
- The directory containing vsjitdebugger.exe must be in your system path. To add it to the system path, follow these steps:
- Open the Control Panel in Classic view, and double-click System.
- In System Properties, click the Advanced tab.
- On the Advanced tab, click Environment Variables.
- In the Environment Variables dialog box, under System variables, select Path, then click the Edit button.
- In the Edit System Variable dialog box, add the directory to Variable value. Use a semicolon to separate it from other entries in the list.
- Click OK to close the Edit System Variable dialog box.
- Click OK to close the Environment Variables dialog box.
- Click OK to close the System Properties dialog box.
Replace myapp.exe in step 4 with DWWIN.exe. Replace vsjitdebugger.exe in step 9 with the name of your own executable that you want to run in place of DWWIN.exe