in Code, HowTo

Crash Reporting For MinGW 32 (Windows) and Clang (macOS) With Qt

I recently had a customer describe some very random crashes with my software. There didn’t seem to be a pattern or a way for me to reproduce the problems locally to debug them. So my first thought was to get him to install a version with some kind of crash reporting tooling so I could get a stack trace to help track down the issue.

I’d looked into implementing some form of crash reporting quite a while ago, but it was never a very high priority for me because I don’t get a lot of bug reports. In this case though it seemed like it would be easiest if I could produce a version of my software with some built-in stack tracing.

The first thing I did was to look at what libraries were available for this. My criteria were:

  • simple to use/integrate with a Qt application
  • works with the MinGW 32-bit compiler on Windows and the clang compiler on macOS
  • inexpensive (or free!)
  • usable in commercial software

The most promising were BreakPad (older) or CrashPad (newer) from Google. From what I understand, Breakpad no longer works on macOS which is why they switched to CrashPad. Unfortunately CrashPad doesn’t handle 32-bit MinGW builds. The reason I’m stuck with the 32-bit version is that Qt currently ships its MinGW builds of the libraries and toolchain using the 32-bit MinGW 4.9.2 compiler.

So after a lot of searching and piecing things together, I’ve created something that works and fits my criteria. It’s very simple – all it does is save the stack trace to a file that the user can send me – and requires some instructions to the user to work with it. If I wanted to get fancier I could have it automatically post the information to a web server, but for now this is simple and it works.

It might work on Linux too since the code path for macOS should be POSIX compliant, though I haven’t tried it. It could also be extended to handle MSVC compiles (or maybe it already does!), but I don’t use that compiler so I can’t test it.

I used many different sites in my search, but my primary sources were Catching Exceptions and Printing Stack Traces for C on Windows, Linux, & Mac by Job Vranish, Printing a Stack Trace with MinGW by Daniel Holden, and the C++ name mangling article on Wikipedia.

Code

The code, along with example usage, may be found on the asmCrashReport GitHub page.

What Makes This One Different?

None of the sources I found online handled the cases quite the same way and they didn’t give the results I was looking for (name demangling for example). What I’ve put together uses ideas from a bunch of different sources.

In addition, since I am targetting Qt applications using C++11, I took the liberty of using Qt classes and methods when putting this together.

Usage

In your .pro file, you need to include the asmCrashreport.pri file. e.g.:

This will define ASM_CRASH_REPORT for the preprocessor and modify the C/CXX and linker flags to include the debug symbols properly.

I usually wrap it in a config option so it can be included using “CONFIG += asmCrashReport” on the command line or in Qt Creator.

In your main.cpp, include the header:

This provides a simple API:

In your main() function, set your signal handler after you have declared your QApplication and set the application name and version number:

Example Results

(Aside: If you can figure out why the macOS stack trace gives junk data for the most recent frame I’ll owe you a beer!)

Some Code Details

I’m not going to go over everything in the code, just some highlights of things that are maybe different from the other posts on this subject.

I have shortened the code and split it up for this post. You can find the full source in the GitHub repo.

_writeLog()

The first part is fairly straightforward – the _writeLog() function writes the log and calls the callback if one was provided.

_addr2line()

This next function – _addr2line() – uses an external tool to translate memory addresses to file and line numbers. On macOS this is the atos tool, on Windows we use the addr2line tool from Cygwin.

Note that on Windows this means we need to ship the add2line tool and supporting DLLs to the end-user. See the GitHub repo README for more details.

If the external tool fails to run it will return an error. If it succeeded but the tool could not find the symbol, it will return an empty string. Otherwise it returns the file and line information as a string.

Windows-Specific

Next up is the Windows-specific code.

Note that I only include the stack frame layout for i386 since we are targetting the MinGW 32-bit compiler.

The other interesting thing here is what I’m doing with the result of addr2line() while walking the stack trace. The addr2line tool does not demangle the C++ symbols, so we end up with symbols like “_ZN9crashTest12divideByZeroEi“. To make them more readable, I identify them with a regular expression, use a function to demangle them properly (abi::__cxa_demangle()), and substitute the human-readable version of the symbol – e.g. “crashTest::divideByZero(int)“.

The last function – winExceptionHandler() – just converts the exception to a human-readable string, gathers the stack trace, and writes the log.

macOS-Specific

The macOS-specific code is similar, but we can use the backtrace() and backtrace_symbols() functions to simplify things.

These functions get us part-way there. The output of these gives us something like:

With a regular expression, I’m matching the _ZN* symbol and sending that to _addr2Line() to look up the file and line number with atos. If it finds it, I’m just substituting the result to get:

The other two functions here – _posixSignalHandler() and _posixSetupSignalHandler() set up the necessary plumbing to capture the signal, convert it a human-readable form, gather the stack trace, and write the log.

setSignalHandler()

Finally, we have the main setup function. All this does is set some of the static variables and call the appropriate signal handler function.

Final Notes

While what I have currently works for me, you might need to adapt it for your own use.

I’ve intentionally kept this code as simple as I can. Some of the possibilities for modification would be to include a working version for MSVC, include a Qt-based way to submit the crash log via a web server, expand the data included in the report to include machine information, etc..

If you make any changes, please feel free to submit pull requests on the github page.

I hope someone finds it useful!

Leave a Reply for DeanFeng Cancel Reply

Write a Comment

Comment

  1. hello Andy.
    I use your crash report code in my qt project(mingw32 on windows)
    but,when some crash in dll file,I can not get the infomation.
    how to solve that?

  2. Thanks for the stack analyzer.

    I integrated the code as mentioned. But, I’m getting the following info only in the stack trace. Could you please let me know if I’m missing something?

    EXCEPTION_ACCESS_VIOLATION

    0x000000007734e486 * Error running command
    D:/tools/addr2line -f -p -e D:\release\test_app.exe 0x000000007734e486
    Process failed to start: Access is denied.

    We are unable to proceed at all. Any pointer should be helpful for us.

    Thanks in advance.

    • Do you have the addr2line program in the right place? (see the readme) Does your program have the correct permissions to run it?

      The paths don’t look quite right there – it should be looking next to the executable for the tools directory to find addr2line.

      You should be able to see what’s happening using your debugger. It’s almost impossible to analyze and debug problems remotely & without the code.

      • Thanks for the reply.
        The path could be an editing mistake.
        This is the actual path. Please let me know if this is right.

        0x000000007734e0e3 * Error running command
        D:/mingwprj/release/tools/addr2line -f -p -e D:\mingwprj\release\st_dab_radio.exe 0x000000007734e0e3
        Process failed to start: Access is denied.

        When we made a deliberate divide by zero error in the code to cross verify, the stack trace was fine. So, I think it has right permissions to run.We don’t have any debugger other than Visual Studio debugger when the QT crash happens. It shows an exception in QTWidgets.dll.

        Please provide your suggestion on the possible debugging options for this issue.
        Thanks in advance.

        • That path looks correct.

          If you are crashing in Qt, then it’s possible it’s finding the wrong Qt version (linked to one version and running with a different one). In my experience, crashes in Qt code itself are fairly rare.

          Other than stepping through the code with the debugger to figure out what’s happening I don’t have any other suggestions.

          I really can’t debug code by going back-and-forth in a comment section like this 🙂

          • Thanks.
            Actually, this crash issue doesn’t happen when we run with a debugger. That’s why an external tool to get the stack trace was required.
            The doubt was in how to interpret this error message “Process failed to start: Access is denied”.
            Anyways, thanks!

          • You could try using the other version of QProcess::start() in asmCrashReport.cpp. Maybe the argument list included as part of the command isn’t being quoted properly?