in Code, HowTo

Crash Reporting For MinGW 32 (Windows) and Clang (macOS) With Qt

I recently had a customer describe some very random crashes with my software. There didn’t seem to be a pattern or a way for me to reproduce the problems locally to debug them. So my first thought was to get him to install a version with some kind of crash reporting tooling so I could get a stack trace to help track down the issue.

I’d looked into implementing some form of crash reporting quite a while ago, but it was never a very high priority for me because I don’t get a lot of bug reports. In this case though it seemed like it would be easiest if I could produce a version of my software with some built-in stack tracing.

The first thing I did was to look at what libraries were available for this. My criteria were:

  • simple to use/integrate with a Qt application
  • works with the MinGW 32-bit compiler on Windows and the clang compiler on macOS
  • inexpensive (or free!)
  • usable in commercial software

The most promising were BreakPad (older) or CrashPad (newer) from Google. From what I understand, Breakpad no longer works on macOS which is why they switched to CrashPad. Unfortunately CrashPad doesn’t handle 32-bit MinGW builds. The reason I’m stuck with the 32-bit version is that Qt currently ships its MinGW builds of the libraries and toolchain using the 32-bit MinGW 4.9.2 compiler.

So after a lot of searching and piecing things together, I’ve created something that works and fits my criteria. It’s very simple – all it does is save the stack trace to a file that the user can send me – and requires some instructions to the user to work with it. If I wanted to get fancier I could have it automatically post the information to a web server, but for now this is simple and it works.

It might work on Linux too since the code path for macOS should be POSIX compliant, though I haven’t tried it. It could also be extended to handle MSVC compiles (or maybe it already does!), but I don’t use that compiler so I can’t test it.

I used many different sites in my search, but my primary sources were Catching Exceptions and Printing Stack Traces for C on Windows, Linux, & Mac by Job Vranish, Printing a Stack Trace with MinGW by Daniel Holden, and the C++ name mangling article on Wikipedia.

Code

The code, along with example usage, may be found on the asmCrashReport GitHub page.

What Makes This One Different?

None of the sources I found online handled the cases quite the same way and they didn’t give the results I was looking for (name demangling for example). What I’ve put together uses ideas from a bunch of different sources.

In addition, since I am targetting Qt applications using C++11, I took the liberty of using Qt classes and methods when putting this together.

Usage

In your .pro file, you need to include the asmCrashreport.pri file. e.g.:

This will define ASM_CRASH_REPORT for the preprocessor and modify the C/CXX and linker flags to include the debug symbols properly.

I usually wrap it in a config option so it can be included using “CONFIG += asmCrashReport” on the command line or in Qt Creator.

In your main.cpp, include the header:

This provides a simple API:

In your main() function, set your signal handler after you have declared your QApplication and set the application name and version number:

Example Results

(Aside: If you can figure out why the macOS stack trace gives junk data for the most recent frame I’ll owe you a beer!)

Some Code Details

I’m not going to go over everything in the code, just some highlights of things that are maybe different from the other posts on this subject.

I have shortened the code and split it up for this post. You can find the full source in the GitHub repo.

_writeLog()

The first part is fairly straightforward – the _writeLog() function writes the log and calls the callback if one was provided.

_addr2line()

This next function – _addr2line() – uses an external tool to translate memory addresses to file and line numbers. On macOS this is the atos tool, on Windows we use the addr2line tool from Cygwin.

Note that on Windows this means we need to ship the add2line tool and supporting DLLs to the end-user. See the GitHub repo README for more details.

If the external tool fails to run it will return an error. If it succeeded but the tool could not find the symbol, it will return an empty string. Otherwise it returns the file and line information as a string.

Windows-Specific

Next up is the Windows-specific code.

Note that I only include the stack frame layout for i386 since we are targetting the MinGW 32-bit compiler.

The other interesting thing here is what I’m doing with the result of addr2line() while walking the stack trace. The addr2line tool does not demangle the C++ symbols, so we end up with symbols like “_ZN9crashTest12divideByZeroEi“. To make them more readable, I identify them with a regular expression, use a function to demangle them properly (abi::__cxa_demangle()), and substitute the human-readable version of the symbol – e.g. “crashTest::divideByZero(int)“.

The last function – winExceptionHandler() – just converts the exception to a human-readable string, gathers the stack trace, and writes the log.

macOS-Specific

The macOS-specific code is similar, but we can use the backtrace() and backtrace_symbols() functions to simplify things.

These functions get us part-way there. The output of these gives us something like:

With a regular expression, I’m matching the _ZN* symbol and sending that to _addr2Line() to look up the file and line number with atos. If it finds it, I’m just substituting the result to get:

The other two functions here – _posixSignalHandler() and _posixSetupSignalHandler() set up the necessary plumbing to capture the signal, convert it a human-readable form, gather the stack trace, and write the log.

setSignalHandler()

Finally, we have the main setup function. All this does is set some of the static variables and call the appropriate signal handler function.

Final Notes

While what I have currently works for me, you might need to adapt it for your own use.

I’ve intentionally kept this code as simple as I can. Some of the possibilities for modification would be to include a working version for MSVC, include a Qt-based way to submit the crash log via a web server, expand the data included in the report to include machine information, etc..

If you make any changes, please feel free to submit pull requests on the github page.

I hope someone finds it useful!

Write a Comment

Comment

22 Comments

  1. Hi, Andy:

    Thanks for your quick reply!

    I just did a quick test with 64-bit mingw on your test program. Everything worked fine until I checked the exported report:

    EXCEPTION_INT_DIVIDE_BY_ZERO

    [0] 0x0000000000406e5f F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [1] 0x0000000000406c01 F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [2] 0x0000000000406bca F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [3] 0x0000000000406dd5 F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [4] 0x000000000040395a F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [5] 0x00000000004052a0 F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [6] 0x00000000004013c7 F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [7] 0x00000000004014cb F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [8] 0x00007ffb5a0f3034 F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized
    [9] 0x00007ffb5b821551 F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\tools\addr2line: F:\keyu_projects\asmCrashReport\build-test-Desktop_Qt_5_12_7_MinGW_64_bit-Release\release\asmCrashReportTest.exe: File format not recognized

    Switch back to 32bit and the report seems OK I guess:

    EXCEPTION_INT_DIVIDE_BY_ZERO

    [0] 0x0000000000405d9a crashTest::_divideByZero(int) at main.cpp:32
    [1] 0x0000000000405b6d crashTest::_function2(int) at main.cpp:39
    [2] 0x0000000000405b4a crashTest::_function1() at main.cpp:44
    [3] 0x0000000000405d17 crashTest::divideByZero() at main.cpp:14
    [4] 0x0000000000403623 qMain(int, char**) at main.cpp:129
    [5] 0x0000000000404802 ?? at qtmain_win.cpp:?
    [6] 0x00000000056b1f38 ?? ??:0
    [7] 0x000000001bbda1b2 ?? ??:0
    [8] 0x000000002d725bb3 ?? ??:0
    [9] 0x00000000972e0d7a ?? ??:0
    [10] 0x000000001587354f ?? ??:0
    [11] 0x00000000615845bb ?? ??:0
    [12] 0x0000000003101736 ?? ??:0
    [13] 0x00000000a5b030bb ?? ??:0
    [14] 0x000000008cc62beb ?? ??:0
    [15] 0x000000008ff95671 ?? ??:0
    [16] 0x00000000bb00eab9 ?? ??:0
    [17] 0x00000000ad3a719a ?? ??:0
    [18] 0x00000000e7344448 ?? ??:0
    [19] 0x00000000fc193413 ?? ??:0
    [20] 0x00000000a0c42693 ?? ??:0

    Thanks,
    Alex

    • I managed to get it to work: Just change the addr2line.exe to its 64-bit version.

      Thanks,
      Alex

  2. Hi,

    The tool is awesome and helpful! I am wondering if it supports(or possibly will in the future) 64-bit windows mingw because now a days most of the qt apps are based on 64-bit arch.

    Thanks!

  3. Hi Andy,
    Nice Solution!!
    The stack traces are generating flawlessly for me in mingw. However since -g option is given, the size of the exe is increased to more than 10 times because all the debug information is present. Is there something like -rdynamic flag which can be used and at least get the function names in case of a crash ? Since the size of the exe is increased so much, it cannot be used in customer location. I have used rdynamic linker flag in linux and then used addr2line, I am getting the stack trace with an increase of only 20% in the binary size.
    But for mingw, i am not finding a way to use rdynamic flag.

    Regards,
    Umang

    • Thanks Aurélien! There is no license. You can do whatever you want with it.

      (I have had this question enough that I might have to write up a blog post on “Why not everything needs needs to be owned by someone” :-))

      Good luck with your project!

      • Thanks for the answer, Andy. Sounds like this is public domain then.

        It would be great to add this to the git repository so anxious people like me are not afraid of using it 😉

        • I understand – but it’s not public domain. It doesn’t have a license.

          I won’t get into it here in this tiny space, but generally I believe the legalese/licensing culture we’ve allowed to develop around sharing is harmful. So for small things like this I choose not to contribute to that culture.

  4. Hello Andy,
    I have studied a bit the code and I find it very useful.
    I have two questions however:
    – Why do you need to setup alternate stack??
    – I have seen that with this code, at least on my Mac core dump file are not generated. Is that intended?
    Thanks!

    • “Why do you need to setup alternate stack??”

      This was a trick from one of the articles I reference to allow this to handle stack overflows.

      “…core dump file are not generated. Is that intended?”

      I didn’t have intent either way. From what I read, macOS doesn’t produce full core dumps unless you turn it on. Or are you saying that using this code prevents that from working?

      My reason for putting this together was that one of my customers had what sounded like random crashes on Windows and I wanted to track it down (turned out to be a hardware problem).

      The stack trace – for my use-cases – is the most useful information a customer can give me. On macOS this is already handled by the OS, so I don’t really need to use this there.

      • According to my tests, even enabling by code core dumps:

        dump file is not generated

        • From the Apple tech note:

          “This does not, however, obviate the need for a writable /cores directory.”

          Did you check this? Did you try changing the limit on the command line to see if that works?

          Other than that I’m afraid I don’t know what else to look at as it’s not something I’ve needed.

  5. Thanks for the stack analyzer.

    I integrated the code as mentioned. But, I’m getting the following info only in the stack trace. Could you please let me know if I’m missing something?

    EXCEPTION_ACCESS_VIOLATION

    0x000000007734e486 * Error running command
    D:/tools/addr2line -f -p -e D:\release\test_app.exe 0x000000007734e486
    Process failed to start: Access is denied.

    We are unable to proceed at all. Any pointer should be helpful for us.

    Thanks in advance.

    • Do you have the addr2line program in the right place? (see the readme) Does your program have the correct permissions to run it?

      The paths don’t look quite right there – it should be looking next to the executable for the tools directory to find addr2line.

      You should be able to see what’s happening using your debugger. It’s almost impossible to analyze and debug problems remotely & without the code.

      • Thanks for the reply.
        The path could be an editing mistake.
        This is the actual path. Please let me know if this is right.

        0x000000007734e0e3 * Error running command
        D:/mingwprj/release/tools/addr2line -f -p -e D:\mingwprj\release\st_dab_radio.exe 0x000000007734e0e3
        Process failed to start: Access is denied.

        When we made a deliberate divide by zero error in the code to cross verify, the stack trace was fine. So, I think it has right permissions to run.We don’t have any debugger other than Visual Studio debugger when the QT crash happens. It shows an exception in QTWidgets.dll.

        Please provide your suggestion on the possible debugging options for this issue.
        Thanks in advance.

        • That path looks correct.

          If you are crashing in Qt, then it’s possible it’s finding the wrong Qt version (linked to one version and running with a different one). In my experience, crashes in Qt code itself are fairly rare.

          Other than stepping through the code with the debugger to figure out what’s happening I don’t have any other suggestions.

          I really can’t debug code by going back-and-forth in a comment section like this 🙂

          • Thanks.
            Actually, this crash issue doesn’t happen when we run with a debugger. That’s why an external tool to get the stack trace was required.
            The doubt was in how to interpret this error message “Process failed to start: Access is denied”.
            Anyways, thanks!

          • You could try using the other version of QProcess::start() in asmCrashReport.cpp. Maybe the argument list included as part of the command isn’t being quoted properly?

  6. hello Andy.
    I use your crash report code in my qt project(mingw32 on windows)
    but,when some crash in dll file,I can not get the infomation.
    how to solve that?