I recently had a customer describe some very random crashes with my software. There didn’t seem to be a pattern or a way for me to reproduce the problems locally to debug them. So my first thought was to get him to install a version with some kind of crash reporting tooling so I could get a stack trace to help track down the issue.
I’d looked into implementing some form of crash reporting quite a while ago, but it was never a very high priority for me because I don’t get a lot of bug reports. In this case though it seemed like it would be easiest if I could produce a version of my software with some built-in stack tracing.
The first thing I did was to look at what libraries were available for this. My criteria were:
- simple to use/integrate with a Qt application
- works with the MinGW 32-bit compiler on Windows and the clang compiler on macOS
- inexpensive (or free!)
- usable in commercial software
The most promising were BreakPad (older) or CrashPad (newer) from Google. From what I understand, Breakpad no longer works on macOS which is why they switched to CrashPad. Unfortunately CrashPad doesn’t handle 32-bit MinGW builds. The reason I’m stuck with the 32-bit version is that Qt currently ships its MinGW builds of the libraries and toolchain using the 32-bit MinGW 4.9.2 compiler.
So after a lot of searching and piecing things together, I’ve created something that works and fits my criteria. It’s very simple – all it does is save the stack trace to a file that the user can send me – and requires some instructions to the user to work with it. If I wanted to get fancier I could have it automatically post the information to a web server, but for now this is simple and it works.
It might work on Linux too since the code path for macOS should be POSIX compliant, though I haven’t tried it. It could also be extended to handle MSVC compiles (or maybe it already does!), but I don’t use that compiler so I can’t test it.
I used many different sites in my search, but my primary sources were Catching Exceptions and Printing Stack Traces for C on Windows, Linux, & Mac by Job Vranish, Printing a Stack Trace with MinGW by Daniel Holden, and the C++ name mangling article on Wikipedia.
Code
The code, along with example usage, may be found on the asmCrashReport GitHub page.
What Makes This One Different?
None of the sources I found online handled the cases quite the same way and they didn’t give the results I was looking for (name demangling for example). What I’ve put together uses ideas from a bunch of different sources.
In addition, since I am targetting Qt applications using C++11, I took the liberty of using Qt classes and methods when putting this together.
Usage
In your .pro file, you need to include the asmCrashreport.pri file. e.g.:
|
1 2 3 |
if ( !include( ../asmCrashReport.pri ) ) { error( Could not find the asmCrashReport.pri file. ) } |
This will define ASM_CRASH_REPORT for the preprocessor and modify the C/CXX and linker flags to include the debug symbols properly.
I usually wrap it in a config option so it can be included using “CONFIG += asmCrashReport” on the command line or in Qt Creator.
|
1 2 3 4 5 |
asmCrashReport { if ( !include( ../asmCrashReport.pri ) ) { error( Could not find the asmCrashReport.pri file. ) } } |
In your main.cpp, include the header:
|
1 2 3 |
#ifdef ASM_CRASH_REPORT #include "asmCrashReport.h" #endif |
This provides a simple API:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
#ifndef ASMCRASHREPORT_H #define ASMCRASHREPORT_H #include <QString> namespace asmCrashReport { /// Function signature for a callback after the log is written. /// @param inLogFileName The full path to the log file which was written. /// @param inSuccess Whether the file was successfully written. typedef void (*logWrittenCallback)( const QString &inLogFileName, bool inSuccess ); ///! Set a signal handler to capture stack trace to a log file. /// /// @param inCrashReportDirPath Path to directory to write our crash report to. /// If not set, it will use Desktop/<App Name> Crash Logs/ /// @param inLogWrittenCallback A function to call after we've written the log. /// You might use this to display a message about where to find the log. void setSignalHandler( const QString &inCrashReportDirPath = QString(), logWrittenCallback inLogWrittenCallback = nullptr ); } #endif |
In your main() function, set your signal handler after you have declared your QApplication and set the application name and version number:
|
1 2 3 4 5 6 7 8 9 10 |
QApplication app( argc, argv ); app.setApplicationName( QStringLiteral( "asmCrashReportExample" ) ); app.setApplicationVersion( QStringLiteral( "1.0.0" ) ); #ifdef ASM_CRASH_REPORT asmCrashReport::setSignalHandler( QString(), [] (const QString &inFileName, bool inSuccess) { // do something with results - I show a QMessageBox (see example) }); #endif |
Example Results
|
1 2 3 4 5 6 7 8 9 10 11 12 |
asmCrashReportExample v1.0.0 07 Aug 2017 @ 09:42:38 Caught SIGFPE: (integer divide by zero) 2 libsystem_platform.dylib 0x00007fffacc42b3a _sigtramp + 26 3 ??? 0x0000000000000000 0x0 + 0 4 asmCrashReportExample 0x0000000100008bd4 crashTest::function2(int) (in asmCrashReportExample) (main.cpp:26) 5 asmCrashReportExample 0x0000000100008baa crashTest::function1() (in asmCrashReportExample) (main.cpp:31) 6 asmCrashReportExample 0x00000001000085d5 crashTest::crashMe() (in asmCrashReportExample) (main.cpp:13) 7 asmCrashReportExample 0x00000001000083de main + 206 8 libdyld.dylib 0x00007fffaca33235 start + 1 |
|
1 2 3 4 5 6 7 8 9 10 11 |
asmCrashReportExample v1.0.0 07 Aug 2017 @ 13:48:22 EXCEPTION_INT_DIVIDE_BY_ZERO [0] 0x00000000004056ea crashTest::divideByZero(int) at C:\dev\asmCrashReport\build-example-Qt_5_9_1_MinGW_32bit/../example/main.cpp:18 [1] 0x0000000000405749 crashTest::function2(int) at C:\dev\asmCrashReport\build-example-Qt_5_9_1_MinGW_32bit/../example/main.cpp:25 [2] 0x0000000000405726 crashTest::function1() at C:\dev\asmCrashReport\build-example-Qt_5_9_1_MinGW_32bit/../example/main.cpp:30 [3] 0x0000000000405707 crashTest::crashMe() at C:\dev\asmCrashReport\build-example-Qt_5_9_1_MinGW_32bit/../example/main.cpp:13 [4] 0x0000000000403388 qMain(int, char**) at C:\dev\asmCrashReport\build-example-Qt_5_9_1_MinGW_32bit/../example/main.cpp:61 [5] 0x0000000000404592 ?? at qtmain_win.cpp:? |
(Aside: If you can figure out why the macOS stack trace gives junk data for the most recent frame I’ll owe you a beer!)
Some Code Details
I’m not going to go over everything in the code, just some highlights of things that are maybe different from the other posts on this subject.
I have shortened the code and split it up for this post. You can find the full source in the GitHub repo.
_writeLog()
The first part is fairly straightforward – the _writeLog() function writes the log and calls the callback if one was provided.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
// ... #includes namespace asmCrashReport { // ... define some static vars void _writeLog( const QString &inSignal, const QStringList &inFrameInfoList ) { bool fileWritten = false; // ... write the log file if ( sLogWrittenCallback != nullptr ) { (*sLogWrittenCallback)( cFileName, fileWritten ); } } |
_addr2line()
This next function – _addr2line() – uses an external tool to translate memory addresses to file and line numbers. On macOS this is the atos tool, on Windows we use the addr2line tool from Cygwin.
Note that on Windows this means we need to ship the add2line tool and supporting DLLs to the end-user. See the GitHub repo README for more details.
If the external tool fails to run it will return an error. If it succeeded but the tool could not find the symbol, it will return an empty string. Otherwise it returns the file and line information as a string.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
// Resolve symbol name & source location QString _addr2line( const QString &inProgramName, void const * const inAddr ) { const QString cAddrStr = QStringLiteral( "0x%1" ).arg( quintptr( inAddr ), 16, 16, QChar( '0' ) ); #ifdef Q_OS_MAC const QString cCommand = QStringLiteral( "atos -o \"%1\" -arch x86_64 %2" ).arg( inProgramName, cAddrStr ); #else const QString cCommand = QStringLiteral( "%1/tools/addr2line -f -p -e %2 %3" ).arg( QCoreApplication::applicationDirPath(), inProgramName, cAddrStr ); #endif sProcess->start( cCommand, QIODevice::ReadOnly ); if ( !sProcess->waitForFinished() ) { return QStringLiteral( "* Error running command\n %1\n %2" ).arg( cCommand, sProcess->errorString() ); } const QString cLocationStr = QString( sProcess->readAll() ).trimmed(); return (cLocationStr == cAddrStr) ? QString() : cLocationStr; } |
Windows-Specific
Next up is the Windows-specific code.
Note that I only include the stack frame layout for i386 since we are targetting the MinGW 32-bit compiler.
The other interesting thing here is what I’m doing with the result of addr2line() while walking the stack trace. The addr2line tool does not demangle the C++ symbols, so we end up with symbols like “_ZN9crashTest12divideByZeroEi“. To make them more readable, I identify them with a regular expression, use a function to demangle them properly (abi::__cxa_demangle()), and substitute the human-readable version of the symbol – e.g. “crashTest::divideByZero(int)“.
The last function – winExceptionHandler() – just converts the exception to a human-readable string, gathers the stack trace, and writes the log.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
#ifdef Q_OS_WIN QStringList _stackTrace( CONTEXT* context ) { HANDLE process = GetCurrentProcess(); HANDLE thread = GetCurrentThread(); SymInitialize( process, 0, true ); STACKFRAME64 stackFrame; memset( &stackFrame, 0, sizeof( STACKFRAME64 ) ); DWORD image; #ifdef _M_IX86 image = IMAGE_FILE_MACHINE_I386; stackFrame.AddrPC.Offset = context->Eip; stackFrame.AddrPC.Mode = AddrModeFlat; stackFrame.AddrStack.Offset = context->Esp; stackFrame.AddrStack.Mode = AddrModeFlat; stackFrame.AddrFrame.Offset = context->Ebp; stackFrame.AddrFrame.Mode = AddrModeFlat; #else // see http://theorangeduck.com/page/printing-stack-trace-mingw #error You need to define the stack frame layout for this architecture #endif QStringList frameList; int frameNumber = 0; while ( StackWalk64( image, process, thread, &stackFrame, context, NULL, SymFunctionTableAccess64, SymGetModuleBase64, NULL ) ) { QString locationStr = _addr2line( sProgramName, (void*)stackFrame.AddrPC.Offset ); // match the mangled name and demangle if we can QRegularExpressionMatch match = sSymbolMatching.match( locationStr ); const QString cSymbol( match.captured( 1 ) ); if ( !cSymbol.isNull() ) { int demangleStatus = 0; const char *cFunctionName = abi::__cxa_demangle( cSymbol.toLatin1().constData(), nullptr, nullptr, &demangleStatus); if ( demangleStatus == 0 ) { locationStr.replace( cSymbol, cFunctionName ); } } frameList += QStringLiteral( "[%1] 0x%2 %3" ) .arg( QString::number( frameNumber ) ) .arg( quintptr( (void*)stackFrame.AddrPC.Offset ), 16, 16, QChar( '0' ) ) .arg( locationStr ); ++frameNumber; } SymCleanup( GetCurrentProcess() ); return frameList; } LONG WINAPI _winExceptionHandler( EXCEPTION_POINTERS *inExceptionInfo ) { const QString cExceptionType = [] ( DWORD code ) { switch( code ) { // ... code to return descriptions of each case } } ( inExceptionInfo->ExceptionRecord->ExceptionCode ); // If this is a stack overflow then we can't walk the stack, so just show // where the error happened QStringList frameInfoList; if ( inExceptionInfo->ExceptionRecord->ExceptionCode == EXCEPTION_STACK_OVERFLOW ) { frameInfoList += _addr2line( sProgramName, (void*)inExceptionInfo->ContextRecord->Eip ); } else { frameInfoList += _stackTrace( inExceptionInfo->ContextRecord ); } _writeLog( cExceptionType, frameInfoList ); return EXCEPTION_EXECUTE_HANDLER; } #else ... |
macOS-Specific
The macOS-specific code is similar, but we can use the backtrace() and backtrace_symbols() functions to simplify things.
These functions get us part-way there. The output of these gives us something like:
|
1 |
4 asmCrashReportExample 0x0000000100008bf4 _ZN9crashTest9function2Ei + 36 |
With a regular expression, I’m matching the _ZN* symbol and sending that to _addr2Line() to look up the file and line number with atos. If it finds it, I’m just substituting the result to get:
|
1 |
4 asmCrashReportExample 0x0000000100008bd4 crashTest::function2(int) (in asmCrashReportExample) (main.cpp:26) |
The other two functions here – _posixSignalHandler() and _posixSetupSignalHandler() set up the necessary plumbing to capture the signal, convert it a human-readable form, gather the stack trace, and write the log.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
... #else constexpr int MAX_STACK_FRAMES = 64; static void *sStackTraces[MAX_STACK_FRAMES]; static uint8_t sAlternateStack[SIGSTKSZ]; QStringList _stackTrace() { int traceSize = backtrace( sStackTraces, MAX_STACK_FRAMES ); char **messages = backtrace_symbols( sStackTraces, traceSize ); // skip the first 2 stack frames (this function and our handler) and skip the last frame (always junk) QStringList frameList; int frameNumber = 0; frameList.reserve( traceSize ); for ( int i = 2; i < (traceSize - 1); ++i ) { QString message( messages[i] ); // match the mangled name if possible and replace with file & line number QRegularExpressionMatch match = sSymbolMatching.match( message ); const QString cSymbol( match.captured( 1 ) ); if ( !cSymbol.isNull() ) { QString locationStr = _addr2line( sProgramName, sStackTraces [i] ); if ( !locationStr.isEmpty() ) { int matchStart = match.capturedStart( 1 ); message.replace( matchStart, message.length() - matchStart, locationStr ); } } frameList += message; ++frameNumber; } if ( messages != nullptr ) { free( messages ); } return frameList; } // prtotype to prevent warning about not returning void _posixSignalHandler( int inSig, siginfo_t *inSigInfo, void *inContext ) __attribute__ ((noreturn)); void _posixSignalHandler( int inSig, siginfo_t *inSigInfo, void *inContext ) { Q_UNUSED( inContext ); const QString cSignalType = [] ( int sig, int inSignalCode ) { switch( sig ) { // ... code to return descriptions of each case } return QStringLiteral( "Unrecognized Signal" ); } ( inSig, inSigInfo->si_code ); const QStringList cFrameInfoList = _stackTrace(); _writeLog( cSignalType, cFrameInfoList ); _Exit(1); } void _posixSetupSignalHandler() { // setup alternate stack stack_t ss{ static_cast<void*>(sAlternateStack), SIGSTKSZ, 0 }; if ( sigaltstack( &ss, nullptr ) != 0 ) { err( 1, "sigaltstack" ); } // register our signal handlers struct sigaction sigAction; sigAction.sa_sigaction = _posixSignalHandler; sigemptyset( &sigAction.sa_mask ); #ifdef __APPLE__ // backtrace() doesn't work on macOS when we use an alternate stack sigAction.sa_flags = SA_SIGINFO; #else sigAction.sa_flags = SA_SIGINFO | SA_ONSTACK; #endif if ( sigaction( SIGSEGV, &sigAction, nullptr ) != 0 ) { err( 1, "sigaction" ); } if ( sigaction( SIGFPE, &sigAction, nullptr ) != 0 ) { err( 1, "sigaction" ); } if ( sigaction( SIGINT, &sigAction, nullptr ) != 0 ) { err( 1, "sigaction" ); } if ( sigaction( SIGILL, &sigAction, nullptr ) != 0 ) { err( 1, "sigaction" ); } if ( sigaction( SIGTERM, &sigAction, nullptr ) != 0 ) { err( 1, "sigaction" ); } if ( sigaction( SIGABRT, &sigAction, nullptr ) != 0 ) { err( 1, "sigaction" ); } } #endif |
setSignalHandler()
Finally, we have the main setup function. All this does is set some of the static variables and call the appropriate signal handler function.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
void setSignalHandler( const QString &inCrashReportDirPath, logWrittenCallback inLogWrittenCallback ) { sProgramName = QCoreApplication::arguments().at( 0 ); sCrashReportDirPath = inCrashReportDirPath; if ( sCrashReportDirPath.isEmpty() ) { sCrashReportDirPath = QStringLiteral( "%1/%2 Crash Logs" ).arg( QStandardPaths::writableLocation( QStandardPaths::DesktopLocation ), QCoreApplication::applicationName() ); } sSymbolMatching.optimize(); sLogWrittenCallback = inLogWrittenCallback; if ( sProcess == nullptr ) { sProcess = new QProcess; sProcess->setProcessChannelMode( QProcess::MergedChannels ); } #ifdef Q_OS_WIN SetUnhandledExceptionFilter( _winExceptionHandler ); #else _posixSetupSignalHandler(); #endif } } |
Final Notes
While what I have currently works for me, you might need to adapt it for your own use.
I’ve intentionally kept this code as simple as I can. Some of the possibilities for modification would be to include a working version for MSVC, include a Qt-based way to submit the crash log via a web server, expand the data included in the report to include machine information, etc..
If you make any changes, please feel free to submit pull requests on the github page.
I hope someone finds it useful!
hello Andy.
I use your crash report code in my qt project(mingw32 on windows)
but,when some crash in dll file,I can not get the infomation.
how to solve that?
Dean:
Would you mind creating an issue for this in the github repo? It will be easier to discuss there.
Thanks for the stack analyzer.
I integrated the code as mentioned. But, I’m getting the following info only in the stack trace. Could you please let me know if I’m missing something?
EXCEPTION_ACCESS_VIOLATION
0x000000007734e486 * Error running command
D:/tools/addr2line -f -p -e D:\release\test_app.exe 0x000000007734e486
Process failed to start: Access is denied.
We are unable to proceed at all. Any pointer should be helpful for us.
Thanks in advance.
Do you have the addr2line program in the right place? (see the readme) Does your program have the correct permissions to run it?
The paths don’t look quite right there – it should be looking next to the executable for the tools directory to find addr2line.
You should be able to see what’s happening using your debugger. It’s almost impossible to analyze and debug problems remotely & without the code.
Thanks for the reply.
The path could be an editing mistake.
This is the actual path. Please let me know if this is right.
0x000000007734e0e3 * Error running command
D:/mingwprj/release/tools/addr2line -f -p -e D:\mingwprj\release\st_dab_radio.exe 0x000000007734e0e3
Process failed to start: Access is denied.
When we made a deliberate divide by zero error in the code to cross verify, the stack trace was fine. So, I think it has right permissions to run.We don’t have any debugger other than Visual Studio debugger when the QT crash happens. It shows an exception in QTWidgets.dll.
Please provide your suggestion on the possible debugging options for this issue.
Thanks in advance.
That path looks correct.
If you are crashing in Qt, then it’s possible it’s finding the wrong Qt version (linked to one version and running with a different one). In my experience, crashes in Qt code itself are fairly rare.
Other than stepping through the code with the debugger to figure out what’s happening I don’t have any other suggestions.
I really can’t debug code by going back-and-forth in a comment section like this 🙂
Thanks.
Actually, this crash issue doesn’t happen when we run with a debugger. That’s why an external tool to get the stack trace was required.
The doubt was in how to interpret this error message “Process failed to start: Access is denied”.
Anyways, thanks!
You could try using the other version of QProcess::start() in asmCrashReport.cpp. Maybe the argument list included as part of the command isn’t being quoted properly?