QTextDocument, HTML, and Unicode: It’s All Greek To Me

To generate reports in my Qt-based software, I create a QTextDocument, which may contain some Unicode characters, convert it to HTML, and then save it to a file.

QFile file( fileName );

if ( !file.open( QIODevice::WriteOnly | QIODevice::Text ) )
   return false;

QTextStream out( &file );
out << mDoc.toHtml();

QFile file( fileName );

if ( !file.open( QIODevice::WriteOnly | QIODevice::Text ) )

return false;

QTextStream out( &file );

out << mDoc.toHtml();

But the Unicode characters were not being being displayed properly when I opened it in a browser:

Unicode Not Displayed Properly

Taking a look at the generated HTML, I noticed that the content encoding was not being set:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html><head><meta name="qrichtext" content="1" />

1 2	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd"> <html><head><meta name="qrichtext" content="1" />

So I thought I could simply specify the encoding in the call to toHtml() which would set the encoding in the HTML header.

QFile file( fileName );

if ( !file.open( QIODevice::WriteOnly | QIODevice::Text ) )
   return false;

QTextStream out( &file );
out << mDoc.toHtml( "utf-8" );

QFile file( fileName );

if ( !file.open( QIODevice::WriteOnly | QIODevice::Text ) )

return false;

QTextStream out( &file );

out << mDoc.toHtml( "utf-8" );

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html><head><meta name="qrichtext" content="1" /><meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

1 2	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd"> <html><head><meta name="qrichtext" content="1" /><meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

That looks better! But wait…

Unicode Still Not Displayed Properly

Uhhh… That looks worse.

After scouring the interwebs, I eventually found the answer. The QTextStream encodes based on the system locale so if you don’t set it explicitly you might not get what you expect. According to the QTextStream docs:

By default, QTextCodec::codecForLocale() is used, and automatic unicode detection is enabled.

For some reason the automatic detection did not work for my case, but the solution is to set the encoding manually using QTextStream::setCodec() like this:

QFile file( fileName );

if ( !file.open( QIODevice::WriteOnly | QIODevice::Text ) )
   return false;

QTextStream out( &file );

out.setCodec( "utf-8" );

out << mDoc.toHtml( "utf-8" );

QFile file( fileName );

if ( !file.open( QIODevice::WriteOnly | QIODevice::Text ) )

return false;

QTextStream out( &file );

out.setCodec( "utf-8" );

out << mDoc.toHtml( "utf-8" );

Aha! Now my Greek looked Greek!

Unicode Displayed Properly

Torque Console Messages and Static Initialization

I figured I’d write this up for the one other person in the world that is going to run into this. Might save someone some time.

Several years ago I used to work at GarageGames on the Torque game engine. I worked on an internal version of the engine that was used for several of our games and by a few third-party games. While this version of the engine was never released, a whole bunch of our work ended up in TGEA at some point. So what I’m describing applies to at least some versions of Torque Game Engine Advanced (TGEA) as well [I have 1.8.2]. It may apply to the current iteration of the Torque engine – Torque 3D – but I don’t have it so I don’t know for sure.

Recently I was reviving a game that was written using an early version of this internal engine. I was updating the FMOD library and ran into a loading problem. While tracking it down, I found myself in sfx/fmod/SFXFMODProvider.cpp where the SFXFMODProvider constructor had a bunch of calls to Con::warnf(), but I wasn’t seeing anything in the console log. The problem was that the console log hadn’t been created yet because the SFXFMODProvider constructor is called during static initialization.

Taking a look at console/console.cpp, Con::printf(), Con::warnf(), and Con::errorf() ultimately call a static function _printf() which starts like this:

static void _printf(ConsoleLogEntry::Level level, ConsoleLogEntry::Type type, const char* fmt, va_list argptr)
{
    if (!active)
        return;
    Con::active = false; 

    char buffer[4096];
    U32 offset = 0;
  ...

static void _printf(ConsoleLogEntry::Level level, ConsoleLogEntry::Type type, const char* fmt, va_list argptr)

{

if (!active)

return;

Con::active = false;

char buffer[4096];

U32 offset = 0;

...

Guess that’s why I wasn’t seeing anything – we’re just throwing away all messages if we aren’t active yet!

So the obvious way to fix this is simply to save our messages instead of throwing them away and then to output them when we do become active. Easy as pie.

static void _printf(ConsoleLogEntry::Level level, ConsoleLogEntry::Type type, const char* fmt, va_list argptr)
{
// [ASM]
#ifdef TORQUE_DEBUG
   static Vector<char *>   preConsoleMessages;

   // IF we aren't active with a log yet
   //    THEN save our message so we can output when we become active
   if ( !active || !consoleLogMode )
   {
      char buffer[4096];

      dVsprintf(buffer, sizeof( buffer ), fmt, argptr);

      preConsoleMessages.push_back( dStrdup( buffer ) );
      return;
   }
   else if ( active && consoleLogMode && preConsoleMessages.size() > 0 )
   {
      // We have some messages saved up, so output them to the log file

      for ( int i = 0; i < preConsoleMessages.size(); ++i )
      {
         log( preConsoleMessages[i] );

         dFree( static_cast<void *>(preConsoleMessages[i]) );
      }
         
      preConsoleMessages.clear();
   }
#else
   if (!active)
      return;
#endif

   Con::active = false; 

   char buffer[4096];
   U32 offset = 0;
 ...

static void _printf(ConsoleLogEntry::Level level, ConsoleLogEntry::Type type, const char* fmt, va_list argptr)

{

// [ASM]

#ifdef TORQUE_DEBUG

static Vector<char *> preConsoleMessages;

// IF we aren't active with a log yet

// THEN save our message so we can output when we become active

if ( !active || !consoleLogMode )

{

char buffer[4096];

dVsprintf(buffer, sizeof( buffer ), fmt, argptr);

preConsoleMessages.push_back( dStrdup( buffer ) );

return;

}

else if ( active && consoleLogMode && preConsoleMessages.size() > 0 )

{

// We have some messages saved up, so output them to the log file

for ( int i = 0; i < preConsoleMessages.size(); ++i )

{

log( preConsoleMessages[i] );

dFree( static_cast<void *>(preConsoleMessages[i]) );

}

preConsoleMessages.clear();

}

#else

if (!active)

return;

#endif

Con::active = false;

char buffer[4096];

U32 offset = 0;

...

Now any messages that are sent using Con::printf(), Con::warnf(), or Con::errorf() during static initialization or before the console is active will show up at the top of console.log.

Hope the one guy who finds this useful in the future buys me a beer…

In-memory ZIP File Access Using Qt

A while ago I was working on a little tool to manage my media files – TV shows and movies that I’d ripped from my DVD collection that I wanted to make available through XBMC on my home theater system. To display things like plot, characters, runtime, etc., XBMC reads NFO files – which are just XML files – one for each media file. Entering this information by hand is not a fun afternoon activity, so I looked for a way to automate it. I found a couple of different sites that provided the data I needed via APIs and eventually settled on TheTVDB.com for TV series and themoviedb.org for movies. So all I needed to do was read the data using the APIs and write it to the NFO files. Simple!

For each TV series, TheTVDB provides a ZIP file containing several XML files which splits up the information to make it more manageable and allows for easier internationalization. A typical ZIP from TheTVDB contains files such as actors.xml, banners.xml, and en.xml. The actors.xml file contains a list of actors, the roles they play, a link to an image, etc.. banners.xml provides links to fan art and thumbnails which XBMC can use for display. en.xml provides the core data: a description of the TV series and a list of all the episodes from the series including plot, runtime, air date, etc. for each of them.

So what’s the best way to download and access this zipped data? Continue reading →

QString::toStdString, QString::fromStdString, and -no-stl

I use the Qt library in my day-to-day development work. I build my Qt libraries directly from the git repository and they are configured specifically for my projects. On the Mac OS X side, this is the command line I use:

./configure -no-qt3support -no-qdbus -no-nis -no-iconv -no-stl -no-accessibility \
            -no-audio-backend -no-svg -no-webkit -no-javascript-jit \
            -arch x86 -platform macx-llvm -opensource -nomake demos -nomake examples

./configure -no-qt3support -no-qdbus -no-nis -no-iconv -no-stl -no-accessibility \

-no-audio-backend -no-svg -no-webkit -no-javascript-jit \

-arch x86 -platform macx-llvm -opensource -nomake demos -nomake examples

The Windows one is similar:

.\configure -no-qt3support -no-qdbus -no-stl -no-accessibility \
            -no-incredibuild-xge -no-plugin-manifests -no-audio-backend -no-webkit \
            -platform win32-msvc2005 -opensource -nomake demos -nomake examples

.\configure -no-qt3support -no-qdbus -no-stl -no-accessibility \

-no-incredibuild-xge -no-plugin-manifests -no-audio-backend -no-webkit \

-platform win32-msvc2005 -opensource -nomake demos -nomake examples

You will notice that I am explicitly removing a whole bunch of stuff – SVG, WebKit, audio, etc.. The time it takes to build the Qt libraries is not insignificant and since I typically track the git repo, I don’t want to waste time building things I will not be using in my projects. The WebKit code takes an extraordinary amount of time to build, for example. I also don’t want to build things into the libs that I’m not going to be using for my commercial software – such as STL. One of the problems this poses, however, is how to handle build problems with other projects I want to build, for example the GUI for the open-source Cppcheck program.

For the most part the projects I’m interested in don’t use any of the capabilities I’m eliminating from my Qt build, but one option has been problematic for a couple of projects: -no-stl. This is because the projects use the functions QString::toStdString() and QString::fromStdString() which do not exist when you build Qt with the -no-stl option. While STL may be available to the source you are building, it was not compiled into the Qt libs, so this causes an error.

When I try to build Cppcheck using my own Qt build, I get errors like this:

../gui/mainwindow.cpp: In member function ‘Settings MainWindow::GetCppcheckSettings()’: ../gui/mainwindow.cpp:457: error: ‘class QString’ has no member named ‘toStdString’

../gui/threadresult.cpp: In member function ‘virtual void ThreadResult::reportOut(const std::string&)’: ../gui/threadresult.cpp:42: error: ‘fromStdString’ is not a member of ‘QString’

Solutions?

Continue reading →

COLLADA Files and SketchUp Component Hierarchies

One of the formats my bloodstain pattern analysis software will export data to is COLLADA. COLLADA is an open format used to exchange 3D scene data between software packages. It is an XML format, meaning it’s readable in a text editor, but it is extremely verbose and not a simple thing to understand unless you know what you’re looking at [and even then…]. Before using it for my current work, I’d used COLLADA with an internal version of the Torque game engine at GarageGames, so I had some familiarity with the format before using it this time.

SketchUp is a powerful free modelling tool from Google. As with any software importing COLLADA, some of the concepts in the format map to concepts within the software, while others do not. One of the things that was not obvious to me when I started exporting COLLADA files for use in Google SketchUp was how to set up my COLLADA so it had hierarchical components upon import. My scenes always showed up as one monolithic component which meant that modifying parts of it to add textures or colours was not easy.

For the bloodstain analysis work, I only have to deal with simple scenes consisting of planar surfaces and lines.

SketchUp COLLADA Import Example

Continue reading →

Running jpegtran From a Qt Application

Continuing on with my obsession with smaller images, I thought I’d give an example of how to include jpegtran with your Qt app and how to call it using QProcess to optimize jpegs. Why do this? The use-case I had was that I was downloading images from the net from within my application and I wanted to ensure that they were optimized. I could have read them in and used my modifications to QImageWriter & co. to write them back out, but we are dealing with JPEG which is a lossy format, so I wanted a lossless way to optimize the images.

Continue reading →

qmake and the Info.plist File

I use the Qt library and qmake to build my bloodstain pattern analysis software. qmake is a great tool for cross-platform development because it lets you use one relatively succinct description [a .pro file] to generate Xcode projects, MSVC projects, and makefiles. On the Mac OS X side of things, I use one qmake .pro file to generate three xcode projects: release, beta, and demo. My directory structure looks like this:

Example File Structure

Overall this works well, but there’s an issue with the Info.plist file.

Continue reading →