Saturday, December 26, 2009

Bye bye data-loss, thanks to root on USB-flash

I still use my server, that is basically a self-made NAS additionally used for irc that's always online (via irssi and screen) large over-night downloads (so I don't have to keep my energy-hungry main pc turned on). I occasionally use it to test out some Linux-stuff (like testing C++-code on different versions of GCC), too.


The server houses 6 SATA drives that are all members of a software-RAID5 and one IDE hard-disk I keep the OS on (previously Debian). I did not want to install the OS on the RAID, because I wanted it to store data only. Since I had bad experience with HDDs in general and IDE-disks especially, this one IDE drive was always something I wanted to replace. Unfortunately, all SATA ports of the motherboard and the additional Adaptec controller are already in use, so I had to stick with the IDE port.


Recently I could not access this box anymore and when I hooked up a monitor to check out what's wrong with it, the console was filled with i/o errors on hda. "Now is the time to replace that last piece of possible failure", I said to myself. A reboot brought no failures, though, and till today I still think the disk is okay and it was a software-failure, but I still wanted to replace the drive with an USB-stick. I had read about booting from USB before and already had an USB-stick with an Ubuntu Live-CD on it. So I ordered another one to use as the OS's root drive. Well actually I used the stick I already had and replaced it's day-to-day usage with the new one, since it was smaller and looked nicer. I did not want to boot a Live-CD, but actually install Ubuntu on the stick, though. When I googled for "Ubuntu USB install" or similar keywords, I always ended up with a Live-CD or a read-only root, and even that took quite a tedious process. But then I stumbled upon this tutorial that made clear the obvious: You can simply put in your Ubuntu CD and your USB stick and install to it like you would to any other drive. Damn! It was so easy :-) The only caveat, as mentioned in the tutorial, is that on the last step of the installation you have to explicitly choose the USB-stick as the boot-device. By the way, it's freaking neat to be able to manually select the boot-device! And on another side-note, the new Ubuntu installer itself is pretty neat, too. It's easy, even not so tech-savvy people could use it without problems. So I burned Ubuntu 9.04 to a CD, booted from it on my main PC (since the server does not have a CD-ROM), plugged in the USB-stick and installed Ubuntu on it. Then I plugged it into the server, did some last fiddling with the settings, copied over the content of the IDE-disk to a backup-directory on the RAID, turned the system off again and unplugged the IDE-disk.


It all worked pretty well, except that the device's name of the USB-stick changed after unplugging the hard-drive. I only had to manually edit GRUB's boot-command (that is another feature that has saved me hours of fiddling already) and later edit it's default configuration. I just guessed that sdh1 has become sdg1, which wasn't too difficult. :-)


Finally! A completely IDE-less system! And additionally, I have minimized the risk of data-loss, because I'd suspect the chances of a head-crash on an USB-drive to be pretty low. Sure, flash-memory wears off after time, but I hope that time-span is much larger than the durability of a hard-disk.

Thursday, December 24, 2009

Qt-widget to visually select the corner of a screen (for popup-windows)

I recently had to create a popup-window that displays active tasks for a user. Because I am frequently annoyed by popups showing at exactly the wrong monitor or position, I included a feature to change the monitor and that monitor's corner that will be used as the popup-position. I'm using QDesktopWidget to find out the available monitors and calculate the correct position for the popup. In the first version of this popup-window, the user had to select the position and monitor using a standard menu. Because that's not very intuitive, I decided to create a widget that displays the available screens in a miniature-preview. The corners of the screens are highlighted and can be selected with the mouse. When the mouse hovers a corner of the preview, a small popup-like window is displayed at that corner's position of the real monitor.


Here are two screenshots:


Screenshot Popup Position Selector on Windows


Screenshot Popup Position Selector on Ubuntu Linux


You can view and download the source in my github-repository and you can download a sample-application for Windows here.

Friday, December 4, 2009

Firebird database-corruption and what you can do about it

There are usually only two reasons why a database can be corrupted in Firebird: 1) You copied the fdb-file while the Server was accessing it and 2) You don't have forced writes enabled and the server shut down unexpectedly (e.g. power-outage).


In these two cases you often get error-messages like "Internal gds software-consistency check", "Database file appears corrupt", "Wrong record length" or similar when trying to access certain tables or datasets. This is quite bad and means that your database is corrupted and that you have to try to recover it. Because recovery may fail and other steps like using IB Surgeon may be neccessary, it's a good advice to first stop the Firebird server, copy the fdb-file and start the server again. This copy can be used to start over using a different approach or send to a database-repair-service.


Database recovery is done in four steps. Here's how you do it:


Validating the database


The first step for recovery is letting gfix validate the database structure. This is done with the parameters -v -f. A call to gfix could look like this:

gfix -v -f -user sysdba -pass masterkey server:/path/to/database.fdb

gfix may then report errors in the database-structure. When it does not, your database is fine and recovery is not necessary.


Preparing for backup


The second step is preparing the database for a backup, using gfix again. The parameter is -mend in this case:

gfix -mend -user sysdba -pass masterkey server:/path/to/database.fdb


Creating a backup


The third step is a critical one. If this step fails, recovery using tools shipped with Firebird may not be possible. In this step we will create a backup of the fdb-file using gbak. gbak may fail to create the backup because unrecoverable errors exist in the database. Creating the backup is the same as always:

gbak -b -v -user sysdba -pass masterkey server:/path/to/database.fdb /path/to/backup.fbk

Note that paths of the database are remote (that means the path has to exist on the server), while the path to the backup-file is local (and has to exist on the machine you are executing gbak at). When you are lucky, gbak finishes this task without errors. You can then be certain that your data has been saved!


Restoring the backup


The fourth and last step is to restore the backup as usual, again with gbak:

gbak -c -r -v -user sysdba -pass masterkey /path/to/backup.fbk server:/path/to/database.fdb


Finished!


Now everything should be fine. But how can we prevent this from happening again? I initially mentioned two reasons for database-corruption to happen, which both can easily be avoided.


Never copy a database-file when the Firebird-server is accessing it.


Or better yet, never touch the file when the Firebird-server is running! That's the safest way to ensure that your database-file is copied uncorrupted and your data stays safe.


Enable forced writes unless you have very good reasons not to do so


Forced writes are a mechanism by the Firebird-server to tell the operating system to disable any disk-caching mechanisms for this file. When forced writes are not enabled, Firebird does not know what parts of the file are physically stored to the medium. That's one reason why having databases on a network-share is not possible and/or highly discouraged. Because disk-caches are not invented without a reason, this comes with a drawback. Writing will be a tad slower with forced writes disabled, so there might be a reason you don't want to enable it. When you disable forced writes, you have to ensure that the server is connected to an UPS and will always properly shut down. This means hitting the reset-button is a big no-no with forced writes disabled!


When following these rules, it should never be neccessary to recover a database again.

Thursday, October 8, 2009

Start application without elevated privileges on Windows Vista and Windows 7

I finally found out how you do this. You sometimes have applications that have something like "install" or "setup" in their names. Windows Vista and Windows 7 are stupid enough to enforce administrator privileges when starting these applications. But often this is absurd and breaks the applications. Let's say it's a command that's called "InstalScript" that installs a new script to your application's settings directory, which is in the user's APPDATA path. Someone uses your application in a company that has a Windows Domain in-place and his user-account does not have administrator privileges, which is usually the case. This user would now be unable to use your script, because he can't run the InstallScript application.


Since the release of Vista I was always wondering how you could prevent this, but never took the time to google for a solution. At some point, I even assumed it was impossible. Well, it nearly is ;-) Since you are reading this article the chances are high you are developing for Windows. And the chances that you stumbled upon manifest-files are high in this case, too. If you ask me, .manifest is short for "Manifestation Of The Royal Pain In The Ass". But heck, this article is not about ranting against manifest-files. In this case, they are here to help you. You can specify which kind of privileges your application needs in your manifest-file, and when you set "requestedExecutionPrivileges" to "asInvoker", you are free to go! That's the information I found on many websites. Unfortunately, little of them mentioned where this actually should go, and because I'm no meanie, here's an example I took from MSDN:


<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0"> 
  <assemblyIdentity version="1.0.0.0"
     processorArchitecture="X86"
     name="ShrinkyInstaler"
     type="win32"/> 
  <description>Installs Shrinky on your computer</description> 
  <!-- Identify the application security requirements. -->
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel
          level="asInvoker"
          uiAccess="false"/>
        </requestedPrivileges>
       </security>
  </trustInfo>
</assembly>


Now I can finally call Shrinky's Installer "ShrinkyInstaller.exe" and still debug it! (I put the manifest-file into the debug directory and named it "ShrinkyInstaller.exe.manifest). Yay!

Sunday, October 4, 2009

Shrinky Pre-Release

I had this idea in mind for a long time now. People are often asking me how they can resize images to a smaller file-size, most often because they want to send them via e-mail and don't want to wait 30 minutes for the e-mail to be sent. You can use Microsoft Paint which ships with every Windows for this and it's actually relatively easy to explain how. But most people can't remember the procedure and are lost when they have to do this the next time. So I came up with the idea to write an extremely easy application that can do this and is optimized for high usability and an easy interface. And here's my shot at it: Shrinky (doesn't that sound cute?).


Shrinky pre-release announcement


The only thing you have to do when loading Shrinky is clicking on the logo above, select the image-file you want to shrink and then hit "Shrink it!". This will create a file that's about 150-200kb in size, which is quite handy for uploading and sending via e-mail. A new file with the same file-name as the original, with a "-small" appended, will be created. You can tell Shrinky how to resize the image by clicking on "I want to decide myself".


Shrinky settings


You can shrink images in three different ways:

  • First, you can enter a file-size you want to have. This can be anything from "100kb" to "1,5 mb" (this is currently the only thing that does not work reliably, because floating-point values are not handled correctly). Shrinky will then try to create a file that matches this file-size as good as possible.

  • The second way to shrink a file is to enter the width and height directly. This can become handy when resizing an image so it fits in a certain space, such as on a webpage or something.

  • And the third way is by entering a percentage of the original resolution. Good for situations where you want to, for example, make an image half as big as it currently is.

When you are done setting up Shrinky the way you want, just hit "Shrink it".


Shrinky is completely written in C++ with Qt and is platform-independent. But obviously, since it's aimed at pc-beginners, the most important platform is Windows. That's why I've created binaries for Windows only (I don't have access to a Mac, anyways). See below for a download-link. You can access the source-code via the public git repository on github. Shrinky's currently translated to English and German only. Japanese will surely follow and some nice folks in #qt on freenode already offered translations, too. Man, open source is really nice. ;-)


There's still much to do to make Shrinky a polished product and publicly release it. Here's my todo-list for this project:

  • Shell-integration for Windows. Easily shrink images via right-click. That's especially important in the file-selection dialogs. With a tight integration, you can easily shrink images while selecting them in your e-mail application.

  • I need to write an installer for Windows, because extracting a ZIP-archive is not that easy for the target audience I am aiming at.

  • Processing of multiple files would be neat, too.

  • A web-page including a donation-facilty via paypal. Albeit being open-source, I'd like to make a few bucks with this application.


You can download the pre-release of Shrinky here. Just extract the zip-archive anywhere on your hard-drive (or usb-stick, for that matter) and run Shrinky.exe. You can post anything regarding your experience with Shrinky as a comment on this blog-post!

Wednesday, September 30, 2009

Setting the mouse-cursor for Qt's item-views via model

I like Qt's Model/View architecture very much. Writing models is, after a few short headaches in the beginning, quite intuitive and straight-forward. And it becomes even easier when you don't use QAbstractItemView directly, but a more specialized class like QAbstractListModel or QAbstractTableModel. The basic idea is that the model provides information to the views that is needed to present the individual items to the user. This does not only include the text of the item, but also a decoration (icon), a tool-tip, font, background-styling and more.


One thing I found missing, however, is the mouse-cursor that should be used for the item. For me, it's a common situation that you display a table and one of the columns is clickable and executes an action (such as deleting the row maybe). The user obviously will not understand that he can click cells in this column, unless you change the cursor to something like Qt::PointingHandCursor, for example.


I guess most people that faced this problem derived from the view they are using and override mouseMoveEvent() and do things there. This solution is not in the mind of Model/View, though. The mouse-cursor for an item is something that belongs into the model. So here's what we are going to do:


  • Create a new Role for the MyMouseCursorRole

  • Create new subclasses from the item-views we want to use (Qt's only item-views are QListView, QTableView and QTreeView)

  • In those sub-classes, make sure that the model's data() is called with the MyMouseCursorRole when the mouse moves to a new item (or moves to a position that is not an item)


After this, we can start writing models and handle the MyMouseCursorRole there. In the example, I derive from QStandardItemModel and only override data() and return different shapes for certain items.


In this blog-post, I'm including a small example for QListView. You can get the full source-code for the other views at the bottom of this post.

Here's the header:


#include <QListView>

// Define our new role that can be used in the model.
const int MyMouseCursorRole = Qt::UserRole + 1;

// Define a new constant for user-defined roles in this application.
const int MyUserRole = MyMouseCursorRole + 1;

// MyListView requests the MyMouseCursorRole from the model when the mouse
// moves over a new row. It does not need to use any tricks like MyTableView
// or MyTreeView, because it does not contain headers.

class MyListView: public QListView
{
Q_OBJECT

public:
MyListView(QWidget *parent = 0);

protected:
virtual void mouseMoveEvent(QMouseEvent *event);

private:
// m_LastRow stores the row the mouse was over the last time
// mouseMoveEvent() was called. This is used to minimize calls to the
// model's data() function. m_lastRow is -1 when no valid row was hovered by
// the mouse.
int m_lastRow;
};

And here's the implementation:
MyListView::MyListView(QWidget *parent):
QListView(parent),
m_lastRow(-1)
{
// We need to enable mouse-tracking because we need to know
// about every mouse-movement.
setMouseTracking(true);
}

void MyListView::mouseMoveEvent(QMouseEvent *event)
{
QAbstractItemModel *m(model());
// Only do something when a model is set.
if (m)
{
QModelIndex index = indexAt(event->pos());
if (index.isValid())
{
// When the index is valid, compare it to the last row.
// Only do something when the the mouse has moved to a new row.
if (index.row() != m_lastRow)
{
m_lastRow = index.row();
// Request the data for the MyMouseCursorRole.
QVariant data = m->data(index, MyMouseCursorRole);

Qt::CursorShape shape = Qt::ArrowCursor;
if (!data.isNull())
shape = static_cast(data.toInt());
setCursor(shape);
}
}
else
{
if (m_lastRow != -1)
// Set he mouse-cursor to the default when it isn't already.
setCursor(Qt::ArrowCursor);
m_lastRow = -1;
}
}
QListView::mouseMoveEvent(event);
}

For QListView, that's all! It gets more complex for QTableView and QTreeView, because they contain headers and mouseMoveEvent() is not called when the mouse moves to these headers. But the cursor set with setCursor() is used for the headers, too. This means that, when the mouse moves from an item with a special cursor-shape directly to a header-control, the mouse-shape does not change back to the default. To circumvent this, we have to implement quite a hack, so I'm not covering it in this blog-post. You can see my solution in the example source-code.


To test our new functionality, we have to create a model that handles MyMouseCursorRole and returns a valid Qt::CursorShape. As noted earlier, I derive from QStandardItemModel, because QStandardItemModel already implements needed standard functionality. QStandardItemModel uses QStandardItems as it's soure of information. Because QStandardItem obviously does not have a property for the cursor's shape, I use a trick to store the cursor-information in the item's text. By convention, when the item's text starts with {hand}, Qt::PointingHandCursor is returned in MyMouseCursorRole and {hand} is snipped off of the text for Qt::DisplayRole.


Here's the header's code, it's quite simple:

class MyTestModel: public QStandardItemModel
{
Q_OBJECT

public:
QVariant data(const QModelIndex &index, int role) const;
};

And here's the implementation's code:
QVariant MyTestModel::data(const QModelIndex &index, int role) const
{
if (role == MyMouseCursorRole)
{
// We use a little hack here so we don't need to add an extra element to
// QStandardItem. When the item's text starts with {hand}, we return
// the Qt::PointingHandCursor cursor-shape.

QVariant text = QStandardItemModel::data(index, Qt::DisplayRole);
if (text.toString().startsWith("{hand}"))
return Qt::PointingHandCursor;
else
return QVariant();
}
else if (role == Qt::DisplayRole)
{
// Cut out the {hand} from actually displayed content.
QVariant result = QStandardItemModel::data(index, role);
QString text = result.toString();
if (text.startsWith("{hand}"))
return text.mid(6);
else
return result;
}
return QStandardItemModel::data(index, role);
}

As you can see, I'm using a trick to call QStandardItemModel::data() because I want to access the "raw" data without re-implementing QStandardItemModel::data()'s functionality. What's quite handy is that you can pass a different Qt::ItemDataRole to data(), so I can get the value for Qt::DisplayRole when preparing the values for MyMouseCursorRole.


Now the only component that's missing to see the result is a demo-application that uses the test-model and the new MyListView-class. That's only a few lines:


#include <QApplication>

int main(int argc, char *argv[])
{
QApplication app(argc, argv);

MyTestModel model;

// Demo data
QList row;
row.push_back(new QStandardItem("A 1"));
row.push_back(new QStandardItem("{hand}B 1"));
row.push_back(new QStandardItem("{hand}C 1"));
model.appendRow(row);

row.clear();
row.push_back(new QStandardItem("{hand}A 1.1"));
row.push_back(new QStandardItem("B 1.2"));
row.push_back(new QStandardItem("{hand}C 1.3"));
model.item(0, 0)->appendRow(row);

row.clear();
row.push_back(new QStandardItem("{hand}A 2"));
row.push_back(new QStandardItem("B 2"));
row.push_back(new QStandardItem("C 2"));
model.appendRow(row);

MyListView listView;
listView.setModel(&model);
listView.show();
app.exec();
}

That's it. I have compiled a complete project that includes derivations from QListView, QTableView and QTreeView, the test-model and the test-widget. However, this is not a complete solution and by far not suitable to, say, be included in Qt's main releases. There are a few things that need to be solved. For example, using setCursor() to set the cursor for the whole widget is completely broken with these changes.


You can download the example source-code here.

Tuesday, September 29, 2009

Exceptions: Do's and Don'ts

Exceptions in C++ are a very complex topic. In addition to the standard-facilities of throwing and catching them, there are things like exception-specifiers and throwing arbitrary objects. Here are some guidelines on what you should do with exceptions and what you definitely should not do.

Do's



  • Use exceptions. But only for exceptional situations. Don't use exceptions for error-conditions that are likely to occur, like misconfigurations by the end-user, etc. However, when the configuration is already supplied by the software and not influenced by the user, it's indeed an exceptional situation where you should throw a C++-exception.

  • Derive all your exception-classes from a standard exception-class such as std::runtime_error or at least std::exception.

  • Create your own base-classes for exceptions in your application or library so you can easily distinguish your exceptions from others'. This is especially important for libraries.

  • Don't create exceptions on the heap. Throw them by value and catch them by value or const-reference.

  • Make your exception's error-information accessible as atomic as possible. This helps the user of your exception to create meaningful error messages for different kinds of end-users. For example, when there's a database-error, make the database-function that was executed, the line and part of the SQL-statement and the error-message accessible separately.

  • Also provide a convenience-function in your exception to create a human-readable error message.

  • The more different exception-classes you provide, the easier it is to handle certain types of exceptions. On the plus side, more exception-classes does not mean more work when you don't care what kind of exception is thrown when following the guideline to have a standard base-class for your application's exceptions.


Don'ts



  • Don't throw anything that does not derive from std::exception.

  • Try not to depend on catch(...) because you can't identify what kind of error has occured. std::exception should be the "widest" scope you catch.

  • Don't use exception-specifiers. The C++ committee has done a half-assed job when inventing those and didn't correct their mistakes in C++0x. There are several reasons why exception-specifiers are broken by design:

    • If any of the called functions throws an exception that's not listed in your exception-specifier, std::terminate will be called. This may happen when modifying a function that's used by any function that uses exception-specifiers, without verifying the whole code after the change.

    • When you use exception-specifiers in functions that call third-party code, it's possible that they break with an update of that library. This update could even silently happen by exchanging a dynamic library that's installed system-wide, without you doing anything.

    • Execption-specifiers always cost speed. Because they are not checked at compile-time, the compiler needs to add code to check for validity of the thrown exceptions, which slows down execution time even when not caring for exceptions at this point.



  • Don't throw exceptions from extern "C" functions. This causes undefined behaviour, and in this case it means a crash on most popular platforms.

  • Don't throw exceptions in destructors. Since destructors usually are called implicitly when the object runs out of scope, they're easy to miss. Additionally, it's not a good idea to prevent your application from freeing up resources.

  • Create your exception-classes exception-safe. For example, using std::string can potentially lead to a std::bad_alloc exception being thrown.