Bye bye data-loss, thanks to root on USB-flash

I still use my server, that is basically a self-made NAS additionally used for irc that's always online (via irssi and screen) large over-night downloads (so I don't have to keep my energy-hungry main pc turned on). I occasionally use it to test out some Linux-stuff (like testing C++-code on different versions of GCC), too.

The server houses 6 SATA drives that are all members of a software-RAID5 and one IDE hard-disk I keep the OS on (previously Debian). I did not want to install the OS on the RAID, because I wanted it to store data only. Since I had bad experience with HDDs in general and IDE-disks especially, this one IDE drive was always something I wanted to replace. Unfortunately, all SATA ports of the motherboard and the additional Adaptec controller are already in use, so I had to stick with the IDE port.

Recently I could not access this box anymore and when I hooked up a monitor to check out what's wrong with it, the console was filled with i/o errors on hda. "Now is the time to replace that last piece of possible failure", I said to myself. A reboot brought no failures, though, and till today I still think the disk is okay and it was a software-failure, but I still wanted to replace the drive with an USB-stick. I had read about booting from USB before and already had an USB-stick with an Ubuntu Live-CD on it. So I ordered another one to use as the OS's root drive. Well actually I used the stick I already had and replaced it's day-to-day usage with the new one, since it was smaller and looked nicer. I did not want to boot a Live-CD, but actually install Ubuntu on the stick, though. When I googled for "Ubuntu USB install" or similar keywords, I always ended up with a Live-CD or a read-only root, and even that took quite a tedious process. But then I stumbled upon this tutorial that made clear the obvious: You can simply put in your Ubuntu CD and your USB stick and install to it like you would to any other drive. Damn! It was so easy :-) The only caveat, as mentioned in the tutorial, is that on the last step of the installation you have to explicitly choose the USB-stick as the boot-device. By the way, it's freaking neat to be able to manually select the boot-device! And on another side-note, the new Ubuntu installer itself is pretty neat, too. It's easy, even not so tech-savvy people could use it without problems. So I burned Ubuntu 9.04 to a CD, booted from it on my main PC (since the server does not have a CD-ROM), plugged in the USB-stick and installed Ubuntu on it. Then I plugged it into the server, did some last fiddling with the settings, copied over the content of the IDE-disk to a backup-directory on the RAID, turned the system off again and unplugged the IDE-disk.

It all worked pretty well, except that the device's name of the USB-stick changed after unplugging the hard-drive. I only had to manually edit GRUB's boot-command (that is another feature that has saved me hours of fiddling already) and later edit it's default configuration. I just guessed that sdh1 has become sdg1, which wasn't too difficult. :-)

Finally! A completely IDE-less system! And additionally, I have minimized the risk of data-loss, because I'd suspect the chances of a head-crash on an USB-drive to be pretty low. Sure, flash-memory wears off after time, but I hope that time-span is much larger than the durability of a hard-disk.

Qt-widget to visually select the corner of a screen (for popup-windows)

I recently had to create a popup-window that displays active tasks for a user. Because I am frequently annoyed by popups showing at exactly the wrong monitor or position, I included a feature to change the monitor and that monitor's corner that will be used as the popup-position. I'm using QDesktopWidget to find out the available monitors and calculate the correct position for the popup. In the first version of this popup-window, the user had to select the position and monitor using a standard menu. Because that's not very intuitive, I decided to create a widget that displays the available screens in a miniature-preview. The corners of the screens are highlighted and can be selected with the mouse. When the mouse hovers a corner of the preview, a small popup-like window is displayed at that corner's position of the real monitor.

Here are two screenshots:

You can view and download the source in my github-repository and you can download a sample-application for Windows here.

Firebird database-corruption and what you can do about it

There are usually only two reasons why a database can be corrupted in Firebird: 1) You copied the fdb-file while the Server was accessing it and 2) You don't have forced writes enabled and the server shut down unexpectedly (e.g. power-outage).

In these two cases you often get error-messages like "Internal gds software-consistency check", "Database file appears corrupt", "Wrong record length" or similar when trying to access certain tables or datasets. This is quite bad and means that your database is corrupted and that you have to try to recover it. Because recovery may fail and other steps like using IB Surgeon may be neccessary, it's a good advice to first stop the Firebird server, copy the fdb-file and start the server again. This copy can be used to start over using a different approach or send to a database-repair-service.

Database recovery is done in four steps. Here's how you do it:

Validating the database

The first step for recovery is letting gfix validate the database structure. This is done with the parameters -v -f. A call to gfix could look like this:

gfix -v -f -user sysdba -pass masterkey server:/path/to/database.fdb

gfix may then report errors in the database-structure. When it does not, your database is fine and recovery is not necessary.

Preparing for backup

The second step is preparing the database for a backup, using gfix again. The parameter is -mend in this case:

gfix -mend -user sysdba -pass masterkey server:/path/to/database.fdb

Creating a backup

The third step is a critical one. If this step fails, recovery using tools shipped with Firebird may not be possible. In this step we will create a backup of the fdb-file using gbak. gbak may fail to create the backup because unrecoverable errors exist in the database. Creating the backup is the same as always:

gbak -b -v -user sysdba -pass masterkey server:/path/to/database.fdb /path/to/backup.fbk

Note that paths of the database are remote (that means the path has to exist on the server), while the path to the backup-file is local (and has to exist on the machine you are executing gbak at). When you are lucky, gbak finishes this task without errors. You can then be certain that your data has been saved!

Restoring the backup

The fourth and last step is to restore the backup as usual, again with gbak:

gbak -c -r -v -user sysdba -pass masterkey /path/to/backup.fbk server:/path/to/database.fdb

Finished!

Now everything should be fine. But how can we prevent this from happening again? I initially mentioned two reasons for database-corruption to happen, which both can easily be avoided.

Never copy a database-file when the Firebird-server is accessing it.

Or better yet, never touch the file when the Firebird-server is running! That's the safest way to ensure that your database-file is copied uncorrupted and your data stays safe.

Enable forced writes unless you have very good reasons not to do so

Forced writes are a mechanism by the Firebird-server to tell the operating system to disable any disk-caching mechanisms for this file. When forced writes are not enabled, Firebird does not know what parts of the file are physically stored to the medium. That's one reason why having databases on a network-share is not possible and/or highly discouraged. Because disk-caches are not invented without a reason, this comes with a drawback. Writing will be a tad slower with forced writes disabled, so there might be a reason you don't want to enable it. When you disable forced writes, you have to ensure that the server is connected to an UPS and will always properly shut down. This means hitting the reset-button is a big no-no with forced writes disabled!

When following these rules, it should never be neccessary to recover a database again.