How to back up a complete hard disk to compact size?

This is my current (2011-09) technique to create a compressed image of a complete hard disk and to restore it later. It works both for disks and partitions wth an operating system and with just data on it (it does not care at all what is on the disk). You should be able to restore it to any hard disk of at least the same size.

Technique using ntfsclone

ntfsclone has the advantage that it can supply zeroes for unused sectors on the fly, eliminating one step of the dd based process (see below).

1. Start from a Linux live CD, such as Ubuntu, and execute this command (naming your hard disk to backup, here /dev/sda is used):

   sudo ntfsclone -o - /dev/sda | bzip2 -c > Backup.yyyy-mm-dd.sda.raw.bz2

2. Note that you can also back up just individual partitions. But then, you should back up the MBR and boot loader separately, and the partition table also in text format. For that, see:
   • Transfer Windows to New Hard Disk with ntfsclone
   • Notes on backing up entire hard disks or partitions
3. For uncompressing later, use a command like this:

   bunzip2 -c Backup.yyyy-mm-dd.sda.raw.bz2 | ntfsclone --overwrite /dev/sda -

Technique using dd and gzip

This involves an additional step for manually zero-ing unused disk space. Also, using gzip instead of bzip2 yields less effective compression, but is faster. You can however supply bzip2, bzcat and bunzip2 for every call to gzip, zcat or gunzip; their parameter usage is compatible for our purposes.

1. To achieve good compression ratios, it is advisable to fill unused space of the disk to back up with zeroes (or ones or any other kind of redundancy). Please do not do that when trying to rescue defective disks and partitions, as you would lose potentially valuable data in unallocated space on your original disk. There are several techniques for that:
• In any kind of file system, when using Linux, you could create a file that is as large as possible, consisting only of zeroes, and after that delete it again:
  dd if=/dev/zero of=fillfile.raw bs=8M; rm fillfile.raw;
• If it’s about a NTFS partition and you can use that in Windows, you could use Microsoft’s SDelete. However, when trying it out in 2008 by using “sdelete -c -p 1 c”, it gave me this error message: “Cannot clean free space for UNC drive.”.
• Again, if it’s about an NTFS partition in Windows, you can use Eraser. In version 5, you could configure an own pattern (just zeroes) to overwrite unused disk space. In version 6.0.8 at least, this is no longer immediately possible. The developer advises to use the “HMG IS5 Baseline” for that, as it would do just that task of overwriting once with zeroes [source]; but in my test, this failed (no meaningful compression ratio afterwards). You can still create an own erasure method according to page 15 of the user manual though [source]. Also note, to be able to erase unused disk space when running it in Windows Vista, you need to run Eraser as Administrator user. For that, close all Eraser instances (no Eraser tray icon must be left!), right-click the Eraser desktop symbol and click “Run as Administrator …” [source].
2. Now connect the hard disk to back up as a second or external USB hard disk to a Linux system, boot its installed Linux system, or boot into a Linux Live CD such as aUbuntu.
3. To back up the hard disk to a gzip-compressed image, use this command (naming your hard disk, here /dev/sda):

   dd if=/dev/sda | gzip -c > Backup.yyyy-mm-dd.sda.dd.gz

   On a 1.3 GHz Pentium 4 processor, this produced a throughput of between 3 MiB/s of hard disk raw data (at the start of the hard disk where compression was difficult) and 10 MiB/s (at the end of the hard disk where the all-zero area was located).

4. Using “kill -USR1 <pid>”, supplying the process ID of the dd process, you can look at the progress of the command.

5. For restoring from the compressed backup, you would do this:

   sudo zcat Backup.yyyy-mm-dd.sda.dd.gz | dd of=/dev/sda

   Or if you want a more exact pendant to the compress command, use this:

   sudo dd if=Backup.yyyy-mm-dd.sda.dd.gz | zcat | dd of=/dev/sda

6. Using the “seek” option of dd, or using cat, it should be possible to split the created file into several ones (for example, to burn them to several DVDs). However, I did not try that out. What definitely works is burning one backup file to a DVD-ROM (e.g. a 8 GB double layer DVD-R). Previously, there were some fixable problems with files >4 GiB on DVD.

Non-working alternatives

The original problem had been rather to back up a complete hard disk using Windows on a  ThinkPad T40. There were two options availale using IBM software, but both did not work (so the above method was used instead):

• The software “ThinkVantage Rescue and Recovery” should normally work as a solution. However, version 4.1 of it refused to install on a ThinkPad T40 because it thought the operating system is Windows Vista rather than XP.
• Alternatively, “IBM Rapid Restore Ultra” from “Access IBM” should be a possible solution. Unfortunately, this is not possible either, because: to create the backup partition, a floppy drive is needed and it was not possible (even not by reinstalling that application) to select the option that would make a USB drive sufficient for creating the backup.