BORG Backup — Deduplicating Archiver

The documentation is very usable, easy to understand. Getting the basic things going is very easy. The code is on github.

It’s worth taking a look and having a go at it! You do know that you NEED a backup, don’t you?!?

I’ve been using dirvish (rsync based) for a good 10 years now. Once set up it is very reliable and can be expanded via pre/post server/client scripts (e.g. to calculate MD5-sums for everything). It supports file-level deduplication (using hard-links), but sadly that doesn’t help with large files that have changed only a tiny bit (like virtual machine disk images). Calculating MD5-sums takes quite some time, especially so as it is done on all backups, so the file-level deduplication saves disk-space but doesn’t help with the MD5-sums being recalculated all the time (on the very same data). A very time-consuming nuisance. Dirvish’s BIGGEST advantage from my point of view however is that the data is easily accessible, it’s just stored in the filesystem you choose (e.g. EXT4). There is no file-level encryption or compression, any system that can mount EXT4 can access the data. Today that is basically any Desktop linux out there.

Back to data deduplication. BORG on the other hand only calculates checksums for NEW chunks of data, so the process is very fast if only little has changed. Yes, I said chunks of data, not files. BORG chops up data into chunks, so it can even handle very large VM disk-images and save some space backing them up. BORG also supports archive compression, password protection and encryption. This immediately brings up the question of long-term accessibility of the data. You really really don’t want to end up with an old backup but vital backup that for some reason cannot be read anymore by new versions of BORG. Fortuitously the developers have released stand-alone versions (see github) that only have minimal requirements (a few standard libs). So far I’ve checked whether these binaries run on a live version of linux (PartedMagic 2015-01-13) and I’m glad to say that the answer is yes. That way it’s quite easily possible to pack a known-good OS / BORG combination onto an USB stick (or ISO image) for future use. In case of a backup that is not based on plain filesystems, it is absolutely VITAL to go that route.

Make a backup, use checksums, verify its integrity, make sure you can access it and restoring actually works. BE PARANOID about your data.

This entry was posted in Computer / Server, Software. and tagged , , , , , , . Bookmark the permalink.