Compared to others

Conserve aims to fill a good niche of being simpler and safer than Duplicity or Obnam, but more friendly to cloud storage than rsync.

The great thing about backup programs is, to some extent, you don't have to pick just one: for safety, make multiple backups with different tools and approaches. So this document is not to knock the other tools or to persuade you not to use them, but rather to explain why it's worth writing Conserve too.

Duplicity addresses a similar case of backup to cloud or dumb servers, and uses librsync delta compression. But it fails several Conserve manifesto items, some of them by design:

• Verification and restoring a single file requires reading the whole archive.

  • Somewhat inherent in having a layer at which the whole backup is represented as one tar file, which does not support random access. This is perhaps not impossible to fix, if we change the format to add an index of blocks and locations within that block.
  • rdiff also done at whole-tar level so even having an index will not help with random access to anything but a full backup.

• I have experienced Duplicity bugs that make the archive unreadable and, although it is based on simpler underlying formats (rdiff, rdiffdir, tar, etc) it is still hard in practice to recover.

• Performance is unpredictable and sometimes bad.

  • Seems partly due to the caching layer? In Conserve we will not rely on a cache (there may be one later) but rather try to reduce the number of files that need to be seen at all for any operation.
  • For example a FUSE filesystem needs to list directories and retrieve individual files. Doing this on the Duplicity format would be unreasonably slow - at best, possible for modest-sized local backups with aggressive caching.

• For decent performance, requires sometimes doing a full backup; but in fact you may often have some files that never change and copying them all the way up seems redundant.

  • Conserve does still have a full-backup tier. It should need to be re-written less often, because there are multiple layers on top of it, not a single chain, therefore performance should degrade as something more like O(log n_backups) rather than O(n). Perhaps we can avoid that in Duplicity, by rearranging and garbage-collecting blocks in the archive? It is worth investigating though it seems at risk of complexity, or doing too many round trips, or avoiding the desirable property of not re-writing existing data.

• Until the first full backup completes, you can't restore anything?

  • But this could in principle be fixed in code without needing a format change: the data is all there and readable.

• Can resume backups but this seems particularly buggy?

  • This too is a code bug not a format bug, though I have a not-rationally-justified feeling the caching interacting with this is complicated.

There is, in fact, a page about replacing tar in Duplicity and Conserve should try to address all of them:

• Not indexed

  • Conserve proposes a two-level index within each band: one giving the first name within each block, and the other listing the filenames in the block. So to search for a filename in a band, we need to read the band index (fairly concise but proportional to the band size), which will identify the block in which it might be present; then that immediately says which data block to read. To find the most recent version of a filename across all bands, we'll have to search back down the stack, so typically repeating this at say four layers. For reading a single known filename we could perhaps add a Bloom filter per band.

• Not seek()able inside container: Because tar does not support encryption/compression on the inside of archives, tarballs nowadays are usually post-processed with gzip or similar. However, once compressed, the tar archive becomes opaque, and it is impossible to seek around inside. Thus if you want to extract the last file in a huge .tar.gz or .tar.gpg archive, it is necessary to read through all the prior data in the archive and discard it!

  • Addressed in Conserve by doing compression at the block level (say 10MB), and having pointers into that content: so to read the last file, you need to identify which block it's in, read and uncompress that whole block, and then pick out the relevant part.

• Doesn't support many filesystem features: Tar also doesn't have support for many features of modern filesystems including access control lists (ACLs), extended attributes (EAs), resource forks or other metadata (as on MacOS for instance), or encoding types for filenames (e.g. UTF-8 vs ascii). The format does not lend itself to extension so adding these features will probably not be easy.

  • Fairly easy to do in Conserve because we're not bound to the tar format, and it's easy to add new fields to the protobufs.

• General cruft: There are some legacy features of tar that make it a pain to work with.

  • Not a problem

• duplicity already doesn't use straight tar

  • Not a problem.

The proposal for replacing tar with a more block-oriented format is not dissimilar to the Conserve approach.

obnam format too complex? Otherwise looks pretty good.

As well as plain rsync, there are many systems built on rsync to pre-configure it for backups or to present a nicer interface. One example is Back in Time.

In some ways a very good option:

• Very fast at copying just the changes to modified files.
• Very good at pipelining to fill high-latency networks.
• Each file that's copied is immediately accessible, and backups can restore by copying the next file.
• Since the tree is stored unpacked, you can directly copy files or subtrees out, and can also use tools like diff and grep directly on the tree.
• If the archive is damaged, you can still copy out some parts of the tree.

But there are some drawbacks:

• rsync requires a smart server and can't be used with dumb cloud storage.
• Stores files unpacked, unencrypted, and uncompressed (unless, of course, this is done by the underlying filesystem)
• Can only represent things the destination filesystem can represent (usernames, permissions, etc) - however, some systems built on top of rsync such as Back in Time can store them externally
• The backup archive files will be writable (when the source files are writable) by the same user. So, it is fairly easy to accidentally and silently modify something in the archive.
• There is no overall stored checksum so no way to tell if the archive has been modified or damaged since the backup was made.

rsync has no built-in mechanism to keep multiple increments of a backup. The typical practice is to build a tree of hardlinks, and rsync can be configured to automatically break the hardlinks when the files change. This has some drawbacks:

• If a small change is made to a large file, only the delta will be sent across the network. However, it will use disk space for a new entire copy.
• It is not easy to see how much space each increment uses (though this can in principle be scripted.)
• One might like to backup to USB, Windows, or CIFS filesystems that don't support (or expose) hardlinks.

And not perfect at incremental operation:

• Recent versions don't require sending a whole file list before sending files but it does always start scanning the whole filesystem from scratch, so a short run may not make progress

Bacula storage format seems very tape-oriented, with no random access.

Does have the interesting feature of recording inaccessible files in the backup and why they were inaccessible: permission denied, etc.

TODO: More study