Backing up your services so that they are not lost

Why backing up

When your service is broken but it worked yesterday you have two options:

  • Spend some time reading logs and debugging what went wrong. Meanwhile the service is unusable and maybe some data is irreversibly lost.
  • Rewind the service to the working state and then debug at a more relaxed pace. Hopefully it was just solar flare or a glitch in the Matrix.

This second, nerve-saving option is enabled by backing up regularly, and even better, automatically.

As an extra benefit, backing up makes it easy to transfer a service from one machine to another with minimal hassle. This is useful for datacenters on fire, if your server provider gets bought out by another corporation or if shareholders decide that it is finally time to make more profit.

This document covers the basic terms and usage of SelfPrivacy backup subsystem.

What is a snapshot?

SelfPrivacy does not make backups of the whole machine. Instead, it saves the states of each service. The state of files used by a service, taken at a certain time, is called a Snapshot. In the interface, you can see a snapshot as having an id, a service it backs up, and a date of creation.

When backups occur?

A snapshot is created in 3 cases:

  • By user’s manual command to back up a service
  • Automatically at certain intervals if Automatic Backups are enabled
  • As a precaution before an inplace restore of a service

How the data is stored?

The service’s files are stored at the cloud of user’s choosing. At the moment we support Backblaze but more are to be added.

All of the service data is encrypted with a local secret which the cloud never receives. Under the hood, we use Restic for transfers of encrypted data.

Clouds like Backblaze have an option to disallow immediate removal of data. SelfPrivacy app uses this option so that in case when the server is hacked the data cannot be erased.

Listing snapshots

There are 2 factors to keep in mind when looking at the list:

  • For the sake of performance, the list is cached. If some snapshots are missing which you think should be there, invalidate the cache so it reloads.
  • If you remove some snapshots, they will disappear from the list, but for some limited time they are still restorable with the help of the cloud.

Restoring a snapshot

When you restore a snapshot, the service is stopped, and all of its files are restored to the state when the snapshot was taken. There are 2 ways to do it.

The safest one, the default one, is to download the snapshot in its entirety, verify that data is not damaged, and replace the service files with the files from the snapshot. This has a downside that you need to have extra space to store the snapshot.

A somewhat riskier way is to overwrite the service files directly, without intermediate storage. This needs less space, but if the transfer goes wrong, you end up with a broken service. To somewhat mitigate this, a pre-restore snapshot is made just before the restoration.

The app does check that we have enough space before attempting a restore.

Forgetting a snapshot

Forgetting makes the snapshot inaccessible from the server, but deletion itself is reversible from cloud UI for some time (30 days for Backblaze by default).

Automatic Backup

If you set up an automatic backup period, all of the services will be backed up regularly according to the period.

Note that backups are independent per service. If you have services A and B backed up automatically every day in the morning, and then you back up service B manually at noon, then service A’s next backup will be in the morning as usual, but B’s backups will occur at noons.

If set to zero, autobackups will be disabled.

Restoring after someone has deleted all the snapshots

  • Go to your Backblaze/other cloud interface directly.
  • Rewind the bucket’s state to prior the deletion event.
  • Open SelfPrivacy app
  • Invalidate snapshot cache
  • List the snapshots
  • Restore from snapshots as usual

Troubleshooting backups

  • If you suspect that the snapshot list is inaccurate, try discarding the cache
  • If an inplace restore failed, make sure that your cloud is accessible and your contract is active, then try to either restore a snapshot you tried to restore, or a pre-restore snapshot generated automatically
  • If you do not have enough space on the disk for a safe restore, try restoring inplace
Last modified September 27, 2023: docs(en): Remove TODO from backups article (b8b1e5b)