Backup

In the Shivering-Isles infrastructure backups take place hourly to daily, depending on the amount of data and their importance.

What is the 3-2-1 Backup strategy?

The current backup concept uses a tiered principle.

  1. Basic snapshots in Longhorn are used to store data within the Kubernetes cluster for quickest recovery. Goal is to undelete data in case accidents happen. These snapshots take place hourly.
  2. Backups from Longhorn to Minio are used to create efficient block-storage based backups to a separate system. Minio uses a ZFS filesystem underneath as part of the TrueNAS setup. Goal is to keep data around independent of longhorn and making sure Volumes that are deleted can be recoved. These backups take place daily.
  3. TrueNAS ships encrypted backups to a remote storage location, usually a backup focussed cloud provider like Backblaze or Storj. Lifecycle rules, objectlocks and bucket versioning are used to manage data. Goal is to keep data around, in case the entire site is lost or a ransomware attack takes place and destroys data and backups. The Object Locks prevent deletion of data by attackers.

Multi-Tier backup, Snapshots on Longhorn, Backups from Longhorn to Minio, ZFS Snapshots, Backups from ZFS to Online S3 Storage

Verification of Backups and data

Currently there is no automatic restore test.

But for particularly important data, there is an implementation for data integrity monitoring, that ensure that photos don't corrupt silently in the background until its too late.

This is intentionally independent of the application, to ensure, it catches potential mistakes by the app developers.