Mounting backups

[gzagatti]

First of all, thanks for putting this package together. I have been using zfs on my system for a while, but I am not yet comfortable with all the features of this tool.

In particular, I am unsure about the correct way of mounting a backup dataset to retrieve a file.

I am using your tool. I have followed the quickstart guide to backup to a local storage, that is:

> sudo zfs set autobackup:local=true rpool
> sudo zpool import -R /mnt/backup backups
> sudo zfs-autobackup -v local backups/autobackup

So far so good. The backup is resting on the local storage and I can see it by listing all the datasets with zfs list.

Now, I would like to mount the backup to retrieve a file. Initially, I tried the following:

> zpool import -R /mnt/backup backups
> zfs mount -laO

However, when I tried to mount I got the following error:

cannot mount '/mnt/bakup/rpool/USERDATA/user_3ab41a': failed to create mountpoint: Read-only file system

To circumvent this problem, I modified the readonly property of the whole pool to readonly=off.

> sudo inherit -rS readonly backup

I exported and imported the pool again. Finally I managed to mount. But I felt this was unsafe. So, I went back and turned the readonly property back again.

> sudo zfs set readonly=on backup

Which somehow set readonly=on to all datasets in the pool. Then, magically I managed to mount most folders read-only. However, I am still facing issues with some datasets like mameara-l1/rpool/ROOT/ubuntu_a43ga/usr/local.

cannot mount '/mnt/mameara/usr/local': failed to create mountpoint: Read-only file system

Are these steps correct? Why am I still not able to mount the /mnt/mameara/usr/local?

Will zfs-autobackup always set readonly=on? More, generally does zfs-autobackup modifies other properties in the target? Can I manually change properties without being concerted that zfs-autobackup will reset then?

FInally, when mounting ZFS asks for the password for every single dataset. Previously, when I was backing up manually. It somehow only requested the password at the top-level dataset.

Thanks for your help.

[psy0rz]

this is a common problem: readonly prevents linux from creating the mountpoint.

there are two options: make sure the mountpoint exists (maybe by creating the directory on the source server)

or change the mountpoint property.

zfs-autobackup wont reset the properties.

edwin

[Scrin]

I am not able to properly answer all of the questions, but for a few;

When working with backups (other than transferring them in full) I always recommend working with clones. For example:

zfs clone backups/something/someset@offsite-20230706030010 backups/recovery-operation-1

and then work on the backups/recovery-operation-1 dataset rather than the backups/something/someset. Change its properties, mount it, do modifications, do whatever you want without risking damaging your actual backups (and zfs destroy the clone once you no longer need it)

For the encryption password related one, I believe this depends on how you backup the datasets (decrypt & re-encrypt, or transfer as-is, is the encryption root backed up as well, ..) and what ends up being their encryption root on the backup target. You can check this with zfs get encryptionroot. If I'd have to guess on the source dataset (and in your previous manual backup) all datasets had the same encryptionroot where they inherited the encryption keys from, and in your current setup they end up having different encryptionroot properties.

In my setup I use decrypt & re-encrypt (in order to be able to change the dataset compression from lz4 to gzip to save space since performance is not a concern on the backups) so I end up with the same encryptionroot, and thus I only need to provide the password for the root dataset, so I'm not able to help here further since I'm not seeing this, but hopefully this provided some pointers on what to search for.

For reference, the command I use in my nightly script is:

zfs-autobackup --ssh-source $HOST --ssh-target $STORAGE_HOST --keep-source 5 --clear-refreservation --filter-properties quota --rollback --decrypt --encrypt --verbose offsite $DATASET

where $HOST is what's being backed up, $STORAGE_HOST is the host that holds the backups, and $DATASET is the dataset to backup to.

This script runs on a dedicated, hardened "backupper machine" that only connects to hosts that needs backing up and hosts that hold the actual backups.

I also clear the refreservation and quota since I don't want those to limit the amount of backups I can hold, I use them on the source hosts to prevent a single service (or other dataset user) from filling up the entire pool, causing everything else to break as well.

[psy0rz]

yes, i forgot to answer that: you need a common encryption parent and use zfs load-key on it i think.

[gzagatti]

@Scrin may I ask you what is the --clear-refreservation flag? The help string says it is recommended and saves space, but it is not turned on. Is there a reason?

I've also noticed that the backup keeps the same autobackup:label property as the original data. Is that expected?

[Scrin]

It clears the refreservation of the dataset (on the receiving side). Since this effectively causes "modifications" to the dataset properties, it's not enabled (=cleared) by default. Same goes for quota. Or at least this is what I believe is the reasoning.

In general it is recommended to use since the refreservation reserves space for the dataset on the pool, that reserved space is not available for other datasets (obviously) and thus the "usable space" for all other datasets is reduced in such case. However when doing so you should be aware that if you restore the entire dataset from a backup, you need to "manually" set the refreservation (and other properties like quota, if you filtered them) after the restoration since those properties are "modified" from the original on the backup.

Also, by default the refreservation is not set, so in a "default pool" the --clear-refreservation doesn't have any effect in the first place. However refreservation is useful on production systems where it can be used to ensure that critical datasets always have some reserved space available even if the pool accidentally gets full, despite quotas limiting the maximum amount a single dataset can hold. A quota can "protect" the pool against one "misbehaving" dataset, but it won't protect against scenarios where multiple datasets suddenly consume the entire pool.

More about quotas and reservations can be found here: https://docs.oracle.com/cd/E19253-01/819-5461/gazvb/index.html

--

About the autobackup property, I believe it's expected as well. Clearing that property would again be a "modification" to the original, and preserving it on the "backups side" can help differentiate between different autobackup instances/runs/schedules/etc

[gzagatti]

Thanks for the detailed answer once again. I'm not yet that familiar with all the ZFS capabilities, so I'll take note of this information to fine tune my backup strategy with time. Also, for my use case I prefer to replicate the backup as close to the original pool as possible, so I'll leave the defaults on.

[gzagatti]

Many thanks for your help! I managed to put together a script to combine zfs-autobackup with borg to perform additional cloud backups. Unfortunately, ZFS providers are still quite pricey, so I will use borg for that.

The script I put together is below. If you have any additional suggestions.

#!/bin/sh

# Borgmatic hook: ZFS Autobackup
#
# usage: borgmatic_hook_zfs_autobackup {clone,destroy} [zfs_mountpoint] [zfs_autobackup_label] [borg_mountpoint]
#
# The script defines two borgmatic hooks to be used as a `before_actions` and `after_actions` hook.
#
# Before borgmatic actions, the hook takes a ZFS snapshot using `zfs-autobackup`. Then
# creates a ZFS clone and mounts the snapshot to prepare it for borg backup.
#
# After borgmatic actions, the hook destroys the ZFS clone.
#
# See: 
# https://github.com/scotte/borgsnap
# https://github.com/mikroskeem/zorg

set -eu

usage() {
  echo "usage: borgmatic_hook_zfs_autobackup {clone,destroy} [zfs_mountpoint] [zfs_autobackup_label] [borg_mountpoint]"
  exit 1
}

destroy() {
  zfs destroy "rpool/borg"
}

assert_no_borg_clone() {
  set +e
  zfs list -t filesystem rpool/borg >/dev/null 2>&1
  if [ $? -eq 0 ]; then
    if [ $# -ne 0 ]; then
      echo "rpool/borg exists, aborting."
      # soft fail
      exit 75
    else
      destroy
      assert_no_borg_clone 'no-retry'
    fi
  fi
  set -e
}

assert_borg_clone() {
  set +e
  zfs list -t filesystem rpool/borg >/dev/null 2>&1
  if [ $? -ne 0 ]; then
    exit 0
  fi
  set -e
}

snapshot() {
  zfs-autobackup --keep-source 2,1d1w,1w1m,1m6m --keep-target 2,1d1w,1w1m,1m6m ${1}
}

findlast() {
  zfs list -d 1 -s name -t snap -o name "${1}" | grep "@${2}-" | tail -1
}

clone() {
  mkdir -p ${1}
  zfs clone -o mountpoint=${1} -o readonly=on -o autobackup:${2}=false ${3} rpool/borg
  echo "rpool/borg"
}

if [ $# -ne 4 ]; then usage; fi

hook=$1
zfs_mountpoint=$2
zfs_autobackup_label=$3
borg_mountpoint=$4

case ${hook} in
  create)
    assert_no_borg_clone
    snapshot ${zfs_autobackup_label}
    last_snapshot=$(findlast ${zfs_mountpoint} ${zfs_autobackup_label})
    echo "Found snapshot: '${last_snapshot}'."
    clone ${borg_mountpoint} ${zfs_autobackup_label} ${last_snapshot}
    ;;
  destroy)
    assert_borg_clone
    destroy
    ;;
  *)
    usage
    exit 1
    ;;
esac

[psy0rz]

Since this is a common problem i've created a wiki page: https://github.com/psy0rz/zfs_autobackup/wiki/Mounting