Backups for docmost + nocodb (#152)

Add some backup cronjobs for nocodb and docmost!

These are kinda rough/ready backups. Both nocodb and docmost have a
storage directory (for images etc), and also of course data in their
respective database.

How does the restore process work? Yes
Where are the backups stored? On a hetzner storage box (another ~3.5
eur/month)
How can we extract the backups if we ever need them? At present, I'm the
only one with the keys here. I'll do some thinking about the best way to
share this. (see below)
Why did we need new containers? I couldn't find any open/maintained
images that had both psql + ssh. It would be nice to not have to manage
some containers, but not the end of the world. Could also manage this
automagically with github actions in theory.

Storage box file structure:

```
|   # root user (me)
\- k3s    # sub-1
   |- nocodb   # sub-2
   |   \- backups
   |       |- pg_backup_<date>.tar.gz
   |       \- file_backup_<date>.tar.gz
   \- docmost   # sub-3
       \- backups
           |- pg_backup_<date>.tar.gz
           \- file_backup_<date>.tar.gz
```

:point_up: Each individual service (docmost/nocodb) has it's own set of
ssh keys to access the storage box. This means if one service gets
pwned, at least it doesn't pwn the other. The good news is that I had
some foresight to have a separate account for the `k3s` directory, which
means we could create some ssh keys there, and share them in the keepass
for emergency access etc.

Author: b-n

infra/containers/README.md

@@ -0,0 +1,81 @@
+# backup containers
+
+This directory contains a collection of containers to help with backing up data.
+
+## Contaienrs
+
+- `quay.io/b-3-n/postgres-backup` - A container that inherits from `postgres:<v>-alpine` and installs a SSH client for remote backups.
+- `quay.io/b-3-n/file-backup` - A container based on `alpine` with an installed SSH client for remote backups.
+
+## Building
+
+Podman/Docker:
+
+```sh
+podman build -t quay.io/b-3-n/<container>:latest -f <contianer>/Containerfile .
+```
+
+e.g.
+
+```sh
+podman build -t quay.io/b-3-n/file-backup:latest -f file-backup/Containerfile .
+```
+
+> [!NOTE]
+> The above command is run from this directory, not the container sub-directory.
+
+## Usage
+
+### `postgres-backup`
+
+Connects to a given Postgres, runs a `pg_dumpall`, throws it into a gzip, and uploads to ssh.
+
+```
+podman run \
+    --rm \
+    --env SSH_HOST=backup.example.com \
+    quay.io/b-3-n/postgres-backup:latest
+```
+
+Required ENV vars:
+
+- `SSH_HOST`: SSH hostname for the backup destination
+
+Optional ENV vars:
+
+- `DEBUG`: Extra logs (default: `0`)
+- `DB_HOST`: Hostname of the Postgres database (default: `localhost`)
+- `DB_PORT`: Port of the Postgres database (default: `5432`)
+- `DB_USER`: Username for the Postgres database (default: `postgres`)
+- `DB_PASSWORD`: Password for the Postgres database (default: empty)
+- `SSH_USER`: SSH username for the backup destination (default: `root`)
+- `SSH_PORT`: SSH port for the backup destination (default: `22`)
+- `SSH_REMOTE_DIR`: Directory on the backup destination to store backups (default: `""`)
+
+### `file-backup`
+
+Takes whatever is in $FILE_PATH, tar+gzips it, uploads to ssh.
+
+```
+podman run \
+    --rm \
+    --env SSH_HOST=backup.example.com \
+    --env FILE_PATH=/data/ \
+    quay.io/b-3-n/file-backup:latest
+```
+
+Required ENV vars:
+
+- `SSH_HOST`: SSH hostname for the backup destination
+- `FILE_PATH`: The directory to back up 
+
+Optional ENV vars:
+
+- `DEBUG`: Extra logs (default: `0`)
+- `DB_HOST`: Hostname of the Postgres database (default: `localhost`)
+- `DB_PORT`: Port of the Postgres database (default: `5432`)
+- `DB_USER`: Username for the Postgres database (default: `postgres`)
+- `DB_PASSWORD`: Password for the Postgres database (default: empty)
+- `SSH_USER`: SSH username for the backup destination (default: `root`)
+- `SSH_PORT`: SSH port for the backup destination (default: `22`)
+- `SSH_REMOTE_DIR`: Directory on the backup destination to store backups (default: `""`)

infra/containers/docker-entrypoint.d/00-logs.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+DEBUG=${DEBUG:-"0"}
+
+function log() {
+  echo "$(date -Iseconds) $1"
+}
+
+function ldebug() {
+  if [ "$DEBUG" = "1" ]; then
+    log "[DEBUG] $1"
+  fi
+}
+
+function linfo() {
+  log "[INFO] $1"
+}
+
+function lerror() {
+  log "[ERROR] $1"
+  exit 1
+}

infra/containers/docker-entrypoint.d/01-ssh.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+
+
+# SSH connection setup
+ssh_user=${SSH_USER:-"root"}
+ssh_port=${SSH_PORT:-"22"}
+ssh_host=${SSH_HOST:-""}
+ssh_key_path=${SSH_KEY_PATH:-"/root/.ssh/id_ed25519"}
+ssh_addr=${ssh_user}@${ssh_host}
+ssh_opts="-o ConnectTimeout=5"
+
+function ssh_setup() {
+  if [ -z "$ssh_host" ]; then
+    lerror "SSH_HOST is not set."
+  fi
+
+  ldebug "SSH connection: ${ssh_addr} on port ${ssh_port}"
+  ldebug "SSH Keyfile Path: ${ssh_key_path}"
+
+  linfo "Testing SSH connection..."
+  ssh -i "${ssh_key_path}" -p$ssh_port ${ssh_opts} "${ssh_addr}" "ls" 1>/dev/null
+  if [ $? -ne 0 ]; then
+    lerror "SSH connection failed connecting to ${ssh_addr} on port ${ssh_port}. Exiting"
+  fi
+  linfo "SSH connection successful."
+}
+
+function ssh_send_file() {
+  local source_path="$1"
+  local dest_path="$2"
+
+  if [ ! -f "$source_path" ]; then
+    lerror "Source file $source_path does not exist."
+  fi
+
+  source_file=${source_path##*/}
+  source_extension="${source_file#*.}"
+  source_filename="${source_file%%.*}"
+  backup_file="${source_filename}_$(date +%Y%m%d_%H%M%S).${source_extension}"
+  backup_path="${backup_file}"
+  if [[ "$dest_path" != "" ]]; then
+    backup_path="${dest_path}/${backup_path}"
+  fi
+  ldebug "Prepared backup file name: ${backup_path}"
+
+  linfo "Transferring backup to remote server..."
+  scp -i "${ssh_key_path}" -P$ssh_port ${ssh_opts} "$source_path" "${ssh_addr}:${backup_path}"
+  linfo "Backup transferred to ${ssh_addr}:${backup_path}"
+}

infra/containers/docker-entrypoint.sh

@@ -0,0 +1,14 @@
+#!/bin/bash
+set -e
+
+for f in /docker-entrypoint.d/*.sh; do
+  source $f
+done
+
+if [ "$1" = 'file-backup.sh' ]; then
+  . file-backup.sh
+elif [ "$1" = 'postgres-backup.sh' ]; then
+  . postgres-backup.sh
+else
+  exec "$@"
+fi

infra/containers/file-backup/Containerfile

@@ -0,0 +1,14 @@
+ARG ALPINE_VERSION=3.22
+FROM alpine:$ALPINE_VERSION
+
+RUN apk update && apk add --no-cache openssh bash
+
+COPY docker-entrypoint.d/ /docker-entrypoint.d/
+COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
+
+COPY file-backup/file-backup.sh /usr/local/bin/file-backup.sh
+
+RUN chmod +x /usr/local/bin/docker-entrypoint.sh /usr/local/bin/file-backup.sh
+
+ENTRYPOINT ["docker-entrypoint.sh"]
+CMD ["file-backup.sh"]

infra/containers/file-backup/file-backup.sh

@@ -0,0 +1,26 @@
+#!/bin/sh
+set -e
+
+# Setup the SSH connection
+ssh_setup
+
+file_path=${FILE_PATH:-""}
+if [ -z "$file_path" ]; then
+  lerror "FILE_PATH is not set. Exiting."
+fi
+if [ ! -e "$file_path" ]; then
+  lerror "File $file_path does not exist. Exiting."
+fi
+
+# Backup process
+backup_dir="/tmp"
+backup_file="file_backup.sql.gz"
+backup_path="${backup_dir}/${backup_file}"
+
+tar -czf $backup_path $file_path
+
+file_info=$(ls -lh "$backup_path")
+ldebug "Backup file at $backup_path info: $file_info"
+
+ssh_dir=${SSH_REMOTE_DIR:-""}
+ssh_send_file "$backup_path" "$ssh_dir"

infra/containers/postgres-backup/Containerfile

@@ -0,0 +1,14 @@
+ARG POSTGRES_VERSION=17-alpine
+FROM postgres:$POSTGRES_VERSION
+
+RUN apk update && apk add --no-cache openssh
+
+COPY docker-entrypoint.d /docker-entrypoint.d
+COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
+
+COPY postgres-backup/postgres-backup.sh /usr/local/bin/postgres-backup.sh
+
+RUN chmod +x /usr/local/bin/docker-entrypoint.sh /usr/local/bin/postgres-backup.sh
+
+ENTRYPOINT ["docker-entrypoint.sh"]
+CMD ["postgres-backup.sh"]

infra/containers/postgres-backup/postgres-backup.sh

@@ -0,0 +1,40 @@
+#!/bin/sh
+set -e
+
+# Database connection setup
+db_host=${DB_HOST:-"localhost"}
+db_port=${DB_PORT:-"5432"}
+db_user=${DB_USER:-"postgres"}
+db_password=${DB_PASSWORD:-""}
+
+function pg_setup() {
+  ldebug "Using database at ${db_host}:${db_port} as user ${db_user}"
+  linfo "Testing DB connection..."
+  PGHOST=$db_host PGPASSWORD=$db_password PGPORT=$db_port PGUSER=$db_user psql -c "SELECT 1;" 1>/dev/null
+  linfo "Database connection successful."
+}
+
+function pg_backup() {
+  local backup_path=$1
+  # Backup process
+  linfo "Starting database backup..."
+  ldebug "Using database at ${db_host}:${db_port} as user ${db_user} for backup"
+  PGHOST=$db_host PGPASSWORD=$db_password PGPORT=$db_port PGUSER=$db_user pg_dumpall | gzip > "$backup_path"
+  linfo "Database backup created at $backup_path"
+}
+
+# SSH connection setup
+pg_setup
+ssh_setup
+
+# Backup process
+backup_dir="/tmp"
+backup_file="pg_backup.sql.gz"
+backup_path="${backup_dir}/${backup_file}"
+pg_backup "$backup_path"
+
+file_info=$(ls -lh "$backup_path")
+ldebug "Backup file at $backup_path info: $file_info"
+
+ssh_dir=${SSH_REMOTE_DIR:-""}
+ssh_send_file "$backup_path" "$ssh_dir"

infra/manifests/docmost/configmap-backup.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: docmost-backup
+data:
+  # For postgres-backup
+  DB_HOST: docmost-postgres.docmost.svc
+  DB_PORT: "5432"
+  DB_USER: docmost
+  # For file-backup
+  FILE_PATH: /data/docmost
+  # Destination SSH server details
+  SSH_HOST: u499668-sub3.your-storagebox.de
+  SSH_USER: u499668-sub3
+  SSH_PORT: "23"
+  SSH_KEY_PATH: /root/.ssh/id_ed25519
+  SSH_REMOTE_DIR: backups
+  DEBUG: "1"

infra/manifests/docmost/cronjob-backup.yaml

@@ -0,0 +1,68 @@
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: docmost-backup
+spec:
+  schedule: "0 3 * * *"
+  concurrencyPolicy: Forbid
+  jobTemplate:
+    spec:
+      template:
+        metadata:
+          labels:
+            app: docmost-backup
+        spec:
+          containers:
+          - name: postgres-backup
+            image: quay.io/b-3-n/postgres-backup:latest # PATCHME
+            imagePullPolicy: Always
+            env:
+            - name: DB_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: docmost-backup
+                  key: DB_PASSWORD
+            envFrom:
+            - configMapRef:
+                name: docmost-backup
+            volumeMounts:
+            - name: ssh
+              mountPath: /root/.ssh/id_ed25519
+              subPath: id_ed25519
+              readOnly: true
+            - name: ssh 
+              mountPath: /root/.ssh/known_hosts
+              subPath: known_hosts
+              readOnly: true
+          - name: data-backup
+            image: quay.io/b-3-n/file-backup:latest # PATCHME
+            imagePullPolicy: Always
+            envFrom:
+            - configMapRef:
+                name: docmost-backup
+            volumeMounts:
+            - name: ssh
+              mountPath: /root/.ssh/id_ed25519
+              subPath: id_ed25519
+              readOnly: true
+            - name: ssh 
+              mountPath: /root/.ssh/known_hosts
+              subPath: known_hosts
+              readOnly: true
+            - name: docmost-data 
+              mountPath: /data/docmost
+              readOnly: true
+          restartPolicy: OnFailure
+          volumes:
+          - name: ssh
+            secret:
+              secretName: docmost-backup
+              defaultMode: 384
+              items:
+              - key: id_ed25519
+                path: id_ed25519
+              - key: SSH_KNOWN_HOSTS
+                path: known_hosts
+          - name: docmost-data
+            persistentVolumeClaim:
+              claimName: docmost-data-docmost-docmost-0