Question on backup and restore process using Velero

[agu3rra]

After reading thru the Influxdb3 core backup and restore process documentation, I understand that the gist of it all is to stop the server process, copy the data folders in a specific order then copy them in the exact opposite order for the restore, then start the server back up.

I was considering using Velero for a full Kubernetes PV (persistent volume) backup and restore process, but now I am realizing it may not respect any sort of copy order.

So my question is: would it just not be recommended to do that? I suppose I can create a script that does that work for me without velero.

Thanks!

[lilic]

👋 @agu3rra could you maybe detail a bit about your setup on k8s, how are you running it? You mentioned PV are you using local storage file system for the storage?

[agu3rra]

It's a Kubernetes cluster I haven't provisioned myself. It is based on https://kyma-project.io and I am using the default storageClass. Whatever the team that provisioned the actual storage (a physical disk or network-based storage) makes no difference for me at this point. From the point of view of the container on Linux, it's like a local file system as you mentioned.

I followed the backup steps as proposed in the documentation and I was able to fully restore a DB on top of a newly created one. It's rather manual, but it works. The gist with Kubernetes and PVC's was, my PVC can only be bound to one Pod at a time, so I basically run helm uninstall influxdb to shut down the InfluxDB instance prior to backing it up. I then mount the same PVC on an Ubuntu based container and run the scripts below to generate the TAR backup then I can transfer it to whatever machine I choose via the network. Once the backup is completed, I kill the Ubuntu container and run helm install influxdb once again.

I will try a more robust setup with Velero, which I assume will simply copy the entire content of the "file system" in the PV along with other Kubernetes resources and see if it also works. Perhaps I can report back here, as it'd be a more robust solution in my opinion.

PS: I also backed up the admin-token for InfluxDB Core.

Scripts I used

Sharing the scripts I used in case they are useful to anyone else. Based on influxdata/docs-v2#6738, I suppose the _catalog_checkpoint piece can be removed.

backup.sh

#!/bin/bash
set -euo pipefail

# Backup script for InfluxDB 3 Core
# Reference: https://docs.influxdata.com/influxdb3/core/admin/backup-restore/

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR="/tmp/backups"
BACKUP_FILE="influxdb-backup-${TIMESTAMP}.tar.gz"
# InfluxDB3 Core is a single-node system, so we back up NODE_ID=0 only
NODE_ID=0
DATA_DIR="/var/lib/influxdb3/data"
PLUGINS_DIR="/var/lib/influxdb3/plugins"
AUTH_DIR="/etc/influxdb/auth"

echo "================================================"
echo "InfluxDB 3 Core Backup Process"
echo "================================================"
echo "Timestamp: ${TIMESTAMP}"
echo "Backup file: ${BACKUP_FILE}"
echo ""

# Create backup directory
mkdir -p "${BACKUP_DIR}"

# Install required tools
echo "Installing required tools..."
apt-get update -qq && apt-get install -y -qq tar gzip > /dev/null

# Check source directories exist
echo "Checking source directories..."
for dir in $DATA_DIR $PLUGINS_DIR $AUTH_DIR; do
  if [ -d "$dir" ]; then
    echo "  ✓ $dir exists"
    du -sh "$dir" 2>/dev/null || echo "    (empty or inaccessible)"
  else
    echo "  ✗ $dir not found (may be empty)"
  fi
done
echo ""

# Create backup following official documentation order
echo "Creating backup archive..."
echo "Following the official backup order from InfluxDB documentation..."

# The official docs specify backing up in this order:

# Copy data directories in the specified order
cp -r $DATA_DIR/node${NODE_ID}/snapshots "$BACKUP_DIR/"
cp -r $DATA_DIR/node${NODE_ID}/dbs "$BACKUP_DIR/"
cp -r $DATA_DIR/node${NODE_ID}/wal "$BACKUP_DIR/"
cp -r $DATA_DIR/node${NODE_ID}/catalog "$BACKUP_DIR/"
# Copy _catalog_checkpoint if it exists
if [ -f "$DATA_DIR/node${NODE_ID}/_catalog_checkpoint" ]; then
  cp $DATA_DIR/node${NODE_ID}/_catalog_checkpoint "$BACKUP_DIR/"
fi

# Backup admin token (critical for authentication)
echo "Backing up admin token..."
if [ -f "$AUTH_DIR/admin-token" ]; then
  mkdir -p "$BACKUP_DIR/auth"
  cp "$AUTH_DIR/admin-token" "$BACKUP_DIR/auth/"
  echo "  ✓ Admin token backed up"
else
  echo "  ⚠ Warning: Admin token not found at $AUTH_DIR/admin-token"
  echo "    Authentication may not work after restore."
fi

# Create tar archive from the backed up files
cd "${BACKUP_DIR}"
tar czf "/tmp/${BACKUP_FILE}" \
  --verbose \
  ./ 2>&1 | tail -20

echo ""
echo "Backup completed successfully!"
echo "Backup file: /tmp/${BACKUP_FILE}"
ls -lh "/tmp/${BACKUP_FILE}"
echo ""
echo "To copy this file to your local machine, run:"
echo "  kubectl cp influxdb-backup-restore:/tmp/${BACKUP_FILE} ./${BACKUP_FILE}"
echo ""

restore.sh

#!/bin/bash
set -euo pipefail

# Restore script for InfluxDB 3 Core
# Reference: https://docs.influxdata.com/influxdb3/core/admin/backup-restore/

BACKUP_FILE="${1:-}"
RESTORE_DIR="/tmp/backups"
NODE_ID=0
DATA_DIR="/var/lib/influxdb3/data"
PLUGINS_DIR="/var/lib/influxdb3/plugins"
AUTH_DIR="/etc/influxdb/auth"

if [ -z "$BACKUP_FILE" ]; then
  echo "Error: No backup file specified"
  echo "Usage: $0 <backup-file.tar.gz>"
  echo ""
  echo "Available backup files in /backups:"
  ls -lh /backups/*.tar.gz 2>/dev/null || echo "  (none found)"
  exit 1
fi

if [ ! -f "$BACKUP_FILE" ]; then
  echo "Error: Backup file not found: $BACKUP_FILE"
  exit 1
fi

echo "================================================"
echo "InfluxDB 3 Core Restore Process"
echo "================================================"
echo "Backup file: ${BACKUP_FILE}"
echo ""

# Install required tools
echo "Installing required tools..."
apt-get update -qq && apt-get install -y -qq tar gzip > /dev/null

# Show current state
echo "Current disk usage:"
for dir in $DATA_DIR $PLUGINS_DIR $AUTH_DIR; do
  if [ -d "$dir" ]; then
    du -sh "$dir" 2>/dev/null || echo "$dir (empty or inaccessible)"
  fi
done
echo ""

# Warning prompt
echo "⚠️  WARNING: This will overwrite existing InfluxDB data!"
echo "Press Ctrl+C within 5 seconds to abort..."
sleep 5
echo ""

# Create restore directory
mkdir -p "${RESTORE_DIR}"

# Extract backup to temporary directory
echo "Extracting backup archive to ${RESTORE_DIR}..."
tar xzf "${BACKUP_FILE}" -C "${RESTORE_DIR}" --verbose 2>&1 | tail -20
echo "  ✓ Extraction complete"
echo ""

# Clear existing data (following official docs - restore requires clean state)
echo "Clearing existing data directories..."
rm -rf $DATA_DIR/node${NODE_ID}/*
echo "  ✓ Data directory cleared"
echo ""

# Restore data directories in REVERSE order (opposite of backup)
# Official docs recommend restoring in reverse order for consistency
echo "Restoring data in reverse order..."

# Restore _catalog_checkpoint first (if it exists in backup)
if [ -f "${RESTORE_DIR}/_catalog_checkpoint" ]; then
  echo "  Restoring _catalog_checkpoint..."
  cp "${RESTORE_DIR}/_catalog_checkpoint" $DATA_DIR/node${NODE_ID}/
fi

# Restore catalog
if [ -d "${RESTORE_DIR}/catalog" ]; then
  echo "  Restoring catalog..."
  cp -r "${RESTORE_DIR}/catalog" $DATA_DIR/node${NODE_ID}/
fi

# Restore WAL
if [ -d "${RESTORE_DIR}/wal" ]; then
  echo "  Restoring wal..."
  cp -r "${RESTORE_DIR}/wal" $DATA_DIR/node${NODE_ID}/
fi

# Restore dbs
if [ -d "${RESTORE_DIR}/dbs" ]; then
  echo "  Restoring dbs..."
  cp -r "${RESTORE_DIR}/dbs" $DATA_DIR/node${NODE_ID}/
fi

# Restore snapshots last
if [ -d "${RESTORE_DIR}/snapshots" ]; then
  echo "  Restoring snapshots..."
  cp -r "${RESTORE_DIR}/snapshots" $DATA_DIR/node${NODE_ID}/
fi

echo "  ✓ All data restored"
echo ""

# Restore admin token (critical for authentication)
echo "Restoring admin token..."
if [ -f "${RESTORE_DIR}/auth/admin-token" ]; then
  mkdir -p "$AUTH_DIR"
  cp "${RESTORE_DIR}/auth/admin-token" "$AUTH_DIR/"
  echo "  ✓ Admin token restored"
else
  echo "  ⚠ Warning: Admin token not found in backup"
  echo "    InfluxDB will generate a new token on startup."
  echo "    You will need to update Grafana with the new token."
fi
echo ""

# Clean up temporary restore directory
echo "Cleaning up temporary files..."
rm -rf "${RESTORE_DIR}"
echo "  ✓ Cleanup complete"
echo ""

# Fix ownership (InfluxDB runs as user/group 1500)
echo "Setting correct ownership (1500:1500)..."
chown -R 1500:1500 $DATA_DIR
chown -R 1500:1500 $PLUGINS_DIR
if [ -d "$AUTH_DIR" ]; then
  chown -R 1500:1500 $AUTH_DIR
fi
echo "  ✓ Ownership updated"
echo ""

echo "Restore completed successfully!"
echo "New disk usage:"
for dir in $DATA_DIR $PLUGINS_DIR $AUTH_DIR; do
  if [ -d "$dir" ]; then
    du -sh "$dir" 2>/dev/null || echo "$dir (empty)"
  fi
done
echo ""
echo "You can now delete this pod and reinstall InfluxDB:"
echo "  kubectl delete pod influxdb-backup-restore -n <namespace>"
echo "  helm install influxdb ./src/infrastructure/helm/influxdb -n <namespace>"
echo ""