Metadata Storage Location within S3 Configuration

[daniel-tigges]

Hi,

I am setting up an self-hosted OpenCloud instance on my Hetzner VPS (using coolify) and want to store my data (incl. meta-data) in a Hetzner S3 Object Storage. In case my VPS dies, I want to be able to restore all my data just having from the S3 Object Storage.

I have setup a docker-compose which starts and configures the OpenCloud instance and I see the data on my S3 Object Storage. However, I guess the metadata is still only on the VPS, which means, that once my VPS dies, my data on the S3 Object Storage is worthless.

What options do I have here to make the system robust, stateless and VPS agnostic?

Here you find my docker-compose which is working:

services:
  opencloud:
    image: 'opencloudeu/opencloud:latest'
    container_name: opencloud
    restart: unless-stopped
    user: '0:0'
    entrypoint:
      - /bin/sh
    command:
      - '-c'
      - "if [ ! -f /etc/opencloud/opencloud.yaml ]; then echo 'Erstelle Config...'; opencloud init --insecure true --admin-password \"${INITIAL_ADMIN_PASSWORD}\"; fi; echo 'Starte Server...'; opencloud server"
    environment:
      - 'OC_DOMAIN=${OC_DOMAIN}'
      - 'INITIAL_ADMIN_PASSWORD=${INITIAL_ADMIN_PASSWORD}'
      - 'IDM_ADMIN_PASSWORD=${INITIAL_ADMIN_PASSWORD}'
      - INSECURE=true
      - 'OC_URL=https://${OC_DOMAIN}'
      - PROXY_TLS=false
      - 'PROXY_HTTP_ADDR=0.0.0.0:9200'
      - PROXY_TRUSTED_PROXIES=0.0.0.0/0
      - OC_CONFIG_DIR=/etc/opencloud
      - OC_DATA_DIR=/var/lib/opencloud
      - STORAGE_USERS_DRIVER=decomposeds3
      - 'STORAGE_USERS_DECOMPOSEDS3_ENDPOINT=xxx'
      - STORAGE_USERS_DECOMPOSEDS3_REGION=eu-central-1
      - STORAGE_USERS_DECOMPOSEDS3_BUCKET=xxx
      - 'STORAGE_USERS_DECOMPOSEDS3_ACCESS_KEY=${S3_ACCESS_KEY}'
      - 'STORAGE_USERS_DECOMPOSEDS3_SECRET_KEY=${S3_SECRET_KEY}'
      - STORAGE_USERS_DECOMPOSEDS3_ROOT=users-files
      - STORAGE_SYSTEM_DRIVER=decomposed
    volumes:
      - 'opencloud_config_stable:/etc/opencloud'
      - 'opencloud_data_stable:/var/lib/opencloud'
    networks:
      - coolify
volumes:
  opencloud_config_stable: null
  opencloud_data_stable: null
networks:
  coolify:
    external: true

[micbar]

Both docker volumes in your setup need to be backed up.

It is a good idea to use snapshots.

To restore the system on a different host, there are some steps involved.

1. the S3 bucket needs to have some retention policies that data is not deleted immediately
2. The S3 bucket needs to be restored to the specific point in time when the storage snapshot with the metadata has been taken.

We have an s3ctl tool created in GitHub for that purpose.

[daniel-tigges]

Thanks for the fast reply.

So for the backup of the docker volumes (config, data), I think about to have a separate service within that docker-compose, which is regularly storing a backup of both volumes to another bucket, just for backups (see yaml-snippet below).

What do you mean regarding a retention policy for the S3 Bucket which contains the actual data? How can this be realized?

opencloud: ...
backup-worker:
    image: 'offen/docker-volume-backup:v2'
    restart: always
    environment:
      - AWS_ENDPOINT=xxx
      - AWS_ACCESS_KEY_ID=${S3_ACCESS_KEY}
      - AWS_SECRET_ACCESS_KEY=${S3_SECRET_KEY}
      - AWS_REGION=eu-central-1
      - BUCKET_NAME=xxx
      - BACKUP_FILENAME=backup-opencloud-%Y-%m-%dT%H-%M-%S.tar.gz
      - BACKUP_CRON_EXPRESSION=0 3 * * *
      - BACKUP_PRUNING_PREFIX=backup-opencloud-
      - BACKUP_RETENTION_DAYS=7
      
    volumes:
      - 'opencloud_config_prod:/backup/config:ro'
      - 'opencloud_data_prod:/backup/data:ro'