Merge pull request #8 from significa/ensure-no-volumes-left-improvements

tofran · web-flow · commit 2ee47a2f018c · 2023-04-28T15:49:41.000+01:00
Allow passing ensure-no-volumes-left from workflow and documentation improvements
diff --git a/.github/workflows/trigger-backup.yaml b/.github/workflows/trigger-backup.yaml
@@ -4,29 +4,34 @@ on:
   workflow_call:
     inputs:
       fly-app:
-        description: 'The Fly app name of your db backup worker.'
+        description: The Fly app name of your db backup worker.
         required: true
         type: string
       volume-size:
-        description: 'The volume size in GB for the volume of the backup worker.'
+        description: The volume size in GB for the volume of the backup worker.
         required: false
         type: number
       machine-size:
-        description: 'Fly machines v2 size (https://fly.io/docs/about/pricing/#machines)'
+        description: Fly machines v2 size (https://fly.io/docs/about/pricing/#machines)
         required: false
         type: string
       region:
-        description: 'The region to run the backup worker from.'
+        description: The region to run the backup worker from.
         required: false
         type: string
       timeout:
-        description: 'Backup GitHub action step timeout, in minutes.'
+        description: Backup GitHub action step timeout, in minutes.
         required: false
         type: number
       docker-image:
-        description: 'The image of the backup worker. Usefull to override with specific versions.'
+        description: The image of the backup worker. Usefull to override with specific versions.
         required: false
         type: string
+      ensure-no-volumes-left:
+        description: >-
+          Makes the action crash if when the backup finishes there are still volumes in the app
+        required: false
+        type: boolean
 
     secrets:
       FLY_API_TOKEN:
@@ -55,3 +60,4 @@ jobs:
           FLY_MACHINE_SIZE: ${{ inputs.machine-size }}
           FLY_VOLUME_SIZE: ${{ inputs.volume-size }}
           DOCKER_IMAGE: ${{ inputs.docker-image }}
+          ENSURE_NO_VOLUMES_LEFT: ${{ inputs.ensure-no-volumes-left }}
diff --git a/README.md b/README.md
@@ -2,9 +2,9 @@
 
 This is ~~a hacky~~ an interesting way to have a Fly app that dumps postgres databases
 that are also on Fly, to AWS S3 buckets.
-It uses a dedicated app for the *backup worker* that is _woken up_ to start the dump.
+It uses a dedicated app for the _backup worker_ that is _woken up_ to start the dump.
 When it finishes it is _scaled_ back to 0, meaning **it is not billable when idle**,
-you only pay for the backup time (it is close to free, and supper affordable even with 
+you only pay for the backup time (it is close to free, and supper affordable even with
 high end machines). It leverages Fly machines to dynamically deploy volumes and servers on demand.
 
 
@@ -25,6 +25,7 @@ Have a look into [create-resources-utils](./create-resources-utils) for scripts
 requirements in a simple way.
 
 1. In a PG shell inside your Fly Postgres instance, create an user with read permissions:
+
    ```sql
    CREATE USER db_backup_worker WITH PASSWORD '<password>';
    GRANT CONNECT ON DATABASE <db_name> TO db_backup_worker;
@@ -43,22 +44,20 @@ requirements in a simple way.
    IAM policy:
    ```json
    {
-        "Version": "2012-10-17",
-        "Statement": [
-            {
-                "Sid": "WriteDatabaseBackups",
-                "Effect": "Allow",
-                "Action": [
-                    "s3:PutObject",
-                    "s3:AbortMultipartUpload",
-                    "s3:ListMultipartUploadParts"
-                ],
-                "Resource": [
-                    "arn:aws:s3:::your-s3-bucket/backup.tar.gz"
-                ]
-            }
-        ]
-    }
+     "Version": "2012-10-17",
+     "Statement": [
+       {
+         "Sid": "WriteDatabaseBackups",
+         "Effect": "Allow",
+         "Action": [
+           "s3:PutObject",
+           "s3:AbortMultipartUpload",
+           "s3:ListMultipartUploadParts"
+         ],
+         "Resource": ["arn:aws:s3:::your-s3-bucket/backup.tar.gz"]
+       }
+     ]
+   }
    ```
 
 
@@ -67,15 +66,17 @@ requirements in a simple way.
 1. Launch your database backup worker with `fly apps create --machines`
 
 2. Set the required fly secrets (env vars). Example:
-    ```env
-    AWS_ACCESS_KEY_ID=XXXX
-    AWS_SECRET_ACCESS_KEY=XXXX
-    DATABASE_URL=postgresql://username:password@my-fly-db-instance.internal:5432/my_database
-    S3_DESTINATION=s3://your-s3-bucket/backup.tar.gz
-    ```
+
+   ```env
+   AWS_ACCESS_KEY_ID=XXXX
+   AWS_SECRET_ACCESS_KEY=XXXX
+   DATABASE_URL=postgresql://username:password@my-fly-db-instance.internal:5432/my_database
+   S3_DESTINATION=s3://your-s3-bucket/backup.tar.gz
+   ```
 
 3. OPTION A: Run `./trigger-backup.sh` whenever you want to start a backup.
-   - `FLY_APP`: (Required) Your fly application.  
+
+   - `FLY_APP`: (Required) Your fly application.
    - `FLY_API_TOKEN`: (Required) Fly token (PAT or Deploy token).
    - `FLY_REGION`: the region of the volume and consequently the region where the worker will run.
      Choose one close to the db and the AWS bucket region. Defaults to `cdg`.
@@ -87,33 +88,32 @@ requirements in a simple way.
      Option to override the default docker image `ghcr.io/significa/fly-pg-dump-to-s3:3`
    - `ENSURE_NO_VOLUMES_LEFT`: When the backup completes and the volume is deleted, checks if there
      are any volumes still available, and crashes if so. This might be useful to alert that there
-     are dangling volumes (that you might want to be paying). Defaults to `true`. If you are making
-     concurrent backups, set it to `false`.
+     are dangling volumes (that you might want to be paying for).
+     Defaults to `false` (warning to stderr only).
 
-   The volume will be deleted when the backup finishes.
-   
-   OPTION B: Optionally you can use the reusable GitHub Actions workflow from found in
+   OPTION B: Call the reusable GitHub Actions workflow found in
    `.github/workflows/trigger-backup.yaml`. Example workflow definition:
 
-    ```yaml
-    name: Backup databases
-    on:
-        workflow_dispatch:
-        schedule:
-            # Runs Every day at 5:00am UTC
-            - cron: "00 5 * * *"
-
-    jobs:
-        backup-databases:
-            name: Backup databases
-            uses: significa/fly-pg-dump-to-s3/.github/workflows/trigger-backup.yaml@v3
-            with:
-                fly-app: my-db-backup-worker
-                volume-size: 3
-                region: ewr
-            secrets:
-                FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
-    ```
+   ```yaml
+   name: Backup databases
+   on:
+     workflow_dispatch:
+     schedule:
+       # Runs Every day at 5:00am UTC
+       - cron: "00 5 * * *"
+
+   jobs:
+     backup-databases:
+       name: Backup databases
+       uses: significa/fly-pg-dump-to-s3/.github/workflows/trigger-backup.yaml@v3
+       with:
+         fly-app: my-db-backup-worker
+         volume-size: 3
+         machine-size: shared-cpu-4x
+         region: ewr
+       secrets:
+         FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }}
+   ```
 
 
 ## Backup history
diff --git a/trigger-backup.sh b/trigger-backup.sh
@@ -11,7 +11,7 @@ FLY_MACHINE_SIZE=${FLY_MACHINE_SIZE:-shared-cpu-4x}
 FLY_VOLUME_SIZE=${FLY_VOLUME_SIZE:-3}
 DEFAULT_DOCKER_IMAGE="ghcr.io/significa/fly-pg-dump-to-s3:3"
 DOCKER_IMAGE=${DOCKER_IMAGE:-$DEFAULT_DOCKER_IMAGE}
-ENSURE_NO_VOLUMES_LEFT=${ENSURE_NO_VOLUMES_LEFT-true}
+ENSURE_NO_VOLUMES_LEFT=${ENSURE_NO_VOLUMES_LEFT-false}
 
 if [[ -z "$FLY_APP" || -z "$FLY_API_TOKEN" ]]; then
   >&2 echo "Env vars FLY_APP and FLY_API_TOKEN must not be empty"
@@ -50,7 +50,7 @@ flyctl machines run \
 sleep "$SLEEP_TIME_SECONDS"
 
 echo "Waiting for volume to become detached."
-until flyctl volumes show "$volume_id" --json | jq -er '.AttachedMachine == null'; do
+until flyctl volumes show "$volume_id" --json | jq -er '.AttachedMachine == null' > /dev/null; do
   printf "."
   sleep 5
 done
@@ -60,9 +60,17 @@ sleep "$SLEEP_TIME_SECONDS"
 echo "Deleting volume $volume_id"
 flyctl volumes delete --yes "$volume_id" 
 
-if "$ENSURE_NO_VOLUMES_LEFT" && fly volumes list --app="$FLY_APP" --json | jq -e 'length != 0' ; then
-  >&2 echo "Backup completed but the app still has volumes. ENSURE_NO_VOLUMES_LEFT is true, exiting"
-  exit 1
+sleep "$SLEEP_TIME_SECONDS"
+
+volumes_left=$(fly volumes list --app="$FLY_APP" --json)
+
+if jq -e 'length != 0' <<< "$volumes_left" > /dev/null ; then
+  >&2 echo -e "WARNING: Backup completed but the app still has volumes. Response:\n$volumes_left"
+
+  if "$ENSURE_NO_VOLUMES_LEFT" ; then
+    >&2 echo "ERROR: ENSURE_NO_VOLUMES_LEFT is true, exiting."
+    exit 1
+  fi
 fi
 
 echo "Done"
