Automation for auto-sync of prod and stage databases

[john-dupuy]

Recently I went through the process of downloading a recent prod database dump and restoring it into the stage ibutsu-server DB.
The steps looked something like:

• Upload recent backup dump file to stage env PVC. This was done using the image quay.io/appuio/oc:v4.9 in the stage env. The steps looked like
  1. oc new-app quay.io/appuio/oc:v4.9
  2. oc env deployment/oc HOME=/tmp
  3. Attach the stage postgresql PVC to the oc deployment (if ReadWriteOnce you have to scale the postgresql DC down to 1 pod)
  4. oc debug deployment/oc
  5. oc login to prod db
  6. oc cp/oc rsync dump file to postgresql PVC
  7. Scale the postgresql DC up again, drop the current ibutsu DB and restore pg_restore -v -C -d postgres < backup-ibutsu-<data>.dump
  8. Scale up the backend, worker, and scheduler DC

It'd be nice if we did this regularly as a cronjob and had some automation around the process. Keeping the stage DB in sync with the prod DB will allow us to catch issues in the master branch of ibutsu that can't be caught in a local dev environment.

[rsnyman]

We should probably add a step in to anonymise the user data once that starts rolling in.