Restoring the Terraform State

[rsignell-usgs]

We have a situation where the bucket containing the Terraform State was moved to a different Amazon account by mistake. The actions started failing with permission denied, and before we realized the bucket still existed, we tried some workarounds with conda-store to upgrade the existing environments.

When we discovered the bucket still existed, we gave list/read/write access to the bucket using the canonical id for the account that controls qhub:

We confirmed that we are able to read/write to this bucket from the aws account that is running qhub,

$ aws s3 ls s3://esipfed-spruce-dev-terraform-state/terraform/ --profile esiplab
2022-02-07 14:48:38     732057 esipfed-spruce-dev.tfstate

but when we tried rerunning the action to add a package to one of the environments, we got the same error:

 [terraform]: Successfully configured the backend "s3"! Terraform will automatically
[terraform]: use this backend unless the backend configuration changes.
[terraform]: Error refreshing state: AccessDenied: Access Denied
[terraform]: 	status code: 403, request id: CRG89HBYQ901G00A, host id: hwtKPNne3c8mjCB/BXp2+zp/gaHNIRpBw6qjPKO6bF+U3N4wLklH+PEuyE1aDXvMzCUVaF3ZKxo=

Problem encountered: Terraform error

Error: Process completed with exit code 1.

@costrouc , any idea how we might recover?

[costrouc]

@rsignell-usgs I don't have a great idea on how to recover from this. There is a infrastructure/state.tf that has the bucket/db that is used. Maybe this needs to be modified? I'm not totally sure.

[tylerpotts]

@rsignell-usgs in the file qhub-spruce/infrastructure/state.tf it lists which bucket the state is stored in. Here's my thoughts:

• Make your changes locally and run qhub render -c qhub-config.yaml -o ./
• Create a new s3 bucket in the account where your qhub is currently deployed
• On line 3 in qhub-spruce/infrastructure/state.tf change this name to your new bucket
• copy the state file from your old bucket to the new bucket, keeping the same file name
• run qhub deploy -c qhub-config --disable-render

You'll have to do the render and manual edit of the bucket name each time you want to do another deploy. Rendering will overwrite to the old bucket name.

[tylerpotts]

The reason you're getting the current error is that you have two AWS accounts involved and can only have the credentials for one active at a time