Redeploying with different 'ec2_instance_type' fails (Terraform)

[arm4b]

User should be able to change the instance type after deploying a default one for example to right-size or increase resources.
The expectation is that the old instance will be destroyed and a new one will be created.
This however fails on Terraform's side.

Steps to reproduce

1. Create a ST2 instance with everything by default
2. Change ec2_instance_type: t2.large and re-run the deployment

The error

https://github.com/bitovi/operations-stackstorm/actions/runs/4058066174/jobs/6984566138#step:2:1504

Error: Provider produced inconsistent final plan

When expanding the plan for
│ module.Stackstorm-Single-VM.local_file.ansible_inventory to include new
│ values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/local" produced an invalid new value for
│ .content: was cty.StringVal("bitops_servers:\n hosts:\n   3.101.57.229\n
│ vars:\n   ansible_ssh_user: ubuntu\n   ansible_ssh_private_key_file:
│ /tmp/tmpaqu6a50z/deployment/terraform/.ssh/bitops-ssh-key.pem\n
│ app_repo_name: operations-stackstorm\n   app_install_root: /home/ubuntu\n
│ ansible_python_interpreter: /usr/bin/python3"), but now
│ cty.StringVal("bitops_servers:\n hosts:\n   13.56.227.31\n vars:\n
│ ansible_ssh_user: ubuntu\n   ansible_ssh_private_key_file:
│ /tmp/tmpaqu6a50z/deployment/terraform/.ssh/bitops-ssh-key.pem\n
│ app_repo_name: operations-stackstorm\n   app_install_root: /home/ubuntu\n
│ ansible_python_interpreter: /usr/bin/python3").
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

[arm4b]

Re-running the job works 2nd time. It's good there are no Terraform conflicts.
So might be a relatively less complex fix to do.

[PhillypHenning]

Yea this smells like a lifecycle block.
Will take a look now

[arm4b]

We couldn't find any reasonably good solutions to fix it for now.

Similar issues:

• https://stackoverflow.com/questions/69008044/terraform-local-file-inconsistent-final-plan

• local_file resource crashes when replacing file with empty content hashicorp/terraform#18316

• Terraform reports an "inconsistent final plan" if a templatefile changes between plan and apply hashicorp/terraform-provider-local#79

• One way is to show the instance IP in the Terraform outputs so the inventory generation is not part of the terraform apply.

• Another way is trying to pull the IP from tf state in Ansible. The advantage is that it would be possible to skip the terraform apply step if there is no state change.

• Maybe something doable with _pre hooks in Ansible?

[PhillypHenning]

Another way is to isolate the teraform for generating ansible files into a separate stack.

Outputs would be populated into an out file, a prehook would convert that into files that ansible could digest

[LeoDiazL]

• We have outputs.sh in D2EC2
  But generating something after the terraform deploy looks hackish to me, and hence, ugly.
• IP from tf-state would be a better option. Thing here is key management. If the instance changes, does the key remains the same?

We could play with terraform depends_on or --target too.

How about this

[LeoDiazL]

Solved in latest release