Beanstalk environment settings always diff

[clstokes]

Beanstalk Environment settings always show a diff after the first up.

Code to reproduce

https://gist.github.com/clstokes/a2ad22a0d9c0ede138f4d56383a67052

Steps to reproduce

1. pulumi up the code above
2. pulumi preview - observe diff of environment settings

[leezen]

Upstream-related issues:
hashicorp/terraform-provider-aws#461
hashicorp/terraform#22563

[clstokes]

Another repro code to try if the one in the original post doesn't work - https://gist.github.com/clstokes/75162674b24a2f35c4c35e52edac8429.

Steps

1. pulumi config set aws:region us-west-2
2. pulumi up
3. preview --diff

(venv) beanstalk-env-order-bug-php % pulumi preview --diff
Previewing update (cameron2):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:cameron2::beanstalk-env-order-bug::pulumi:pulumi:Stack::beanstalk-env-order-bug-cameron2]
    ~ aws:elasticbeanstalk/environment:Environment: (update)
        [id=e-ww3rt3srsh]
        [urn=urn:pulumi:cameron2::beanstalk-env-order-bug::aws:elasticbeanstalk/environment:Environment::beanstalk-env-order-bug-cameron2]
        [provider=urn:pulumi:cameron2::beanstalk-env-order-bug::pulumi:providers:aws::default_1_27_0::7a821537-ea2e-4fd9-9ebe-efdc61edf383]
      ~ settings: [
          ~ [0]: {
                  ~ name     : "ELBScheme" => "ELBScheme"
                  ~ namespace: "aws:ec2:vpc" => "aws:ec2:vpc"
                  - resource : ""
                  ~ value    : "public" => "public"
                }
          ~ [1]: {
                  ~ name     : "ProxyServer" => "MinSize"
                  ~ namespace: "aws:elasticbeanstalk:environment:proxy" => "aws:autoscaling:asg"
                  + resource : "AWSEBAutoScalingGroup"
                  ~ value    : "nginx" => "2"
                }
          ~ [2]: {
                  ~ name     : "InstanceType" => "MaxSize"
                  ~ namespace: "aws:autoscaling:launchconfiguration" => "aws:autoscaling:asg"
                  + resource : "AWSEBAutoScalingGroup"
                  ~ value    : "t3.medium" => "2"
                }
          ~ [3]: {
                  ~ name     : "ServiceRole" => "InstanceType"
                  ~ namespace: "aws:elasticbeanstalk:environment" => "aws:autoscaling:launchconfiguration"
                  ~ value    : "arn:aws:iam::052848974346:role/aws-elasticbeanstalk-service-role" => "t3.medium"
                }
          + [4]: {
                  + name      : "IamInstanceProfile"
                  + namespace : "aws:autoscaling:launchconfiguration"
                  + resource  : "AWSEBAutoScalingLaunchConfiguration"
                  + value     : "aws-elasticbeanstalk-ec2-role"
                }
          + [5]: {
                  + name      : "ServiceRole"
                  + namespace : "aws:elasticbeanstalk:environment"
                  + value     : "arn:aws:iam::052848974346:role/aws-elasticbeanstalk-service-role"
                }
          + [6]: {
                  + name      : "ProxyServer"
                  + namespace : "aws:elasticbeanstalk:environment:proxy"
                  + value     : "nginx"
                }
        ]
    --outputs:--
  ~ url            : "awseb-e-w-AWSEBLoa-M925PUWWTLEJ-30063682.us-west-2.elb.amazonaws.com" => output<string>
Resources:              
    ~ 1 to update
    3 unchanged
Permalink: https://app.pulumi.com/clstokes/beanstalk-env-order-bug/cameron2/previews/0d26a7f5-73d3-4fad-ae8f-200fc1cd97cd

[packplusplus]

+1 to this fix.

I've been "working around" this problem by having a config variable api:ebIgnoreChanges: '[ "settings" ]' and then doing something like this to create a beanstalk environment.

  return new Environment(name, {
    name: name,
    application: application,
    solutionStackName: name),
    settings: ebSettings
  }, { ignoreChanges: config.requireObject("ebIgnoreChanges") });

When I need to overwrite the changes I pulumi config set api:ebIgnoreChanges "'[]'", and do a pulumi up, then set it back to ignoring settings.

The significant downside to this approach is you cannot detect configuration drift. You have zero human consumable ways to tell what's going to happen to the environment during the run since EVERYTHING looks changed.

[joeduffy]

In the original repro, what I see is that any inputs that are ARNs get translated into just the naked IDs by the AWS APIs. So the inputs are ARNs and outputs are IDs and then subsequent previews show a diff. If you change instance_profile.arn to instance_profile.id, I see the diffs go away. I don't know if this is just for specific resources or universal for all ARNs.

[lukehoban]

So the inputs are ARNs and outputs are IDs and then subsequent previews show a diff. If you change instance_profile.arn to instance_profile.id, I see the diffs go away.

Note that this particular issue appears to be hashicorp/terraform-provider-aws#280 (comment).

I'm not positive that itself addresses all of the cases raised so far - if others have repros of beanstalk diffs unrelated to ARN/ID changes, please do share the repro so we can look into those seperately.

[lukehoban]

Here's another repro:

import pulumi
import pulumi_aws

app = pulumi_aws.elasticbeanstalk.Application("app")

env = pulumi_aws.elasticbeanstalk.Environment("env",
    application=app.name,
    solution_stack_name=pulumi_aws.elasticbeanstalk.get_solution_stack(
        most_recent=True, name_regex=r"64bit Amazon Linux .+ running Docker .+"
    ).name,
    settings=[
        {
            "namespace": "aws:elasticbeanstalk:customoption",
            "name": "MyOption",
            "value": "Foo"
        },        
    ]
)

Deploying once, then running pulumi up immediately following gives:

Previewing update (dev):
     Type                                 Name             Plan       Info
     pulumi:pulumi:Stack                  pybeanstalk-dev             
 ~   └─ aws:elasticbeanstalk:Environment  env              update     [diff: ~settings]
 
Resources:
    ~ 1 to update
    2 unchanged

Do you want to perform this update? details
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:dev::pybeanstalk::pulumi:pulumi:Stack::pybeanstalk-dev]
    ~ aws:elasticbeanstalk/environment:Environment: (update)
        [id=e-hifejkyxst]
        [urn=urn:pulumi:dev::pybeanstalk::aws:elasticbeanstalk/environment:Environment::env]
        [provider=urn:pulumi:dev::pybeanstalk::pulumi:providers:aws::default_2_1_0::3f7bcd02-8b58-4077-8ae7-5efe16586638]
      ~ settings: [
          + [0]: {
                  + name      : "MyOption"
                  + namespace : "aws:elasticbeanstalk:customoption"
                  + value     : "Foo"
                }
        ]

And indeed MyOption is not in the outputs section of the state file at all.

stack.txt

[lukehoban]

aws:elasticbeanstalk:customoption

It turns out, this kind of option doesn't appear to get persisted in Beanstalk itself at all - it is not returned by aws elasticbeanstalk describe-configuration-settings. Terraform doesn't see the constant diff in this case because it automatically refreshes, which removes this setting from the state file entirely. Running pulumi refresh has the same result - and will avoid seeing a diff here. Though only because in both cases, the IaC tool forgets this was ever provided as an input.

[lukehoban]

I'm going to close this out and continue tracking the few different issues identified here in other issues.

1. ID => ARN normalization: This appears to be by-design for Beanstalk, but could potentially be handled/suppressed better by the provider in Elastic Beanstalk settings are not sticking hashicorp/terraform-provider-aws#280 (comment).

2. aws:elasticbeanstalk:customoption diffs: These aren't actually persisted to Beanstalk. A pulumi refresh should solve these. We are considering whether to refresh by default in Pulumi in Consider automatically refreshing on preview/update/destroy pulumi#2247, and this is another case that argues in favor of that.

3. Display of set diffs is confusing: I've opened Present clearer diff fo single element changes to sets  pulumi-terraform-bridge#186 to track improving the display of set diffs where a single item is changing. That would help identify where there are (unexpected) changes happening more easily.

If anyone sees other cases of unexpected diffs with Beanstalk that are not addressed by any of the above, please do open a new issue with a repro!