Pulumi preview and up incorrectly says that it will change / is changing a service

[JiriKovar]

Describe what happened

Hello,

In case of some of our service, the pulumi preview and pulumi up commands shows that it will do "something" with a service (the service root is flagged with ~, prints details of the service, but in the details, nothing is flagged to be changed (no line starts with ~/-/+).

In case of pulumi up, it says that its updating the services, the service is listed in the number of the updated services, but at the end of the day, it does nothing at all = not even a new version of the fastly service is created, which shows that at the runtime the Pulumi detects that there is no change - version management requires extra requests AFAIK and its not running even these requests. Even so, the next round of pulumi preview and pulumi up shows the same results.

Have you ever observed the same behaviour? Do you have any idea what could be the cause? In case of some of the types of our Fastly services, it happens only in certain configuration (for QA configurations some objects like ACLs or WAF are missing), in some of them it doesn't happen (even though they have ACLs and WAF), in some of them it happens consistently no matter the configuration.

This has been bothering us for quite a while, we had a couple of theories, but none of them proved to be right - it could be some handling of whitespaces, or whatnot. I'm reporting it as a bug, since the logs and the actual behaviour does not match, but at the same time I'm asking for assistance, because I cannot pinpoint the cause and give you a reasonable repro.

Sample program

import { ServiceVcl } from '@pulumi/fastly';
import type {
  ServiceVclBackend,
  ServiceVclCacheSetting,
  ServiceVclDictionary,
  ServiceVclDomain,
  ServiceVclHeader,
} from '@pulumi/fastly/types/input';
import * as pulumi from '@pulumi/pulumi';
import { WebTest, type WebTestArgs } from '../../../@components/azure/webTest';
import type { FastlyMonitoringConfig } from '../../../@components/fastly/fastlyMonitoringConfig';
import type { Domains, ServiceBackend } from '../../../@components/fastly/types';
import { Waf, type WafConfig } from '../../../@components/fastly/waf/waf';
import type { ResourceContext } from '../../../@components/utils/resourceContext';

export interface FastlyServiceWhereItAlwaysHappensArgs {
  context: ResourceContext;
  domains: Domains;
  monitoring: FastlyMonitoringConfig | undefined;
  serviceBackend: ServiceBackend;
  waf: WafConfig | undefined;
}

export class FastlyServiceWhereItAlwaysHappens {
  public readonly id: pulumi.Output<string>;
  public readonly name: pulumi.Output<string>;
  public readonly primaryDomain: pulumi.Output<string>;

  constructor(name: string, args: FastlyServiceWhereItAlwaysHappensArgs) {
    const domains: ServiceVclDomain[] = [args.domains.primary, ...args.domains.other].map((domain) => ({
      name: domain,
    }));
    const primaryDomain = args.domains.primary;

    const cacheSettings: ServiceVclCacheSetting[] = [
      {
        name: 'Do not cache',
        action: 'pass',
        ttl: 0,
        staleTtl: 0,
      },
    ];

    const backends: ServiceVclBackend[] = [
      {
        address: args.serviceBackend.address,
        name: `Backend_${args.serviceBackend.name.replace(/[^a-zA-Z0-9]+/, '_')}`,
        overrideHost: args.serviceBackend.address,
        port: 443,
        useSsl: true,
        sslCertHostname: '*.azurewebsites.net',
        autoLoadbalance: false,
        betweenBytesTimeout: 20000,
        connectTimeout: 20000,
        firstByteTimeout: 300000,
      },
    ];

    const headers: ServiceVclHeader[] = [
      {
        name: 'X-Azure-FDID',
        action: 'set',
        type: 'request',
        destination: 'http.X-Azure-FDID',
        source: pulumi.interpolate`\"${args.serviceBackend.azureFrontDoorId}\"`,
        priority: 10,
      },
    ];

    const dictionaries: ServiceVclDictionary[] = [
      {
        name: 'Edge_Security',
      },
    ];

    const serviceVcl = new ServiceVcl(
      name,
      {
        backends,
        cacheSettings,
        defaultTtl: 3600,
        dictionaries,
        domains,
        headers,
        dynamicsnippets: [],
      },
      {
        ignoreChanges: ['versionComment', 'dynamicsnippets'],
      },
    );

    if (args.waf) {
      new Waf(
        `${name}-waf`,
        {
          ...args.waf,
          origins: backends.map((b) => b.address),
          serviceId: serviceVcl.id,
        },
        {
          parent: serviceVcl,
        },
      );
    }

    if (args.monitoring) {
      for (const domain of [primaryDomain, ...args.domains.other]) {
        new WebTest(
          `subscription-cdn-webtest-${domain.replace(/[^a-zA-Z0-9]/g, '-').toLowerCase()}`,
          {
            resourceGroup: args.monitoring.resourceGroup,
            webTestProperties: {
              name: `subscription-${domain.replace(/[^a-zA-Z0-9]/g, '-').toLowerCase()}`,
              url: `https://${domain}/`,
              expectedHttpStatusCode: 200,
            },
            azureSubscriptionId: args.monitoring.azureSubscriptionId,
            applicationInsights: args.monitoring.applicationInsights,
            alert: {
              description: '      Website is DOWN',
              monitoringEmails: [args.monitoring.emails.service],
              actionGroups: [args.monitoring.actionGroupId.azureAlertMessageFormatter],
              severityLevel: 3,
            },
          } as WebTestArgs,
          {
            parent: serviceVcl,
          },
        );
      }
    }

    this.id = serviceVcl.id;
    this.name = serviceVcl.name;
    this.primaryDomain = pulumi.output(primaryDomain);
  }
}

Log output

@ updating....
    ~ fastly:index/serviceVcl:ServiceVcl: (update)
        [id=<REDACTED_SERVICE_ID>]
        [urn=urn:pulumi:qa::persistent-fastly::fastly:index/serviceVcl:ServiceVcl::qa-serviceWhereItAlwaysHappens]
        [provider=urn:pulumi:qa::persistent-fastly::pulumi:providers:fastly::default_8_11_0::021b2e85-0f62-46e3-9bea-a3381e86b531]
        activate       : true
        backends       : [secret]
        cacheSettings  : [
            [0]: {
                action        : "pass"
                cacheCondition: ""
                name          : "Do not cache"
                staleTtl      : 0
                ttl           : 0
            }
        ]
        comment        : "Managed by Terraform"
        defaultTtl     : 3600
        dictionaries   : [
            [0]: {
                forceDestroy: false
                name        : "Edge_Security"
                writeOnly   : false
            }
        ]
        domains        : [
            [0]: {
                name      : "serviceWhereItAlwaysHappens.ourinternaldomain.com"
            }
            [1]: {
                name      : "qa-serviceWhereItAlwaysHappens.global.ssl.fastly.net"
            }
        ]
        dynamicsnippets: []
        headers        : [
            [0]: {
                action           : "set"
                cacheCondition   : ""
                destination      : "http.X-Azure-FDID"
                ignoreIfSet      : false
                name             : "X-Azure-FDID"
                priority         : 10
                requestCondition : ""
                responseCondition: ""
                source           : "\"<REDACTED_GUID>\""
                type             : "request"
            }
        ]
        http3          : false
        name           : "qa-serviceWhereItAlwaysHappens"
        staleIfError   : false
        staleIfErrorTtl: 43200
Resources:
    ~ 1 updated
    116 unchanged

Affected Resource(s)

No response

Output of pulumi about

CLI          
Version      3.136.1
Go Version   go1.23.2
Go Compiler  gc

Plugins
KIND      NAME    VERSION
language  nodejs  unknown

Host     
OS       Microsoft Windows Server 2022 Standard
Version  10.0.20348 Build 20348
Arch     x86_64

This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v22.9.0'

Backend        
Name           Agent-06
URL            azblob://qastacks?storage_account=***
User           Agent-06\Administrator
Organizations  
Token type     personal

Pulumi locates its logs in C:\Users\ADMINI~1\AppData\Local\Temp by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[JiriKovar]

FYI I have also tried to run pulumi refresh --diff --skip-preview --non-interactive -j and analyzed the JSON output, where there are no differences between the "new" and the "old" objects in the results are the same and the "detailedDiff" is null.

[guineveresaenger]

Hi @JiriKovar, thank you for reporting this. We do have a category of bugs involving spurious diffs, and we understand they're confusing an unwieldy for our users.

Some of these bugs stem from the pulumi-terraform bridge rather than an individual provider. In fact, we're currently working on improved diffing logic in the bridge.
Other bugs of this type are rooted in the upstream provider implementation, such as this one on ServiceVcl bigquery logging.

Even though they may have different causes, we'd love it if you could send us a sample program with such an unexplained diff.

[JiriKovar]

Hi,

interresting is that we used to manage these Fastly services via Terraform until recently and with Terraform, we werent having these issues. The service in mention does not use any logging and the service itself is as simple as mentioned in the aforementioned code. I will try to come up with a minimal repro when I have a few moments to spare, but I have a couple of questions that could help me with that:

• The objects dependant on the very VLC service can cause this. Is this correct? (In this case most probably the WAF configuration that adds a dynamic snippet, even though its specified to ignore them).
• It might also be caused by some kind of incorrect interpolation in some of the attributes. Is this correct?
• Do you have any other ideas for source of this issue?

Thank you very much!

[guineveresaenger]

Hi @JiriKovar -

The objects dependant on the very VLC service can cause this. Is this correct? (In this case most probably the WAF configuration that adds a dynamic snippet, even though its specified to ignore them).

It is possible; it is also possible that ignoreChanges isn't working as we think it should. The fact that you see no diff between the olds and the news makes me think this is unlikely.

It might also be caused by some kind of incorrect interpolation in some of the attributes. Is this correct?
I don't think that should cause a spurious diff, but again, a repro would hlep here.

it is helpful to know that you do not see this issue in Terraform. There are some significant differences between the way Pulumi creates a diff preview and the way Terraform does, and you may be tripping over some of them.

This leads me to believe this is almost certainly a bug on our end with Diff. Please update here for a repro once you have it.

In the meantime... I don't think we've addressed anything quite like this, but the team has put in a lot of work on Diff in the bridge in the last several weeks. I've triggered a patch release of this provider for you to try out. Once https://github.com/pulumi/pulumi-fastly/actions/runs/11674513405 is done building, try v8.12.2 and let us know if your troubles persist.

[JiriKovar]

Hello @guineveresaenger,

Unfortunatelly the patch v8.12.2 produces the same results.

And even worse, I was unable to come up with a reliable minimal repro. Parallel to that I was tryting to remove some parts of the affected service and I have noticed that whenever I remove WAF from an affected service, it fixed the issue, until I put the WAF back. But at the same time, not all services that use WAF are affected (which is why I dont have a reliable minimal repro yet).

Another little piece of information is that when I create the Fastly Service with WAF, everything is OK (pulumi preview and pulumi up shows no changes) until the first refresh is run, then it starts behaving according to the issue. This should mean that it is not something between the pulumi code and stack, but rather something that changes with the first refresh. I have tried to compare the stacks before and after the refresh and it could be summarized as:

• enabling WAF publishes a new version that the pulumi state does not know about until the refresh (this is presumably due to the dynamic snippets that needs to be added)
• backend inputs get encripted as secrets (they are written in the stack in the plain text after the apply which is probably OK and to be expected)
• the version and modified date gets updated to match the aforementioned new version created by WAF
• the empty propertyDependencies object is removed (for some reason pulumi adds it with each apply and removes it with each refresh)

This list that it is indeed caused by something around either WAF, or dynamic snippets.

Based on that I'm thinking it might be the best option to move this conversation to the slack / Fastly support ticket, where we could safely share customer IDs and Fastly service IDs so that you can possibly look at the differences between the affected and unaffected services. Is that OK with you?

[JiriKovar]

Hi @guineveresaenger,

I finally have a repro:

• clone this repo: git clone https://github.com/JiriKovar/PulumiFastlyEmptyChangesRepro.git
• pulumi install
• pulumi stack init -s jiris
• pulumi stack select jiris
• pulumi config set fastly:apiKey --secret 'YOUR_API_KEY'
• pulumi config set sigSciApiKey --secret 'YOUR_WAF_API_KEY'
• pulumi config set sigSciSite 'YOUR_WAF_SITE_NAME'
• pulumi config set sigSciEmail 'YOUR_WAF_EMAIL'
• pulumi up --yes --skip-preview --diff leading to the creation of the services:
  
• pulumi up --yes --skip-preview --diff leading the expected result showing no changes:
  
• pulumi refresh --diff --skip-preview --non-interactive initiating the problem (updaring the stack as explained above):
  
• pulumi up --yes --skip-preview --diff now showing that the service is to be changed, although no part is flagged with ~ and nothing is changed & no new version is created:
  
• you can repeat the last command as many times you want and the result will still be same

[guineveresaenger]

Hi @JiriKovar - thank you so much for your perseverance here in chasing down a repro for us.

A couple points:

• This provider is not managed by Fastly - we are a team at Pulumi. Moving the conversation to a ticket at Fastly wouldn't help here - but you can join the Pulumi community Slack to get in touch with other Pulumi users and engineers.

• Would you be able to try one more thing for us - could you try PULUMI_TF_BRIDGE_ACCURATE_BRIDGE_PREVIEW=true pulumi up and tell us if you see a difference?

We'll take a look at the repro and see what's going on here as well.

[JiriKovar]

Hi @guineveresaenger,

I'm running on Windows, so I presume the exact command you had in mind is this: $env:PULUMI_TF_BRIDGE_ACCURATE_BRIDGE_PREVIEW="true"; pulumi up --yes --skip-preview --diff

When I tried that on an empty stack, the first up has succeeded, creating the 3 resources, the second (and any following one) has resulted in the following error:

Updating (jiris):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:jiris::PulumiFastlyEmptyChangesRepro::pulumi:pulumi:Stack::PulumiFastlyEmptyChangesRepro-jiris]
panic: fatal: A failure has occurred: Unexpected value marshaled: {&{{}}}

goroutine 102 [running]:

github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.137.0/go/common/util/contract/failfast.go:23

github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Failf({0x1f983ef?, 0xc000f5c268?}, {0xc000f5c1e8?, 0x1e2a760?, 0xc00090c8d0?})

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.137.0/go/common/util/contract/fail.go:32 +0xcb

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*conversionContext).makeTerraformInput(0xc0008ec580, {0x1f76086, 0xe}, {{0x0?, 0x0?}}, {{0x1b953e0, 0xc00095bba0}}, {0x265d960, 0xc000a72500}, 0x0)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/schema.go:553 +0xbf6

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*conversionContext).makeObjectTerraformInputs(0xc0008ec580, 0x0, 0xc000ab6390, {0x264b7e0, 0xc00090c8d0}, 0x0)     

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/schema.go:652 +0x2e5

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*conversionContext).makeTerraformInput(0xc0008ec580, {0xc0005a9e40, 0xb}, {{0x0?, 0x0?}}, {{0x1e9dae0, 0xc000ab6390}}, {0x265d738, 0xc0002b18c0}, 0x0)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/schema.go:528 +0xc6d

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*conversionContext).makeTerraformInput(0xc0008ec580, {0xc0005a9e30, 0x8}, {{0x0?, 0x0?}}, {{0x1bc61e0, 0xc001304f00}}, {0x265d960, 0xc000a72c80}, 0x0)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/schema.go:471 +0xfe5

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.makeSingleTerraformInput({0x2646510, 0xc0006c0900}, {0xc0005a9e30, 0x8}, {{0x1bc61e0?, 0xc001304f00?}}, {0x265d960, 0xc000a72c80}, 0x0)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/schema.go:359 +0x125

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.detailedDiffer.calculateSetHashIndexMap({{0x2646510, 0xc0006c0900}, {0x264b7e0, 0xc00090c7e0}, 0xc0005ac360, 0xc0010428d0}, {0xc00095be60, 0x1, 0x1}, {0xc00095bbe0, ...})

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/detailed_diff.go:177 +0x153

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.detailedDiffer.makeSetDiff({{0x2646510, 0xc0006c0900}, {0x264b7e0, 0xc00090c7e0}, 0xc0005ac360, 0xc0010428d0}, {0xc00095be60, 0x1, 0x1}, {{0x1bc61e0, ...}}, ...)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/detailed_diff.go:403 +0x145

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.detailedDiffer.makePropDiff({{0x2646510, 0xc0006c0900}, {0x264b7e0, 0xc00090c7e0}, 0xc0005ac360, 0xc0010428d0}, {0xc00095be60, 0x1, 0x1}, {{0x1bc61e0, ...}}, ...)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/detailed_diff.go:283 +0x413

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.detailedDiffer.makeDetailedDiffPropertyMap({{0x2646510, 0xc0006c0900}, {0x264b7e0, 0xc00090c7e0}, 0xc0005ac360, 0xc0010428d0}, 0xc000ab6330, 0xc000d07c20)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/detailed_diff.go:462 +0x232

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.makeDetailedDiffV2({0x2646510, 0xc0006c0900}, {0x264b7e0, 0xc00090c7e0}, 0xc0005ac360, {0x2650950?, 0xc0000a6e80?}, {0x265ec00, 0xc00050f1a0}, {0x2646d68, ...}, ...)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/detailed_diff.go:510 +0x218

github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff(0xc000879988, {0x2646510?, 0xc0006c04e0?}, 0xc000a3b5e0)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/v3@v3.94.0/pkg/tfbridge/provider.go:1172 +0xb93

github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x2646510?, 0xc0006c04e0?}, {0x1e9dc00?, 0xc000a3b5e0?})

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.137.0/proto/go/provider_grpc.pb.go:645 +0xcb

github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x2646510, 0xc0006c0000}, {0x1e9dc00, 0xc000a3b5e0}, 0xc000cf3a60, 0xc0009f6030)     

        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3db

github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x1f2e820, 0xc000879988}, {0x2646510, 0xc0006c0000}, 0xc000d34d00, 0xc000a23c40)

        /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.137.0/proto/go/provider_grpc.pb.go:647 +0x143

google.golang.org/grpc.(*Server).processUnaryRPC(0xc000a11000, {0x2646510, 0xc000f3a870}, {0x26569e0, 0xc0002b5860}, 0xc000ea9e60, 0xc000b6f320, 0x34bbf60, 0x0)

        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1394 +0xe49

google.golang.org/grpc.(*Server).handleStream(0xc000a11000, {0x26569e0, 0xc0002b5860}, 0xc000ea9e60)

        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1805 +0xe8b

google.golang.org/grpc.(*Server).serveStreams.func2.1()

        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1029 +0x8b

created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 99

        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.67.1/server.go:1040 +0x125

error: error reading from server: read tcp 127.0.0.1:55522->127.0.0.1:55521: wsarecv: An existing connection was forcibly closed by the remote host.

When I followed by running: $env:PULUMI_TF_BRIDGE_ACCURATE_BRIDGE_PREVIEW="False"; pulumi up --yes --skip-preview --diff, everything went back to normal (3 unchanged). Running refresh did not change anything - when the variable is set to "True" it fails with the aforementioned error, when running with the value "False", it behaves according to the issue description. Also, running the preview or refresh with the variable set to "True" fails with the same error.