Do we need to include the "last successful pact" when turning on pending?

[bethesque]

Timothy Jones (pact-js) 21 hours ago
I have some set of endpoints, X1, including a contract that tests interactions on X1.
The client publishes a new version X2, which introduces a new endpoint
The provider team are not working on X2 yet, so X2 is pending and failures are ignored
The provider team make some unrelated change that breaks X1. But X1 is no longer tested.
I want to know that X1 is broken ASAP, ignoring deployment cycles.

Timothy Jones (pact-js) 21 hours ago
The reason I want to know immediately is because if I only find out when we try to deploy the provider, then we have to go back over potentially many test results to find out where the regression was

Timothy Jones (pact-js) 21 hours ago
(also maybe nothing is deployed yet)

Matt (pactflow.io / pact-js / pact-go):house_with_garden: 20 hours ago
In that scenario, are you saying that because pending was enabled it tested X1 and had a positive result, and then it didn’t get tested again because it was no longer pending?

Matt (pactflow.io / pact-js / pact-go):house_with_garden: 20 hours ago
(and so the regression in a subsequent build wasn’t caught?)

Matt (pactflow.io / pact-js / pact-go):house_with_garden: 20 hours ago
Regardless of pending, you need a configuration to check the X1 contract right?

Timothy Jones (pact-js) 18 hours ago
Without pending, you don't need it, because the build is failing so you know something is wrong. With pending, you no longer know something is wrong.

Timothy Jones (pact-js) 17 hours ago
enabling pending pacts makes the build evergreen when there is a change to the contract that hasn't been verified properly yet. I don't want that, because I want to know if something has broken.
Pending is neat, because it says "if this hasn't passed before, I don't expect it to pass now". But that means the build is either detecting regressions (when the latest contract isn't pending) or it's evergreen, which means it's no longer providing much value.
So, I'd like to configure it so that I get the normal regression protection, plus the "we don't think this passes yet" of pending pacts.

Timothy Jones (pact-js) 17 hours ago
I see what you're saying about "regardless of pending, you'll need to do something special to check the X1 contract", but the thing is - without pending, I don't publish contracts to main that shouldn't pass yet.

Timothy Jones (pact-js) 17 hours ago
Pending lets me publish to main, which is nicer for the consumer team

Timothy Jones (pact-js) 17 hours ago
but without the ability to ask the broker for the last verified + the current pending pact, I have the inverse problem.

Timothy Jones (pact-js) 17 hours ago
pending makes the build harder to reason about

Timothy Jones (pact-js) 17 hours ago
it passed! Does that mean there are regressions? Who knows

Timothy Jones (pact-js) 17 hours ago
without pending, if you did merge, you can say "it failed! That means the contract isn't fulfilled"

Timothy Jones (pact-js) 17 hours ago
"if this hasn't passed before, I don't expect it to pass now".
^ This is AWESOME, and I want that.
But, I also want:
"Pact lets you know if you've broken your consumer"
Which I don't get with pending.

Timothy Jones (pact-js) 17 hours ago
maybe if you pretend there's an environment called "main" then I can use can-i-deploy?

Timothy Jones (pact-js) 17 hours ago
but can-i-deploy isn't necessary. Until you use pending pacts.

Timothy Jones (pact-js) 17 hours ago
Also I think I'm then just pushing the failure further down the track.

Timothy Jones (pact-js) 17 hours ago
Essentially the same problem as this:
https://pact-foundation.slack.com/archives/C9VPNUJR2/p1631742512138400

Francisco Moreno Sanz
Builds are always passing. Sometimes I prefer them to break
Thread in #pact-broker | Yesterday at 7:48 AM | View message

Timothy Jones (pact-js) 17 hours ago
Thinking about it a bit more - I'm not sure there's a situation where you wouldn't want pending to select the head pact, plus the last successfully verified pact on that tag (which might be the same pact).

[bethesque]

I have some set of endpoints, X1, including a contract that tests interactions on X1.
Assuming this pact is passing.
The client publishes a new version X2, which introduces a new endpoint
On the same branch? The consumer team is stopping themselves from being able to deploy until the provider has both implemented and deployed this feature. That is against the recommended workflow, and seems like a bad outcome for the consumer.
The provider team are not working on X2 yet, so X2 is pending and failures are ignored
Without the pending feature enabled, the provider team cannot deploy at all, because their build is broken.
The provider team make some unrelated change that breaks X1. But X1 is no longer tested.
I want to know that X1 is broken ASAP, ignoring deployment cycles.

Pending pacts isn't supposed to enable the consumer to start putting stuff into the main pact that isn't supported yet - it's just supposed to protect the provider team's build from the consumer team if they do.

New features are still supposed to go on a branch of the consumer, or you need to follow the feature branch tagging approach that I documented.

Even with pending pacts disabled, this problem still exists - but the provider team is also shitty because they can't deploy. Enabling PP just means that the consumer pays for their own "unrecommended" practice, instead of the provider, which is the main purpose of the feature.

I think the reason you've come up with this scenario Tim is that you never use branches (and tbh, I don't know how you ever use pact without using branches or the feature toggle tagging strategy!). I think what this does highlight is that we need better support for feature toggle workflows, or the ability for users to manually mark certain interactions as pending within a pact.

Just thinking through some more scenarios. Assuming we always also verified the last successful pact (from a branch).

Imagine consumer publishes pact 1,
Provider verifies successfully
Consumer publishes pact 2 which adds an interaction
Provider verifies successfully
Consumer decides that was a mistake, and publishes pact 3 which removes that interaction again
Provider tries to remove the code for that interaction, but finds their build breaks because they're also required to verify the last successful pact, which was pact 2.

One of the things that makes the pact style of verification better than just straight "can't make a backwards compatible change" checks is that it allows people to make backwards incompatible changes but only if they're safe (ie. it will not break any deployed/released versions, and the latest "planned" version). So APIs can evolve over time rather than having to add a major version each time something needs to be removed.

Yes, there is a risk that there is a breaking change made between the time a pact is the "latest main" and when it is deployed to a test environment", but it's not a risk that we could put a number on, and my gut says it's unlikely enough to happen day to day, and when it does, can-i-deploy catches it. Always verifying the last successful version introduces its own problems.

I may end up changing my view on this, but this is where I've got to after working through the scenario this time at least! I appreciate you bringing it up and making me work though it Tim.

[TimothyJones]

I think the reason you've come up with this scenario Tim is that you never use branches

So, that's possibly part of it - my strategy before has been "it's trunk based, and yes when new features come in, the provider's build breaks, and no I don't think that's a problem, because I want to encourage the provider to implement the consumer's contract". However, my current project is using branches, and so we enabled pending pacts. Maybe we're doing the branches wrong? I don't want to have to mandate the names for the branches, but I think we probably have to?

Your example is a good one. Maybe the behaviour for pending that I'm looking for is to verify latest + last-success, and only fail the build if they both fail?

Either way, it feels to me like using pending pacts enforces the need for can-i-deploy, so making it the default makes it harder for people to use pact in a lightweight casual way - it means you have to lean into using the broker, webhooks, and all the CI tools.

I think pending pacts as the default is probably the right solution for experienced users, but I'm not sure about for new / casual users.

[TimothyJones]

Pending pacts isn't supposed to enable the consumer to start putting stuff into the main pact that isn't supported yet

Hmm, that's what I thought it was for. Maybe I have more learning to do about it :/

Possibly we need documentation on how to do pending pacts:

• In pure trunk based (If the point isn't to allow merging of things that aren't supported yet, then maybe it's supposed to be disabled here? I'm not sure)
• In trunk based with feature branches ("github flow") - I think this is what's focussed on in most of the docs so far, which makes sense because it's certainly the most common branching strategy
• With environment branches ("git flow") - I guess this is the same as for feature branches, except you would get "environment" tags for free because merging to an environment branch is the same as deployment

[TimothyJones]

Is it correct to say that the one sentence summary of pending pacts is:

"Using pending pacts moves the verification failure from test time to the time the deployment check is run"?