fix: Retry foghorn LighthouseJob update using original variable instead of DeepCopy variable

[dippynark]

We are using Lighthouse foghorn v1.14.4 and very occasionally it is getting into a continuous loop trying to update a LighthouseJob when there is a conflict.

Initially there is a failure with the following message:

Failed to update LighthouseJob status: Operation cannot be fulfilled on lighthousejobs.lighthouse.jenkins.io "[LIGHTHOUSE_JOB_NAME]": the object has been modified; please apply your changes to the latest version and try again

It does try again and succeeds: took 2 attempts to update Job

However, I think this is false positive because jobCopy (instead of &job) is being passed into retryModifyJob so on retry r.client.Get is overwriting jobCopy and reverting the updated status, meaning the retry is just updating the LighthouseJob to its current value.

This PR instead passes &job into retryModifyJob which matches with the Tekton controller:

lighthouse/pkg/engines/tekton/controller.go

Line 146 in 867b0b5

err = r.retryModifyJob(ctx, req.NamespacedName, &job, f)

I don't fully understand why foghorn is getting into a continuous loop, I would have thought the update just wouldn't happen (perhaps the noop update is actually triggering another event and foghorn is continuously conflicting with itself), but either way this should fix the problem because on next reconciliation we shouldn't get past the reflect.DeepEqual check.

For a future improvement, Lighthouse foghorn and other controllers could instead use RetryOnConflict from client-go to handle conflicts instead of the custom retryModifyJob function: https://pkg.go.dev/k8s.io/client-go/util/retry#RetryOnConflict

[jenkins-x-bot]

Hi @dippynark. Thanks for your PR.

I'm waiting for a jenkins-x member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.

[msvticket]

/ok-to-test

[msvticket]

/approve

[jenkins-x-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msvticket

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [msvticket]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jenkins-x-bot]

Failed to merge this PR due to:

failed merging [1591]: [Method Not Allowed]