Retry failed runs with ObserveAndDelete policy

[d-honeybadger]

Description of your changes

Fixes #295
On backoff: Since Observe returns error whenever a retry run fails, crossplane-runtime will automatically apply backoff
We don't have control over the interval, max steps etc., but it's going to be the same default backoff that crossplane-runtime applies whenever Update fails, so we should have the bahavior that matches CheckWhenObserve policy's updates, and also matches other providers.

I have:

• Read and followed Crossplane's [contribution process].
• Run make reviewable to ensure this PR is ready for review.
• Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

1. Deploy modified version of provider-ansible in a k8s cluster (make local.xpkg.deploy.provider.provider-ansible)
2. Create an ansiblerun w/ the default run policy ObserveAndDelete and a playbook that will fail. Here's my example:

apiVersion: ansible.crossplane.io/v1alpha1
kind: AnsibleRun
metadata:
  annotations:
  name: test-run
spec:
  forProvider:
    executableInventory: false
    inventoryInline: |
      cluster:
        hosts:
          test:
            ansible_host: <VM IP>
    playbookInline: |
      - hosts: cluster
        tasks:
        - file:
            path: /test-file-missing
            state: present
    vars:
      ansible_ssh_private_key_file: ./ssh_id
  providerConfigRef:
    name: test-config
3. Watch the logs to see retries
4. Fix the playbook, e.g.:
 playbookInline: |
      - hosts: cluster
        tasks:
        - file:
            path: /test-file-missing
            state: touch
5. Check the logs to see that, after a successful run, no more runs happen during regular sync.

[contribution process]: https://git.io/fj2m9

[d-honeybadger]

Idk if this flag is a good idea: retrying failed syncs, to me, is something that a k8s controller should always do, it shouldn't even be configurable. And it's unconditionally retried with CheckWhenObserve policy.
But there might be provider-ansible users who already rely on the existing no-retries behavior, which is the only reason I added the flag. LMK what you'd prefer

[morningspace]

I agree. retryFailed seems to be the usual case most of the time.

there might be provider-ansible users who already rely on the existing no-retries behavior

Do you have a concrete use case for this? If not, I'd prefer taking this as default behavior and open another issue to track the potential necessity of introducing this flag.

BTW: It looks the retryFailed flag is not really being used through out the code in this PR.

[d-honeybadger]

🤦 yup forgot to consume it in handleLastApplied
Anyway, removed the flag

[d-honeybadger]

The list of parameters that apply only to the ansiblerun controller and not to the config controller was getting too long as I added another flag, so I decided to refactor a bit and out this into a struct

[d-honeybadger]

This whole block has just changed indentation cause it was moved out of the if. There aren't actually any logic changes

[d-honeybadger]

@morningspace ready for review 🙏

[morningspace]

@d-honeybadger There are some lint errors and conflicts need to be resolved.

[d-honeybadger]

@morningspace sorry about the linting errors! Turns out I commented out the linting line in the makefile cause at some point it was broken for me, and of course forgot that I did that...
Got it working now and all linting passing

[morningspace]

It looks awesome.