Upgrade CAPA to v2.x including v1beta2 CRDs

[AndiDog]

Task

Upgrade CAPA to v2.x including v1beta2 CRDs.

Background

CAPA is currently installed from a custom version (PR). Once that fix landed upstream, we can go ahead upgrading CAPA. Since that involves upgrading to v1beta2 CRDs, we must be careful since a rollback is not easy.

This is required so that operators can be upgraded to use new CRDs as well. In #1469, we only upgraded until v1.5.2 because of this prerequisite.

Hint:

• Changelog for CAPA v2.0.0
• https://github.com/giantswarm/cluster-api-provider-aws-app – make generate in that repo may create all the CRD files for the Helm chart
• Check if operators can only be upgraded after CAPA was deployed on the respective MC. If so, be careful with the order of releases.

TODOs

• Upgrade CAPA together with CAPI on one MC;
• Check if everything is working fine;
• Create issues for any problems found;
• Ensure that the next steps are discussed in the team if too many problems are found;
• Check which CRD CAPI versions are used by our operators and upgrade where needed (e.g. dns-operator-aws - upgrade/migrate CAPI imports to v1beta1 #1328).

[AndiDog]

This took much more effort than expected, and we're not done yet. Progress:

• New CAPA controller (~ v2.0.2) was logging reconciliation like crazy. I fixed the busy loop in Move defaulting before creation of patch helper so that no differences will be detected unnecessarily kubernetes-sigs/cluster-api-provider-aws#4025 and Use patch instead of update method to avoid ping-pong of errors the object has been modified; please apply your changes to the latest version and try again dns-operator-aws#180.
• CAPA wrongly assumed network load balancer type and therefore failed to reconcile existing AWSMachine objects such as those of AWSCluster/golem (the MC on which I'm testing). It's because of missing field value defaulting. Writing the test took ages since I'm new to the code base and simulating non-defaulted values is not easy. Fix pending in upstream PR Default value of load balancer type since code does not handle an empty string kubernetes-sigs/cluster-api-provider-aws#4029.
• Other, already merged fixes to work locally with CAPA code and tests: Run tests on localhost to increase security and avoid macOS firewall popup dialogs kubernetes-sigs/cluster-api-provider-aws#4024, Fix make test for Apple M1 (darwin-arm64), fail if kubebuilder assets cannot be found kubernetes-sigs/cluster-api-provider-aws#4023
• We cannot really upgrade CAPA to v2.x anywhere yet: hard blocker CAPA v1beta2 breaks the subnet config #1932. I therefore need to keep our management cluster golem in limbo state. Or we roll back, but that would be nasty.

[AndiDog]

We decided to try a workaround for #1932 – only in our fork – as short-time solution. Basically make the subnet ID optional again and test if it works. We will nevertheless get strongly involved in the draft and real solution for re-adding "generate the subnets instead of bring-your-own" in CAPA (will follow up in the other ticket).

[AndiDog]

Another blocker: https://github.com/giantswarm/giantswarm/issues/22689 must be finished, since only then we can release irsa-operator including the switch to CAPA v1beta1 types

[AndiDog]

Regarding https://github.com/giantswarm/giantswarm/issues/22689, we decided to disable pushing new versions to vintage and do the release of irsa-operator. More fixes happened afterwards.

I'm now testing private cluster creation with all the new versions + CAPA v2.x, and then we can finish here.

[AndiDog]

CAPA v2 was well tested incl. on MCs for private cluster support, so we decided to roll out the change (giantswarm/cluster-api-provider-aws-app#154).