✨ Support storing Ignition user data in S3 bucket for AWSMachinePool

[AndiDog]

(Replaces #5148 so I can do PR stacking)

What this PR does / why we need it:

EC2 user data is insecure and very limited in size (16 KB). Use the S3 bucket storage also for machine pools. The difference to AWSMachine is that the S3 objects must stay around because a new instance could be started by the ASG at any time, and user data with a valid token must exist. Both a lifecycle policy for regular deletion, and explicit S3 object deletion requests, are used for cleanup, so that normally, only 1-3 launch template versions' user data are stored in S3. Each launch template version relates to one S3 object.

What type of PR is this?

I marked this as breaking change for now, since the default storage – in the case of AWSMachinePool + Ignition user data format – changes to IgnitionStorageTypeOptionClusterObjectStore if nothing is explicitly specified. For me, this makes sense because the AWSMachine controller already defaulted to that storage type. It's very likely that an administrator who uses Ignition user data has also configured an S3 bucket, so I see low chances of breaking someone's workflow. If not, the reconciler will log an error and not continue, and the administrator could set AWSMachinePool.spec.ignition.storageType to UnencryptedUserData in order to keep machine pool related user data stored in plaintext on each EC2 instance.

/kind feature

Which issue(s) this PR fixes:

Didn't find an existing issue. My company ran into EC2 user data size limit problems.

Special notes for your reviewer:

This change has been successfully used in the Giant Swarm fork of CAPA since July, on production clusters. I have adapted the change to the latest upstream changes, in particular the introduction of IgnitionStorageTypeOption (#4700). Therefore, I retested this feature using Tilt on a plain CAPI@main + CAPA@main setup, with AWSCluster + Flatcar AMI + AWSMachinePool in my manifests.

Checklist:

• squashed commits
• includes documentation
• includes emojis
• adds unit tests
• adds or updates e2e tests

Release note:

Support storing Ignition user data in S3 bucket for `AWSMachinePool`

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from andidog. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[richardcase]

If this is a breaking change, then we will need to wait for an api version bump.

[richardcase]

Whilst we discuss the plan to get this in (i.e. which version):

/hold

[AndiDog]

We decided in the office hours to switch this to a non-breaking change, i.e. default to EC2 instance storage of user data, not S3 bucket storage (for AWSMachinePools)

[AndiDog]

Switched this to a non-breaking change by introducing a separate infrav1.DefaultMachinePoolIgnitionStorageType = IgnitionStorageTypeOptionUnencryptedUserData which we could change in newer API versions.