Better ownership-aligned workflows on CAPA

[Rotfuks]

Motivation

As we moved a lot of component ownerships across KaaS Teams and this new ownership is not along the lines of repositories, but sometimes multiple teams own parts of a repository we now face some issues in our engineering workflows. We see some potential on improving the structure of how we organise the code of our systems to better match the ownership boundaries and create better flow.

Stories

Preview Give feedback

1. https://github.com/giantswarm/giantswarm/issues/27020
2. Implementing Phase 1: extract Cluster API resources #2742
   area/kaas team/turtles +2
   
3. Implementing Phase 1, part 2/5: Refactor cluster-aws Helm values #2954
   15 of 15
   provider/cluster-api-aws team/turtles topic/capi topic/cluster-schema +4
   
4. Implementing Phase 1, part 3/5: Use cluster chart in cluster-aws and remove Cluster API resource from cluster-aws #3047
   28 of 28
   provider/cluster-api-aws team/turtles topic/capi topic/cluster-schema +4
   
5. Implementing Phase 1, part 4/5: Release new cluster-aws version and upgrade existing CAPA clusters
6. Refactor capi-migration-cli to use new cluster-aws schema #3203
   2 of 2
   team/turtles +1
   
7. Use node-pool resources from cluster chart #3194
   0 of 4
   area/kaas team/turtles +2
   
8. Document the use of the cluster chart #3195
   2 of 2
   area/kaas team/turtles +2
   
9. Implementing Phase 2: extract Flux Helm Releases and Repos #2851
   5 of 5
   area/kaas team/turtles +2
   
10. Implementing Phase 3: extract Flux Helm Releases cleanup #2852
    area/kaas team/turtles +2
    

Outcome

• We have a clear structure for organising code that improves our overall efficiency by improving our engineering workflows.

[nprokopic]

In Team Turtles we have continued working on restructuring of the cluster-<provider> apps.

For more high-level overview (from which you will be able to dive into more details as we add them to specific issues) see epic Better ownership-aligned workflows on CAPA.

The original (investigation/discussion) issue suggested multiple approaches, all of which have their own pros and cons. After analyzing multiple options and mocking and trying out some of the suggested approaches, we have decided to continue with 3.2.1. Provider-specific chart referencing provider-independent chart. In practice this means that all the provider-independent resources will be moved to a separate cluster chart, which will then be added as a subchart to the existing cluster-<provider> apps.

Some of the reasons why we picked this approach:

• All existing cluster-<provider> apps continue to exist, which minimizes impact on development, (automated/manual) testing and cluster lifecycle (creation/upgrade/deletion) UI/UX.
• We get to have all provider-independent resources in a centralized place. This enables us to do all the provider-independent development work once in one place, and then test same change on all providers (by updating cluster chart version in cluster-<provider> apps).
• cluster chart will be highly configurable via Helm values, so cloud/onprem integration teams will still keep high level of control over resources in cluster chart.
  • The goal is then to understand better what is the same and similar across all providers, and then have more defaults in cluster chart that do not have to be configured explicitly from cluster-<provider> apps.
• It allows us to continue working with an iterative approach, starting with a single provider and moving one group of provider-independent resources to a provider-independent cluster chart, and then incrementally move more provider-independent resources and update more providers to use the new cluster chart.
• Once all providers have been switched to use provider-independent cluster chart, the cluster chart can progress at its own pace, and all providers can be upgraded to newer cluster chart whenever they are ready (ideally immediately, but the reality probably won't allow that).

One of the next steps is to show how all of this would look like, which some folks also asked for, so it's easier to see and understand the impact that this change will have on existing cluster apps and KaaS teams. This is already in progress and soon you will see the pull requests in existing cluster-<provider> apps and we can then dive into more details.

Here is a diagram (Miro board) that shows how the restructuring will look like in phases (exported image below). The current state is on the left, then there are few iterative phases, with the final target state being on the right.

The diagram is drawn with cluster-aws in mind. As soon as we're done with cluster-aws/AWS/CAPA, we'll continue immediately with other providers. Also, while working on cluster chart for CAPA, we're also looking into other providers so we don't miss something and design cluster chart poorly because of that.

[Rotfuks]

All Done