AutoX feature

[sriumcp]

AutoX enables automated experimentation in Kubernetes. It does so by orchestrating install/upgrade/delete of experiments that are packaged as Helm charts automatically. It uses labels on Kubernetes resource objects as the trigger for these actions.

Imagine this from the perspective of Iter8 experiments.

1. We will use informers as an implementation mechanism for observing changes to Kubernetes resources. But, informers are not reliable (events can be missed; events are not assumed to be have any order) -- for e.g., object was deleted and then created; the informer might provide the creation event first, and the deletion event next. Req: we do not assume events from the informer are the source of truth. Informers are merely an optimization mechanism (to react to changes faster, if in an unreliable, manner).
2. When you upgrade autoX (without changing its config) -- from an experimentation perspective, you do not want any experiments to be disturbed. Req: If an experiment was running before, it should simply continue on its course, as if nothing has changed. Maybe, there is a case here to distinguish between breaking and non-breaking changes to autoX.
3. Req: Above behavior (of not disturbing experiment) need to happen even when autox crashes and is restarted by Kubernetes.
4. Req: User removes a chartGroup from the autoX config -- user shouldn't have to do further cleanup at this stage; any experiment related artifact needs to be automatically cleaned up, without any burden on user.

[sriumcp]

High level design idea #1.

1. Cleanup on start.

namespaces: # namespaces that we want autox to keep clean on start
- a
- b
- c
specs:
  spec1:
    trigger:
      group:
      version:
      resource:
      name:
      namespace:
    releaseSpecs:
      spec1:
      spec2:
      spec3:
    ...
  spec2:
... 

2. Validation: A single trigger object cannot be part of multiple autox specs.
3. Source of truth for us is not the informer presented objects, but the Kubernetes cluster. What is being investigated depends on the informer update.
4. Design feature: AutoX will use the same function for handling add, update and delete events from the informer, and during its periodic checks. And this function will be idempotent.
5. Reconcile function does the following (remember: goal is idempotence): (we're invoking these functions in iter8 CLI already) (under mutex)
   1. WANT: No Helm releases for a trigger object -- (force) delete the relevant Helm releases for that trigger (based on config). No labels on the trigger object; or, no trigger object in cluster.
   2. WANT: Helm releases for a trigger object -- for each release we want, check if that required release already exists (required release name/namespace/values).
      a. No release with required name/namespace: Helm install
      b. Release with the required name/namespace but without the required values: Helm upgrade with new values
      c. Release with the required name/namespace/values: do nothing
6. Non-MVP: Provide the user with the option of using Secret/Configmap for AutoX. Both of them are persistent drivers that Helm supports.
7. Non-MVP: Handle SIGTERM: If a reconcile loop is on, attempt to let it complete before shutdown. Attempt to achieve atomicity of chartGroup operations.
8. Non-MVP: iter8 autox list --namespace ns iter8 autox list --namespace ns --orphan, iter8 autox start # convert current iter8 autox

[sriumcp]

Note: Helm will recreate only objects that have changed. We still do not know if this is true if Helm charts are coupled with ArgoCD/FluxCD, etc.