diff --git a/0051-v1beta1-schema/README.md b/0051-v1beta1-schema/README.md new file mode 100644 index 0000000..02ba76a --- /dev/null +++ b/0051-v1beta1-schema/README.md @@ -0,0 +1,546 @@ + + +# ZEP-0051: v1beta1 schema + + + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + + +## Summary + + + +Several fields in the v1alpha1 ZarfPackageConfig can be restructured to provide a more intuitive experience. Other fields that have a poor user experience and add unnecessary overhead to Zarf should be removed. A new schema version, v1beta1, provides the opportunity to make these changes. + +## Motivation + + + +There are several open issues requesting enhancements to the schema. The general theme of these changes is to make it easier to create Zarf packages. +- [Refactor charts definition in zarf.yaml #2245](https://github.com/zarf-dev/zarf/issues/2245) +- [Breaking Change: make components required by default #2059](https://github.com/zarf-dev/zarf/issues/2059) +- [Use kstatus as the engine behind zarf tools wait-for and .wait.cluster #4077](https://github.com/zarf-dev/zarf/issues/4077) + +Additionally, users often struggle to use data injections. Usually, they would be better served by using a Kubernetes native solution [#3926](https://github.com/zarf-dev/zarf/issues/3926). + +### Goals + + + +- Detail the schema changes in the ZarfPackageConfig from v1alpha1 to v1beta1. + +### Non-Goals + + + +- Discuss how the Zarf codebase will shift to handle multiple API versions. This is detailed in [0048-schema-upgrade-process](https://github.com/zarf-dev/proposals/pull/49) + +## Proposal + + + +Zarf will determine the schema of the package definition using the existing optional field `apiVersion`. If `apiVersion` is not set, then Zarf will assume it is a v1alpha1 package. Users will be able to automatically upgrade their package to the v1beta1 schema by running `zarf dev convert`. + +The v1beta1 schema will remove, replace, and rename several fields. View this [zarf.yaml](zarf.yaml) to see a package definition with reasonable values for each key. + +### Removed Fields + +If a package has these fields defined then `zarf dev convert` will error with and print a recommendation for an alternative. + +- `.components.[x].group` will be removed. Users will be recommended to use `components[x].only.flavor` instead. +- `.components.[x].dataInjections` will be removed. There will be a guide in Zarf's documentation for alternatives. See [#3926](https://github.com/zarf-dev/zarf/issues/3926). +- `.components.[x].charts.[x].variables` will be removed. Its successor is [Zarf values](../0021-zarf-values/), but there will be no automated migration with `zarf dev convert`. +- `.component.[x].default` will be removed. It set the default option for groups and (y/n) interactive prompts for optional components. Groups are removed, and we've generally seen the user base shift away from optional components. + +### Replaced / Restructured Fields + +`zarf dev convert` will automatically migrate these fields. + +- `.components.[x].actions.[onAny].onSuccess` will be removed. Any `onSuccess` actions will be appended to the `actions.[onAny].after` list. +- `.components[x].actions.[onAny].setVariable` will be removed. This field is already deprecated and will be migrated to the existing field `.components[x].actions.[onAny].setVariables`. +- `.components.[x].scripts` will be removed. This field is already deprecated and will be migrated to the existing field `.components.[x].actions`. +- `.metadata` fields `image`, `source`, `documentation`, `url`, `authors`, `vendors` will be removed. `zarf dev convert` will move these fields under `.metadata.annotations`, which is a generic map of strings. +- `.components[x].actions.[onAny].wait.cluster` will receive a new required sub field, `.apiVersion`. During conversion, `.apiVersion` will be added to the object but kept empty. Users will be warned that they must fill this field out, otherwise create will error. +- `.components.[x].healthChecks` will be removed and appended to `.components.[x].actions.onDeploy.After.wait.cluster`. This will be accompanied by a behavior change in `zarf tools wait-for` to perform kstatus style readiness checks when `.wait.cluster.condition` is empty. See [Zarf Tools wait-for Changes](#zarf-tools-wait-for-changes). +- `.component.[x].charts` will be restructured to move fields into different sub-objects depending on the method of consuming the chart. See [Helm Chart Changes](#zarf-helm-chart-changes) + +### Renamed Fields + +`zarf dev convert` will automatically migrate these fields. + +- `.metadata.aggregateChecksum` will move to `.build.aggregateChecksum`. +- `.metadata.yolo` will be renamed to `.metadata.airgap`. `airgap` will default to true. +- `.components[x].required` will be renamed to `.components[x].optional`. `optional` will default to false. Since `required` currently defaults to false, components will now default to being required. +- `noWait` will be renamed to `wait`. `wait` will default to true. This change will happen on both `.components.[x].manifests` and `.components.[x].charts`. +- `.components.[x].actions.[default/onAny].maxRetries` will be renamed to `.components.[x].actions.[default/onAny].retries`. +- `.components.[x].actions.[default/onAny].maxTotalSeconds` will be renamed to `.components.[x].actions.[default/onAny].timeout`, which must be in a [Go recognized duration string format](https://pkg.go.dev/time#ParseDuration). + +### User Stories (Optional) + + + +#### Story 1 + +As a user of Helm charts in my package, I have the following existing `zarf.yaml`: + +```yaml +kind: ZarfPackageConfig +metadata: + name: helm-charts +components: + - name: demo-helm-charts + required: true + charts: + - name: podinfo-local + version: 6.4.0 + namespace: podinfo-from-local-chart + localPath: chart + valuesFiles: + - values.yaml + + - name: podinfo-oci + version: 6.4.0 + namespace: podinfo-from-oci + url: oci://ghcr.io/stefanprodan/charts/podinfo + valuesFiles: + - values.yaml + + - name: podinfo-git + version: 6.4.0 + namespace: podinfo-from-git + url: https://github.com/stefanprodan/podinfo.git + gitPath: charts/podinfo + valuesFiles: + - values.yaml + + - name: podinfo-repo + version: 6.4.0 + namespace: podinfo-from-repo + url: https://stefanprodan.github.io/podinfo + repoName: podinfo + releaseName: cool-release-name + valuesFiles: + - values.yaml +``` + +I want to upgrade to the v1beta1 schema, so I run `zarf dev convert`, which produces: + +```yaml +apiVersion: v1beta1 +kind: ZarfPackageConfig +metadata: + name: helm-charts + description: Example showcasing multiple ways to deploy helm charts + version: 0.0.1 + +components: + - name: demo-helm-charts + optional: false # Changed from `required: true` + charts: + - name: podinfo-local + namespace: podinfo-from-local-chart + local: + path: chart # Changed from `localPath` + # version field removed - uses version from local chart.yaml + valuesFiles: + - values.yaml + - name: podinfo-oci + namespace: podinfo-from-oci + oci: + url: oci://ghcr.io/stefanprodan/charts/podinfo + version: 6.4.0 + valuesFiles: + - values.yaml + + - name: podinfo-git + namespace: podinfo-from-git + git: + url: https://github.com/stefanprodan/podinfo.git@6.4.0 + path: charts/podinfo # Changed from `gitPath` + # version field removed - uses version from chart.yaml at git tag + valuesFiles: + - values.yaml + + - name: podinfo-repo + namespace: podinfo-from-repo + helmRepo: + url: https://stefanprodan.github.io/podinfo + name: podinfo # Changed from `repoName` + version: 6.4.0 + releaseName: cool-release-name + valuesFiles: + - values.yaml +``` + +#### Story 2 + +As a user of health checks in my package, I have the following `zarf.yaml` in v1alpha1: + +```yaml +kind: ZarfPackageConfig +metadata: + name: health-checks + description: Deploys a simple pod to test health checks + +components: + - name: health-checks + required: true + manifests: + - name: ready-pod + namespace: health-checks + noWait: true + files: + - ready-pod.yaml + healthChecks: + - name: ready-pod + namespace: health-checks + apiVersion: v1 + kind: Pod +``` + +I want to move to the latest schema, so I run `zarf dev convert`, which produces: + +```yaml +apiVersion: v1beta1 +kind: ZarfPackageConfig +metadata: + name: health-checks + description: Deploys a simple pod to test health checks + +components: + - name: health-checks + optional: false # Changed from `required: true` + manifests: + - name: ready-pod + namespace: health-checks + wait: false # Changed from `noWait: true` + files: + - ready-pod.yaml + actions: + onDeploy: + after: + - wait: + cluster: + kind: Pod + name: ready-pod + namespace: health-checks + apiVersion: v1 + # condition is empty, so Kstatus will be used for readiness check +``` + +### Risks and Mitigations + + + +The field `.components.[x].dataInjections` will be removed without a direct replacement in the schema. There must be documentation to present to users so they know what alternatives they can use to achieve a similar result. + +The alpha field `.components.[x].charts.[x].variables` has seen significant adoption and there will be no automatic conversion to its replacement Zarf values. There must be documentation on how users can utilize Zarf values as an alternative to chart variables. + +## Design Details + + + +### Zarf Helm Chart Changes + +The ZarfChart object will be restructured to match the code block below. Exactly one of sub-objects `helmRepo`, `git`, `oci`, or `local` is required for each entry in `components.[x].charts`. The fields `localPath`, `gitPath`, `URL`, and `repoName` will be removed from the top level of `components.[x].charts`. See [#2245](https://github.com/defenseunicorns/zarf/issues/2245). + +During conversion, Zarf will detect the method of consuming the chart and create the proper sub-objects. If a git repo is used, then `@` + the `.version` value will be appended to `.gitRepoSource.URL`. This is consistent with the current Zarf behavior. + +Zarf uses the top level `version` field to determine where in the package layout file structure it will place charts. This makes the field necessary for deploy, and therefore it must be carried over using the strategy defined in the removed fields section of [0048](https://github.com/zarf-dev/proposals/pull/49/files). Newer versions of Zarf will ensure that Zarf works whether or not `version` is set. Packages created with the v1beta1 schema will leave `version` empty, and therefore will not work with earlier versions of Zarf. When support is dropped for v1alpha1 packages, the `version` field will be dropped entirely. Note, this process is applied to internal conversion so that there is no change in behavior when v1alpha1 packages use function signatures that contain v1beta1 objects. `zarf dev convert` will simply move the top level `version` field to the right sub object, or drop it when not applicable. + +```go +// ZarfChart defines a helm chart to be deployed. +type ZarfChart struct { + // The name of the chart within Zarf; note that this must be unique and does not need to be the same as the name in the chart repo. + Name string `json:"name"` + // The version of the chart. This field is removed for the schema, but kept as a backwards compatibility shim so v1alpha1 packages can be converted to v1beta1 + version string + // The Helm repo where the chart is stored + HelmRepo HelmRepoSource `json:"helmRepo,omitempty"` + // The Git repo where the chart is stored + Git GitRepoSource `json:"git,omitempty"` + // The local path where the chart is stored + Local LocalRepoSource `json:"local,omitempty"` + // The OCI registry where the chart is stored + OCI OCISource `json:"oci,omitempty"` + // The namespace to deploy the chart to. + Namespace string `json:"namespace,omitempty"` + // The name of the Helm release to create (defaults to the Zarf name of the chart). + ReleaseName string `json:"releaseName,omitempty"` + // Whether to not wait for chart resources to be ready before continuing. + Wait *bool `json:"wait,omitempty"` + // List of local values file paths or remote URLs to include in the package; these will be merged together when deployed. + ValuesFiles []string `json:"valuesFiles,omitempty"` + // [alpha] List of values sources to their Helm override target + Values []ZarfChartValue `json:"values,omitempty"` +} + +// HelmRepoSource represents a Helm chart stored in a Helm repository. +type HelmRepoSource struct { + // The name of a chart within a Helm repository + RepoName string `json:"repoName,omitempty"` + // The URL of the chart repository where the helm chart is stored. + URL string `json:"url"` + // The version of the chart to deploy; for git-based charts this is also the tag of the git repo by default (when not using the '@' syntax for 'repos'). + Version string `json:"version"` +} + +// GitRepoSource represents a Helm chart stored in a Git repository. +type GitRepoSource struct { + // The URL of the git repository where the helm chart is stored. + URL string `json:"url"` + // The sub directory to the chart within a git repo. + Path string `json:"path,omitempty"` +} + +// LocalRepoSource represents a Helm chart stored locally. +type LocalRepoSource struct { + // The path to a local chart's folder or .tgz archive. + Path string `json:"path"` +} + +// OCISource represents a Helm chart stored in an OCI registry. +type OCISource struct { + // The URL of the OCI registry where the helm chart is stored. + URL string `json:"url"` + Version string `json:"version"` +} +``` + +#### Zarf Tools wait-for Changes + +`zarf tools wait-for` is the underlying engine to `.wait.cluster`. Currently, `zarf tools wait-for` shells out to `zarf tools kubectl wait`. In the future, `wait-for` will optionally accept an API version alongside the resource kind. If API version is set and condition is empty, then kstatus will be used as `wait-for`'s engine. + +v1alpha1 packages do not have `.apiVersion` as a sub field under `.wait.cluster`, so they will always use the existing engine, avoiding breaking changes. v1beta1 packages will require that `apiVersion` is set. + +### Test Plan + + + +[X] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this proposal. + +There will be e2e tests for creating, deploying, and publishing a v1beta1 package. As the schema nears towards GA, existing tests will shift to use the v1beta1 schema. + +### Graduation Criteria + + + +- Alpha: Fields are subject to change or rename. No backwards compatibility guarantees. +- Beta: Fields will not change in a way that is not fully backwards compatible. +- GA: Users have provided feedback that the new schema improves the UX. Examples and tests in Zarf shift to using the v1beta1 schema. + +Deprecation: +- This schema will likely be deprecated one day in favor of a v1 schema. It will not be deprecated until after the next schema version is generally available. Once deprecated, Zarf will still support the v1beta1 schema for at least one year. + +### Upgrade / Downgrade Strategy + + + +See proposal in ZEP-0048. + +### Version Skew Strategy + + + +See version skew strategy in ZEP-0048. + +## Implementation History + + + +- 2025-10-21: Proposal submitted + +## Drawbacks + + + +## Alternatives + + diff --git a/0051-v1beta1-schema/zarf.yaml b/0051-v1beta1-schema/zarf.yaml new file mode 100644 index 0000000..6323b57 --- /dev/null +++ b/0051-v1beta1-schema/zarf.yaml @@ -0,0 +1,440 @@ +kind: ZarfPackageConfig +apiVersion: v1beta1 +metadata: + name: everything-zarf-package + description: A zarf package with a non empty value for every key - this demonstrates the changes proposed by ZEP-0051 + version: v1.0.0 + uncompressed: false + architecture: amd64 + airgap: true # changed from yolo + annotations: # All of these are v0 fields that will be deprecated in favor of a choose your own adventure label map + authors: cool-kidz + documentation: https://my-package-documentation.com + source: https://my-git-server/my-package + url: https://my-package-website.com + vendor: my-vendor + image: https://my-image-url-to-use-in-deprecated-zarf-ui + anyway-you-want-it: thats-the-way-you-need-it + allowNamespaceOverride: true +build: # Everything here is created by Zarf not be users + terminal: my-computer + user: my-user + architecture: amd64 + timestamp: 2021-09-01T00:00:00Z + version: v1.0.0 + migrations: + - scripts-to-actions + versionRequirements: + - version: 0.65.0 + reason: "the package structure of Helm charts changed" + registryOverrides: + gcr.io: my-gcr.com + differential: true + differentialPackageVersion: "v0.99.9" + flavor: cool-flavor + aggregateChecksum: shasum # this is moved from .metadata +values: + files: + - values.yaml + schema: values-schema.json +components: +- name: a-component + description: Zarf description + optional: false # Component is required (default behavior) + only: + localOS: darwin + cluster: + architecture: amd64 + distros: + - ubuntu + flavor: a-flavor # this will only be used when there are multiple components + import: + name: other-component-name + path: ABCD # Only path or URL will be used, not both + url: oci:// + manifests: + - name: manifest + namespace: manifest-ns + files: + - a-file.yaml + kustomizeAllowAnyDirectory: false + kustomizations: + - a-kustomization.yaml + wait: false + template: true # Enable go-template processing + charts: + - name: chart + namespace: chart-ns + releaseName: chart-release + wait: true + valuesFiles: + - values.yaml + values: + - sourcePath: ".app.name" + targetPath: ".appName" + schemaValidation: true + helmRepo: # Only one of helm, git, oci, url, or local is allowed + url: https://stefanprodan.github.io/podinfo + name: podinfo # replaces repoName since it's only applicable in this situation + version: 6.4.0 + git: + url: https://stefanprodan.github.io/podinfo + path: charts/podinfo + oci: + url: oci://ghcr.io/stefanprodan/charts/podinfo + version: 6.4.0 + local: + path: chart + files: + - source: source-file.txt + target: target-file.txt + shasum: shasum + executable: false + symlinks: + - /path/to/symlink + extractPath: /path/to/extract + template: false # Disable go-template processing for this file + images: + - podinfo@v1 + repos: + - https://github.com/defenseunicorns/zarf + actions: + onCreate: + defaults: + mute: true + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + shell: + darwin: sh + linux: sh + windows: powershell + before: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + after: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + onFailure: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + onDeploy: + defaults: + mute: true + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + shell: + darwin: sh + linux: sh + windows: powershell + before: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + after: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + onFailure: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + onRemove: + defaults: + mute: true + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + shell: + darwin: sh + linux: sh + windows: powershell + before: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + after: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 + onFailure: + - mute: false + timeout: 1m + retries: 0 + dir: dir + env: + - ENV_VAR=FOO + cmd: echo hello + shell: + darwin: sh + linux: sh + windows: powershell + setVariables: + - name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + setValues: + - key: .json + description: action-description + wait: + cluster: # Only one of cluster / network can be used + apiVersion: v1 + kind: pod + name: my-pod + namespace: pod-ns + condition: ready + network: + protocol: http + address: github.com + code: 200 +constants: +- name: CONSTANT + value: constant-value + description: constant-value + autoIndent: false + pattern: ".+" +variables: +- name: VAR + sensitive: false + autoIndent: true + pattern: ".+" + type: raw + description: var + default: whatever + prompt: false diff --git a/0051-v1beta1-schema/zep.yaml b/0051-v1beta1-schema/zep.yaml new file mode 100644 index 0000000..9d6cccf --- /dev/null +++ b/0051-v1beta1-schema/zep.yaml @@ -0,0 +1,29 @@ +schema-version: 1.0.0 + +title: v1beta1 schema +zep-number: 0051 +authors: + - "@austinabro321" +status: provisional +creation-date: 2025-10-21 +reviewers: + - "@zarf-dev" +approvers: + - "@zarf-dev" + +see-also: + - "/0048-schema-upgrade-process" + +# The target maturity stage in the current dev cycle for this ZEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this ZEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "TBD" + beta: "TBD" + stable: "TBD""