From 173e4930531b93230ebc3d01c05a933073a175f1 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Wed, 31 Aug 2022 07:38:21 +0200 Subject: [PATCH 01/35] List existing proposals --- proposals/ytt/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/ytt/README.md b/proposals/ytt/README.md index 31fcbb025..335cde8aa 100644 --- a/proposals/ytt/README.md +++ b/proposals/ytt/README.md @@ -3,3 +3,5 @@ Feature proposals for `ytt`. See each proposal for their status. - [Schemas](001-schemas/) +- [Raw data values](002-raw-data-values/) +- [OpenAPI Schemas](003-openapi-schemas/) From 3d6ac38d5618ad2874f9d977dc35350b5a269f7e Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Wed, 31 Aug 2022 07:38:44 +0200 Subject: [PATCH 02/35] Propose semver support in ytt's standard library --- proposals/ytt/005-semver/README.md | 217 +++++++++++++++++++++++++++++ proposals/ytt/README.md | 1 + 2 files changed, 218 insertions(+) create mode 100644 proposals/ytt/005-semver/README.md diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md new file mode 100644 index 000000000..35607ab89 --- /dev/null +++ b/proposals/ytt/005-semver/README.md @@ -0,0 +1,217 @@ +--- +title: "Semver support in the standard library" +authors: [ "Max Brauer " ] +status: "draft" +approvers: [ ? ] +--- + +# Semver + +## Problem Statement + +Configuration authors frequently encounter strings which denote the version of a +software. [Semantic versioning](https://semver.org/) is a widely adopted scheme for versioning +software. In this proposal the term "semver" is used both for _the_ scheme itself as well _an_ instance of a version. + +A semantic version is a string, e.g. `3.2.12`, `0.23.0-beta`, `1.99.2-next+build.5`, which contains structured +information about the version. Configuration authors may want +to read a version's constituents, mutate them and compare different versions. For example, when authoring `ytt` +templates: + +* I want to assert that a given string is a well-formed semver. +* I want to change a semver's representation from string another to data structure, e.g. object or array. +* I want to compare semvers, e.g. expect a given version to be larger than a known version. +* I want to (de,in)crement the major, minor and/or patch version of a semver. +* I want to add, change or remove a semver's prerelease identifier or metadata. + +Starlark's standard library provides the means to achieve this. However, it sets a high bar and burdens configuration +authors to write boilerplate code for common, industry standard operations. + +## Terminology / Concepts + +A semantic version encodes different properties about the version in a string of the +form `..(-)(+)`. + +Semantic versions are strictly sortable. + +For its full specification refer to [Semantic versioning](https://semver.org/). + +## Proposal + +The proposal is to include a built-in `@ytt:semver` library. Configuration authors can then load it and programmatically +work with strings representing semvers. + +The semver library contains means for turning a semver string into a structured representation. This can result in an +error in the case of a malformed semver. If the operation succeeds, it returns a `struct` representing the semver. + +Now, it is possible to read the _major_, _minor_, _patch_ versions as integers, as well as the _pre-release_ +version and _build metadata_ as a strings. This enables authors to change its representation at will. + +It is also possible to programmatically construct a semver `struct` by providing _major_, _minor_, _patch_ versions as +required integers, as well as the _pre-release_ version and _build metadata_ as optional strings. This enables authors +to construct semvers either from values known in advance or which are received in a representation other than a semver. + +Given two semver `structs`, they can be compared. This enables authors to test if a semver is large or small enough, +or whether it is within a range. + +For its implementation, the suggestion is to use [blang/semver](https://github.com/blang/semver). It is a +well-established module, contains an API to address all conceivable case and is already used by Carvel's `vendir` for +its `tools sort-semver` subcommand. + +### Goals and Non-goals + +* Enable configuration authors to programmatically work with semvers out-of-the-box. + +* Education around and proliferation of the semver scheme is not considered to be `ytt`'s responsibility. + +### Specification / Use Cases + +#### semver·version + +`semver.version([major=int, minor=int, patch=int, prerelease=string, metadata=string])` returns a struct representing a +semantic version. The fields `major`, `minor` and `patch` are strings and default to `0`. The fields `prerelease` and +`metadata` are strings and default to `None`. If any of the arguments are of the wrong type or contain forbidden +characters, it is a dynamic error. + +```yaml +#@ load("@ytt:semver", "semver") + +#@ semver.version() #! 0.0.0 +#@ semver.version(1, 2, 3) #! 1.2.3 +#@ semver.version(1, 2, 3, "beta") #! 1.2.3-beta +#@ semver.version(1, 2, 3, metadata="build.5") #! 1.2.3+build.5 +#@ semver.version(1, 2, 3, "next", "nightly") #! 1.2.3-next+nightly +#@ semver.version(minor=1, prerelease="alpha") #! 0.1.0-alpha +#@ semver.version(1, 2, -5) #! ⚡️ error +#@ semver.version(prerelease="$send-ytt-coins$") #! ⚡️ error +#@ semver.version(meta="__dunder-score") #! ⚡️ error +``` + +Use cases: + +```yaml +#@ load("@ytt:semver", "semver") +#@ load("@ytt:yaml", "yaml") +#@ load("@ytt:json", "json") +#@ load("@ytt:assert", "assert") + +#@ v = semver.version(1, 2, 3) +#@ d = dict(major=v.major, minor=v.minor, patch=v.patch, prerelease=v.prerelease, metadata=v.metadata) + +--- #@ d + +--- +version.yaml: #@ yaml.encode(d) +version.json: #@ json.encode(d) + +#@ d.update(major=d["major"]+1) +#@ next_major = semver.version(**d) + +#@ if v.prerelease: +#@ assert.fail("Sorry, we don't do prereleases") +#@ end + +#@ forbidden = ["nightly", "dev"] +#@ if v.metadata in forbidden: +#@ assert.fail("Sorry, that version is forbidden") +#@ end +``` + +#### semver·from_str + +`semver.from_str(s)` parses `s` as a semantic version and returns its `semver.version()` struct representation. If `s` +is malformed, it is a dynamic error. + +```yaml +#@ load("@ytt:semver", "semver") + +#@ semver.from_str("1.2.3") #! semver.version(1, 2, 3) +#@ semver.from_str("1.2.3-pre") #! semver.version(1, 2, 3, "pre") +#@ semver.from_str("1.2.3+meta") #! semver.version(1, 2, 3, metadata="meta") +#@ semver.from_str("1.2.3-pre+meta") #! semver.version(1, 2, 3, "pre", "meta") +#@ semver.from_str("1.2.-5") #! ⚡️ error +#@ semver.from_str("Homer Simpson") #! ⚡️ error +``` + +#### version·cmp + +`version.cmp(other)` compares two semantic versions and returns `0` if they are equal, `-1` if `other` is greater +and `1` if other is smaller. If `other` is not a `version` struct, it is a dynamic error. + +Usage of `version.(eq, lt, lte, gt, gte)` is recommended instead. However, there may be cases where a numeric return +value is preferred. + +```yaml +#@ load("@ytt:semver", "semver") + +#@ semver.version().cmp(semver.version()) #! 0 +#@ semver.version(1).cmp(semver.version(2)) #! -1 +#@ semver.version(1, 2, 3).cmp(semver.version(2, 0, 3)) #! 1 +#@ semver.version(1, 2, 3).cmp(dict()) #! ⚡️ error +``` + +#### version·(eq, lt, lte, gt, gte) + +All of `version.eq(version)`, `version.lt(version)`, `version.lte(version)`, `version.gt(version)` +and `version.gte(version)` compare `version` with `other` and return `True` if the operator holds and `False` otherwise. +If `other` is not a `version` struct, it is a dynamic error. + +```yaml +#@ load("@ytt:data", "data") +#@ load("@ytt:semver", "semver") +#@ load("@ytt:assert", "assert") + +#@ v1_2_3 = semver.version(1, 2, 3) +#@ given_version = semver.from_str(data.values.version) + +--- +cool_config: nice +#@ if/end given_version.lt(v1_2_3) +ye_olde_confyg: they don't make 'em like this any more + +#@ if given_version.gte(semver.version(10, 20, 30)) +#@ assert.fail("Sorry, but that's too hot for us") +#@ end +``` + +#### version·to_str + +`version.to_str()` returns the string representation of the `version` struct. + +```yaml +#@ load("@ytt:semver", "semver") + +#@ semver.version().to_str() #! "0.0.0" +#@ semver.version(1, 2, 3).to_str() #! "1.2.3" +#@ semver.version(1, 2, 3, "beta").to_str() #! "1.2.3-beta" +#@ semver.version(1, 2, 3, metadata="build.5").to_str() #! "1.2.3+build.5" +#@ semver.version(1, 2, 3, "next", "nightly").to_str() #! "1.2.3-next+nightly" +#@ semver.version(minor=1, prerelease="alpha").to_str() #! "0.1.0-alpha" +``` + +Use cases: + +```yaml +#@ load("@ytt:semver", "semver") + +#@ all_the_minor_versions = [semver.version(minor=i).to_str() for i in range(10)] +--- +allowed_versions: #@ all_the_minor_versions +``` + +### Other Approaches Considered + +* A Github-hosted Starlark library like `awesome-ytt/semver` containing a similar API could be fetched with `vendir` and + then loaded. This wouldn't in reach for the widest possible spectrum of configuration authors. It would require to + reimplement what existing, established Go modules already have done. + +## Open Questions + +* There's a host of conceivable additional operations available for semver, e.g. test against range, sort list of + versions, bumping, etc. It is debatable if that should be part of the standard library. Testing against a range and + sorting are likelier candidates for a standard library. Bumping is possible with the primitives presented. +* Maybe configuration authors would benefit from a `version.to_dict()`(or even `version.to_list()` + and `version.to_tuple()`) to ease the transformation into a serializable data structure. On the other hand, at least + in the case of `version.to_dict()` the author may disagree with the keys. + +## Answered Questions diff --git a/proposals/ytt/README.md b/proposals/ytt/README.md index 335cde8aa..c575d9562 100644 --- a/proposals/ytt/README.md +++ b/proposals/ytt/README.md @@ -5,3 +5,4 @@ Feature proposals for `ytt`. See each proposal for their status. - [Schemas](001-schemas/) - [Raw data values](002-raw-data-values/) - [OpenAPI Schemas](003-openapi-schemas/) +- [Semver](005-semver/) From d669d327f091acb4ed86e8e2417920a74b8c0fa6 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Fri, 2 Sep 2022 08:02:48 +0200 Subject: [PATCH 03/35] List Dmitriy and John as approvers --- proposals/ytt/005-semver/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index 35607ab89..91e1d282c 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -2,7 +2,7 @@ title: "Semver support in the standard library" authors: [ "Max Brauer " ] status: "draft" -approvers: [ ? ] +approvers: [ "Dmitriy Kalinin ", "John Ryan " ] --- # Semver From b45faee2826bf800c0c89ff12189ec78c3f60c87 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Sun, 11 Sep 2022 08:09:02 +0200 Subject: [PATCH 04/35] Improve prose --- proposals/ytt/005-semver/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index 91e1d282c..d83180538 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -19,7 +19,7 @@ to read a version's constituents, mutate them and compare different versions. Fo templates: * I want to assert that a given string is a well-formed semver. -* I want to change a semver's representation from string another to data structure, e.g. object or array. +* I want to change a semver's representation from string to a data structure of my choice, e.g. object or array. * I want to compare semvers, e.g. expect a given version to be larger than a known version. * I want to (de,in)crement the major, minor and/or patch version of a semver. * I want to add, change or remove a semver's prerelease identifier or metadata. From 7c951fca3e1d45ef6ec0254fc47f4c0e4d9f7fe5 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Wed, 14 Sep 2022 08:39:20 +0200 Subject: [PATCH 05/35] Include version bumping and range testing --- proposals/ytt/005-semver/README.md | 134 +++++++++++++++++++++++++---- 1 file changed, 118 insertions(+), 16 deletions(-) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index d83180538..65827a2e1 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -64,7 +64,31 @@ its `tools sort-semver` subcommand. * Education around and proliferation of the semver scheme is not considered to be `ytt`'s responsibility. -### Specification / Use Cases +### Specification + +The built-in `@ytt:semver` library exposes the following API: + +* [semver·version](#semverversion) +* [semver·from_str](#semverfrom_str) +* [version·to_str](#versionto_str) +* [version·cmp](#versioncmp) +* [version·(lt, lte, gt, gte)](#versioneq-lt-lte-gt-gte) +* [version·(next_major, next_minor, next_patch)](#versionnext_major-next_minor-next_patch) +* [semver·range](#semverrange) +* [range·contains](#rangecontains) +* [range·to_str](#rangeto_str) + +The library can be loaded like so: + +```yaml +#! config.yaml +#@ load("@ytt:semver", "semver") +``` + +```star +# config.star +load("@ytt:semver", "semver") +``` #### semver·version @@ -133,6 +157,31 @@ is malformed, it is a dynamic error. #@ semver.from_str("Homer Simpson") #! ⚡️ error ``` +#### version·to_str + +`version.to_str()` returns the string representation of the `version` struct. + +```yaml +#@ load("@ytt:semver", "semver") + +#@ semver.version().to_str() #! "0.0.0" +#@ semver.version(1, 2, 3).to_str() #! "1.2.3" +#@ semver.version(1, 2, 3, "beta").to_str() #! "1.2.3-beta" +#@ semver.version(1, 2, 3, metadata="build.5").to_str() #! "1.2.3+build.5" +#@ semver.version(1, 2, 3, "next", "nightly").to_str() #! "1.2.3-next+nightly" +#@ semver.version(minor=1, prerelease="alpha").to_str() #! "0.1.0-alpha" +``` + +Use cases: + +```yaml +#@ load("@ytt:semver", "semver") + +#@ all_the_minor_versions = [semver.version(minor=i).to_str() for i in range(10)] +--- +allowed_versions: #@ all_the_minor_versions +``` + #### version·cmp `version.cmp(other)` compares two semantic versions and returns `0` if they are equal, `-1` if `other` is greater @@ -174,31 +223,83 @@ ye_olde_confyg: they don't make 'em like this any more #@ end ``` -#### version·to_str +#### version·(next_major, next_minor, next_patch) -`version.to_str()` returns the string representation of the `version` struct. +All of `version.next_major()`, `version.next_minor()` and `version.next_patch()` return a new `version` and bump the +respective components. Existing `prerelease` and `metadata` are reset. ```yaml #@ load("@ytt:semver", "semver") -#@ semver.version().to_str() #! "0.0.0" -#@ semver.version(1, 2, 3).to_str() #! "1.2.3" -#@ semver.version(1, 2, 3, "beta").to_str() #! "1.2.3-beta" -#@ semver.version(1, 2, 3, metadata="build.5").to_str() #! "1.2.3+build.5" -#@ semver.version(1, 2, 3, "next", "nightly").to_str() #! "1.2.3-next+nightly" -#@ semver.version(minor=1, prerelease="alpha").to_str() #! "0.1.0-alpha" +#@ semver.from_str("1.2.3").next_patch() #! semver.version(1, 2, 4) +#@ semver.from_str("1.2.3").next_minor() #! semver.version(1, 3, 0) +#@ semver.from_str("1.2.3").next_major() #! semver.version(2, 0, 0) +#@ semver.from_str("1.2.3-alpha").next_major() #! semver.version(2, 0, 0) +#@ semver.from_str("1.2.3+nightly").next_minor() #! semver.version(1, 3, 0) +#@ semver.from_str("1.2.3-beta+dev").next_patch() #! semver.version(1, 2, 4) ``` -Use cases: +#### semver·range + +`semver.range(range=string)` returns a struct representing a range of semantic versions. The struct has no fields. If +the range argument is not a `string` or cannot be parsed as a range of semantic versions, it is a dynamic error. + +Valid ranges are all strings which are accepted +by [blang/semver@v4.range.ParseRange](https://pkg.go.dev/github.com/blang/semver/v4?utm_source=godoc#ParseRange) without +error. ```yaml #@ load("@ytt:semver", "semver") -#@ all_the_minor_versions = [semver.version(minor=i).to_str() for i in range(10)] ---- -allowed_versions: #@ all_the_minor_versions +#@ semver.range("<1.0.0") #! less than 1.0.0 +#@ semver.range("<=1.0.0") #! less than or equal 1.0.0 +#@ semver.range(">1.0.0") #! larger than 1.0.0 +#@ semver.range(">=1.0.0") #! larger than or equal 1.0.0 +#@ semver.range("1.0.0") #! equal 1.0.0 +#@ semver.range("=1.0.0") #! equal 1.0.0 +#@ semver.range("==1.0.0") #! equal 1.0.0 +#@ semver.range("!1.0.0") #! not equal 1.0.0 +#@ semver.range("!=1.0.0") #! not equal 1.0.0 +#@ semver.range(">1.0.0 <2.0.0") #! between 1.0.0 and 2.0.0 +#@ semver.range(">1.0.0 <3.0.0 !2.0.3-beta.2") #! between 1.0.0 and 3.0.0 but not equal 2.0.3-beta.2 +#@ semver.range("<2.0.0 || >=3.0.0") #! less than 2.0.0 or larger than or equal 3.0.0 +#@ semver.range(">1.0.0 <2.0.0 || >3.0.0 !4.2.1") #! ... +#@ semver.range("~1.2.3") #! ⚡️ error +#@ semver.range("not a range") #! ⚡️ error ``` +#### range·contains + +`range.contains(version=version)` returns `True` if `range` contains the given `version` and `False` otherwise. + +```yaml +#@ load("@ytt:semver", "semver") + +#@ semver.range("<1.0.0").contains(semver.from_str("0.1.0)) #! True +#@ semver.range("<1.0.0").contains(semver.from_str("1.0.0)) #! False +#@ semver.range(">1.0.0 <2.0.0").contains(semver.from_str("1.23.45)) #! True +#@ semver.range(">1.0.0 <3.0.0 !2.0.3-beta.2").contains(semver.from_str("2.0.3-beta.2")) #! False +#@ semver.range(">1.0.0 <3.0.0 !2.0.3-beta.2").contains(semver.from_str("2.0.3")) #! True +``` + +#### range·to_str + +`range.to_str()` returns the initially given string representation of `range`. + +```yaml +#@ load("@ytt:semver", "semver") + +#@ semver.range("<1.0.0").to_str() #! "<1.0.0" +#@ semver.range(">=1.0.0").to_str() #! ">=1.0.0" +#@ semver.range("1.0.0").to_str() #! "1.0.0" +#@ semver.range("=1.0.0").to_str() #! "=1.0.0" +#@ semver.range("==1.0.0").to_str() #! "==1.0.0" +#@ semver.range(">1.0.0 <2.0.0").to_str() #! ">1.0.0 <2.0.0" +#@ semver.range(">1.0.0 <3.0.0 !2.0.3-beta.2").to_str() #! ">1.0.0 <3.0.0 !2.0.3-beta.2" +``` + +### Use Cases + ### Other Approaches Considered * A Github-hosted Starlark library like `awesome-ytt/semver` containing a similar API could be fetched with `vendir` and @@ -207,9 +308,10 @@ allowed_versions: #@ all_the_minor_versions ## Open Questions -* There's a host of conceivable additional operations available for semver, e.g. test against range, sort list of - versions, bumping, etc. It is debatable if that should be part of the standard library. Testing against a range and - sorting are likelier candidates for a standard library. Bumping is possible with the primitives presented. +* There's a host of conceivable additional operations available for semver, e.g. sort list of + versions, bumping prereleases (1.2.3-build.5 → 1.2.3-build.6), etc. It is debatable if that should be part of the + built-in library. If the primitives provided by the library are expressive enough, then authors should be able to + cover these use cases by themselves. * Maybe configuration authors would benefit from a `version.to_dict()`(or even `version.to_list()` and `version.to_tuple()`) to ease the transformation into a serializable data structure. On the other hand, at least in the case of `version.to_dict()` the author may disagree with the keys. From d9da438b3a13261603016b72f8b80173574cd980 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Wed, 14 Sep 2022 07:40:41 +0200 Subject: [PATCH 06/35] Rephrase config author's needs --- proposals/ytt/005-semver/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index 65827a2e1..1a3d1cd86 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -19,7 +19,7 @@ to read a version's constituents, mutate them and compare different versions. Fo templates: * I want to assert that a given string is a well-formed semver. -* I want to change a semver's representation from string to a data structure of my choice, e.g. object or array. +* I want to access a semver's components so that I can easily change its representation. * I want to compare semvers, e.g. expect a given version to be larger than a known version. * I want to (de,in)crement the major, minor and/or patch version of a semver. * I want to add, change or remove a semver's prerelease identifier or metadata. From 81aa1dd86ce3c7f29369321f5fc9a538d0f40862 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Wed, 14 Sep 2022 07:41:33 +0200 Subject: [PATCH 07/35] Rephrase semver sorting --- proposals/ytt/005-semver/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index 1a3d1cd86..432d212de 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -32,7 +32,7 @@ authors to write boilerplate code for common, industry standard operations. A semantic version encodes different properties about the version in a string of the form `..(-)(+)`. -Semantic versions are strictly sortable. +Semantic versions can be sorted based on [precedence](https://semver.org/#spec-item-11). For its full specification refer to [Semantic versioning](https://semver.org/). From 2781093d8957a02acf5510d1691d93f1077f8843 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Wed, 14 Sep 2022 08:46:14 +0200 Subject: [PATCH 08/35] Add todo for collecting use cases --- proposals/ytt/005-semver/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index 432d212de..a9ccd6712 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -300,6 +300,11 @@ error. ### Use Cases +**todo** collect ^ use cases here and make them prettier, and more specific + +**todo** add use cases / needs from [kapp-controller's in-development downward API](https://hackmd.io/6FQhzRIPR_uceqtTqSIXaA) + + ### Other Approaches Considered * A Github-hosted Starlark library like `awesome-ytt/semver` containing a similar API could be fetched with `vendir` and From 05d938f0aa4bcf7d9121f6d506abc519e6ef87b7 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Wed, 14 Sep 2022 08:48:08 +0200 Subject: [PATCH 09/35] =?UTF-8?q?Rename=20field=20'version.{metadata=20?= =?UTF-8?q?=E2=86=92=20build}'?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- proposals/ytt/005-semver/README.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index a9ccd6712..c18cf6c23 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -22,7 +22,7 @@ templates: * I want to access a semver's components so that I can easily change its representation. * I want to compare semvers, e.g. expect a given version to be larger than a known version. * I want to (de,in)crement the major, minor and/or patch version of a semver. -* I want to add, change or remove a semver's prerelease identifier or metadata. +* I want to add, change or remove a semver's prerelease identifier or build metadata. Starlark's standard library provides the means to achieve this. However, it sets a high bar and burdens configuration authors to write boilerplate code for common, industry standard operations. @@ -30,7 +30,7 @@ authors to write boilerplate code for common, industry standard operations. ## Terminology / Concepts A semantic version encodes different properties about the version in a string of the -form `..(-)(+)`. +form `..(-)(+)`. Semantic versions can be sorted based on [precedence](https://semver.org/#spec-item-11). @@ -92,9 +92,9 @@ load("@ytt:semver", "semver") #### semver·version -`semver.version([major=int, minor=int, patch=int, prerelease=string, metadata=string])` returns a struct representing a +`semver.version([major=int, minor=int, patch=int, prerelease=string, build=string])` returns a struct representing a semantic version. The fields `major`, `minor` and `patch` are strings and default to `0`. The fields `prerelease` and -`metadata` are strings and default to `None`. If any of the arguments are of the wrong type or contain forbidden +`build` are strings and default to `None`. If any of the arguments are of the wrong type or contain forbidden characters, it is a dynamic error. ```yaml @@ -103,7 +103,7 @@ characters, it is a dynamic error. #@ semver.version() #! 0.0.0 #@ semver.version(1, 2, 3) #! 1.2.3 #@ semver.version(1, 2, 3, "beta") #! 1.2.3-beta -#@ semver.version(1, 2, 3, metadata="build.5") #! 1.2.3+build.5 +#@ semver.version(1, 2, 3, build="build.5") #! 1.2.3+build.5 #@ semver.version(1, 2, 3, "next", "nightly") #! 1.2.3-next+nightly #@ semver.version(minor=1, prerelease="alpha") #! 0.1.0-alpha #@ semver.version(1, 2, -5) #! ⚡️ error @@ -120,7 +120,7 @@ Use cases: #@ load("@ytt:assert", "assert") #@ v = semver.version(1, 2, 3) -#@ d = dict(major=v.major, minor=v.minor, patch=v.patch, prerelease=v.prerelease, metadata=v.metadata) +#@ d = dict(major=v.major, minor=v.minor, patch=v.patch, prerelease=v.prerelease, build=v.build) --- #@ d @@ -136,7 +136,7 @@ version.json: #@ json.encode(d) #@ end #@ forbidden = ["nightly", "dev"] -#@ if v.metadata in forbidden: +#@ if v.build in forbidden: #@ assert.fail("Sorry, that version is forbidden") #@ end ``` @@ -151,7 +151,7 @@ is malformed, it is a dynamic error. #@ semver.from_str("1.2.3") #! semver.version(1, 2, 3) #@ semver.from_str("1.2.3-pre") #! semver.version(1, 2, 3, "pre") -#@ semver.from_str("1.2.3+meta") #! semver.version(1, 2, 3, metadata="meta") +#@ semver.from_str("1.2.3+meta") #! semver.version(1, 2, 3, build="meta") #@ semver.from_str("1.2.3-pre+meta") #! semver.version(1, 2, 3, "pre", "meta") #@ semver.from_str("1.2.-5") #! ⚡️ error #@ semver.from_str("Homer Simpson") #! ⚡️ error @@ -167,7 +167,7 @@ is malformed, it is a dynamic error. #@ semver.version().to_str() #! "0.0.0" #@ semver.version(1, 2, 3).to_str() #! "1.2.3" #@ semver.version(1, 2, 3, "beta").to_str() #! "1.2.3-beta" -#@ semver.version(1, 2, 3, metadata="build.5").to_str() #! "1.2.3+build.5" +#@ semver.version(1, 2, 3, build="build.5").to_str() #! "1.2.3+build.5" #@ semver.version(1, 2, 3, "next", "nightly").to_str() #! "1.2.3-next+nightly" #@ semver.version(minor=1, prerelease="alpha").to_str() #! "0.1.0-alpha" ``` @@ -226,7 +226,7 @@ ye_olde_confyg: they don't make 'em like this any more #### version·(next_major, next_minor, next_patch) All of `version.next_major()`, `version.next_minor()` and `version.next_patch()` return a new `version` and bump the -respective components. Existing `prerelease` and `metadata` are reset. +respective components. Existing `prerelease` and `build` are reset. ```yaml #@ load("@ytt:semver", "semver") From ab2c40e06a89b64e367efd8f8a6c66a457c0a1ea Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Fri, 23 Sep 2022 15:26:21 +0200 Subject: [PATCH 10/35] Iterate on proposal with use cases, touches and cosmetics --- proposals/ytt/005-semver/README.md | 386 ++++++++++++++++++++--------- 1 file changed, 267 insertions(+), 119 deletions(-) diff --git a/proposals/ytt/005-semver/README.md b/proposals/ytt/005-semver/README.md index c18cf6c23..51f25dcaa 100644 --- a/proposals/ytt/005-semver/README.md +++ b/proposals/ytt/005-semver/README.md @@ -1,36 +1,59 @@ --- title: "Semver support in the standard library" -authors: [ "Max Brauer " ] +authors: ["Max Brauer "] status: "draft" -approvers: [ "Dmitriy Kalinin ", "John Ryan " ] +approvers: ["Dmitriy Kalinin ", "John Ryan "] --- # Semver +- [Semver](#semver) + - [Problem Statement](#problem-statement) + - [Terminology / Concepts](#terminology--concepts) + - [Proposal](#proposal) + - [Goals](#goals) + - [Non-goals](#non-goals) + - [Specification](#specification) + - [semver·version](#semverversion) + - [semver·from_str](#semverfrom_str) + - [version·to_str](#versionto_str) + - [version·cmp](#versioncmp) + - [version·(eq, lt, lte, gt, gte)](#versioneq-lt-lte-gt-gte) + - [version·(next_major, next_minor, next_patch)](#versionnext_major-next_minor-next_patch) + - [semver·range](#semverrange) + - [range·contains](#rangecontains) + - [range·to_str](#rangeto_str) + - [Use Cases](#use-cases) + - [Data value validation](#data-value-validation) + - [_kapp-controller_'s _Downward_ API](#kapp-controllers-downward-api) + - [Drive releases with _conventional commits_](#drive-releases-with-conventional-commits) + - [Considered Alternatives](#considered-alternatives) + - [Open Questions](#open-questions) + - [Answered Questions](#answered-questions) + ## Problem Statement -Configuration authors frequently encounter strings which denote the version of a -software. [Semantic versioning](https://semver.org/) is a widely adopted scheme for versioning -software. In this proposal the term "semver" is used both for _the_ scheme itself as well _an_ instance of a version. +Configuration authors frequently encounter strings which denote the version of a software. +[Semantic versioning](https://semver.org/) is a widely adopted scheme for versioning software. In this proposal the term +"semver" is used both for _the_ scheme itself as well _an_ instance of a version. A semantic version is a string, e.g. `3.2.12`, `0.23.0-beta`, `1.99.2-next+build.5`, which contains structured -information about the version. Configuration authors may want -to read a version's constituents, mutate them and compare different versions. For example, when authoring `ytt` -templates: +information about the version. Configuration authors may want to read a version's constituents, mutate them and compare +different versions. For example, when authoring `ytt` templates: -* I want to assert that a given string is a well-formed semver. -* I want to access a semver's components so that I can easily change its representation. -* I want to compare semvers, e.g. expect a given version to be larger than a known version. -* I want to (de,in)crement the major, minor and/or patch version of a semver. -* I want to add, change or remove a semver's prerelease identifier or build metadata. +- I want to assert that a given string is a well-formed semver. +- I want to access a semver's components so that I can easily change its representation. +- I want to compare semvers, e.g. expect a given version to be larger than a known version. +- I want to (de,in)crement the major, minor and/or patch version of a semver. +- I want to add, change or remove a semver's prerelease identifier or build metadata. Starlark's standard library provides the means to achieve this. However, it sets a high bar and burdens configuration authors to write boilerplate code for common, industry standard operations. ## Terminology / Concepts -A semantic version encodes different properties about the version in a string of the -form `..(-)(+)`. +A semantic version encodes different properties about the version in a string of the form +`..(-)(+)`. Semantic versions can be sorted based on [precedence](https://semver.org/#spec-item-11). @@ -39,46 +62,38 @@ For its full specification refer to [Semantic versioning](https://semver.org/). ## Proposal The proposal is to include a built-in `@ytt:semver` library. Configuration authors can then load it and programmatically -work with strings representing semvers. +work with strings representing semantic verssions. -The semver library contains means for turning a semver string into a structured representation. This can result in an -error in the case of a malformed semver. If the operation succeeds, it returns a `struct` representing the semver. +The library contains means for turning a semver string into a structured representation. This can result in an error in +the case of a malformed semver. If the operation succeeds, it returns a `struct` representing the semver. -Now, it is possible to read the _major_, _minor_, _patch_ versions as integers, as well as the _pre-release_ -version and _build metadata_ as a strings. This enables authors to change its representation at will. +Now, it is possible to read the _major_, _minor_, _patch_ versions as integers, as well as the _pre-release_ version and +_build metadata_ as a strings. This enables authors to change its representation at will. It is also possible to programmatically construct a semver `struct` by providing _major_, _minor_, _patch_ versions as required integers, as well as the _pre-release_ version and _build metadata_ as optional strings. This enables authors to construct semvers either from values known in advance or which are received in a representation other than a semver. -Given two semver `structs`, they can be compared. This enables authors to test if a semver is large or small enough, -or whether it is within a range. - -For its implementation, the suggestion is to use [blang/semver](https://github.com/blang/semver). It is a -well-established module, contains an API to address all conceivable case and is already used by Carvel's `vendir` for -its `tools sort-semver` subcommand. +Given two semver `structs`, they can be compared. This enables authors to test if a semver is large or small enough, or +whether it is within a range. -### Goals and Non-goals +For its implementation, the suggestion is to use [k14s/semver](https://github.com/k14s/semver), a Carvel-owned fork of +[blang/semver](https://github.com/blang/semver) It is a well-established module, contains an API to address all +conceivable case and is already used by Carvel's `vendir` for its `tools sort-semver` subcommand. -* Enable configuration authors to programmatically work with semvers out-of-the-box. +### Goals -* Education around and proliferation of the semver scheme is not considered to be `ytt`'s responsibility. +- Enable configuration authors to programmatically work with semvers out-of-the-box. +- Use and promote terminology and APIs consistent with existing concepts in other Carvel tools, e.g. _kapp-controller_'s [constraints](https://carvel.dev/kapp-controller/docs/develop/package-consumer-concepts/#constraints) +- Where other Carvel tools deviate from the semver spec, be consistent with them, e.g. _vendir_'s [semver](https://carvel.dev/vendir/docs/v0.31.0/versions/#semver) constraint -### Specification +### Non-goals -The built-in `@ytt:semver` library exposes the following API: +- Education around and proliferation of the semver scheme is not considered to be `ytt`'s responsibility. -* [semver·version](#semverversion) -* [semver·from_str](#semverfrom_str) -* [version·to_str](#versionto_str) -* [version·cmp](#versioncmp) -* [version·(lt, lte, gt, gte)](#versioneq-lt-lte-gt-gte) -* [version·(next_major, next_minor, next_patch)](#versionnext_major-next_minor-next_patch) -* [semver·range](#semverrange) -* [range·contains](#rangecontains) -* [range·to_str](#rangeto_str) +### Specification -The library can be loaded like so: +As is the case for all `ytt` libraries, it can be loaded by YAML-embedded or pure _Starlark_ programs: ```yaml #! config.yaml @@ -103,7 +118,7 @@ characters, it is a dynamic error. #@ semver.version() #! 0.0.0 #@ semver.version(1, 2, 3) #! 1.2.3 #@ semver.version(1, 2, 3, "beta") #! 1.2.3-beta -#@ semver.version(1, 2, 3, build="build.5") #! 1.2.3+build.5 +#@ semver.version(1, 2, 3, build="build.5") #! 1.2.3+build.5 #@ semver.version(1, 2, 3, "next", "nightly") #! 1.2.3-next+nightly #@ semver.version(minor=1, prerelease="alpha") #! 0.1.0-alpha #@ semver.version(1, 2, -5) #! ⚡️ error @@ -111,36 +126,6 @@ characters, it is a dynamic error. #@ semver.version(meta="__dunder-score") #! ⚡️ error ``` -Use cases: - -```yaml -#@ load("@ytt:semver", "semver") -#@ load("@ytt:yaml", "yaml") -#@ load("@ytt:json", "json") -#@ load("@ytt:assert", "assert") - -#@ v = semver.version(1, 2, 3) -#@ d = dict(major=v.major, minor=v.minor, patch=v.patch, prerelease=v.prerelease, build=v.build) - ---- #@ d - ---- -version.yaml: #@ yaml.encode(d) -version.json: #@ json.encode(d) - -#@ d.update(major=d["major"]+1) -#@ next_major = semver.version(**d) - -#@ if v.prerelease: -#@ assert.fail("Sorry, we don't do prereleases") -#@ end - -#@ forbidden = ["nightly", "dev"] -#@ if v.build in forbidden: -#@ assert.fail("Sorry, that version is forbidden") -#@ end -``` - #### semver·from_str `semver.from_str(s)` parses `s` as a semantic version and returns its `semver.version()` struct representation. If `s` @@ -172,20 +157,10 @@ is malformed, it is a dynamic error. #@ semver.version(minor=1, prerelease="alpha").to_str() #! "0.1.0-alpha" ``` -Use cases: - -```yaml -#@ load("@ytt:semver", "semver") - -#@ all_the_minor_versions = [semver.version(minor=i).to_str() for i in range(10)] ---- -allowed_versions: #@ all_the_minor_versions -``` - #### version·cmp -`version.cmp(other)` compares two semantic versions and returns `0` if they are equal, `-1` if `other` is greater -and `1` if other is smaller. If `other` is not a `version` struct, it is a dynamic error. +`version.cmp(other)` compares two semantic versions and returns `0` if they are equal, `-1` if `other` is greater and +`1` if other is smaller. If `other` is not a `version` struct, it is a dynamic error. Usage of `version.(eq, lt, lte, gt, gte)` is recommended instead. However, there may be cases where a numeric return value is preferred. @@ -201,26 +176,26 @@ value is preferred. #### version·(eq, lt, lte, gt, gte) -All of `version.eq(version)`, `version.lt(version)`, `version.lte(version)`, `version.gt(version)` -and `version.gte(version)` compare `version` with `other` and return `True` if the operator holds and `False` otherwise. -If `other` is not a `version` struct, it is a dynamic error. +All of `version.eq(version)`, `version.lt(version)`, `version.lte(version)`, `version.gt(version)` and +`version.gte(version)` compare `version` with `other` and return `True` if the operator holds and `False` otherwise. If +`other` is not a `version` struct, it is a dynamic error. ```yaml -#@ load("@ytt:data", "data") #@ load("@ytt:semver", "semver") -#@ load("@ytt:assert", "assert") -#@ v1_2_3 = semver.version(1, 2, 3) -#@ given_version = semver.from_str(data.values.version) - ---- -cool_config: nice -#@ if/end given_version.lt(v1_2_3) -ye_olde_confyg: they don't make 'em like this any more - -#@ if given_version.gte(semver.version(10, 20, 30)) -#@ assert.fail("Sorry, but that's too hot for us") -#@ end +#@ semver.version().eq(semver.version()) #! True +#@ semver.from_str("1.2.3-pre+build")).eq(semver.version(1, 2, 3, "pre", "build")) #! True +#@ semver.version(1, 2, 3).eq(semver.version(3, 2, 1)) #! False +#@ semver.version(1, 2, 3).lt(semver.version(3, 2, 1)) #! True +#@ semver.version(1, 2, 3).lt(semver.version(1, 2, 3)) #! False +#@ semver.version(1, 2, 3).lte(semver.version(3, 2, 1)) #! True +#@ semver.version(1, 2, 3).lte(semver.version(1, 2, 3)) #! True +#@ semver.version(1, 2, 3).lte(semver.version(1, 0, 0)) #! False +#@ semver.version(3, 2, 1).gt(semver.version(1, 2, 3)) #! True +#@ semver.version(3, 2, 1).gt(semver.version(4, 0, 0)) #! False +#@ semver.version(3, 2, 1).gte(semver.version(1, 2, 3)) #! True +#@ semver.version(3, 2, 1).gte(semver.version(3, 2, 1)) #! True +#@ semver.version(3, 2, 1).gte(semver.version(4, 0, 0)) #! False ``` #### version·(next_major, next_minor, next_patch) @@ -244,15 +219,15 @@ respective components. Existing `prerelease` and `build` are reset. `semver.range(range=string)` returns a struct representing a range of semantic versions. The struct has no fields. If the range argument is not a `string` or cannot be parsed as a range of semantic versions, it is a dynamic error. -Valid ranges are all strings which are accepted -by [blang/semver@v4.range.ParseRange](https://pkg.go.dev/github.com/blang/semver/v4?utm_source=godoc#ParseRange) without +Valid ranges are all strings which are accepted by +[range.ParseRange](https://pkg.go.dev/github.com/k14s/semver/v4?utm_source=godoc#ParseRange) without error. ```yaml #@ load("@ytt:semver", "semver") -#@ semver.range("<1.0.0") #! less than 1.0.0 -#@ semver.range("<=1.0.0") #! less than or equal 1.0.0 +#@ semver.range("<1.0.0") #! less than 1.0.0 +#@ semver.range("<=1.0.0") #! less than or equal 1.0.0 #@ semver.range(">1.0.0") #! larger than 1.0.0 #@ semver.range(">=1.0.0") #! larger than or equal 1.0.0 #@ semver.range("1.0.0") #! equal 1.0.0 @@ -275,8 +250,8 @@ error. ```yaml #@ load("@ytt:semver", "semver") -#@ semver.range("<1.0.0").contains(semver.from_str("0.1.0)) #! True -#@ semver.range("<1.0.0").contains(semver.from_str("1.0.0)) #! False +#@ semver.range("<1.0.0").contains(semver.from_str("0.1.0)) #! True +#@ semver.range("<1.0.0").contains(semver.from_str("1.0.0)) #! False #@ semver.range(">1.0.0 <2.0.0").contains(semver.from_str("1.23.45)) #! True #@ semver.range(">1.0.0 <3.0.0 !2.0.3-beta.2").contains(semver.from_str("2.0.3-beta.2")) #! False #@ semver.range(">1.0.0 <3.0.0 !2.0.3-beta.2").contains(semver.from_str("2.0.3")) #! True @@ -289,9 +264,9 @@ error. ```yaml #@ load("@ytt:semver", "semver") -#@ semver.range("<1.0.0").to_str() #! "<1.0.0" -#@ semver.range(">=1.0.0").to_str() #! ">=1.0.0" -#@ semver.range("1.0.0").to_str() #! "1.0.0" +#@ semver.range("<1.0.0").to_str() #! "<1.0.0" +#@ semver.range(">=1.0.0").to_str() #! ">=1.0.0" +#@ semver.range("1.0.0").to_str() #! "1.0.0" #@ semver.range("=1.0.0").to_str() #! "=1.0.0" #@ semver.range("==1.0.0").to_str() #! "==1.0.0" #@ semver.range(">1.0.0 <2.0.0").to_str() #! ">1.0.0 <2.0.0" @@ -302,23 +277,196 @@ error. **todo** collect ^ use cases here and make them prettier, and more specific -**todo** add use cases / needs from [kapp-controller's in-development downward API](https://hackmd.io/6FQhzRIPR_uceqtTqSIXaA) +The presented use cases demonstrate the proposed API's utility in different templating scenarios. +The scenarios aspire to be representative but not exhaustive of all conveivable applications. + +#### Data value validation + +Use `@ytt:semver` to validate and constrain a data value which represents a semantic version. + +```yaml +#! schema.yaml + +#@ load("@ytt:data", "data") +#@ load("@ytt:semver", "semver") +#@ load("@ytt:assert", "assert") + +#@ compatible_versions = semver.range("<3.0.0 >=2.0.0 !2.5.3") + +#@ def is_semver(s): +#@ """Returns True if s is a semantic version and an error otherwise.""" +#@ _, err = assert.try_to(lambda: semver.from_str(s)) +#@ return True if err == Null else err +#@ end + +#@ def is_compatible(v): +#@ """Returns True if v is within the allowed range and optionally a release candidate, but not a pre-release. Returns False otherwise.""" +#@ return compatible_versions.contains(v) and (v.prerelease == "rc" or !v.prerelease) +#@ end + +#@schema/title "The version" +#@schema/desc "The version to use must be a compatible semantic version" +#@schema/validation ("is_semver", is_semver), ("is_compatible", is_compatible) +version: "" +``` + +#### _kapp-controller_'s _Downward_ API + +Consume _kapp-controller_'s _Downward_ API and conditionally template Kubernetes resources. + +```yaml +#! app.yaml + +apiVersion: kappctrl.k14s.io/v1alpha1 +kind: App +metadata: + name: my-app + namespace: my-app +spec: + serviceAccountName: my-app-sa + template: + - ytt: + inline: + valuesFrom: + - downwardAPI: + items: + name: kubernetes_version + kubernetesVersion: {} + paths: + deployment.yaml: #! ... + service.yaml: #! ... + ingress.yaml: | + #@ load("@ytt:data", "data") + #@ load("@ytt:semver", "semver") + + #! The Ingress API graduated to stable in Kubernetes 1.19. + #@ stable_ingress_kubernetes_versions = semver.range(">=1.19.0") + #@ kubernetes_version = semver.from_str(data.values.kubernetes_version) + #@ use_stable_ingress = stable_ingress_kubernetes_versions.contains(kubernetes_version) + + --- + #@ if use_stable_ingress: + apiVersion: networking.k8s.io/v1 + #@ else + apiVersion: networking.k8s.io/v1beta1 + #@ end + kind: Ingress + #! ... + #! ... +``` + +#### Drive releases with _conventional commits_ + +Release software by templating _kapp-controller_ packaging resources driven with [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/#summary) + +```yaml +#! release.yaml +#@ load("@ytt:data", "data") +#@ load("@ytt:semver", "semver") +#@ load("@ytt:assert", "assert") + +#@ last_version = semver.from_str(data.values.version) + +#! Detect the semantic change +#@ has_fixes = False +#@ has_features = False +#@ has_breaking_changes = False +#@ for commit in data.values.new_commits: +#@ if commit.startswith("feat: "): +#@ if commit.count("BREAKING CHANGE: "): +#@ has_breaking_changes = True +#@ else +#@ has_features = True +#@ end +#@ elif commit.startswith("fix: "): +#@ has_fixes = True +#@ end +#@ end + +#! Bump next version accordingly +#@ if has_breaking_changes: +#@ next_version = last_version.next_major() +#@ elif has_features: +#@ next_version = last_version.next_minor() +#@ elif has_fixes: +#@ next_version = last_version.next_patch() +#@ else: +#@ assert.fail("No changes since last release") +#@ end + +#! If prerelease, increment prerelease "build.x" identifier +#@ if data.values.prerelease: +#@ prerelease = data.values.prerelease +#@ separator = "." +#@ counter = 1 +#@ if last_version.prerelease: +#@ prerelease, _, counter_str = last_version.build.partition(separator) +#@ counter = int(counter_str)+1 +#@ end +#@ next_version.prerelease = "{}{}{}".format(prerelease, separator, counter) +#@ end + +--- +apiVersion: data.packaging.carvel.dev/v1alpha1 +kind: Package +metadata: + name: #@ "my-package.example.com.{}".format(next_version.to_str()) + namespace: my-packages +spec: + refName: my-package.example.com + version: #@ next_version.to_str() + # ... + +--- +apiVersion: data.packaging.carvel.dev/v1alpha1 +kind: PackageInstall +metadata: + name: my-package-install + namespace: my-packages +spec: + packageRef: + refName: my-package.example.com + versionSelection: + constraints: #@ next_version.to_str() + #@ if/end next_version.prerelease: + prereleases: {} + # ... +``` + +```shell +# Release 1.2.4-build.1 +ytt \ + --filename release.yaml \ + --data-value last_version=1.2.3 \ + --data-value prerelease=beta \ + --data-value-file new_commits=only_fixes.yaml +``` + +```shell +# Release 2.0.0 +ytt \ + --filename release.yaml \ + --data-value last_version=1.2.3-alpha.15 \ + --data-value prerelease="" \ + --data-value-file new_commits=breaking_changes.yaml +``` -### Other Approaches Considered +### Considered Alternatives -* A Github-hosted Starlark library like `awesome-ytt/semver` containing a similar API could be fetched with `vendir` and +- A Github-hosted Starlark library like `awesome-ytt/semver` containing a similar API could be fetched with `vendir` and then loaded. This wouldn't in reach for the widest possible spectrum of configuration authors. It would require to reimplement what existing, established Go modules already have done. ## Open Questions -* There's a host of conceivable additional operations available for semver, e.g. sort list of - versions, bumping prereleases (1.2.3-build.5 → 1.2.3-build.6), etc. It is debatable if that should be part of the - built-in library. If the primitives provided by the library are expressive enough, then authors should be able to - cover these use cases by themselves. -* Maybe configuration authors would benefit from a `version.to_dict()`(or even `version.to_list()` - and `version.to_tuple()`) to ease the transformation into a serializable data structure. On the other hand, at least - in the case of `version.to_dict()` the author may disagree with the keys. +- Should we prefer the term `constraint` over `range` as it might be more consistent with other Carvel tools? +- There's a host of conceivable additional operations for semver, e.g. sorting list of versions, bumping + prereleases (1.2.3-build.5 → 1.2.3-build.6), etc. It is debatable if that should be part of the built-in library. If + the primitives provided by the library are expressive enough, then authors should be able to cover these use cases by + themselves. +- Maybe configuration authors would benefit from a `version.to_dict()`(or even `version.to_list()` and + `version.to_tuple()`) to ease the transformation into a serializable data structure. On the other hand, at least in + the case of `version.to_dict()` the author may disagree with the keys. ## Answered Questions From 626eca69bf8dd73f21f479f6cc445c40fd2b7301 Mon Sep 17 00:00:00 2001 From: Renu Yarday Date: Mon, 26 Sep 2022 16:10:17 +0530 Subject: [PATCH 11/35] Update weekly-content-sharing.md --- processes/weekly-content-sharing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/processes/weekly-content-sharing.md b/processes/weekly-content-sharing.md index 88d79cf21..aa0de76e2 100644 --- a/processes/weekly-content-sharing.md +++ b/processes/weekly-content-sharing.md @@ -33,6 +33,6 @@ And, as a thank you for sharing your knowledge with the community, we would love | August 17, 2022 | [Ollie Hughes](https://github.com/ojhughes) | Local development workflow with Tilt and Carvel | Blog | N/A | | August 19, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020), [Soumik Majumder](https://github.com/100mik) | Introducing kctrl package authoring commands | Blog | N/A | | September 7, 2022 | [Aaron Hurley](https://github.com/aaronshurley) | Carvel in August | Blog | [@AarHurley](https://twitter.com/AarHurley)| -| September 14, 2022 | | | | -| September 21, 2022 | | | | | September 28, 2022 | | | | +| October 6, 2022 | | | | +| October 13, 2022 | | | | From 54a7d18e06808d81685863c570eab740e1ddd0da Mon Sep 17 00:00:00 2001 From: Renu Yarday Date: Mon, 26 Sep 2022 16:18:55 +0530 Subject: [PATCH 12/35] Update ROADMAP.md --- ROADMAP.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index ddf52cfc2..f4bc1f55a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -18,16 +18,16 @@ Please take the timelines & dates as proposals and goals, not commitments. Prior `Last Updated: September 2022` |Theme|Feature|Stage|Timeline| |---|---|---|---| -| Package Author Experience | **[kctrl]** [kctrl commands for package authors - Alpha Release Milestone 2.](https://github.com/vmware-tanzu/carvel-kapp-controller/issues/632): CLI-based Package Author commands to enable Package Authors easily create a Carvel package of their software | Build | September 2022| +| Package Author Experience | **[kctrl]** [kctrl commands for package authors - Milestone 3.](https://github.com/vmware-tanzu/carvel-kapp-controller/issues/632): CLI-based Package Author commands to enable Package Authors easily create a Carvel package of their software. Prioritized list in Milestone-3 | Build | October 2022| | Package Author Experience | **[ytt]** [Schema Validations](https://hackmd.io/pODV3wzbT56MbQTxbQOOKQ#Part-7-Validating-Documents): Configuration authors can specify the valid range or format of the data values. | Test | September 2022 | | Stability and Delight | **[kapp-controller]** [Version safeguards](https://github.com/vmware-tanzu/carvel-kapp-controller/issues/781): Expose cluster versions and capabilities to templating steps. | Build | September 2022 | -| Stability and Delight | **[kapp]** [Do not exit on first error](https://github.com/vmware-tanzu/carvel-kapp/issues/426): Summarize all errors found at the end of execution |Build| September 2022| -| Stability and Delight | **[kapp]** [Versioned resource based on a predetermined interval](https://github.com/vmware-tanzu/carvel-kapp/issues/224): kapp deploy to trigger a new versioned asset based on a predetermined interval | Build | September 2022| +| Stability and Delight | **[kapp]** [Versioned resource based on a predetermined interval](https://github.com/vmware-tanzu/carvel-kapp/issues/224): kapp deploy to trigger a new versioned asset based on a predetermined interval | Build | October 2022| | Stability and Delight | **[kapp-controller]** [Resilient to network failures](https://github.com/vmware-tanzu/carvel-kapp-controller/issues/664): kapp-controller continues to reconcile despite a lost connection to a registry. | Build | September 2022 | | Stability and Delight | **[ytt]** [Guides & Examples](https://github.com/vmware-tanzu/carvel-ytt/issues/314): Provide more guides and examples so that ytt is easy to get started with and details how it can be incorporate in different workflows. | Awaiting Proposal | October 2022 | | Stability and Delight | **[Carvel]** Refreshed website to improve the "getting started" experience for new users. | Awaiting Proposal | October 2022 | | Stability and Delight | **[kapp-controller]** Stable when running 1,000 workloads. | Awaiting Proposal | December 2022| | Stability and Delight | **[kapp]** [Use kapp as Go module](https://github.com/vmware-tanzu/carvel-kapp/issues/564): Use as a Go module, help improve the error handling. | Awaiting Proposal | TBD | +| Stability and Delight | **[kapp]** [Long running: Performance enhancements](https://github.com/vmware-tanzu/carvel-kapp/issues/599): minimise the list calls. | Awaiting Proposal | TBD | | Package Author Experience | **[carvel]** Carvel supports the ability to sign and verify assets (such as images, bundles, pkg/pkgr). |Awaiting Proposal| TBD | Please note that the maintainers are actively monitoring other Carvel tools that are not explicitly listed in the roadmap, e.g. kbld, vendir etc. While the maintainers have prioritized the big features listed above, if you would like us to address issues that are important to you please don't hesitate to share them with us. One way to share your feedback is by voting on an existing issue or you could simply bring them up during our community meeting. From 181e83295dddab0141fbada5b27594b84784a19c Mon Sep 17 00:00:00 2001 From: John Ryan Date: Mon, 26 Sep 2022 16:57:52 -0700 Subject: [PATCH 13/35] Draft --- site/content/blog/ytt-validations-released.md | 126 ++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 site/content/blog/ytt-validations-released.md diff --git a/site/content/blog/ytt-validations-released.md b/site/content/blog/ytt-validations-released.md new file mode 100644 index 000000000..f06ab492f --- /dev/null +++ b/site/content/blog/ytt-validations-released.md @@ -0,0 +1,126 @@ +--- +title: "The Hidden Costs of Misconfiguration" +slug: ytt-validations-released +date: 2022-09-26 +author: John Ryan, Varsha Munishwar +excerpt: "When your end-users give the wrong input, the best thing you can do is give them immediate, concise, and actionable feedback. Now, you can do that with ytt Data Values Validations!" +image: /img/ytt.svg +tags: ['ytt', 'data values', 'validation'] +--- + +## The Hidden Costs of Misconfiguration + +Take a look at this error message: + +``` +... +Updating resource service/petc (serving.knative.dev/v1) + API server says: + admission webhook "validation.webhook.serving.knative.dev" denied the request: + validation failed: + "PORT" is a reserved environment variable: spec.template.spec.containers[0].env[0].name +... +``` + +What you're looking at is the tail end of a **30-minute circuitous journey locating and collecting logs** after a +particular service apparently failed to deploy. 🥵 + +The person trying to decipher this cryptic-to-them error message wasn't versed in the intricacies of +Knative services. What they _did_ know was that there's a configuration option in the package they are using +to inject environment variables. + +Here's the simplified schema: + +```yaml +#! schema.yaml +#@data/values-schema +--- +service_name: "" +additional_env_values: + - name: "" + value: "" +``` + +and they had supplied these data values: + +```yaml +# values.yaml +--- +service_name: hello +additional_env_values: + - name: PORT + value: "5432" +``` + +When they deployed their application, all seemed to be in order... until it wasn't. + +Now, in this reduced example, the source of the error is probably obvious to the reader. But consider when +there are dozens (if not more) Data Values involved... or if there weren't so obvious matchable strings in +the error message and inputs. + +And at least in _this_ case, there's an admission webhook involved. Imagine if the workload simply quietly failed +to run? Or ignored the invalid configuration altogether??!? + +It's a recipe for pain. 😖 + +## Shift-Left Configuration Errors + +The key to avoiding this kind of suffering is to "shift left" the validating input values. + +By "[shift left](https://devopedia.org/shift-left)", we mean moving a test or check to an earlier step in the process. +This has two key effects: +1. the end-user gets immediate feedback, saving them (potentially hours) of troubleshooting. +2. the message they get is in terms of the inputs they provided, not in places where that input was used. + +We've actually grown to expect this kind of thing from web applications we use every day. Signing up for +a service, we're often asked for our email address. Usually, we get _immediate_ feedback when we put in a value that +is not a well-formed email address. + + +## Introducing `ytt` Data Values Validations + +As of `ytt` v0.43.0, you now have the ability to give your user that more useful/pleasant experience of reporting an +erroneous input right away. + +Using our example above, the author could simply include a validation annotation: + +```yaml +#@data/values-schema +--- +service_name: "" +additional_env_values: + - + #@schema/validation ("environment variable name, expect PORT (which is reserved)", lambda v: v != "PORT") + name: "" + value: "" +``` +Here: +- each value given for `additional_env_values` will be validated. +- `name:` has a validation rule defined: + - the first parameter is a user-friendly message, describing what a valid value is. + - the second parameter is an expression that _implements_ that rule (here, the value of `name:`, passed to `v` can't equal `"PORT"`) + +Instead of having to make heads-or-tails of the error message at the top of this article, they see this: + +```console +$ ytt -f schema.yaml --data-values-file values.yaml +ytt: Error: Validating final data values: + additional_env_values[0].name + from: values.yaml:4 + - must be: environment variable name, expect PORT (which is reserved) (by: schema.yaml:6) +``` + +Here: +- the data value they supplied `additional_env_values[0].name` is directly referenced (including filename and line number) +- the definition of a valid value is given. + +It's seconds, not minutes (or hours) to learn what when wrong, how, and what they can do to fix it. + +Now _that's_ a delightful experience. 🥳 + +## How do I get started? + +1. upgrade to ytt v0.43.0 +2. TODO: pointer into the Guide, Cheat Sheet, and reference. + + From 26d23919796751f8ce75a66837b087a9541f1b86 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Tue, 27 Sep 2022 11:34:15 -0700 Subject: [PATCH 14/35] Reformat source table of Content Sharing Co-Authored-by: Varsha Munishwar --- processes/weekly-content-sharing.md | 58 ++++++++++++++--------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/processes/weekly-content-sharing.md b/processes/weekly-content-sharing.md index aa0de76e2..8217b15a9 100644 --- a/processes/weekly-content-sharing.md +++ b/processes/weekly-content-sharing.md @@ -7,32 +7,32 @@ Below is a signup for individuals to reserve weeks in advance. And, as a thank you for sharing your knowledge with the community, we would love to send a t-shirt to anyone who is a not a maintainer of Carvel that participates. Please send an email to [Nanci Lancaster](mailto:nancil@vmware.com) with your contact information so that we can get the right sized t-shirt shipped to you. -| Date | Author(s) | Topic | Type of Medium (Blog, Video, etc.) | Twitter handle | -|-------------------|-------------------------------------------------------|------------------------------------------------| --- | --- | -| February 2, 2022 | [Nanci Lancaster](https://github.com/microwavables) | Example | Blog | [@microwavables](https://twitter.com/microwavables) | -| February 9, 2022 | [Daniel Helfand](https://github.com/danielhelfand) | Using vclusters with Carvel | Blog | [@danielhelfand](https://twitter.com/danielhelfand) | -| February 17, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020) | Carvelizing Helm Charts | Blog | N/A | -| February 24, 2022 | [Cari Lynn](https://github.com/cari-lynn) | Argo CD + ytt | Blog | N/A | -| March 3, 2022 | [Praveen Rewar](https://github.com/praveenrewar) | Moving existing resources to a new kapp app | Blog | [@praveen_rewar](https://twitter.com/praveen_rewar) | -| March 10, 2022 | [Yash Sethiya](https://github.com/sethiyash) | Multi environment deployment with ytt and kapp | Blog | N/A | -| March 17, 2022 | [Soumik Majumder](https://github.com/100mik) | Introducing kctrl, kapp-controller’s native CLI | Blog | N/A | -| March 24, 2022 | [Joao Pereira](https://github.com/joaopapereira) | imgpkg image collocation and tagging | Blog | N/A | -| March 31, 2022 | [John Ryan](https://github.com/pivotaljohn) | Introduction to ytt overlays | Blog/Vlog | [@jtigger](https://twitter.com/jtigger) | -| April 7, 2022 | [Leigh Capili](https://github.com/stealthybox) | Manage Kubernetes configs with vendir + ytt | Video | [@capileigh](https://twitter.com/capileigh) | -| April 14, 2022 | [Garrett Cheadle](https://github.com/gcheadle-vmware) | Parameterizing your Project Configuration with ytt | Blog | N/A | -| April 21, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020) | Identify ghost diff during kapp controller reconcilliation | Blog |N/A | -| April 28, 2022 | [Joe Kimmel](https://github.com/joe-kimmel-vmw) | kapp Rebase Rules | Blog | N/A | -| May 5, 2022 | [Soumik Majumder](https://github.com/100mik) | Getting to know App resources better with kctrl | Blog| [@\_100mik\_](https://twitter.com/_100mik_) | -| May 26, 2022 | [John Ryan](https://github.com/pivotaljohn) | Preview of `ytt` Valdations (and Experiments) | Blog | [@jtigger](https://twitter.com/jtigger) | -| June 9, 2022 | [Varsha Munishwar](https://github.com/vmunishwar) | Getting started with contributing to open-source projects like ytt | Blog| N/A | | | | -| June 30, 2022 | [Kumari Tanushree](https://github.com/kumaritanushree) | Blogs targeting the practical scenarios e.g. update deployment when configmap is updated, etc. | Blog |N/A| -| July 7, 2022 | [Neil Hickey](https://github.com/neil-hickey)| How to template a Concourse pipeline using ytt | Blog | N/A | -| July 14, 2022 | [Renu Yarday](https://github.com/renuy)| Using kapp in dagger pipeline| Blog | [@renu_yarday](https://twitter.com/renu_yarday) | -| July 21, 2022 | [Yash Sethiya](https://github.com/sethiyash) | Setup kapp deploy with OIDC Github + GKE/AWS | Blog | N/A | -| August 11, 2022 | [Joe Kimmel](https://github.com/joe-kimmel-vmw) | Carvel: July In Review | Blog | N/A | -| August 17, 2022 | [Ollie Hughes](https://github.com/ojhughes) | Local development workflow with Tilt and Carvel | Blog | N/A | -| August 19, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020), [Soumik Majumder](https://github.com/100mik) | Introducing kctrl package authoring commands | Blog | N/A | -| September 7, 2022 | [Aaron Hurley](https://github.com/aaronshurley) | Carvel in August | Blog | [@AarHurley](https://twitter.com/AarHurley)| -| September 28, 2022 | | | | -| October 6, 2022 | | | | -| October 13, 2022 | | | | +| Date | Author(s) | Topic | Type of Medium (Blog, Video, etc.) | Twitter handle | +|--------------------|-------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------|------------------------------------|-----------------------------------------------------| +| February 2, 2022 | [Nanci Lancaster](https://github.com/microwavables) | Example | Blog | [@microwavables](https://twitter.com/microwavables) | +| February 9, 2022 | [Daniel Helfand](https://github.com/danielhelfand) | Using vclusters with Carvel | Blog | [@danielhelfand](https://twitter.com/danielhelfand) | +| February 17, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020) | Carvelizing Helm Charts | Blog | N/A | +| February 24, 2022 | [Cari Lynn](https://github.com/cari-lynn) | Argo CD + ytt | Blog | N/A | +| March 3, 2022 | [Praveen Rewar](https://github.com/praveenrewar) | Moving existing resources to a new kapp app | Blog | [@praveen_rewar](https://twitter.com/praveen_rewar) | +| March 10, 2022 | [Yash Sethiya](https://github.com/sethiyash) | Multi environment deployment with ytt and kapp | Blog | N/A | +| March 17, 2022 | [Soumik Majumder](https://github.com/100mik) | Introducing kctrl, kapp-controller’s native CLI | Blog | N/A | +| March 24, 2022 | [Joao Pereira](https://github.com/joaopapereira) | imgpkg image collocation and tagging | Blog | N/A | +| March 31, 2022 | [John Ryan](https://github.com/pivotaljohn) | Introduction to ytt overlays | Blog/Vlog | [@jtigger](https://twitter.com/jtigger) | +| April 7, 2022 | [Leigh Capili](https://github.com/stealthybox) | Manage Kubernetes configs with vendir + ytt | Video | [@capileigh](https://twitter.com/capileigh) | +| April 14, 2022 | [Garrett Cheadle](https://github.com/gcheadle-vmware) | Parameterizing your Project Configuration with ytt | Blog | N/A | +| April 21, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020) | Identify ghost diff during kapp controller reconcilliation | Blog | N/A | +| April 28, 2022 | [Joe Kimmel](https://github.com/joe-kimmel-vmw) | kapp Rebase Rules | Blog | N/A | +| May 5, 2022 | [Soumik Majumder](https://github.com/100mik) | Getting to know App resources better with kctrl | Blog | [@\_100mik\_](https://twitter.com/_100mik_) | +| May 26, 2022 | [John Ryan](https://github.com/pivotaljohn) | Preview of `ytt` Valdations (and Experiments) | Blog | [@jtigger](https://twitter.com/jtigger) | +| June 9, 2022 | [Varsha Munishwar](https://github.com/vmunishwar) | Getting started with contributing to open-source projects like ytt | Blog | N/A | | | | +| June 30, 2022 | [Kumari Tanushree](https://github.com/kumaritanushree) | Blogs targeting the practical scenarios e.g. update deployment when configmap is updated, etc. | Blog | N/A | +| July 7, 2022 | [Neil Hickey](https://github.com/neil-hickey) | How to template a Concourse pipeline using ytt | Blog | N/A | +| July 14, 2022 | [Renu Yarday](https://github.com/renuy) | Using kapp in dagger pipeline | Blog | [@renu_yarday](https://twitter.com/renu_yarday) | +| July 21, 2022 | [Yash Sethiya](https://github.com/sethiyash) | Setup kapp deploy with OIDC Github + GKE/AWS | Blog | N/A | +| August 11, 2022 | [Joe Kimmel](https://github.com/joe-kimmel-vmw) | Carvel: July In Review | Blog | N/A | +| August 17, 2022 | [Ollie Hughes](https://github.com/ojhughes) | Local development workflow with Tilt and Carvel | Blog | N/A | +| August 19, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020), [Soumik Majumder](https://github.com/100mik) | Introducing kctrl package authoring commands | Blog | N/A | +| September 7, 2022 | [Aaron Hurley](https://github.com/aaronshurley) | Carvel in August | Blog | [@AarHurley](https://twitter.com/AarHurley) | +| September 28, 2022 | | | | +| October 6, 2022 | | | | +| October 13, 2022 | | | | From 056245cd0b3e759aa4cf871b02cd7fe7f1ec1d46 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Tue, 27 Sep 2022 11:41:45 -0700 Subject: [PATCH 15/35] Add current and next blog articles to content sharing table --- processes/weekly-content-sharing.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/processes/weekly-content-sharing.md b/processes/weekly-content-sharing.md index 8217b15a9..cd6534eab 100644 --- a/processes/weekly-content-sharing.md +++ b/processes/weekly-content-sharing.md @@ -33,6 +33,6 @@ And, as a thank you for sharing your knowledge with the community, we would love | August 17, 2022 | [Ollie Hughes](https://github.com/ojhughes) | Local development workflow with Tilt and Carvel | Blog | N/A | | August 19, 2022 | [Rohit Aggarwal](https://github.com/rohitagg2020), [Soumik Majumder](https://github.com/100mik) | Introducing kctrl package authoring commands | Blog | N/A | | September 7, 2022 | [Aaron Hurley](https://github.com/aaronshurley) | Carvel in August | Blog | [@AarHurley](https://twitter.com/AarHurley) | -| September 28, 2022 | | | | -| October 6, 2022 | | | | -| October 13, 2022 | | | | +| September 28, 2022 | [John Ryan](https://github.com/pivotaljohn), [Varsha Munishwar](https://github.com/vmunishwar) | The Hidden Costs of Misconfiguration | Blog | | +| October 6, 2022 | | | | | +| October 13, 2022 | [Varsha Munishwar](https://github.com/vmunishwar) | ytt Getting Started Tutorials, Part 1 | Video | | From b4b59e462c43a668329227fb3755b9ae1998a1e8 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Tue, 27 Sep 2022 11:42:50 -0700 Subject: [PATCH 16/35] Review/update ytt validations release announcement Co-Authored-by: Varsha Munishwar --- site/content/blog/ytt-validations-released.md | 55 +++++++++++-------- 1 file changed, 33 insertions(+), 22 deletions(-) diff --git a/site/content/blog/ytt-validations-released.md b/site/content/blog/ytt-validations-released.md index f06ab492f..d3c3eb363 100644 --- a/site/content/blog/ytt-validations-released.md +++ b/site/content/blog/ytt-validations-released.md @@ -3,12 +3,12 @@ title: "The Hidden Costs of Misconfiguration" slug: ytt-validations-released date: 2022-09-26 author: John Ryan, Varsha Munishwar -excerpt: "When your end-users give the wrong input, the best thing you can do is give them immediate, concise, and actionable feedback. Now, you can do that with ytt Data Values Validations!" +excerpt: "When your end-users give the wrong input, the best thing you can do is give them immediate, concise, and actionable feedback. Now you can, with ytt Validations!" image: /img/ytt.svg tags: ['ytt', 'data values', 'validation'] --- -## The Hidden Costs of Misconfiguration +## A Cryptic Error Take a look at this error message: @@ -63,13 +63,13 @@ to run? Or ignored the invalid configuration altogether??!? It's a recipe for pain. 😖 -## Shift-Left Configuration Errors +## Shift Left Configuration Errors The key to avoiding this kind of suffering is to "shift left" the validating input values. By "[shift left](https://devopedia.org/shift-left)", we mean moving a test or check to an earlier step in the process. This has two key effects: -1. the end-user gets immediate feedback, saving them (potentially hours) of troubleshooting. +1. the end-user gets immediate feedback, saving them (potentially hours) of troubleshooting; 2. the message they get is in terms of the inputs they provided, not in places where that input was used. We've actually grown to expect this kind of thing from web applications we use every day. Signing up for @@ -77,30 +77,32 @@ a service, we're often asked for our email address. Usually, we get _immediate_ is not a well-formed email address. -## Introducing `ytt` Data Values Validations +## Introducing `ytt` Validations As of `ytt` v0.43.0, you now have the ability to give your user that more useful/pleasant experience of reporting an erroneous input right away. Using our example above, the author could simply include a validation annotation: -```yaml -#@data/values-schema ---- -service_name: "" -additional_env_values: - - - #@schema/validation ("environment variable name, expect PORT (which is reserved)", lambda v: v != "PORT") - name: "" - value: "" +```diff + #! schema.yaml + #@data/values-schema + --- + service_name: "" + additional_env_values: + - ++ #@schema/validation ("environment variable name, expect PORT (which is reserved)", lambda v: v != "PORT") + name: "" + value: "" ``` Here: -- each value given for `additional_env_values` will be validated. +- each value given for `additional_env_values` will be validated; - `name:` has a validation rule defined: - - the first parameter is a user-friendly message, describing what a valid value is. - - the second parameter is an expression that _implements_ that rule (here, the value of `name:`, passed to `v` can't equal `"PORT"`) + - the first parameter is a user-friendly message, describing what a valid value is; + - the second parameter is an expression that _implements_ that rule (here, the value of `name:`, passed to `v` can't equal `"PORT"`). -Instead of having to make heads-or-tails of the error message at the top of this article, they see this: +Instead of having to find and make heads-or-tails of [the error message at the top of this article](#a-cryptic-error)...\ +... as soon as `ytt` is invoked, they see this: ```console $ ytt -f schema.yaml --data-values-file values.yaml @@ -111,16 +113,25 @@ ytt: Error: Validating final data values: ``` Here: -- the data value they supplied `additional_env_values[0].name` is directly referenced (including filename and line number) +- the data value they supplied `additional_env_values[0].name` is directly referenced (including filename and line number); - the definition of a valid value is given. It's seconds, not minutes (or hours) to learn what when wrong, how, and what they can do to fix it. Now _that's_ a delightful experience. 🥳 -## How do I get started? +## Where to, from here? -1. upgrade to ytt v0.43.0 -2. TODO: pointer into the Guide, Cheat Sheet, and reference. +Learn more about `ytt` Validations: +- See [a demo of validations in action](https://www.youtube.com/watch?v=GBMSru3WBJg) (click "show more" for a table of contents of the video). +- [Upgrade to ytt v0.43.0](/ytt/docs/v0.43.0/install/) or later. +- Get a gentle introduction through our How To [Write Validations](/ytt/docs/v0.43.0/how-to-write-validations). +- Get started quick with our [Validations Cheat Sheet](/ytt/docs/v0.43.0/schema-validations-cheat-sheet). +- Dive into the syntax and inventory of out-of-the-box rules in the [@schema/validation reference](/ytt/docs/v0.43.0/lang-ref-ytt-schema/#schemavalidation). +## Join the Carvel Community +We are excited to hear from you and learn with you! Here are several ways you can get involved: +* Join Carvel's slack channel, [#carvel in Kubernetes]({{% named_link_url "slack_url" %}}) workspace, and connect with over 1000+ Carvel users. +* Find us on [GitHub](https://github.com/vmware-tanzu/carvel). Suggest how we can improve the project, the docs, or share any other feedback. +* Attend our Community Meetings! Check out the [Community page](/community/) for full details on how to attend. From 776ca1124738c2eab8b78a9ea190f0234fc5d1b2 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Tue, 27 Sep 2022 11:48:56 -0700 Subject: [PATCH 17/35] Publish next article on Wednesday. --- site/content/blog/ytt-validations-released.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/blog/ytt-validations-released.md b/site/content/blog/ytt-validations-released.md index d3c3eb363..6eab8e2ca 100644 --- a/site/content/blog/ytt-validations-released.md +++ b/site/content/blog/ytt-validations-released.md @@ -1,7 +1,7 @@ --- title: "The Hidden Costs of Misconfiguration" slug: ytt-validations-released -date: 2022-09-26 +date: 2022-09-28 author: John Ryan, Varsha Munishwar excerpt: "When your end-users give the wrong input, the best thing you can do is give them immediate, concise, and actionable feedback. Now you can, with ytt Validations!" image: /img/ytt.svg From 053f49714ed2070e0caae2512a5e91572ee742f8 Mon Sep 17 00:00:00 2001 From: Max Brauer Date: Sun, 11 Sep 2022 08:18:46 +0200 Subject: [PATCH 18/35] Document ytt overlay/insert's via kwarg with an example --- .../ytt/docs/develop/lang-ref-ytt-overlay.md | 22 ++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/site/content/ytt/docs/develop/lang-ref-ytt-overlay.md b/site/content/ytt/docs/develop/lang-ref-ytt-overlay.md index 487a1ba42..90cbaad4e 100644 --- a/site/content/ytt/docs/develop/lang-ref-ytt-overlay.md +++ b/site/content/ytt/docs/develop/lang-ref-ytt-overlay.md @@ -403,7 +403,7 @@ See also: ### @overlay/insert -Inserts "right" node before/after the matched "left" node. +Inserts "right" node before/after the matched "left" node. The inserted node is either the "right" node or that provided by a function. **Valid on:** Document, Array Item. @@ -412,6 +412,26 @@ Inserts "right" node before/after the matched "left" node. ``` - **`before=`**`Bool` whether to insert the "right" node immediately in front of the matched "left" node. - **`after=`**`Bool` whether to insert the "right" node immediately following the matched "left" node. +- **`via=`**`Function(left, right): (any)` _(optional)_ determines the value to substitute in. If omitted, the value is `right`. + - `left` ([`yamlfragment`](lang-ref-yaml-fragment.md) or scalar) — the matched node's value + - `right` ([`yamlfragment`](lang-ref-yaml-fragment.md) or scalar) — the value of the annotated node + +**Examples:** + +Add a `ConfigMap` into each `Namespace`: +```yaml +#@ def configMap(namespace): +apiVersion: v1 +kind: ConfigMap +metadata: + name: insert + namespace: #@ namespace.metadata.name +#@ end + +#@overlay/match overlay.subset({"kind": "Namespace"}) +#@overlay/insert after=True, via=lambda namespace, _: configMap(namespace) +--- +``` ### @overlay/append From c0198cfb13a71ca7e6005f38a647f2bc40f5f05c Mon Sep 17 00:00:00 2001 From: John Ryan Date: Wed, 12 Jan 2022 09:52:08 -0800 Subject: [PATCH 19/35] Inception (incomplete) --- proposals/ytt/004-schema-validation/README.md | 1523 +++++++++++++++++ 1 file changed, 1523 insertions(+) create mode 100644 proposals/ytt/004-schema-validation/README.md diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md new file mode 100644 index 000000000..14ba65069 --- /dev/null +++ b/proposals/ytt/004-schema-validation/README.md @@ -0,0 +1,1523 @@ +--- +title: "Validations" +authors: [ "John S. Ryan " ] +status: "draft" +approvers: [] +--- + + +# Validations + +## Problem Statement + +To date, ytt Library / Carvel Package **authors have had to "roll their own" logic to check that customer's inputs are valid.** That is: that data values are a) present/non-empty; and b) within expected range(s). + +Given the expressiveness of Starlark, producing the actual logic, itself, is not too onerous. + +However, Authors have had to... +- write the "plumbing" code around validations (or offer their consumers a degraded UX): + - collecting all violations (instead of failing on first violation) + - ordering validation evaluation — leaves to root + - handling common scenarios (empty values, enums, unions, conditional) +- hand-document validations by adding comments to their Data Values (Schema) describing those constraints. + - it's double the work + - there are now two places that need to be updated whenever a constraint changes (see [Shalloway's Law](https://netobjectivesthoughts.com/shalloways-law-and-shalloways-principle/#:~:text=%E2%80%9CWhen%20N%20things%20need%20to,%2D1%20of%20these%20things.%E2%80%9D&text=It's%20a%20law!)). + +**The time our users spend in developing and maintaining this code, is time lost to making progress in their first-order work.** + +In the Kubernetes space, as cluster deployments become **increasingly complex**, the need to **vet declarative configuration becomes ever more critical**. Users are rightfully expecting their tooling to help them catch errors as early as possible, out-of-the-box. + +## Terminology / Concepts + +- **assertion** — a function that — given a value — if the value meets the assertion's criteria, quietly succeeds, but if the value fails the criteria, fails. +- **CEL** — [Common Expression Language](https://github.com/google/cel-spec) — language supported (in alpha) by Kubernetes for CRD validation rules. +- **module** — a file that can be loaded into a template in order to reuse its functions (or variables). The `ytt` "Standard Library" is comprised of modules that begin with the `@ytt:` prefix (e.g. @ytt:data, @ytt:assert, @ytt:overlay, ...) +- **node** — a single piece of YAML: a document set, document, map, map item, array, array item. One "annotates" a node by placing a `ytt` annotation-shaped comment just above the node (see also: [YAML Primer](https://carvel.dev/ytt/docs/v0.39.0/yaml-primer/)). +- **required value** — a Data Value that the consumer must set. Mechanically, a Data Value that does not (yet) have a value that is "valid." +- **rule** — the combination of: + - a textual description of what constitutes a valid value; + - a function that asserts the rule against an actual value; + - a violation message template, used to report invalid values. + - **custom rule** — a rule based on a function (either one written by a user or provided in a `ytt` module). + - **named rule** — a rule provided with `ytt` expressed through a keyword argument to `@assert/validate`. +- **validation** — the binding of one or more "rule"s to a "node". +- **violation** — an instance of where a given value of a "node" fails a "validation". + +## Proposal + +Implement all plumbing logic required to provide high-quality data validation over Data Values and output documents. + +Mechanically, this means: + +Extend the `@ytt:assert` module, to include the ability to validate: +- introduce an annotation (`@assert/validate`) which attaches one or more "rules" to a "node", defining what a valid value is for that node; +- supply a set of common assertions (functions in the `@ytt:assert` module) that covers most validation use-cases; +- provide guidance for how to express most commonly intended validations. + +Further, integrate this feature into `ytt`'s processing pipeline: +- immediately after a final Data Values is calculated for a library, automatically validate that document (and halt processing if there are any violations); +- if the user is inspecting schema, include validations in the output; +- immediately after the Output Document Set (i.e. the final result of evaluating a library) is produced, automatically validate each document in the set (and halt processing if there are any violations). + +Finally, achieve all this in a way that complements existing validation tooling: +- be 100% compatible with Kubernetes validation facilities (i.e. OpenAPI v3 / JSON Schema validations + [Kubernetes Validation Rules](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules)). + +Deliver this functionality in the following increments: + +1. core validation plumbing, integrated into Schema, (at least) minimum set of built-in validations. +2. include validations in exports to OpenAPI v3 formatted Schema "inspect" / exports. +3. address advanced use-cases by adding the ability to edit annotations in the overlay module. +4. integrate evaluating validations in output documents. + +### Use Cases + +Specific scenarios within the overall job of validating data include: + +- [Ensure consumer provides a value (Required Input)](#use-case-required-input) + - [... when the data value is a type that has length](#required-input-for-types-with-length) + - [... when the data value is a type that has an empty value](#required-input-for-types-with-empty-value) + - [... for all other cases](#required-input-for-types-with-no-empty-value) +- [Require value to be from a defined set (Enumeration)](#use-case-enumeration) +- [Sections of configuration are mutually exclusive (Union Structures)](#use-case-union-structures) +- [Validations involving multiple Data Values (Multiple Nodes)](#use-case-multiple-nodes) +- [Validate nullable Data Values](#use-case-validations-over-nullables) +- [Apply validations in specific situations (Conditional Validations)](#use-case-conditional-validations) +- [Disable all validations in an "emergency" (Disable Validations)](#use-case-disable-validations) +- [Reuse a set of validations over a commonly-shaped set of Data Values (Programatic Validations)](#use-case-programmatic-validations) +- [Export Schema in OpenAPI v3 format](#use-case-exporting-schema-in-openapi-format) +- [Validate an output document](#use-case-validating-an-output-document) + +The following sections detail each use case, in turn... + +#### Use Case: Required input + +When the author cannot/will not provide a default value for a data value and requires the consumer to provide it. + +Where possible, authors are encouraged to use a type-specific "non-empty" validation. As a fallback, they may opt for nullable+not_null. + +##### Required input for types with length + +**String Values** + +The recommended way for an author to indicate required string input is to expect a non-empty string: + +```yaml +#@data/values-schema +--- +dex: + #@assert/validate min_len=1 + namespace: "" + #@assert/validate min_len=1 + username: "" +``` + +This case is so common (it is our understanding that most string-based data values are meant to be non-empty), we offer this convenience: + +```yaml +#@data/values-schema +#@assert/validate-string-defaults min_len=1 +--- +dex: + namespace: "" + username: "" +``` + +Which is equivalent to the prior example, but applies the default to any descendent that contains a string-typed value. If a particular node already has an `@assert/validate` annotation, it is skipped. + +**Array Values** + +Similarly, a non-empty value of an array has at least one element: + +```yaml +#@data/values-schema +--- +dex: + oauth2: + #@assert/validate min_len=1 + responseTypes: + - "" +``` + +Array-typed values are less common. And when present are not necessarily _as_ common to require a non-empty value as string-typed values. So, there is no corresponding shorthand annotation for requiring a non-empty array. + + +##### Required input for types with empty value + +For some integer-typed (or float-typed) data values, a zero value may be invalid and thus can represent "empty". + +```yaml +#@data/values-schema +dex: + #@assert/validate min=1024, max=65535 + port: 0 +``` + +Authors are encouraged to favor this approach over [marking the data values as nullable](#required-input-for-types-with-no-empty-value) to keep their schema as simple as possible. + +##### Required input for types with no empty value + +Some types have no natural (or easily detectable) "empty" value. In these cases, authors can force the consumer to supply a value by marking the value as "nullable" and constraint it to be "not null": + +```yaml +#@data/values-schema +--- +#@schema/nullable +#@assert/validate not_null=("Cloud credentials are required.", True) +credential: + name: + secretContents: + cloud: +``` + +_(also considered: [`@schema/required`](#schemarequired))._ + +#### Use Case: Enumeration + +Values can be constrained to be one of a finite site (i.e. of an enumeration). + +```yaml +#@data/values-schema +--- +volumeSnapshotLocation: + spec: + #@assert/validate one_of=["aws", "azure", "vsphere"] + provider: "" +``` + +#### Use Case: Union Structures + +An author can require that exactly one of a set of sibling node has a value: + +```yaml +#@data/values-schema +--- +dex: + #@assert/validate one_not_null=["oidc", "ldap"] + config: + #@schema/nullable + oidc: + CLIENT_ID: null #! required if oidc enabled + CLIENT_SECRET: null #! required if oidc enabled + issuer: null #! is required if oidc enabled + #@schema/nullable + ldap: + host: null #! is required if ldap enabed + bindDN: null + bindPW: null +``` + +This approach is simpler than (and therefore recommended over) using a discriminator: + +```yaml +#@data/values +--- +dex: + config: + type: "oidc" + oidc: + CLIENT_ID: null #! required if oidc enabled + CLIENT_SECRET: null #! required if oidc enabled + issuer: null #! is required if oidc enabled + ldap: + host: null #! is required if ldap enabed + bindDN: null + bindPW: null +``` + +#### Use Case: Multiple Nodes + +When a validation involves more than just one Node's value: + +```python +def validate_registry(): + if data.values.persistence.imageChartStorage.type == "filesystem": + if data.values.persistence.persistentVolumeClaim.registry.accessMode == "ReadWriteOnce": + data.values.registry.replicas == 1 or assert.fail("The registry replicas must be 1 when the image storage is filesystem and the access mode of persistentVolumeClaim is ReadWriteOnce") + end + end +end +``` +(ref: [vmware-tanzu/community-edition/.../harbor/2.2.3/config/values.star](https://github.com/vmware-tanzu/community-edition/blob/cc7eb24b0e13875a06ac8578cd73f83a217ec4d9/addons/packages/harbor/2.2.3/bundle/config/values.star#L45-L53)) + +The author places the validation at the lowest-common parent Node: + +```yaml +#@ def one_registry_if_pvc_is_filesystem(val): +#@ if val.persistence.imageChartStorage.type == "filesystem" and +#@ val.persistence.persistentVolumeClaim.registry.accessMode == "ReadWriteOnce": +#@ return val.registry.replicas == 1 \ +#@ or (False, "There can be only one registry replica when the image storage is filesystem and the access mode of persistentVolumeClaim is ReadWriteOnce; there are {} replicas.".format(val.registry.replicas)) +#@ end +#@ end + +#@data/values-schema +#@assert/validate ("There should be exactly one (1) registry replica if Helm Charts are stored on the filesystem.", one_registry_if_pvc_is_filesystem) +--- +persistence: + imageChartStorage: + type: filesystem + persistentVolumeClaim: + registry: + accessMode: ReadWriteOnce +registry: + replicas: 2 +``` + +Note: the `@ytt:data` module's `data.values` struct is _not_ populated until _after_ the Data Values Pre-Processing phase is complete. As such, the `data.values` variable is not useful within validations. + +Authors are encouraged to minimize their use of this technique: it fixes the location of data values making it more expensive to reorganize them. + +#### Use Case: Validations over Nullables + +Whenever a node is annotated `@schema/nullable`, then its default value is `null`. + +```yaml +#@data/values-schema +--- +#@assert/validate min=42, max=42 +#@schema/nullable +foo: 13 +``` + +The author may opt to short-circuit validations when the actual value is `null`: + +```yaml +#@data/values-schema +--- +#@assert/validate min=42, max=42, when_null_skip=True +#@schema/nullable +foo: 13 +``` + +(This is preferred instead of requiring _all_ validators to perform a null check to keep the overall complexity of the system to a minimum.) + +_(also considered: [automatically including `when_null_skip` when Data Value is nullable](#automatically-including-when_null_skip-for-nullable-data-values))._ + +When the author has set the Data Value as "nullable" with the intent to require the consumer to supply their own value, this is the [Required Input (for types with no empty value)](#required-input-for-types-with-no-empty-value) use case. + +This keyword is a special case of [`when=`](#use-case-conditional-validations) because it is so common. + +#### Use Case: Conditional Validations + +The author can short-circuit validations under certain conditions: + +```yaml +#@data/values-schema +--- +#@assert/validate ("", valid_service_config), when=lambda v: v.enabled +service: + enabled: true + type: NodePort +``` + +Before any validations are checked, the value of `when=` (itself being a validation) is checked.\ +If `false`, then none of the other validations are checked and the value is assumed "valid." + +Here, the contents of the `service:` are validated (via `valid_service_config()`) only if `service.enabled` is `true`. + +#### Use Case: Disable Validations + +A consumer can skip validating Data Values: + +```console +$ ytt ... --data-value-schema-disable-validation +``` + +or the programmatic equivalent: + +```python +lib.eval(data_value_schema_disable_validation=True) +``` + +Likewise, a consumer can skip validating output documents: + +```console +$ ytt ... --output-disable-validation=True +``` + +or the programmatic equivalent: + +```python +lib.eval(output_disable_validation=True) +``` + +#### Use Case: Programmatic Validations + +> **TODO:** +- [ ] briefly talk through how common structures themselves are best duplicated _or_ captured in a fragment function — including validations + +_(Also considered: [programmatically invoking multiple validations](#programmatically-invoking-multiple-validations).)_ + + +#### Use Case: Exporting Schema in OpenAPI format + +> **TODO:** +> - [ ] detail how rule descriptions are included in documentation. +> - [x] for each OpenAPI/JSON Schema validation keyword, show exactly how that would be expressed in `ytt` validation. +> - [ ] describe the fallback strategy where authors can represent their rule where there _is_ no OpenAPI equivalent. + +When the user inspects the schema: + +``` +$ ytt ... --data-values-schema-inspect -o openapi-v3 +``` + +Validations defined on schema are included in that export. + + +OpenAPI v3 / JSON Schema validation keywords: + +Supported (ytt rule ==> OpenAPI v3 / JSON Schema property): +- `max=` / `@ytt:assert.max()` ==> [maximum](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.2) +- `min=` / `@ytt:assert.min()` ==> [minimum](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.4) +- `max_len=` / `@ytt:assert.max_len()`: + - type: `string` ==> [maxLength](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.6) + - type: `array` ==> [maxItems](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.10) + - type: `map` ==> [maxProperties](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.13) +- `min_len=` / `@ytt:assert.min_len()`: + - type: `string` ==> [minLength](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.7) + - type: `array` ==> [minItems](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.11) + - type: `map` ==> [minProperties](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.14) +- `one_of=` / `@ytt:assert.one_of()`: [enum](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.20) +- `format=` / `@ytt:assert.format()` : [format](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7) \ + Format names are identical: + - [date-time](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.1) + - [email](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.2) + - [hostname](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.3) + - [ipv4](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.4) + - [ipv6](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.5) + - [uri](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.6) + - [uriref](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.7) + +The following are deferred: + +- uniqueItems +- exclusiveMaximum (is this useful?) +- exclusiveMinimum (is this useful?) +- [multipleOf](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.1) +- pattern (This string SHOULD be a valid regular expression, according to the ECMA 262 regular - expression dialect) + +The following JSON Schema properties are better supported as part of `ytt` schema: +- [items](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.9) +- [additionalItems](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.9) +- [additionalProperties](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.18) + +The following are references to JSON Schema and will not be generated: +- [properties](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.16) +- [patternProperties](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.17) +- [dependencies](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.19) +- [allOf](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.22) +- [anyOf](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.23) +- [not](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.25) +- [oneOf](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.24) + +References: +- https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.0.md#schemaObject +- https://262.ecma-international.org/5.1/#sec-7.8.5 +- https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00 + +#### Use Case: Exporting Schema with Kubernetes Extensions + +> **TODO:** +- [ ] consider how to allow for or translate `ytt` validations to CEL rules. (directly support CEL?) + + +#### Use Case: Validating an output document + +> **TODO:** +> - [ ] illustrate validating a YAML document from within template code. + +A template author can help ensure that not just the inputs, but also the final output is valid. + +```yaml +#@ load("@ytt:assert", "assert") + +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + labels: + app: nginx +spec: + #@assert/validate max=4 + replicas: #@ calc_replicas() + selector: + matchLabels: #@ labels() + template: + metadata: + #@assert/validate ("Istio sidecar inbound port must be present and >= 9000", lambda v: ("excludeInboundPorts" in v) and int(v["excludeInboundPorts"]) > 9000) + annotations: + excludeInboundPorts: #@ "{}".format(data.values.istioPortLimit) + labels: #@ labels() + spec: + containers: + - name: nginx + image: nginx:1.14.2 + ports: + - containerPort: 80 + +``` + +Note: in order for validations to be applied via an overlay in any practical way, we would need to [define merge semantics of annotations in overlays](#defining-merge-semantics-for-annotations-in-an-overlay). + +Sources: +- [cloudfoundry/cf-for-k8s/tests](https://github.com/cloudfoundry/cf-for-k8s/tree/95c77d741706ab2a27979db90873fb4461a1ac3c/tests) + +### Specification + +#### @assert/validate + +Defines validity for the value of the annotated node. + +``` +@assert/validate [rule0, rule1, ...] [,] [,when=] [,when_null_skip=] +``` + +where: +- `ruleX` — any number of custom rules: a tuple `(description, assertion, [message])` + - `description` (`string`) — a description of what a valid value is. + - `assertion` (`function(value) : None` | `function(value) : bool`) — that either `assert.fail()`s or returns `False` when `value` is not valid. + - `value` (`string` | `int` | `float` | `bool` | [`yamlfragment`]()) — the value of the annotated node. + - `message` (`string`) — (optional) overrides the default violation message with a custom template in [`string.format()`](https://github.com/google/starlark-go/blob/bb14e15/doc/spec.md#stringformat) syntax supplied with the following fields: + - `{key}` — the key portion of the map item (or index if an array item or document). + - `{value}` — the value of the annotated node (supplied as [`repr(value)`](https://github.com/google/starlark-go/blob/bb14e15/doc/spec.md#repr)) + - `{desc}` — the description supplied in the 0th item in this rule. + - `{failure}` — when `assertion` fails, the message from that `assert.fail()` or empty string. +- `` — any of the keyword arguments described in [Named Rules](#named-rules), below. +- `when=` (`function(value) : None` | `function(value) : bool`) — criteria for when the validation rules should be checked. If the criteria is met (function either returns `True` or `None`), the validations are checked; otherwise (function either returns `False` or `assert.fail()`s, none of the validations are checked. +- `when_null_skip=` (`bool`) — a special-case of `when=` that checks if the value of the annotated node is `null`. + +Each rule is evaluated, in the order they appear in the annotation (left-to-right): +- if all rules pass (either returns `True` or `None`), then the value is valid. +- if a rule returns `False` (not `None`) or `assert.fail()`s, then the valid is invalid. + +The default `message` is: +``` +\"{key}\" requires a valid value: {desc}; {failure}. +``` +- centering around the word "...requires..." helps the message work both in the case where the consumer supplied no value (see [Use Case: Required input](#use-case-required-input)) _and_ when they supplied an invalid value. +- `{value}` is omitted to not contribute to the risk of leaking secrets. + + +_Example 1: Named rules_ + +```yaml +#@data/values-schema +--- + +#@assert/validate min=49142 max=65535 +adminPort: 1024 +``` + +_Example 2: Custom assertion-based rule_ + +```yaml +#@data/values-schema +--- +#@ def is_dynamic_port(n): +#@ n >= 49142 and n <= 65535 or assert.fail("{} is not in the dynamic port range".format(n)) +#@ end + +#@assert/validate ("a TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port) +adminPort: 1024 +``` + +Yields: +> "adminPort" requires a valid value: a TCP/IP port in the "dynamic" range: 49142 and 65535, inclusive; 1024 is not in the dynamic port range. + +_Example 3: Custom violation message_ + +```yaml +#@data/values-schema +--- +#@ def is_dynamic_port(n): +#@ n >= 49142 and n <= 65535 or assert.fail("{} is not in the dynamic port range".format(n)) +#@ end + +#@assert/validate ("a TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port, "\"{key}\" (={value}) must be between 49142 and 65535") +adminPort: 1024 +``` + +Yields: +> "adminPort" (=1024) must be between 49142 and 65535 + +_Example 4: Custom predicate-based rule_ + +```yaml +#@data/values-schema +--- +#@ def is_valid_port(n): +#@ return n >= 49142 and n <= 65535 +#@ end + +#@assert/validate ("A TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port) +adminPort: 1024 +``` + +Yields: +> "adminPort" requires a valid value: a TCP/IP port in the "dynamic" range: 49142 and 65535, inclusive; \"is_valid_port()\" returned False." + +Note that `ytt` detects that `is_valid_port()` returns a false-y value and formats `{failure}` as `\"{key}\" returned False.` + +> **TODO:** +> - [ ] determine how anonymous functions (e.g. lambdas) can be effectively referred (e.g. "third rule"?) + +##### Named Rules + +A number of rules are so common that in addition to their function form, they warrant a shorthand in the form of a keyword argument. + +They are: +- `min_len=` (see [`@ytt:assert.min_len()`](#yttassertmin_len)) +- `max_len=` (see [`@ytt:assert.max_len()`](#yttassertmax_len)) +- `min=` (see [`@ytt:assert.min()`](#yttassertmin)) +- `max=` (see [`@ytt:assert.max()`](#yttassertmax)) +- `one_of=` — enumeration (see [`@ytt:assert.one_of()`](#yttassertone_of)) +- `not_null=` (see [`@ytt:assert.not_null()`](#yttassertone_not_null)) +- `one_not_null=` — union structure (see [`@ytt:assert.one_not_null()`](#yttassertone_not_null)) + +General-purpose string assertions: +- `starts_with=` (see [`@ytt:assert.starts_with()`](#yttassertstarts_with)) +- `ends_with=` (see [`@ytt:assert.ends_with()`](#yttassertends_with)) +- `contains=` (see [`@ytt:assert.contains()`](#yttassertcontains)) +- `matches=` (see [`@ytt:assert.matches()`](#yttassertmatches)) +- `format=` (see [`@ytt:assert.format()`](#yttassertformat)) + +###### Named rule arguments + +The argument of a named rule can either be just the value: +``` +@assert/validate min_len=1 +username: "" +``` +and invalid values with be reported with the underlying validation's default violation message. + +> `"username" requires a valid value (a length of at least 1); it is a length of 0.` + + +The value can also be a tuple of a custom definition of a valid value, along with the value: +``` +@assert/validate min_len=("a non-empty string", 1) +username: "" +``` + +> `"username" requires a valid value (a non-empty string); it is a length of 0.` + +Note: if an author wishes to provide a custom violation message, they would use the corresponding built-in rule as a custom rule. + +#### @assert/validate-string-defaults + +> **TODO:** +> - [ ] firm-up name of this annotation: is it a bug or a feature that it is type-specific? + +Defines what a valid value is for all descendants which hold a "string" value. + +``` +@assert/validate-string-defaults [rule0, rule1...] [,] [,when=] [,when_null_skip=] +``` + +See also: [Consideration: Setting validation defaults for strings in a schema overlay](#consideration-setting-validation-defaults-for-strings-in-a-schema-overlay). + +#### @ytt:assert.validate() + +Programmatically asserts that a given value is valid based on the set of supplied rules. + +```python +assert.validate(key, value, rule0, rule1, ... [,] [,when=] [,when_null_skip=]) +``` +where: +- `key` (`string`) — the name of the value being validated. +- `value` (any) — the value being validated. + +This function is the programmatic equivalent of [@assert/validate](#assertvalidate). + +#### @ytt:assert.is_valid() + +Asserts any validations present on a Node (and descendents). + +```python +assert.is_valid(node) +``` + +- if all validation rules are satisfied, nothing happens. +- if any validation rule is violated, `assert.fail()`s with the corresponding validation message(s). + + +#### Included Assertion Functions + +- the functions in this section are technically higher-order functions that _produce_ assertions: + ```python + def assert.min(x): + return lambda v: assert.fail("at least {}", x) if v >= x else None + end + ``` + of which the return value _is_ an assertion. +- violation messages should be consistent across these functions. +- these functions should attempt to support as many types as possible: + +##### @ytt:assert.contains() + +##### @ytt:assert.ends_with() + +##### @ytt:assert.format() + +Supported formats: +- "quantity" — https://kubernetes.io/docs/reference/glossary/?all=true#term-quantity +- Additional keywords will be added as needs are clear. Likely candidates come from common domain-specific formats: +[JSON Schema Validation: §7 Vocabularies for Semantic Content With "format"](https://json-schema.org/draft/2020-12/json-schema-validation.html#rfc.section.7). + +##### @ytt:assert.len() + +The length of the value is _exactly_ that of the given length. + +##### @ytt:assert.matches() + +##### @ytt:assert.max() + +The value is _at most_ (inclusive) of the given minimum. + +##### @ytt:assert.max_len() + +The length of the value is _at most_ (inclusive) that of the given minimum. + + +##### @ytt:assert.min() + +The value is _at least_ (inclusive) of the given minimum. + +##### @ytt:assert.min_len() + +The length of the value is _at least_ (inclusive) that of the given minimum. + +##### @ytt:assert.not_null() + +When present, this validator is always checked first. + +##### @ytt:assert.one_not_null() + +##### @ytt:assert.one_of() + +##### @ytt:assert.starts_with() + +#### Custom Validation Functions + +> **TODO:** +> - [ ] Determine: does more need to be specified about these kinds of functions? Or are descriptions from elsewhere sufficient? +#### Consideration: Order of Validations + +Validations are checked, leaf-upwards: a parent value can't be valid if one of its children are invalid. + +#### Consideration: Merging Validations on Schema Nodes + +Some preliminary design work has been done describe how to augment the Overlay module to better support editing of annotations. (see [Defining merge semantics for annotations in an overlay](#defining-merge-semantics-for-annotations-in-an-overlay) for details). + +However, for the few that find themselves in a situation where they need to edit existing schema, they have workarounds +available with existing mechanisms: +- one can knock-out any overly-constraining validation by writing an overlay that removes the existing node and subsequently replaces it with a node that has the desired annotations. +- if all else fails, one can supply a "valid" input and then override the desired output value via an overlay (over templated result). + +### Open Questions + +1. **Q:** How are `one_not_null` expressed in CRDs (or built-in types) e.g. go look at Volumes (for each array item). \ + https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/_print/#validation +2. **Q:** `@assert/validate-string-defaults` feels overly specific; is there a way to both generalize the notion of defaulting while keeping a reasonable syntax? +3. **Q:** in the case where a function is placed higher up on the tree, but really the value being checked is one of the children, should we give the facility to be able to name the path of the target key? In this way, the violation message can be associated with the child (as well). +4. **Q:** When evaluating a private library, should validations be _automatically_ evaluated? +5. **Q:** Should we provide a "break-the-glass" means of capturing a form of a validation in CEL? + +### Answered Questions + +1. **Q:** (backwards compatibility) should we be validating against non-empty strings, by default? \ + **A:** no, there's no known way to do this without creating a breaking change. Instead, we'll provide a convenient +2. **Q:** Should we accept more than one `@assert/validate` annotation (to make it easier to stack them)? + - what happens today when we have multiple instances? (merge vs. replace behavior) + - what would it mean to have multiple annotation of other kinds? + - ideally, this is a generic rule about how annotations merge + - if an annotation is applied at the doc level, how does that work when merging to an existing schema? (think string defaults to min_len=0) + + **A:** No. These scenarios can be deferred until a general mechanism for editing annotations within an overlay are worked out. +3. **Q:** Should keyword argument validations perform type-checking? \ + **A:** No. Given that schema will type-check values _before_ applying validations, it's a rare case when the author will place the wrong validation on a data value and not realize it. This is likely unnecessary complexity. +4. **Q:** Are there any special considerations for libraries of validations? (e.g. Kubernetes, cloud-provider specific values) \ + **A:** If we ensure that we map well to Kubernetes validation features, + +### Complete Examples + +- https://github.com/vmware-tanzu/community-edition/tree/main/addons/packages + - https://github.com/vmware-tanzu/community-edition/blob/cc7eb24b0e13875a06ac8578cd73f83a217ec4d9/addons/packages/harbor/2.2.3/bundle/config/values.star + - https://github.com/vmware-tanzu/community-edition/blob/cc7eb24b0e13875a06ac8578cd73f83a217ec4d9/addons/packages/vsphere-cpi/1.22.4/bundle/config/values.star + - https://github.com/vmware-tanzu/community-edition/blob/cc7eb24b0e13875a06ac8578cd73f83a217ec4d9/addons/packages/velero/1.6.3/bundle/config/values.star + - https://github.com/vmware-tanzu/community-edition/blob/cc7eb24b0e13875a06ac8578cd73f83a217ec4d9/addons/packages/pinniped/0.12.0/bundle/config/upstream/_ytt_lib/supervisor/helpers.lib.yaml +- https://github.com/vmware-tanzu/tanzu-framework + - https://github.com/vmware-tanzu/tanzu-framework/blob/2b3c557f5651a0bfe79dac1b19e12d5925178bef/pkg/v1/providers/ytt/lib/validate.star + - e.g. `validate_configuration()` used by https://github.com/vrabbi/tkg-resources/blob/0e302afc87997bee8312cc7da25b76815ea01b0c/TKG%20Customization/tkg/providers/infrastructure-vsphere/v0.7.1/ytt/overlay.yaml +- https://github.com/cloudfoundry/cf-for-k8s/blob/develop/config/get_missing_parameters.star +- String values that are typically/ok to be empty by default: https://github.com/tomkennedy513/carvel-package-kpack/blob/main/config/schema.yaml + +#### Harbor + +From: +- vmware-tanzu/community-edition//addons/packages/harbor/2.2.3/bundle/ + - [config/values.yaml](https://github.com/vmware-tanzu/community-edition/blob/cc7eb24b0e13875a06ac8578cd73f83a217ec4d9/addons/packages/harbor/2.2.3/bundle/config/values.yaml) + - [config/values.star](https://github.com/vmware-tanzu/community-edition/blob/cc7eb24b0e13875a06ac8578cd73f83a217ec4d9/addons/packages/harbor/2.2.3/bundle/config/values.star) + +To: + +```yaml +#@ load("@ytt:validation", "validation") +#@ +#@ def one_registry_if_pvc_is_filesystem(val): +#@ if val.persistence.imageChartStorage.type == "filesystem" and +#@ val.persistence.persistentVolumeClaim.registry.accessMode == "ReadWriteOnce": +#@ return val.registry.replicas == 1 \ +#@ or (False, "Charts are being stored in a 'filesystem' type PVC and the Registry's PVC access mode is 'ReadWriteOnce': there must be exactly one (1) replica of the registry but {} replicas are configured.".format(val.registry.replicas)) +#@ end +#@ end + +#@data/values-schema +#@assert/validate ("There can be exactly one (1) registry replica if Helm Charts are stored on the filesystem.", one_registry_if_pvc_is_filesystem) +#@assert/validate-string-defaults min_len=1 +--- +#@schema/desc "The namespace to install Harbor" +namespace: harbor + +#@schema/desc "The FQDN for accessing Harbor admin UI and Registry service." +#@assert/validate min_len=1, format="hostname" +hostname: harbor.yourdomain.com +#@schema/desc "The network port of the Envoy service in Contour or other Ingress Controller." +port: + #@assert/validate min=1, max=65535 + https: 443 + +#@schema/desc "The log level of core, exporter, jobservice, registry." +#@assert/validate one_of=["debug", "info", "warning", "error", "fatal"] +logLevel: info + +#@ text = """ +#@ The certificate for the ingress if you want to use your own TLS certificate. +#@ We will issue the certificate by cert-manager when it's empty. +#@ """ +#@schema/desc text +#@schema/nullable +tlsCertificate: + #@schema/desc "The certificate" + tls.crt: "" + #@schema/desc "The private key" + tls.key: "" + #@schema/desc "The certificate of CA, this enables the download link on portal to download the certificate of CA." + #@assert/validate min_len=0 + ca.crt: "" + +#@schema/desc "Use contour http proxy instead of the ingress." +enableContourHttpProxy: true + +#@schema/desc "The initial password of Harbor admin." +harborAdminPassword: "" + +#@schema/desc "The secret key used for encryption." +#@assert/validate len=16 +secretKey: "" + +database: + #@schema/desc "The initial password of the postgres database." + password: "" + +core: + replicas: 1 + #@schema/desc "Secret is used when core server communicates with other components." + secret: "" + #@schema/desc "The XSRF key." + #@assert/validate len=32 + xsrfKey: "" +jobservice: + replicas: 1 + #@schema/desc "Secret is used when job service communicates with other components." + secret: "" +registry: + replicas: 1 + #@ text = """ + #@ Secret used to secure the upload state from client and registry storage backend. + #@ See: https://github.com/docker/distribution/blob/master/docs/configuration.md#http + #@ """ + #@schema/desc text + secret: "" +notary: + #@schema/desc "Whether to install Notary" + enabled: true + +#@schema/desc "Trivy scanner configuration" +#@schema/nullable +trivy: + replicas: 1 + #@schema/desc "gitHubToken the GitHub access token to download Trivy DB" + gitHubToken: "" + #@ text = """ + #@ The flag to disable Trivy DB downloads from GitHub + #@ + #@ You might want to set the value of this flag to `true` in test or CI/CD environments to avoid GitHub rate limiting issues. + #@ If the value is set to `true` you have to manually download the `trivy.db` file and mount it in the + #@ `/home/scanner/.cache/trivy/db/trivy.db` path. + #@ """ + #@schema/desc text + skipUpdate: false + +#@ text = """ +#@ The persistence is always enabled and a default StorageClass +#@ is needed in the k8s cluster to provision volumes dynamically. +#@ Specify another StorageClass in the "storageClass" or set "existingClaim" +#@ if you have already existing persistent volumes to use. +#@ +#@ For storing images and charts, you can also use "azure", "gcs", "s3", +#@ "swift" or "oss". Set it in the "imageChartStorage" section +#@ """ +#@schema/desc text +persistence: + #@ pvc_existing_claim_desc = "Use the existing PVC which must be created manually before bound, and specify the 'subPath' if the PVC is shared with other components" + + #@ pvc_storage_class_desc = """ + #@ Specify the 'storageClass' used to provision the volume. Or the default + #@ StorageClass will be used(the default). + #@ Set it to '-' to disable dynamic provisioning + #@ """ + #@assert/validate-string-defaults min_len=0 + persistentVolumeClaim: + #@assert/validate ("Either existingClaim or storageClass can be specified, but not both", lambda v: v.existing.....) + registry_b: #@ persistent_volume_claim_config(size="10Gi") + #@assert/validate ("", validPersistentVolumeClaimConfig) + registry: + #@schema/desc pvc_existing_claim_desc + existingClaim: "" + #@schema/desc pvs_storage_class_desc + storageClass: "" + subPath: "" + accessMode: ReadWriteOnce + size: 10Gi + #@assert/validate ("", validPersistentVolumeClaimConfig) + jobservice: + #@schema/desc pvc_existing_claim_desc + existingClaim: "" + #@schema/desc pvs_storage_class_desc + storageClass: "" + subPath: "" + accessMode: ReadWriteOnce + size: 1Gi + #@assert/validate ("", validPersistentVolumeClaimConfig) + database: + #@schema/desc pvc_existing_claim_desc + existingClaim: "" + #@schema/desc pvs_storage_class_desc + storageClass: "" + subPath: "" + accessMode: ReadWriteOnce + size: 1Gi + #@assert/validate ("", validPersistentVolumeClaimConfig) + redis: + #@schema/desc pvc_existing_claim_desc + existingClaim: "" + #@schema/desc pvs_storage_class_desc + storageClass: "" + subPath: "" + accessMode: ReadWriteOnce + size: 1Gi + #@assert/validate ("", validPersistentVolumeClaimConfig) + trivy: + #@schema/desc pvc_existing_claim_desc + existingClaim: "" + #@schema/desc pvs_storage_class_desc + storageClass: "" + subPath: "" + accessMode: ReadWriteOnce + size: 5Gi + + #! Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift", "oss" by filling in the information + #! in the corresponding section. + #! You must use "filesystem" if you want to use persistent volumes for registry and chartmuseum + #@assert/validate one_not_null=["filesystem", "azure", "gcs", "s3", "swift", "oss"] + + #@ text = """ + #@ Define which storage backend is used for registry and chartmuseum to store + #@ images and charts. Refer to + #@ https://github.com/docker/distribution/blob/master/docs/configuration.md#storage + #@ for the detail. + #@ """ + #@schema/desc text + imageChartStorage: + #@ text = """ + #@ Specify whether to disable `redirect` for images and chart storage, for + #@ backends which not supported it (such as using minio for `s3` storage type), please disable + #@ it. To disable redirects, simply set `disableredirect` to `true` instead. + #@ Refer to + #@ https://github.com/docker/distribution/blob/master/docs/configuration.md#redirect + #@ for the detail. + #@ """ + #@schema/desc text + disableredirect: false + #@ text = """ + #@ Specify the "caBundleSecretName" if the storage service uses a self-signed certificate. + #@ The secret must contain keys named "ca.crt" which will be injected into the trust store + #@ of registry's and chartmuseum's containers. + #@ """ + #@schema/desc text + #@assert/validate min_len=0 + caBundleSecretName: "" + + #@schema/nullable + filesystem: + rootdirectory: /storage + #@schema/nullable + maxthreads: 100 + #@schema/nullable + azure: + accountname: "" + #@schema/desc "base-64 encoded account key" + accountkey: "" + container: "" + #@assert/validate min_len=0 + #@schema/examples ("Using the default realm", "core.windows.net") + realm: "" + #@schema/nullable + gcs: + bucket: "" + #@schema/desc The base64 encoded json file which contains the key + #@assert/validate min_len=0 + encodedkey: "" + #@assert/validate min_len=0 + rootdirectory: "" + chunksize: 5242880 + #@schema/nullable + s3: + #@assert/validate one_of=["af-south-1", "ap-east-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-southeast-1", "ap-southeast-2", "ap-southeast-3", "ca-central-1", "eu-central-1", "eu-north-1", "eu-south-1", "eu-west-1", "eu-west-2", "eu-west-3", "me-south-1", "sa-east-1", "us-east-1", "us-east-2", "us-gov-east-1", "us-gov-west-1", "us-west-1", "us-west-2"] + region: us-west-1 + bucket: "" + #@schema/examples ("", "awsaccesskey") + accesskey: "" + #@schema/examples ("", "awssecretkey") + secretkey: "" + #@schema/examples ("Local endpoint", "http://myobjects.local") + #@assert/validate min_len=0 + regionendpoint: "" + encrypt: false + #@schema/examples ("", "mykeyid") + keyid: "" + secure: true + skipverify: false + v4auth: true + #@schema/nullable + chunksize: 0 + #@assert/validate min_len=0 + rootdirectory: "" + #@assert/validate one_of=["DEEP_ARCHIVE", "GLACIER", "INTELLIGENT_TIERING", "ONEZONE_IA", "REDUCED_REDUNDANCY", "STANDARD", "STANDARD_IA"] + storageclass: STANDARD + #@schema/nullable + multipartcopychunksize: 0 + #@schema/nullable + multipartcopymaxconcurrency: 0 + #@schema/nullable + multipartcopythresholdsize: 0 + + #@assert/validate one_of=["1", "10"], when_null_skip=True + #@schema/nullable + swift: + #@schema/examples ("Example", "https://storage.myprovider.com/v3/auth") + authurl: "" + username: "" + password: "" + container: "" + #@schema/examples ("France", "fr") + region: "" + #@schema/examples ("", "tenantname") + tenant: "" + #@schema/examples ("", "tenantid") + tenantid: "" + #@schema/examples ("", "domainname") + domain: "" + #@schema/examples ("", "domainid") + domainid: "" + #@schema/examples ("", "trustid") + trustid: "" + #@schema/nullable + insecureskipverify: false + #@schema/nullable + #@assert/validate format="quantity", when_null_skip=True + #@schema/examples ("Five meg chunks", "5M") + chunksize: "" + #@schema/nullable + prefix: "" + #@schema/nullable + #@schema/examples ("", "secretkey") + secretkey: "" + #@schema/nullable + #@schema/examples ("", "accesskey") + accesskey: "" + #@schema/nullable + authversion: 3 + #@schema/nullable + #@schema/examples ("", "public") + endpointtype: "" + #@schema/nullable + tempurlcontainerkey: false + #@schema/nullable + tempurlmethods: "" + #@schema/nullable + oss: + #@schema/examples ("", "accesskeyid") + accesskeyid: "" + #@schema/examples ("", "accesskeysecret") + accesskeysecret: "" + #@schema/examples ("", "regionname") + region: "" + #@schema/examples ("", "bucketname") + bucket: "" + #@schema/nullable + endpoint: "" + #@schema/nullable + internal: false + #@schema/nullable + encrypt: false + #@schema/nullable + secure: true + #@schema/nullable + #@schema/examples ("Ten megabytes", "10M") + chunksize: "" + #@schema/nullable + rootdirectory: "" + +#@schema/desc "The http/https network proxy for core, exporter, jobservice, trivy" +proxy: + httpProxy: + httpsProxy: + noProxy: 127.0.0.1,localhost,.local,.internal + +#! The PSP names used by Harbor pods. The names are separated by ','. 'null' means all PSP can be used. +pspNames: null + +#! The metrics used by core, registry and exporter +metrics: + enabled: false + core: + path: /metrics + port: 8001 + registry: + path: /metrics + port: 8001 + exporter: + path: /metrics + port: 8001 +``` + +### Other Approaches Considered + +#### Default non-empty validation for strings + +From the beginning of our design of this feature, we expected to provide a default validation for strings: + +```yaml +#@data/values-schema +--- +foo: "" +``` + +would be equivalent to: + +```yaml +#@data/values-schema +--- +#@schema/validation min_len=1 +foo: "" +``` + +However, doing so would result in a breaking change: anyone using a ytt library that adopted Schema would suddenly see an error message that they needed to specify a value for a data value they previously had not. + +Example: + +```yaml +#@data/values-schema +--- +kp_default_repository: "" +kp_default_repository_username: "" +kp_default_repository_password: "" + +http_proxy: "" +https_proxy: "" +no_proxy: "" +``` +(src: [tomkennedy513/carvel-package-kpack:/config/schema.yaml](https://github.com/tomkennedy513/carvel-package-kpack/blob/2d7ffd7276db244c6a574019ebb80603115d1009/config/schema.yaml)) + +#### Automatically including `when_null_skip` for nullable data values + +Had considered automatically short-circuiting validations on a Data Value that was also annotated `@schema/nullable`. Thinking: "when would someone _ever_ want the validations to fire when the value is null?!?!" + +Providing the `when_null_skip=` keyword has these advantages: +- makes the behavior explicit; +- carries the behavior into other contexts (e.g. validation on an output document); +- make schema and validations features as orthogonal as possible. + +#### More feature rich Validation Functions + +- `validation_func` (`function`) — zero-to-many "validation functions" + - predicate: `func (value) : bool | (bool, string)` + - assertion: `func (value) : None` (that `assert.fail()`s when condition is not met) + - validator: `func () : {"description": string, "type": string}` + - a "predicate" or "assertion" that when invoked with no arguments returns a specification of the validator. + +**Validation Function Specification** + +A validation function can improve the consumer's experience by supplying a "validation function specification". This is +a dictionary value returned when the function is invoked with no arguments: +```python +{ + "description": "(a definition of what a valid value is/means.)", + "type": "int" +} +``` + +- `description` (`string`) will be used in two cases: a) when generating documentation; b) as a hint when reporting a violation. +- `type` (enum: Starlark type) enables `ytt` to automatically check the type of the annotated node to ensure the validation is appropriate for that node. + +#### Programmatically invoking multiple validations + +At one point, it seemed tempting to provide a facility by which authors could name a set of validations to run over a tree of nodes. + +Here's one "prototype": + +```yaml +#@ load("@ytt:assert", "assert") + +#! Checks that a Persistent Volume configuration contains valid values. +#@ def validPV(pv): +#@ return assert.validate_all( +#@ ("accessMode", pv.accessMode, ("a PV access mode", validation.one_of["ReadWriteOnce", "ReadWriteOncePod", "ReadWriteMany", "ReadOnlyMany"]))), +#@ ("size", pv.size, ("", validation.format("quantity")) +#@ ) +#@ end +``` + +But this kind of move separates the validation from the node it is targeting, which then requires that the system provide a means for naming nodes (i.e. providing the path to the node). + +Instead of creating and solving that problem, if we encourage users to stay _within_ the structure (i.e. attach validations on the node against which violation messages will be reported), none of this work need be done. + +For now, we've decided to not pursue this approach until we can get clearer signal that the facilities we _do_ provide are too awkward in a sufficient number of cases as to warrant absorbing that complexity into the tool. In 2022 Q1, this seemed doubtful. + +#### @schema/required + +In consideration is whether another annotation would be practically useful in schema to indicate that a given data value is required. + +https://github.com/vmware-tanzu/carvel-ytt/issues/556 + +For now, we're deferring this approach until we get a clearer indication of how the "require non-empty" approach is working out (or not). + + +#### Allow multiple validations using sequence numbers + +re: [Merging validations on schema nodes](#consideration-merging-validations-on-schema-nodes)... + +Accept a sequence as part of the annotation name: + +```yaml +#@assert/validate ... +#@assert/validate1 ... +``` + +...which merges nicely with... + +```yaml +#@assert/validate2 +``` + +#### Defining merge semantics for annotations in an overlay + +... so that (for example) a consumer can edit `@assert/validate` annotations + +##### Replacing an existing annotation + +```yaml +#! base-schema.yml + +#@data/values-schema +--- +#@assert/validate is_even +foo: 42 +bar: 13 +``` + +...overlayed with... + +```yaml +#! schema-overlay.yml + +#@data/values-schema +--- +#@assert/validate lambda n: n < 1000 +#@overlay/replace-ann +foo: 13 +#@assert/validate lambda n: n < 1000 +#@overlay/replace-ann +bar: 1001 +``` +- requiring the consumer to supply the + +... yields ... + +```yaml +#@data/values-schema +--- +#@assert/validate lambda n: n < 1000 +foo: 13 +#@assert/validate lambda n: n < 1000 +bar: 1001 +``` + +##### Merging over an existing annotation + +- new arguments are appended, +- new keyword arguments are appended, +- existing keyword arguments are replaced. + +```yaml +#! base-schema.yml + +#@data/values-schema +--- +#@assert/validate is_even, not_null=True +foo: 42 +bar: 13 +``` + +...overlayed with... + +```yaml +#! schema-overlay.yml + +#@data/values-schema +--- +#@assert/validate lambda n: n < 1000, min=10, not_null=False +#@overlay/noop +#@overlay/merge-anns +foo: ~ +#@assert/validate lambda n: n < 1000 +#@overlay/noop +#@overlay/merge-anns +bar: ~ +``` + +... yields ... + +```yaml +#@data/values-schema +--- +#@assert/validate is_even, lambda n: n < 1000, not_null=False, min=10 +foo: 42 +#@assert/validate lambda n: n < 1000 +bar: 13 +``` + +##### Removing an existing annotation + +```yaml +#! base-schema.yml + +#@data/values-schema +--- +#@assert/validate is_even, not_null=True +foo: 42 +#@schema/nullable +#@assert/validate not_null=True +bar: 13 +``` + +...overlayed with... + +_(or with an overlay-namespaced annotation)_ +```yaml +#! schema-overlay.yml + +#@data/values-schema +--- +#@overlay/noop +#@overlay/remove-anns "schema/validate" +foo: 13 +#@overlay/noop +#@overlay/remove-anns "schema/validate" +#@overlay/validate min=1, max=1000 +bar: 0 +``` +- include a `via=` kwarg takes a predicate, called once for each existing annotation? + +... yields ... + +```yaml +#@data/values-schema +--- +foo: 42 +#@overlay/nullable +#@overlay/validate min=1, max=1000 +bar: 13 +``` + +## Implementation Considerations + +### Consideration: Attaching Validations to Nodes + +In a recent refactoring, `yamlmeta.Node` presents programmers with the ability to attach any kind of metadata to a Node. +This proposal presumes that validations are attached to Nodes using this facility (rather than say as a property of a `schema.Type`). + +In this way, validation is orthogonal to, but readily composable with Schema functionality. + +### Consideration: `@assert/validate` Keyword arguments are syntactic sugar over `@ytt:assert` functions + +Consider prior art around how functionality that's exposed through a Starlark built-in is reused in other contexts: + +https://github.com/vmware-tanzu/carvel-ytt/blob/c43dcf06798d3ed246e7c458f829a0d63f956e05/pkg/yttlibrary/yaml.go#L31-L69 + +For example `@ytt:assert.not_null()`'s functionality is likely literally used in implementing `@assert/validate not_null=True`. + + +### Consideration: Decode yamlfragments to Starlark values in assertion functions + + + +### Consideration: Setting validation defaults for strings in a schema overlay + +When exploring how to implement [@assert/validate-string-defaults](#assertvalidate-string-defaults), there are challenges. + +Setting up an example: + +Two (2) schema files: a base and an overlay. + +```yaml +#! base-schema.yml + +#@data/values-schema +--- +#@assert/validate min_len=5 +hostname: "" + +proxy: "" +``` + +```yaml +#! schema-overlay.yml + +#@data/values-schema +#@overlay/match-child-defaults missing_ok=True +#@assert/validate-string-defaults min_len=1 +--- +kp_default_repository: "" +#@assert/validate min_len=0 +kp_username: "" +limit: #@ 30 +``` +_(the `#@ 30` expression is here to remind us that we can't determine the type of the value of a node until it is evaluated — not just parsed.)_ + +#### Challenge A: No merge semantics for annotations + +Currently, `ytt`'s overlaying mechanism specifically preserves all annotations of the "left" side of the overlay operation (i.e. the base). The only way to change the annotations on the "left" is to completely replace that node. + +So, merging our example with the current release of `ytt` (v0.38.0) results in: + +```yaml +#@data/values-schema +--- +#@assert/validate min_len=5 +hostname: "" + +proxy: "" +kp_default_repository: "" +#@assert/validate min_len=0 +kp_username: "" +limit: 30 +``` +where: +- 👎 the `@assert/validate-string-defaults` annotation is effectively ignored. + +#### Solution A1: Implement merge semantics for annotations + +... let's stipulate that we _do_ implement merging of annotations (as described in [Defining merge semantics for annotations in an overlay](#defining-merge-semantics-for-annotations-in-an-overlay), then we hit the next challenge... + + +#### Challenge B: Unintended scoping of "normalized" annotations + +We classify `@assert/validate-string-defaults` as a "normalized annotation" — by that we mean that it represents a factoring out of a set of annotations within a tree of nodes. In our example, it's the "factoring out" of `@assert/validate min_len=1` from `kp_default_repository:`. + +The next challenge is that the naive merge causes the normalized annotation to apply to not just the nodes of the overlay, but of the entire overlayed result: + +```yaml +#@data/values-schema +#@assert/validate-string-defaults min_len=1 +--- +#@assert/validate min_len=5 +hostname: "" + +http_proxy: "" + +kp_default_repository: "" +#@assert/validate min_len=0 +kp_username: "" +limit: 30 +``` +where: +- 👎 `http_proxy:` suddenly _also_ had the `min_len=1` default applied to it (surprising) + + +#### Solution B1: Denormalize annotations before overlaying + +One approach is to "distribute" the annotation to all the descendents _before_ the overlay is applied: + +```yaml +#! schema-overlay.yml + +#@data/values-schema +#@overlay/match-child-defaults missing_ok=True +--- +#@assert/validate-string-defaults min_len=1 +kp_default_repository: "" +#@assert/validate min_len=0 +#@assert/validate-string-defaults min_len=1 +kp_username: "" +#@assert/validate-string-defaults min_len=1 +limit: 30 +``` +Note: +- resist the temptation to check the type of the value of each node; instead wait until we have formal schema type information available (after all schema has been merged). +- we take care to retain the "blame"/Position of the original/normalized annotation. + +Which results in a net schema: + +```yaml +#@data/values-schema +--- +#@assert/validate min_len=5 +hostname: "" + +proxy: "" +#@assert/validate-string-defaults min_len=1 +kp_default_repository: "" +#@assert/validate min_len=0 +#@assert/validate-string-defaults min_len=1 +kp_username: "" +#@assert/validate-string-defaults min_len=1 +limit: 30 +``` + +... that can be compiled into a `schema.Type` composite tree. + +In type compilation process, the `@assert/validate-string-defaults` is properly combined with the existing annotations (if any): + +```yaml +#@data/values-schema +--- +#@assert/validate min_len=5 +hostname: "" + +proxy: "" +#@assert/validate min_len=1 +kp_default_repository: "" +#@assert/validate min_len=0 +kp_username: "" +limit: 30 +``` +where: +- `kp_default_repository:` obtains its validation from the defaults. +- `kp_username` — already having a `@assert/validate` present, ignores the defaults. +- `limit` — having a type other than `"string"`, ignores the defaults. + From bb3eb193d40930cd55632b7e0d715212437bd8a3 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Wed, 16 Feb 2022 15:40:52 -0800 Subject: [PATCH 20/35] Add @schema/validation annotation Following discussions with [gcheadle-vmware](https://github.com/vmware-tanzu/carvel-ytt/issues/604#issuecomment-1034353974) and @cppforlife, it became clear that @assert/validate ought to always be consumed in a step after overlays are applied for the current pipeline. When writing schema, the author intends not for the schema document _itself_ to be validated, but to validate the _produced_ document (of which the schema is a basis for). Authors need a way to declare the intention for a validation to be attached to the type composite: @schema/validation. --- proposals/ytt/004-schema-validation/README.md | 230 +++++++++++------- 1 file changed, 136 insertions(+), 94 deletions(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index 14ba65069..03aaa3563 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -39,7 +39,8 @@ In the Kubernetes space, as cluster deployments become **increasingly complex**, - a function that asserts the rule against an actual value; - a violation message template, used to report invalid values. - **custom rule** — a rule based on a function (either one written by a user or provided in a `ytt` module). - - **named rule** — a rule provided with `ytt` expressed through a keyword argument to `@assert/validate`. + - **named rule** — a rule provided with `ytt` expressed through a keyword argument to an `@schema/validation` or `@assert/validate`. +- **schema base document** — the document instantiated from a schema; when the schema describes Data Values, it is the annotated default data values document. - **validation** — the binding of one or more "rule"s to a "node". - **violation** — an instance of where a given value of a "node" fails a "validation". @@ -49,6 +50,10 @@ Implement all plumbing logic required to provide high-quality data validation ov Mechanically, this means: +Extend `@schema/...` to be able to annotate validations in schema: +- introduce `@schema/validation` which attaches a validation to a type. +- when generating default Data Values (generally, the schema base document), when a type has a validation, annotate the corresponding node with an `@assert/validate` with that validation. + Extend the `@ytt:assert` module, to include the ability to validate: - introduce an annotation (`@assert/validate`) which attaches one or more "rules" to a "node", defining what a valid value is for that node; - supply a set of common assertions (functions in the `@ytt:assert` module) that covers most validation use-cases; @@ -105,9 +110,9 @@ The recommended way for an author to indicate required string input is to expect #@data/values-schema --- dex: - #@assert/validate min_len=1 + #@schema/validation min_len=1 namespace: "" - #@assert/validate min_len=1 + #@schema/validation min_len=1 username: "" ``` @@ -115,7 +120,7 @@ This case is so common (it is our understanding that most string-based data valu ```yaml #@data/values-schema -#@assert/validate-string-defaults min_len=1 +#@schema/validation-defaults-for-strings min_len=1 --- dex: namespace: "" @@ -133,7 +138,7 @@ Similarly, a non-empty value of an array has at least one element: --- dex: oauth2: - #@assert/validate min_len=1 + #@schema/validation min_len=1 responseTypes: - "" ``` @@ -148,7 +153,7 @@ For some integer-typed (or float-typed) data values, a zero value may be invalid ```yaml #@data/values-schema dex: - #@assert/validate min=1024, max=65535 + #@schema/validation min=1024, max=65535 port: 0 ``` @@ -162,7 +167,7 @@ Some types have no natural (or easily detectable) "empty" value. In these cases, #@data/values-schema --- #@schema/nullable -#@assert/validate not_null=("Cloud credentials are required.", True) +#@schema/validation not_null=("Cloud credentials are required.", True) credential: name: secretContents: @@ -180,7 +185,7 @@ Values can be constrained to be one of a finite site (i.e. of an enumeration). --- volumeSnapshotLocation: spec: - #@assert/validate one_of=["aws", "azure", "vsphere"] + #@schema/validation one_of=["aws", "azure", "vsphere"] provider: "" ``` @@ -192,7 +197,7 @@ An author can require that exactly one of a set of sibling node has a value: #@data/values-schema --- dex: - #@assert/validate one_not_null=["oidc", "ldap"] + #@schema/validation one_not_null=["oidc", "ldap"] config: #@schema/nullable oidc: @@ -251,7 +256,7 @@ The author places the validation at the lowest-common parent Node: #@ end #@data/values-schema -#@assert/validate ("There should be exactly one (1) registry replica if Helm Charts are stored on the filesystem.", one_registry_if_pvc_is_filesystem) +#@schema/validation ("There should be exactly one (1) registry replica if Helm Charts are stored on the filesystem.", one_registry_if_pvc_is_filesystem) --- persistence: imageChartStorage: @@ -274,7 +279,7 @@ Whenever a node is annotated `@schema/nullable`, then its default value is `null ```yaml #@data/values-schema --- -#@assert/validate min=42, max=42 +#@schema/validation min=42, max=42 #@schema/nullable foo: 13 ``` @@ -284,7 +289,7 @@ The author may opt to short-circuit validations when the actual value is `null`: ```yaml #@data/values-schema --- -#@assert/validate min=42, max=42, when_null_skip=True +#@schema/validation min=42, max=42, when_null_skip=True #@schema/nullable foo: 13 ``` @@ -304,7 +309,7 @@ The author can short-circuit validations under certain conditions: ```yaml #@data/values-schema --- -#@assert/validate ("", valid_service_config), when=lambda v: v.enabled +#@schema/validation ("", valid_service_config), when=lambda v: v.enabled service: enabled: true type: NodePort @@ -499,13 +504,20 @@ The default `message` is: - centering around the word "...requires..." helps the message work both in the case where the consumer supplied no value (see [Use Case: Required input](#use-case-required-input)) _and_ when they supplied an invalid value. - `{value}` is omitted to not contribute to the risk of leaking secrets. +> **Note** \ +> Examples below illustrate how an author would specify a validation (using `@schema/validation`) that becomes an `@assert/validate` as this is the driving use case at this time. _Example 1: Named rules_ ```yaml #@data/values-schema --- - +#@schema/validation min=49142 max=65535 +adminPort: 1024 +``` +which yields a schema base document: +```yaml +--- #@assert/validate min=49142 max=65535 adminPort: 1024 ``` @@ -519,6 +531,12 @@ _Example 2: Custom assertion-based rule_ #@ n >= 49142 and n <= 65535 or assert.fail("{} is not in the dynamic port range".format(n)) #@ end +#@schema/validation ("a TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port) +adminPort: 1024 +``` +which yields a schema base document: +```yaml +--- #@assert/validate ("a TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port) adminPort: 1024 ``` @@ -535,10 +553,17 @@ _Example 3: Custom violation message_ #@ n >= 49142 and n <= 65535 or assert.fail("{} is not in the dynamic port range".format(n)) #@ end +#@schema/validation ("a TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port, "\"{key}\" (={value}) must be between 49142 and 65535") +adminPort: 1024 +``` +which yields a schema base document: +```yaml +--- #@assert/validate ("a TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port, "\"{key}\" (={value}) must be between 49142 and 65535") adminPort: 1024 ``` + Yields: > "adminPort" (=1024) must be between 49142 and 65535 @@ -551,6 +576,12 @@ _Example 4: Custom predicate-based rule_ #@ return n >= 49142 and n <= 65535 #@ end +#@schema/validation ("A TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port) +adminPort: 1024 +``` +which yields a schema base document: +```yaml +--- #@assert/validate ("A TCP/IP port in the \"dynamic\" range: 49142 and 65535, inclusive", is_valid_port) adminPort: 1024 ``` @@ -605,19 +636,6 @@ username: "" Note: if an author wishes to provide a custom violation message, they would use the corresponding built-in rule as a custom rule. -#### @assert/validate-string-defaults - -> **TODO:** -> - [ ] firm-up name of this annotation: is it a bug or a feature that it is type-specific? - -Defines what a valid value is for all descendants which hold a "string" value. - -``` -@assert/validate-string-defaults [rule0, rule1...] [,] [,when=] [,when_null_skip=] -``` - -See also: [Consideration: Setting validation defaults for strings in a schema overlay](#consideration-setting-validation-defaults-for-strings-in-a-schema-overlay). - #### @ytt:assert.validate() Programmatically asserts that a given value is valid based on the set of supplied rules. @@ -699,6 +717,30 @@ When present, this validator is always checked first. ##### @ytt:assert.starts_with() +#### @schema/validation + +Attaches a validation to the type being declared by the annotated node. + +``` +@schema/validation [rule0, rule1, ...] [,] [,when=] [,when_null_skip=] +``` + +When the schema base document is generated, the corresponding node is annotated with `@assert/validate` with the same configuration provide _this_ annotation. + +See [@assert/validate](#assertvalidate) for details about arguments to this annotation. + +#### @schema/validation-defaults-for-strings + +> **TODO:** +> - [ ] firm-up name of this annotation: is it a bug or a feature that it is type-specific? + +Defines what a valid value is for all descendants which hold a "string" value. + +``` +@schema/validation-defaults-for-strings [rule0, rule1...] [,] [,when=] [,when_null_skip=] +``` + +See also: [Consideration: Setting validation defaults for strings in a schema overlay](#consideration-setting-validation-defaults-for-strings-in-a-schema-overlay). #### Custom Validation Functions > **TODO:** @@ -720,7 +762,7 @@ available with existing mechanisms: 1. **Q:** How are `one_not_null` expressed in CRDs (or built-in types) e.g. go look at Volumes (for each array item). \ https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/_print/#validation -2. **Q:** `@assert/validate-string-defaults` feels overly specific; is there a way to both generalize the notion of defaulting while keeping a reasonable syntax? +2. **Q:** `@schema/validation-defaults-for-strings` feels overly specific; is there a way to both generalize the notion of defaulting while keeping a reasonable syntax? 3. **Q:** in the case where a function is placed higher up on the tree, but really the value being checked is one of the children, should we give the facility to be able to name the path of the target key? In this way, the violation message can be associated with the child (as well). 4. **Q:** When evaluating a private library, should validations be _automatically_ evaluated? 5. **Q:** Should we provide a "break-the-glass" means of capturing a form of a validation in CEL? @@ -764,33 +806,33 @@ From: To: ```yaml -#@ load("@ytt:validation", "validation") +#@ load("@ytt:assert", "assert") #@ #@ def one_registry_if_pvc_is_filesystem(val): #@ if val.persistence.imageChartStorage.type == "filesystem" and #@ val.persistence.persistentVolumeClaim.registry.accessMode == "ReadWriteOnce": #@ return val.registry.replicas == 1 \ -#@ or (False, "Charts are being stored in a 'filesystem' type PVC and the Registry's PVC access mode is 'ReadWriteOnce': there must be exactly one (1) replica of the registry but {} replicas are configured.".format(val.registry.replicas)) +#@ or assert.fail("Charts are being stored in a 'filesystem' type PVC and the Registry's PVC access mode is 'ReadWriteOnce': there must be exactly one (1) replica of the registry but {} replicas are configured.".format(val.registry.replicas)) #@ end #@ end #@data/values-schema -#@assert/validate ("There can be exactly one (1) registry replica if Helm Charts are stored on the filesystem.", one_registry_if_pvc_is_filesystem) -#@assert/validate-string-defaults min_len=1 +#@schema/validation ("There can be exactly one (1) registry replica if Helm Charts are stored on the filesystem.", one_registry_if_pvc_is_filesystem) +#@schema/validation-defaults-for-strings min_len=1 --- #@schema/desc "The namespace to install Harbor" namespace: harbor #@schema/desc "The FQDN for accessing Harbor admin UI and Registry service." -#@assert/validate min_len=1, format="hostname" +#@schema/validation min_len=1, format="hostname" hostname: harbor.yourdomain.com #@schema/desc "The network port of the Envoy service in Contour or other Ingress Controller." port: - #@assert/validate min=1, max=65535 + #@schema/validation min=1, max=65535 https: 443 #@schema/desc "The log level of core, exporter, jobservice, registry." -#@assert/validate one_of=["debug", "info", "warning", "error", "fatal"] +#@schema/validation one_of=["debug", "info", "warning", "error", "fatal"] logLevel: info #@ text = """ @@ -805,7 +847,7 @@ tlsCertificate: #@schema/desc "The private key" tls.key: "" #@schema/desc "The certificate of CA, this enables the download link on portal to download the certificate of CA." - #@assert/validate min_len=0 + #@schema/validation min_len=0 ca.crt: "" #@schema/desc "Use contour http proxy instead of the ingress." @@ -815,7 +857,7 @@ enableContourHttpProxy: true harborAdminPassword: "" #@schema/desc "The secret key used for encryption." -#@assert/validate len=16 +#@schema/validation len=16 secretKey: "" database: @@ -827,7 +869,7 @@ core: #@schema/desc "Secret is used when core server communicates with other components." secret: "" #@schema/desc "The XSRF key." - #@assert/validate len=32 + #@schema/validation len=32 xsrfKey: "" jobservice: replicas: 1 @@ -879,11 +921,11 @@ persistence: #@ StorageClass will be used(the default). #@ Set it to '-' to disable dynamic provisioning #@ """ - #@assert/validate-string-defaults min_len=0 + #@schema/validation-defaults-for-strings min_len=0 persistentVolumeClaim: - #@assert/validate ("Either existingClaim or storageClass can be specified, but not both", lambda v: v.existing.....) + #@schema/validation ("Either existingClaim or storageClass can be specified, but not both", lambda v: v.existing.....) registry_b: #@ persistent_volume_claim_config(size="10Gi") - #@assert/validate ("", validPersistentVolumeClaimConfig) + #@schema/validation ("", validPersistentVolumeClaimConfig) registry: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -892,7 +934,7 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 10Gi - #@assert/validate ("", validPersistentVolumeClaimConfig) + #@schema/validation ("", validPersistentVolumeClaimConfig) jobservice: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -901,7 +943,7 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 1Gi - #@assert/validate ("", validPersistentVolumeClaimConfig) + #@schema/validation ("", validPersistentVolumeClaimConfig) database: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -910,7 +952,7 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 1Gi - #@assert/validate ("", validPersistentVolumeClaimConfig) + #@schema/validation ("", validPersistentVolumeClaimConfig) redis: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -919,7 +961,7 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 1Gi - #@assert/validate ("", validPersistentVolumeClaimConfig) + #@schema/validation ("", validPersistentVolumeClaimConfig) trivy: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -932,7 +974,7 @@ persistence: #! Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift", "oss" by filling in the information #! in the corresponding section. #! You must use "filesystem" if you want to use persistent volumes for registry and chartmuseum - #@assert/validate one_not_null=["filesystem", "azure", "gcs", "s3", "swift", "oss"] + #@schema/validation one_not_null=["filesystem", "azure", "gcs", "s3", "swift", "oss"] #@ text = """ #@ Define which storage backend is used for registry and chartmuseum to store @@ -958,7 +1000,7 @@ persistence: #@ of registry's and chartmuseum's containers. #@ """ #@schema/desc text - #@assert/validate min_len=0 + #@schema/validation min_len=0 caBundleSecretName: "" #@schema/nullable @@ -972,21 +1014,21 @@ persistence: #@schema/desc "base-64 encoded account key" accountkey: "" container: "" - #@assert/validate min_len=0 + #@schema/validation min_len=0 #@schema/examples ("Using the default realm", "core.windows.net") realm: "" #@schema/nullable gcs: bucket: "" #@schema/desc The base64 encoded json file which contains the key - #@assert/validate min_len=0 + #@schema/validation min_len=0 encodedkey: "" - #@assert/validate min_len=0 + #@schema/validation min_len=0 rootdirectory: "" chunksize: 5242880 #@schema/nullable s3: - #@assert/validate one_of=["af-south-1", "ap-east-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-southeast-1", "ap-southeast-2", "ap-southeast-3", "ca-central-1", "eu-central-1", "eu-north-1", "eu-south-1", "eu-west-1", "eu-west-2", "eu-west-3", "me-south-1", "sa-east-1", "us-east-1", "us-east-2", "us-gov-east-1", "us-gov-west-1", "us-west-1", "us-west-2"] + #@schema/validation one_of=["af-south-1", "ap-east-1", "ap-northeast-1", "ap-northeast-2", "ap-northeast-3", "ap-south-1", "ap-southeast-1", "ap-southeast-2", "ap-southeast-3", "ca-central-1", "eu-central-1", "eu-north-1", "eu-south-1", "eu-west-1", "eu-west-2", "eu-west-3", "me-south-1", "sa-east-1", "us-east-1", "us-east-2", "us-gov-east-1", "us-gov-west-1", "us-west-1", "us-west-2"] region: us-west-1 bucket: "" #@schema/examples ("", "awsaccesskey") @@ -994,7 +1036,7 @@ persistence: #@schema/examples ("", "awssecretkey") secretkey: "" #@schema/examples ("Local endpoint", "http://myobjects.local") - #@assert/validate min_len=0 + #@schema/validation min_len=0 regionendpoint: "" encrypt: false #@schema/examples ("", "mykeyid") @@ -1004,9 +1046,9 @@ persistence: v4auth: true #@schema/nullable chunksize: 0 - #@assert/validate min_len=0 + #@schema/validation min_len=0 rootdirectory: "" - #@assert/validate one_of=["DEEP_ARCHIVE", "GLACIER", "INTELLIGENT_TIERING", "ONEZONE_IA", "REDUCED_REDUNDANCY", "STANDARD", "STANDARD_IA"] + #@schema/validation one_of=["DEEP_ARCHIVE", "GLACIER", "INTELLIGENT_TIERING", "ONEZONE_IA", "REDUCED_REDUNDANCY", "STANDARD", "STANDARD_IA"] storageclass: STANDARD #@schema/nullable multipartcopychunksize: 0 @@ -1015,7 +1057,7 @@ persistence: #@schema/nullable multipartcopythresholdsize: 0 - #@assert/validate one_of=["1", "10"], when_null_skip=True + #@schema/validation one_of=["1", "10"], when_null_skip=True #@schema/nullable swift: #@schema/examples ("Example", "https://storage.myprovider.com/v3/auth") @@ -1038,7 +1080,7 @@ persistence: #@schema/nullable insecureskipverify: false #@schema/nullable - #@assert/validate format="quantity", when_null_skip=True + #@schema/validation format="quantity", when_null_skip=True #@schema/examples ("Five meg chunks", "5M") chunksize: "" #@schema/nullable @@ -1226,7 +1268,7 @@ Accept a sequence as part of the annotation name: #### Defining merge semantics for annotations in an overlay -... so that (for example) a consumer can edit `@assert/validate` annotations +... so that (for example) a consumer can edit `@schema/validation` annotations ##### Replacing an existing annotation @@ -1235,7 +1277,7 @@ Accept a sequence as part of the annotation name: #@data/values-schema --- -#@assert/validate is_even +#@schema/validation is_even foo: 42 bar: 13 ``` @@ -1247,10 +1289,10 @@ bar: 13 #@data/values-schema --- -#@assert/validate lambda n: n < 1000 +#@schema/validation lambda n: n < 1000 #@overlay/replace-ann foo: 13 -#@assert/validate lambda n: n < 1000 +#@schema/validation lambda n: n < 1000 #@overlay/replace-ann bar: 1001 ``` @@ -1261,9 +1303,9 @@ bar: 1001 ```yaml #@data/values-schema --- -#@assert/validate lambda n: n < 1000 +#@schema/validation lambda n: n < 1000 foo: 13 -#@assert/validate lambda n: n < 1000 +#@schema/validation lambda n: n < 1000 bar: 1001 ``` @@ -1278,7 +1320,7 @@ bar: 1001 #@data/values-schema --- -#@assert/validate is_even, not_null=True +#@schema/validation is_even, not_null=True foo: 42 bar: 13 ``` @@ -1290,11 +1332,11 @@ bar: 13 #@data/values-schema --- -#@assert/validate lambda n: n < 1000, min=10, not_null=False +#@schema/validation lambda n: n < 1000, min=10, not_null=False #@overlay/noop #@overlay/merge-anns foo: ~ -#@assert/validate lambda n: n < 1000 +#@schema/validation lambda n: n < 1000 #@overlay/noop #@overlay/merge-anns bar: ~ @@ -1305,9 +1347,9 @@ bar: ~ ```yaml #@data/values-schema --- -#@assert/validate is_even, lambda n: n < 1000, not_null=False, min=10 +#@schema/validation is_even, lambda n: n < 1000, not_null=False, min=10 foo: 42 -#@assert/validate lambda n: n < 1000 +#@schema/validation lambda n: n < 1000 bar: 13 ``` @@ -1318,10 +1360,10 @@ bar: 13 #@data/values-schema --- -#@assert/validate is_even, not_null=True +#@schema/validation is_even, not_null=True foo: 42 #@schema/nullable -#@assert/validate not_null=True +#@schema/validation not_null=True bar: 13 ``` @@ -1378,7 +1420,7 @@ For example `@ytt:assert.not_null()`'s functionality is likely literally used in ### Consideration: Setting validation defaults for strings in a schema overlay -When exploring how to implement [@assert/validate-string-defaults](#assertvalidate-string-defaults), there are challenges. +When exploring how to implement [@schema/validation-defaults-for-strings](#schemavalidation-defaults-for-strings), there are challenges. Setting up an example: @@ -1389,7 +1431,7 @@ Two (2) schema files: a base and an overlay. #@data/values-schema --- -#@assert/validate min_len=5 +#@schema/validation min_len=5 hostname: "" proxy: "" @@ -1400,10 +1442,10 @@ proxy: "" #@data/values-schema #@overlay/match-child-defaults missing_ok=True -#@assert/validate-string-defaults min_len=1 +#@schema/validation-defaults-for-strings min_len=1 --- kp_default_repository: "" -#@assert/validate min_len=0 +#@schema/validation min_len=0 kp_username: "" limit: #@ 30 ``` @@ -1418,17 +1460,17 @@ So, merging our example with the current release of `ytt` (v0.38.0) results in: ```yaml #@data/values-schema --- -#@assert/validate min_len=5 +#@schema/validation min_len=5 hostname: "" proxy: "" kp_default_repository: "" -#@assert/validate min_len=0 +#@schema/validation min_len=0 kp_username: "" limit: 30 ``` where: -- 👎 the `@assert/validate-string-defaults` annotation is effectively ignored. +- 👎 the `@schema/validation-defaults-for-strings` annotation is effectively ignored. #### Solution A1: Implement merge semantics for annotations @@ -1437,21 +1479,21 @@ where: #### Challenge B: Unintended scoping of "normalized" annotations -We classify `@assert/validate-string-defaults` as a "normalized annotation" — by that we mean that it represents a factoring out of a set of annotations within a tree of nodes. In our example, it's the "factoring out" of `@assert/validate min_len=1` from `kp_default_repository:`. +We classify `@schema/validation-defaults-for-strings` as a "normalized annotation" — by that we mean that it represents a factoring out of a set of annotations within a tree of nodes. In our example, it's the "factoring out" of `@schema/validation min_len=1` from `kp_default_repository:`. The next challenge is that the naive merge causes the normalized annotation to apply to not just the nodes of the overlay, but of the entire overlayed result: ```yaml #@data/values-schema -#@assert/validate-string-defaults min_len=1 +#@schema/validation-defaults-for-strings min_len=1 --- -#@assert/validate min_len=5 +#@schema/validation min_len=5 hostname: "" http_proxy: "" kp_default_repository: "" -#@assert/validate min_len=0 +#@schema/validation min_len=0 kp_username: "" limit: 30 ``` @@ -1469,12 +1511,12 @@ One approach is to "distribute" the annotation to all the descendents _before_ t #@data/values-schema #@overlay/match-child-defaults missing_ok=True --- -#@assert/validate-string-defaults min_len=1 +#@schema/validation-defaults-for-strings min_len=1 kp_default_repository: "" -#@assert/validate min_len=0 -#@assert/validate-string-defaults min_len=1 +#@schema/validation min_len=0 +#@schema/validation-defaults-for-strings min_len=1 kp_username: "" -#@assert/validate-string-defaults min_len=1 +#@schema/validation-defaults-for-strings min_len=1 limit: 30 ``` Note: @@ -1486,38 +1528,38 @@ Which results in a net schema: ```yaml #@data/values-schema --- -#@assert/validate min_len=5 +#@schema/validation min_len=5 hostname: "" proxy: "" -#@assert/validate-string-defaults min_len=1 +#@schema/validation-defaults-for-strings min_len=1 kp_default_repository: "" -#@assert/validate min_len=0 -#@assert/validate-string-defaults min_len=1 +#@schema/validation min_len=0 +#@schema/validation-defaults-for-strings min_len=1 kp_username: "" -#@assert/validate-string-defaults min_len=1 +#@schema/validation-defaults-for-strings min_len=1 limit: 30 ``` ... that can be compiled into a `schema.Type` composite tree. -In type compilation process, the `@assert/validate-string-defaults` is properly combined with the existing annotations (if any): +In type compilation process, the `@schema/validation-defaults-for-strings` is properly combined with the existing annotations (if any): ```yaml #@data/values-schema --- -#@assert/validate min_len=5 +#@schema/validation min_len=5 hostname: "" proxy: "" -#@assert/validate min_len=1 +#@schema/validation min_len=1 kp_default_repository: "" -#@assert/validate min_len=0 +#@schema/validation min_len=0 kp_username: "" limit: 30 ``` where: - `kp_default_repository:` obtains its validation from the defaults. -- `kp_username` — already having a `@assert/validate` present, ignores the defaults. +- `kp_username` — already having a `@schema/validation` present, ignores the defaults. - `limit` — having a type other than `"string"`, ignores the defaults. From 1c22300410ac60e17b16363db472a25b3db6e292 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Wed, 16 Feb 2022 15:46:24 -0800 Subject: [PATCH 21/35] Rename assert.is_valid to assert.valid The former is a predicate (expected to return True or False), the later is an assertion. --- proposals/ytt/004-schema-validation/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index 03aaa3563..74bb67a2d 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -649,12 +649,12 @@ where: This function is the programmatic equivalent of [@assert/validate](#assertvalidate). -#### @ytt:assert.is_valid() +#### @ytt:assert.valid() Asserts any validations present on a Node (and descendents). ```python -assert.is_valid(node) +assert.valid(node) ``` - if all validation rules are satisfied, nothing happens. From 4cb6a2db376ba51bfe560922c529cff88f8b9961 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Wed, 16 Feb 2022 16:02:09 -0800 Subject: [PATCH 22/35] Make "Merging annotation semantics" an impl consideration It seems quite likely that we'll need such a facility for more advanced cases. Placed in the "Other approaches considered" means that the ideas are found without sufficient merit; not the case here. --- proposals/ytt/004-schema-validation/README.md | 268 +++++++++--------- 1 file changed, 134 insertions(+), 134 deletions(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index 74bb67a2d..d802741e0 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -464,7 +464,7 @@ spec: ``` -Note: in order for validations to be applied via an overlay in any practical way, we would need to [define merge semantics of annotations in overlays](#defining-merge-semantics-for-annotations-in-an-overlay). +Note: in order for validations to be applied via an overlay in any practical way, we would need to [define merge semantics of annotations in overlays](#consideration-defining-merge-semantics-for-annotations-in-an-overlay). Sources: - [cloudfoundry/cf-for-k8s/tests](https://github.com/cloudfoundry/cf-for-k8s/tree/95c77d741706ab2a27979db90873fb4461a1ac3c/tests) @@ -751,7 +751,7 @@ Validations are checked, leaf-upwards: a parent value can't be valid if one of i #### Consideration: Merging Validations on Schema Nodes -Some preliminary design work has been done describe how to augment the Overlay module to better support editing of annotations. (see [Defining merge semantics for annotations in an overlay](#defining-merge-semantics-for-annotations-in-an-overlay) for details). +Some preliminary design work has been done describe how to augment the Overlay module to better support editing of annotations. (see [Defining merge semantics for annotations in an overlay](#consideration-defining-merge-semantics-for-annotations-in-an-overlay) for details). However, for the few that find themselves in a situation where they need to edit existing schema, they have workarounds available with existing mechanisms: @@ -1265,137 +1265,6 @@ Accept a sequence as part of the annotation name: ```yaml #@assert/validate2 ``` - -#### Defining merge semantics for annotations in an overlay - -... so that (for example) a consumer can edit `@schema/validation` annotations - -##### Replacing an existing annotation - -```yaml -#! base-schema.yml - -#@data/values-schema ---- -#@schema/validation is_even -foo: 42 -bar: 13 -``` - -...overlayed with... - -```yaml -#! schema-overlay.yml - -#@data/values-schema ---- -#@schema/validation lambda n: n < 1000 -#@overlay/replace-ann -foo: 13 -#@schema/validation lambda n: n < 1000 -#@overlay/replace-ann -bar: 1001 -``` -- requiring the consumer to supply the - -... yields ... - -```yaml -#@data/values-schema ---- -#@schema/validation lambda n: n < 1000 -foo: 13 -#@schema/validation lambda n: n < 1000 -bar: 1001 -``` - -##### Merging over an existing annotation - -- new arguments are appended, -- new keyword arguments are appended, -- existing keyword arguments are replaced. - -```yaml -#! base-schema.yml - -#@data/values-schema ---- -#@schema/validation is_even, not_null=True -foo: 42 -bar: 13 -``` - -...overlayed with... - -```yaml -#! schema-overlay.yml - -#@data/values-schema ---- -#@schema/validation lambda n: n < 1000, min=10, not_null=False -#@overlay/noop -#@overlay/merge-anns -foo: ~ -#@schema/validation lambda n: n < 1000 -#@overlay/noop -#@overlay/merge-anns -bar: ~ -``` - -... yields ... - -```yaml -#@data/values-schema ---- -#@schema/validation is_even, lambda n: n < 1000, not_null=False, min=10 -foo: 42 -#@schema/validation lambda n: n < 1000 -bar: 13 -``` - -##### Removing an existing annotation - -```yaml -#! base-schema.yml - -#@data/values-schema ---- -#@schema/validation is_even, not_null=True -foo: 42 -#@schema/nullable -#@schema/validation not_null=True -bar: 13 -``` - -...overlayed with... - -_(or with an overlay-namespaced annotation)_ -```yaml -#! schema-overlay.yml - -#@data/values-schema ---- -#@overlay/noop -#@overlay/remove-anns "schema/validate" -foo: 13 -#@overlay/noop -#@overlay/remove-anns "schema/validate" -#@overlay/validate min=1, max=1000 -bar: 0 -``` -- include a `via=` kwarg takes a predicate, called once for each existing annotation? - -... yields ... - -```yaml -#@data/values-schema ---- -foo: 42 -#@overlay/nullable -#@overlay/validate min=1, max=1000 -bar: 13 -``` - ## Implementation Considerations ### Consideration: Attaching Validations to Nodes @@ -1474,7 +1343,7 @@ where: #### Solution A1: Implement merge semantics for annotations -... let's stipulate that we _do_ implement merging of annotations (as described in [Defining merge semantics for annotations in an overlay](#defining-merge-semantics-for-annotations-in-an-overlay), then we hit the next challenge... +... let's stipulate that we _do_ implement merging of annotations (as described in [Defining merge semantics for annotations in an overlay](#consideration-defining-merge-semantics-for-annotations-in-an-overlay), then we hit the next challenge... #### Challenge B: Unintended scoping of "normalized" annotations @@ -1563,3 +1432,134 @@ where: - `kp_username` — already having a `@schema/validation` present, ignores the defaults. - `limit` — having a type other than `"string"`, ignores the defaults. + +### Consideration: Defining merge semantics for annotations in an overlay + +... so that (for example) a consumer can edit `@schema/validation` annotations + +#### Replacing an existing annotation + +```yaml +#! base-schema.yml + +#@data/values-schema +--- +#@schema/validation is_even +foo: 42 +bar: 13 +``` + +...overlayed with... + +```yaml +#! schema-overlay.yml + +#@data/values-schema +--- +#@schema/validation lambda n: n < 1000 +#@overlay/replace-ann +foo: 13 +#@schema/validation lambda n: n < 1000 +#@overlay/replace-ann +bar: 1001 +``` +- requiring the consumer to supply the + +... yields ... + +```yaml +#@data/values-schema +--- +#@schema/validation lambda n: n < 1000 +foo: 13 +#@schema/validation lambda n: n < 1000 +bar: 1001 +``` + +#### Merging over an existing annotation + +- new arguments are appended, +- new keyword arguments are appended, +- existing keyword arguments are replaced. + +```yaml +#! base-schema.yml + +#@data/values-schema +--- +#@schema/validation is_even, not_null=True +foo: 42 +bar: 13 +``` + +...overlayed with... + +```yaml +#! schema-overlay.yml + +#@data/values-schema +--- +#@schema/validation lambda n: n < 1000, min=10, not_null=False +#@overlay/noop +#@overlay/merge-anns +foo: ~ +#@schema/validation lambda n: n < 1000 +#@overlay/noop +#@overlay/merge-anns +bar: ~ +``` + +... yields ... + +```yaml +#@data/values-schema +--- +#@schema/validation is_even, lambda n: n < 1000, not_null=False, min=10 +foo: 42 +#@schema/validation lambda n: n < 1000 +bar: 13 +``` + +#### Removing an existing annotation + +```yaml +#! base-schema.yml + +#@data/values-schema +--- +#@schema/validation is_even, not_null=True +foo: 42 +#@schema/nullable +#@schema/validation not_null=True +bar: 13 +``` + +...overlayed with... + +_(or with an overlay-namespaced annotation)_ +```yaml +#! schema-overlay.yml + +#@data/values-schema +--- +#@overlay/noop +#@overlay/remove-anns "schema/validate" +foo: 13 +#@overlay/noop +#@overlay/remove-anns "schema/validate" +#@overlay/validate min=1, max=1000 +bar: 0 +``` +- include a `via=` kwarg takes a predicate, called once for each existing annotation? + +... yields ... + +```yaml +#@data/values-schema +--- +foo: 42 +#@overlay/nullable +#@overlay/validate min=1, max=1000 +bar: 13 +``` + From d22edfea156569151119688ce8bd007c717352f6 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Fri, 18 Feb 2022 11:11:54 -0800 Subject: [PATCH 23/35] Remove cruft from Harbor example --- proposals/ytt/004-schema-validation/README.md | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index d802741e0..db75fcb6c 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -812,7 +812,7 @@ To: #@ if val.persistence.imageChartStorage.type == "filesystem" and #@ val.persistence.persistentVolumeClaim.registry.accessMode == "ReadWriteOnce": #@ return val.registry.replicas == 1 \ -#@ or assert.fail("Charts are being stored in a 'filesystem' type PVC and the Registry's PVC access mode is 'ReadWriteOnce': there must be exactly one (1) replica of the registry but {} replicas are configured.".format(val.registry.replicas)) +#@ or assert.fail("{} replicas are configured".format(val.registry.replicas)) #@ end #@ end @@ -923,9 +923,6 @@ persistence: #@ """ #@schema/validation-defaults-for-strings min_len=0 persistentVolumeClaim: - #@schema/validation ("Either existingClaim or storageClass can be specified, but not both", lambda v: v.existing.....) - registry_b: #@ persistent_volume_claim_config(size="10Gi") - #@schema/validation ("", validPersistentVolumeClaimConfig) registry: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -934,7 +931,6 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 10Gi - #@schema/validation ("", validPersistentVolumeClaimConfig) jobservice: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -943,7 +939,6 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 1Gi - #@schema/validation ("", validPersistentVolumeClaimConfig) database: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -952,7 +947,6 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 1Gi - #@schema/validation ("", validPersistentVolumeClaimConfig) redis: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -961,7 +955,6 @@ persistence: subPath: "" accessMode: ReadWriteOnce size: 1Gi - #@schema/validation ("", validPersistentVolumeClaimConfig) trivy: #@schema/desc pvc_existing_claim_desc existingClaim: "" @@ -1048,7 +1041,7 @@ persistence: chunksize: 0 #@schema/validation min_len=0 rootdirectory: "" - #@schema/validation one_of=["DEEP_ARCHIVE", "GLACIER", "INTELLIGENT_TIERING", "ONEZONE_IA", "REDUCED_REDUNDANCY", "STANDARD", "STANDARD_IA"] + #@schema/validation one_of=["REDUCED_REDUNDANCY", "STANDARD"] storageclass: STANDARD #@schema/nullable multipartcopychunksize: 0 From fc6c4a02f4ffaaa907801ddc43a4b55af5681b67 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Fri, 18 Feb 2022 11:22:22 -0800 Subject: [PATCH 24/35] Add top-level TOC for easier navigation --- proposals/ytt/004-schema-validation/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index db75fcb6c..6b0a90ba4 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -8,6 +8,17 @@ approvers: [] # Validations +- [Problem Statement](#problem-statement) +- [Terminology / Concepts](#terminology--concepts) +- [Proposal](#proposal) + - [Use Cases](#use-cases) + - [Specification](#specification) + - [Open Questions](#open-questions) + - [Answered Questions](#answered-questions) + - [Complete Examples](#complete-examples) + - [Other Approaches Considered](#other-approaches-considered) +- [Implementation Considerations](#implementation-considerations) + ## Problem Statement To date, ytt Library / Carvel Package **authors have had to "roll their own" logic to check that customer's inputs are valid.** That is: that data values are a) present/non-empty; and b) within expected range(s). From 08d3d8be7270313c5356e3c46ad035b72c168262 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Fri, 4 Mar 2022 12:22:44 -0800 Subject: [PATCH 25/35] Add/promote named rules that are useful (from conversation with @vrabbi; that you Scott!) --- proposals/ytt/004-schema-validation/README.md | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index 6b0a90ba4..bce3c004d 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -404,13 +404,13 @@ Supported (ytt rule ==> OpenAPI v3 / JSON Schema property): - [ipv6](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.5) - [uri](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.6) - [uriref](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-7.3.7) +- `multiple_of=` / `@ytt:assert.multiple_of()` : [multipleOf](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.1) The following are deferred: - uniqueItems - exclusiveMaximum (is this useful?) - exclusiveMinimum (is this useful?) -- [multipleOf](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.1) - pattern (This string SHOULD be a valid regular expression, according to the ECMA 262 regular - expression dialect) The following JSON Schema properties are better supported as part of `ytt` schema: @@ -620,11 +620,16 @@ They are: General-purpose string assertions: - `starts_with=` (see [`@ytt:assert.starts_with()`](#yttassertstarts_with)) -- `ends_with=` (see [`@ytt:assert.ends_with()`](#yttassertends_with)) -- `contains=` (see [`@ytt:assert.contains()`](#yttassertcontains)) +- `ends_with=` (see [`@ytt:assert.ends_with()`](#yttassertends_with)) — e.g. ".git" for a git source in GitOps CRDs in a CI/CD system. +- `contains=` (see [`@ytt:assert.contains()`](#yttassertcontains)) — e.g. registry name within a URL, "@sha" to validate an image reference is digest-resolved. - `matches=` (see [`@ytt:assert.matches()`](#yttassertmatches)) - `format=` (see [`@ytt:assert.format()`](#yttassertformat)) +Kubernetes-common: +- `even=` (see [`@ytt:assert.even()`](#yttasserteven)) — e.g. memory resources should be even. +- `odd=` (see [`@ytt:assert.odd()`](#yttassertodd)) — e.g. replicas should be odd numbers in stateful applications. +- `multiple_of=` (see [`@ytt:assert.multiple_of()`](#yttassertmultiple_of)) — e.g. memory sizes ought to be multiples of 1024 + ###### Named rule arguments The argument of a named rule can either be just the value: @@ -669,7 +674,8 @@ assert.valid(node) ``` - if all validation rules are satisfied, nothing happens. -- if any validation rule is violated, `assert.fail()`s with the corresponding validation message(s). +- all violated validation rules result ..., `assert.fail()`s with the corresponding validation message(s). + - `violations` (`list`) #### Included Assertion Functions @@ -688,6 +694,8 @@ assert.valid(node) ##### @ytt:assert.ends_with() +##### @ytt:assert.even() + ##### @ytt:assert.format() Supported formats: @@ -718,10 +726,14 @@ The value is _at least_ (inclusive) of the given minimum. The length of the value is _at least_ (inclusive) that of the given minimum. +##### @ytt:assert.multiple_of() + ##### @ytt:assert.not_null() When present, this validator is always checked first. +##### @ytt:assert.odd() + ##### @ytt:assert.one_not_null() ##### @ytt:assert.one_of() @@ -1441,6 +1453,9 @@ where: ... so that (for example) a consumer can edit `@schema/validation` annotations +For example to: +- be able to remove/disable/edit a validation on a given Data Value (or output field). + #### Replacing an existing annotation ```yaml From b1d034d6bd53275bd1535111e0c852cd175ce239 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Mon, 7 Mar 2022 17:20:27 -0800 Subject: [PATCH 26/35] Note where docs will need special attention --- proposals/ytt/004-schema-validation/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index bce3c004d..202c84ee3 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -1582,3 +1582,14 @@ foo: 42 bar: 13 ``` +### Consideration: Proactively Address Likely-Challenging Topics in Documentation + +- The concept of declaring [Required Input](#use-case-required-input) by setting the default value to be invalid is not intuitive for many users. We anticipate this being a heavily-used feature and need to make sure it is grokable. + - documenting approaches around this idea should particularly _start_ from the user/reader's intention. + - sometimes acknowledging that a concept is counter-intuitive for some and buttressing the explanation with the explicit benefits for the user can help. + - is there an analogy that can help bridge the gap? + - see also: https://github.com/vmware-tanzu/carvel/pull/331#discussion_r820794818 +- When guiding users through the [Union Structures use case](#use-case-union-structures), be mindful there is a body of existing configuration that uses the discriminator style. + - It's a breaking change for them to modify their configuration to adopt the union-style we recommend. + - even as we recommend the union-style over discriminator, given the pervasive existence of the latter, it would ease adoption if there were specific guidance on how to implement/work with the former. + - see also: https://github.com/vmware-tanzu/carvel/pull/331#discussion_r820796795 From d6ef2db5917713082c5ba45f0fd56d18b0f1d250 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Thu, 25 Aug 2022 16:22:10 -0700 Subject: [PATCH 27/35] Capture need for "preserve unknown fields" --- proposals/ytt/004-schema-validation/README.md | 20 +++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index 202c84ee3..87a957efa 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -371,6 +371,23 @@ _(Also considered: [programmatically invoking multiple validations](#programmati > - [ ] detail how rule descriptions are included in documentation. > - [x] for each OpenAPI/JSON Schema validation keyword, show exactly how that would be expressed in `ytt` validation. > - [ ] describe the fallback strategy where authors can represent their rule where there _is_ no OpenAPI equivalent. +> - [ ] Incorporate feedback about key schema needs +> > Maybe these have already been added and I haven't been paying attention, but are there equivalents for the following in OpenAPI schema: +> > ```yaml +> > labelSelector: +> > type: object +> > properties: +> > matchLabels: +> > type: object +> > x-kubernetes-preserve-unknown-fields: true +> > additionalProperties: +> > type: string +> > ``` +> > +> > That is: +> > additionalProperties so can force values must be of certain type in an object. +> > x-kubernetes-preserve-unknown-fields so can allow any number of keys in object. +> > These are the two main things I would care about. When the user inspects the schema: @@ -380,8 +397,7 @@ $ ytt ... --data-values-schema-inspect -o openapi-v3 Validations defined on schema are included in that export. - -OpenAPI v3 / JSON Schema validation keywords: +##### OpenAPI v3 / JSON Schema validation keywords: Supported (ytt rule ==> OpenAPI v3 / JSON Schema property): - `max=` / `@ytt:assert.max()` ==> [maximum](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-validation-00#section-5.2) From 9b6ab1e42976bbe772b18c8902db4f52112edb97 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Tue, 6 Sep 2022 17:16:36 -0700 Subject: [PATCH 28/35] Remove defunct when_null_skip= kwarg from design Identified as redundant as described in vmware-tanzu/carvel-ytt#732. --- proposals/ytt/004-schema-validation/README.md | 28 ++++++++++++------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index 87a957efa..cb4d40f23 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -295,18 +295,19 @@ Whenever a node is annotated `@schema/nullable`, then its default value is `null foo: 13 ``` -The author may opt to short-circuit validations when the actual value is `null`: +By default, validations are skipped when the actual value is `null`. Authors can insist that validations are run by including the `not_null=` rule: ```yaml #@data/values-schema --- -#@schema/validation min=42, max=42, when_null_skip=True +#@schema/validation min=42, max=42, not_null=True #@schema/nullable foo: 13 ``` -(This is preferred instead of requiring _all_ validators to perform a null check to keep the overall complexity of the system to a minimum.) +When present, the `not_null=` rule runs first. If it fails, all other rules are guaranteed to fail, so they are not run. +_(also considered: [Opt-out of skipping validations when value is `null`](#opt-out-of-skipping-validations-when-value-is-null))_ _(also considered: [automatically including `when_null_skip` when Data Value is nullable](#automatically-including-when_null_skip-for-nullable-data-values))._ When the author has set the Data Value as "nullable" with the intent to require the consumer to supply their own value, this is the [Required Input (for types with no empty value)](#required-input-for-types-with-no-empty-value) use case. @@ -503,7 +504,7 @@ Sources: Defines validity for the value of the annotated node. ``` -@assert/validate [rule0, rule1, ...] [,] [,when=] [,when_null_skip=] +@assert/validate [rule0, rule1, ...] [,] [,when=] ``` where: @@ -518,7 +519,6 @@ where: - `{failure}` — when `assertion` fails, the message from that `assert.fail()` or empty string. - `` — any of the keyword arguments described in [Named Rules](#named-rules), below. - `when=` (`function(value) : None` | `function(value) : bool`) — criteria for when the validation rules should be checked. If the criteria is met (function either returns `True` or `None`), the validations are checked; otherwise (function either returns `False` or `assert.fail()`s, none of the validations are checked. -- `when_null_skip=` (`bool`) — a special-case of `when=` that checks if the value of the annotated node is `null`. Each rule is evaluated, in the order they appear in the annotation (left-to-right): - if all rules pass (either returns `True` or `None`), then the value is valid. @@ -673,7 +673,7 @@ Note: if an author wishes to provide a custom violation message, they would use Programmatically asserts that a given value is valid based on the set of supplied rules. ```python -assert.validate(key, value, rule0, rule1, ... [,] [,when=] [,when_null_skip=]) +assert.validate(key, value, rule0, rule1, ... [,] [,when=] ``` where: - `key` (`string`) — the name of the value being validated. @@ -761,7 +761,7 @@ When present, this validator is always checked first. Attaches a validation to the type being declared by the annotated node. ``` -@schema/validation [rule0, rule1, ...] [,] [,when=] [,when_null_skip=] +@schema/validation [rule0, rule1, ...] [,] [,when=] ``` When the schema base document is generated, the corresponding node is annotated with `@assert/validate` with the same configuration provide _this_ annotation. @@ -776,7 +776,7 @@ See [@assert/validate](#assertvalidate) for details about arguments to this anno Defines what a valid value is for all descendants which hold a "string" value. ``` -@schema/validation-defaults-for-strings [rule0, rule1...] [,] [,when=] [,when_null_skip=] +@schema/validation-defaults-for-strings [rule0, rule1...] [,] [,when=] ``` See also: [Consideration: Setting validation defaults for strings in a schema overlay](#consideration-setting-validation-defaults-for-strings-in-a-schema-overlay). @@ -1089,7 +1089,7 @@ persistence: #@schema/nullable multipartcopythresholdsize: 0 - #@schema/validation one_of=["1", "10"], when_null_skip=True + #@schema/validation one_of=["1", "10"] #@schema/nullable swift: #@schema/examples ("Example", "https://storage.myprovider.com/v3/auth") @@ -1112,7 +1112,7 @@ persistence: #@schema/nullable insecureskipverify: false #@schema/nullable - #@schema/validation format="quantity", when_null_skip=True + #@schema/validation format="quantity" #@schema/examples ("Five meg chunks", "5M") chunksize: "" #@schema/nullable @@ -1217,6 +1217,14 @@ no_proxy: "" ``` (src: [tomkennedy513/carvel-package-kpack:/config/schema.yaml](https://github.com/tomkennedy513/carvel-package-kpack/blob/2d7ffd7276db244c6a574019ebb80603115d1009/config/schema.yaml)) +#### Opt-out of skipping validations when value is `null` + +Initially, the design included a keyword argument — `when_null_skip=`. This flag allowed the user to opt-out of skipping validations when the value was `null`. + +The rationale for including the keyword is documented in [Automatically including `when_null_skip` for nullable data values](#automatically-including-when_null_skip-for-nullable-data-values). + +However, the implementation turned out not to rely on any schema features. Further, after [conducting usability testing](https://github.com/vmware-tanzu/carvel-ytt/issues/707), it became clear that this was an unnecessary feature. + #### Automatically including `when_null_skip` for nullable data values Had considered automatically short-circuiting validations on a Data Value that was also annotated `@schema/nullable`. Thinking: "when would someone _ever_ want the validations to fire when the value is null?!?!" From 9960ea1d93a8ba8f7322095fafefcdf3274c8732 Mon Sep 17 00:00:00 2001 From: John Ryan Date: Tue, 13 Sep 2022 16:35:53 -0700 Subject: [PATCH 29/35] Update specs around "when=" keyword. --- proposals/ytt/004-schema-validation/README.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/proposals/ytt/004-schema-validation/README.md b/proposals/ytt/004-schema-validation/README.md index cb4d40f23..e6e6cf061 100644 --- a/proposals/ytt/004-schema-validation/README.md +++ b/proposals/ytt/004-schema-validation/README.md @@ -332,6 +332,17 @@ If `false`, then none of the other validations are checked and the value is assu Here, the contents of the `service:` are validated (via `valid_service_config()`) only if `service.enabled` is `true`. +Similarly, the validation can be dependent on _other_ data values. + +```yaml +#@data/values-schema +--- +service: + enabled: true + #@schema/validation one_of=["NodePort", "LoadBalancer"], when=lambda _, ctx: ctx.parent["enabled"] + type: NodePort +``` + #### Use Case: Disable Validations A consumer can skip validating Data Values: @@ -518,7 +529,7 @@ where: - `{desc}` — the description supplied in the 0th item in this rule. - `{failure}` — when `assertion` fails, the message from that `assert.fail()` or empty string. - `` — any of the keyword arguments described in [Named Rules](#named-rules), below. -- `when=` (`function(value) : None` | `function(value) : bool`) — criteria for when the validation rules should be checked. If the criteria is met (function either returns `True` or `None`), the validations are checked; otherwise (function either returns `False` or `assert.fail()`s, none of the validations are checked. +- `when=` (`function(value[, context]) : bool`) — criteria for when the validation rules should be checked. If the criteria is met (function returns `True`), the validations are checked; otherwise (function either returns `False` or `assert.fail()`s, none of the validations are checked. Each rule is evaluated, in the order they appear in the annotation (left-to-right): - if all rules pass (either returns `True` or `None`), then the value is valid. From 5e7961b9e1bdef806f22797736514cfed993657c Mon Sep 17 00:00:00 2001 From: Varsha Munishwar Date: Wed, 28 Sep 2022 08:59:16 -0700 Subject: [PATCH 30/35] update the blog date --- site/content/blog/ytt-validations-released.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/blog/ytt-validations-released.md b/site/content/blog/ytt-validations-released.md index 6eab8e2ca..c09d977b4 100644 --- a/site/content/blog/ytt-validations-released.md +++ b/site/content/blog/ytt-validations-released.md @@ -1,7 +1,7 @@ --- title: "The Hidden Costs of Misconfiguration" slug: ytt-validations-released -date: 2022-09-28 +date: 2022-09-27 author: John Ryan, Varsha Munishwar excerpt: "When your end-users give the wrong input, the best thing you can do is give them immediate, concise, and actionable feedback. Now you can, with ytt Validations!" image: /img/ytt.svg From ad609a19934121a64e928fea50d634be02aa30f7 Mon Sep 17 00:00:00 2001 From: Praveen Rewar <8457124+praveenrewar@users.noreply.github.com> Date: Thu, 29 Sep 2022 14:43:36 +0530 Subject: [PATCH 31/35] Bump kapp docs (#567) --- site/config.yaml | 5 +- site/content/kapp/docs/v0.52.0/_index.md | 2 +- .../kapp/docs/v0.52.0/apply-ordering.md | 2 +- .../kapp/docs/v0.52.0/apply-waiting.md | 2 +- site/content/kapp/docs/v0.52.0/apply.md | 2 +- site/content/kapp/docs/v0.52.0/apps.md | 2 +- site/content/kapp/docs/v0.52.0/cheatsheet.md | 2 +- site/content/kapp/docs/v0.52.0/config.md | 2 +- .../kapp/docs/v0.52.0/configmap-migration.md | 2 +- .../kapp/docs/v0.52.0/dangerous-flags.md | 2 +- site/content/kapp/docs/v0.52.0/diff.md | 2 +- site/content/kapp/docs/v0.52.0/faq.md | 2 +- site/content/kapp/docs/v0.52.0/gitops.md | 2 +- .../docs/v0.52.0/hpa-deployment-rebase.md | 2 +- site/content/kapp/docs/v0.52.0/install.md | 2 +- .../v0.52.0/integrating-with-other-tools.md | 2 +- .../content/kapp/docs/v0.52.0/merge-method.md | 2 +- site/content/kapp/docs/v0.52.0/rbac.md | 2 +- site/content/kapp/docs/v0.52.0/rebase-pvc.md | 2 +- site/content/kapp/docs/v0.52.0/security.md | 2 +- .../kapp/docs/v0.52.0/state-namespace.md | 2 +- site/content/kapp/docs/v0.53.0/_index.md | 44 ++ .../kapp/docs/v0.53.0/apply-ordering.md | 94 +++++ .../kapp/docs/v0.53.0/apply-waiting.md | 47 +++ site/content/kapp/docs/v0.53.0/apply.md | 123 ++++++ site/content/kapp/docs/v0.53.0/apps.md | 41 ++ site/content/kapp/docs/v0.53.0/cheatsheet.md | 135 +++++++ site/content/kapp/docs/v0.53.0/config.md | 377 +++++++++++++++++ .../kapp/docs/v0.53.0/configmap-migration.md | 125 ++++++ .../kapp/docs/v0.53.0/dangerous-flags.md | 34 ++ site/content/kapp/docs/v0.53.0/diff.md | 378 ++++++++++++++++++ site/content/kapp/docs/v0.53.0/faq.md | 128 ++++++ site/content/kapp/docs/v0.53.0/gitops.md | 17 + .../docs/v0.53.0/hpa-deployment-rebase.md | 42 ++ site/content/kapp/docs/v0.53.0/install.md | 58 +++ .../v0.53.0/integrating-with-other-tools.md | 27 ++ .../content/kapp/docs/v0.53.0/merge-method.md | 27 ++ site/content/kapp/docs/v0.53.0/rbac.md | 70 ++++ site/content/kapp/docs/v0.53.0/rebase-pvc.md | 102 +++++ site/content/kapp/docs/v0.53.0/security.md | 8 + .../kapp/docs/v0.53.0/state-namespace.md | 53 +++ site/data/kapp/docs/kapp-v0-53-0-toc.yml | 53 +++ site/data/kapp/docs/toc-mapping.yml | 1 + 43 files changed, 2007 insertions(+), 22 deletions(-) create mode 100644 site/content/kapp/docs/v0.53.0/_index.md create mode 100644 site/content/kapp/docs/v0.53.0/apply-ordering.md create mode 100644 site/content/kapp/docs/v0.53.0/apply-waiting.md create mode 100644 site/content/kapp/docs/v0.53.0/apply.md create mode 100644 site/content/kapp/docs/v0.53.0/apps.md create mode 100644 site/content/kapp/docs/v0.53.0/cheatsheet.md create mode 100644 site/content/kapp/docs/v0.53.0/config.md create mode 100644 site/content/kapp/docs/v0.53.0/configmap-migration.md create mode 100644 site/content/kapp/docs/v0.53.0/dangerous-flags.md create mode 100644 site/content/kapp/docs/v0.53.0/diff.md create mode 100644 site/content/kapp/docs/v0.53.0/faq.md create mode 100644 site/content/kapp/docs/v0.53.0/gitops.md create mode 100644 site/content/kapp/docs/v0.53.0/hpa-deployment-rebase.md create mode 100644 site/content/kapp/docs/v0.53.0/install.md create mode 100644 site/content/kapp/docs/v0.53.0/integrating-with-other-tools.md create mode 100644 site/content/kapp/docs/v0.53.0/merge-method.md create mode 100644 site/content/kapp/docs/v0.53.0/rbac.md create mode 100644 site/content/kapp/docs/v0.53.0/rebase-pvc.md create mode 100644 site/content/kapp/docs/v0.53.0/security.md create mode 100644 site/content/kapp/docs/v0.53.0/state-namespace.md create mode 100644 site/data/kapp/docs/kapp-v0-53-0-toc.yml diff --git a/site/config.yaml b/site/config.yaml index ab4864e46..329632655 100644 --- a/site/config.yaml +++ b/site/config.yaml @@ -59,15 +59,16 @@ params: name: kapp short_name: kapp root_link: /kapp/ - latest_docs_link: /kapp/docs/v0.52.0/ + latest_docs_link: /kapp/docs/v0.53.0/ github_url: https://github.com/vmware-tanzu/carvel-kapp search: true search_index_name: carvel-kapp search_api_key: a38560864c2e9128ae57d5734df438ff versioning: true - version_latest: v0.52.0 + version_latest: v0.53.0 versions: - develop + - v0.53.0 - v0.52.0 - v0.51.0 - v0.50.0 diff --git a/site/content/kapp/docs/v0.52.0/_index.md b/site/content/kapp/docs/v0.52.0/_index.md index 96b22697f..17f1a49ba 100644 --- a/site/content/kapp/docs/v0.52.0/_index.md +++ b/site/content/kapp/docs/v0.52.0/_index.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/] + title: "About kapp" toc: "false" cascade: diff --git a/site/content/kapp/docs/v0.52.0/apply-ordering.md b/site/content/kapp/docs/v0.52.0/apply-ordering.md index 6a60e1044..6f870b2ec 100644 --- a/site/content/kapp/docs/v0.52.0/apply-ordering.md +++ b/site/content/kapp/docs/v0.52.0/apply-ordering.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/apply-ordering] + title: Apply Ordering --- diff --git a/site/content/kapp/docs/v0.52.0/apply-waiting.md b/site/content/kapp/docs/v0.52.0/apply-waiting.md index b33e6638e..5b70036f0 100644 --- a/site/content/kapp/docs/v0.52.0/apply-waiting.md +++ b/site/content/kapp/docs/v0.52.0/apply-waiting.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/apply-waiting] + title: Apply Waiting --- diff --git a/site/content/kapp/docs/v0.52.0/apply.md b/site/content/kapp/docs/v0.52.0/apply.md index 29a58b478..3a4e9299e 100644 --- a/site/content/kapp/docs/v0.52.0/apply.md +++ b/site/content/kapp/docs/v0.52.0/apply.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/apply] + title: Apply stage --- diff --git a/site/content/kapp/docs/v0.52.0/apps.md b/site/content/kapp/docs/v0.52.0/apps.md index ebaf13fab..2abf48417 100644 --- a/site/content/kapp/docs/v0.52.0/apps.md +++ b/site/content/kapp/docs/v0.52.0/apps.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/apps] + title: Applications --- diff --git a/site/content/kapp/docs/v0.52.0/cheatsheet.md b/site/content/kapp/docs/v0.52.0/cheatsheet.md index 61e9bef11..31cca3575 100644 --- a/site/content/kapp/docs/v0.52.0/cheatsheet.md +++ b/site/content/kapp/docs/v0.52.0/cheatsheet.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/cheatsheet] + title: Cheatsheet --- diff --git a/site/content/kapp/docs/v0.52.0/config.md b/site/content/kapp/docs/v0.52.0/config.md index ca35e2c11..273b0113c 100644 --- a/site/content/kapp/docs/v0.52.0/config.md +++ b/site/content/kapp/docs/v0.52.0/config.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/config] + title: Configuration --- diff --git a/site/content/kapp/docs/v0.52.0/configmap-migration.md b/site/content/kapp/docs/v0.52.0/configmap-migration.md index 9d1d3a4f7..d31a49ff5 100644 --- a/site/content/kapp/docs/v0.52.0/configmap-migration.md +++ b/site/content/kapp/docs/v0.52.0/configmap-migration.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/configmap-migration] + title: Configmap Migration (experimental) --- diff --git a/site/content/kapp/docs/v0.52.0/dangerous-flags.md b/site/content/kapp/docs/v0.52.0/dangerous-flags.md index 25969c7d6..25478b6a3 100644 --- a/site/content/kapp/docs/v0.52.0/dangerous-flags.md +++ b/site/content/kapp/docs/v0.52.0/dangerous-flags.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/dangerous-flags] + title: Dangerous Flags --- diff --git a/site/content/kapp/docs/v0.52.0/diff.md b/site/content/kapp/docs/v0.52.0/diff.md index f4a8237e1..1db56762e 100644 --- a/site/content/kapp/docs/v0.52.0/diff.md +++ b/site/content/kapp/docs/v0.52.0/diff.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/diff] + title: Diff stage --- ## Overview diff --git a/site/content/kapp/docs/v0.52.0/faq.md b/site/content/kapp/docs/v0.52.0/faq.md index a75b85f77..c3a759bab 100644 --- a/site/content/kapp/docs/v0.52.0/faq.md +++ b/site/content/kapp/docs/v0.52.0/faq.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/faq] + title: FAQ --- diff --git a/site/content/kapp/docs/v0.52.0/gitops.md b/site/content/kapp/docs/v0.52.0/gitops.md index 156a06c19..c7ce67312 100644 --- a/site/content/kapp/docs/v0.52.0/gitops.md +++ b/site/content/kapp/docs/v0.52.0/gitops.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/gitops] + title: GitOps --- diff --git a/site/content/kapp/docs/v0.52.0/hpa-deployment-rebase.md b/site/content/kapp/docs/v0.52.0/hpa-deployment-rebase.md index 722c9bc68..ce4ce0d2e 100644 --- a/site/content/kapp/docs/v0.52.0/hpa-deployment-rebase.md +++ b/site/content/kapp/docs/v0.52.0/hpa-deployment-rebase.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/hpa-deployment-rebase] + title: HPA and Deployment rebase --- ## HPA and Deployment rebase diff --git a/site/content/kapp/docs/v0.52.0/install.md b/site/content/kapp/docs/v0.52.0/install.md index e2bda94da..da00c259f 100644 --- a/site/content/kapp/docs/v0.52.0/install.md +++ b/site/content/kapp/docs/v0.52.0/install.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/install] + title: Install --- diff --git a/site/content/kapp/docs/v0.52.0/integrating-with-other-tools.md b/site/content/kapp/docs/v0.52.0/integrating-with-other-tools.md index aa84510bf..0bdc967b8 100644 --- a/site/content/kapp/docs/v0.52.0/integrating-with-other-tools.md +++ b/site/content/kapp/docs/v0.52.0/integrating-with-other-tools.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/integrating-with-other-tools] + title: Integrating with Other Tools --- diff --git a/site/content/kapp/docs/v0.52.0/merge-method.md b/site/content/kapp/docs/v0.52.0/merge-method.md index b19c51b09..bc2b84e48 100644 --- a/site/content/kapp/docs/v0.52.0/merge-method.md +++ b/site/content/kapp/docs/v0.52.0/merge-method.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/merge-method] + title: "Resource Merge Method" --- diff --git a/site/content/kapp/docs/v0.52.0/rbac.md b/site/content/kapp/docs/v0.52.0/rbac.md index 7eb89bd96..33b568951 100644 --- a/site/content/kapp/docs/v0.52.0/rbac.md +++ b/site/content/kapp/docs/v0.52.0/rbac.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/rbac] + title: Permissions --- diff --git a/site/content/kapp/docs/v0.52.0/rebase-pvc.md b/site/content/kapp/docs/v0.52.0/rebase-pvc.md index 13e9e6383..092e24c13 100644 --- a/site/content/kapp/docs/v0.52.0/rebase-pvc.md +++ b/site/content/kapp/docs/v0.52.0/rebase-pvc.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/rebase-pvc] + title: PersistentVolumeClaim rebase --- ## PersistentVolumeClaim rebase diff --git a/site/content/kapp/docs/v0.52.0/security.md b/site/content/kapp/docs/v0.52.0/security.md index 6b16ba957..30752f23a 100644 --- a/site/content/kapp/docs/v0.52.0/security.md +++ b/site/content/kapp/docs/v0.52.0/security.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/security] + title: Security --- diff --git a/site/content/kapp/docs/v0.52.0/state-namespace.md b/site/content/kapp/docs/v0.52.0/state-namespace.md index 1c278ca29..27c19c585 100644 --- a/site/content/kapp/docs/v0.52.0/state-namespace.md +++ b/site/content/kapp/docs/v0.52.0/state-namespace.md @@ -1,5 +1,5 @@ --- -aliases: [/kapp/docs/latest/state-namespace] + title: Namespace for State Storage --- diff --git a/site/content/kapp/docs/v0.53.0/_index.md b/site/content/kapp/docs/v0.53.0/_index.md new file mode 100644 index 000000000..e0e988e81 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/_index.md @@ -0,0 +1,44 @@ +--- +aliases: [/kapp/docs/latest/] +title: "About kapp" +toc: "false" +cascade: + version: v0.53.0 + toc: "true" + type: docs + layout: docs +--- + +`kapp` (pronounced: `kap`) CLI encourages Kubernetes users to manage resources in bulk by working with "Kubernetes applications" (sets of resources with the same label). It focuses on resource diffing, labeling, deployment and deletion. Unlike tools like Helm, `kapp` considers YAML templating and management of packages outside of its scope, though it works great with tools that generate Kubernetes configuration. + +![Kapp Deploy](/images/kapp/kapp-deploy-screenshot.png) + +Features: + +- Works with standard Kubernetes YAMLs +- Focuses exclusively on deployment workflow, not packaging or templating + - but plays well with tools (such as [ytt](/ytt)) that produce Kubernetes configuration +- Converges application resources (creates, updates and/or deletes resources) in each deploy + - based on comparison between provided files and live objects in the cluster +- Separates calculation of changes ([diff stage](diff.md)) from application of changes ([apply stage](apply.md)) +- [Waits for resources](apply-waiting.md) to be "ready" +- Creates CRDs and Namespaces first and supports [custom change ordering](apply-ordering.md) +- Works [without admin privileges](rbac.md) and does not use custom CRDs + - making it possible to use kapp as a regular user in a single namespace +- Records application deployment history +- Opt-in resource version management + - for example, to trigger Deployment rollout when ConfigMap changes +- Optionally streams Pod logs during deploy +- Works with any group of labeled resources (`kapp -a label:tier=web inspect -t`) +- Works without server side components +- GitOps friendly (`kapp app-group deploy -g all-apps --directory .`) + +## Blog posts + +- [Deploying Kubernetes Applications with ytt, kbld, and kapp](/blog/deploying-apps-with-ytt-kbld-kapp) + +## Talks + +- [ytt and kapp @ TGI Kubernetes 079](https://www.youtube.com/watch?v=CSglwNTQiYg) with Joe Beda +- [Managing Applications in Production: Helm vs ytt & kapp @ Kubecon 2020](https://www.youtube.com/watch?v=WJw1MDFMVuk) +- [Introduction to Carvel @ Rawkode Live](https://www.youtube.com/watch?v=LBCmMTofNxw) diff --git a/site/content/kapp/docs/v0.53.0/apply-ordering.md b/site/content/kapp/docs/v0.53.0/apply-ordering.md new file mode 100644 index 000000000..6a60e1044 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/apply-ordering.md @@ -0,0 +1,94 @@ +--- +aliases: [/kapp/docs/latest/apply-ordering] +title: Apply Ordering +--- + +## Overview + +kapp includes builtin rules to make sure certain changes are applied in particular order: + +- Creates/updates + - CRDs are created/updated before custom resources + - Namespaces are created/updated before namespaced resources + - Pod related resources (ServiceAccount, ConfigMap, Secret, etc.) are created/updated before other resources (v0.25.0+) + - RBAC related resources (Role, RoleBinding, etc.) are created/updated before other resources (v0.25.0+) +- Deletions (below is order as of v0.29.0+) + - Custom resources are deleted first + - CRDs are deleted next + - Rest of resoures are deleted + +As of v0.25.0+, builtin rules are specified via [changeGroupBindings and changeRuleBindings](config.md#changegroupbindings) configurations. Custom rules can be added via same mechanism. + +Additionally kapp allows to customize order of changes via following resource annotations: + +- `kapp.k14s.io/change-group` annotation to group one or more resource changes into arbitrarily named group. Example: `apps.big.co/db-migrations`. You can specify multiple change groups by suffixing each annotation with a `.x` where `x` is unique identifier (e.g. `kapp.k14s.io/change-group.istio-sidecar-order`). +- `kapp.k14s.io/change-rule` annotation to control when resource change should be applied (created, updated, or deleted) relative to other changes. You can specify multiple change rules by suffixing each annotation with a `.x` where `x` is unique identifier (e.g. `kapp.k14s.io/change-rule.istio-sidecar-order`). + +`kapp.k14s.io/change-rule` annotation value format is as follows: `(upsert|delete) (after|before) (upserting|deleting) `. For example: + +- `kapp.k14s.io/change-rule: "upsert after upserting apps.big.co/db-migrations"` +- `kapp.k14s.io/change-rule: "delete before upserting apps.big.co/service"` + +As of v0.41.0+, kapp provides change group placeholders, which can be used in change-group and change-rule annotation values and are later replaced by values from the resource manifest of the resource they are associated with. For example: + +- `kapp.k14s.io/change-group: apps.co/db-migrations-{name}` - Here `{name}` would later be replaced by the name of the resource. +- `kapp.k14s.io/change-rule: upsert after upserting apps.co/namespaces-{namespace}` - Here `{namespace}` would later be replaced by the namespace of the resource. + +kapp provides the following placeholders: + +- `{api-group}` - apiGroup +- `{kind}` - kind +- `{name}` - name +- `{namespace}` - namespace +- `{crd-kind}` - spec.names.kind (available for CRDs only) +- `{crd-group}` - spec.group (available for CRDs only) + +These placeholders can also be used in changeGroupBindings and changeRuleBindings. By default, they are used for CRDs, CRs, namespaces and namespaced resources. Due to this, CRs now wait for their respective CRDs only and namespaced resources now wait for their respective namespaces only. + +## Example + +Following example shows how to run `job/migrations`, start and wait for `deployment/app`, and finally `job/app-health-check`. + +```yaml +kind: ConfigMap +metadata: + name: app-config + annotations: {} +#... +--- +kind: Job +metadata: + name: migrations + annotations: + kapp.k14s.io/change-group: "apps.big.co/db-migrations" +#... +--- +kind: Service +metadata: + name: app + annotations: + kapp.k14s.io/change-group: "apps.big.co/deployment" +#... +--- +kind: Ingress +metadata: + name: app + annotations: + kapp.k14s.io/change-group: "apps.big.co/deployment" +#... +--- +kind: Deployment +metadata: + name: app + annotations: + kapp.k14s.io/change-group: "apps.big.co/deployment" + kapp.k14s.io/change-rule: "upsert after upserting apps.big.co/db-migrations" +#... +--- +kind: Job +metadata: + name: app-health-check + annotations: + kapp.k14s.io/change-rule: "upsert after upserting apps.big.co/deployment" +#... +``` diff --git a/site/content/kapp/docs/v0.53.0/apply-waiting.md b/site/content/kapp/docs/v0.53.0/apply-waiting.md new file mode 100644 index 000000000..b33e6638e --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/apply-waiting.md @@ -0,0 +1,47 @@ +--- +aliases: [/kapp/docs/latest/apply-waiting] +title: Apply Waiting +--- + +## Overview + +kapp includes builtin rules on how to wait for the following resource types: + +- any resource with `metadata.deletionTimestamp`: wait for resource to be fully removed +- any resource matching Config's waitRules: [see "Custom waiting behaviour" below](#custom-waiting-behaviour) +- [`apiextensions.k8s.io//CustomResourceDefinition`](https://github.com/vmware-tanzu/carvel-kapp/blob/develop/pkg/kapp/resourcesmisc/api_extensions_vx_crd.go): wait for Established and NamesAccepted conditions to be `True` (note that this is wait rule for CustomResourceDefinition resource itself, not CRs) +- `apps/v1/DaemonSet`: wait for `status.numberUnavailable` to be 0 +- `apps/v1/Deployment`: [see "apps/v1/Deployment resource" below](#apps-v1-deployment-resource) +- `apps/v1/ReplicaSet`: wait for `status.replicas == status.availableReplicas` +- `batch/v1/Job`: wait for `Complete` or `Failed` conditions to appear +- `batch//CronJob`: immediately considered done +- `/v1/Pod`: looks at `status.phase` +- `/v1/Service`: wait for `spec.clusterIP` and/or `status.loadBalancer.ingress` to become set +- `apps/v1/StatefulSet`: [see "apps/v1/StatefulSet resource" below](#appsv1statefulset-resource) + +If resource is not affected by the above rules, its waiting behaviour depends on aggregate of waiting states of its associated resources (associated resources are resources that share same `kapp.k14s.io/association` label value). + +## Controlling waiting via resource annotations + +- `kapp.k14s.io/disable-wait` annotation controls whether waiting will happen at all. Possible values: "". +- `kapp.k14s.io/disable-associated-resources-wait` annotation controls whether associated resources impact resource's waiting state. Possible values: "". + +## apps/v1/Deployment resource + +kapp by default waits for `apps/v1/Deployment` resource to have `status.unavailableReplicas` equal to zero. Additionally waiting behaviour can be controlled via following annotations: + +- `kapp.k14s.io/apps-v1-deployment-wait-minimum-replicas-available` annotation controls how many new available replicas are enough to consider waiting successful. Example values: `"10"`, `"5%"`. + +## apps/v1/StatefulSet resource + +Available in v0.32.0+. + +kapp will wait for any pods created from the updated template to be ready based on StatefulSet's status. This behaviour depends on the [update strategy](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies) used. + +Note: kapp does not do anything special when `OnDelete` strategy is used. It will wait for StatefulSet to report it's reconciled (expecting some actor in the system to delete Pods per `OnDelete` requirements). + +## Custom waiting behaviour + +Available in v0.29.0+. + +kapp can be extended with custom waiting behaviour by specifying [wait rules as additional config](config.md#wait-rules). (If this functionality is not enough to wait for resources in your use case, please reach out on Slack to discuss further.) diff --git a/site/content/kapp/docs/v0.53.0/apply.md b/site/content/kapp/docs/v0.53.0/apply.md new file mode 100644 index 000000000..29a58b478 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/apply.md @@ -0,0 +1,123 @@ +--- +aliases: [/kapp/docs/latest/apply] +title: Apply stage +--- + +## Overview + +Once change set is calculated (see [Diff](diff.md) section for details), kapp asks for user confirmation (unless `--yes` flag is specified) to proceed with changes. + +Changes are applied in particular order as described in [Apply ordering](apply-ordering.md). + +All created resources are labeled with several labels: + +- `kapp.k14s.io/app` to track which application "owns" resource +- `kapp.k14s.io/identity` to identify preferred API version used when creating resource +- `kapp.k14s.io/association` to track (best effort) parent-child relationships between resources + +Every time application is deployed, new application change record is saved. They can be viewed via `kapp app-change ls -a app-name`. + +Related: [ownership label rules](config.md#ownershiplabelrules) and [label scoping rules](config.md#labelscopingrules). + +## Controlling apply via resource annotations + +### kapp.k14s.io/create-strategy + +`kapp.k14s.io/create-strategy` annotation controls create behaviour (rarely necessary) + +Possible values: "" (default), `fallback-on-update`, `fallback-on-update-or-noop`. + +In some cases creation of a resource may conflict with that resource being created in the cluster by other means (often automated). An example of that is creation of default ServiceAccount by kapp racing with Kubernetes service accounts controller doing the same thing. By specifying `fallback-on-update` value, kapp will catch resource creation conflicts and apply resource as an update. + +`fallback-on-update-or-noop` (Available in v0.47.0+) also allows to use `noop` operation if `kapp.k14.io/noop` is added through rebase rules, else it behaves the same way as `fallback-on-update`. + +### kapp.k14s.io/update-strategy + +`kapp.k14s.io/update-strategy` annotation controls update behaviour + +Possible values: "" (default), `fallback-on-replace`, `always-replace`, `skip`. + +In some cases entire resources or subset resource fields are immutable which forces kapp users to specify how to apply wanted update. + +- "" means to issue plain update call +- `fallback-on-replace` causes kapp to fallback to resource replacement if update call results in `Invalid` error. Note that if resource is replaced (= delete + create), it may be negatively affected (loss of persistent data, loss of availability, etc.). For example, if Deployment or DaemonSet is first deleted and then created then associated Pods will be recreated as well, but all at the same time (even if rolling update is enabled), which likely causes an availability gap. +- `always-replace` causes kapp to always delete and then create resource (See note above as well.) +- `skip` causes kapp to not apply update (it will show up in a diff next time). Available in v0.33.0+. + +### kapp.k14s.io/delete-strategy + +`kapp.k14s.io/delete-strategy` annotation controls deletion behaviour + +Possible values: "" (default), `orphan`. + +By default resource is deleted, however; choosing `orphan` value will make kapp forget about this resource. Note that if this resource is owned by a different resource that's being deleted, it might still get deleted. Orphaned resources are labeled with `kapp.k14s.io/orphaned` label. As of v0.31.0+, resource is also disassociated from owning app so that it can be owned by future apps. + +### kapp.k14s.io/owned-for-deletion + +`kapp.k14s.io/owned-for-deletion` annotation controls resource deletion during `kapp delete` command + +Possible values: "". + +By default non-kapp owned resources are not explicitly deleted by kapp, but expected to be deleted by the cluster (for example Endpoints resource for each Service). In some cases it's desired to annotate non-kapp owned resource so that it does get explicitly deleted, possibly because cluster does not plan to delete it (e.g. PVCs created by StatefulSet are not deleted by StatefulSet controller; [https://github.com/vmware-tanzu/carvel-kapp/issues/36](https://github.com/vmware-tanzu/carvel-kapp/issues/36)). + +### kapp.k14s.io/nonce + +`kapp.k14s.io/nonce` annotation allows to inject unique ID + +Possible values: "" (default). + +Annotation value will be replaced with a unique ID on each deploy. This allows to force resource update as value changes every time. + +### kapp.k14s.io/deploy-logs + +`kapp.k14s.io/deploy-logs` annotation indicates which Pods' log output to show during deploy + +Possible values: + +- "" (default; equivalent to `for-new`) +- `for-new` (only newly created Pods are tailed) +- `for-existing` (only existing Pods are tailed) +- `for-new-or-existing` (both newly created and existing Pods are tailed) + +Especially useful when added to Jobs. For example, see [examples/resource-ordering/sync-check.yml](https://github.com/vmware-tanzu/carvel-kapp/blob/develop/examples/resource-ordering/sync-check.yml) + +### kapp.k14s.io/deploy-logs-container-names + +`kapp.k14s.io/deploy-logs-container-names` annotation indicates which containers' log output to show during deploy + +Possible values: "" (default), `containerName1`, `containerName1,containerName2` + +### kapp.k14s.io/exists + +Available in v0.43.0+ + +`kapp.k14s.io/exists` will ensure that resource exists in Kubernetes. It will not be considered to be part of the app (not labeled). + +If the resource is not present already, then kapp uses the `exists` operation and ensures that the resource exists in Kubernetes. + +If the resource already exists, kapp does not perform any operation on it (the `noop` operation is used). + +Possible values: "". + +Especially useful in scenarios where an external agency such as a controller might be creating a resource that we want to wait for. + +### kapp.k14s.io/noop + +Available in v0.43.0+ + +`kapp.k14s.io/noop` ensures that kapp is aware of the resource. It will not be considered to be part of the app (not labeled). + +kapp always uses the `noop` operation for these resources. + +Possible values: "". + +--- +## Controlling apply via deploy flags + +- `--apply-ignored=bool` explicitly applies ignored changes; this is useful in cases when controllers lose track of some resources instead of for example deleting them +- `--apply-default-update-strategy=string` controls default strategy for all resources (see `kapp.k14s.io/update-strategy` annotation above) +- `--apply-exit-status=bool` (default `false`) controls exit status (`0`: unused, `1`: any error, `2`: no changes applied, `3`: at least one change applied) +- `--wait=bool` (default `true`) controls whether kapp will wait for resource to "stabilize". See [Apply waiting](apply-waiting.md) +- `--wait-ignored=bool` controls whether kapp will wait for ignored changes (regardless whether they were initiated by kapp or by controllers) +- `--logs=bool` (default `true`) controls whether to show logs as part of deploy output for Pods annotated with `kapp.k14s.io/deploy-logs: ""` +- `--logs-all=bool` (deafult `false`) controls whether to show all logs as part of deploy output for all Pods diff --git a/site/content/kapp/docs/v0.53.0/apps.md b/site/content/kapp/docs/v0.53.0/apps.md new file mode 100644 index 000000000..ebaf13fab --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/apps.md @@ -0,0 +1,41 @@ +--- +aliases: [/kapp/docs/latest/apps] +title: Applications +--- + +## Overview + +kapp considers a set of resources with the same label as an application. These resources could span any number of namespaces or could be cluster-wide (e.g. CRDs). + +kapp has two methods of finding resources: + +1. via unique-to-Namespace application name (via `-a my-name` flag), or +2. via user provided label (via `-a label:my-label=val` flag) + +First approach is most common as kapp generates a unique label for each tracked application and associates that with an application name. + +## List + +Applications can be listed via `ls` command: + +```bash +$ kapp ls +``` + +## Deploy + +To create or update an application use `deploy` command: + +```bash +$ kapp deploy -a my-name -f my-app-config/ +``` + +Deploy command consists of two stages: [resource "diff" stage](diff.md), and [resource "apply" stage](apply.md). + +## Delete + +To delete an application use `delete` command: + +```bash +$ kapp delete -a my-name +``` diff --git a/site/content/kapp/docs/v0.53.0/cheatsheet.md b/site/content/kapp/docs/v0.53.0/cheatsheet.md new file mode 100644 index 000000000..61e9bef11 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/cheatsheet.md @@ -0,0 +1,135 @@ +--- +aliases: [/kapp/docs/latest/cheatsheet] +title: Cheatsheet +--- + + +## List + +List all app in the cluster (across all namespaces) + +```bash +kapp ls -A +``` + +Show only specific columns while listing apps + +```bash +kapp ls --column=namespace,name,label +``` + +## Deploy + +Deploy app named `app1` with configuration from `config/`: + +```bash +kapp deploy -a app1 -f config/ -c +``` + +Deploy app named `app1` with configuration piped in (see alternative that does not require `--yes` next): + +```bash +ytt -f config/ | kapp deploy -a app1 -f- -c -y +``` + +Deploy app named `app1` with configuration generated inline and with confirmation dialog: + +```bash +kapp deploy -a app1 -f <(ytt -f config/ ) +``` + +Show more diff context when reviewing changes during deploy: + +```bash +kapp deploy -a app1 -f config/ -c --diff-context=10 +``` + +Show diff and exit successfully (without applying any changes): + +```bash +kapp deploy -a app1 -f config/ --diff-run +``` + +Show logs from all app `Pods` throughout deploy: + +```bash +kapp deploy -a app1 -f config/ --logs-all +``` + +Rewrite all resources to specify `app1-ns` namespace: + +```bash +kapp deploy -a app1 -f config/ --into-ns app1-ns +``` + +## Inspect + +Show summary of all resources in app `app1`: + +```bash +kapp inspect -a app1 +``` + +Show summary organized as a tree of all resources in app `app1`: + +```bash +kapp inspect -a app1 --tree +``` + +Show status subresources for each resource in app `app1`: + +```bash +kapp inspect -a app1 --status +``` + +Show all resources in the cluster: + +```bash +kapp inspect -a 'label:' +``` + +Show all resources in particular namespace (note that it currently does namespace filtering client-side): + +```bash +kapp inspect -a 'label:' --filter-ns some-ns +``` + +Show all resources labeled `tier=web` in the cluster: + +```bash +kapp inspect -a 'label:tier=web' +``` + +Show all `Deployment` resources in the cluster **not** managed by kapp: + +```bash +kapp inspect -a 'label:!kapp.k14s.io/app' --filter-kind Deployment +``` + +## Delete + +Delete resources under particular label (in this example deleting resources associated with some app): + +```bash +kapp delete -a 'label:kapp.k14s.io/app=1578599579922603000' +``` + +## Misc + +See which labels are used in your cluster (add `--values` to see label values): + +```bash +kapp tools list-labels +``` + +Shows app labels that are still present in the cluster (could be combined with delete command below): + +```bash +kapp tools list-labels --values --tty=false | grep kapp.k14s.io/app +``` + +Delete all app changes older than 500h (v0.12.0+): + +```bash +kapp deploy -a label:kapp.k14s.io/is-app-change --filter-age 500h+ --dangerous-allow-empty-list-of-resources --apply-ignored +``` diff --git a/site/content/kapp/docs/v0.53.0/config.md b/site/content/kapp/docs/v0.53.0/config.md new file mode 100644 index 000000000..ca35e2c11 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/config.md @@ -0,0 +1,377 @@ +--- +aliases: [/kapp/docs/latest/config] +title: Configuration +--- + +## Overview + +kapp supports custom `Config` resource to specify its own configuration. It's expected to be included with your other Kubernetes configuration. Config resource is never applied to the cluster, though it follows general Kubernetes resource format. Multiple config resources are allowed. + +kapp comes with __built-in configuration__ (see it via `kapp deploy-config`) that includes rules for common resources. + +## Format + +```yaml +apiVersion: kapp.k14s.io/v1alpha1 +kind: Config + +minimumRequiredVersion: 0.23.0 + +rebaseRules: +- path: [spec, clusterIP] + type: copy + sources: [new, existing] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: v1, kind: Service} + +ownershipLabelRules: +- path: [metadata, labels] + resourceMatchers: + - allMatcher: {} + +labelScopingRules: +- path: [spec, selector] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: v1, kind: Service} + +templateRules: +- resourceMatchers: + - apiVersionKindMatcher: {apiVersion: v1, kind: ConfigMap} + affectedResources: + objectReferences: + - path: [spec, template, spec, containers, {allIndexes: true}, env, {allIndexes: true}, valueFrom, configMapKeyRef] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} + - path: [spec, template, spec, containers, {allIndexes: true}, envFrom, {allIndexes: true}, configMapRef] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} + +additionalLabels: + department: marketing + cost-center: mar201 + +diffAgainstLastAppliedFieldExclusionRules: +- path: [metadata, annotations, "deployment.kubernetes.io/revision"] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} + +diffMaskRules: +- path: [data] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: v1, kind: Secret} +``` + +### minimumRequiredVersion + +`minimumRequiredVersion` forces kapp to exit with a validation error if kapp's version is below minimum required version. Available in v0.23.0+. + +### rebaseRules + +`rebaseRules` specify origin of field values. + +kapp rebase rules explicitly define how to merge resources during an update. To read more about why rebase rules are necessary, see [Resource Merge Method](merge-method.md). +For examples of rebase rules in use, see [HPA and Deployment rebase](hpa-deployment-rebase.md) or [PersistentVolumeClaim rebase](rebase-pvc.md). + +- `rebaseRules` (array) list of rebase rules + - `path` (array of strings) specifies location within a resource to rebase. Mutually exclusive with `paths`. Example: `[spec, clusterIP]` + - `paths` (array of `path`) specifies multiple locations within a resource to rebase. This is a convenience for specifying multiple rebase rules with only different paths. Mutually exclusive with `path`. Available in v0.27.0+. + - `type` (string) specifies strategy to modify field values. Allowed values: `copy` or `remove`. `copy` will update the field value; `remove` will delete the field. + - `sources` (array of `new` or `existing`) specifies a preference order for the source of the referenced field value being rebased. `new` refers to an updated resource from user input, where `existing` refers to a resource already in the cluster. If the field value being rebased is not found in any of the sources provided, kapp will error. Only used with `type: copy`. \ + Examples: + - `[existing, new]` – If field value is present in the `existing` resource on cluster, use that value, otherwise use the value in the `new` user input. + - `[existing]` – Only look for field values in resources already on cluster, corresponding value you provide in new resource will be overwritten. + - `resourceMatchers` (array) specifies rules to find matching resources. See various resource matchers below. + - `ytt` specifies choice as [ytt](https://carvel.dev/ytt/) for rebase rule. Available in v0.38.0+. + - `overlayContractV1` allows to use ytt overlay to modify provided resource based on existing resource. + - `overlay.yml` overlay YAML file. + - Following fields are accessible via `data.values` inside ytt: + - `data.values.existing` resource from live cluster + - `data.values.new` resource from config (post-prep) + - `data.values._current` resource after previous rebase rules already applied + +Rebase rule to `copy` the `clusterIP` field value to `Service`/`v1` resources; if `clusterIp` is present in the `new` user input, use that value, otherwise use the value in `existing` resource on cluster: +```yaml +rebaseRules: +- path: [spec, clusterIP] + type: copy + sources: [new, existing] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: v1, kind: Service} +``` + +Rebase rule to `copy` the `clusterIP` and `healthCheckNodePort` field values from the `existing` resource on cluster, to `Service`/`v1` resources: +```yaml +rebaseRules: +- paths: + - [spec, clusterIP] + - [spec, healthCheckNodePort] + type: copy + sources: [existing] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: v1, kind: Service} +``` + +See [ytt rebase rule](https://github.com/vmware-tanzu/carvel-kapp/blob/d3ee9a01b5f0d7d5632b6a157ea7d0338730d497/pkg/kapp/config/default.go#L123-L154) (included in default configuration) for retaining cluster added token secret in ServiceAccount's secrets array. + +### ownershipLabelRules + +`ownershipLabelRules` specify locations for inserting kapp generated labels. These labels allow kapp to track which resources belong to which application. For resources that describe creation of other resources (e.g. `Deployment` or `StatefulSet`), configuration may need to specify where to insert labels for child resources that will be created. `kapp.k14s.io/disable-default-ownership-label-rules: ""` (value must be empty) annotation can be be used to exclude an individual resource from default onwership label rules. + +### labelScopingRules + +`labelScopingRules` specify locations for inserting kapp generated labels that scope resources to resources within current application. `kapp.k14s.io/disable-default-label-scoping-rules: ""` (as of v0.33.0+, or use `kapp.k14s.io/disable-label-scoping: ""` in earlier versions) annotation can be used to exclude an individual resource from label scoping. + +### waitRules + +Available in v0.29.0+. + +`waitRules` specify how to wait for resources that kapp does not wait for by default. Each rule provides a way to specify which `status.conditions` indicate success or failure. Once any of the condition matchers successfully match against one of the resource's conditions, kapp will stop waiting for the matched resource and report any failures. (If this functionality is not enough to wait for resources in your use case, please reach out on Slack to discuss further.) + +```yaml +waitRules: +- supportsObservedGeneration: true + conditionMatchers: + - type: Failed + status: "True" + failure: true + - type: Deployed + status: "True" + success: true + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: corp.com/v1, kind: DatabaseInstance} +``` + +```yaml +waitRules: +- supportsObservedGeneration: true + conditionMatchers: + - type: Ready + status: "False" + failure: true + - type: Ready + status: "True" + success: true + supportsObservedGeneration: true # available at condition level from v0.47.0+ + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: corp.com/v1, kind: Application} +``` + +Available in v0.48.0+. + +ytt `waitRules` can be for Custom Resources that don't have `conditions` field in their `status`. This allows users to configure arbitrary rules. `is_done(resource)` method can be defined as part of a ytt waitRule to return the done state based on resource fields. + +```yaml +waitRules: + - ytt: + funcContractV1: + resource.star: | + def is_done(resource): + state = resource.status.currentState + if state == "Failed": + return {"done": True, "successful": False, "message": "Current state as Failed"} + elif state == "Running": + return {"done": True, "successful": True, "message": "Current state as Running"} + else: + return {"done": False, "successful": False, "message": "Not in Failed or Running state"} + end + end + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: , kind: } +``` + +Available in v0.50.0+ + +`unblockChanges` can be used for conditions to unblock any dependent resources. These conditions are treated as non success/failure conditions. It can also be used along with ytt waitRules. + +```yaml +waitRules: +- conditionMatchers: + - type: Progressing + status: "True" + unblockChanges: true + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: corp.com/v1, kind: Application} +``` + +### templateRules + +`templateRules` specify how versioned resources affect other resources. In above example, versioned config maps are said to affect deployments. [Read more about versioned resources](diff.md#versioned-resources). + +### additionalLabels + +`additionalLabels` specify additional labels to apply to all resources for custom uses by the user (added based on `ownershipLabelRules`). + +### diffAgainstLastAppliedFieldExclusionRules + +`diffAgainstLastAppliedFieldExclusionRules` specify which fields should be removed before diff-ing against last applied resource. These rules are useful for fields are "owned" by the cluster/controllers, and are only later updated. For example `Deployment` resource has an annotation that gets set after a little bit of time after resource is created/updated (not during resource admission). It's typically not necessary to use this configuration. + +### diffMaskRules + +`diffMaskRules` specify which field values should be masked in diff. By default `v1/Secret`'s `data` fields are masked. Currently only applied to `deploy` command. + +### changeGroupBindings + +Available in v0.25.0+. + +`changeGroupBindings` bind specified change group to resources matched by resource matchers. This is an alternative to using `kapp.k14s.io/change-group` annotation to add change group to resources. See `kapp deploy-config` for default bindings. + +### changeRuleBindings + +Available in v0.25.0+. + +`changeRuleBindings` bind specified change rules to resources matched by resource matchers. This is an alternative to using `kapp.k14s.io/change-rule` annotation to add change rules to resources. See `kapp deploy-config` for default bindings. + +--- +## Resource matchers + +Resource matchers (as used by `rebaseRules`, `ownershipLabelRules`, `labelScopingRules`, `templateRules`, `diffAgainstLastAppliedFieldExclusionRules`, and `diffMaskRules`): + +### allMatcher + +Matches all resources + +```yaml +allMatcher: {} +``` + +### anyMatcher + +Matches resources that match one of matchers + +```yaml +anyMatcher: + matchers: + - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} + - apiVersionKindMatcher: {apiVersion: extensions/v1alpha1, kind: Deployment} +``` + +### notMatcher + +Matches any resource that does not match given matcher + +```yaml +notMatcher: + matcher: + apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} +``` + +### andMatcher + +Matches any resource that matches all given matchers + +```yaml +andMatcher: + matchers: + - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} + - hasNamespaceMatcher: {} +``` + +### apiGroupKindMatcher + +```yaml +apiGroupKindMatcher: {apiGroup: apps, kind: Deployment} +``` + +### apiVersionKindMatcher + +```yaml +apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} +``` + +### kindNamespaceNameMatcher + +```yaml +kindNamespaceNameMatcher: {kind: Deployment, namespace: mysql, name: mysql} +``` + +### hasAnnotationMatcher + +Matches resources that have particular annotation + +```yaml +hasAnnotationMatcher: + keys: + - kapp.k14s.io/change-group +``` + +### hasNamespaceMatcher + +Matches any resource that has a non-empty namespace + +```yaml +hasNamespaceMatcher: {} +``` + +Matches any resource with namespace that equals to one of the specified names + +```yaml +hasNamespaceMatcher: + names: [app1, app2] +``` + +### customResourceMatcher + +Matches any resource that is not part of builtin k8s API groups (e.g. apps, batch, etc.). It's likely that over time some builtin k8s resources would not be matched. + +```yaml +customResourceMatcher: {} +``` + +### emptyFieldMatcher + +Available in v0.34.0+. + +Matches any resource that has empty specified field + +```yaml +emptyFieldMatcher: + path: [aggregationRule] +``` + +--- +## Paths + +Path specifies location within a resource (as used `rebaseRules` and `ownershipLabelRules`): + +``` +[spec, clusterIP] +``` + +``` +[spec, volumeClaimTemplates, {allIndexes: true}, metadata, labels] +``` + +``` +[spec, volumeClaimTemplates, {index: 0}, metadata, labels] +``` + +--- +## Config wrapped in ConfigMap + +Available of v0.34.0+. + +Config resource could be wrapped in a ConfigMap to support same deployment configuration by tools that do not understand kapp's `Config` resource directly. ConfigMap carrying kapp config must to be labeled with `kapp.k14s.io/config` and have `config.yml` data key. Such config maps will be applied to the cluster, unlike config given as `Config` resource. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: my-kapp-config + labels: + kapp.k14s.io/config: "" +data: + config.yml: | + apiVersion: kapp.k14s.io/v1alpha1 + kind: Config + rebaseRules: + - path: [rules] + type: copy + sources: [existing, new] + resourceMatchers: + - notMatcher: + matcher: + emptyFieldMatcher: + path: [aggregationRule] +``` + +NOTE: `kapp` is _only_ affected by a `Config` (whether wrapped in a `ConfigMap` or not) when supplied as a direct input (i.e. as a `-f` argument). Any `ConfigMap` containing a `Config` is already present in the target cluster has _no affect whatsoever_ on `kapp`'s behavior. diff --git a/site/content/kapp/docs/v0.53.0/configmap-migration.md b/site/content/kapp/docs/v0.53.0/configmap-migration.md new file mode 100644 index 000000000..9d1d3a4f7 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/configmap-migration.md @@ -0,0 +1,125 @@ +--- +aliases: [/kapp/docs/latest/configmap-migration] +title: Configmap Migration (experimental) +--- + +## Overview + +Kapp internally uses a configmap to store information about an application. + +This configmap name has defaulted to the app name supplied during a deploy. `kapp deploy -a `. + +Example: + +```bash +kapp deploy -a my-app -f app.yml --yes + +$ kapp ls +Namespace Name Namespaces Lcs Lca +default my-app default true 7s + +$ kubectl get configmaps + +NAME DATA AGE +my-app 1 1m +``` + +This is challenging when users also want to create a configmap named `my-app` for their application. It is not expected that `kapp` is already using this configmap name. + +## Enabling Configmap migration + +As of v0.47.0+, kapp now supports a new optional boolean environment variable `KAPP_FQ_CONFIGMAP_NAMES` which can be used to migrate **both new and existing configmaps** to the new naming convention: `.apps.k14s.io`. + +- `KAPP_FQ_CONFIGMAP_NAMES=true` opts into the new kapp behavior. +- `KAPP_FQ_CONFIGMAP_NAMES=false` maintains the current kapp behavior. + +*Important Note: The app name is not being changed, only the configmap name, all references to the app name can remain the same.* + +### Examples + +#### Deploy new App + +```bash +export KAPP_FQ_CONFIGMAP_NAMES=true + +kapp deploy -a my-app -f app.yml --yes + +$ kapp ls +Namespace Name Namespaces Lcs Lca +default my-app default true 7s + +$ kubectl get configmaps + +NAME DATA AGE +my-app.apps.k14s.io 1 1m +``` + +#### Deploy existing App + +```bash +$ kapp ls +Namespace Name Namespaces Lcs Lca +default my-app default true 7s + +export KAPP_FQ_CONFIGMAP_NAMES=true + +$ kapp deploy -a my-app -f app.yml --yes + +$ kubectl get configmaps + +NAME DATA AGE +my-app.apps.k14s.io 1 1m +``` + +#### Delete + +```bash +# With migration enabled +$ KAPP_FQ_CONFIGMAP_NAMES=true kapp delete -a my-app + +Changes + +Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri +default simple-app Deployment 2/2 t 28m delete - delete ok - + +# With migration disabled +$ KAPP_FQ_CONFIGMAP_NAMES=false kapp delete -a my-app + +App 'my-app' (namespace: default) does not exist + +``` + +### Caveats + +1. Migrated apps will show up with the suffix with previous versions of kapp (0.46.0-): + +```bash +export KAPP_FQ_CONFIGMAP_NAMES=true + +kapp deploy -a my-app -f app.yml --yes + +$ kapp ls +Namespace Name Namespaces Lcs Lca +default my-app default true 7s + +# With old kapp versions +$ kapp ls +Namespace Name Namespaces Lcs Lca +default my-app.apps.k14s.io default true 7s +``` + +### Opting out after migration + +To return to the previous configmap naming convention, the following steps must be followed: + +1. `kubectl get configmap my-app.apps.k14s.io -o yaml > app.yml` + +2. Find the `metadata.name` field in `app.yml` and remove the suffix `.apps.k14s.io` + +3. Find the annotation named `kapp.k14s.io/is-configmap-migrated` in `metadata.annotations` and remove it + +4. `kubectl create -f app.yml` + +5. `kubectl delete configmap my-app.apps.k14s.io` + +*Important Note: Ensure the configmap with suffix `apps.k14s.io` is deleted after opting-out!* diff --git a/site/content/kapp/docs/v0.53.0/dangerous-flags.md b/site/content/kapp/docs/v0.53.0/dangerous-flags.md new file mode 100644 index 000000000..25969c7d6 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/dangerous-flags.md @@ -0,0 +1,34 @@ +--- +aliases: [/kapp/docs/latest/dangerous-flags] +title: Dangerous Flags +--- + +## Overview + +There are several flags in `kapp deploy/delete/etc.` commands that might be helpful in rare cases, but can cause problems if used improperly. These are their stories: + +## `--dangerous-allow-empty-list-of-resources` + +This flag allows `kapp deploy` to accept empty set of new resources. Given that kapp deploy converges set of resources, when empty set is provided, kapp will delete all existing resources. + +This commonly happens unintentionally. When configuration is piped into kapp (e.g. `ytt -f config/ | kapp deploy -f- ...`) and resource producing command fails (ytt in this example), kapp will not receive any resources by the time is closes. Since providing empty set of resources intentionally is pretty rare, this functionality is behind a flag. + +## `--dangerous-override-ownership-of-existing-resources` + +This flag allows `kapp deploy` to take ownership of resources that are already associated with another application (i.e. already has `kapp.k14s.io/app` label with a different value). + +Most commonly user may have _unintentionally_ included resource that is already deployed, hence by default we do not want to override that resource with a new copy. This may happen when multiple apps accidently specified same resource (i.e. same name under same namespace). In most cases this is not what user wants. + +This flag may be useful in cases when multiple applications (managed by kapp) need to be merged into one, or may be previously owning application have been deleted but its resources were kept. + +Note that by default if resource is given to kapp and it already exists in the cluster, and is not owned by another application, kapp will label it to belong to deploying app. + +## `--dangerous-ignore-failing-api-services` + +In some cases users may encounter that they have misbehaving `APIServices` within they cluster. Since `APIServices` affect how one finds existing resources within a cluster, by default kapp will show error similar to below and stop: + +``` +Error: ... unable to retrieve the complete list of server APIs: <...>: the server is currently unable to handle the request +``` + +In cases when APIService cannot be fixed, this flag can be used to let kapp know that it is okay to proceed even though it's not able to see resources under that `APIService`. Note when this flag is used, kapp will effectively think that resources under misbehaving `APIService` do not exist. diff --git a/site/content/kapp/docs/v0.53.0/diff.md b/site/content/kapp/docs/v0.53.0/diff.md new file mode 100644 index 000000000..f4a8237e1 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/diff.md @@ -0,0 +1,378 @@ +--- +aliases: [/kapp/docs/latest/diff] +title: Diff stage +--- +## Overview + +kapp compares resources specified in files against resources that exist in Kubernetes API. Once change set is calculated, it provides an option to apply it (see [Apply](apply.md) section for further details). + +There are five different types of operations: `create`, `update`, `delete`, `noop` (shown as empty), `exists` (added in v0.43.0). Seen in `Op` column of diff summary table. Additionally there is `Op strategy` column (shorted as `Op st.`), added in v0.31.0+, that shows supplemental information how operation will be performed (for example [`fallback on replace`](apply.md#kappk14sioupdate-strategy) for `update` operation). + +There are three different types of waiting: `reconcile` (waits until resource has converged to its desired state; see [apply waiting](apply-waiting.md) for waiting semantics), `delete` (waits until resource is gone), `noop` (shown as empty). Seen in `Wait to` column of diff summary table. + +## Diff strategies + +There are two diff strategies used by kapp: + +1. kapp compares against last applied resource content (previously applied by kapp; stored in annotation `kapp.k14s.io/original`) **if** there were no outside changes done to the resource (i.e. done outside of kapp, for example, by another team member or controller); kapp tries to use this strategy as much as possible to produce more user-friendly diffs. + +2. kapp compares against live resource content **if** it detects there were outside changes to the resource (hence, sometimes you may see a diff that shows several deleted fields even though these fields are not specified in the original file) + +Strategy is selected for each resource individually. You can control which strategy is used for all resources via `--diff-against-last-applied=bool` flag. + +Related: [rebase rules](config.md/#rebaserules). + +## Versioned Resources + +In some cases it's useful to represent an update to a resource as an entirely new resource. Common example is a workflow to update ConfigMap referenced by a Deployment. Deployments do not restart their Pods when ConfigMap changes making it tricky for wide variety of applications for pick up ConfigMap changes. kapp provides a solution for such scenarios, by offering a way to create uniquely named resources based on an original resource. + +Anytime there is a change to a resource marked as a versioned resource, entirely new resource will be created instead of updating an existing resource. + +To make resource versioned, add `kapp.k14s.io/versioned` annotation with an empty value. Created resource follow `{resource-name}-ver-{n}` naming pattern by incrementing `n` any time there is a change. + +Example: +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: secret-sa-sample + annotations: + kapp.k14s.io/versioned: "" +``` +This will create versioned resource named `secret-sa-sample-ver-1` + +```bash +Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri +default secret-sa-sample-ver-1 Secret - - create - reconcile - - + +Op: 1 create, 0 delete, 0 update, 0 noop, 0 exists +Wait to: 1 reconcile, 0 delete, 0 noop +``` + +Additionally kapp follows configuration rules (default ones, and ones that can be provided as part of application) to find and update object references (since new resource name is not something that configuration author knew about). + +{{< detail-tag "Example" >}} +Sample Config +```yaml +apiVersion: kapp.k14s.io/v1alpha1 +kind: Config +templateRules: + - resourceMatchers: + - apiVersionKindMatcher: {apiVersion: v1, kind: ConfigMap} + affectedResources: + objectReferences: + - path: [spec, template, spec, containers, {allIndexes: true}, env, {allIndexes: true}, valueFrom, configMapKeyRef] + resourceMatchers: + - apiVersionKindMatcher: {apiVersion: apps/v1, kind: Deployment} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: special-config + annotations: + kapp.k14s.io/versioned: "" +data: + special.how: very-good +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment + labels: + app: nginx +spec: + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.14.2 + ports: + - containerPort: 80 + env: + - name: SPECIAL_LEVEL_KEY + valueFrom: + configMapKeyRef: + name: special-config + key: special.how +``` +Here we have specified the configuration rules that will update the ConfigMap object reference in resources of Kind Deployment. Here `ConfigMap` special-config is marked as versioned so anytime there is an update it will create a new resource with name `special-config-ver-{n}` and update the same name in resource of kind `Deployment` under `configMapKeyRef`. This example is part of [default configuration rule](https://github.com/vmware-tanzu/carvel-kapp/blob/28b17b775558ef4c64ce27a5655b81c00c8a2f59/pkg/kapp/config/default.go#L299) that kapp follows. +{{< /detail-tag >}} + +As of v0.38.0+, `kapp.k14s.io/versioned-keep-original` annotation can be used in conjunction with `kapp.k14s.io/versioned` to have the original resource (resource without `-ver-{n}` suffix in name) along with versioned resource. + +Example: +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: secret-sa-sample + annotations: + kapp.k14s.io/versioned: "" + kapp.k14s.io/versioned-keep-original: "" +``` +This will create two resources one with original name `secret-sa-sample` and one with `-ver-{n}` suffix in name `secret-sa-sample-ver-1`. +```bash +Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri +default secret-sa-sample Secret - - create - reconcile - - +^ secret-sa-sample-ver-1 Secret - - create - reconcile - - + +Op: 2 create, 0 delete, 0 update, 0 noop, 0 exists +Wait to: 2 reconcile, 0 delete, 0 noop +``` + +You can control number of kept resource versions via `kapp.k14s.io/num-versions=int` annotation. + +As of v0.41.0+, the `kapp.k14s.io/versioned-explicit-ref` can be used to explicitly refer to a versioned resource. This annotation allows a resource to be updated whenever a new version of the referred resource is created. + +Multiple annotations with the prefix `kapp.k14s.io/versioned-explicit-ref.`(Note the "." at the end) can be used to define multiple explicit references. + +Example: +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-1 + annotations: + kapp.k14s.io/versioned: "" +data: + foo: bar +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-2 + annotations: + kapp.k14s.io/versioned-explicit-ref: | + apiVersion: v1 + kind: ConfigMap + name: config-1 +data: + foo: bar +``` +Here, `config-2` explicitly refers `config-1` and is updated with the latest versioned name when `config-1` is versioned. +```bash +@@ create configmap/config-1-ver-2 (v1) namespace: default @@ + ... + 1, 1 data: + 2 - foo: bar + 2 + foo: alpha + 3, 3 kind: ConfigMap + 4, 4 metadata: +@@ update configmap/config-2 (v1) namespace: default @@ + ... + 8, 8 kind: ConfigMap + 9 - name: config-1-ver-1 + 9 + name: config-1-ver-2 + 10, 10 creationTimestamp: "2021-09-29T17:27:34Z" + 11, 11 labels: + +Changes + +Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri +default config-1-ver-2 ConfigMap - - create - reconcile - - +^ config-2 ConfigMap - 14s update - reconcile ok - +``` + +Try deploying [redis-with-configmap example](https://github.com/vmware-tanzu/carvel-kapp/tree/develop/examples/gitops/redis-with-configmap) and changing `ConfigMap` in a next deploy. + +--- +## Controlling diff via resource annotations + +### kapp.k14s.io/disable-original + +kapp, by default, records the resource copy into its annotation `kapp.k14s.io/original` while applying the resource to the cluster. + +{{< detail-tag "Example" >}} +Sample Config +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-1 + namespace: default +data: + foo: bar +``` +After deploying the resource, kapp added the annotation `kapp.k14s.io/original` with the content of the resource that was given to kapp: + +```bash +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-1 + namespace: default + annotations: + kapp.k14s.io/original: '{ "apiVersion": "v1", "kind": "ConfigMap", ...snip... }' +data: + foo: bar +``` +{{< /detail-tag >}} + +`kapp.k14s.io/disable-original` annotation controls whether to record provided resource copy (rarely wanted) + +Possible values: "" (empty). In some cases it's not possible or wanted to record applied resource copy into its annotation `kapp.k14s.io/original`. One such case might be when resource is extremely lengthy (e.g. long ConfigMap or CustomResourceDefinition) and will exceed annotation value max length of 262144 bytes. + +{{< detail-tag "Example" >}} +Sample Config +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-1 + namespace: default + annotations: + kapp.k14s.io/disable-original: "" +data: + foo: bar +``` +After deploying the resource, kapp didn't add the annotation `kapp.k14s.io/original` this time: + +```bash +apiVersion: v1 +kind: ConfigMap +metadata: + name: config-1 + namespace: default + annotations: + kapp.k14s.io/disable-original: "" +data: + foo: bar +``` +{{< /detail-tag >}} + +--- +## Controlling diff via deploy flags + +Diff summary shows quick information about what's being changed: +- `--diff-summary=bool` (default `true`) shows diff summary, listing how resources have changed + {{< detail-tag "Example" >}} +Sample config +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: sample +stringData: + foo: bar +``` +```bash +$ kapp deploy -a sample-secret -f config.yaml --diff-summary=true +Target cluster 'https://127.0.0.1:56540' (nodes: kind-control-plane) + +Changes + +Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri +default sample Secret - - create - reconcile - - + +Op: 1 create, 0 delete, 0 update, 0 noop, 0 exists +Wait to: 1 reconcile, 0 delete, 0 noop + +Continue? [yN]: +``` + {{< /detail-tag >}} + +Diff changes (line-by-line diffs) are useful for looking at actual changes, when app is re-deployed: +- `--diff-changes=bool` (`-c`) (default `false`) shows line-by-line diffs +- `--diff-context=int` (default `2`) controls number of lines to show around changed lines +- `--diff-mask=bool` (default `true`) controls whether to mask sensitive fields + {{< detail-tag "Example" >}} +Sample config +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: sample +stringData: + foo: bar +``` +```bash +# deploy sample-secre app +$ kapp deploy -a sample-secret -f config.yaml + +#update config +... +stringData: + foo: bars +... + +# re-deploy sample-secret app with required diff-changes flag to see line by line changes +$ kapp deploy -a sample-secret -f config.yaml --diff-changes=true --diff-context=4 +Target cluster 'https://127.0.0.1:56540' (nodes: kind-control-plane) + +@@ update secret/sample (v1) namespace: default @@ + ... + 30, 30 resourceVersion: "244751" + 31, 31 uid: b2453c2a-8dc8-4ed1-9b59-791547f78ea8 + 32, 32 stringData: + 33 - foo: <-- value not shown (#1) + 33 + foo: <-- value not shown (#2) + 34, 34 + +Changes + +Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri +default sample Secret - 7m update - reconcile ok - + +Op: 0 create, 0 delete, 1 update, 0 noop, 0 exists +Wait to: 1 reconcile, 0 delete, 0 noop + +Continue? [yN]: + +# --diff-mask=true by default, note the masked value for secret data + +# try out kapp deploy -a sample-secret -f config.yaml --diff-mask=false --diff-changes=true --diff-context=2 +``` + {{< /detail-tag >}} + +Controlling how diffing is done: + +- `--diff-against-last-applied=bool` (default `true`) forces kapp to use particular diffing strategy (see above). +- `--diff-run=bool` (default `false`) set the flag to true, to stop after showing diff information. +- `--diff-exit-status=bool` (default `false`) controls exit status for diff runs (`0`: unused, `1`: any error, `2`: no changes, `3`: pending changes) + {{< detail-tag "Example" >}} + Sample config +```yaml +--- +apiVersion: v1 +kind: Secret +metadata: + name: sample +stringData: + foo: bar +``` +```bash +# deploy secret-sample app +$ kapp deploy -a secret-sample -f config.yaml --diff-run=true --diff-exit-status=true +Target cluster 'https://127.0.0.1:56540' (nodes: kind-control-plane) + +Changes + +Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri +default sample Secret - - create - reconcile - - + +Op: 1 create, 0 delete, 0 update, 0 noop, 0 exists +Wait to: 1 reconcile, 0 delete, 0 noop + +kapp: Error: Exiting after diffing with pending changes (exit status 3) + +# note that kapp exits after diff and displays the exit status + +``` + {{< /detail-tag >}} + +Diff filter allows to filter changes based on operation (add/update/delete), newResource (configuration provided to kapp) and existingResource (resources in Kubernetes cluster) + +- `--diff-filter='{"and":[{"ops":["update"]},{"existingResource":{"kinds":["Deployment"]}]}'` will keep the resources which are getting updated and were of kind Deployment. diff --git a/site/content/kapp/docs/v0.53.0/faq.md b/site/content/kapp/docs/v0.53.0/faq.md new file mode 100644 index 000000000..a75b85f77 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/faq.md @@ -0,0 +1,128 @@ +--- +aliases: [/kapp/docs/latest/faq] +title: FAQ +--- + +## Migrating from `kubectl apply` to kapp + +Switching from `kubectl apply` to `kapp deploy` will allow kapp to adopt resources mentioned in a given config. +However, kapp will try to insert a few of its labels in bodies of some resources, like Deployments, which may fail due to those resources having immutable fields that kapp tries to update (spec.selector on Deployments). + +To prevent this failure, add the [`kapp.k14s.io/disable-default-label-scoping-rules: ""` annotation](config.md#labelscopingrules) as a [kapp configuration](config.md) to prevent kapp from touching the immutable fields when adopting a resource. + +Additional Resources: [GitHub Issue](https://github.com/vmware-tanzu/carvel-kapp/issues/204), [Slack Thread](https://kubernetes.slack.com/archives/CH8KCCKA5/p1606079730457700) + +## `Error: Asking for confirmation: EOF` + +This probably means you have piped configuration into kapp and did not specify `--yes` (`-y`) flag to continue. It's necessary because kapp can no longer ask for confirmation via stdin. Feel free to re-run the command with `--diff-changes` (`-c`) to make sure pending changes are correct. Instead of using a pipe you can also use an anonymous fifo keeping stdin free for the confirmation prompt, e.g. `kapp deploy -a app1 -f <(ytt -f config/)` + +--- +## Where to store app resources (i.e. in which namespace)? + +See [state namespace](state-namespace.md) doc page. + +--- +## `... Field is immutable` error + +> After changing the labels/selectors in one of my templates, I'm getting the `MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable (reason: Invalid)` errors on deployment resource. Is there a way to tell kapp to force the change? + +[via slack](https://kubernetes.slack.com/archives/CH8KCCKA5/p1565600090224400) + +Some fields on a resource are immutable. kapp provides a `kapp.k14s.io/update-strategy` annotation that controls how kapp will update resource. One of the strategies is `fallback-on-replace` which will have kapp recreate an object (delete, wait, then create) if initial update results in `Invalid` error. See [Controlling apply via resource annotations](apply.md#controlling-apply-via-resource-annotations) for details. + +--- +## `Job.batch is invalid: ... spec.selector: Required value` error + +`batch.Job` resource is augmented by the Job controller with unique labels upon its creation. When using kapp to subsequently update existing Job resource, API server will return `Invalid` error since given configuration does not include `spec.selector`, and `job-name` and `controller-uid` labels. kapp's [rebase rules](config.md#rebaserules) can be used to copy over necessary configuration from server side copy; however, since Job resource is mostly immutable, we recommend to use [`kapp.k14s.io/update-strategy` annotation](apply.md#kappk14sioupdate-strategy) set to `fallback-on-replace` to recreate Job resource with any updates. + +--- +## Updating Deployments when ConfigMap changes + +> Can kapp force update on ConfigMaps in Deployments/DaemonSets? Just noticed that it didn't do that and I somehow expected it to. + +[via slack](https://kubernetes.slack.com/archives/CH8KCCKA5/p1565624685226400) + +kapp has a feature called [versioned resources](diff.md#versioned-resources) that allows kapp to create uniquely named resources instead of updating resources with changes. Resources referencing versioned resources are forced to be updated with new names, and therefore are changed, thus solving a problem of how to propagate changes safely. + +--- +## Quick way to find common kapp command variations + +See [cheatsheet](cheatsheet.md). + +--- +## Limit number of ReplicaSets for Deployments + +> Everytime I do a new deploy w/ kapp I see a new replicaset, along with all of the previous ones. + +[via slack](https://kubernetes.slack.com/archives/CH8KCCKA5/p1565887856281400) + +`Deployment` resource has a field `.spec.revisionHistoryLimit` that controls how many previous `ReplicaSets` to keep. See [Deployment's clean up polciy](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#clean-up-policy) for more details. + +--- +## Changes detected immediately after successful deploy + +Sometimes Kubernetes API server will convert submitted field values into their canonical form server-side. This will be detected by kapp as a change during a next deploy. To avoid such changes in future, you will have to change your provided field values to what API server considers as canonical. + +``` +... +186 - cpu: "2" +187 - memory: 1Gi + 170 + cpu: 2000m + 171 + memory: 1024Mi +... +``` + +Consider using [ytt](/ytt) and [its overlay feature](/ytt/docs/latest/lang-ref-ytt-overlay/) to change values if you do not control source configuration. + +--- +## Changes detected after resource is modified server-side + +There might be cases where other system actors (various controllers) may modify resource outside of kapp. Common example is Deployment's `spec.replicas` field is modified by Horizontal Pod Autoscaler controller. To let kapp know of such external behaviour use custom `rebaseRules` configuration (see [HPA and Deployment rebase](hpa-deployment-rebase.md) for details). + +--- +## Colors are not showing up in my CI build, in my terminal, etc. + +Try setting `FORCE_COLOR=1` environment variable to force enabling color output. Available in v0.23.0+. + +--- +## How can I version apps deployed by kapp? + +kapp itself does not provide any notion of versioning, since it's just a tool to reconcile config. We recommend to include a ConfigMap in your deployment with application metadata e.g. git commit, release notes, etc. + +--- +## `Resource ... is associated with a different label value` + +Resource ownership is tracked by app labels. kapp expects that each resource is owned by exactly one app. + +If you are receiving this error and are using correct app name, it might be that you are targeting wrong namespace where app is located. Use `--namespace` to set correct namespace. + +Additional resources: [State Namespace](state-namespace.md), [Slack Thread](https://kubernetes.slack.com/archives/CH8KCCKA5/p1589264289257000) + +--- +## Why does kapp hang when trying to delete a resource? + +By default, kapp won't delete resources it didn't create. You can see which resources are owned by kapp in output of `kapp inspect -a app-name` in its `Owner` column. You can force kapp to apply this ignored change using `--apply-ignored` [flag](apply.md#controlling-apply-via-deploy-flags). Alternatively if you are able to set [kapp.k14s.io/owned-for-deletion](apply.md#kappk14sioowned-for-deletion) annotation on resource that will be created, kapp will take that as a request to "own it" for deletion. This comes in handy for example with PVCs created by StatefulSet. + +--- +## How does kapp handle merging? + +kapp explicitly decided against _basic_ 3 way merge, instead allowing the user to specify how to resolve conflicts via rebase rules. + +Resources: [merge method](merge-method.md), [rebase rules](config.md#rebaserules) + +--- +## Can I force an update for a change that does not produce a diff? + +If kapp does not detect changes, it won't perform an update. To force changes every time you can set [`kapp.k14s.io/nonce`](apply.md#kappk14siononce) annotation. That way, every time you deploy the resource will appear to have changes. + +--- +## How can I remove decorative headings from kapp inspect output? + +Use `--tty=false` flag which will disable decorative output. Example: `kapp inspect --raw --tty=false`. + +Additional resources: [tty flag in kapp code](https://github.com/vmware-tanzu/carvel-kapp/blob/3f3e207d7198cdedd6985761ecb0d9616a84e305/pkg/kapp/cmd/ui_flags.go#L20) + +--- +## How can I get kapp to skip waiting on some resources? + +kapp allows to control waiting behavior per resource via [resource annotations](apply-waiting.md#controlling-waiting-via-resource-annotations). When used, the resource will be applied to the cluster, but will not wait to reconcile. diff --git a/site/content/kapp/docs/v0.53.0/gitops.md b/site/content/kapp/docs/v0.53.0/gitops.md new file mode 100644 index 000000000..156a06c19 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/gitops.md @@ -0,0 +1,17 @@ +--- +aliases: [/kapp/docs/latest/gitops] +title: GitOps +--- + +## Using kapp with GitOps workflow + +kapp provides a set of commands to make GitOps workflow very easy. Assuming that you have a CI environment or some other place where `kapp` can run based on a trigger (e.g. for every Git repo change) or continuously (e.g. every 5 mins), you can use following command: + +```bash +$ ls my-repo +. .. app1/ app2/ app3/ + +$ kapp app-group deploy -g my-env --directory my-repo +``` + +Above command will deploy an application for each subdirectory in `my-repo` directory (in this case `app1`, `app2` and `app3`). It will also remove old applications if subdirectories are deleted. diff --git a/site/content/kapp/docs/v0.53.0/hpa-deployment-rebase.md b/site/content/kapp/docs/v0.53.0/hpa-deployment-rebase.md new file mode 100644 index 000000000..722c9bc68 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/hpa-deployment-rebase.md @@ -0,0 +1,42 @@ +--- +aliases: [/kapp/docs/latest/hpa-deployment-rebase] +title: HPA and Deployment rebase +--- +## HPA and Deployment rebase + +Here is an example on how to use custom `rebaseRules` to "prefer" server chosen value for `spec.replicas` field for a particular Deployment. + +```yaml +apiVersion: kapp.k14s.io/v1alpha1 +kind: Config +rebaseRules: +- path: [spec, replicas] + type: copy + sources: [existing, new] + resourceMatchers: + - kindNamespaceNameMatcher: + kind: Deployment + namespace: my-ns + name: my-app +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: my-app + namespace: my-ns +... +--- +apiVersion: autoscaling/v1 +kind: HorizontalPodAutoscaler +metadata: + name: my-app + namespace: my-ns +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: my-app + minReplicas: 1 + maxReplicas: 10 + targetCPUUtilizationPercentage: 50 +``` diff --git a/site/content/kapp/docs/v0.53.0/install.md b/site/content/kapp/docs/v0.53.0/install.md new file mode 100644 index 000000000..e2bda94da --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/install.md @@ -0,0 +1,58 @@ +--- +aliases: [/kapp/docs/latest/install] +title: Install +--- + +## Via script (macOS or Linux) + +(Note that `install.sh` script installs other Carvel tools as well.) + +Install binaries into specific directory: + +```bash +$ mkdir local-bin/ +$ curl -L https://carvel.dev/install.sh | K14SIO_INSTALL_BIN_DIR=local-bin bash + +$ export PATH=$PWD/local-bin/:$PATH +$ kapp version +``` + +Or system wide: + +```bash +$ wget -O- https://carvel.dev/install.sh > install.sh + +# Inspect install.sh before running... +$ sudo bash install.sh +$ kapp version +``` + +## Via Homebrew (macOS or Linux) + +Based on [github.com/vmware-tanzu/homebrew-carvel](https://github.com/vmware-tanzu/homebrew-carvel). + +```bash +$ brew tap vmware-tanzu/carvel +$ brew install kapp +$ kapp version +``` + +## Specific version from a GitHub release + +To download, click on one of the assets in a [chosen GitHub release](https://github.com/vmware-tanzu/carvel-kapp/releases), for example for 'kapp-darwin-amd64'. + +```bash +# **Compare binary checksum** against what's specified in the release notes +# (if checksums do not match, binary was not successfully downloaded) +$ shasum -a 256 ~/Downloads/kapp-darwin-amd64 +08b25d21675fdc77d4281c9bb74b5b36710cc091f30552830604459512f5744c /Users/pivotal/Downloads/kapp-darwin-amd64 + +# Move binary next to your other executables +$ mv ~/Downloads/kapp-darwin-amd64 /usr/local/bin/kapp + +# Make binary executable +$ chmod +x /usr/local/bin/kapp + +# Check its version +$ kapp version +``` diff --git a/site/content/kapp/docs/v0.53.0/integrating-with-other-tools.md b/site/content/kapp/docs/v0.53.0/integrating-with-other-tools.md new file mode 100644 index 000000000..aa84510bf --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/integrating-with-other-tools.md @@ -0,0 +1,27 @@ +--- +aliases: [/kapp/docs/latest/integrating-with-other-tools] +title: Integrating with Other Tools +--- + +**Note:** This is a non-exhaustive list of integrations + +## ytt and kbld + +We recommend to use kapp with [ytt](/ytt) and [kbld](/kbld) to cover your configuration templating and image building needs. Typical workflow may look like this: + +```bash +ytt -f config/ | kbld -f - | kapp deploy -a app1 -f- -c -y +``` + +## Helm + +If you want to take advantage of both Helm templating and kapp deployment mechanisms, you can use `helm template` command to build configuration, and have kapp apply to the cluster: + +```bash +helm template ... | kapp deploy -a app1 -f- -c -y +``` + +## PV labeling controller + +If you want to have better visibility into which persistent volumes (PVs) are associated with persistent volume claims (PVCs), you can install [https://github.com/k14s/pv-labeling-controller](https://github.com/k14s/pv-labeling-controller) so that it copies several kapp applied labels to associated PVs. Once that's done you will see PVs in `kapp inspect` output. + diff --git a/site/content/kapp/docs/v0.53.0/merge-method.md b/site/content/kapp/docs/v0.53.0/merge-method.md new file mode 100644 index 000000000..b19c51b09 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/merge-method.md @@ -0,0 +1,27 @@ +--- +aliases: [/kapp/docs/latest/merge-method] +title: "Resource Merge Method" +--- + +## Why not basic 3 way merge? + +kapp explicitly decided to _not_ do basic 3 way merge, and instead allow the user to specify how to resolve "conflicts". Here is our thinking: + +- you as an operator have a set of files (input files given to kapp via -f) which describe desired configuration +- cluster has resources that need to be converged to whatever input files specify, with one exception: in some cases, cluster is the source of truth for certain information (but not most) and should keep that state on resources (common examples: some annotation on Deployment, clusterIP on Service, etc.) + +Given information above there are multiple ways to converge: + +- make assumptions about how to merge things (basic 3 way merge, what kubectl and helm does afaik) +- be explicit about how to merge things (kapp with rebase rules) +- or, just override + +Overriding is not really an option as it removes potentially important cluster changes (e.g. removes replicas value as scaled by HPA). + +Regarding explicit vs implicit: we decided to go with the explicit option. kapp allows users to add [rebase rules](config.md#rebaserules) to specify exactly which information to retain from existing resources. That gives control to the user to decide what's important to be kept based on cluster state and what's not. This method ensures that there are no _surprising_ changes left in the cluster (if basic 3 way merge was used, then user cannot confidently know how final resource will look like; ... imagine if you had a field `allowUnauthenticatedRequests: true` in some resource that someone flipped on in your cluster, and your configs never specified it; it would not be removed unless you decide to also specify this field in your configs). + +kapp comes with some common k8s rebase rules. you can see them via `kapp deploy-config`. + +tldr: kapp takes user provided config as the only source of truth, but also allows to explicitly specify that certain fields are cluster controlled. This method guarantees that clusters don't drift, which is better than what basic 3 way merge provides. + +Originally answered [here](https://github.com/vmware-tanzu/carvel-kapp/issues/58#issuecomment-559214883). diff --git a/site/content/kapp/docs/v0.53.0/rbac.md b/site/content/kapp/docs/v0.53.0/rbac.md new file mode 100644 index 000000000..7eb89bd96 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/rbac.md @@ -0,0 +1,70 @@ +--- +aliases: [/kapp/docs/latest/rbac] +title: Permissions +--- + +## Running kapp under restricted permissions + +In a multi-tenant Kubernetes cluster, user's actions may be limited to one or more namespaces via `Role` and `RoleBinding` configuration. + +Following setup is currently expected by kapp (v0.10.0+): + +- [required] kapp requires list/get/create/update/delete for `v1/ConfigMap` in [state namespace](state-namespace.md) so that it can store record of application and deployment history. +- [optional] kapp requires one `ClusterRole` rule: listing of namespaces. This requirement is necessary for kapp to find all namespaces so that it can search in each namespace resources that belong to a particular app (via a label). As of v0.11.0+, kapp will fallback to only [state namespace](state-namespace.md) if it is forbidden to list all namespaces. +- otherwise, kapp does _not_ require permissions to resource types that are not used in deployed configuration. In other words, if you are not deploying `Job` resource then kapp does not need any permissions for `Job`. Note that some resources are "cluster" created (e.g. `Pods` are created by k8s deployment controller when `Deployment` resource is created) hence users may not see all app associated resources in `kapp inspect` command if they are restricted (this could be advantageous and disadvantegeous in different setups). + +Please reach out to us in #carvel channel in k8s slack (linked at the bottom of the page) if current kapp permissions model isn't compatible with your use cases. We are eager to learn about your setup and potentially improve kapp. + +Example of `Namespace` listing permission needed by kapp: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kapp-restricted-cr +rules: +- apiGroups: [""] + resources: ["namespaces"] + verbs: ["list"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: kapp-restricted-cr-binding +subjects: +- kind: ServiceAccount + name: # ??? + namespace: # ??? (some tenant ns) +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: kapp-restricted-cr +``` + +Example of `ConfigMap` permissions needed by kapp: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: kapp-restricted-role + namespace: # ??? (some tenant ns) +rules: +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["list", "get", "create", "update", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: kapp-restricted-role-binding + namespace: # ??? (some tenant ns) +subjects: +- kind: ServiceAccount + name: # ??? + namespace: # ??? (some tenant ns) +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: kapp-restricted-role +``` diff --git a/site/content/kapp/docs/v0.53.0/rebase-pvc.md b/site/content/kapp/docs/v0.53.0/rebase-pvc.md new file mode 100644 index 000000000..13e9e6383 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/rebase-pvc.md @@ -0,0 +1,102 @@ +--- +aliases: [/kapp/docs/latest/rebase-pvc] +title: PersistentVolumeClaim rebase +--- +## PersistentVolumeClaim rebase + +Here is an example on how to use custom `rebaseRules` to "prefer" server chosen value for several annotations added by PVC controller (in other words, cluster owned fields), instead of removing them based on given configuration. + +Let's deploy via `kapp deploy -a test -f config.yml -c` with following configuration `config.yml`: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: mysqlclaim +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi +``` + +Without additional rebase rules following diff will be presented upon next deploy, stating that several annotations will be removed (since they were not present in the initial configuration): + +```bash +$ kapp deploy -a test -f config.yml -c + +Target cluster 'https://x.x.x.x' (nodes: gke-dk-jan-9-default-pool-a218b1c9-55sl, 3+) + +--- update persistentvolumeclaim/mysqlclaim (v1) namespace: default + ... + 2, 2 metadata: + 3 - annotations: + 4 - pv.kubernetes.io/bind-completed: "yes" + 5 - pv.kubernetes.io/bound-by-controller: "yes" + 6 - volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd + 7, 3 creationTimestamp: "2020-03-08T22:17:29Z" + 8, 4 finalizers: + ... + 24, 20 storageClassName: standard + 25 - volumeMode: Filesystem + 26, 21 volumeName: pvc-1be63b2b-20de-429c-863a-9e7eb062f5d3 + 27, 22 status: + +Changes + +Namespace Name Kind Conds. Age Op Wait to Rs Ri +default mysqlclaim PersistentVolumeClaim - 43s update reconcile ok - + +Op: 0 create, 0 delete, 1 update, 0 noop, 0 exists +Wait to: 1 reconcile, 0 delete, 0 noop + +Continue? [yN]: +``` + +To let kapp know that these annotations should be copied from the live resource copy, we can augment deploys with following configuration `kapp-config.yml`: + +```yaml +--- +apiVersion: kapp.k14s.io/v1alpha1 +kind: Config + +rebaseRules: +- path: [metadata, annotations, pv.kubernetes.io/bind-completed] + type: copy + sources: [new, existing] + resourceMatchers: &pvcs + - apiVersionKindMatcher: + apiVersion: v1 + kind: PersistentVolumeClaim + +- path: [metadata, annotations, pv.kubernetes.io/bound-by-controller] + type: copy + sources: [new, existing] + resourceMatchers: *pvcs + +- path: [metadata, annotations, volume.beta.kubernetes.io/storage-provisioner] + type: copy + sources: [new, existing] + resourceMatchers: *pvcs + +- path: [spec, volumeMode] + type: copy + sources: [new, existing] + resourceMatchers: *pvcs +``` + +```bash +$ kapp deploy -a test -f config.yml -f rules.yml -c + +Target cluster 'https://x.x.x.x' (nodes: gke-dk-jan-9-default-pool-a218b1c9-55sl, 3+) + +Changes + +Namespace Name Kind Conds. Age Op Wait to Rs Ri + +Op: 0 create, 0 delete, 0 update, 0 noop, 0 exists +Wait to: 0 reconcile, 0 delete, 0 noop + +Succeeded +``` diff --git a/site/content/kapp/docs/v0.53.0/security.md b/site/content/kapp/docs/v0.53.0/security.md new file mode 100644 index 000000000..6b16ba957 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/security.md @@ -0,0 +1,8 @@ +--- +aliases: [/kapp/docs/latest/security] +title: Security +--- + +## Vulnerability Disclosure + +If you believe you have found a security issue in `kapp`, please privately and responsibly disclose it by following the directions in our [security policy](/shared/docs/latest/security-policy). diff --git a/site/content/kapp/docs/v0.53.0/state-namespace.md b/site/content/kapp/docs/v0.53.0/state-namespace.md new file mode 100644 index 000000000..1c278ca29 --- /dev/null +++ b/site/content/kapp/docs/v0.53.0/state-namespace.md @@ -0,0 +1,53 @@ +--- +aliases: [/kapp/docs/latest/state-namespace] +title: Namespace for State Storage +--- + +## Overview + +To show list of deployed applications (via `kapp ls`), kapp manages metadata `ConfigMap` for each saved application. Each metadata `ConfigMap` contains generated label used to label all application resources. Additionally kapp creates `ConfigMap` per each deploy to record deployment history (seen via `kapp app-change list -a app1`). + +`-n` (`--namespace`) flag allows to control which namespace is used for finding and storing metadata `ConfigMaps`. If namespace is not explicitly specified your current namespace is selected from kube config (typically `~/.kube/config`). + +There are currently two approaches to deciding which namespace to use for storing metadata `ConfigMaps`: + +- for each application, keep metadata `ConfigMap` and app resources themselves in the same namespace. That namespace will have to be created before running `kapp deploy` since kapp will first want to create a `ConfigMap` representing application. + + ```bash + $ kubectl create ns app1 + $ kapp deploy -n app1 -f config.yml + $ kapp ls -n app1 + ``` + +- create a dedicated namespace to store metadata `ConfigMaps` representing apps, and have kapp create `Namespace` resources for applications from their config. With this approach namespace management (creation and deletion) is tied to a particular app configuration which makes it a bit easier to track `Namespaces` via configuration. + + ```bash + $ kubectl create ns apps + $ kapp deploy -n apps -f app1/config.yml + $ kapp deploy -n apps -f app2/config.yml + $ kapp ls -n apps + ``` + + for example, `app1/config.yml` may look like this: + + ```yaml + apiVersion: v1 + kind: Namespace + metadata: + name: app1 + --- + apiVersion: apps/v1 + kind: Deployment + metadata: + name: dep + namespace: app1 + ... + ``` + +Note: It's currently not possible to have kapp place app `ConfigMap` resource into `Namespace` that kapp creates for that application. + +## App Changes + +As mentioned above, app changes (stored as `ConfigMap`) are stored in state namespace. App changes do not store any information necessary for kapp to operate, but rather act as informational records. There is currently no cap on how many app changes are kept per app. + +To remove older app changes, use `kapp app-change gc -a app1` which by default will keep 200 most recent changes (as of v0.12.0). Alternatively use `--app-changes-max-to-keep` flag on the `deploy` command to control number of changes kept at the time of deploy. diff --git a/site/data/kapp/docs/kapp-v0-53-0-toc.yml b/site/data/kapp/docs/kapp-v0-53-0-toc.yml new file mode 100644 index 000000000..b12678ee6 --- /dev/null +++ b/site/data/kapp/docs/kapp-v0-53-0-toc.yml @@ -0,0 +1,53 @@ +toc: + - title: Introduction + subfolderitems: + - page: About kapp + url: / + - page: Install + url: /install + - page: Applications + url: /apps + - title: Deploy command + subfolderitems: + - page: Diff stage + url: /diff + - page: Apply stage + url: /apply + - page: Apply Ordering + url: /apply-ordering + - page: Apply Waiting + url: /apply-waiting + - title: Reference + subfolderitems: + - page: Configuration + url: /config + - page: Permissions + url: /rbac + - page: Namespace for State Storage + url: /state-namespace + - page: Cheatsheet + url: /cheatsheet + - page: Dangerous Flags + url: /dangerous-flags + - page: Configmap Migration (experimental) + url: /configmap-migration + - title: Integrations + subfolderitems: + - page: Integrating With Other Tools + url: /integrating-with-other-tools + - page: GitOps + url: /gitops + - title: FAQ + subfolderitems: + - page: FAQ + url: /faq + - page: Resource Merge Method + url: /merge-method + - page: Code of Conduct + shared_url: /code-of-conduct + - page: Contributing + shared_url: /contributing + - page: Carvel Development Guidelines + shared_url: /development_guidelines + - page: Security + shared_url: /security-policy diff --git a/site/data/kapp/docs/toc-mapping.yml b/site/data/kapp/docs/toc-mapping.yml index aea42dbe5..85cdf691a 100644 --- a/site/data/kapp/docs/toc-mapping.yml +++ b/site/data/kapp/docs/toc-mapping.yml @@ -8,3 +8,4 @@ v0.49.0: kapp-v0-49-0-toc v0.50.0: kapp-v0-50-0-toc v0.51.0: kapp-v0-51-0-toc v0.52.0: kapp-v0-52-0-toc +v0.53.0: kapp-v0-53-0-toc From 3eb9d576c1f55062033a484c22951b9d9469cfe2 Mon Sep 17 00:00:00 2001 From: carvel-bot Date: Fri, 30 Sep 2022 16:06:57 +0000 Subject: [PATCH 32/35] chore: autopublish 2022-09-30T16:06:57Z --- site/static/install.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/site/static/install.sh b/site/static/install.sh index bc16d2556..2b24f63a2 100755 --- a/site/static/install.sh +++ b/site/static/install.sh @@ -24,7 +24,7 @@ install() { binary_type=darwin-amd64 ytt_checksum=579012ac80cc0d55c3a6dde2dfc0ff5bf8a4f74c775295be99faf691cc18595e - imgpkg_checksum=f6044cd0134fe94ff1117b15545e8b6063b04e6bc602eace79faecfbfa821348 + imgpkg_checksum=7e5a95b48f2f8f674f7f5d1d0207cda6c6813734fa5bb0ed7fa6f5522ae0b5bb kbld_checksum=f0fa574d1dd1c4dcd6a763d38e746a918477ac61a6cd52e8e9e1bba6714259c9 kapp_checksum=2b466b9f8bbc8719334cadf917769b27affc10c95c9ded3e76be283cfd3d4721 kwt_checksum=555d50d5bed601c2e91f7444b3f44fdc424d721d7da72955725a97f3860e2517 @@ -34,7 +34,7 @@ install() { binary_type=linux-amd64 ytt_checksum=29e647beeacbcc2be5f2f481e405c73bcd6d7563bd229ff924a7997b6f2edd5f - imgpkg_checksum=bfc210872c18fd45e8012aadba27588f51c53cb237e397439abe5fe999e3a6fd + imgpkg_checksum=10a8327490ca3dbfa3d00f90ae442838d364e4d7d158786e94aa1ff0438dab27 kbld_checksum=1c3f0e4171063eef70cdabf1730d3557af992aeafb423755e71e9d5f1aea2c9b kapp_checksum=c2c7381a152216c8600408b4dee26aee48390f1e23d8ef209af8d9eb1edd60fc kwt_checksum=92a1f18be6a8dca15b7537f4cc666713b556630c20c9246b335931a9379196a0 @@ -53,11 +53,11 @@ install() { echo "Installed ${dst_dir}/ytt v0.43.0" echo "Installing imgpkg..." - $dl_bin github.com/vmware-tanzu/carvel-imgpkg/releases/download/v0.32.0/imgpkg-${binary_type} > /tmp/imgpkg + $dl_bin github.com/vmware-tanzu/carvel-imgpkg/releases/download/v0.33.0/imgpkg-${binary_type} > /tmp/imgpkg echo "${imgpkg_checksum} /tmp/imgpkg" | shasum -c - mv /tmp/imgpkg ${dst_dir}/imgpkg chmod +x ${dst_dir}/imgpkg - echo "Installed ${dst_dir}/imgpkg v0.32.0" + echo "Installed ${dst_dir}/imgpkg v0.33.0" echo "Installing kbld..." $dl_bin github.com/vmware-tanzu/carvel-kbld/releases/download/v0.35.0/kbld-${binary_type} > /tmp/kbld From 8a8340cc98f2851f5fb8830818f1c3857b31180f Mon Sep 17 00:00:00 2001 From: Joao Pereira Date: Fri, 30 Sep 2022 11:10:05 -0500 Subject: [PATCH 33/35] Bump imgpkg to 0.33.0 Signed-off-by: Joao Pereira --- site/config.yaml | 5 +- site/content/imgpkg/docs/v0.32.0/_index.md | 2 +- .../docs/v0.32.0/air-gapped-workflow.md | 2 +- site/content/imgpkg/docs/v0.32.0/auth.md | 2 +- .../docs/v0.32.0/automation-workflow.md | 2 +- .../imgpkg/docs/v0.32.0/basic-workflow.md | 2 +- .../imgpkg/docs/v0.32.0/ca-certs-windows.md | 2 +- site/content/imgpkg/docs/v0.32.0/commands.md | 2 +- site/content/imgpkg/docs/v0.32.0/debugging.md | 2 +- .../imgpkg/docs/v0.32.0/faq-generic.md | 2 +- site/content/imgpkg/docs/v0.32.0/install.md | 2 +- site/content/imgpkg/docs/v0.32.0/proxy.md | 2 +- site/content/imgpkg/docs/v0.32.0/resources.md | 2 +- site/content/imgpkg/docs/v0.32.0/security.md | 2 +- .../v0.32.0/working-directly-with-images.md | 2 +- site/content/imgpkg/docs/v0.33.0/_index.md | 21 +++ .../docs/v0.33.0/air-gapped-workflow.md | 154 ++++++++++++++++ site/content/imgpkg/docs/v0.33.0/auth.md | 166 +++++++++++++++++ .../docs/v0.33.0/automation-workflow.md | 141 +++++++++++++++ .../imgpkg/docs/v0.33.0/basic-workflow.md | 150 ++++++++++++++++ .../imgpkg/docs/v0.33.0/ca-certs-windows.md | 28 +++ site/content/imgpkg/docs/v0.33.0/commands.md | 169 ++++++++++++++++++ site/content/imgpkg/docs/v0.33.0/debugging.md | 46 +++++ .../imgpkg/docs/v0.33.0/faq-generic.md | 30 ++++ site/content/imgpkg/docs/v0.33.0/install.md | 58 ++++++ site/content/imgpkg/docs/v0.33.0/proxy.md | 33 ++++ site/content/imgpkg/docs/v0.33.0/resources.md | 145 +++++++++++++++ site/content/imgpkg/docs/v0.33.0/security.md | 8 + .../v0.33.0/working-directly-with-images.md | 8 + site/data/imgpkg/docs/imgpkg-v0-33-0-toc.yml | 43 +++++ site/data/imgpkg/docs/toc-mapping.yml | 1 + 31 files changed, 1218 insertions(+), 16 deletions(-) create mode 100644 site/content/imgpkg/docs/v0.33.0/_index.md create mode 100644 site/content/imgpkg/docs/v0.33.0/air-gapped-workflow.md create mode 100644 site/content/imgpkg/docs/v0.33.0/auth.md create mode 100644 site/content/imgpkg/docs/v0.33.0/automation-workflow.md create mode 100644 site/content/imgpkg/docs/v0.33.0/basic-workflow.md create mode 100644 site/content/imgpkg/docs/v0.33.0/ca-certs-windows.md create mode 100644 site/content/imgpkg/docs/v0.33.0/commands.md create mode 100644 site/content/imgpkg/docs/v0.33.0/debugging.md create mode 100644 site/content/imgpkg/docs/v0.33.0/faq-generic.md create mode 100644 site/content/imgpkg/docs/v0.33.0/install.md create mode 100644 site/content/imgpkg/docs/v0.33.0/proxy.md create mode 100644 site/content/imgpkg/docs/v0.33.0/resources.md create mode 100644 site/content/imgpkg/docs/v0.33.0/security.md create mode 100644 site/content/imgpkg/docs/v0.33.0/working-directly-with-images.md create mode 100644 site/data/imgpkg/docs/imgpkg-v0-33-0-toc.yml diff --git a/site/config.yaml b/site/config.yaml index 329632655..cc51e95ca 100644 --- a/site/config.yaml +++ b/site/config.yaml @@ -82,15 +82,16 @@ params: name: imgpkg short_name: imgpkg root_link: /imgpkg/ - latest_docs_link: /imgpkg/docs/v0.32.0/ + latest_docs_link: /imgpkg/docs/v0.33.0/ github_url: https://github.com/vmware-tanzu/carvel-imgpkg search: true search_index_name: carvel-imgpkg search_api_key: a38560864c2e9128ae57d5734df438ff versioning: true - version_latest: v0.32.0 + version_latest: v0.33.0 versions: - develop + - v0.33.0 - v0.32.0 - v0.31.0 - v0.30.0 diff --git a/site/content/imgpkg/docs/v0.32.0/_index.md b/site/content/imgpkg/docs/v0.32.0/_index.md index 04c1d1380..f40a1cc09 100644 --- a/site/content/imgpkg/docs/v0.32.0/_index.md +++ b/site/content/imgpkg/docs/v0.32.0/_index.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/] + title: "About imgpkg" toc: "false" cascade: diff --git a/site/content/imgpkg/docs/v0.32.0/air-gapped-workflow.md b/site/content/imgpkg/docs/v0.32.0/air-gapped-workflow.md index a3d589126..146fff0b7 100644 --- a/site/content/imgpkg/docs/v0.32.0/air-gapped-workflow.md +++ b/site/content/imgpkg/docs/v0.32.0/air-gapped-workflow.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/air-gapped-workflow] + title: Air-gapped Workflow --- diff --git a/site/content/imgpkg/docs/v0.32.0/auth.md b/site/content/imgpkg/docs/v0.32.0/auth.md index df015c0a0..f4a10756b 100644 --- a/site/content/imgpkg/docs/v0.32.0/auth.md +++ b/site/content/imgpkg/docs/v0.32.0/auth.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/auth] + title: Authentication --- diff --git a/site/content/imgpkg/docs/v0.32.0/automation-workflow.md b/site/content/imgpkg/docs/v0.32.0/automation-workflow.md index 0a6c87d5b..3884ad3e8 100644 --- a/site/content/imgpkg/docs/v0.32.0/automation-workflow.md +++ b/site/content/imgpkg/docs/v0.32.0/automation-workflow.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/automation-workflow] + title: Automation Workflow --- diff --git a/site/content/imgpkg/docs/v0.32.0/basic-workflow.md b/site/content/imgpkg/docs/v0.32.0/basic-workflow.md index f752ade58..c1f00c072 100644 --- a/site/content/imgpkg/docs/v0.32.0/basic-workflow.md +++ b/site/content/imgpkg/docs/v0.32.0/basic-workflow.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/basic-workflow] + title: Basic Workflow --- diff --git a/site/content/imgpkg/docs/v0.32.0/ca-certs-windows.md b/site/content/imgpkg/docs/v0.32.0/ca-certs-windows.md index 2f0404248..950141a4e 100644 --- a/site/content/imgpkg/docs/v0.32.0/ca-certs-windows.md +++ b/site/content/imgpkg/docs/v0.32.0/ca-certs-windows.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/ca-certs-windows] + title: CA Certs on Windows --- diff --git a/site/content/imgpkg/docs/v0.32.0/commands.md b/site/content/imgpkg/docs/v0.32.0/commands.md index 3589ec7f3..b62a635d9 100644 --- a/site/content/imgpkg/docs/v0.32.0/commands.md +++ b/site/content/imgpkg/docs/v0.32.0/commands.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/commands] + title: Commands --- diff --git a/site/content/imgpkg/docs/v0.32.0/debugging.md b/site/content/imgpkg/docs/v0.32.0/debugging.md index 9000ae487..e669f95e8 100644 --- a/site/content/imgpkg/docs/v0.32.0/debugging.md +++ b/site/content/imgpkg/docs/v0.32.0/debugging.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/debugging] + title: Debugging --- diff --git a/site/content/imgpkg/docs/v0.32.0/faq-generic.md b/site/content/imgpkg/docs/v0.32.0/faq-generic.md index 879d4bd09..5ff292950 100644 --- a/site/content/imgpkg/docs/v0.32.0/faq-generic.md +++ b/site/content/imgpkg/docs/v0.32.0/faq-generic.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/faq-generic] + title: FAQ --- diff --git a/site/content/imgpkg/docs/v0.32.0/install.md b/site/content/imgpkg/docs/v0.32.0/install.md index 058244d52..cb14ae645 100644 --- a/site/content/imgpkg/docs/v0.32.0/install.md +++ b/site/content/imgpkg/docs/v0.32.0/install.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/install] + title: Install --- diff --git a/site/content/imgpkg/docs/v0.32.0/proxy.md b/site/content/imgpkg/docs/v0.32.0/proxy.md index 0998e7dc2..cc509f507 100644 --- a/site/content/imgpkg/docs/v0.32.0/proxy.md +++ b/site/content/imgpkg/docs/v0.32.0/proxy.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/proxy] + title: Proxy --- diff --git a/site/content/imgpkg/docs/v0.32.0/resources.md b/site/content/imgpkg/docs/v0.32.0/resources.md index 6a0300834..7ae340dca 100644 --- a/site/content/imgpkg/docs/v0.32.0/resources.md +++ b/site/content/imgpkg/docs/v0.32.0/resources.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/resources] + title: Resources --- diff --git a/site/content/imgpkg/docs/v0.32.0/security.md b/site/content/imgpkg/docs/v0.32.0/security.md index ca69db28d..c70b75d5e 100644 --- a/site/content/imgpkg/docs/v0.32.0/security.md +++ b/site/content/imgpkg/docs/v0.32.0/security.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/security] + title: Security --- diff --git a/site/content/imgpkg/docs/v0.32.0/working-directly-with-images.md b/site/content/imgpkg/docs/v0.32.0/working-directly-with-images.md index cdac04918..21dc3ac34 100644 --- a/site/content/imgpkg/docs/v0.32.0/working-directly-with-images.md +++ b/site/content/imgpkg/docs/v0.32.0/working-directly-with-images.md @@ -1,5 +1,5 @@ --- -aliases: [/imgpkg/docs/latest/working-directly-with-images] + title: Working directly with images --- diff --git a/site/content/imgpkg/docs/v0.33.0/_index.md b/site/content/imgpkg/docs/v0.33.0/_index.md new file mode 100644 index 000000000..13d24299d --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/_index.md @@ -0,0 +1,21 @@ +--- +aliases: [/imgpkg/docs/latest/] +title: "About imgpkg" +toc: "false" +cascade: + version: v0.33.0 + toc: "true" + type: docs + layout: docs +--- + +`imgpkg` is a tool that allows users to store a set of arbitrary files as an OCI image. One of the driving use cases is to store Kubernetes configuration (plain YAML, ytt templates, Helm templates, etc.) in OCI registry as an image. + +`imgpkg`'s primary concept is a [bundle](resources.md#bundle), which is an OCI image that holds 0+ arbitrary files and 0+ references to dependent OCI images (which *may* also be [bundles](resources.md/#nested-bundle)). With this concept, `imgpkg` is able to copy bundles and their dependent images across registries (both online and offline). + +![Bundle diagram](/images/imgpkg/bundle-diagram.png) + +## Workflows + +- [Basic Workflow](basic-workflow.md) shows how to create, push, and pull bundles with a simple Kubernetes application +- [Air-gapped Workflow](air-gapped-workflow.md) shows how to copy bundles from one registry to another, to enable running Kubernetes applications without relying on external (public) registries diff --git a/site/content/imgpkg/docs/v0.33.0/air-gapped-workflow.md b/site/content/imgpkg/docs/v0.33.0/air-gapped-workflow.md new file mode 100644 index 000000000..a3d589126 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/air-gapped-workflow.md @@ -0,0 +1,154 @@ +--- +aliases: [/imgpkg/docs/latest/air-gapped-workflow] +title: Air-gapped Workflow +--- + +## Scenario + +You want to ensure Kubernetes application does not rely on images from external registries when deployed. + +This scenario _also_ applies when trying to ensure that all images are consolidated into a single registry, even if that registry is not air-gapped. + +## Prerequisites + +To complete this workflow you will need access to an OCI registry like Docker Hub, and optionally, +a Kubernetes cluster. (If you would like to use a local registry and Kubernetes cluster, try using [Kind](https://kind.sigs.k8s.io/docs/user/local-registry/)) + +If you would like to deploy the results of this scenario to your Kubernetes cluster, you will additionally need [`kbld`](/kbld) and kubectl. + +If any of your bundles contain [non-distributable layers](commands.md#non-distributable-or-foreign-layers) you will need to include +the `--include-non-distributable-layers` flag to each copy command in the examples provided. + +--- +## Step 1: Finding bundle in source registry + +If you have already pushed a bundle to the registry, continue to the next step. + +If you are trying to bundle your own or third-part software, you will need to create a bundle. Refer to basic workflow's ["Step 1: Creating the bundle" and "Step 2: Pushing the bundle to registry"](basic-workflow.md#step-1-creating-the-bundle). + +--- +## Step 2: Two methods of copying bundles + +You have two options how to transfer bundle from one registry to another: + +- Option 1: From a common location connected to both registries. This option is more efficient because only changed image layers will be transfered between registries. +- Option 2: With intermediate tarball. This option works best when registries have no common network access. + +### Option 1: From a location connected to both registries + +1. Get to a location that can access both registries + + This may be a server that has access to both internal and external networks. If there is no such location, you will have to use "Option 2" below. + +1. [Authenticate](auth.md) with both source, and destination registries + +1. Run following command to copy bundle from one registry to another: + + ```bash-plain + $ imgpkg copy -b index.docker.io/user1/simple-app-bundle:v1.0.0 --to-repo registry.corp.com/apps/simple-app-bundle + + copy | exporting 2 images... + copy | will export index.docker.io/user1/simple-app-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + copy | will export index.docker.io/user1/simple-app-bundle@sha256:70225df0a05137ac385c95eb69f89ded3e7ef3a0c34db43d7274fd9eba3705bb + copy | exported 2 images + copy | importing 2 images... + copy | importing index.docker.io/user1/simple-app-bundle@sha256:70225df0a05137ac385c95eb69f89ded3e7ef3a0c34db43d7274fd9eba3705bb + -> registry.corp.com/apps/simple-app-bundle@sha256:70225df0a05137ac385c95eb69f89ded3e7ef3a0c34db43d7274fd9eba3705bb... + copy | importing index.docker.io/user1/simple-app-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + -> registry.corp.com/apps/simple-app-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0... + copy | imported 2 images + Succeeded + ``` + + The bundle, and all images referenced in the bundle, are copied to the destination registry. + + Flags used in the command: + * `-b` (`--bundle`) indicates the bundle location in the source registry + * `--to-repo` indicates the registry where the bundle and associated images should be copied to + +### Option 2: With intermediate tarball + +1. Get to a location that can access source registry + +1. [Authenticate with the source registry](auth.md) + +1. Save the bundle to a tarball + + ```bash-plain + $ imgpkg copy -b index.docker.io/user1/simple-app-bundle:v1.0.0 --to-tar /tmp/my-image.tar + + copy | exporting 2 images... + copy | will export index.docker.io/user1/simple-app-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + copy | will export index.docker.io/user1/simple-app-bundle@sha256:70225df0a05137ac385c95eb69f89ded3e7ef3a0c34db43d7274fd9eba3705bb + copy | exported 2 images + copy | writing layers... + copy | done: file 'manifest.json' (13.71µs) + copy | done: file 'sha256-233f1d0dbdc8cf675af965df8639b0dfd4ef7542dfc9fcfd03bfc45c570b0e4d.tar.gz' (47.616µs) + copy | done: file 'sha256-8ece9ac45f2b7228b2ed95e9f407b4f0dc2ac74f93c62ff1156f24c53042ba54.tar.gz' (43.204905ms) + Succeeded + ``` + + Flags used in the command: + * `-b` (`--bundle`) indicates the bundle location in the source registry + * `--to-tar` indicates the local location to write a tar file containing the bundle assets + +1. Transfer the local tarball `/tmp/my-image.tar` to a location with access to the destination registry + +1. [Authenticate with the destination registry](auth.md) + +1. Import the bundle from your tarball to the destination registry: + + ```bash-plain + $ imgpkg copy --tar /tmp/my-image.tar --to-repo registry.corp.com/apps/simple-app-bundle + + copy | importing 2 images... + copy | importing index.docker.io/user1/simple-app-bundle@sha256:70225df0a05137ac385c95eb69f89ded3e7ef3a0c34db43d7274fd9eba3705bb -> registry.corp.com/apps/simple-app-bundle@sha256:70225df0a05137ac385c95eb69f89ded3e7ef3a0c34db43d7274fd9eba3705bb... + copy | importing index.docker.io/user1/simple-app-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> registry.corp.com/apps/simple-app-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0... + copy | imported 2 images + Succeeded + ``` + + The bundle, and all images referenced in the bundle, are copied to the destination registry. + + Flags used in the command: + * `--tar` indicates the path to a tar file containing the assets to be copied to a registry + * `--to-repo` indicates destination bundle location in the registry + +--- +## Step 3: Pulling bundle from destination registry + +1. [Authenticate with the destination registry](auth.md) + +1. Pull the bundle from the destination registry: + + ```bash-plain + $ imgpkg pull -b registry.corp.com/apps/simple-app-bundle:v1.0.0 -o /tmp/bundle + + Pulling image 'registry.corp.com/apps/simple-app-bundle@sha256:70225df0a05137ac385c95eb69f89ded3e7ef3a0c34db43d7274fd9eba3705bb' + Extracting layer 'sha256:233f1d0dbdc8cf675af965df8639b0dfd4ef7542dfc9fcfd03bfc45c570b0e4d' (1/1) + Locating image lock file images... + All images found in bundle repo; updating lock file: /tmp/bundle/.imgpkg/images.yml + + Succeeded + ``` + + Flags used in the command: + * `-b` (`--bundle`) indicates to pull a particular bundle from a registry + * `-o` (`--output`) indicates the local folder where the bundle will be unpacked + + Note that the `.imgpkg/images.yml` file was updated with the destination registry locations of the images. This happened because, in the prior step, the images referenced by the bundle were copied into the destination registry. + + ```bash-plain + $ cat /tmp/bundle/.imgpkg/images.yml + apiVersion: imgpkg.carvel.dev/v1alpha1 + kind: ImagesLock + images: + - image: registry.corp.com/apps/simple-app-bundle@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + annotations: + kbld.carvel.dev/id: docker.io/dkalinin/k8s-simple-app + ``` + +--- +## Step 4: Use pulled bundle contents + +Regardless which location the bundle is downloaded from, source registry or destination registry, use of the pulled bundle contents remains the same. Continue with ["Step 4: Use pulled bundle contents"](basic-workflow.md#step-4-use-pulled-bundle-contents) in the basic workflow. diff --git a/site/content/imgpkg/docs/v0.33.0/auth.md b/site/content/imgpkg/docs/v0.33.0/auth.md new file mode 100644 index 000000000..df015c0a0 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/auth.md @@ -0,0 +1,166 @@ +--- +aliases: [/imgpkg/docs/latest/auth] +title: Authentication +--- + +# Ordering + +imgpkg has multiple ways to provide authentication details to registries. + +The order at which imgpkg chooses which authentication details to use is the following: + +1. [Via Environment Variables](#via-environment-variables) +1. [Via IaaS](#via-iaas) +1. [Via Command Flags](#via-command-flags) +1. [Via Docker Config](#via-docker-config) + +## Via Environment Variables + +As of v0.7.0+, `imgpkg` can also use following environment variables: + +- `IMGPKG_REGISTRY_HOSTNAME` to specify registry hostname (e.g. gcr.io, docker.io, https://gcr.io, docker.io/v2/) + - As of v0.18.0+ `IMGPKG_REGISTRY_HOSTNAME` also supports providing glob wildcards. for e.g. `*.*.docker.io` will match `bar.foo.docker.io`. + - Note: if there is overlap between 2 HOSTNAMES, one using globbing and the other not, the HOSTNAME not using globbing will be applied. e.g. `IMGPKG_REGISTRY_HOSTNAME_0=*.docker.io` vs `IMGPKG_REGISTRY_HOSTNAME_1=foo.docker.io` for the image `foo.docker.io/image` will result in auth details from `IMGPKG_REGISTRY_HOSTNAME_1` being used. + - As of v0.18.0+ `IMGPKG_REGISTRY_HOSTNAME` also supports providing the fully qualified repository. for e.g. `gcr.io/repo/image`. +- `IMGPKG_REGISTRY_USERNAME` to specify registry username +- `IMGPKG_REGISTRY_PASSWORD` to specify registry password +- `IMGPKG_REGISTRY_IDENTITY_TOKEN` to authenticate the user and get an access token for the registry via an [oauth2 refresh token grant type](https://docs.docker.com/registry/spec/auth/oauth/). +- `IMGPKG_REGISTRY_REGISTRY_TOKEN` to specify the access token to be used in the Authorization Header as a [Bearer Token](https://docs.docker.com/registry/spec/auth/token/#using-the-bearer-token). + +Since you may need to provide multiple registry credentials, the environment variables above may be specified multiple times with a suffix of 1+ alphanumeric characters, + +e.g. If you had 2 registries you wish to provide authentication credentials for, you would require 2 sets of env variables. + +For Registry #1: + +``` +IMGPKG_REGISTRY_HOSTNAME_0=hostname.for.registry.1 +IMGPKG_REGISTRY_USERNAME_0=username +IMGPKG_REGISTRY_PASSWORD_0=password +``` + +For Registry #2: + +``` +IMGPKG_REGISTRY_HOSTNAME_1=hostname.for.registry.2 +IMGPKG_REGISTRY_IDENTITY_TOKEN_1=token +``` + +When imgpkg interacts with `hostname.for.registry.1`, it will use the env variables with the suffix `_0`. And when interacting with `hostname.for.registry.2`, it will use the env variables with the suffix `_1` + + +Note: Credentials provided via an env variable for a specific registry will take precedence over Command Flags. + +## Via IaaS + +By default, `imgpkg` will **NOT** attempt to authenticate itself via the underlying IaaS: + +To activate this behavior you can set the environment variable `IMGPKG_ACTIVE_KEYCHAINS` with the keychains to the IaaS that you are currently using. + +*Note:* To mimic the old behavior of `imgpkg` set the environment variable as follows `export IMGPKG_ACTIVE_KEYCHAINS=gke,aks,ecr` + +Below is a list of IaaS providers that `imgpkg` can authenticate with: + +- [GCP](https://cloud.google.com/compute/docs/metadata/overview) + + To activate it use `export IMGPKG_ACTIVE_KEYCHAINS=gke` + +- [AWS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) + + To activate it use `export IMGPKG_ACTIVE_KEYCHAINS=ecr` + For more information [check the helper](https://github.com/awslabs/amazon-ecr-credential-helper#configuration) + +- [Azure](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-managed-identities-work-vm) + + To activate it use `export IMGPKG_ACTIVE_KEYCHAINS=aks` + For more information check [this library](https://github.com/chrismellard/docker-credential-acr-env) + +- Github + + To activate use `export IMGPKG_ACTIVE_KEYCHAINS=github` + Requires the environment variable `GITHUB_TOKEN` to be set to connect to ghcr.io + +**Deprecation:** The environment variable `IMGPKG_ENABLE_IAAS_AUTH` can be used only to activate all the keychains. +This behavior will be removed in a future version. + + +## Via Command Flags + +You can explicitly specify credentials via command flags or associated environment variables. See `imgpkg push -h` for further details. + +- `--registry-username` (or `$IMGPKG_USERNAME`) +- `--registry-password` (or `$IMGPKG_PASSWORD`) +- `--registry-token` (or `$IMGPKG_TOKEN`): to specify the access token to be used in the Authorization Header as a [Bearer Token](https://docs.docker.com/registry/spec/auth/token/#using-the-bearer-token). +- `--registry-anon` (or `$IMGPKG_ANON=true`): used for anonymous access (commonly for pulling) + +## Via Docker config + +Even though `imgpkg` commands use registry APIs directly, by default it uses credentials stored in `~/.docker/config.json` which are typically generated via a `docker login` command. + +Example generated `~/.docker/config.json`: + +```json +{ + "auths": { + "https://index.docker.io/v1/": { + "auth": "dXNlcjpwYXNzd29yZA==" + }, + }, + "HttpHeaders": { + "User-Agent": "Docker-Client/18.09.6 (darwin)" + } +} +``` + +where `dXNlcjpwYXNzd29yZA==` is `base64("username:password")`. + +## gcr.io + +- Create a service account with "Storage Admin" permissions for push access + - See [Permissions and Roles](https://cloud.google.com/container-registry/docs/access-control#permissions_and_roles) +- Download a JSON service account key and place it somewhere on filesystem (e.g. `/tmp/key`) + - See [Advanced authentication](https://cloud.google.com/container-registry/docs/advanced-authentication#json_key_file) +- Run `cat /tmp/key | docker login -u _json_key --password-stdin https://gcr.io` to authenticate + +## AWS ECR + +- Create an ECR repository +- Create an IAM user with an ECR policy that allows read/write + - See [Amazon ECR Policies](https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html) +- Run `aws configure` and specify access key ID, secret access key and region + - To install on Ubuntu, run `apt-get install pip3` and `pip3 install awscli` + - See [Installing the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) +- Run `eval $(aws ecr get-login --no-include-email)` to authenticate + - See [get-login command](https://docs.aws.amazon.com/cli/latest/reference/ecr/get-login.html) + +Example ECR policy from [Amazon ECR](https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html): + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ecr:GetAuthorizationToken", + "ecr:BatchCheckLayerAvailability", + "ecr:GetDownloadUrlForLayer", + "ecr:GetRepositoryPolicy", + "ecr:DescribeRepositories", + "ecr:ListImages", + "ecr:DescribeImages", + "ecr:BatchGetImage", + "ecr:InitiateLayerUpload", + "ecr:UploadLayerPart", + "ecr:CompleteLayerUpload", + "ecr:PutImage" + ], + "Resource": "*" + } + ] +} +``` + +## Harbor + +You may have to provide `--registry-ca-cert-path` flag with a path to a CA certificate file for Harbor Registry API. diff --git a/site/content/imgpkg/docs/v0.33.0/automation-workflow.md b/site/content/imgpkg/docs/v0.33.0/automation-workflow.md new file mode 100644 index 000000000..0a6c87d5b --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/automation-workflow.md @@ -0,0 +1,141 @@ +--- +aliases: [/imgpkg/docs/latest/automation-workflow] +title: Automation Workflow +--- + +## Scenario + +When using an automated CI tool you might want to promote a given Bundle between steps of the pipeline + +## Prerequisites + +To complete this workflow you will need access to an OCI registry like Docker Hub. + +### Step 1: Creating the Bundle + +1. Prepare bundle contents + + The [examples/basic-step-1/](https://github.com/vmware-tanzu/carvel-imgpkg/tree/develop/examples/basic-step-1) + directory has a `config.yml` file, which contains a very simple Kubernetes application. Your application may have as + many configuration files as necessary in various formats such as plain YAML, ytt templates, Helm templates, etc. + + In our example `config.yml` includes an image reference to `docker.io/dkalinin/k8s-simple-app`. This reference does + not point to an exact image (via digest) meaning that it may change over time. To ensure we get precisely the bits we + expect, we will lock it down to an exact image next. + +1. Add `.imgpkg/` directory + + [examples/basic-step-2](https://github.com/vmware-tanzu/carvel-imgpkg/tree/develop/examples/basic-step-2) shows what + a `.imgpkg/` directory may look like. It contains: + + - **optional** [bundle.yml](resources.md#bundle-metadata): a file which records informational metadata + - **required** [images.yml](resources.md#imageslock): a file which records image references used by the + configuration + + ```bash-plain + examples/basic-step-2 + ├── .imgpkg + │   ├── bundle.yml + │   └── images.yml + └── config.yml + ``` + + Note that `.imgpkg/images.yml` contains a list of images, each with fully resolved digest reference ( + e.g `index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4...`) and a some additional metadata ( + e.g. `annotations` section). See [ImagesLock configuration](resources.md#imageslock-configuration) for details. + + ```yaml + apiVersion: imgpkg.carvel.dev/v1alpha1 + kind: ImagesLock + images: + - image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + annotations: + kbld.carvel.dev/id: docker.io/dkalinin/k8s-simple-app + ``` + +--- + +### Step 2: Creating the Bundle + +1. [Authenticate with a registry](auth.md) where we will push our bundle + +2. Push the bundle to the registry + + You can push the bundle with our specified contents to an OCI registry using the following command: + + ```bash-plain + $ imgpkg push -b index.docker.io/user1/simple-app-bundle:v1.0.0 -f examples/basic-step-2 --lock-output /tmp/bundle-lock.yml + + dir: . + dir: .imgpkg + file: .imgpkg/bundle.yml + file: .imgpkg/images.yml + file: config.yml + Pushed 'index.docker.io/user1/simple-app-bundle@sha256:5c2dafe3c70c13990190d643c91e9f67b8129b179257674888178868474f6511' + + Succeeded + ``` + + Flags used in the command: + - `-b` (`--bundle`) refers to a location for a bundle within an OCI registry + - `-f` (`--file`) indicates directory contents to include + - `--lock-output` indicates the destination of the [BundleLock](resources.md#bundlelock-configuration) file + +--- + +## Step 3: Promoting the BundleLock file + +Since in the previous step we generated a BundleLock we can promote this file and in the next steps of the pipeline we +can reference it. + +Examples of usage: + +1. Promote the Bundle to a different registry + + ```bash-plain + $ imgpkg copy --lock /tmp/bundle-lock.yml --to-repo production.registry.io/simple-app-bundle + copy | exporting 2 images... + copy | will export index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + copy | will export production.registry.io/simple-app-bundle@sha256:5c2dafe3c70c13990190d643c91e9f67b8129b179257674888178868474f6511 + copy | exported 2 images + copy | importing 2 images... + + 3.56 MiB / 3.57 MiB [========================================================================================================================================================================] 99.68% 8.80 MiB/s 0s + + copy | done uploading images + + Succeeded + ``` + + Flags used in the command: + - `--lock` refers to a location for a BundleLock file + - `--to-repo` indicates the destination Repository where the Bundle is copied to + +2. Download the Bundle contents to disk + + ```bash-plain + $ imgpkg pull --lock /tmp/bundle-lock.yml -o /tmp/simple-app-bundle + + Pulling image 'index.docker.io/user1/simple-app-bundle@sha256:ec3f870e958e404476b9ec67f28c598fa8f00f819b8ae05ee80d51bac9f35f5d' + Extracting layer 'sha256:7906b9650be657359ead106e354f2728e16c8f317e1d87f72b05b5c5ec3d89cc' (1/1) + + Locating image lock file images... + The bundle repo (index.docker.io/user1/simple-app-bundle@sha256:5c2dafe3c70c13990190d643c91e9f67b8129b179257674888178868474f6511) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml) + + Succeeded + ``` + + Flags used in the command: + - `--lock`e`) refers to a location for a BundleLock file + - `-o` (`--output`) indicates the destination directory on your local machine where the bundle contents will be + placed + + Bundle contents will be extracted into `/tmp/simple-app-bundle` directory: + + ```bash-plain + /tmp/simple-app-bundle + ├── .imgpkg + │   ├── bundle.yml + │   └── images.yml + └── config.yml + ``` diff --git a/site/content/imgpkg/docs/v0.33.0/basic-workflow.md b/site/content/imgpkg/docs/v0.33.0/basic-workflow.md new file mode 100644 index 000000000..f752ade58 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/basic-workflow.md @@ -0,0 +1,150 @@ +--- +aliases: [/imgpkg/docs/latest/basic-workflow] +title: Basic Workflow +--- + +## Scenario + +You want to create an immutable artifact containing Kubernetes configuration and images used in that configuration. Later, you want to grab that artifact and deploy it to Kubernetes. + +## Prerequisites + +To complete this workflow you will need access to an OCI registry like Docker Hub, and optionally, +a Kubernetes cluster. (If you would like to use a local registry and Kubernetes cluster, try using [Kind](https://kind.sigs.k8s.io/docs/user/local-registry/)) + +If you would like to deploy the results of this scenario to your Kubernetes cluster, you will additionally need [`kbld`](/kbld) and kubectl. + +## Step 1: Creating the bundle + +1. Prepare bundle contents + + The [examples/basic-step-1/](https://github.com/vmware-tanzu/carvel-imgpkg/tree/develop/examples/basic-step-1) directory has a `config.yml` file, which contains a very simple Kubernetes application. Your application may have as many configuration files as necessary in various formats such as plain YAML, ytt templates, Helm templates, etc. + + In our example `config.yml` includes an image reference to `docker.io/dkalinin/k8s-simple-app`. This reference does not point to an exact image (via digest) meaning that it may change over time. To ensure we get precisely the bits we expect, we will lock it down to an exact image next. + +1. Add `.imgpkg/` directory + + [examples/basic-step-2](https://github.com/vmware-tanzu/carvel-imgpkg/tree/develop/examples/basic-step-2) shows what a `.imgpkg/` directory may look like. It contains: + + - **optional** [bundle.yml](resources.md#bundle-metadata): a file which records informational metadata + - **required** [images.yml](resources.md#imageslock): a file which records image references used by the configuration + + ```bash-plain + examples/basic-step-2 + ├── .imgpkg + │   ├── bundle.yml + │   └── images.yml + └── config.yml + ``` + + Note that `.imgpkg/images.yml` contains a list of images, each with fully resolved digest reference (e.g `index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4...`) and a little bit of additional metadata (e.g. `annotations` section). See [ImagesLock configuration](resources.md#imageslock-configuration) for details. + + ```yaml + apiVersion: imgpkg.carvel.dev/v1alpha1 + kind: ImagesLock + images: + - image: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + annotations: + kbld.carvel.dev/id: docker.io/dkalinin/k8s-simple-app + ``` + + This allows us to record the exact image that will be used by our Kubernetes configuration. We expect that `.imgpkg/images.yml` would be created either manually, or in an automated way. Our recommendation is to use [kbld](/kbld) to generate `.imgpkg/images.yml`: + + ```bash-plain + $ cd examples/basic-bundle/ + + $ kbld -f config.yml --imgpkg-lock-output .imgpkg/images.yml + ``` + +--- +## Step 2: Pushing the bundle to a registry + +1. [Authenticate with a registry](auth.md) where we will push our bundle + +1. Push the bundle to the registry + + You can push the bundle with our specified contents to an OCI registry using the following command: + + ```bash-plain + $ imgpkg push -b index.docker.io/user1/simple-app-bundle:v1.0.0 -f examples/basic-step-2 + + dir: . + dir: .imgpkg + file: .imgpkg/bundle.yml + file: .imgpkg/images.yml + file: config.yml + Pushed 'index.docker.io/user1/simple-app-bundle@sha256:ec3f870e958e404476b9ec67f28c598fa8f00f819b8ae05ee80d51bac9f35f5d' + + Succeeded + ``` + + Flags used in the command: + * `-b` (`--bundle`) refers to a location for a bundle within an OCI registry + * `-f` (`--file`) indicates directory contents to include + +1. The pushed bundle is now available at `index.docker.io/user1/simple-app-bundle:v1.0.0` + +--- +## Step 3: Pulling the bundle to registry + +Now that we have pushed a bundle to a registry, other users can pull it. + +1. [Authenticate with the registry](auth.md) from which we'll pull our bundle + +1. Download the bundle by running the following command: + + ```bash-plain + $ imgpkg pull -b index.docker.io/user1/simple-app-bundle:v1.0.0 -o /tmp/simple-app-bundle + + Pulling image 'index.docker.io/user1/simple-app-bundle@sha256:ec3f870e958e404476b9ec67f28c598fa8f00f819b8ae05ee80d51bac9f35f5d' + Extracting layer 'sha256:7906b9650be657359ead106e354f2728e16c8f317e1d87f72b05b5c5ec3d89cc' (1/1) + Locating image lock file images... + One or more images not found in bundle repo; skipping lock file update + + Succeeded + ``` + + Flags used in the command: + * `-b` (`--bundle`) refers to a location for a bundle within an OCI registry + * `-o` (`--output`) indicates the destination directory on your local machine where the bundle contents will be placed + + Bundle contents will be extracted into `/tmp/simple-app-bundle` directory: + + ```bash-plain + /tmp/simple-app-bundle + ├── .imgpkg + │   ├── bundle.yml + │   └── images.yml + └── config.yml + ``` + + __Note:__ The message `One or more images not found in bundle repo; skipping lock file update` is expected, and indicates that `/tmp/simple-app-bundle/.imgpkg/images.yml` (ImagesLock configuration) was not modified. + + If imgpkg had been able to find all images that were referenced in the [ImagesLock configuration](resources.md#imageslock-configuration) in the registry where bundle is located, then it would update `.imgpkg/images.yml` file to point to the registry-local locations. + + See what happens to the lock file if you run the same pull command after [copying](air-gapped-workflow.md#option-1-from-a-location-connected-to-both-registries) the bundle to another registry! + +--- +## Step 4: Use pulled bundle contents + +1. Now that we have have pulled bundle contents to a local directory, we can deploy Kubernetes configuration: + + Before we apply Kubernetes configuration, let's use [kbld](/kbld) to ensure that Kubernetes configuration uses exact image reference from `.imgpkg/images.yml`. (You can of course use other tools to take advantage of data stored in `.imgpkg/images.yml`). + + ```bash-plain + $ cd /tmp/simple-app-bundle/ + + $ kbld -f ./config.yml -f .imgpkg/images.yml | kubectl apply -f- + + resolve | final: docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + resolve | final: index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 -> index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + + service/simple-app configured + deployment/simple-app configured + ``` + + kbld found `docker.io/dkalinin/k8s-simple-app` in Kubernetes configuration and replaced it with `index.docker.io/dkalinin/k8s-simple-app@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0` before forwarding configuration to kubectl. + +## Next steps + +In this workflow we saw how to publish and download a bundle to distribute a Kubernetes application. Next, follow the [Air-gapped workflow](air-gapped-workflow.md) to see how we can use the `imgpkg copy` command to copy a bundle between registries. diff --git a/site/content/imgpkg/docs/v0.33.0/ca-certs-windows.md b/site/content/imgpkg/docs/v0.33.0/ca-certs-windows.md new file mode 100644 index 000000000..2f0404248 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/ca-certs-windows.md @@ -0,0 +1,28 @@ +--- +aliases: [/imgpkg/docs/latest/ca-certs-windows] +title: CA Certs on Windows +--- + +## Known issue verifying certificates on Windows + +If you are using imgpkg v0.19.0 or earlier, and use imgpkg with a registry over https, you will likely encounter the following error: +``` +imgpkg: Error: Fetching image: + Get "https://some.registry/v2/": x509: certificate signed by unknown authority +``` + +imgpkg v0.20.0+ supports loading Windows root ca certs. Meaning, that imgpkg is able to verify registry certificates signed by a trusted certificate authority! + +## Known issue providing custom ca certificates on Windows + +imgpkg allows specifying the `--registry-ca-cert-path` flag as a way to add custom ca certificates to use when verifying a registry server certificate. + +However, on Windows, the entire set of ca certificates to use during verify is loaded from the flag (Windows root ca store is skipped in this case). +Meaning that if you are targeting multiple registries, and some are signed with a trusted certificate authority and others signed with a custom ca certificate, +both ca certificates will need to be provided. (via the `--registry-ca-cert-path` flag) + +An example workflow: +1. Build a single ca certificate file (containing multiple ca certificates) from a trusted source. e.g. [extract ca certs provided by Mozilla](https://github.com/curl/curl/blob/4d2f8006777d6354d9b62eae38ebd0a0256d0f94/lib/firefox-db2pem.sh) +1. Provide that single ca certificate file to imgpkg. `--registry-ca-cert-path ./mozilla-ca-certs.pem` +1. Provide any additional custom ca certificates to imgpkg. `--registry-ca-cert-path ./dev-registry.pem` + diff --git a/site/content/imgpkg/docs/v0.33.0/commands.md b/site/content/imgpkg/docs/v0.33.0/commands.md new file mode 100644 index 000000000..3589ec7f3 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/commands.md @@ -0,0 +1,169 @@ +--- +aliases: [/imgpkg/docs/latest/commands] +title: Commands +--- + +## Push + +### Overview + +`push` command allows users to create a bundle in the registry from files and/or directories on their local file systems. For example, + +```bash-plain +$ imgpkg push -b index.docker.io/k8slt/sample-bundle -f my-bundle/ +``` + +will push a bundle contents containing in the `my-bundle/` directory to `index.docker.io/k8slt/sample-bundle`. + +Use the `-b`/`--bundle` flag to specify the destination of the push. If the specified destination does not include a tag, the artifact will be pushed with the default tag `:latest`. + +The `-f` flag can be used multiple times to add different files or directories to the bundle. + +### Generating a BundleLock + +`push` command can output a [`BundleLock` configuration](resources.md#bundlelock-configuration) for users that would like a deterministic reference to a pushed bundle. For example, running: + +```bash-plain +$ impgpkg push -b index.docker.io/k8slt/sample-bundle:v0.1.0 -f my-bundle/ --lock-output +/tmp/bundle.lock.yml +``` + +will create `/tmp/bundle.lock.yml` with BundleLock configuration. If another bundle image in the repository is later given the same tag (`v0.1.0`), the BundleLock configuration will continue to provide immutable reference (via digest) to the original pushed bundle. + +--- +## Pull + +### Overview + +After pushing bundles to a registry, users can retrieve them with `imgpkg pull`. For example, + +```bash-plain +$ imgpkg pull -b index.docker.io/k8slt/sample-bundle -o my-bundle/ +``` + +will pull a bundle from `index.docker.io/k8slt/sample-bundle` and extract its contents into the `my-bundle/` directory, which gets created if it does not already exist. + +When pulling a bundle, imgpkg ensures that the referenced images are updated to account for any relocations. It will search for each referenced image by digest in the same repository as the bundle. If all referenced digests are found, imgpkg will update image references in the bundle's [`.imgpkg/images.yml` file](resources.md#imgpkg-directory). If any of the digests are not found in the repository, imgpkg will not update any references. + +### Pulling via lock file + +[BundleLock configuration](resources.md#bundlelock-configuration) can be used as input to the pull command via the `--lock` flag. + +```bash-plain +$ imgpkg pull --lock bundle.lock.yml -o my-bundle/ +``` + +### Pulling nested bundles + +If pulling a bundle that references another bundle (via it's ImagesLock file), in order to *also* pull down the contents of every nested bundle, use the `--recursive` flag. + +```bash-plain +$ imgpkg pull --recursive -b bundle-with-nested-bundles +``` + +Contents of *every* nested bundle are written to the 'parent' bundle's `.imgpkg/bundles` directory, namespaced by the bundle's sha256. + +For e.g. pulling a bundle with a nested bundle having sha of `123` would result in: +``` +parent-bundle-path/.imgpkg/bundles/sha256-123/ +``` + +--- +## Copy + +### Overview + +The `copy` command copies a bundle from a registry to another registry (as long as both registries are accessible from where the command is running): + +```bash-plain +$ imgpkg copy -b index.docker.io/k8slt/sample-bundle --to-repo registry.corp.com/user2/sample-bundle-name +``` + +Alternatively `copy` can copy a bundle between registries which are not both accessible from a single location, by creating an intermediate tarball: + +```bash-plain +$ imgpkg copy -b index.docker.io/k8slt/sample-bundle --to-tar=/Volumes/secure-thumb/bundle.tar +# ... take thumb driver to a different location ... +$ imgpkg copy --tar=/Volumes/secure-thumb/bundle.tar --to-repo registry.corp.com/user2/sample-bundle-name +``` + +In either case, the bundle image and all dependent images are copied to the destination location `registry.corp.com/user2/sample-bundle-name`. + +**Note:** To generate tags that provide information on the origin of the images use the flag `--repo-based-tags` + + +### Resume copy of image or bundle to tar + +If the copy to tar was interrupted by a network problem it can be resumed by providing the flag `--resume` to the `copy` command + +```bash-plain +$ imgpkg copy -b index.docker.io/k8slt/sample-bundle --to-tar=/Volumes/secure-thumb/bundle.tar --resume +``` + +### Copying via lock file + +[BundleLock configuration](resources.md#bundlelock-configuration) can be used as input to the copy command via the `--lock` flag. + +```bash-plain +$ imgpkg copy --lock bundle.lock.yml --to-repo registry.corp.com/user2/sample-bundle-name --lock-output /tmp/new-bundle.lock.yml +``` + +### Non-Distributable or Foreign Layers + +Some images contain layers which should not be uploaded when copying, such as a proprietary base image. +Instead, to comply with license requirements, it is expected to get them directly from the source registry. +These layers are interchangeably known as +[Non-Distributable](https://github.com/opencontainers/image-spec/blob/79b036d80240ae530a8de15e1d21c7ab9292c693/layer.md#non-distributable-layers) +(by the OCI) or +[Foreign](https://docs.docker.com/registry/spec/manifest-v2-2/) (by Docker) and denoted in the layer's MediaType. + +By default, imgpkg will not relocate any layers marked as non-distributable. + +This can cause issues when dealing with [air-gapped environments](air-gapped-workflow.md) as they may be unable to reach the external registries. +To allow this use case, imgpkg supports the `--include-non-distributable-layers` flag to copy all layers, even those marked as non-distributable. + +Note that usage of this flag shall not preclude your obligation to comply with the terms of the image license(s). + +### Image Signatures + +`imgpkg` can copy Signature created by [cosign](https://github.com/sigstore/cosign). By +default `imgpkg` will not search for Signatures for Images. To enable the search and copy of the signatures the +flag `--cosign-signatures` needs to be provided to copy command + +```bash-plain +$ imgpkg copy -b index.docker.io/k8slt/sample-bundle --to-repo some.repo.io/some-bundle --cosign-signatures +``` + +This feature will work while copying to a different repository as well as copying to a tarball. + +--- + +## Tag + +`imgpkg tag` supports a `list` subcommand that allows users to list the image tags pushed to registries. The command features an `--image`/`-i` option that allows a user to specify an image name. + +An example of this is shown below: + +```bash-plain +$ imgpkg tag list -i index.docker.io/k8slt/image +``` + +The output shows the names of all tags associated with the image, along with its digest. + +--- + +## Describe + +`imgpkg describe` Provides a summary of all the images that are part of the provided Bundle. + +An example of this is shown below: + +```bash-plain +$ imgpkg describe -b carvel.dev/app1-bundle +``` + +This command provides 2 different types of output, `yaml` and `text`, that can be selected via the flag `--output-type`. +By default `text` is selected. + +The flag `--cosign-artifacts` provides the user the ability to select if they want or not `imgpkg` to check and display +any [cosign](https://github.com/sigstore/cosign) artifact that is part of the Bundle. diff --git a/site/content/imgpkg/docs/v0.33.0/debugging.md b/site/content/imgpkg/docs/v0.33.0/debugging.md new file mode 100644 index 000000000..9000ae487 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/debugging.md @@ -0,0 +1,46 @@ +--- +aliases: [/imgpkg/docs/latest/debugging] +title: Debugging +--- + +## Debugging + +In the process of communicating with remote OCI registries, it is possible that an error will occur. In order to help debug an error situation, use the `--debug` command line argument. Specifying this argument will output detailed logs of all communications between `imgpkg` and the OCI registries. + +> This feature is available in v0.20.0 and later + +As an example, consider this pull command along with the additional information logged. The record of all HTTP communication will be displayed to assist in resolving a problem or error condition. + +```bash-plain +imgpkg pull -b registry.example.com/foo/bar:latest -o temp --debug + +2021/09/21 20:50:12 --> GET https://registry.example.com/v2/ +2021/09/21 20:50:12 GET /v2/ HTTP/1.1 +Host: registry.example.com +User-Agent: Go-http-client/1.1 +Accept-Encoding: gzip + +2021/09/21 20:50:12 <-- 401 https://registry.example.com/v2/ (207.596107ms) +2021/09/21 20:50:12 HTTP/1.1 401 Unauthorized +Content-Length: 76 +Connection: keep-alive +Content-Type: application/json; charset=utf-8 +Date: Wed, 22 Sep 2021 02:50:12 GMT +Docker-Distribution-Api-Version: registry/2.0 +Set-Cookie: sid=f9752c01ce47ab50791d4a845a78d996; Path=/; HttpOnly; Secure +Strict-Transport-Security: max-age=31536000; includeSubDomains +Www-Authenticate: Bearer realm="https://registry.example.com/service/token",service="harbor-registry" +X-Request-Id: 2fe97b25-ca40-4012-9105-bbf8284995b6 + +{"errors":[{"code":"UNAUTHORIZED","message":"unauthorized: unauthorized"}]} + +2021/09/21 20:50:12 --> GET https://registry.example.com/service/token?scope=repository%3Afoo%2Fbar%3Apull&service=harbor-registry [body redacted: basic token response contains credentials] +2021/09/21 20:50:12 GET /service/token?scope=repository%3Afoo%2Fbar%3Apull&service=harbor-registry HTTP/1.1 +Host: registry.example.com +User-Agent: go-containerregistry/v0.6.0 +Authorization: +Accept-Encoding: gzip +... +``` + +> Note that sensitive information, such as basic authentication parameters and Authorization strings, are not displayed. diff --git a/site/content/imgpkg/docs/v0.33.0/faq-generic.md b/site/content/imgpkg/docs/v0.33.0/faq-generic.md new file mode 100644 index 000000000..879d4bd09 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/faq-generic.md @@ -0,0 +1,30 @@ +--- +aliases: [/imgpkg/docs/latest/faq-generic] +title: FAQ +--- + +## Using `registry:2` and non-distributable layer + +**We do not recommend the usage of `registry:2` in production** + +There is a known issue when using `registry:2` image as the registry and using it as the destination of a Bundle or +Image that contains non-distributable layers. + +The problem is surfaced with an error similar to + +```shell +imgpkg copy \ + -b my.registry.io/some-bundle:0.0.1 \ + --to-repo localhost:5000/some-bundle +imgpkg: Error: PUT http://localhost:5000/v2/some-bundle/manifests/sha256-6195153fbf1af788bb68124fe2e0b016a1d0b6d3d8ca16cc6d23823d8a7b5445.imgpkg: multiple errors returned: + UNKNOWN: unknown error; UNKNOWN: unknown error; map[]; MANIFEST_BLOB_UNKNOWN: blob unknown to registry; sha256:3a78847ea829208edc2d7b320b7e602b9d12e47689499d5180a9cc7790dec4d7 +``` + +This error happens because the `registry:2` registry does a validation on non-distributable layers and checks the URL +against the provided allowed list, which is empty so it fails. + +For local development this validation can be turned off. To do so start the registry using the following command + +```shell +docker run --env REGISTRY_VALIDATION_DISABLED=true -d -p 5000:5000 --restart always --name registry registry:2 +``` diff --git a/site/content/imgpkg/docs/v0.33.0/install.md b/site/content/imgpkg/docs/v0.33.0/install.md new file mode 100644 index 000000000..058244d52 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/install.md @@ -0,0 +1,58 @@ +--- +aliases: [/imgpkg/docs/latest/install] +title: Install +--- + +## Via script (macOS or Linux) + +(Note that `install.sh` script installs other Carvel tools as well.) + +Install binaries into specific directory: + +```bash +$ mkdir local-bin/ +$ curl -L https://carvel.dev/install.sh | K14SIO_INSTALL_BIN_DIR=local-bin bash + +$ export PATH=$PWD/local-bin/:$PATH +$ imgpkg version +``` + +Or system wide: + +```bash +$ wget -O- https://carvel.dev/install.sh > install.sh + +# Inspect install.sh before running... +$ sudo bash install.sh +$ imgpkg version +``` + +## Via Homebrew (macOS or Linux) + +Based on [github.com/vmware-tanzu/homebrew-carvel](https://github.com/vmware-tanzu/homebrew-carvel). + +```bash +$ brew tap vmware-tanzu/carvel +$ brew install imgpkg +$ imgpkg version +``` + +## Specific version from a GitHub release + +To download, click on one of the assets in a [chosen GitHub release](https://github.com/vmware-tanzu/carvel-imgpkg/releases), for example for 'imgpkg-darwin-amd64'. + +```bash +# **Compare binary checksum** against what's specified in the release notes +# (if checksums do not match, binary was not successfully downloaded) +$ shasum -a 256 ~/Downloads/imgpkg-darwin-amd64 +08b25d21675fdc77d4281c9bb74b5b36710cc091f30552830604459512f5744c /Users/pivotal/Downloads/imgpkg-darwin-amd64 + +# Move binary next to your other executables +$ mv ~/Downloads/imgpkg-darwin-amd64 /usr/local/bin/imgpkg + +# Make binary executable +$ chmod +x /usr/local/bin/imgpkg + +# Check its version +$ imgpkg version +``` diff --git a/site/content/imgpkg/docs/v0.33.0/proxy.md b/site/content/imgpkg/docs/v0.33.0/proxy.md new file mode 100644 index 000000000..0998e7dc2 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/proxy.md @@ -0,0 +1,33 @@ +--- +aliases: [/imgpkg/docs/latest/proxy] +title: Proxy +--- + +## Using Proxy + +When using `imgpkg` to connect with a registry via a proxy you will need to provide one of following environment variables + +- `HTTP_PROXY` or `http_proxy` when using the flag `--registry-insecure` +- `HTTPS_PROXY` or `https_proxy` when the communication with the registry need to be using TLS + +### No TLS example + +Assuming the proxy to access the registry is located in `http://proxy.company.com` + +When executing `imgpkg` do the following: +```bash +export http_proxy=http://proxy.company.com + +imgpkg pull -b registry.company.com/my-image@sha256:265d4a5ed8bf0df27d1107edb00b70e658ee9aa5acb3f37336c5a17db634481e -o folder --registry-insecure +``` + +### TLS example + +Assuming the proxy to access the registry is located in `https://proxy.company.com` + +When executing `imgpkg` do the following: +```bash +export https_proxy=https://proxy.company.com + +imgpkg pull -b registry.company.com/my-image@sha256:265d4a5ed8bf0df27d1107edb00b70e658ee9aa5acb3f37336c5a17db634481e -o folder +``` diff --git a/site/content/imgpkg/docs/v0.33.0/resources.md b/site/content/imgpkg/docs/v0.33.0/resources.md new file mode 100644 index 000000000..6a0300834 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/resources.md @@ -0,0 +1,145 @@ +--- +aliases: [/imgpkg/docs/latest/resources] +title: Resources +--- + +## Image + +An OCI image is an artifact that lives within an OCI registry (such as DockerHub). It can contain an arbitrary number of files. + +--- +## Bundle + +A bundle is an OCI image that holds 0+ arbitrary files _and_ 0+ references to dependent OCI images (which *may* also be [bundles](#nested-bundle)). By tracking dependent images, imgpkg can copy bundles across registries. + +Referenced images are stored within the [`.imgpkg` directory](#imgpkg-directory) at the root level of the bundle image. + +![Bundle diagram](/images/imgpkg/bundle-diagram.png) + +Implementation note: A bundle OCI image has the `dev.carvel.imgpkg.bundle` [label](https://docs.docker.com/config/labels-custom-metadata/) set. + +--- +## `.imgpkg` directory + +`.imgpkg` directory contains metadata files describing bundle: + +- `images.yml` (required) contains [ImagesLock configuration](#imageslock-configuration) that describes 0+ dependent OCI images. Consumers of bundles can rely on this file being always present. + +- `bundle.yml` (optional) file contains [Bundle configuration](#bundle-configuration) that contains details about bundle authors, associated websites, etc. + +Restrictions for location of `.imgpkg` directory: + +- Only one `.imgpkg` directory is allowed across all directories provided via `-f` to the `push` command. This restriction ensures there is a single source of bundle metadata and referenced images. + +- The `.imgpkg` directory must be a direct child of one of the input directories. This prevents any confusion around the scope of the `.imgpkg` metadata. + +--- +## Bundle configuration + +Used by bundle creators to store general information about the bundle. Stored in `.imgpkg/bundle.yml`. + +Example: + +```yaml +apiVersion: imgpkg.carvel.dev/v1alpha1 +kind: Bundle +metadata: + name: my-bundle +authors: +- name: Full Name + email: name@example.com +websites: +- url: example.com +``` + +- `authors` (array of structs; optional) + - `name` (string) Author name + - `email` (string) Author email +- `websites` (array of structs; optional) + - `url` (string) Related website URL + +--- +## ImagesLock configuration + +An ImagesLock configuration is used to track a collection of image references. + +Bundle's `.imgpkg/images.yml` contains ImagesLock configuration. That's how bundle knows which OCI images it references. When copying a bundle `imgpkg` uses this configuration to know which images to copy. + +It can be conveniently generated with [kbld](/kbld): + +```bash-plain +$ kbld -f config.yml --imgpkg-lock-output .imgpkg/images.yml +``` + +Example: + +```yaml +apiVersion: imgpkg.carvel.dev/v1alpha1 +kind: ImagesLock +images: +- image: docker.io/user1/my-app@sha256:42462d0cb227497976754bb67348bdd7471c7bd159819d6bd63fdf479eb7eb19 + annotations: + kbld.carvel.dev/id: "my-app:v1" +- image: gcr.io/projectX/controller@sha256:6ecba6f14373a449f8d54fa4286f57fb8ef37c4ffa637969551f2fda52672206 +``` + +- `images` (array of images): 0+ images + - `image` (string; required) digest reference to OCI image (tag references are not allowed) + - `annotations` (map[string]string; optional) arbitrary additional data about image reference. Expected to be used by tools that create or read ImagesLock configuration. Example: [kbld](/kbld) uses annotations to store an identifier that can later tell it which location(s) within a Kubernetes configuration to update with the digest reference. + +Advanced non-bundle use: See [copying via lock files](commands.md#copying-via-lock-file). + +--- +## BundleLock configuration + +Stores a digest reference to a bundle (as well as the tag it was pushed with). + +This configuration is generated by the `--lock-output` flag during a `push` command. + +```yaml +$ imgpkg push -b ... --lock-output /tmp/lock.yml + +$ cat /tmp/lock.yml + +apiVersion: imgpkg.carvel.dev/v1alpha1 +kind: BundleLock +bundle: + image: docker.io/my-app@sha256:b12026c7a0a6a1756a82a2a74ac759e9a7036523faca0e33dbddebc214e097df + tag: v1.0 +``` + +--- +## Nested Bundle + +A nested bundle is a bundle referenced from a 'parent' bundle in its `ImagesLock` configuration. + +Having a bundle 'reference' another bundle is no different from referencing any other OCI image. The copy and pull commands work the same as dealing with any OCI image. + +![Nested Bundle diagram](/images/imgpkg/nested-bundle-diagram.png) + +One key difference between nested bundles and other OCI images, is the directory structure when `imgpkg pull` writes the nested bundle's content to disk. + +Bundles can be nested repeatedly without limits on depth or breadth. +Imgpkg optimizes both network requests and storage on the destination, so we +would not expect any issues short of hard storage limits at the destination +repository. + +For further details refer to [pulling a nested bundle.](commands.md#pulling-nested-bundles) + +--- +## Locations OCI Image + +`imgpkg` when copying Bundles and Images now creates a new OCI Images that will act as a Cache that contain information +about the Images that were copied and if these Images are a Bundle or not. This OCI Image will contain a single layer +with a single file `image-locations.yml` at the root of the image. This is the file structure + +```yaml +apiVersion: imgpkg.carvel.dev/v1alpha1 +kind: ImageLocations +images: +- image: some.image.io/test@sha256:4c8b96d4fffdfae29258d94a22ae4ad1fe36139d47288b8960d9958d1e63a9d0 + isBundle: true +``` + +The OCI Image will be pushed into the same repository as the Bundle and will have the tag +`sha256-{Bundle SHA}.image-locations.imgpkg` diff --git a/site/content/imgpkg/docs/v0.33.0/security.md b/site/content/imgpkg/docs/v0.33.0/security.md new file mode 100644 index 000000000..ca69db28d --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/security.md @@ -0,0 +1,8 @@ +--- +aliases: [/imgpkg/docs/latest/security] +title: Security +--- + +## Vulnerability Disclosure + +If you believe you have found a security issue in `imgpkg`, please privately and responsibly disclose it by following the directions in our [security policy](/shared/docs/latest/security-policy). diff --git a/site/content/imgpkg/docs/v0.33.0/working-directly-with-images.md b/site/content/imgpkg/docs/v0.33.0/working-directly-with-images.md new file mode 100644 index 000000000..cdac04918 --- /dev/null +++ b/site/content/imgpkg/docs/v0.33.0/working-directly-with-images.md @@ -0,0 +1,8 @@ +--- +aliases: [/imgpkg/docs/latest/working-directly-with-images] +title: Working directly with images +--- + +In rare cases imgpkg's [bundle](resources.md#bundle) concept is not wanted (or necessary). imgpkg provides a `--image` flag for push, pull and copy commands. When the `--image` flag is used, there is no need for a `.imgpkg` directory to store metadata. + +For most use cases, we suggest using the bundle concept and `--bundle` flag. diff --git a/site/data/imgpkg/docs/imgpkg-v0-33-0-toc.yml b/site/data/imgpkg/docs/imgpkg-v0-33-0-toc.yml new file mode 100644 index 000000000..a04ec6eeb --- /dev/null +++ b/site/data/imgpkg/docs/imgpkg-v0-33-0-toc.yml @@ -0,0 +1,43 @@ +toc: + - title: Introduction + subfolderitems: + - page: About imgpkg + url: / + - page: Install + url: /install + - title: Workflows + subfolderitems: + - page: Basic workflow + url: /basic-workflow + - page: Air-gapped workflow + url: /air-gapped-workflow + - page: Automation workflow + url: /automation-workflow + - title: Reference + subfolderitems: + - page: Authentication + url: /auth + - page: Resources + url: /resources + - page: Commands + url: /commands + - page: Working directly with images + url: /working-directly-with-images + - page: Proxy + url: /proxy + - title: FAQ + subfolderitems: + - page: General + url: /faq-generic + - page: Code of Conduct + shared_url: /code-of-conduct + - page: Contributing + shared_url: /contributing + - page: Carvel Development Guidelines + shared_url: /development_guidelines + - page: Security + shared_url: /security-policy + - page: CA Certs on Windows + url: /ca-certs-windows + - page: Debugging + url: /debugging diff --git a/site/data/imgpkg/docs/toc-mapping.yml b/site/data/imgpkg/docs/toc-mapping.yml index 91c64cbab..e728c7989 100644 --- a/site/data/imgpkg/docs/toc-mapping.yml +++ b/site/data/imgpkg/docs/toc-mapping.yml @@ -8,3 +8,4 @@ v0.29.0: imgpkg-v0-29-0-toc v0.30.0: imgpkg-v0-30-0-toc v0.31.0: imgpkg-v0-31-0-toc v0.32.0: imgpkg-v0-32-0-toc +v0.33.0: imgpkg-v0-33-0-toc From 19c78836888c63e75454cce4cb5bf14ebcaaf74a Mon Sep 17 00:00:00 2001 From: Joao Pereira Date: Tue, 20 Sep 2022 13:57:44 -0500 Subject: [PATCH 34/35] Update maintainers.md - Add Joao as a Lead for `imgpkg` Signed-off-by: Joao Pereira --- MAINTAINERS.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 53ffc1c11..6845f354f 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -5,20 +5,20 @@ The Carvel [GOVERNANCE.md](https://github.com/vmware-tanzu/carvel/blob/develop/G ## Maintainers Maintainers may focus on one or many tools. Some maintainers have different responsibilities for different tools (such as being an Approver or Reviewer). This is in the parenthesis next to their tool focus below. -| Maintainer | GitHub ID | Affiliation | Carvel Tool Focus | -| --------------- | --------- | ----------- | ----------- | -| Dmitriy Kalinin | [cppforlife](https://github.com/cppforlife) | [VMware](https://www.github.com/vmware/) | All Tools (Approver) | -| John Ryan | [pivotaljohn](https://github.com/pivotaljohn) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [ytt](https://github.com/vmware-tanzu/carvel-ytt) (Approver) | -| João Pereira | [joaopapereira](https://github.com/joaopapereira) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg) (Approver), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [ytt](https://github.com/vmware-tanzu/carvel-ytt) | -| Varsha Munishwar | [vmunishwar](https://github.com/vmunishwar)| [VMware](https://www.github.com/vmware/) |[ytt](https://github.com/vmware-tanzu/carvel-ytt) | -| Joe Kimmel | [joe-kimmel-vmw](https://github.com/joe-kimmel-vmw) | [VMware](https://www.github.com/vmware/) | [kapp-controller](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [vendir](https://github.com/vmware-tanzu/carvel-vendir), [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) | +| Maintainer | GitHub ID | Affiliation | Carvel Tool Focus | +| --------------- | --------- | ----------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Dmitriy Kalinin | [cppforlife](https://github.com/cppforlife) | [VMware](https://www.github.com/vmware/) | All Tools (Approver) | +| John Ryan | [pivotaljohn](https://github.com/pivotaljohn) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [ytt](https://github.com/vmware-tanzu/carvel-ytt) (Approver) | +| João Pereira | [joaopapereira](https://github.com/joaopapereira) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg) (Lead), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [vendir](https://github.com/vmware-tanzu/carvel-vendir) | +| Varsha Munishwar | [vmunishwar](https://github.com/vmunishwar)| [VMware](https://www.github.com/vmware/) | [ytt](https://github.com/vmware-tanzu/carvel-ytt) | +| Joe Kimmel | [joe-kimmel-vmw](https://github.com/joe-kimmel-vmw) | [VMware](https://www.github.com/vmware/) | [kapp-controller](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [vendir](https://github.com/vmware-tanzu/carvel-vendir), [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) | | Neil Hickey | [neil-hickey](https://github.com/neil-hickey) | [VMware](https://www.github.com/vmware/) | [kapp-controller](https://github.com/vmware-tanzu/carvel-kapp-controller) (Reviewer), [vendir](https://github.com/vmware-tanzu/carvel-vendir), [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) | -| Praveen Rewar | [praveenrewar](https://github.com/praveenrewar) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver)| -| Yash Sethiya | [sethiyash](https://github.com/sethiyash) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | -| Soumik Majumder | [100mik](https://github.com/100mik) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver)| -| Rohit Aggarwal | [rohitagg2020](https://github.com/rohitagg2020) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | -| John Brunton | [jbrunton](https://github.com/jbrunton) | [converge.io](https://converge.io/) | [carvel-setup-action](https://github.com/vmware-tanzu/carvel-setup-action) (Approver) | -| Kumari Tanushree | [kumaritanushree](https://github.com/kumaritanushree) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | +| Praveen Rewar | [praveenrewar](https://github.com/praveenrewar) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver) | +| Yash Sethiya | [sethiyash](https://github.com/sethiyash) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | +| Soumik Majumder | [100mik](https://github.com/100mik) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver) | +| Rohit Aggarwal | [rohitagg2020](https://github.com/rohitagg2020) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | +| John Brunton | [jbrunton](https://github.com/jbrunton) | [converge.io](https://converge.io/) | [carvel-setup-action](https://github.com/vmware-tanzu/carvel-setup-action) (Approver) | +| Kumari Tanushree | [kumaritanushree](https://github.com/kumaritanushree) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | ## Emeritus Maintainers From 780f4209b37bc57e61f4015aec1784cc9a24903b Mon Sep 17 00:00:00 2001 From: Joao Pereira Date: Tue, 20 Sep 2022 13:58:48 -0500 Subject: [PATCH 35/35] Reformat tables to make them easier to edit Signed-off-by: Joao Pereira --- MAINTAINERS.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/MAINTAINERS.md b/MAINTAINERS.md index 6845f354f..0561b70a9 100644 --- a/MAINTAINERS.md +++ b/MAINTAINERS.md @@ -5,19 +5,19 @@ The Carvel [GOVERNANCE.md](https://github.com/vmware-tanzu/carvel/blob/develop/G ## Maintainers Maintainers may focus on one or many tools. Some maintainers have different responsibilities for different tools (such as being an Approver or Reviewer). This is in the parenthesis next to their tool focus below. -| Maintainer | GitHub ID | Affiliation | Carvel Tool Focus | -| --------------- | --------- | ----------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Dmitriy Kalinin | [cppforlife](https://github.com/cppforlife) | [VMware](https://www.github.com/vmware/) | All Tools (Approver) | -| John Ryan | [pivotaljohn](https://github.com/pivotaljohn) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [ytt](https://github.com/vmware-tanzu/carvel-ytt) (Approver) | -| João Pereira | [joaopapereira](https://github.com/joaopapereira) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg) (Lead), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [vendir](https://github.com/vmware-tanzu/carvel-vendir) | -| Varsha Munishwar | [vmunishwar](https://github.com/vmunishwar)| [VMware](https://www.github.com/vmware/) | [ytt](https://github.com/vmware-tanzu/carvel-ytt) | -| Joe Kimmel | [joe-kimmel-vmw](https://github.com/joe-kimmel-vmw) | [VMware](https://www.github.com/vmware/) | [kapp-controller](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [vendir](https://github.com/vmware-tanzu/carvel-vendir), [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) | -| Neil Hickey | [neil-hickey](https://github.com/neil-hickey) | [VMware](https://www.github.com/vmware/) | [kapp-controller](https://github.com/vmware-tanzu/carvel-kapp-controller) (Reviewer), [vendir](https://github.com/vmware-tanzu/carvel-vendir), [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) | -| Praveen Rewar | [praveenrewar](https://github.com/praveenrewar) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver) | -| Yash Sethiya | [sethiyash](https://github.com/sethiyash) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | -| Soumik Majumder | [100mik](https://github.com/100mik) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver) | -| Rohit Aggarwal | [rohitagg2020](https://github.com/rohitagg2020) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | -| John Brunton | [jbrunton](https://github.com/jbrunton) | [converge.io](https://converge.io/) | [carvel-setup-action](https://github.com/vmware-tanzu/carvel-setup-action) (Approver) | +| Maintainer | GitHub ID | Affiliation | Carvel Tool Focus | +|------------------|-------------------------------------------------------|------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Dmitriy Kalinin | [cppforlife](https://github.com/cppforlife) | [VMware](https://www.github.com/vmware/) | All Tools (Approver) | +| John Ryan | [pivotaljohn](https://github.com/pivotaljohn) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [ytt](https://github.com/vmware-tanzu/carvel-ytt) (Approver) | +| João Pereira | [joaopapereira](https://github.com/joaopapereira) | [VMware](https://www.github.com/vmware/) | [imgpkg](https://github.com/vmware-tanzu/carvel-imgpkg) (Lead), [kbld](https://github.com/vmware-tanzu/carvel-kbld), [vendir](https://github.com/vmware-tanzu/carvel-vendir) | +| Varsha Munishwar | [vmunishwar](https://github.com/vmunishwar) | [VMware](https://www.github.com/vmware/) | [ytt](https://github.com/vmware-tanzu/carvel-ytt) | +| Joe Kimmel | [joe-kimmel-vmw](https://github.com/joe-kimmel-vmw) | [VMware](https://www.github.com/vmware/) | [kapp-controller](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [vendir](https://github.com/vmware-tanzu/carvel-vendir), [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) | +| Neil Hickey | [neil-hickey](https://github.com/neil-hickey) | [VMware](https://www.github.com/vmware/) | [kapp-controller](https://github.com/vmware-tanzu/carvel-kapp-controller) (Reviewer), [vendir](https://github.com/vmware-tanzu/carvel-vendir), [secretgen-controller](https://github.com/vmware-tanzu/carvel-secretgen-controller) | +| Praveen Rewar | [praveenrewar](https://github.com/praveenrewar) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver) | +| Yash Sethiya | [sethiyash](https://github.com/sethiyash) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | +| Soumik Majumder | [100mik](https://github.com/100mik) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) (Reviewer), [kctrl](https://github.com/vmware-tanzu/carvel-kapp-controller/tree/develop/cli) (Approver) | +| Rohit Aggarwal | [rohitagg2020](https://github.com/rohitagg2020) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | +| John Brunton | [jbrunton](https://github.com/jbrunton) | [converge.io](https://converge.io/) | [carvel-setup-action](https://github.com/vmware-tanzu/carvel-setup-action) (Approver) | | Kumari Tanushree | [kumaritanushree](https://github.com/kumaritanushree) | [VMware](https://www.github.com/vmware/) | [kapp](https://github.com/vmware-tanzu/carvel-kapp) | @@ -32,10 +32,10 @@ Maintainers may focus on one or many tools. Some maintainers have different resp ## Carvel Stakeholders -| Carvel Feature Area | Lead | -| ----------------------------- | :---------------------: | -| All | Dmitriy Kalinin (cppforlife), Project Tech Lead | -| imgpkg, kapp-controller, kbld, secretgen-controller, vendir, ytt | Aaron Hurley (aaronshurley), Engineering Manager | -| kapp, kctrl | Renu Yarday (renuy), Engineering Manager | -| [Windows Application Packages Contributor](https://github.com/adriens?tab=repositories&q=chocolatey-&type=&language=powershell&sort=stargazers) | Adrien Sales (adriens) | -| Community Management | Nanci Lancaster (microwavables) | +| Carvel Feature Area | Lead | +|-------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------:| +| All | Dmitriy Kalinin (cppforlife), Project Tech Lead | +| imgpkg, kapp-controller, kbld, secretgen-controller, vendir, ytt | Aaron Hurley (aaronshurley), Engineering Manager | +| kapp, kctrl | Renu Yarday (renuy), Engineering Manager | +| [Windows Application Packages Contributor](https://github.com/adriens?tab=repositories&q=chocolatey-&type=&language=powershell&sort=stargazers) | Adrien Sales (adriens) | +| Community Management | Nanci Lancaster (microwavables) |