diff --git a/dev/pages/remote_repositories/index.html b/dev/pages/remote_repositories/index.html
index e8bcc37b3..6cdb5eaa0 100644
--- a/dev/pages/remote_repositories/index.html
+++ b/dev/pages/remote_repositories/index.html
@@ -18,7 +18,7 @@
diff --git a/dev/search/search_index.json b/dev/search/search_index.json
index ad3035eb9..2ef20b841 100644
--- a/dev/search/search_index.json
+++ b/dev/search/search_index.json
@@ -1 +1 @@
-{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Kapitan: Keep your ship together","text":"Kapitan aims to be your one-stop configuration management solution to help you manage the ever growing complexity of your configurations by enabling Platform Engineering and GitOps workflows.
It streamlines complex deployments across heterogeneous environments while providing a secure and adaptable framework for managing infrastructure configurations. Kapitan's inventory-driven model, powerful templating capabilities, and native secret management tools offer granular control, fostering consistency, reducing errors, and safeguarding sensitive data.
Empower your team to make changes to your infrastructure whilst maintaining full control, with a GitOps approach and full transparency.
- Join the community
#kapitan - Help us grow: give us a star or even better sponsor our project
"},{"location":"#why-do-i-need-kapitan","title":"Why do I need Kapitan?","text":""},{"location":"#video-tutorials-to-get-started","title":"Video Tutorials to get started","text":"Kapitan Youtube Channel
InventoryReferencesHelm and Generators integrationRawkode: Introduction to Kapitan "},{"location":"ADOPTERS/","title":"Who uses Kapitan","text":"If you're using Kapitan in your organization, please let us know by adding to this list on the docs/ADOPTERS.md file.
"},{"location":"FAQ/","title":"FAQ","text":""},{"location":"FAQ/#why-do-i-need-kapitan","title":"Why do I need Kapitan?","text":"See Why do I need Kapitan?
"},{"location":"FAQ/#ask-your-question","title":"Ask your question","text":"Please use the comments facility below to ask your question
"},{"location":"getting_started/","title":"Kapitan Overview","text":""},{"location":"getting_started/#setup-your-installation","title":"Setup your installation","text":"Using our reference repositories you can easily get started with Kapitan
"},{"location":"getting_started/#examples-repository","title":"Examples repository","text":"kapicorp/kapitan-reference repository is meant show you many working examples of things you can do with Kapitan. You can use this to get familiar with Kapitan
$ git clone git@github.com:kapicorp/kapitan-reference.git kapitan-templates\n$ cd kapitan-templates\n\n$ ./kapitan compile\nCompiled postgres-proxy (1.51s)\nCompiled tesoro (1.70s)\nCompiled echo-server (1.64s)\nCompiled mysql (1.67s)\nCompiled gke-pvm-killer (1.17s)\nCompiled prod-sockshop (4.74s)\nCompiled dev-sockshop (4.74s)\nCompiled tutorial (1.68s)\nCompiled global (0.76s)\nCompiled examples (2.60s)\nCompiled pritunl (2.03s)\nCompiled sock-shop (4.36s)\n
"},{"location":"getting_started/#minimal-repository","title":"Minimal repository","text":"Using cruft based cookiecutter
pip3 install cruft\n
cruft create http://github.com/kapicorp/kapitan-reference --checkout cookiecutter --no-input\nDependency https://github.com/kapicorp/generators.git: saved to system/lib\nDependency https://github.com/kapicorp/generators.git: saved to system/generators/kubernetes\nDependency https://github.com/kapicorp/generators.git: saved to system/generators/terraform\nRendered inventory (1.74s)\nCompiled echo-server (0.14s)\n
"},{"location":"getting_started/#running-kapitan","title":"running Kapitan","text":"recommended
"},{"location":"getting_started/#kapitan-wrapper-script","title":"kapitan wrapper script","text":"If you use the provided repository, we already package a kapitan shell script that wraps the docker command to run Kapitan
$ ./kapitan compile\nCompiled postgres-proxy (1.51s)\nCompiled tesoro (1.70s)\nCompiled echo-server (1.64s)\nCompiled mysql (1.67s)\nCompiled gke-pvm-killer (1.17s)\nCompiled prod-sockshop (4.74s)\nCompiled dev-sockshop (4.74s)\nCompiled tutorial (1.68s)\nCompiled global (0.76s)\nCompiled examples (2.60s)\nCompiled pritunl (2.03s)\nCompiled sock-shop (4.36s)\n
"},{"location":"getting_started/#other-installation-methods","title":"Other installation methods","text":""},{"location":"getting_started/#docker","title":"Docker","text":"recommended
"},{"location":"getting_started/#docker_1","title":"Docker","text":"LinuxMac alias kapitan=\"docker run -t --rm -u $(id -u) -v $(pwd):/src:delegated kapicorp/kapitan\"\nkapitan -h\n
alias kapitan=\"docker run -t --rm -v $(pwd):/src:delegated kapicorp/kapitan\"\nkapitan -h\n
"},{"location":"getting_started/#pip","title":"Pip","text":""},{"location":"getting_started/#install-python","title":"Install Python","text":"LinuxMac sudo apt-get update && sudo apt-get install -y python3.8-dev python3-pip python3-yaml\n
brew install python3 libyaml libmagic\n
"},{"location":"getting_started/#install-kapitan-using-pip","title":"Install Kapitan using pip","text":""},{"location":"getting_started/#user","title":"User","text":"LinuxMac kapitan will be installed in $HOME/.local/lib/python3.7/bin
pip3 install --user --upgrade kapitan\n
kapitan will be installed in $HOME/Library/Python/3.7/bin
pip3 install --user --upgrade kapitan\n
"},{"location":"getting_started/#system-wide","title":"System-wide","text":"not recommended
sudo pip3 install --upgrade kapitan\n
"},{"location":"proposals/","title":"Kapitan proposals","text":"","tags":["community"]},{"location":"proposals/#introduction","title":"Introduction","text":"Proposals can be submitted for review by performing a pull request against this repository. If approved the proposal will be published here for further review by the Kapitan community. Proposals tend to be improvements or design consideration for new features.
","tags":["community"]},{"location":"proposals/#existing-proposals","title":"Existing proposals","text":"Kadet input type
External dependency management
Helm charts input type
Kubernetes scheme validation
Portable standalone Kapitan executable
Ref types redesign
Hashicorp vault secrets
","tags":["community"]},{"location":"references/","title":"Kapitan References (formally Secrets)","text":"One of the motivations behing Kapitan's design is that we believe that everything about your setup should be tracked, and Kapitan takes this to the extreme. Sometimes, however, we have to manage values that we do not think they belong to the Inventory: perhaps they are either too variable (for instance, a Git commit sha that changes with every build) or too sensitive, like a password or a generic secret, and then they should always be encrypted.
Kapitan has a built in support for References, which you can use to manage both these use cases.
Kapitan References supports the following backends:
Backend Description Encrypted plain Plain text, (e.g. commit sha) base64 Base64, non confidential but with base64 encoding gpg Support for https://gnupg.org/ gkms GCP KMS awskms AWS KMS azkms Azure Key Vault env Environment vaultkv Hashicorp Vault (RO) vaulttransit Hashicorp Vault (encrypt, decrypt, update_key, rotate_key)"},{"location":"references/#setup","title":"Setup","text":"Some reference backends require configuration, both in the Inventory and to configure the actual backend.
Get started
If you want to get started with references but don't want to deal with the initial setup, you can use the plain and base64 reference types. These are great for demos, but we will see they are extremely helpful even in Production environments.
Danger
Both plain and base64 references do not support encryption: they are intended for development or demo purposes only. DO NOT use plain or base64 for storing sensitive information!
Backend configuration
Configuration for each backend varies, and it is perfomed by configuring the inventory under parameters.kapitan.secrets.
plainbase64gpggkmsawskmsazkmsenvvaultkvvaulttransit No configuration needed
No configuration needed
parameters:\n kapitan:\n secrets:\n gpg:\n recipients:\n - name: example@kapitan.dev\n fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C\n
parameters:\n kapitan:\n secrets:\n gkms:\n key: 'projects/<project>/locations/<location>/keyRings/<keyRing>/cryptoKeys/<key>'\n
parameters:\n kapitan:\n secrets:\n awskms:\n key: 'alias/nameOfKey'\n
parameters:\n kapitan:\n secrets:\n azkms:\n key: 'https://<keyvault-name>.vault.azure.net/keys/<object-name>/<object-version>'\n
parameters:\n ...\n mysql:\n root_password: ?{env:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n kapitan:\n secrets:\n vaultkv:\n VAULT_ADDR: http://127.0.0.1:8200\n auth: token\n mount: secret\n
parameters:\n kapitan:\n secrets:\n vaulttransit:\n VAULT_ADDR: https://vault.example.com\n VAULT_TOKEN: s.mqWkI0uB6So0aHH0v0jyDs97\n VAULT_SKIP_VERIFY: \"False\" # Recommended\n auth: token\n mount: mytransit\n key: 2022-02-13-test\n
Organize your configuration in classes
Just like any other inventory parameters, these configurations can be inherited from a common class or defined per target.
inventory/classes/common.yml
classes:\n- security.backend\n...\n
inventory/classes/security/backend.yml
parameters:\n kapitan:\n secrets:\n <backend>: <configuration>\n
ADVANCED: Mix-and-Match backends Remember that you can use multiple backends at the same time, and also use variable interpolation for an even greater flexibility.
In a multi-cloud setup, for instance, you could configure both GKMS
GCP configuration
inventory/classes/cloud/gcp.yml
classes:\n- security.backends.gkms\n...\n
inventory/classes/security/backends/gkms.yml
# Configuration for GCP targets\nparameters:\n backend: gkms\n kapitan:\n secrets:\n gkms: <configuration>\n
AWS configuration
inventory/classes/security/backends/awskms.yml
# Configuration for AWS targets\nparameters:\n backend: awskms\n kapitan:\n secrets:\n awskms: <configuration>\n
inventory/classes/cloud/aws.yml
classes:\n- security.backends.awskms\n...\n
Now because they both set the parameters.backend variable, you can define a reference whose backend changes based on what class is assigned to the target
inventory/targets/cloud/gcp/acme.yml
classes:\n- cloud.aws\n\nparameters:\n ...\n mysql:\n # the secret backend will change based on the cloud assigned to this target\n root_password: ?{${backend}:targets/${target_name}/mysql/root_password}\n ...\n
"},{"location":"references/#define-references","title":"Define references","text":"References can be defined in the inventory following the syntax spaces added for clarity:
?{ <backend_id> : <reference_path> }
expand for advanced features The syntax also supports for process functions and create_functions which we will discuss later, which brings the full syntax to
?{ <backend_id> : <reference_path> } |<process_function> ||<create_function>
plainbase64gpggkmsawskmsazkmsenvvaultkvvaulttransit parameters:\n ...\n mysql:\n root_password: ?{plain:targets/${target_name}/mysql/root_password}\n ...\n
not encrypted
This reference type does not support encryption: it is intended for non sensitive data only. DO NOT use plain for storing sensitive information!
parameters:\n ...\n mysql:\n root_password: ?{base64:targets/${target_name}/mysql/root_password}\n ...\n
not encrypted
This reference type does not support encryption: it is intended for non sensitive data only. DO NOT use base64 for storing sensitive information!
parameters:\n ...\n mysql:\n root_password: ?{gpg:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{gkms:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{awskms:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{azkms:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{env:targets/${target_name}/mysql/root_password}\n ...\n
read-only
parameters:\n ...\n mysql:\n root_password: ?{vaultkv:targets/${target_name}/mysql/root_password}\n ...\n
read-write: parameters:\n ...\n mysql:\n root_password: ?{vaultkv:targets/${target_name}/mysql/root_password:mount:path/in/vault:mykey}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{vaulttransit:targets/${target_name}/mysql/root_password}\n ...\n
"},{"location":"references/#assign-a-value","title":"Assign a value","text":""},{"location":"references/#manually","title":"Manually","text":"You can assign values to your reference using the command line. Both reading from a file and pipes are supported.
Please Note
Kapitan will fail compilation if a reference is not found. Please see how to assign a value automatically in the next section
plainbase64gpggkmsawskmsazkmsenvvaultkvvaulttransit kapitan refs --write plain:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write plain:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write base64:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write base64:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write gpg:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write gpg:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write gkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write gkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write vaulttransit:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write vaulttransit:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write azkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write azkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
Setting default value only
The env backend works in a slightly different ways, as it allows you to reference environment variables at runtime.
For example, for a reference called {?env:targets/envs_defaults/mysql_port_${target_name}}, Kapitan would look for an environment variable called KAPITAN_ENV_mysql_port_${TARGET_NAME}.
If that variable cannot be found in the Kapitan environment, the default will be taken from the refs/targets/envs_defaults/mysql_port_${TARGET_NAME} file instead.
kapitan refs --write env:refs/targets/envs_defaults/mysql_port_${TARGET_NAME} -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write env:refs/targets/envs_defaults/mysql_port_${TARGET_NAME} -t ${TARGET_NAME} -f -\n
kapitan refs --write vaultkv:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write vaultkv:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
This backend expects the value to be stored as a key:value pair.
echo \"a_key:a_value\" | kapitan refs --write vaulttransit:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
When reading from disk, the input file should be formatted accordingly.
"},{"location":"references/#automatically","title":"Automatically","text":"Kapitan has built in capabilities to initialise its references on creation, using an elegant combination of primary and secondary functions. This is extremely powerful because it allows for you to make sure they are always initialised with sensible values.
"},{"location":"references/#primary-functions","title":"primary functions","text":"To automate the creation of the reference, you can add one of the following primary functions to the reference tag by using the syntax ||primary_function:param1:param2
For instance, to automatically initialise a reference with a random string with a lenght of 32 characters, you can use the random primary function
```yaml\nparameters:\n ...\n mysql:\n root_password: ?{${backend}:targets/${target_name}/mysql/root_password||random:str:32}\n ...\n```\n
Initialise non existent references
The first operator here || is more similar to a logical OR.
- If the reference file does not exist, Kapitan will use the function to initialise it
- If the reference file exists, no functions will run.
Automate secret rotation with ease
You can take advantage of it to implement easy rotation of secrets. Simply delete the reference files, and run kapitan compile: let Kapitan do the rest.
randomprivate keysbasicauthreveal strintloweralphaupperalphaloweralphanumupperalphanumspecial Generator function for alphanumeric characters, will be url-token-safe
?{${backend}:targets/${target_name}/mysql/root_password||random:str}\n
generator function for digits (0-9)
?{${backend}:targets/${target_name}/mysql/root_password||random:int}\n
generator function for lowercase letters (a-z)
?{${backend}:targets/${target_name}/mysql/root_password||random:loweralpha}\n
generator function for uppercase letters (A-Z)
?{${backend}:targets/${target_name}/mysql/root_password||random:upperalpha}\n
generator function for lowercase letters and numbers (a-z and 0-9)
?{${backend}:targets/${target_name}/mysql/root_password||random:loweralphanum}\n
generator function for uppercase letters and numbers (A-Z and 0-9)
?{${backend}:targets/${target_name}/mysql/root_password||random:upperalphanum}\n
generator function for alphanumeric characters and given special characters
?{${backend}:targets/${target_name}/mysql/root_password||random:special}\n
rsaed25519publickeyrsapublic Generates an RSA 4096 private key (PKCS#8). You can optionally pass the key size
?{${backend}:targets/${target_name}/private_key||rsa}\n
Generates a ed25519 private key (PKCS#8)
?{${backend}:targets/${target_name}/private_key||ed25519}\n
Derives the public key from a revealed private key
?{${backend}:targets/${target_name}/private_key||rsa}\n?{${backend}:targets/${target_name}/public_key||reveal:targets/${target_name}/private_key|publickey}\n
DEPRECATED: use ||publickey
Generates a base64 encoded pair of username:password
?{${backend}:targets/${target_name}/apache_basicauth||basicauth:username:password}\n
Reveals the content of another reference, useful when deriving public keys or a reference requires a different encoding or the same value.
?{${backend}:targets/${target_name}/secret||random:str}\n?{${backend}:targets/${target_name}/base64_secret||reveal:targets/${target_name}/secret|base64}\n
attention when rotating secrets used with reveal
If you use reveal to initialise a reference, like my_reference||reveal:source_reference the my_reference will not be automatically updated if source_reference changes. Please make sure you also re-initialise my_reference correctly
"},{"location":"references/#secondary-functions","title":"secondary functions","text":"base64sha256 base64 encodes your reference
?{${backend}:targets/${target_name}/secret||random:str|base64}\n
sha256 hashes your reference param1: salt
?{${backend}:targets/${target_name}/secret||random:str|sha256}\n
"},{"location":"references/#reveal-references","title":"Reveal references","text":"You can reveal the secrets referenced in the outputs of kapitan compile via:
```shell\nkapitan refs --reveal -f path/to/rendered/template\n```\n
For example, compiled/minikube-mysql/manifests/mysql_secret.yml with the following content:
```yaml\napiVersion: v1\ndata:\n MYSQL_ROOT_PASSWORD: ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de}\n MYSQL_ROOT_PASSWORD_SHA256: ?{gpg:targets/minikube-mysql/mysql/password_sha256:122d2732}\nkind: Secret\nmetadata:\n annotations: {}\n labels:\n name: example-mysql\n name: example-mysql\n namespace: minikube-mysql\ntype: Opaque\n```\n
can be revealed as follows:
```shell\nkapitan refs --reveal -f compiled/minikube-mysql/manifests/mysql_secret.yml\n```\n
This will substitute the referenced secrets with the actual decrypted secrets stored at the referenced paths and display the file content.
You can also use:
```shell\nkapitan refs --reveal --ref-file refs/targets/all-glob/mysql/password\n```\n
or
```shell\nkapitan refs --reveal --tag \"?{base64:targets/all-glob/mysql/password}\"\n# or\nkapitan refs --reveal --tag \"?{base64:targets/all-glob/mysql/password:3192c15c}\"\n```\n
for more convenience.
"},{"location":"references/#embedded-refs","title":"Embedded refs","text":"Please refer to the CLI reference
"},{"location":"references/#yaml-subvars-references","title":"YAML SubVars References","text":"Kapitan is also able to use access specific keys in YAML content by using subvars.
For instance given a reference plain:larder with content:
```yaml\nfood:\n apples: 1\n```\n
I could now have an inventory variable like:
```yaml\nparameters:\n number_of_apples: ?{plain:larder@food.apple}\n```\n
"},{"location":"references/#using-subvars-to-ingest-yaml-from-command-line-tools","title":"Using subvars to ingest yaml from command line tools","text":"Subvars can have a very practical use for storing YAML outputs coming straight from other tools. For instance, I could use the GCP gcloud command to get all the information about a cluster, and write it into a reference
```shell\ngcloud container clusters describe \\\n --project ${TARGET_NAME}-project \\\n gke-cluster --zone europe-west1 --format yaml \\\n | kapitan refs --write plain:clusters/${TARGET_NAME}/cluster -t ${TARGET_NAME} -f -\n```\n
knowing the output of gcloud to produce yaml that contain the following values:
```yaml\n...\nname: gke-cluster\nreleaseChannel:\n channel: REGULAR\nselfLink: https://container.googleapis.com/v1/projects/kapicorp/locations/europe-west1/clusters/gke-cluster\n...\n```\n
I can not reference the link to the cluster in the inventory using:
```yaml\nparameters:\n cluster:\n name: ?{plain:clusters/${target_name}/cluster@name}\n release_channel: ?{plain:clusters/${target_name}/cluster@releaseChannel.channel}\n link: ?{plain:clusters/${target_name}/cluster@selfLink}\n```\n
Combined with a Jinja template, I could write automatically documentation containing the details of the clusters I use.
```text\n{% set p = inventory.parameters %}\n# Documentation for {{p.target_name}}\n\nCluster [{{p.cluster.name}}]({{p.cluster.link}}) has release channel {{p.cluster.release_channel}}\n```\n
"},{"location":"references/#hashicorp-vault","title":"Hashicorp Vault","text":""},{"location":"references/#vaultkv","title":"vaultkv","text":"Considering a key-value pair like my_key:my_secret in the path secret/foo/bar in a kv-v2(KV version 2) secret engine on the vault server, to use this as a secret use:
```shell\necho \"foo/bar:my_key\" | kapitan refs --write vaultkv:path/to/secret_inside_kapitan -t <target_name> -f -\n```\n
To write a secret in the vault with kapitan use a ref tag with following structure:
```yaml\nparameters:\n ...\n secret:\n my_secret: ?{vaultkv:targets/${target_name}/mypath:mount:path/in/vault:mykey||<functions>}\n ...\n```\n
Leave mount empty to use the specified mount from vault params from the inventory (see below). Same applies to the path/in/vault where the ref path in kapitan gets taken as default value.
Parameters in the secret file are collected from the inventory of the target we gave from CLI -t <target_name>. If target isn't provided then kapitan will identify the variables from the environment when revealing secret.
The environment variables which can also be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Note that providing these variables through the inventory in envvar style is deprecated. Users should update their inventory to set these values in keys without the VAULT_ prefix and in all lowercase. For example VAULT_ADDR: https://127.0.0.1:8200 should be given as addr: https://127.0.0.1:8200 in the inventory. Please note that configuring one of these values in both kapitan.secrets.vaultkv in the inventory and in the environment will cause a validation error.
Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=alpha-secret/foo/bar then mount: alpha-secret (default secret)engine: secret engine used, either kv-v2 or kv (default kv-v2)
The environment variables which cannot be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID.
```yaml\n parameters:\n kapitan:\n secrets:\n vaultkv:\n auth: userpass\n engine: kv-v2\n mount: team-alpha-secret\n addr: http://127.0.0.1:8200\n namespace: CICD-alpha\n skip_verify: false\n client_key: /path/to/key\n client_cert: /path/to/cert\n ```\n
"},{"location":"references/#vaulttransit","title":"vaulttransit","text":"Considering a key-value pair like my_key:my_secret in the path secret/foo/bar in a transit secret engine on the vault server, to use this as a secret use:
```shell\necho \"any.value:whatever-you_may*like\" | kapitan refs --write vaulttransit:my_target/to/secret_inside_kapitan -t <target_name> -f -\n```\n
Parameters in the secret file are collected from the inventory of the target we gave from CLI -t <target_name>. If target isn't provided then kapitan will identify the variables from the environment when revealing secret.
Environment variables that can be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=my_mount (default transit)crypto_key: Name of the encryption key defined in vault-
always_latest: Always rewrap ciphertext to latest rotated crypto_key version Environment variables cannot be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID.
parameters:\n kapitan:\n vars:\n target: my_target\n namespace: my_namespace\n secrets:\n vaulttransit:\n VAULT_ADDR: http://vault.example.com:8200\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n VAULT_SKIP_VERIFY: \"True\"\n auth: token\n mount: transit\n crypto_key: new_key\n always_latest: False\nparameters:\n target_name: secrets\n kapitan:\n secrets:\n vaulttransit:\n VAULT_ADDR: http://127.0.0.1:8200\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n VAULT_SKIP_VERIFY: \"True\"\n auth: token\n mount: transit\n crypto_key: key\n always_latest: False\n
"},{"location":"references/#azure-kms-secret-backend","title":"Azure KMS Secret Backend","text":"To encrypt secrets using keys stored in Azure's Key Vault, a key_id is required to identify an Azure key object uniquely. It should be of the form https://{keyvault-name}.vault.azure.net/{object-type}/{object-name}/{object-version}.
"},{"location":"references/#defining-the-kms-key","title":"Defining the KMS key","text":"This is done in the inventory under parameters.kapitan.secrets.
```yaml\nparameters:\n kapitan:\n vars:\n target: ${target_name}\n namespace: ${target_name}\n secrets:\n azkms:\n key: 'https://<keyvault-name>.vault.azure.net/keys/<object-name>/<object-version>'\n```\n
The key can also be specified using the --key flag
"},{"location":"references/#creating-a-secret","title":"Creating a secret","text":"Secrets can be created using any of the methods described in the \"creating your secret\" section.
For example, if the key is defined in the prod target file
```shell\necho \"my_encrypted_secret\" | kapitan refs --write azkms:path/to/secret_inside_kapitan -t prod -f -\n```\n
Using the --key flag and a key_id
```shell\necho \"my_encrypted_secret\" | kapitan refs --write azkms:path/to/secret_inside_kapitan --key=<key_id> -f -\n```\n
"},{"location":"references/#referencing-and-revealing-a-secret","title":"Referencing and revealing a secret","text":"Secrets can be referenced and revealed in any of the ways described above.
For example, to reveal the secret stored at path/to/secret_inside_kapitan
```shell\nkapitan refs --reveal --tag \"?{azkms:path/to/secret_inside_kapitan}\"\n```\n
Note: Cryptographic algorithm used for encryption is rsa-oaep-256.
"},{"location":"related/","title":"Related projects","text":" - Tesoro - Kubernetes Admission Controller for Kapitan Secrets
- Kapitan Reference - Reference repository to get started
- sublime-jsonnet-syntax - Jsonnet syntax highlighting for Sublime Text
- language-jsonnet - Jsonnet syntax highlighting for Atom
- vim-jsonnet - Jsonnet plugin for Vim (requires a vim plugin manager)
"},{"location":"support/","title":"Get support with Kapitan","text":""},{"location":"support/#community","title":"Community","text":" - Join us on kubernetes.slack.com
#kapitan(Get invited) - Follow us on Twitter @kapitandev.
- Website
https://kapitan.dev - Mailing List kapitan-discuss@googlegroups.com(Subscribe)
"},{"location":"support/#resources","title":"Resources","text":" - Main Blog, articles and tutorials: Kapitan Blog
- Generators and reference kapitan repository: Kapitan Reference
- Kapitan Reference: our reference repository to get started with Kapitan.
"},{"location":"kap_proposals/kap_0_kadet/","title":"Kadet","text":"This introduces a new experimental input type called Kadet.
Kadet is essentially a Python module offering a set of classes and functions to define objects which will compile to JSON or YAML. A complete example is available in examples/kubernetes/components/nginx.
Author: @ramaro
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#overview","title":"Overview","text":"","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#baseobj","title":"BaseObj","text":"BaseObj implements the basic object implementation that compiles into JSON or YAML. Setting keys in self.root means they will be in the compiled output. Keys can be set as an hierarchy of attributes (courtesy of addict) The self.body() method is reserved for setting self.root on instantiation:
The example below:
class MyApp(BaseObj):\n def body(self):\n self.root.name = \"myapp\"\n self.root.inner.foo = \"bar\"\n self.root.list = [1, 2, 3]\n
compiles into:
---\nname: myapp\ninner:\n foo: bar\nlist:\n - 1\n - 2\n - 3\n
The self.new() method can be used to define a basic constructor. self.need() checks if a key is set and errors if it isn't (with an optional custom error message). kwargs that are passed onto a new instance of BaseObj are always accessible via self.kwargs In this example, MyApp needs name and foo to be passed as kwargs.
class MyApp(BaseObj):\n def new(self):\n self.need(\"name\")\n self.need(\"foo\", msg=\"please provide a value for foo\")\n\n def body(self):\n self.root.name = self.kwargs.name\n self.root.inner.foo = self.kwargs.foo\n self.root.list = [1, 2, 3]\n\nobj = MyApp(name=\"myapp\", foo=\"bar\")\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#setting-a-skeleton","title":"Setting a skeleton","text":"Defining a large body with Python can be quite hard and repetitive to read and write. The self.update_root() method allows importing a YAML/JSON file to set the skeleton of self.root.
MyApp's skeleton can be set instead like this:
#skel.yml\n---\nname: myapp\ninner:\n foo: bar\nlist:\n - 1\n - 2\n - 3\n
class MyApp(BaseObj):\n def new(self):\n self.need(\"name\")\n self.need(\"foo\", msg=\"please provide a value for foo\")\n self.update_root(\"path/to/skel.yml\")\n
Extending a skeleton'd MyApp is possible just by implementing self.body():
class MyApp(BaseObj):\n def new(self):\n self.need(\"name\")\n self.need(\"foo\", msg=\"please provide a value for foo\")\n self.update_root(\"path/to/skel.yml\")\n\n def body(self):\n self.set_replicas()\n self.root.metadata.labels = {\"app\": \"mylabel\"}\n\ndef set_replicas(self):\n self.root.spec.replicas = 5\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#inheritance","title":"Inheritance","text":"Python inheritance will work as expected:
class MyOtherApp(MyApp):\n def new(self):\n super().new() # MyApp's new()\n self.need(\"size\")\n\ndef body(self):\n super().body() # we want to extend MyApp's body\n self.root.size = self.kwargs.size\n del self.root.list # get rid of \"list\"\n\nobj = MyOtherApp(name=\"otherapp1\", foo=\"bar2\", size=3)\n
compiles to:
---\nname: otherapp1\ninner:\n foo: bar2\nreplicas: 5\nsize: 3\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#components","title":"Components","text":"A component in Kadet is a python module that must implement a main() function returning an instance ofBaseObj. The inventory is also available via the inventory() function.
For example, a tinyapp component:
# components/tinyapp/__init__.py\nfrom kapitan.inputs.kadet import BaseOBj, inventory\ninv = inventory() # returns inventory for target being compiled\n\nclass TinyApp(BaseObj):\n def body(self):\n self.root.foo = \"bar\"\n self.root.replicas = inv.parameters.tinyapp.replicas\n\ndef main():\n obj = BaseOb()\n obj.root.deployment = TinyApp() # will compile into deployment.yml\n return obj\n
An inventory class must be created for tinyapp:
# inventory/classes/components/tinyapp.yml\n\nparameters:\n tinyapp:\n replicas: 1\n kapitan:\n compile:\n - output_path: manifests\n input_type: kadet\n output_type: yaml\n input_paths:\n - components/tinyapp\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#common-components","title":"Common components","text":"A library in --search-paths (which now defaults to . and lib/) can also be a module that kadet components import. It is loaded using the load_from_search_paths():
kubelib = load_from_search_paths(\"kubelib\") # lib/kubelib/__init__.py\n\ndef main():\n obj = BaseObj()\n obj.root.example_app_deployment = kubelib.Deployment(name=\"example-app\")\n return obj\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_10_azure_key_vault/","title":"Support for Azure Key Management","text":"This feature will enable users to encrypt secrets using keys stored in Azure's Key Vault. The azkms keyword will be used to access the azure key management backend.
"},{"location":"kap_proposals/kap_10_azure_key_vault/#specification","title":"Specification","text":"key_id uniquely identifies an Azure key object and it's version stored in Key Vault. It is of the form https://{keyvault-name}.vault.azure.net/{object-type}/{object-name}/{object-version}. It needs to be made accessible to kapitan in one of the following ways:
parameters:\n kapitan:\n secrets:\n azkms:\n key: key_id #eg https://kapitanbackend.vault.azure.net/keys/myKey/deadbeef\n
kapitan refs --key=<key_id> --write azkms:/path/to/secret -f file_with_secret_data.txt\n
"},{"location":"kap_proposals/kap_10_azure_key_vault/#using-a-key-to-encrypt-a-secret","title":"Using a key to encrypt a secret","text":"The following command will be used to encrypt a secret (using the specified key from Key Vault) and save it in the refs-path along with it's metadata
echo \"my_treasured_secret\" | kapitan refs --write azkms:path/to/secret_inside_kapitan -t <target_name> -f -\n
The -t <target_name> is used to get the information about key_id.
Once the secret is Base64 encoded and encrypted using the key, it will be stored in path/to/secret_inside_kapitan as
data: bXlfdHJlYXN1cmVkX3NlY3JldAo=\nencoding: original\nkey: https://kapitanbackend.vault.azure.net/keys/myKey/deadbeef\ntype: azkms\n
note Cryptographic algorithm used for encryption would be rsa-oaep-256. Optimal Asymmetric Encryption Padding (OAEP) is a padding scheme often used together with RSA encryption.
"},{"location":"kap_proposals/kap_10_azure_key_vault/#referencing-a-secret","title":"referencing a secret","text":"Secrets can be refered using ?{azkms:path/to/secret_id} e.g.
parameter:\n mysql:\n storage: 10G\n storage_class: standard\n image: mysql:latest\n users:\n root:\n password: ?{azkms:path/to/secret}\n
"},{"location":"kap_proposals/kap_10_azure_key_vault/#revealing-a-secret","title":"Revealing a secret","text":"After compilation, the secret reference will be postfixed with 8 characters from the sha256 hash of the retrieved password/secret
apiVersion: v1\ndata:\n MYSQL_ROOT_PASSWORD: ?{azkms:path/to/secret:deadbeef}\nkind: Secret\nmetadata:\n labels:\n name: example-mysql\n name: example-mysql\n namespace: minikube-mysql\ntype: Opaque\n
To reveal the secret, the following command will be used $ kapitan ref --reveal -f compiled/file/containing/secret
"},{"location":"kap_proposals/kap_10_azure_key_vault/#dependencies","title":"Dependencies","text":" - azure-keyvault-keys
- azure-identity
note Kapitan will not be responsible for authentication or access management to Azure
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/","title":"Hashicorp Vault Transit","text":"This feature allows the user to fetch secrets from Hashicorp Vault, with the new secret backend keyword 'vaulttransit'.
Author: @xqp @Moep90
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/#specification","title":"Specification","text":"The following variables need to be exported to the environment(depending on authentication used) where you will run kapitan refs --reveal in order to authenticate to your HashiCorp Vault instance:
- VAULT_ADDR: URL for vault
- VAULT_SKIP_VERIFY=true: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing
- VAULT_TOKEN: token for vault or file (~/.vault-tokens)
- VAULT_ROLE_ID: required by approle
- VAULT_SECRET_ID: required by approle
- VAULT_USERNAME: username to login to vault
- VAULT_PASSWORD: password to login to vault
- VAULT_CLIENT_KEY: the path to an unencrypted PEM-encoded private key matching the client certificate
- VAULT_CLIENT_CERT: the path to a PEM-encoded client certificate for TLS authentication to the Vault server
- VAULT_CACERT: the path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate
- VAULT_CAPATH: the path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate
- VAULT_NAMESPACE: specify the Vault Namespace, if you have one
Considering any stringdata like any.value:whatever-you_may*like ( in our case let\u2019s encrypt any.value:whatever-you_may*like with vault transit ) using the key 2022-02-13-test in a transit secret engine with mount mytransit on the vault server, to use this as a secret either follow:
echo \"any.value:whatever-you_may*like\" > somefile.txt\nkapitan refs --write vaulttransit:<target_name>/to/secret_inside_kapitan --file somefile.txt --target <target_name>\n
or in a single line
echo \"any.value:whatever-you_may*like\" | kapitan refs --write vaulttransit:<target_name>/to/secret_inside_kapitan -t <target_name> -f -\n
The entire string \"any.value:whatever-you_may*like\" will be encrypted by vault and looks like this in return: vault:v2:Jhn3UzthKcJ2s+sEiO60EUiDmuzqUC4mMBWp2Vjg/DGl+GDFEDIPmAQpc5BdIefkplb6yrJZq63xQ9s=. This then gets base64 encoded and stored in the secret_inside_kapitan. Now secret_inside_kapitan contains the following
data: dmF1bHQ6djI6SmhuM1V6dGhLY0oycytzRWlPNjBFVWlEbXV6cVVDNG1NQldwMlZqZy9ER2wrR0RGRURJUG1BUXBjNUJkSWVma3BsYjZ5ckpacTYzeFE5cz0=\nencoding: original\ntype: vaulttransit\nvault_params:\n VAULT_ADDR: http://127.0.0.1:8200\n VAULT_SKIP_VERIFY: 'True'\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n auth: token\n crypto_key: key\n mount: transit\n always_latest: false\n
Encoding tells the type of data given to kapitan, if it is original then after decoding base64 we'll get the original secret and if it is base64 then after decoding once we still have a base64 encoded secret and have to decode again. Parameters in the secret file are collected from the inventory of the target we gave from CLI --target my_target. If target isn't provided then kapitan will identify the variables from the environment, but providing auth is necessary as a key inside target parameters like the one shown:
parameters:\n kapitan:\n vars:\n target: my_target\n namespace: my_namespace\n secrets:\n vaulttransit:\n VAULT_ADDR: http://vault.example.com:8200\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n VAULT_SKIP_VERIFY: \"True\"\n auth: token\n mount: transit\n crypto_key: new_key\n always_latest: False\n
Environment variables that can be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=alpha-secret/foo/bar then mount: alpha-secret (default secret)crypto_key: Name of the encryption key defined in vaultalways_latest: Always rewrap ciphertext to latest rotated crypto_key version Environment variables should NOT be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID. This makes the secret_inside_kapitan file accessible throughout the inventory, where we can use the secret whenever necessary like ?{vaulttransit:${target_name}/secret_inside_kapitan}
Following is the example file having a secret and pointing to the vault ?{vaulttransit:${target_name}/secret_inside_kapitan}
parameters:\n releases:\n app_version: latest\n app:\n image: app:app-tag\n release: ${releases:app_version}\n replicas: ${replicas}\n args:\n - --verbose=${verbose}\n - --password=?{vaulttransit:${target_name}/secret_inside_kapitan||random:str}\n
when ?{vaulttransit:${target_name}/secret_inside_kapitan} is compiled, it will look same with an 8 character prefix of sha256 hash added at the end like:
kind: Deployment\nmetadata:\n name: app\n namespace: my_namespace\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: app\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=?{vaulttransit:${target_name}/secret_inside_kapitan||random:str}\n image: app:app-tag\n name: app\n
Only the user with the required tokens/permissions can reveal the secrets. Please note that the roles and permissions will be handled at the Vault level. We need not worry about it within Kapitan. Use the following command to reveal the secrets:
kapitan refs --reveal -f compile/file/containing/secret\n
Following is the result of the app-deployment.md file after Kapitan reveal.
kind: Deployment\nmetadata:\n name: app\n namespace: my_namespace\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: app\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=\"any.value:whatever-you_may*like\"\n image: app:app-tag\n name: app\n
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/#vault-policies","title":"Vault policies","text":"path \"mytransit/encrypt/2022-02-13-test\" {\n capabilities = [ \"create\", \"update\" ]\n}\n\npath \"mytransit/decrypt/2022-02-13-test\" {\n capabilities = [ \"create\", \"update\" ]\n}\n
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/#dependencies","title":"Dependencies","text":" - hvac is a python client for Hashicorp Vault
"},{"location":"kap_proposals/kap_1_external_dependencies/","title":"External dependencies","text":"This features allows kapitan to fetch files from online repositories/sources during compile and store in a particular target directory.
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_1_external_dependencies/#specification","title":"Specification","text":"Specify the files to be fetched as follows:
parameters:\n kapitan:\n dependencies:\n - type: git | http[s]\n output_path: <output_path>\n source: <git/http[s]_url>\n
The output path is the path to save the dependency into. For example, it could be /components/external/manifest.jsonnet. Then, the user can specify the fetched file as a kapitan.compile item along with the locally-created files.
Git type may also include ref and subdir parameters as illustrated below:
- type: git\n output_path: <output_path>\n source: <git_url>\n subdir: relative/path/in/repository\n ref: <commit_hash/branch/tag>\n force_fetch: <bool>\n
If the file already exists at output_path, the fetch will be skipped. For fresh fetch of the dependencies, users may add --fetch option as follows:
kapitan compile --fetch\n
Users can also add the force_fetch: true option to the kapitan.dependencies in the inventory in order to force fetch of the dependencies of the target every time.
"},{"location":"kap_proposals/kap_1_external_dependencies/#implementation-details","title":"Implementation details","text":""},{"location":"kap_proposals/kap_1_external_dependencies/#dependencies","title":"Dependencies","text":" - GitPython module (and git executable) for git type
- requests module for http[s]
- (optional) tqdm for reporting download progress
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/","title":"Helm Charts Input Type","text":"This will allow kapitan, during compilation, to overwrite the values in user-specified helm charts using its inventory by calling the Go & Sprig template libraries. The helm charts can be specified via local path, and users may download the helm chart via external-dependency feature (of http[s] type).
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#specification","title":"Specification","text":"This feature basically follows the helm template command available. This will run after the fetching of the external dependencies takes place, such that users can simultaneously specify the fetch as well as the import of a helm chart dependency.
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#semantics","title":"Semantics","text":"kapitan:\n compile:\n - input_type: helm\n input_path: <path_to_chart_dir>\n output_path: <output_path>\n set-file:\n - <optional_file_path>\n - ...\n values_file: <optional_values_file>\n namespace: <optional_namespace>\n
This mostly maps to the options available to helm template command (refer to here).
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#implementation-details","title":"Implementation details","text":"C-binding between Helm (Go) and Kapitan (Python) will be created. Helm makes use of two template libraries, namely, text/template and Sprig. The code for helm template command will be converted into shared object (.so) using CGo, which exposes C interface that kapitan (i.e. CPython) could use. The source code for helm template command is found here. This file will be modified to
- remove redundant options
- expose C-interface for Kapitan
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#dependencies","title":"Dependencies","text":""},{"location":"kap_proposals/kap_3_schema_validation/","title":"Schema Validation (for k8s)","text":"If a yaml/json output is to be used as k8s manifest, users may specify its kind and have kapitan validate its structure during kapitan compile. The plan is to have this validation feature extendable to other outputs as well, such as terraform.
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_3_schema_validation/#specification","title":"Specification","text":"The following inventory will validate the structure of Kubernetes Service manifest file in .
parameters:\n kapitan:\n validate:\n - output_type: kubernetes.service\n version: 1.6.6\n output_path: relative/path/in/target\n
version parameter is optional: if omitted, the version will be set to the stable release of kubernetes (tbc).
"},{"location":"kap_proposals/kap_3_schema_validation/#implementation","title":"Implementation","text":" - The schemas will be downloaded by requests from this repository.
- Caching of schema will also be implemented.
"},{"location":"kap_proposals/kap_3_schema_validation/#dependencies","title":"Dependencies","text":" - jsonschema to validate the output yaml/json against the correct schema
"},{"location":"kap_proposals/kap_4_standalone_executable/","title":"Standalone Kapitan Executable (Discontinued)","text":"Create a portable (i.e. static) kapitan binary for users. This executable will be made available for each release on Github. The target/tested platform is Debian 9 (possibly Windows to be supported in the future).
Criteria:
- speed of the resulting binary
- size of the resulting binary
- portability of the binary (single-file executable or has an accompanying folder)
- cross-platform
- actively maintained
- supports Python 3.6, 3.7
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_4_standalone_executable/#tools-to-be-explored","title":"Tools to be explored","text":" - (tentative first-choice) Pyinstaller
- (Alternative) nuitka (also part of GSoC 2019. It might soon support single-file executable output).
"},{"location":"kap_proposals/kap_5_ref_types_redesign/","title":"Ref Types Redesign","text":"Redesign Kapitan Secrets and rename them as References or Ref.
Breaking changes:
$ kapitan secrets is replaced with $ kapitan refs- the default secrets directory
./secrets/ changes to ./refs/ - the
--secrets-path flag changes to --refs-path - ref ref type is renamed to base64 e.g.
?{ref:some/ref} into ?{base64:some/ref}
Status: In progress
Author: @ramaro
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#proposal","title":"Proposal","text":"Rename Secrets into Ref (or References) to improve consistency and meaning of the backend types by removing the ref backend and introducting new backends:
Type Description Encrypted? Compiles To gpg GnuPG Yes hashed tag gkms Google KMS Yes hashed tag awskms Amazon KMS Yes hashed tag base64 base64 No hashed tag plain plain text No plain text The type value will now need to be representative of the way a reference is stored via its backend.
A new plain backend type is introduced and will compile into revealed state instead of a hashed tag.
A new base64 backend type will store a base64 encoded value as the backend suggests (replacing the old badly named ref backend).
The command line for secrets will be instead:
kapitan refs --write gpg:my/secret1 ...\nkapitan refs --write base64:my/file ...\nkapitan refs --write plain:my/info ...\n
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#plain-backend","title":"plain backend","text":"The plain backend type will allow referring to external state by updating refs programmatically (e.g. in your pipeline)
For example, one can update the value of an environment variable and use ?{plain:my/user} as a reference in a template:
echo $USER | kapitan refs --write plain:my/user -f -\n
Or update a docker image value as ref ?{plain:images/dev/envoy}:
echo 'envoyproxy/envoy:v1.10.0' | kapitan refs --write plain:images/dev/envoy -f -\n
These references will be compiled into their values instead of hashed tags.
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#base64-backend","title":"base64 backend","text":"The base64 backend type will function as the original ref type. Except that this time, the name is representative of what is actually happening :)
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#refs-path","title":"Refs path","text":"Refs will be stored by default in the ./refs path set by --refs-path replacing the --secrets-path flag.
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#background","title":"Background","text":""},{"location":"kap_proposals/kap_5_ref_types_redesign/#kapitan-secrets","title":"Kapitan Secrets","text":"Kapitan Secrets allow referring to restricted information (passwords, private keys, etc...) in templates while also securely storing them.
On compile, secret tags are updated into hashed tags which validate and instruct Kapitan how to reveal tags into decrypted or encoded information.
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#kapitan-secrets-example","title":"Kapitan Secrets example","text":"The following command creates a GPG encrypted secret with the contents of file.txt for recipient ramaro@google.com to read:
kapitan secrets --write gpg:my/secret1 -f file.txt --recipients ramaro@google.com\n
This secret can be referred to in a jsonnet compoment:
{\n \"type\": \"app\",\n \"name\": \"test_app\",\n \"username\": \"user_one\",\n \"password\": \"?{gpg:my/secret1}\"\n}\n
When this compoment is compiled, it looks like (note the hashed tag):
type: app\nname: test_app\nusername: user_one\npassword: ?{gpg:my/secret1:deadbeef}\n
A user with the required permissions can reveal the compiled component:
$ kapitan secrets --reveal -f compiled/mytarget/manifests/component.yml\n\ntype: app\nname: test_app\nusername: user_one\npassword: secret_content_of_file.txt\n
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#secret-backend-comparison","title":"Secret Backend Comparison","text":"Kapitan today offers multiple secret backends:
Type Description Encrypted? Compiles To gpg GnuPG Yes hashed tag gkms Google KMS Yes hashed tag awskms Amazon KMS Yes hashed tag ref base64 No hashed tag However, not all backends are encrypted - this is not consistent!
The ref type is not encrypted as its purpose is to allow getting started with the Kapitan Secrets workflow without the need of setting up the encryption backends tooling (gpg, gcloud, boto, etc...)
"},{"location":"kap_proposals/kap_6_hashicorp_vault/","title":"Hashicorp Vault","text":"This feature allows the user to fetch secrets from Hashicorp Vault, with the new secret backend keyword 'vaultkv'.
Author: @vaibahvk @daminisatya
"},{"location":"kap_proposals/kap_6_hashicorp_vault/#specification","title":"Specification","text":"The following variables need to be exported to the environment(depending on authentication used) where you will run kapitan refs --reveal in order to authenticate to your HashiCorp Vault instance:
- VAULT_ADDR: URL for vault
- VAULT_SKIP_VERIFY=true: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing
- VAULT_TOKEN: token for vault or file (~/.vault-tokens)
- VAULT_ROLE_ID: required by approle
- VAULT_SECRET_ID: required by approle
- VAULT_USERNAME: username to login to vault
- VAULT_PASSWORD: password to login to vault
- VAULT_CLIENT_KEY: the path to an unencrypted PEM-encoded private key matching the client certificate
- VAULT_CLIENT_CERT: the path to a PEM-encoded client certificate for TLS authentication to the Vault server
- VAULT_CACERT: the path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate
- VAULT_CAPATH: the path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate
- VAULT_NAMESPACE: specify the Vault Namespace, if you have one
Considering a key-value pair like my_key:my_secret ( in our case let\u2019s store hello:batman inside the vault ) in the path secret/foo in a kv-v2(KV version 2) secret engine on the vault server, to use this as a secret either follow:
echo \"foo:hello\" > somefile.txt\nkapitan refs --write vaultkv:path/to/secret_inside_kapitan --file somefile.txt --target dev-sea\n
or in a single line
echo \"foo:hello\" | kapitan refs --write vaultkv:path/to/secret_inside_kapitan -t dev-sea -f -\n
The entire string \"foo:hello\" is base64 encoded and stored in the secret_inside_kapitan. Now secret_inside_kapitan contains the following
data: Zm9vOmhlbGxvCg==\nencoding: original\ntype: vaultkv\nvault_params:\n auth: token\n
Encoding tells the type of data given to kapitan, if it is original then after decoding base64 we'll get the original secret and if it is base64 then after decoding once we still have a base64 encoded secret and have to decode again. Parameters in the secret file are collected from the inventory of the target we gave from CLI --target dev-sea. If target isn't provided then kapitan will identify the variables from the environment, but providing auth is necessary as a key inside target parameters like the one shown:
parameters:\n kapitan:\n secrets:\n vaultkv:\n auth: userpass\n engine: kv-v2\n mount: team-alpha-secret\n VAULT_ADDR: http://127.0.0.1:8200\n VAULT_NAMESPACE: CICD-alpha\n VAULT_SKIP_VERIFY: false\n VAULT_CLIENT_KEY: /path/to/key\n VAULT_CLIENT_CERT: /path/to/cert\n
Environment variables that can be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=alpha-secret/foo/bar then mount: alpha-secret (default secret)engine: secret engine used, either kv-v2 or kv (default kv-v2) Environment variables cannot be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID. This makes the secret_inside_kapitan file accessible throughout the inventory, where we can use the secret whenever necessary like ?{vaultkv:path/to/secret_inside_kapitan}
Following is the example file having a secret and pointing to the vault ?{vaultkv:path/to/secret_inside_kapitan}
parameters:\n releases:\n cod: latest\n cod:\n image: alledm/cod:${cod:release}\n release: ${releases:cod}\n replicas: ${replicas}\n args:\n - --verbose=${verbose}\n - --password=?{vaultkv:path/to/secret_inside_kapitan}\n
when ?{vaultkv:path/to/secret_inside_kapitan} is compiled, it will look same with an 8 character prefix of sha256 hash added at the end like:
kind: Deployment\nmetadata:\n name: cod\n namespace: dev-sea\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: cod\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=?{vaultkv:path/to/secret_inside_kapitan:57d6f9b7}\n image: alledm/cod:v2.0.0\n name: cod\n
Only the user with the required tokens/permissions can reveal the secrets. Please note that the roles and permissions will be handled at the Vault level. We need not worry about it within Kapitan. Use the following command to reveal the secrets:
kapitan refs --reveal -f compile/file/containing/secret\n
Following is the result of the cod-deployment.md file after Kapitan reveal.
kind: Deployment\nmetadata:\n name: cod\n namespace: dev-sea\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: cod\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=batman\n image: alledm/cod:v2.0.0\n name: cod\n
"},{"location":"kap_proposals/kap_6_hashicorp_vault/#dependencies","title":"Dependencies","text":" - hvac is a python client for Hashicorp Vault
"},{"location":"kap_proposals/kap_7_remote_inventory/","title":"Remote Inventory Federation","text":"This feature would add the ability to Kapitan to fetch parts of the inventory from remote locations (https/git). This would allow users to combine different inventories from different sources and build modular infrastructure reusable across various repos.
Author: @alpharoy14
"},{"location":"kap_proposals/kap_7_remote_inventory/#specification","title":"Specification","text":"The configuration and declaration of remote inventories would be done in the inventory files.
The file specifications are as follows:
parameters:\n kapitan:\n inventory:\n - type: <inventory_type> #git\\https\n source: <source_of_inventory>\n output_path: <relative_output_path>\n
On executing the $ kapitan compile --fetch command, first the remote inventories will be fetched followed by fetching of external dependencies and finally merge the inventory to compile.
"},{"location":"kap_proposals/kap_7_remote_inventory/#copying-inventory-files-to-the-output-location","title":"Copying inventory files to the output location","text":"The output path is the path to save the inventory items into. The path is relative to the inventory/ directory. For example, it could be /classes/. The contents of the fetched inventory will be recursively copied.
The fetched inventory files will be cached in the .dependency_cache directory if --cache is set. eg. $ kapitan compile --fetch --cache
"},{"location":"kap_proposals/kap_7_remote_inventory/#force-fetching","title":"Force fetching","text":"While fetching, the output path will be recursively checked to see if it contains any file with the same name. If so, kapitan will skip fetching it.
To overwrite the files with the newly downloaded inventory items, we can add the --force-fetch flag to the compile command, as shown below.
$ kapitan compile --force-fetch
"},{"location":"kap_proposals/kap_7_remote_inventory/#url-type","title":"URL type","text":"The URL type can be either git or http(s). Depending on the URL type, the configuration file may have additional arguments.
E.g Git type may also include aditional ref parameter as illustrated below:
inventory:\n - type: git #git\\https\n source: <source_of_inventory>\n output_path: <output_path>\n ref: <commit_hash/branch/tag>\n
"},{"location":"kap_proposals/kap_7_remote_inventory/#implementation-details","title":"Implementation details","text":"TODO
"},{"location":"kap_proposals/kap_7_remote_inventory/#dependencies","title":"Dependencies","text":" - GitPython module (and git executable) for git type
- requests module for http[s]
"},{"location":"kap_proposals/kap_8_google_secret_management/","title":"Support for Google Secret Manager","text":"This feature will enable users to retrieve secrets from Google Secret Manager API using the gsm keyword.
"},{"location":"kap_proposals/kap_8_google_secret_management/#specification","title":"Specification","text":"project_id uniquely identifies GCP projects, and it needs to be made accessible to kapitan in one of the following ways:
parameters:\n kapitan:\n secrets:\n gsm:\n project_id: Project_Id\n
kapitan refs --google-project-id=<Project_Id> --write gsm:/path/to/secret_id -f secret_id_file.txt\n
- As an environment variable
export PROJECT_ID=<Project_Id>\n
"},{"location":"kap_proposals/kap_8_google_secret_management/#using-a-secret","title":"Using a secret","text":"In GCP, a secret contains one or more secret versions, along with its metadata. The actual contents of a secret are stored in a secret version. Each secret is identified by a name. We call that variable secret_id e.g. my_treasured_secret. The URI of the secret becomes projects/<Project_Id>/secrets/my_treasured_secret
The following command will be used to add a secret_id to kapitan.
echo \"my_treasured_secret\" | kapitan refs --write gsm:path/to/secret_inside_kapitan -t <target_name> -f -\n
The -t <target_name> is used to get the information about Project_ID.
The secret_id is Base64 encoded and stored in path/to/secret_inside_kapitan as
data: bXlfdHJlYXN1cmVkX3NlY3JldAo=\nencoding: original\ntype: gsm\ngsm_params:\n project_id: Project_ID\n
"},{"location":"kap_proposals/kap_8_google_secret_management/#referencing-a-secret","title":"referencing a secret","text":"Secrets can be refered using ?{gsm:path/to/secret_id:version_id} e.g.
parameter:\n mysql:\n storage: 10G\n storage_class: standard\n image: mysql:latest\n users:\n root:\n password: ?{gsm:path/to/secret_id:version_id}\n
Here, version_id will be an optional argument. By default it will point to latest.
"},{"location":"kap_proposals/kap_8_google_secret_management/#revealing-a-secret","title":"Revealing a secret","text":"After compilation, the secret reference will be postfixed with 8 characters from the sha256 hash of the retrieved password
apiVersion: v1\ndata:\n MYSQL_ROOT_PASSWORD: ?{gsm:path/to/secret_id:version_id:deadbeef}\nkind: Secret\nmetadata:\n labels:\n name: example-mysql\n name: example-mysql\n namespace: minikube-mysql\ntype: Opaque\n
To reveal the secret, the following command will be used $ kapitan ref --reveal -f compiled/file/containing/secret
"},{"location":"kap_proposals/kap_8_google_secret_management/#dependencies","title":"Dependencies","text":" - google-cloud-secret-manager
note Kapitan will not be responsible for authentication or access management to GCP
"},{"location":"kap_proposals/kap_8_modularize_kapitan/","title":"Modularize Kapitan","text":"Kapitan is packaged in PYPI and as a binary along with all its dependencies. Adding an extra key/security backend means that we need to ship another dependency with that PYPI package, making deploying changes more complicated. This project would modularize kapitan into core dependencies and extra modules.
"},{"location":"kap_proposals/kap_8_modularize_kapitan/#usage","title":"Usage","text":"pip3 install --user kapitan # to install only core dependencies\nPip3 install --user kapitan[gkms] \u200b# gkms is the module\n
"},{"location":"kap_proposals/kap_8_modularize_kapitan/#implementation","title":"Implementation","text":" - The main module includes the essential kapitan dependencies and reclass dependencies, which will be included in the \u200brequirement.txt\u200b file.
- The extra module pypi extras will be defined in the
s\u200betup.py\u200b file. - The extra dependencies are of secret backends like (AWS Key backend, Google KMS Key backend, Vault Key backend etc.) and Helm support.
"},{"location":"kap_proposals/kap_9_bring_your_own_helm/","title":"Bring Your Own Helm Proposal","text":""},{"location":"kap_proposals/kap_9_bring_your_own_helm/#the-problem","title":"The Problem","text":"Currently the helm binding can't be run on Mac OSX. Attempts to fix this have been made on several occasions:
- https://github.com/kapicorp/kapitan/pull/414
- https://github.com/kapicorp/kapitan/pull/547
- https://github.com/kapicorp/kapitan/pull/568
There are some issues with the current bindings besides the lack of Mac OSX support. The golang runtime (1.14) selected will effect older versions helm templates: https://github.com/helm/helm/issues/7711. Users can't select the version of helm they'd like to use for templating.
"},{"location":"kap_proposals/kap_9_bring_your_own_helm/#solution","title":"Solution","text":"Users supply their own helm binary. This allows them to control the version of golang runtime and version of helm they'd like to use.
In Kapitan we could rewrite the interface to use subprocess and perform commands. The cli of helm 2 vs helm 3 is slightly different but shouldn't be difficult to codify.
This would be great to get rid of cffi and golang which will reduce complexity and build time of the project.
Depending on how this goes, this could pave the way for a \"bring your own binary\" input type.
"},{"location":"pages/external_dependencies/","title":"External dependencies","text":"Kapitan has the functionality to fetch external dependencies from remote locations.
Supported dependencies types are:
"},{"location":"pages/external_dependencies/#usage","title":"Usage","text":"Kapitan by default will not attempt to download any dependency, and rely on what is already available.
"},{"location":"pages/external_dependencies/#basic-fetching","title":"Basic fetching","text":"You can use the fetch option to explicitly fetch the dependencies:
clidotfile kapitan compile --fetch\n
.kapitan
to make it default, then simply use kapitan compile
...\ncompile:\n fetch: true\n
This will download the dependencies and store them at their respective output_path.
"},{"location":"pages/external_dependencies/#overwrite-local-changes","title":"Overwrite local changes","text":"When fetching a dependency, Kapitan will refuse to overwrite existing files to preserve your local modifications.
Use the force-fetch option to force overwrite your local files in the output_path.
clidotfile kapitan compile --force-fetch\n
.kapitan
to make it default, then simply use kapitan compile
...\ncompile:\n force-fetch: true\n
"},{"location":"pages/external_dependencies/#caching","title":"Caching","text":"Kapitan also supports caching Use the --cache flag to cache the fetched items in the .dependency_cache directory in the root project directory.
```shell\nkapitan compile --cache --fetch\n```\n
"},{"location":"pages/external_dependencies/#defining-dependencies","title":"Defining dependencies","text":"githttphelm"},{"location":"pages/external_dependencies/#syntax","title":"Syntax","text":"parameters:\n kapitan:\n dependencies:\n - type: git\n output_path: path/to/dir\n source: git_url # mkdocs (1)!\n subdir: relative/path/from/repo/root (optional) # mkdocs (2)!\n ref: tag, commit, branch etc. (optional) # mkdocs (3)!\n submodules: true/false (optional) # mkdocs (4)!\n
- Git types can fetch external
git repositories through either HTTP/HTTPS or SSH URLs. - Optional supports for cloning just a sub-directory
- Optional support for accessing them in specific commits and branches (refs).
- Optional support to disable fetching the submodules of a repo.
Note
This type depends on the git binary installed on your system and available to Kapitan.
"},{"location":"pages/external_dependencies/#example","title":"Example","text":"Say we want to fetch the source code from our kapitan repository, specifically, kapicorp/kapitan/kapitan/version.py. Let's create a very simple target file inventory/targets/kapitan-example.yml.
parameters:\n kapitan:\n vars:\n target: kapitan-example\n dependencies:\n - type: git\n output_path: source/kapitan\n source: git@github.com:kapicorp/kapitan.git\n subdir: kapitan\n ref: master\n submodules: true\n compile:\n - input_paths:\n - source/kapitan/version.py\n input_type: jinja2 # just to copy the file over to target\n output_path: .\n
"},{"location":"pages/external_dependencies/#syntax_1","title":"Syntax","text":"parameters:\n kapitan:\n dependencies:\n - type: http | https # mkdocs (2)!\n output_path: path/to/file # mkdocs (1)!\n source: http[s]://<url> # mkdocs (2)!\n unpack: True | False # mkdocs (3)!\n
output_path must fully specify the file name. For example:- http[s] types can fetch external dependencies available at
http:// or https:// URL. - archive mode: download and unpack
"},{"location":"pages/external_dependencies/#example_1","title":"Example","text":"Single fileArchive Say we want to download kapitan README.md file. Since it's on Github, we can access it as https://raw.githubusercontent.com/kapicorp/kapitan/master/README.md. Using the following inventory, we can copy this to our target folder:
parameters:\n kapitan:\n vars:\n target: kapitan-example\n dependencies:\n - type: https\n output_path: README.md\n source: https://raw.githubusercontent.com/kapicorp/kapitan/master/README.md\n compile:\n - input_paths:\n - README.md\n input_type: jinja2\n output_path: .\n
"},{"location":"pages/external_dependencies/#syntax_2","title":"Syntax","text":"parameters:\n kapitan:\n dependencies:\n - type: helm\n output_path: path/to/chart\n source: http[s]|oci://<helm_chart_repository_url>\n version: <specific chart version>\n chart_name: <name of chart>\n helm_path: <helm binary>\n
Fetches helm charts and any specific subcharts in the requirements.yaml file.
helm_path can be used to specify where the helm binary name or path. It defaults to the value of the KAPITAN_HELM_PATH environment var or simply to helm if neither is set. You should specify only if you don't want the default behavior.
source can be either the URL to a chart repository, or the URL to a chart on an OCI registry (supported since Helm 3.8.0).
"},{"location":"pages/external_dependencies/#example_2","title":"Example","text":"If we want to download the prometheus helm chart we simply add the dependency to the monitoring target. We want a specific version 11.3.0 so we put that in.
parameters:\n kapitan:\n vars:\n target: monitoring\n dependencies:\n - type: helm\n output_path: charts/prometheus\n source: https://kubernetes-charts.storage.googleapis.com\n version: 11.3.0\n chart_name: prometheus\n compile:\n - input_type: helm\n output_path: .\n input_paths:\n - charts/prometheus\n helm_values:\n alertmanager:\n enabled: false\n helm_params:\n namespace: monitoring\n name: prometheus\n
"},{"location":"pages/kapitan_overview/","title":"Kapitan Overview","text":""},{"location":"pages/kapitan_overview/#kapitan-at-a-glance","title":"Kapitan at a glance","text":"Kapitan is a powerful configuration management tool designed to help engineers manage complex systems through code. It centralizes and simplifies the management of configurations with a structured approach that revolves around a few core concepts.
Kapitan diagram %%{ init: { securityLevel: 'loose'} }%%\ngraph LR\n classDef pink fill:#f9f,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef blue fill:#00FFFF,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n TARGET1 --> KAPITAN\n TARGET2 --> KAPITAN\n TARGETN --> KAPITAN\n KAPITAN --> EXTERNAL\n KAPITAN --> GENERATORS\n KAPITAN --> HELM\n KAPITAN --> JINJA\n KAPITAN --> JSONNET\n KAPITAN --> KADET\n EXTERNAL --> OUTPUT\n GENERATORS --> OUTPUT\n JINJA --> OUTPUT\n JSONNET --> OUTPUT\n KADET --> OUTPUT\n HELM --> OUTPUT\n GKMS --> REFERENCES\n AWSKMS --> REFERENCES\n VAULT --> REFERENCES\n OTHER --> REFERENCES\n PLAIN --> REFERENCES\n OUTPUT --> TARGETN_OUTPUT\n OUTPUT --> TARGET1_OUTPUT\n OUTPUT --> TARGET2_OUTPUT\n REFERENCES --> KAPITAN\n TARGET1_OUTPUT --> DOCUMENTATION\n TARGET1_OUTPUT --> KUBERNETES\n TARGET1_OUTPUT --> SCRIPTS\n TARGET1_OUTPUT --> TERRAFORM\n CLASSES --> TARGET1\n CLASSES --> TARGET2\n CLASSES --> TARGETN\n\n subgraph \"Inventory\"\n CLASSES[classes]\n TARGET1([\"target 1\"]):::pink\n TARGET2([\"target 2\"])\n TARGETN([\"target N\"])\n end\n\n subgraph \"references\"\n direction TB\n GKMS[\"GCP KMS\"]\n AWSKMS[\"AWS KMS\"]\n VAULT[\"Hashicorp Vault\"]\n OTHER[\"others\"]\n PLAIN[\"plain\"]\n REFERENCES[\"references\"]\n end\n\n KAPITAN((\"<img src='/images/kapitan_logo.png'; width='80'/>\")):::blue\n click EXTERNAL \"/compile#external\"\n\n subgraph \"Input Types\"\n EXTERNAL[\"external\"]\n GENERATORS[\"generators\"]\n HELM[\"helm\"]\n JINJA[\"jinja\"]\n JSONNET[\"jsonnet\"]\n KADET[\"kadet\"]\n end\n\n OUTPUT{{\"compiled output\"}}:::blue\n\n\n\n subgraph \" \"\n TARGET1_OUTPUT([target1]):::pink\n DOCUMENTATION[\"docs\"]\n KUBERNETES[\"manifests\"]\n SCRIPTS[\"scripts\"]\n TERRAFORM[\"terraform\"]\n end\n\n TARGET2_OUTPUT([\"target 2\"])\n TARGETN_OUTPUT([\"target N\"])\n
Let's explore these concepts in a way that's accessible to new users:
"},{"location":"pages/kapitan_overview/#inventory","title":"Inventory","text":"At the core of Kapitan lies the Inventory, a structured database of variables meticulously organized in YAML files. This hierarchical setup serves as the single source of truth (SSOT) for your system's configurations, making it easier to manage and reference the essential components of your infrastructure. Whether you're dealing with Kubernetes configurations, Terraform resources, or even business logic, the Inventory allows you to define and store these elements efficiently. This central repository then feeds into Kapitan's templating engines, enabling seamless reuse across various applications and services.
"},{"location":"pages/kapitan_overview/#input-types","title":"Input Types","text":"Kapitan takes the information stored in the Inventory and brings it to life through its templating engines upon compilation. This process transforms static data into dynamic configurations, capable of generating a wide array of outputs like Kubernetes manifests, Terraform plans, documentation, and scripts. It's about making your configurations work for you, tailored to the specific needs of your projects.
See Input Types for more
"},{"location":"pages/kapitan_overview/#generators","title":"Generators","text":"Generators offer a straightforward entry point into using Kapitan, requiring minimal to no coding experience. These are essentially pre-made templates that allow you to generate common configuration files, such as Kubernetes manifests, directly from your Inventory data. Kapitan provides a wealth of resources, including the Kapitan Reference GitHub repository and various blog posts, to help users get up and running with generators.
"},{"location":"pages/kapitan_overview/#kadet","title":"Kadet","text":"For those looking to leverage the full power of Kapitan, Kadet introduces a method to define and reuse complex configurations through Python. This internal library facilitates the creation of JSON and YAML manifests programmatically, offering a higher degree of customization and reuse. Kadet empowers users to craft intricate configurations with the simplicity and flexibility of Python.
"},{"location":"pages/kapitan_overview/#references","title":"References","text":"Kapitan References provide a secure way to store passwords, settings, and other essential data within your project. Think of them as special code placeholders.
- Flexibility: Update a password once, and Kapitan updates it everywhere automatically.
- Organization: References tidy up your project, especially when you're juggling multiple settings or environments (dev, staging, production). Security: Protect sensitive information like passwords with encryption
Tip
Use Tesoro, our Kubernetes Admission Controller, to complete your integration with Kubernetes for secure secret decryption on-the-fly.
"},{"location":"pages/remote_repositories/","title":"Remote Inventories","text":"Kapitan is capable of recursively fetching inventory items stored in remote locations and copy it to the specified output path. This feature can be used by specifying those inventory items in classes or targets under parameters.kapitan.inventory. Supported types are:
Class items can be specified before they are locally available as long as they are fetched in the same run. Example of this is given below.
"},{"location":"pages/remote_repositories/#git-type","title":"Git type","text":"Git types can fetch external inventories available via HTTP/HTTPS or SSH URLs. This is useful for fetching repositories or their sub-directories, as well as accessing them in specific commits and branches (refs).
Note: git types require git binary on your system.
"},{"location":"pages/remote_repositories/#definition","title":"Definition","text":"parameters:\n kapitan:\n inventory:\n - type: git\n output_path: path/to/dir\n source: git_url\n subdir: relative/path/from/repo/root (optional)\n ref: tag, commit, branch etc. (optional)\n
"},{"location":"pages/remote_repositories/#example","title":"Example","text":"Lets say we want to fetch a class from our kapitan repository, specifically kapicorp/kapitan/tree/master/examples/docker/inventory/classes/dockerfiles.yml.
Lets create a simple target file docker.yml
Note
external dependencies are used to fetch dependency items in this example.
targets/docker.yml
classes:\n - dockerfiles\nparameters:\n kapitan:\n vars:\n target: docker\n inventory:\n - type: git\n source: https://github.com/kapicorp/kapitan\n subdir: examples/docker/inventory/classes/\n output_path: classes/\n dependencies:\n - type: git\n source: https://github.com/kapicorp/kapitan\n subdir: examples/docker/components\n output_path: components/\n - type: git\n source: https://github.com/kapicorp/kapitan\n subdir: examples/docker/templates\n output_path: templates/\n dockerfiles:\n - name: web\n image: amazoncorretto:11\n - name: worker\n image: amazoncorretto:8\n
kapitan compile --fetch\n
click to expand output [WARNING] Reclass class not found: 'dockerfiles'. Skipped!\n[WARNING] Reclass class not found: 'dockerfiles'. Skipped!\nInventory https://github.com/kapicorp/kapitan: fetching now\nInventory https://github.com/kapicorp/kapitan: successfully fetched\nInventory https://github.com/kapicorp/kapitan: saved to inventory/classes\nDependency https://github.com/kapicorp/kapitan: saved to components\nDependency https://github.com/kapicorp/kapitan: saved to templates\nCompiled docker (0.11s)\n
"},{"location":"pages/remote_repositories/#http-type","title":"http type","text":"http[s] types can fetch external inventories available at http:// or https:// URL.
"},{"location":"pages/remote_repositories/#definition_1","title":"Definition","text":"parameters:\n kapitan:\n inventory:\n - type: http | https\n output_path: full/path/to/file.yml\n source: http[s]://<url>\n unpack: True | False # False by default\n
"},{"location":"pages/remote_repositories/#example_1","title":"Example","text":"targets/mysql-generator-fetch.yml
classes:\n - common\n - kapitan.generators.kubernetes\nparameters:\n kapitan:\n inventory:\n - type: https\n source: https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml\n output_path: classes/kapitan/generators/kubernetes.yml\n components:\n mysql:\n image: mysql\n
kapitan compile --fetch\n
click to expand output ./kapitan compile -t mysql-generator-fetch --fetch\nInventory https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml: fetching now\nInventory https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml: successfully fetched\nInventory https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml: saved to inventory/classes/kapitan/generators/kubernetes.yml\n\n...\ncut\n...\n\nCompiled mysql-generator-fetch (0.06s)\n
"},{"location":"pages/blog/","title":"Blog","text":""},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/","title":"5 Years of Kapitan","text":"Last October we quietly celebrated 5 years of Kapitan.
In 5 years, we've been able to witness a steady and relentless of Kapitan, which has however never caught the full attention of the majority of the community.
The main issue has always been around an embarassing lack of documentation, and we've worked hard to improve on that, with more updates due soon.
Let this first blog post from a revamped website be a promise to our community of a better effort in explaining what sets Kapitan apart, and makes it the only tool of its kind.
And let's start with a simple question: Why do you even need Kapitan?
Credits
In reality Kapitan's heatbeat started about 9 months earlier at DeepMind Health, created by [**Ricardo Amaro**](https://github.com/ramaro) with the help of some of my amazing team: in no particular order [Adrian Chifor](https://github.com/adrianchifor), [Paul S](https://github.com/uberspot) and [Luis Buriola](https://github.com/gburiola). It was then kindly released to the community by Google/DeepMind and is has so been improved thanks to more than [50 contributors](https://github.com/kapicorp/kapitan/graphs/contributors).\n
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#why-do-i-need-kapitan","title":"Why do I need Kapitan?","text":"Kapitan is a hard sell, but a rewarding one. For these main reasons:
- Kapitan solves problems that some don\u2019t know/think to have.
- Some people by now have probably accepted the Status Quo and think that some suffering is part of their job descriptions.
- Objectively, Kapitan requires an investment of effort to learn how to use a new tool, and this adds friction.
All I can say it is very rewarding once you get to use it, so stick with me while I try to explain the problems that Kapitan is solving
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#the-problems","title":"The problems","text":"It would be reductive to list the problems that Kapitan solves, because sometimes we ourselves are stunned by what Kapitan is being used for, so I will start with some common relatable ones, and perhaps that will give you the right framing to understand how to use it with your setup.
In its most basic explanation, Kapitan solves the problem of avoiding duplication of configuration data: by consolidating it in one place (the Inventory), and making it accessible by all the tools and languages it integrates with (see Input Types).
This configuration data is then used by Kapitan (templates) to configure and operate a number of completely distinct and unaware tools which would normally not be able to share their configurations.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#without-kapitan","title":"Without Kapitan","text":"Let's consider the case where you want to define a new bucket, with a given bucket_name. Without Kapitan you would probably need to:
- Write a PR on your Terraform repository to create the new bucket.
- Which name should I use? Make sure to write it down!
CTRL-C - Write a PR for your
values.yaml file to configure your Helm chart: <CTRL-V> - Write somewhere some documentation to write down the bucket name and why it exists. Another
<CTRL-V> - Another PR to change some
**kustomize** configuration for another service to tell it to use the new bucket <CTRL-V> - Days after, time to upload something to that bucket:
gsutil cp my_file wait_what_was_the_bucket_name_again.. Better check the documentation: CTRL-C + <CTRL-V>
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#with-kapitan","title":"With Kapitan","text":"When using Kapitan, your changes are likely to be contained within one PR, from which you can have a full view of everything that is happening. What happens is explained in this flow
\n%%{ init: { securityLevel: 'loose'} }%%\ngraph LR\n classDef pink fill:#f9f,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef blue fill:#00FFFF,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef bold color:#000,font-weight: bold;\n\n DATA --> KAPITAN\n BUCKET --> DATA\n KAPITAN --> KUBERNETES\n KAPITAN --> TERRAFORM\n KAPITAN --> DOCUMENTATION\n KAPITAN --> SCRIPT\n KAPITAN --> HELM\n KUBERNETES --> BUCKET_K8S\n TERRAFORM --> BUCKET_TF\n DOCUMENTATION --> BUCKET_DOC\n SCRIPT --> BUCKET_SCRIPT\n HELM --> BUCKET_HELM\n\n\n DATA[(\"All your data\")]\n BUCKET(\"bucket_name\")\n KAPITAN((\"<img src='/images/kapitan_logo.png'; width='150'/>\")):::blue\n\n\n subgraph \" \"\n KUBERNETES([\"Kubernetes\"]):::pink\n BUCKET_K8S(\".. a ConfigMap uses bucket_name\"):::bold\n end\n subgraph \" \"\n TERRAFORM([\"Terraform\"]):::pink\n BUCKET_TF(\"..creates the bucket bucket_name\"):::bold\n end\n subgraph \" \"\n DOCUMENTATION([\"Documentation\"]):::pink\n BUCKET_DOC(\"..references a link to bucket_name\"):::bold\n end\n subgraph \" \"\n SCRIPT([\"Canned Script\"]):::pink\n BUCKET_SCRIPT(\"..knows how to upload files to bucket_name\"):::bold\n end\n subgraph \" \"\n HELM([\"Helm\"]):::pink\n BUCKET_HELM(\"..configures a chart to use the bucket_name\"):::bold\n end
Thanks to its flexiblility, you can use Kapitan to generate all sorts of configurations: Kubernetes and Terraform resources, ArgoCD pipelines, Docker Compose files, random configs, scripts, documentations and anything else you find relevant. The trick is obviously on how to drive these changes, but it is not as complicated as it sounds. We'll get there soon enough!
Let's see now another example of things that are so established in the way to do things that become elusivly impossible to see. As a way to highlight the potential issues with this way of doing things, let's ask some questions on your current setup. We pick on Kubernetes this time.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#kubernetes","title":"Kubernetes","text":"I\u2019ll start with Kubernetes, such a popular and brilliant solution to problems most people should not be concerned with (jokes apart, I adore Kubernetes). To most, Kubernetes is that type of solution that quickly turns into a problem of its own right.
So.. how do you deploy to Kubernetes right now?
Helm comes to mind first, right?
Kapitan + Helm: BFF
In spite of Kapitan being initially considered (even by ourselves) as an alternative to Helm, we\u2019ve actually enjoyed the benefits of integrating with this amazing tool and the ecosystem it gives us access to. So yes, good news: you can use Helm right from within Kapitan!.
Well, let\u2019s put that to a test. How do you manage your Helm charts? I\u2019ll attempt to break these questions down into categories.
Code OrganizationDRYMaintenanceOperationsDocumentationSecrets managementEverything else - Where do you keep your Helm charts?
- In a single repository?
- How many repositories?
- Alongside the code you develop?
- What about the official ones that you didn't create yourself?
- How many
values.yaml files do you have? - How much consistency is there between them? any snowflakes?
- If you change something, like with the
bucket_name example above: - how many places do you need to go and update?
- And how many times do you get it wrong?
- Don't you feel all your charts look the same?
- Yet how many times do you need to deviate from the one you thought captured everything?
- What if you need to make a change to all your charts at once: how do you deal with it?
- What about configuration files, how do you deal with templating those?
- How do you deal with \u201cofficial\u201d charts, do they always cover what you want to do?
- How do you deal with modifications that you need to apply to your own version of a an official chart?
- What if you need to make a change that affects ALL your charts?
- Or if the change is for all the charts for a set of microservices?
- How many times you find yourself seting parameters on the command line of Helm and other tools?
- How many times did you connect to the wrong context in Kubernetes
- How many of your colleagues have the same clean context setup as you have?
- How many things are there that you wish you were tracking?
- How do I connect to the production database? Which user is it again?
- How easy is it for you to create a new environment from scratch?
- Are you sure?
- When was the last time you tried?
- How easy is it to keep your configuration up to date?
- Does your documentation need to be \u201cunderstood\u201d or can be just executed on?
- Would you be able to follow those instructions at 3am on a Sunday morning?
- How do you handle secrets in your repository?
- Do you know how to create your secrets from scratch?
- Do you remember that token you created 4 months ago? How did you do that?
- How long would it take you?
- Is the process of creating them \u201csecure\u201d?
- Or does it leave you with random certificates and tokens unencrypted on your \u201cDownloads\u201d folder?
- The above concerns: do they also apply to other things you manage?
- Terraform?
- Pipelines?
- Random other systems you interact with?
I\u2019ll stop here because I do not want to lose you, and neither do I want to discourage you.
But if you look around it\u2019s true, you do have a very complicated setup. And Kapitan can help you streamline it for you. In fact, Kapitan can leave you with a consistent and uniform way to manage all these concerns at once.
My job here is done: you have awakened and you won't look at your setup in the same way. Keep tuned and learn about how Kapitan can change the way you do things.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-new-kapitan-release--v0310/","title":"New Kapitan release v0.31.0","text":"The Kapicorp team is happy to to announce a new release of Kapitan.
This release is yet another great bundle of features and improvements over the past year, the majority of which have been contributions from our community!
Head over our release page on GitHub for a full list of features and contributors.
If you missed it, have a look at our latest blog post here 5 years of Kapitan
Please help us by visiting our Sponsor Kapitan page.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-new-kapitan-release--v0320/","title":"New Kapitan release v0.32.0","text":"The Kapicorp team is happy to to announce a new release of Kapitan.
This release contains loads of improvements for the past 6 months, the majority of which have been contributions from our community!
Head over our release page on GitHub for a full list of features and contributors.
Please help us by visiting our Sponsor Kapitan page.
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/","title":"Deploying Keda with Kapitan","text":"We have worked hard to bring out a brand new way of experience Kapitan, through something that we call generators
Although the concept is something we've introduced in 2020 with our blog post Keep your ship together with Kapitan, the sheer amount of new capabilities (and frankly, the embarassing lack of documentation and examples) forces me to show you the new capabilities using a practicle example: deploying Keda.
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#objective-of-this-tutorial","title":"Objective of this tutorial","text":"We are going to deploy Keda using the helm chart approach. While Kapitan supports a native way to deploy helm charts using the helm input type, we are going instead to use a generator based approach using the \"charts\" generator.
This tutorial will show you how to configure kapitan to:
- download a helm chart
- compile a helm chart
- modify a helm chart using mutations
The content of this tutorial is already available on the kapitan-reference
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#deploying-keda","title":"Deploying KEDA","text":""},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#define-parameters","title":"Define parameters","text":"## inventory/classes/components/keda.yml\nparameters:\n keda:\n params:\n # Variables to reference from other places\n application_version: 2.11.2\n service_account_name: keda-operator\n chart_name: keda\n chart_version: 2.11.2\n chart_dir: system/sources/charts/${keda:params:chart_name}/${keda:params:chart_name}/${keda:params:chart_version}/${keda:params:application_version}\n namespace: keda\n helm_values: {}\n...\n
Override Helm Values
As an example we could be passing to helm an override to the default values parameters to make the operator deploy 2 replicas.
helm_values:\n operator:\n replicaCount: 2\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#download-the-chart","title":"Download the chart","text":"Kapitan supports downloading dependencies, including helm charts.
When Kapitan is run with the --fetch, it will download the dependency if not already present. Use --force-fetch if you want to download it every time. Learn more about External dependencies
## inventory/classes/components/keda.yml\n...\n kapitan:\n dependencies:\n # Tells kapitan to download the helm chart into the chart_dir directory\n - type: helm\n output_path: ${keda:params:chart_dir}\n source: https://kedacore.github.io/charts\n version: ${keda:params:chart_version}\n chart_name: ${keda:params:chart_name}\n...\n
Parameter interpolation
Notice how we are using parameter interpolation from the previously defined keda.params section. This will make it easier in the future to override some aspects of the configuration on a per-target base.
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#generate-the-chart","title":"Generate the chart","text":"## inventory/classes/components/keda.yml\n...\n charts:\n # Configures a helm generator to compile files for the given chart\n keda:\n chart_dir: ${keda:params:chart_dir}\n helm_params:\n namespace: ${keda:params:namespace}\n name: ${keda:params:chart_name}\n helm_values: ${keda:params:helm_values}\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#compile","title":"Compile","text":"Before we can see any effect, we need to attach the class to a target. We will create a simple target which looks like
# inventory/targets/tutorials/keda.yml\nclasses:\n- common\n- components.keda\n
Now when we run kapitan compile we will see the chart being donwloaded and the manifests being produced.
./kapitan compile -t keda --fetch\nDependency keda: saved to system/sources/charts/keda/keda/2.11.2/2.11.2\nRendered inventory (1.87s)\nCompiled keda (2.09s)\n
kapitan compile breakdown
--fetch tells kapitan to fetch the chart if it is not found locally-t keda tells kapitan to compile only the previously defined keda.yml target
ls -l compiled/keda/manifests/\ntotal 660\n-rw-r--r-- 1 ademaria root 659081 Aug 29 10:25 keda-bundle.yml\n-rw-r--r-- 1 ademaria root 79 Aug 29 10:25 keda-namespace.yml\n-rw-r--r-- 1 ademaria root 7092 Aug 29 10:25 keda-rbac.yml\n-rw-r--r-- 1 ademaria root 1783 Aug 29 10:25 keda-service.yml\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#using-mutations","title":"Using mutations","text":"Now let's do a couple of things that would not be easy to do with helm natively.
You can already notice that the content of the chart is being splitted into multiple files: this is because the Generator is configured to separate different resources types into different files for convenience and consistency. The mechanism behing it is the \"Mutation\" of type \"bundle\" which tells Kapitan which file to save a resource into.
Here are some example \"mutation\" which separates different kinds into different files
mutations:\n bundle:\n - conditions:\n kind: [Ingress]\n filename: '{content.component_name}-ingress'\n ...\n - conditions:\n kind: [HorizontalPodAutoscaler, PodDisruptionBudget, VerticalPodAutoscaler]\n filename: '{content.component_name}-scaling'\n - conditions:\n kind: ['*']\n filename: '{content.component_name}-bundle'\n
Catch-all rule
Notice the catchall rule at the end that puts everything that has not matched into the bundle.yml file
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#bundle-mutation","title":"bundle mutation","text":"Currently most of the keda related resources are bundled into the -bundle.yml file Instead, we want to separate them into their own file.
Let's add this configuration:
charts:\n # Configures a helm generator to compile files for the given chart\n keda:\n chart_dir: ${keda:params:chart_dir}\n ...\n mutations:\n bundle:\n - conditions:\n # CRDs need to be setup separately\n kind: [CustomResourceDefinition]\n filename: '{content.component_name}-crds'\n
Upon compile, you can now see that the CRD are being moved to a different file:
ls -l compiled/keda/manifests/\ntotal 664\n-rw-r--r-- 1 ademaria root 11405 Aug 29 10:56 keda-bundle.yml\n-rw-r--r-- 1 ademaria root 647672 Aug 29 10:56 keda-crds.yml\n-rw-r--r-- 1 ademaria root 79 Aug 29 10:56 keda-namespace.yml\n-rw-r--r-- 1 ademaria root 7092 Aug 29 10:56 keda-rbac.yml\n-rw-r--r-- 1 ademaria root 1783 Aug 29 10:56 keda-service.yml\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#patch-mutation","title":"patch mutation","text":"As we are using Argo, we want to pass a special argocd.argoproj.io/sync-options annotation to the CRD only so that ArgoCD can handle them properly.
For this we are going to use the patch mutation:
...\n mutations:\n...\n patch:\n - conditions:\n kind: [CustomResourceDefinition]\n patch:\n metadata:\n annotations:\n argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true,Replace=true\n
Upon compile, you can now see that the CRDs have been modified as required:
diff --git a/compiled/keda/manifests/keda-crds.yml b/compiled/keda/manifests/keda-crds.yml\nindex 2662bf3..9306c3a 100644\n--- a/compiled/keda/manifests/keda-crds.yml\n+++ b/compiled/keda/manifests/keda-crds.yml\n@@ -2,6 +2,7 @@ apiVersion: apiextensions.k8s.io/v1\n kind: CustomResourceDefinition\n metadata:\n annotations:\n+ argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true,Replace=true\n controller-gen.kubebuilder.io/version: v0.12.0\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#summary","title":"Summary","text":"With this tutorial have explored some capabilities of Kapitan to manage and perform changes to helm charts. Next tutorial will show how to make use of Keda and deploy a generator for Keda resources
"},{"location":"pages/blog/12/02/2024/kapitan-logo-new-kapitan-release--v0331/","title":"New Kapitan release v0.33.1","text":"The Kapicorp team is happy to to announce a new release of Kapitan.
This release contains loads of improvements for the past 8 months, the majority of which have been contributions from our community!
Head over our release page on GitHub for a full list of features and contributors.
Please help us by visiting our Sponsor Kapitan page.
"},{"location":"pages/commands/kapitan_compile/","title":"CLI Reference | kapitan compile","text":""},{"location":"pages/commands/kapitan_compile/#kapitan-compile","title":"kapitan compile","text":"Merges inventory and inputs and produces generated files in the output folder (/compiled by default)
"},{"location":"pages/commands/kapitan_compile/#compile-all-targets","title":"Compile all targets","text":"kapitan compile\n
click to expand output Compiled mysql-generator-fetch (0.18s)\nCompiled vault (0.25s)\nCompiled pritunl (0.22s)\nCompiled gke-pvm-killer (0.05s)\nCompiled examples (0.30s)\nCompiled mysql (0.08s)\nCompiled postgres-proxy (0.06s)\nCompiled echo-server (0.06s)\nCompiled global (0.03s)\nCompiled guestbook-argocd (0.08s)\nCompiled tutorial (0.13s)\nCompiled kapicorp-project-123 (0.03s)\nCompiled kapicorp-demo-march (0.03s)\nCompiled kapicorp-terraform-admin (0.03s)\nCompiled sock-shop (0.32s)\nCompiled tesoro (0.09s)\nCompiled dev-sockshop (0.32s)\nCompiled prod-sockshop (0.38s)\nCompiled argocd (2.29s)\n
"},{"location":"pages/commands/kapitan_compile/#selective-compilation","title":"Selective compilation","text":""},{"location":"pages/commands/kapitan_compile/#using-target-names","title":"Using target names","text":"Compiles one or more targets selected by name using --targets or -t
kapitan compile -t mysql tesoro\n
click to expand output Compiled mysql (0.06s)\nCompiled tesoro (0.09s)\n
"},{"location":"pages/commands/kapitan_compile/#using-labels","title":"Using labels","text":"Compiles one or more targets selected matching labels with --labels or -l
Info
This works if you have labelled your targets using the following syntax:
parameters:\n ...\n kapitan:\n ...\n labels:\n customer: acme\n
see Labels for more details
$ kapitan compile -l customer=acme\nCompiled acme-project (0.14s)\nCompiled acme-pipelines (0.10s)\n
"},{"location":"pages/commands/kapitan_compile/#fetch-on-compile","title":"Fetch on compile","text":"Use the --fetch flag to fetch Remote Inventories and the External Dependencies.
kapitan compile --fetch\n
This will download the dependencies according to their configurations By default, kapitan does not overwrite an existing item with the same name as that of the fetched inventory items.
Use the --force-fetch flag to force fetch (update cache with freshly fetched items) and overwrite inventory items of the same name in the output_path.
kapitan compile --force-fetch\n
Use the --cache flag to cache the fetched items in the .dependency_cache directory in the root project directory.
kapitan compile --cache --fetch\n
"},{"location":"pages/commands/kapitan_compile/#embed-references","title":"Embed references","text":"By default, Kapitan references are stored encrypted (for backends that support encription) in the configuration repository under the /refs directory.
For instance, a reference tag ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de} would point to a phisical file on disk under /refs like:
refs/targets/minikube-mysql/mysql/password
data: hQEMA8uOJKdm07XTAQgAp5i [[ CUT ]] BwqYc3g7PI09HCJZdU=\nencoding: base64\nrecipients:\n- fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C\ntype: gpg\n
The --embed-refs flags tells Kapitan to embed these references on compile, alongside the generated output. By doing so, compiled output is self-contained and can be revealed by Tesoro or other tools.
kapitan compile --embed-refs\n
See how the compiled output for this specific target changes to embed the actul encrypted content, (marked by ?{gpg: :embedded} to indicate it is a gpg reference) rather than just holding a reference to it (like in this case ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de} which points to ).
click to expand output diff --git a/examples/kubernetes/compiled/minikube-mysql/manifests/mysql_app.yml b/examples/kubernetes/compiled/minikube-mysql/manifests/mysql_app.yml\n[[ CUT ]]\napiVersion: v1\ndata:\n- MYSQL_ROOT_PASSWORD: ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de}\n- MYSQL_ROOT_PASSWORD_SHA256: ?{gpg:targets/minikube-mysql/mysql/password_sha256:122d2732}\n+ MYSQL_ROOT_PASSWORD: ?{gpg:eyJkYXRhIjogImhR [[ CUT ]] gInR5cGUiOiAiZ3BnIn0=:embedded}\n+ MYSQL_ROOT_PASSWORD_SHA256: ?{gpg:eyJkYXRhI [[ CUT ]] eXBlIjogImdwZyJ9:embedded}\n
"},{"location":"pages/commands/kapitan_compile/#help","title":"help","text":"kapitan compile --help\n
click to expand output usage: kapitan compile [-h] [--inventory-backend {reclass}]\n [--search-paths JPATH [JPATH ...]]\n [--jinja2-filters FPATH] [--verbose] [--prune]\n [--quiet] [--output-path PATH] [--fetch]\n [--force-fetch] [--force] [--validate]\n [--parallelism INT] [--indent INT]\n [--refs-path REFS_PATH] [--reveal] [--embed-refs]\n [--inventory-path INVENTORY_PATH] [--cache]\n [--cache-paths PATH [PATH ...]]\n [--ignore-version-check] [--use-go-jsonnet]\n [--compose-target-name] [--schemas-path SCHEMAS_PATH]\n [--yaml-multiline-string-style STYLE]\n [--yaml-dump-null-as-empty]\n [--targets TARGET [TARGET ...] | --labels\n [key=value ...]]\n\noptions:\n -h, --help show this help message and exit\n --inventory-backend {reclass,reclass-rs}\n Select the inventory backend to use (default=reclass)\n --search-paths JPATH [JPATH ...], -J JPATH [JPATH ...]\n set search paths, default is [\".\"]\n --jinja2-filters FPATH, -J2F FPATH\n load custom jinja2 filters from any file, default is\n to put them inside lib/jinja2_filters.py\n --verbose, -v set verbose mode\n --prune prune jsonnet output\n --quiet set quiet mode, only critical output\n --output-path PATH set output path, default is \".\"\n --fetch fetch remote inventories and/or external dependencies\n --force-fetch overwrite existing inventory and/or dependency item\n --force overwrite existing inventory and/or dependency item\n --validate validate compile output against schemas as specified\n in inventory\n --parallelism INT, -p INT\n Number of concurrent compile processes, default is 4\n --indent INT, -i INT Indentation spaces for YAML/JSON, default is 2\n --refs-path REFS_PATH\n set refs path, default is \"./refs\"\n --reveal reveal refs (warning: this will potentially write\n sensitive data)\n --embed-refs embed ref contents\n --inventory-path INVENTORY_PATH\n set inventory path, default is \"./inventory\"\n --cache, -c enable compilation caching to .kapitan_cache and\n dependency caching to .dependency_cache, default is\n False\n --cache-paths PATH [PATH ...]\n cache additional paths to .kapitan_cache, default is\n []\n --ignore-version-check\n ignore the version from .kapitan\n --use-go-jsonnet use go-jsonnet\n --compose-target-name Create same subfolder structure from inventory/targets\n inside compiled folder\n --schemas-path SCHEMAS_PATH\n set schema cache path, default is \"./schemas\"\n --yaml-multiline-string-style STYLE, -L STYLE\n set multiline string style to STYLE, default is\n 'double-quotes'\n --yaml-dump-null-as-empty\n dumps all none-type entries as empty, default is\n dumping as 'null'\n --targets TARGET [TARGET ...], -t TARGET [TARGET ...]\n targets to compile, default is all\n --labels [key=value ...], -l [key=value ...]\n compile targets matching the labels, default is all\n
"},{"location":"pages/commands/kapitan_dotfile/","title":"CLI Reference | .kapitan config file","text":""},{"location":"pages/commands/kapitan_dotfile/#kapitan","title":".kapitan","text":"Kapitan allows you to coveniently override defaults by specifying a local .kapitan file in the root of your repository (relative to the kapitan configuration):
This comes handy to make sure Kapitan runs consistently for your specific setup.
Info
Any Kapitan command can be overridden in the .kapitan dotfile, but here are some of the most common examples.
"},{"location":"pages/commands/kapitan_dotfile/#version","title":"version","text":"To enforce the Kapitan version used for compilation (for consistency and safety), you can add version to .kapitan:
version: 0.30.0\n\n...\n
This constrain can be relaxed to allow minor versions to be also accepted:
version: 0.30 # Allows any 0.30.x release to run\n\n...\n
"},{"location":"pages/commands/kapitan_dotfile/#command-line-flags","title":"Command line flags","text":"You can also permanently define all command line flags in the .kapitan config file. For example:
...\n\ncompile:\n indent: 4\n parallelism: 8\n
would be equivalent to running:
kapitan compile --indent 4 --parallelism 8\n
For flags which are shared by multiple commands, you can either selectively define them for single commmands in a section with the same name as the command, or you can set any flags in section global, in which case they're applied for all commands. If you set a flag in both the global section and a command's section, the value from the command's section takes precedence over the value from the global section.
As an example, you can configure the inventory-path in the global section of the Kapitan dotfile to make sure it's persisted across all Kapitan runs.
...\n\nglobal:\n inventory-path: ./some_path\n
which would be equivalent to running any command with --inventory-path=./some_path.
Another flag that you may want to set in the global section is inventory-backend to select a non-default inventory backend implementation.
global:\n inventory-backend: reclass\n
which would be equivalent to always running Kapitan with --inventory-backend=reclass.
Please note that the inventory-backend flag currently can't be set through the command-specific sections of the Kapitan config file.
"},{"location":"pages/commands/kapitan_inventory/","title":"CLI Reference | kapitan inventory","text":""},{"location":"pages/commands/kapitan_inventory/#kapitan-inventory","title":"kapitan inventory","text":"Renders the resulting inventory values for a specific target.
For example, rendering the inventory for the mysql target:
kapitan inventory -t mysql\n
click to expand output __reclass__:\n environment: base\n name: mysql\n node: mysql\n timestamp: Wed Nov 23 23:19:28 2022\n uri: yaml_fs:///src/inventory/targets/examples/mysql.yml\napplications: []\nclasses:\n - kapitan.kube\n - kapitan.generators.kubernetes\n - kapitan.generators.argocd\n - kapitan.generators.terraform\n - kapitan.generators.rabbitmq\n - kapitan.common\n - common\n - components.mysql\nenvironment: base\nexports: {}\nparameters:\n _reclass_:\n environment: base\n name:\n full: mysql\n short: mysql\n components:\n mysql:\n config_maps:\n config:\n data:\n mysql.cnf:\n value: ignore-db-dir=lost+found\n mytemplate.cnf:\n template: components/mysql/mytemplate.cnf.j2\n values:\n mysql:\n client:\n port: 3306\n socket: /var/run/mysqld/mysqld.sock\n mysqld:\n bind-address: 127.0.0.1\n max_allowed_packet: 64M\n thread_concurrency: 8\n mount: /etc/mysql/conf.d/\n env:\n MYSQL_DATABASE: ''\n MYSQL_PASSWORD:\n secretKeyRef:\n key: mysql-password\n MYSQL_ROOT_PASSWORD:\n secretKeyRef:\n key: mysql-root-password\n MYSQL_USER: ''\n image: mysql:5.7.28\n ports:\n mysql:\n service_port: 3306\n secrets:\n secrets:\n data:\n mysql-password:\n value: ?{plain:targets/mysql/mysql-password||randomstr|base64}\n mysql-root-password:\n value: ?{plain:targets/mysql/mysql-root-password||randomstr:32|base64}\n versioned: true\n type: statefulset\n volume_claims:\n datadir:\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 10Gi\n storageClassName: standard\n volume_mounts:\n datadir:\n mountPath: /var/lib/mysql\n docs:\n - templates/docs/README.md\n generators:\n manifest:\n default_config:\n annotations:\n manifests.kapicorp.com/generated: 'true'\n service_account:\n create: false\n type: deployment\n kapitan:\n compile:\n - input_paths:\n - components/generators/kubernetes\n input_type: kadet\n output_path: manifests\n output_type: yml\n - input_params:\n function: generate_docs\n template_path: templates/docs/service_component.md.j2\n input_paths:\n - components/generators/kubernetes\n input_type: kadet\n output_path: docs\n output_type: plain\n - input_params:\n function: generate_pre_deploy\n input_paths:\n - components/generators/kubernetes\n input_type: kadet\n output_path: pre-deploy\n output_type: yml\n - input_paths:\n - components/generators/argocd\n input_type: kadet\n output_path: argocd\n output_type: yml\n - input_params:\n generator_root: resources.tf\n input_paths:\n - components/generators/terraform\n input_type: kadet\n output_path: terraform\n output_type: json\n - ignore_missing: true\n input_paths:\n - resources/state/mysql/.terraform.lock.hcl\n input_type: copy\n output_path: terraform/\n - input_paths:\n - components/generators/rabbitmq\n input_type: kadet\n output_path: rabbitmq\n output_type: yml\n - input_paths:\n - templates/docs/README.md\n input_type: jinja2\n output_path: docs\n - input_paths: []\n input_type: jinja2\n output_path: scripts\n - input_paths: []\n input_type: jsonnet\n output_path: manifests\n output_type: yml\n dependencies:\n - output_path: lib/kube.libsonnet\n source: https://raw.githubusercontent.com/bitnami-labs/kube-libsonnet/master/kube.libsonnet\n type: https\n - output_path: lib/kube-platforms.libsonnet\n source: https://raw.githubusercontent.com/bitnami-labs/kube-libsonnet/master/kube-platforms.libsonnet\n type: https\n - output_path: components/generators/kubernetes\n ref: master\n source: https://github.com/kapicorp/kapitan-reference.git\n subdir: components/generators/kubernetes\n type: git\n - output_path: components/generators/terraform\n ref: master\n source: https://github.com/kapicorp/kapitan-reference.git\n subdir: components/generators/terraform\n type: git\n vars:\n target: mysql\n manifests: []\n mysql:\n settings:\n client:\n port: 3306\n socket: /var/run/mysqld/mysqld.sock\n mysqld:\n bind-address: 127.0.0.1\n max_allowed_packet: 64M\n thread_concurrency: 8\n namespace: mysql\n scripts: []\n target_name: mysql\n
"},{"location":"pages/commands/kapitan_lint/","title":"CLI Reference | kapitan lint","text":""},{"location":"pages/commands/kapitan_lint/#kapitan-lint","title":"kapitan lint","text":"Perform a checkup on your inventory or refs.
./kapitan lint\n
click to expand output Running yamllint on all inventory files...\n\n.yamllint not found. Using default values\nFile ./inventory/classes/components/echo-server.yml has the following issues:\n 95:29: forbidden implicit octal value \"0550\" (octal-values)\nFile ./inventory/classes/terraform/gcp/services.yml has the following issues:\n 15:11: duplication of key \"enable_compute_service\" in mapping (key-duplicates)\n\nTotal yamllint issues found: 2\n\nChecking for orphan classes in inventory...\n\nNo usage found for the following 6 classes:\n{'components.argoproj.cd.argocd-server-oidc',\n'components.helm.cert-manager-helm',\n'components.rabbitmq-operator.rabbitmq-configuration',\n'components.rabbitmq-operator.rabbitmq-operator',\n'features.gkms-demo',\n'projects.localhost.kubernetes.katacoda'}\n
"},{"location":"pages/commands/kapitan_searchvar/","title":"CLI Reference | kapitan searchvar","text":""},{"location":"pages/commands/kapitan_searchvar/#kapitan-searchvar","title":"kapitan searchvar","text":"Shows all inventory files where a variable is declared:
./kapitan searchvar parameters.components.*.image\n
click to expand output ./inventory/classes/components/vault.yml ${vault:image}\n./inventory/classes/components/logstash.yml eu.gcr.io/antha-images/logstash:7.5.1\n./inventory/classes/components/gke-pvm-killer.yml estafette/estafette-gke-preemptible-killer:1.2.5\n./inventory/classes/components/mysql.yml mysql:5.7.28\n./inventory/classes/components/postgres-proxy.yml gcr.io/cloudsql-docker/gce-proxy:1.16\n./inventory/classes/components/echo-server.yml jmalloc/echo-server\n./inventory/classes/components/trivy.yml ${trivy:image}\n./inventory/classes/components/filebeat.yml ${filebeat:image}:${filebeat:version}\n./inventory/classes/components/pritunl/pritunl-mongo.yml docker.io/bitnami/mongodb:4.2.6-debian-10-r23\n./inventory/classes/components/pritunl/pritunl.yml alledm/pritunl\n./inventory/classes/components/weaveworks/user-db.yml weaveworksdemos/user-db:0.3.0\n./inventory/classes/components/weaveworks/catalogue.yml weaveworksdemos/catalogue:0.3.5\n./inventory/classes/components/weaveworks/user.yml weaveworksdemos/user:0.4.7\n./inventory/classes/components/weaveworks/session-db.yml redis:alpine\n./inventory/classes/components/weaveworks/catalogue-db.yml weaveworksdemos/catalogue-db:0.3.0\n./inventory/classes/components/weaveworks/carts-db.yml mongo\n./inventory/classes/components/weaveworks/orders-db.yml mongo\n./inventory/classes/components/weaveworks/orders.yml weaveworksdemos/orders:0.4.7\n./inventory/classes/components/weaveworks/shipping.yml weaveworksdemos/shipping:0.4.8\n./inventory/classes/components/weaveworks/queue-master.yml weaveworksdemos/queue-master:0.3.1\n./inventory/classes/components/weaveworks/rabbitmq.yml rabbitmq:3.6.8-management\n./inventory/classes/components/weaveworks/payment.yml weaveworksdemos/payment:0.4.3\n./inventory/classes/components/weaveworks/front-end.yml weaveworksdemos/front-end:0.3.12\n./inventory/classes/components/weaveworks/carts.yml weaveworksdemos/carts:0.4.8\n./inventory/classes/components/kapicorp/tesoro.yml kapicorp/tesoro\n
"},{"location":"pages/commands/kapitan_validate/","title":"CLI Reference | kapitan validate","text":""},{"location":"pages/commands/kapitan_validate/#kapitan-validate","title":"kapitan validate","text":"Validates the schema of compiled output. Validate options are specified in the inventory under parameters.kapitan.validate. Supported types are:
"},{"location":"pages/commands/kapitan_validate/#usage","title":"Usage","text":"standalonemanual with kapitan compileautomatic with .kapitan dotfile kapitan validate\n
click to expand output created schema-cache-path at ./schemas\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_secret.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_jsonnet.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_simple.yml\n
kapitan compile --validate\n
click to expand output Rendered inventory (0.27s)\nCompiled labels (0.23s)\nCompiled removal (0.00s)\nCompiled busybox (0.24s)\nCompiled minikube-nginx-jsonnet (0.49s)\nCompiled minikube-nginx-kadet (0.25s)\nCompiled minikube-mysql (0.59s)\nCompiled minikube-es (1.17s)\nCompiled all-glob (1.55s)\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_secret.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_jsonnet.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_simple.yml\n
You can leverage the .kapitan dotfile to make sure validate runs every time you run compile.
example .kapitan
...\n\ncompile:\n validate: true\n
The validate command will now be implied for every compile run
kapitan compile\n
click to expand output Rendered inventory (0.27s)\nCompiled labels (0.23s)\nCompiled removal (0.00s)\nCompiled busybox (0.24s)\nCompiled minikube-nginx-jsonnet (0.49s)\nCompiled minikube-nginx-kadet (0.25s)\nCompiled minikube-mysql (0.59s)\nCompiled minikube-es (1.17s)\nCompiled all-glob (1.55s)\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_secret.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_jsonnet.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_simple.yml\n
"},{"location":"pages/commands/kapitan_validate/#kubernetes-setup","title":"Kubernetes Setup","text":"Kubernetes has different resource kinds, for instance:
servicedeploymentstatefulset
Kapitan has built in support for validation of Kubernetes kinds, and automatically integrates with https://kubernetesjsonschema.dev. See github.com/instrumenta/kubernetes-json-schema for more informations.
Info
Kapitan will automatically download the schemas for Kubernetes Manifests directly from https://kubernetesjsonschema.dev
By default, the schemas are cached into ./schemas/, which can be modified with the --schemas-path option.
override permanently schema-path
Remember to use the .kapitan dotfile configuration to override permanently the schema-path location.
$ cat .kapitan\n# other options abbreviated for clarity\nvalidate:\n schemas-path: custom/schemas/cache/path\n
"},{"location":"pages/commands/kapitan_validate/#example","title":"Example","text":"Refer to the mysql example.
kubernetes/inventory/classes/component/mysql.yml validate: \n - type: kubernetes # mkdocs (1)! \n output_paths: # mkdocs (2)! \n - manifests/mysql_secret.yml\n kind: secret # temporarily replaced with 'deployment' during test # mkdocs (3)! \n version: 1.14.0 # optional, defaults to 1.14.0 # mkdocs (4)! \n - type: kubernetes\n output_paths:\n - manifests/mysql_service_jsonnet.yml\n - manifests/mysql_service_simple.yml\n kind: service\n version: 1.14.0\n
type | currently only Kubernetes is supportedoutput_paths | list of files to validatekind | a Kubernetes resource kindversion | a Kubernetes API version, defaults to 1.14.0
"},{"location":"pages/contribute/code/","title":"Kapitan code","text":"Many of our features come from contributions from external collaborators. Please help us improve Kapitan by extending it with your ideas, or help us squash bugs you discover.
It's simple, just send us a PR with your improvements!
","tags":["community"]},{"location":"pages/contribute/code/#submitting-code","title":"Submitting code","text":"We would like ask you to fork Kapitan project and create a Pull Request targeting master branch. All submissions, including submissions by project members, require review.
","tags":["community"]},{"location":"pages/contribute/code/#setup","title":"Setup","text":"We build kapitan using poetry.
-
Install poetry
pip install poetry\n
-
Install dependencies
poetry install --all-extras --with dev --with test --with docs\n
Poetry creates a virtual environment with the required dependencies installed.
-
Run kapitan with the own compiled code
poetry run kapitan <your command>\n
Because we are using a pinned version of reclass which is added as a submodule into Kapitan's repository, you need to pull it separately by executing the command below:
git submodule update --init\n
","tags":["community"]},{"location":"pages/contribute/code/#troubleshoot","title":"Troubleshoot","text":"Check if gcc is installed:
brew install gcc@5\n
","tags":["community"]},{"location":"pages/contribute/code/#testing","title":"Testing","text":"Run make test to run all tests. If you modify anything in the examples/ folder make sure you replicate the compiled result of that in tests/test_kubernetes_compiled. If you add new features, run make test_coverage && make test_formatting to make sure the test coverage remains at current or better levels and that code formatting is applied.
If you would like to evaluate your changes by running your version of Kapitan, you can do that by running bin/kapitan from this repository or even setting an alias to it.
python3 -m unittest tests/test_vault_transit.py\n
","tags":["community"]},{"location":"pages/contribute/code/#code-style","title":"Code Style","text":"To make sure you adhere to the Style Guide for Python (PEP8) Python Black is used to apply the formatting so make sure you have it installed with pip3 install black.
","tags":["community"]},{"location":"pages/contribute/code/#apply-via-git-hook","title":"Apply via Git hook","text":" - Run
pip3 install pre-commit to install precommit framework. - In the Kapitan root directory, run
pre-commit install - Git add/commit any changed files you want.
","tags":["community"]},{"location":"pages/contribute/code/#apply-manually","title":"Apply manually","text":"Run make format_codestyle before submitting.
","tags":["community"]},{"location":"pages/contribute/code/#release-process","title":"Release process","text":" - Create a branch named
release-v<NUMBER>. Use v0.*.*-rc.* if you want pre-release versions to be uploaded. - Update CHANGELOG.md with the release changes.
- Once reviewed and merged, Github Actions will auto-release.
- The merge has to happen with a merge commit not with squash/rebase so that the commit message still mentions
kapicorp/release-v* inside.
","tags":["community"]},{"location":"pages/contribute/code/#packaging-extra-resources-in-python-package","title":"Packaging extra resources in python package","text":"To package any extra resources/files in the pip package, make sure you modify both MANIFEST.in.
","tags":["community"]},{"location":"pages/contribute/code/#leave-a-comment","title":"Leave a comment","text":"","tags":["community"]},{"location":"pages/contribute/documentation/","title":"Documentation","text":"Our documentation usully prevents new users from adopting Kapitan. Help us improve by contributing with fixes and keeping it up-to-date.
","tags":["community"]},{"location":"pages/contribute/documentation/#articles","title":"Articles","text":"Write articles on Kapitan and share your way of working. Inspire others, and reach out to have your article published / endorsed by us.
","tags":["community"]},{"location":"pages/contribute/documentation/#this-website","title":"This Website","text":"Find something odd? Let us know or change it yourself: you can edit pages of this website on Github by clicking the pencil icon at the top right of this page!
","tags":["community"]},{"location":"pages/contribute/documentation/#update-documentation","title":"Update documentation","text":"We use mkdocs to generate our gh-pages from .md files under docs/ folder.
Updating our gh-pages is therefore a two-step process.
","tags":["community"]},{"location":"pages/contribute/documentation/#update-the-markdown","title":"Update the markdown","text":"Submit a PR for our master branch that updates the .md file(s). Test how the changes would look like when deployed to gh-pages by serving it on localhost:
- Edit the
strict property in mkdocs.yml and set it to false. make local_serve_documentation- Now the documentation site should be available at
localhost:8000.
","tags":["community"]},{"location":"pages/contribute/documentation/#submit-a-pr","title":"Submit a PR","text":"Once the above PR has been merged, our CI will deploy your docs automatically.
","tags":["community"]},{"location":"pages/contribute/sponsor/","title":"Sponsor Kapitan","text":"Do you want to help the project? Great! There are many ways to do it
We accept donations throught GitHubs Sponsors. Alternatively reach out for other ways to support us.
Companies and individuals sponsoring us on a regular base will be recognised and called out on our website
","tags":["community"]},{"location":"pages/contribute/talk/","title":"Talk about Kapitan","text":"Our project needs your support to get noticed! Please let everyone know that you are using Kapitan
- Help us grow: give us a star
- Join us on kubernetes.slack.com
#kapitan(Get invited) - Tweet about us on Twitter . Remember to add @kapitandev to your tweets
- Share our website
https://kapitan.dev - Write tutorials and blog posts and join the many who have done it already! Get published on the Kapitan Blog
- Share what Kapitan does for you and for your company
- Inspire your colleagues and network on LinkedIn
"},{"location":"pages/input_types/copy/","title":"Copy","text":"This input type simply copies the input templates to the output directory without any rendering/processing. For Copy, input_paths can be either a file or a directory: in case of a directory, all the templates in the directory will be copied and outputted to output_path.
Supported output types: N/A (no need to specify output_type)
Example
kapitan:\n compile:\n - input_type: copy\n ignore_missing: true # Do not error if path is missing. Defaults to False\n input_paths:\n - resources/state/${target_name}/.terraform.lock.hcl\n output_path: terraform/\n
"},{"location":"pages/input_types/external/","title":"External","text":"This input type executes an external script or binary. This can be used to manipulate already compiled files or execute binaries outside of kapitan that generate or manipulate files.
For example, ytt is a useful yaml templating tool. It is not built into the kapitan binary, however, with the external input type, we could specify the ytt binary to be executed with specific arguments and environment variables.
In this example, we're removing a label from a k8s manifests in a directory ingresses and placing it into the compiled target directory.
parameters:\n target_name: k8s-manifests\n kapitan:\n vars:\n target: ${target_name}\n compile:\n - input_type: external\n input_paths:\n - /usr/local/bin/ytt # path to ytt on system\n output_path: .\n args:\n - -f\n - ingresses/ # directory with ingresses\n - -f\n - ytt/remove.yaml # custom ytt script\n - \">\"\n - \\${compiled_target_dir}/ingresses/ingresses.yaml # final merged result\n
Supported output types: N/A (no need to specify output_type)
Additionally, the input type supports field env_vars, which can be used to set environment variables for the external command. By default, the external command doesn't inherit any environment variables from Kapitan's environment. However, if environment variables $PATH or $HOME aren't set in env_vars, they will be propagated from Kapitan's environment to the external command's environment.
Finally, Kapitan will substitute ${compiled_target_dir} in both the command's arguments and the environment variables. This variable needs to be escaped in the configuration to ensure that reclass won't interpret it as a reclass reference.
"},{"location":"pages/input_types/helm/","title":"Input Type | Helm","text":"This is a Python binding to helm template command for users with helm charts. This does not require the helm executable, and the templates are rendered without the Tiller server.
Unlike other input types, Helm input types support the following additional parameters under kapitan.compile:
parameters:\n kapitan:\n compile:\n - output_path: <output_path>\n input_type: helm\n input_paths:\n - <chart_path>\n helm_values:\n <object_with_values_to_override>\n helm_values_files:\n - <values_file_path>\n helm_path: <helm binary>\n helm_params:\n name: <chart_release_name>\n namespace: <substitutes_.Release.Namespace>\n output_file: <string>\n validate: true\n \u2026\n
helm_values is an object containing values specified that will override the default values in the input chart. This has exactly the same effect as specifying --values custom_values.yml for helm template command where custom_values.yml structure mirrors that of helm_values.
helm_values_files is an array containing the paths to helm values files used as input for the chart. This has exactly the same effect as specifying --file my_custom_values.yml for the helm template command where my_custom_values.yml is a helm values file. If the same keys exist in helm_values and in multiple specified helm_values_files, the last indexed file in the helm_values_files will take precedence followed by the preceding helm_values_files and at the bottom the helm_values defined in teh compile block. There is an example in the tests. The monitoring-dev(kapitan/tests/test_resources/inventory/targets/monitoring-dev.yml) and monitoring-prd(kapitan/tests/test_resources/inventory/targets/monitoring-prd.yml) targets both use the monitoring(tests/test_resources/inventory/classes/component/monitoring.yml) component. This component has helm chart input and takes a common.yml helm_values file which is \"shared\" by any target that uses the component and it also takes a dynamically defined file based on a kapitan variable defined in the target.
helm_path can be use to provide the helm binary name or path. helm_path defaults to the value of KAPITAN_HELM_PATH env var if it is set, else it defaults to helm
helm_params correspond to the flags for helm template. Most flags that helm supports can be used here by replacing '-' by '_' in the flag name.
Flags without argument must have a boolean value, all other flags require a string value.
Special flags:
name: equivalent of helm template [NAME] parameter. Ignored if name_template is also specified. If neither name_template nor name are specified, the --generate-name flag is used to generate a name.-
output_file: name of the single file used to output all the generated resources. This is equivalent to call helm template without specifing output dir. If not specified, each resource is generated into a distinct file.
-
include_crds and skip_tests: These flags are enabled by default and should be set to false to be removed.
debug: prints the helm debug output in kapitan debug log.-
namespace: note that due to the restriction on helm template command, specifying the namespace does not automatically add metadata.namespace property to the resources. Therefore, users are encouraged to explicitly specify it in all resources:
metadata:\n namespace: {{ .Release.Namespace }} # or any other custom values\n
See the helm doc for further detail.
"},{"location":"pages/input_types/helm/#example","title":"Example","text":"Let's use nginx-ingress helm chart as the input. Using kapitan dependency manager, this chart can be fetched via a URL as listed in https://helm.nginx.com/stable/index.yaml.
On a side note, https://helm.nginx.com/stable/ is the chart repository URL which you would helm repo add, and this repository should contain index.yaml that lists out all the available charts and their URLs. By locating this index.yaml file, you can locate all the charts available in the repository.
We can use version 0.3.3 found at https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz. We can create a simple target file as inventory/targets/nginx-from-chart.yml whose content is as follows:
parameters:\n kapitan:\n vars:\n target: nginx-from-chart\n dependencies:\n - type: https\n source: https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz\n unpack: True\n output_path: components/charts\n compile:\n - output_path: .\n input_type: helm\n input_paths:\n - components/charts/nginx-ingress\n helm_values:\n controller:\n name: my-controller\n image:\n repository: custom_repo\n helm_params:\n name: my-first-release-name\n namespace: my-first-namespace\n
To compile this target, run:
$ kapitan compile --fetch\nDependency https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz : fetching now\nDependency https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz : successfully fetched\nDependency https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz : extracted to components/charts\nCompiled nginx-from-chart (0.07s)\n
The chart is fetched before compile, which creates components/charts/nginx-ingress folder that is used as the input_paths for the helm input type. To confirm if the helm_values actually has overridden the default values, we can try:
$ grep \"my-controller\" compiled/nginx-from-chart/nginx-ingress/templates/controller-deployment.yaml\n name: my-controller\n app: my-controller\n app: my-controller\n
"},{"location":"pages/input_types/helm/#building-the-binding-from-source","title":"Building the binding from source","text":"Run
cd kapitan/inputs/helm\n./build.sh\n
This requires Go 1.14.
"},{"location":"pages/input_types/helm/#helm-subcharts","title":"Helm subcharts","text":"There is an external dependency manager of type helm which enables you to specify helm charts to download, including subcharts.
"},{"location":"pages/input_types/introduction/","title":"Introduction","text":"Note: make sure to read up on inventory before moving on.
"},{"location":"pages/input_types/introduction/#phases-of-the-compile-command","title":"Phases of the compile command","text":"Now that we have a basic understanding of Kapitan inventory, we can talk about the kapitan compile command.
The command has five distinct phases:
graph LR\n classDef pink fill:#f9f,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef blue fill:#00FFFF,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n INVENTORY[\"Inventory\"]:::pink\n\n COMPILE[\"Compile\"]:::pink\n FINISH[\"Finish\"]:::pink\n COPY[\"Copy\"]:::pink\n\n\n subgraph \"fetch\"\n F{\"fetch?\"}\n FETCH[\"fetch dependencies\"]\n end\n\n subgraph \"validate\"\n V{\"validate?\"}\n VALIDATE[\"Validate\"]\n end\n\n subgraph \"reveal\"\n REVEAL[\"Reveal\"]\n R{\"reveal?\"}\n end\n\n INVENTORY --> F\n F --> |yes| FETCH\n FETCH --> COMPILE\n F ==> |no| COMPILE\n COMPILE ==> R\n R ==> |no| COPY\n R --> |yes| REVEAL\n REVEAL --> COPY\n COPY --> V\n V --> |yes| VALIDATE\n V ==> |no| FINISH\n VALIDATE --> FINISH\n\n
Step Flag Description Configuration Inventory Kapitan uses reclass to render a final version of the inventory. Fetch --fetch Kapitan fetches external dependencies parameters.kapitan.dependencies Compile Kapitan compiles the input types for each target parameters.kapitan.compile Reveal --reveal Kapitan reveals the secrets directly in the compiled output parameters.kapitan.secrets Copy Kapitan moves the output files from the tmp directory to /compiled Validate --validate Kapitan validates the schema of compiled output. parameters.kapitan.validate Finish Kapitan has completed all tasks"},{"location":"pages/input_types/introduction/#supported-input-types","title":"Supported input types","text":"Input types can be specified in the inventory under kapitan.compile in the following format:
jinja2jsonnetkadethelmcopy parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: jinja2\n input_params: # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n
- a dict passed to the template
Please see Jinja
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: jsonnet\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: [`yaml` | `json`]\n
- (Default: global --prune)
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: kadet\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: [`yaml` | `json`]\n
- (Default: global --prune)
Please see Kadet
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: helm\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: <output_type_specific_to_input_type>\n
- (Default: global --prune)
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: copy\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: <output_type_specific_to_input_type>\n
- (Default: global --prune)
"},{"location":"pages/input_types/jinja/","title":"Input Type | Jinja2","text":"This input type is probably the most simple input type to use: it is very versatile and is commonly used to create scripts and documentation files.
It renders jinja2 templates.
"},{"location":"pages/input_types/jinja/#example-configuration","title":"Example configuration","text":"Here's some configuration from the nginx example
examples/kubernetes/inventory/classes/component/nginx-common.yml
templates: #(1)!\n - docs/nginx/README.md\n - components/nginx-deploy.sh\n\n kapitan:\n compile:\n - output_path: . #(2)!\n input_type: jinja2\n input_paths: ${templates} #(3)!\n
- We define a list with all the templates we want to compile with this input type
- Then input type will render the files a the root of the target compiled folder e.g.
compiled/${target_name} - We pass the list as
input_paths
Notice how make use of variable interpolation to use the convenience of a list to add all the files we want to compile. You can now simply add to that list from any other place in the inventory that calls that class.
input_paths can either be a file, or a directory: in case of a directory, all the templates in the directory will be rendered.input_params (optional) can be used to pass extra parameters, helpful when needing to use a similar template for multiple components in the same target.
"},{"location":"pages/input_types/jinja/#documentation","title":"Documentation","text":"We usually store documentation templates under the templates/docs directory.
examples/kubernetes/docs/nginx/README.md
{% set i = inventory.parameters %}\n\n# Welcome to the README!\n\nTarget *{{ i.target_name }}* is running:\n\n* {{ i.nginx.replicas }} replicas of *nginx* running nginx image {{ i.nginx.image }}\n* on cluster {{ i.cluster.name }}\n
Compiled result
# Welcome to the README!\n\nTarget *minikube-nginx-jsonnet* is running:\n\n* 1 replicas of *nginx* running nginx image nginx:1:15.8\n* on cluster minikube\n
"},{"location":"pages/input_types/jinja/#scripts","title":"Scripts","text":"When we use Jinja to render scripts, we tend to call them \"canned scripts\" to indicate that these scripts have everything needed to run without extra parameters.
We usually store script templates under the templates/scripts directory.
examples/kubernetes/components/nginx-deploy.sh
#!/bin/bash -e\nDIR=$(dirname ${BASH_SOURCE[0]})\n{% set i = inventory.parameters %} #(1)!\n\nKUBECTL=\"kubectl -n {{i.namespace}}\" #(2)!\n\n# Create namespace before anything else\n${KUBECTL} apply -f ${DIR}/pre-deploy/namespace.yml\n\nfor SECTION in manifests\ndo\n echo \"## run kubectl apply for ${SECTION}\"\n ${KUBECTL} apply -f ${DIR}/${SECTION}/ | column -t\ndone\n
- We import the
inventory as a Jinja variable - We use to set the
namespace explicitly
Compiled result
#!/bin/bash -e\nDIR=$(dirname ${BASH_SOURCE[0]})\n #(1)!\n\nKUBECTL=\"kubectl -n minikube-nginx-jsonnet\" #(2)!\n\n# Create namespace before anything else\n${KUBECTL} apply -f ${DIR}/pre-deploy/namespace.yml\n\nfor SECTION in manifests\ndo\n echo \"## run kubectl apply for ${SECTION}\"\n ${KUBECTL} apply -f ${DIR}/${SECTION}/ | column -t\ndone\n
- The script is now a \"canned script\" and ready to be used for this specif target.
- You can see that the namespace has been replaced with the target's one.
"},{"location":"pages/input_types/jinja/#accessing-the-inventory","title":"Accessing the inventory","text":"Templates will be provided at runtime with 3 variables:
inventory: To access the inventory for that specific target.inventory_global: To access the inventory of all targets.input_params: To access the optional dictionary provided to the input type.
Use of inventory_global
inventory_global can be used to generate a \"global\" README.md that contains a link to all generated targets.
| *Target* |\n|------------------------------------------------------------------------|\n{% for target in inventory_global | sort() %}\n{% set p = inventory_global[target].parameters %}\n|[{{target}}](../{{target}}/docs/README.md) |\n{% endfor %}\n
Compiled result
| *Target* |\n|------------------------------------------------------------------------|\n| [argocd](../argocd/docs/README.md) |\n| [dev-sockshop](../dev-sockshop/docs/README.md) |\n| [echo-server](../echo-server/docs/README.md) |\n| [examples](../examples/docs/README.md) |\n| [gke-pvm-killer](../gke-pvm-killer/docs/README.md) |\n| [global](../global/docs/README.md) |\n| [guestbook-argocd](../guestbook-argocd/docs/README.md) |\n| [kapicorp-demo-march](../kapicorp-demo-march/docs/README.md) |\n| [kapicorp-project-123](../kapicorp-project-123/docs/README.md) |\n| [kapicorp-terraform-admin](../kapicorp-terraform-admin/docs/README.md) |\n| [mysql](../mysql/docs/README.md) |\n| [postgres-proxy](../postgres-proxy/docs/README.md) |\n| [pritunl](../pritunl/docs/README.md) |\n| [prod-sockshop](../prod-sockshop/docs/README.md) |\n| [sock-shop](../sock-shop/docs/README.md) |\n| [tesoro](../tesoro/docs/README.md) |\n| [tutorial](../tutorial/docs/README.md) |\n| [vault](../vault/docs/README.md) |\n
"},{"location":"pages/input_types/jinja/#jinja2-custom-filters","title":"Jinja2 custom filters","text":"We support the following custom filters for use in Jinja2 templates:
EncodingTimeRegexpfileglobboolternaryshufflereveal_maybe sha256yamltomlb64encodeb64decode SHA256 hashing of text
{{ text | sha256 }}
Dump text as YAML
{{ text | yaml }}
Dump text as TOML
{{ text | toml }}
base64 encode text
{{ text | b64encode }}
base64 decode text
{{ text | b64decode }}
to_datetimestrftime return datetime object for string
{{ \"2019-03-07 13:37:00\" | to_datetime }}
return current date string for format
{{ \"%a, %d %b %Y %H:%M\" | strftime }}
regex_replaceregex_escaperegex_searchregex_findall perform a re.sub returning a string
{{ hello world | regex_replace(pattern=\"world\", replacement=\"kapitan\")}}
escape all regular expressions special characters from string
{{ \"+s[a-z].*\" | regex_escape}}
perform re.search and return the list of matches or a backref
{{ hello world | regex_search(\"world.*\")}}
perform re.findall and return the list of matches as array
{{ hello world | regex_findall(\"world.*\")}}
return list of matched regular files for glob
{{ ./path/file* | fileglob }}
return the bool for value
{{ yes | bool }}
value ? true_val : false_val
{{ condition | ternary(\"yes\", \"no\")}}
randomly shuffle elements of a list
{{ [1, 2, 3, 4, 5] | shuffle }}
reveal ref/secret tag only if compile --reveal flag is set
{{ \"?{base64:my_ref}\" | reveal_maybe}}
Tip
You can also provide path to your custom filter modules in CLI. By default, you can put your filters in lib/jinja2_filters.py and they will automatically get loaded.
"},{"location":"pages/input_types/jsonnet/","title":"Input Type | Jsonnet","text":"Jsonnet is a superset of json format that includes features such as conditionals, variables and imports. Refer to jsonnet docs to understand how it works.
Note: unlike jinja2 templates, one jsonnet template can output multiple files (one per object declared in the file).
"},{"location":"pages/input_types/jsonnet/#accessing-the-inventory","title":"Accessing the inventory","text":"Typical jsonnet files would start as follows:
local kap = import \"lib/kapitan.libjsonnet\"; #(1)!\nlocal inv = kap.inventory(); #(2)!\nlocal p = inv.parameters; #(3)!\n\n{\n \"data_java_opts\": p.elasticsearch.roles.data.java_opts, #(4)!\n}\n
- Import the Kapitan inventory library.
- Assign the content of the full inventory for this specific target to the
inv variable. - Assign the content of the
inventory.parameters to a variable p for convenience. - Use the
p variable fo access a specific intentory value
Note: The dictionary keys of the jsonnet object are used as filenames for the generated output files. If your jsonnet is not a dictionary, but is a valid json(net) object, then the output filename will be the same as the input filename. E.g. 'my_string' is inside templates/input_file.jsonnet so the generated output file will be named input_file.json for example and will contain \"my_string\".
"},{"location":"pages/input_types/jsonnet/#jinja2-templating","title":"Jinja2 templating","text":"Kapitan allows you to compile a Jinja template from within Jsonnet:
local kap = import \"lib/kapitan.libjsonnet\";\n\n{\n \"jon_snow\": kap.jinja2_template(\"templates/got.j2\", { is_dead: false }),\n}\n
"},{"location":"pages/input_types/jsonnet/#callback-functions","title":"Callback functions","text":"In addition, importing kapitan.libjsonnet makes available the following native_callback functions gluing reclass to jsonnet (amongst others):
inventoryjinja2_templateyamlfile I/Osha256_stringgzip_b64jsonschema returns a dictionary with the inventory for target
renders the jinja2 file with context specified
yaml_loadyaml_load_streamyaml_dumpyaml_dump_stream returns a json string of the specified yaml file
returns a list of json strings of the specified yaml file
returns a string yaml from a json string
returns a string yaml stream from a json string
file_readfile_existsdir_files_listdir_files_read reads the file specified
returns informative object if a file exists
returns a list of file in a dir
returns an object with keys - file_name and values - file contents
returns sha256 of string
returns base64 encoded gzip of obj
validates obj with schema, returns object with 'valid' and 'reason' keys
"},{"location":"pages/input_types/jsonnet/#jsonschema-validation","title":"Jsonschema validation","text":"Given the follow example inventory:
mysql:\n storage: 10G\n storage_class: standard\n image: mysql:latest\n
The yaml inventory structure can be validated with the new jsonschema() function:
local schema = {\n type: \"object\",\n properties: {\n storage: { type: \"string\", pattern: \"^[0-9]+[MGT]{1}$\"},\n image: { type: \"string\" },\n }\n};\n// run jsonschema validation\nlocal validation = kap.jsonschema(inv.parameters.mysql, schema);\n// assert valid, otherwise error with validation.reason\nassert validation.valid: validation.reason;\n
If validation.valid is not true, it will then fail compilation and display validation.reason.
Fails validation because storage has an invalid pattern (10Z)
Jsonnet error: failed to compile /code/components/mysql/main.jsonnet:\nRUNTIME ERROR: '10Z' does not match '^[0-9]+[MGT]{1}$'\n\nFailed validating 'pattern' in schema['properties']['storage']:\n {'pattern': '^[0-9]+[MGT]{1}$', 'type': 'string'}\n\nOn instance['storage']:\n '10Z'\n\n/code/mysql/main.jsonnet:(19:1)-(43:2)\n\nCompile error: failed to compile target: minikube-mysql\n
"},{"location":"pages/input_types/kadet/","title":"Input Type | Kadet","text":"Kadet is an extensible input type for Kapitan that enables you to generate templates using Python.
The key benefit being the ability to utilize familiar programing principles while having access to Kapitan's powerful inventory system.
A library that defines resources as classes using the Base Object class is required. These can then be utilized within components to render output.
The following functions are provided by the class BaseObj().
Method definitions:
new(): Provides parameter checking capabilitiesbody(): Enables in-depth parameter configuration
Method functions:
root(): Defines values that will be compiled into the outputneed(): Ability to check & define input parametersupdate_root(): Updates the template file associated with the class
A class can be a resource such as a Kubernetes Deployment as shown here:
class Deployment(BaseObj): # (1)!\n def new(self): # (2)!\n self.need(\"name\", \"name string needed\")\n self.need(\"labels\", \"labels dict needed\")\n self.need(\"containers\", \"containers dict needed\")\n self.update_root(\"lib/kubelib/deployment.yml\")\n\n def body(self): # (3)!\n self.root.metadata.name = self.kwargs.name # (4)!\n self.root.metadata.namespace = inv.parameters.target_name\n self.root.spec.template.metadata.labels = self.kwargs.labels\n self.root.spec.template.spec.containers = self.kwargs.containers\n
- The deployment is an
BaseObj() which has two main functions. new(self) is used to perform parameter validation & template compilationbody(self) is utilized to set those parameters to be rendered.self.root.metadata.name is a direct reference to a key in the corresponding yaml.
Kadet supports importing libraries as you would normally do with Python. These libraries can then be used by the components to generate the required output.
...\nkubelib = kadet.load_from_search_paths(\"kubelib\") #(1)!\n...\nname = \"nginx\"\nlabels = kadet.BaseObj.from_dict({\"app\": name})\nnginx_container = kubelib.Container( #(2)!\n name=name, image=inv.parameters.nginx.image, ports=[{\"containerPort\": 80}]\n)\n...\ndef main():\n output = kadet.BaseObj() #(3)!\n output.root.nginx_deployment = kubelib.Deployment(name=name, labels=labels, containers=[nginx_container]) #(4)!\n output.root.nginx_service = kubelib.Service( #(5)!\n name=name, labels=labels, ports=[svc_port], selector=svc_selector\n )\n return output #(6)!\n
- We import a library called
kubelib using load_from_search_paths() - We use
kubelib to create a Container - We create an output of type
BaseObj and we will be updating the root element of this output. - We use
kubelib to create a Deployment kind. The Deployment makes use of the Container created. - We use
kubelib to create a Service kind. - We return the object. Kapitan will render everything under
output.root
Kadet uses a library called addict to organise the parameters inline with the yaml templates. As shown above we create a BaseObject() named output. We update the root of this output with the data structure returned from kubelib. This output is what is then returned to kapitan to be compiled into the desired output type.
For a deeper understanding please refer to github.com/kapicorp/kadet
Supported output types:
"},{"location":"pages/input_types/remove/","title":"Remove","text":"This input type simply removes files or directories. This can be helpful if you can't control particular files generated during other compile inputs.
For example, to remove a file named copy_target, specify an entry to input_paths, compiled/${kapitan:vars:target}/copy_target.
parameters:\n target_name: removal\n kapitan:\n vars:\n target: ${target_name}\n compile:\n - input_type: copy\n input_paths:\n - copy_target\n output_path: .\n # test removal of a file\n - input_type: remove\n input_paths:\n - compiled/${kapitan:vars:target}/copy_target\n output_path: .\n
As a reminder, each input block within the compile array is run sequentially for a target in Kapitan. If we reversed the order of the inputs above like so:
parameters:\n target_name: removal\n kapitan:\n vars:\n target: ${target_name}\n compile:\n - input_type: remove\n input_paths:\n - compiled/${kapitan:vars:target}/copy_target\n output_path: .\n - input_type: copy\n input_paths:\n - copy_target\n output_path: .\n
The first input block would throw an error because the copy input command hasn't run yet to produce the file being removed by the remove input block.
Supported output types: N/A (no need to specify output_type)
"},{"location":"pages/inventory/advanced/","title":"Advanced Inventory Features","text":""},{"location":"pages/inventory/advanced/#target-labels","title":"Target labels","text":"Kapitan allows you to define labels in your inventory, which can then be used to group together targets with similar labels.
For instance you could define the following:
Defines a class to add the customer label to selected targets
inventory/classes/type/customer_project.yml
parameters:\n customer_name: ${target_name} # Defaults to the target_name\n kapitan:\n labels:\n customer: ${customer_name}\n
Apply the class to the target for customer acme
inventory/targets/customers/acme.yml
classes:\n...\n- type.customer_project\n\nparameters:\n...\n
You can now selectively compile targets for customer acme using the following (see see Labels for more details )
kapitan compile -l customer=acme\nCompiled acme (0.06s)\nCompiled acme-documentation (0.09s)\n
"},{"location":"pages/inventory/classes/","title":"Classes","text":""},{"location":"pages/inventory/classes/#usage","title":"Usage","text":"The next thing you want to learn about the inventory are classes. A class is a yaml file containing a fragment of yaml that we want to import and merge into the inventory.
Classes are fragments of yaml: feature sets, commonalities between targets. Classes let you compose your Inventory from smaller bits, eliminating duplication and exposing all important parameters from a single, logically organised place. As the Inventory lets you reference other parameters in the hierarchy, classes become places where you can define something that will then get referenced from another section of the inventory, allowing for composition.
Classes are organised under the inventory/classes directory substructure. They are organised hierarchically in subfolders, and the way they can be imported into a target or other classes depends on their location relative to the inventory/classes directory.
"},{"location":"pages/inventory/classes/#importing-classes","title":"Importing classes","text":"To import a class from within another file of the Inventory, you can follow these instructions:
- take the file path relative to the
inventory/classes/ directory - remove the
.yml file extension - replace
/ with .
For example, this will import the class inventory/classes/applications/sock-shop.yaml
classes:\n- applications.sock-shop\n
"},{"location":"pages/inventory/classes/#definition","title":"Definition","text":"Let's take a look at the common class which appears in the example above:
As explained, because the common.yaml is directly under the inventory/classes subdirectory, it can be imported directly into a target with:
classes:\n- common\n
If we open the file, we find another familiar yaml fragment.
inventory/classes/common.yml
classes:\n- kapitan.common\n\nparameters:\n namespace: ${target_name}\n target_name: ${_reclass_:name:short}\n
Notice that this class includes an import definition for another class, kapitan.common. We've already learned this means that kapitan will import a file on disk called inventory/classes/kapitan/common.yml
You can also see that in the parameters section we now encounter a new syntax which unlocks another powerful inventory feature: parameters interpolation!
"},{"location":"pages/inventory/introduction/","title":"Introduction","text":""},{"location":"pages/inventory/introduction/#overview","title":"Overview","text":"The Inventory is a core component of Kapitan: this section aims to explain how it works and how to best take advantage of it.
The Inventory is a hierarchical YAML based structure which you use to capture anything that you want to make available to Kapitan, so that it can be passed on to its templating engines.
The first concept to learn about the Inventory is the target. A target is a file, found under the inventory/targets substructure, that tells Kapitan what you want to compile. It will usually map to something you want to do with Kapitan.
For instance, you might want to define a target for each environment that you want to deploy using Kapitan.
The Inventory lets you also define and reuse common configurations through YAML files that are referred to as classes: by listing classes into target, their content gets merged together and allows you to compose complex configurations without repetitions.
By combining target and classes, the Inventory becomes the SSOT for your whole configuration, and learning how to use it will unleash the real power of Kapitan.
Info
The Kapitan Inventory is based on an open source project called reclass and you can find the full documentation on our Github clone. However we discourage you to look directly at the reclass documentation before you learn more about Kapitan, because Kapitan uses a fork of reclass and greatly simplifies the reclass experience.
Info
Kapitan allows users to switch the inventory backend to reclass-rs. You can switch the backend to reclass-rs by passing --inventory-backend=reclass-rs on the command line. Alternatively, you can define the backend in the .kapitan config file.
See the reclass-rs inventory backend documentation for more details.
Note
Kapitan enforces very little structure for the Inventory, so that you can adapt it to your specific needs: this might be overwhelming at the beginning: don\u2019t worry, we will explain best practice and give guidelines soon.
By default, Kapitan will search for its Inventory under inventory/classes and inventory/targets.
inventory/\n\u251c\u2500\u2500 classes\n\u2502 \u251c\u2500\u2500 applications\n\u2502 \u251c\u2500\u2500 components\n\u2502 \u251c\u2500\u2500 features\n\u2502 \u251c\u2500\u2500 kapitan\n\u2502 \u251c\u2500\u2500 projects\n\u2502 \u2514\u2500\u2500 terraform\n\u2514\u2500\u2500 targets\n \u251c\u2500\u2500 examples\n \u251c\u2500\u2500 kapicorp\n \u2514\u2500\u2500 terraform\n
"},{"location":"pages/inventory/parameters_interpolation/","title":"Parameters Interpolation","text":"Note
as a shorthand, when we encounter deep yaml structures like the following:
parameters:\n components:\n nginx:\n image: nginx:latest\n
Usually when we want to talk about the image subkey, we normally use either of the following:
parameters.components.nginx.imagecomponents.nginx.image
However, when used in parameter expansion, remember to:
- replace the
. with : - omit the
parameters initial key which is implied - wrap it into the
${} variable interpolation syntax
The correct way to reference parameters.nginx.image then becomes ${components:nginx:image}.
The Inventory allows you to refer to other values defined elsewhere in the structure, using parameter interpolation.
Given the example:
parameters:\n cluster:\n location: europe\n\n application:\n location: ${cluster:location}\n\n namespace: ${target_name}\n target_name: dev\n
Here we tell Kapitan that:
namespace should take the same value defined in target_nametarget_name should take the literal string devapplication.location should take the same value as defined in cluster.location
It is important to notice that the inventory can refer to values defined in other classes, as long as they are imported by the target. So for instance with the following example
classes:\n - project.production\n\n parameters:\n application:\n location: ${cluster.location}\n
Here in this case application.location refers to a value location which has been defined elsewhere, perhaps (but not necessarily) in the project.production class.
Also notice that the class name (project.production) is not in any ways influencing the name or the structed of the yaml it imports into the file
"},{"location":"pages/inventory/reclass-rs/","title":"The reclass-rs inventory backend","text":""},{"location":"pages/inventory/reclass-rs/#overview","title":"Overview","text":"Reclass-rs is a reimplementation of Kapitan's Reclass fork in Rust. Please note that the Rust implementation doesn't support all the features of Kapitan's Reclass fork yet.
However, reclass-rs improves rendering time for the inventory significantly, especially if you're making heavy use of parameter references in class includes. If some of the Reclass features or options that you're using are missing in reclass-rs, don't hesitate to open an issue in the reclass-rs project.
"},{"location":"pages/inventory/reclass-rs/#installation","title":"Installation","text":"The reclass-rs Python package is an optional dependency of Kapitan. You can install it as follows:
pip install kapitan[reclass-rs]\n
"},{"location":"pages/inventory/reclass-rs/#usage","title":"Usage","text":"To use the reclass-rs inventory backend, you need to pass --inventory-backend=reclass-rs on the command line. If you want to permanently switch to the reclass-rs inventory backend, you can select the inventory backend in the .kapitan config file:
global:\n inventory-backend: reclass-rs\n
"},{"location":"pages/inventory/reclass-rs/#performance-comparison","title":"Performance comparison","text":"For the performance comparison, a real Kapitan inventory which makes heavy use of parameter interpolation in class includes was rendered with both Reclass and reclass-rs. The example inventory that was used for the performance comparison contains 325 classes and 56 targets. The example inventory renders to a total of 25MB of YAML.
"},{"location":"pages/inventory/reclass-rs/#reclass","title":"Reclass","text":"$ time kapitan inventory -v --inventory-backend=reclass > inv.yml\n[ ... some output omitted ... ]\nkapitan.resources DEBUG Using reclass as inventory backend\nkapitan.inventory.backends.reclass DEBUG Inventory reclass: No config file found. Using reclass inventory config defaults\nkapitan.inventory.backends.reclass DEBUG Inventory rendering with reclass took 0:01:06.037057\n\nreal 1m23.840s\nuser 1m23.520s\nsys 0m0.287s\n
Reclass takes 1 minute and 6 seconds to render the example inventory. The rest of the runtime (roughly 18 seconds) is spent in writing the resulting 25MB of YAML to the output file.
"},{"location":"pages/inventory/reclass-rs/#reclass-rs","title":"reclass-rs","text":"$ time kapitan inventory -v --inventory-backend=reclass-rs > inv-rs.yml\n[ ... some output omitted ... ]\nkapitan.resources DEBUG Using reclass-rs as inventory backend\nkapitan.inventory.backends.reclass DEBUG Inventory reclass: No config file found. Using reclass inventory config defaults\nreclass-config.yml entry 'storage_type=yaml_fs' not implemented yet, ignoring...\nreclass-config.yml entry 'inventory_base_uri=./inventory' not implemented yet, ignoring...\nreclass-config.yml entry 'allow_none_override=true' not implemented yet, ignoring...\nkapitan.inventory.backends.reclass_rs DEBUG Inventory rendering with reclass-rs took 0:00:01.717107\n\nreal 0m19.921s\nuser 0m35.586s\nsys 0m1.066s\n
reclass-rs takes 1.7 seconds to render the example inventory. The rest of the runtime (roughly 18 seconds) is spent in writing the resulting 25MB of YAML to the output file.
"},{"location":"pages/inventory/targets/","title":"Targets","text":""},{"location":"pages/inventory/targets/#usage","title":"Usage","text":"A target is a file that lives under the inventory/targets subdirectory, and that tells Kapitan what you want it to do for you.
Kapitan will recognise all YAML files in the inventory/targets subtree as targets.
Note
Only use .yml as extension for Inventory files. .yaml will not be recognised as a valid Inventory file.
What you do with a target is largely up to you and your setup. Common examples:
- clusters: Map each target to a cluster, capturing all configurations needed for a given cluster. For instance:
targets/clusters/production-cluster1.yml - applications: When using Kapitan to manage Kubernetes applications, you might define a target for everything that you would normally deploy in a single namespace, including all its resources, scripts, secrets and documentation. For instance:
targets/mysql.yml - environments: You might have want to define a different target for each environment you have, like
dev.yml, test.yml and prod.yml - cloud projects: When working with Terraform, it may be convenient to group target by cloud project. For instance:
targets/gcp/projects/engineering-prod.yml. - single tenancy: When deploying a single-tenancy application, you might combine the approaches above, and have a target
acme.yml that is used to define both Terraform and Kubernetes resources for a given tenant, perhaps also with some ArgoCD or Spinnaker pipelines to go with it.
Example
If you have configured your kapitan repository like in Quick Start instructions, you can run the commands we give during the course of this documentation.
kapitan compile
Compiled gke-pvm-killer (0.09s)\nCompiled vault (0.18s)\nCompiled pritunl (0.17s)\nCompiled mysql (0.07s)\nCompiled examples (0.25s)\nCompiled postgres-proxy (0.06s)\nCompiled echo-server (0.08s)\nCompiled global (0.05s)\nCompiled tutorial (0.09s)\nCompiled guestbook-argocd (0.08s)\nCompiled sock-shop (0.30s)\nCompiled kapicorp-demo-march (0.04s)\nCompiled kapicorp-project-123 (0.03s)\nCompiled kapicorp-terraform-admin (0.08s)\nCompiled tesoro (0.09s)\nCompiled prod-sockshop (0.34s)\nCompiled dev-sockshop (0.41s)\nCompiled argocd (2.53s)\n
When you run kapitan compile, you instruct Kapitan to generate for each given target a directory under compiled with the same name. Under this directory you will find all the files that have been generated by Kapitan for that target.
tree compiled/mysql/
compiled/mysql/\n\u251c\u2500\u2500 argocd\n\u251c\u2500\u2500 docs\n\u2502 \u251c\u2500\u2500 mysql-readme.md\n\u2502 \u2514\u2500\u2500 README.md\n\u251c\u2500\u2500 manifests\n\u2502 \u251c\u2500\u2500 mysql-bundle.yml\n\u2502 \u251c\u2500\u2500 mysql-config.yml\n\u2502 \u251c\u2500\u2500 mysql-namespace.yml\n\u2502 \u2514\u2500\u2500 mysql-secret.yml\n\u251c\u2500\u2500 pre-deploy\n\u251c\u2500\u2500 rabbitmq\n\u251c\u2500\u2500 scripts\n\u2514\u2500\u2500 terraform\n\n7 directories, 6 files\n
"},{"location":"pages/inventory/targets/#definition","title":"Definition","text":"A typical target might look like this:
inventory/targets/acme/dev.yaml
classes:\n - common\n - components.acme.frontend\n - components.acme.backend\n\nparameters:\n target_name: dev\n
Note that it is made of 2 sections:
classes is a list of class files you will want to import.parameters allows for local override of what is unique to this target.
Info
the kapitan key under the root parameters is reserved for kapitan usage. Some examples:
parameters:\n kapitan:\n compile: # input types configuration section\n dependencies: # dependencies configuration section to download resources\n secrets: # secret encryption/decryption configuration section\n validate: # items which indicate which compiled output to validate\n vars: # which are also passed down to input types as context\n
"},{"location":"tags/","title":"Tags","text":""},{"location":"tags/#community","title":"community","text":" - Proposals
- Kapitan Code
- Documentation
- Sponsor Us
"},{"location":"tags/#kadet","title":"kadet","text":""},{"location":"tags/#kubernetes","title":"kubernetes","text":""}]}
\ No newline at end of file
+{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Kapitan: Keep your ship together","text":"Kapitan aims to be your one-stop configuration management solution to help you manage the ever growing complexity of your configurations by enabling Platform Engineering and GitOps workflows.
It streamlines complex deployments across heterogeneous environments while providing a secure and adaptable framework for managing infrastructure configurations. Kapitan's inventory-driven model, powerful templating capabilities, and native secret management tools offer granular control, fostering consistency, reducing errors, and safeguarding sensitive data.
Empower your team to make changes to your infrastructure whilst maintaining full control, with a GitOps approach and full transparency.
- Join the community
#kapitan - Help us grow: give us a star or even better sponsor our project
"},{"location":"#why-do-i-need-kapitan","title":"Why do I need Kapitan?","text":""},{"location":"#video-tutorials-to-get-started","title":"Video Tutorials to get started","text":"Kapitan Youtube Channel
InventoryReferencesHelm and Generators integrationRawkode: Introduction to Kapitan "},{"location":"ADOPTERS/","title":"Who uses Kapitan","text":"If you're using Kapitan in your organization, please let us know by adding to this list on the docs/ADOPTERS.md file.
"},{"location":"FAQ/","title":"FAQ","text":""},{"location":"FAQ/#why-do-i-need-kapitan","title":"Why do I need Kapitan?","text":"See Why do I need Kapitan?
"},{"location":"FAQ/#ask-your-question","title":"Ask your question","text":"Please use the comments facility below to ask your question
"},{"location":"getting_started/","title":"Kapitan Overview","text":""},{"location":"getting_started/#setup-your-installation","title":"Setup your installation","text":"Using our reference repositories you can easily get started with Kapitan
"},{"location":"getting_started/#examples-repository","title":"Examples repository","text":"kapicorp/kapitan-reference repository is meant show you many working examples of things you can do with Kapitan. You can use this to get familiar with Kapitan
$ git clone git@github.com:kapicorp/kapitan-reference.git kapitan-templates\n$ cd kapitan-templates\n\n$ ./kapitan compile\nCompiled postgres-proxy (1.51s)\nCompiled tesoro (1.70s)\nCompiled echo-server (1.64s)\nCompiled mysql (1.67s)\nCompiled gke-pvm-killer (1.17s)\nCompiled prod-sockshop (4.74s)\nCompiled dev-sockshop (4.74s)\nCompiled tutorial (1.68s)\nCompiled global (0.76s)\nCompiled examples (2.60s)\nCompiled pritunl (2.03s)\nCompiled sock-shop (4.36s)\n
"},{"location":"getting_started/#minimal-repository","title":"Minimal repository","text":"Using cruft based cookiecutter
pip3 install cruft\n
cruft create http://github.com/kapicorp/kapitan-reference --checkout cookiecutter --no-input\nDependency https://github.com/kapicorp/generators.git: saved to system/lib\nDependency https://github.com/kapicorp/generators.git: saved to system/generators/kubernetes\nDependency https://github.com/kapicorp/generators.git: saved to system/generators/terraform\nRendered inventory (1.74s)\nCompiled echo-server (0.14s)\n
"},{"location":"getting_started/#running-kapitan","title":"running Kapitan","text":"recommended
"},{"location":"getting_started/#kapitan-wrapper-script","title":"kapitan wrapper script","text":"If you use the provided repository, we already package a kapitan shell script that wraps the docker command to run Kapitan
$ ./kapitan compile\nCompiled postgres-proxy (1.51s)\nCompiled tesoro (1.70s)\nCompiled echo-server (1.64s)\nCompiled mysql (1.67s)\nCompiled gke-pvm-killer (1.17s)\nCompiled prod-sockshop (4.74s)\nCompiled dev-sockshop (4.74s)\nCompiled tutorial (1.68s)\nCompiled global (0.76s)\nCompiled examples (2.60s)\nCompiled pritunl (2.03s)\nCompiled sock-shop (4.36s)\n
"},{"location":"getting_started/#other-installation-methods","title":"Other installation methods","text":""},{"location":"getting_started/#docker","title":"Docker","text":"recommended
"},{"location":"getting_started/#docker_1","title":"Docker","text":"LinuxMac alias kapitan=\"docker run -t --rm -u $(id -u) -v $(pwd):/src:delegated kapicorp/kapitan\"\nkapitan -h\n
alias kapitan=\"docker run -t --rm -v $(pwd):/src:delegated kapicorp/kapitan\"\nkapitan -h\n
"},{"location":"getting_started/#pip","title":"Pip","text":""},{"location":"getting_started/#install-python","title":"Install Python","text":"LinuxMac sudo apt-get update && sudo apt-get install -y python3.8-dev python3-pip python3-yaml\n
brew install python3 libyaml libmagic\n
"},{"location":"getting_started/#install-kapitan-using-pip","title":"Install Kapitan using pip","text":""},{"location":"getting_started/#user","title":"User","text":"LinuxMac kapitan will be installed in $HOME/.local/lib/python3.7/bin
pip3 install --user --upgrade kapitan\n
kapitan will be installed in $HOME/Library/Python/3.7/bin
pip3 install --user --upgrade kapitan\n
"},{"location":"getting_started/#system-wide","title":"System-wide","text":"not recommended
sudo pip3 install --upgrade kapitan\n
"},{"location":"proposals/","title":"Kapitan proposals","text":"","tags":["community"]},{"location":"proposals/#introduction","title":"Introduction","text":"Proposals can be submitted for review by performing a pull request against this repository. If approved the proposal will be published here for further review by the Kapitan community. Proposals tend to be improvements or design consideration for new features.
","tags":["community"]},{"location":"proposals/#existing-proposals","title":"Existing proposals","text":"Kadet input type
External dependency management
Helm charts input type
Kubernetes scheme validation
Portable standalone Kapitan executable
Ref types redesign
Hashicorp vault secrets
","tags":["community"]},{"location":"references/","title":"Kapitan References (formally Secrets)","text":"One of the motivations behing Kapitan's design is that we believe that everything about your setup should be tracked, and Kapitan takes this to the extreme. Sometimes, however, we have to manage values that we do not think they belong to the Inventory: perhaps they are either too variable (for instance, a Git commit sha that changes with every build) or too sensitive, like a password or a generic secret, and then they should always be encrypted.
Kapitan has a built in support for References, which you can use to manage both these use cases.
Kapitan References supports the following backends:
Backend Description Encrypted plain Plain text, (e.g. commit sha) base64 Base64, non confidential but with base64 encoding gpg Support for https://gnupg.org/ gkms GCP KMS awskms AWS KMS azkms Azure Key Vault env Environment vaultkv Hashicorp Vault (RO) vaulttransit Hashicorp Vault (encrypt, decrypt, update_key, rotate_key)"},{"location":"references/#setup","title":"Setup","text":"Some reference backends require configuration, both in the Inventory and to configure the actual backend.
Get started
If you want to get started with references but don't want to deal with the initial setup, you can use the plain and base64 reference types. These are great for demos, but we will see they are extremely helpful even in Production environments.
Danger
Both plain and base64 references do not support encryption: they are intended for development or demo purposes only. DO NOT use plain or base64 for storing sensitive information!
Backend configuration
Configuration for each backend varies, and it is perfomed by configuring the inventory under parameters.kapitan.secrets.
plainbase64gpggkmsawskmsazkmsenvvaultkvvaulttransit No configuration needed
No configuration needed
parameters:\n kapitan:\n secrets:\n gpg:\n recipients:\n - name: example@kapitan.dev\n fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C\n
parameters:\n kapitan:\n secrets:\n gkms:\n key: 'projects/<project>/locations/<location>/keyRings/<keyRing>/cryptoKeys/<key>'\n
parameters:\n kapitan:\n secrets:\n awskms:\n key: 'alias/nameOfKey'\n
parameters:\n kapitan:\n secrets:\n azkms:\n key: 'https://<keyvault-name>.vault.azure.net/keys/<object-name>/<object-version>'\n
parameters:\n ...\n mysql:\n root_password: ?{env:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n kapitan:\n secrets:\n vaultkv:\n VAULT_ADDR: http://127.0.0.1:8200\n auth: token\n mount: secret\n
parameters:\n kapitan:\n secrets:\n vaulttransit:\n VAULT_ADDR: https://vault.example.com\n VAULT_TOKEN: s.mqWkI0uB6So0aHH0v0jyDs97\n VAULT_SKIP_VERIFY: \"False\" # Recommended\n auth: token\n mount: mytransit\n key: 2022-02-13-test\n
Organize your configuration in classes
Just like any other inventory parameters, these configurations can be inherited from a common class or defined per target.
inventory/classes/common.yml
classes:\n- security.backend\n...\n
inventory/classes/security/backend.yml
parameters:\n kapitan:\n secrets:\n <backend>: <configuration>\n
ADVANCED: Mix-and-Match backends Remember that you can use multiple backends at the same time, and also use variable interpolation for an even greater flexibility.
In a multi-cloud setup, for instance, you could configure both GKMS
GCP configuration
inventory/classes/cloud/gcp.yml
classes:\n- security.backends.gkms\n...\n
inventory/classes/security/backends/gkms.yml
# Configuration for GCP targets\nparameters:\n backend: gkms\n kapitan:\n secrets:\n gkms: <configuration>\n
AWS configuration
inventory/classes/security/backends/awskms.yml
# Configuration for AWS targets\nparameters:\n backend: awskms\n kapitan:\n secrets:\n awskms: <configuration>\n
inventory/classes/cloud/aws.yml
classes:\n- security.backends.awskms\n...\n
Now because they both set the parameters.backend variable, you can define a reference whose backend changes based on what class is assigned to the target
inventory/targets/cloud/gcp/acme.yml
classes:\n- cloud.aws\n\nparameters:\n ...\n mysql:\n # the secret backend will change based on the cloud assigned to this target\n root_password: ?{${backend}:targets/${target_name}/mysql/root_password}\n ...\n
"},{"location":"references/#define-references","title":"Define references","text":"References can be defined in the inventory following the syntax spaces added for clarity:
?{ <backend_id> : <reference_path> }
expand for advanced features The syntax also supports for process functions and create_functions which we will discuss later, which brings the full syntax to
?{ <backend_id> : <reference_path> } |<process_function> ||<create_function>
plainbase64gpggkmsawskmsazkmsenvvaultkvvaulttransit parameters:\n ...\n mysql:\n root_password: ?{plain:targets/${target_name}/mysql/root_password}\n ...\n
not encrypted
This reference type does not support encryption: it is intended for non sensitive data only. DO NOT use plain for storing sensitive information!
parameters:\n ...\n mysql:\n root_password: ?{base64:targets/${target_name}/mysql/root_password}\n ...\n
not encrypted
This reference type does not support encryption: it is intended for non sensitive data only. DO NOT use base64 for storing sensitive information!
parameters:\n ...\n mysql:\n root_password: ?{gpg:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{gkms:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{awskms:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{azkms:targets/${target_name}/mysql/root_password}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{env:targets/${target_name}/mysql/root_password}\n ...\n
read-only
parameters:\n ...\n mysql:\n root_password: ?{vaultkv:targets/${target_name}/mysql/root_password}\n ...\n
read-write: parameters:\n ...\n mysql:\n root_password: ?{vaultkv:targets/${target_name}/mysql/root_password:mount:path/in/vault:mykey}\n ...\n
parameters:\n ...\n mysql:\n root_password: ?{vaulttransit:targets/${target_name}/mysql/root_password}\n ...\n
"},{"location":"references/#assign-a-value","title":"Assign a value","text":""},{"location":"references/#manually","title":"Manually","text":"You can assign values to your reference using the command line. Both reading from a file and pipes are supported.
Please Note
Kapitan will fail compilation if a reference is not found. Please see how to assign a value automatically in the next section
plainbase64gpggkmsawskmsazkmsenvvaultkvvaulttransit kapitan refs --write plain:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write plain:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write base64:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write base64:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write gpg:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write gpg:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write gkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write gkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write vaulttransit:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write vaulttransit:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
kapitan refs --write azkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write azkms:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
Setting default value only
The env backend works in a slightly different ways, as it allows you to reference environment variables at runtime.
For example, for a reference called {?env:targets/envs_defaults/mysql_port_${target_name}}, Kapitan would look for an environment variable called KAPITAN_ENV_mysql_port_${TARGET_NAME}.
If that variable cannot be found in the Kapitan environment, the default will be taken from the refs/targets/envs_defaults/mysql_port_${TARGET_NAME} file instead.
kapitan refs --write env:refs/targets/envs_defaults/mysql_port_${TARGET_NAME} -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write env:refs/targets/envs_defaults/mysql_port_${TARGET_NAME} -t ${TARGET_NAME} -f -\n
kapitan refs --write vaultkv:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f <input file>\n
which also works with pipes
cat input_file | kapitan refs --write vaultkv:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
This backend expects the value to be stored as a key:value pair.
echo \"a_key:a_value\" | kapitan refs --write vaulttransit:refs/targets/${TARGET_NAME}/mysql/root_password -t ${TARGET_NAME} -f -\n
When reading from disk, the input file should be formatted accordingly.
"},{"location":"references/#automatically","title":"Automatically","text":"Kapitan has built in capabilities to initialise its references on creation, using an elegant combination of primary and secondary functions. This is extremely powerful because it allows for you to make sure they are always initialised with sensible values.
"},{"location":"references/#primary-functions","title":"primary functions","text":"To automate the creation of the reference, you can add one of the following primary functions to the reference tag by using the syntax ||primary_function:param1:param2
For instance, to automatically initialise a reference with a random string with a lenght of 32 characters, you can use the random primary function
```yaml\nparameters:\n ...\n mysql:\n root_password: ?{${backend}:targets/${target_name}/mysql/root_password||random:str:32}\n ...\n```\n
Initialise non existent references
The first operator here || is more similar to a logical OR.
- If the reference file does not exist, Kapitan will use the function to initialise it
- If the reference file exists, no functions will run.
Automate secret rotation with ease
You can take advantage of it to implement easy rotation of secrets. Simply delete the reference files, and run kapitan compile: let Kapitan do the rest.
randomprivate keysbasicauthreveal strintloweralphaupperalphaloweralphanumupperalphanumspecial Generator function for alphanumeric characters, will be url-token-safe
?{${backend}:targets/${target_name}/mysql/root_password||random:str}\n
generator function for digits (0-9)
?{${backend}:targets/${target_name}/mysql/root_password||random:int}\n
generator function for lowercase letters (a-z)
?{${backend}:targets/${target_name}/mysql/root_password||random:loweralpha}\n
generator function for uppercase letters (A-Z)
?{${backend}:targets/${target_name}/mysql/root_password||random:upperalpha}\n
generator function for lowercase letters and numbers (a-z and 0-9)
?{${backend}:targets/${target_name}/mysql/root_password||random:loweralphanum}\n
generator function for uppercase letters and numbers (A-Z and 0-9)
?{${backend}:targets/${target_name}/mysql/root_password||random:upperalphanum}\n
generator function for alphanumeric characters and given special characters
?{${backend}:targets/${target_name}/mysql/root_password||random:special}\n
rsaed25519publickeyrsapublic Generates an RSA 4096 private key (PKCS#8). You can optionally pass the key size
?{${backend}:targets/${target_name}/private_key||rsa}\n
Generates a ed25519 private key (PKCS#8)
?{${backend}:targets/${target_name}/private_key||ed25519}\n
Derives the public key from a revealed private key
?{${backend}:targets/${target_name}/private_key||rsa}\n?{${backend}:targets/${target_name}/public_key||reveal:targets/${target_name}/private_key|publickey}\n
DEPRECATED: use ||publickey
Generates a base64 encoded pair of username:password
?{${backend}:targets/${target_name}/apache_basicauth||basicauth:username:password}\n
Reveals the content of another reference, useful when deriving public keys or a reference requires a different encoding or the same value.
?{${backend}:targets/${target_name}/secret||random:str}\n?{${backend}:targets/${target_name}/base64_secret||reveal:targets/${target_name}/secret|base64}\n
attention when rotating secrets used with reveal
If you use reveal to initialise a reference, like my_reference||reveal:source_reference the my_reference will not be automatically updated if source_reference changes. Please make sure you also re-initialise my_reference correctly
"},{"location":"references/#secondary-functions","title":"secondary functions","text":"base64sha256 base64 encodes your reference
?{${backend}:targets/${target_name}/secret||random:str|base64}\n
sha256 hashes your reference param1: salt
?{${backend}:targets/${target_name}/secret||random:str|sha256}\n
"},{"location":"references/#reveal-references","title":"Reveal references","text":"You can reveal the secrets referenced in the outputs of kapitan compile via:
```shell\nkapitan refs --reveal -f path/to/rendered/template\n```\n
For example, compiled/minikube-mysql/manifests/mysql_secret.yml with the following content:
```yaml\napiVersion: v1\ndata:\n MYSQL_ROOT_PASSWORD: ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de}\n MYSQL_ROOT_PASSWORD_SHA256: ?{gpg:targets/minikube-mysql/mysql/password_sha256:122d2732}\nkind: Secret\nmetadata:\n annotations: {}\n labels:\n name: example-mysql\n name: example-mysql\n namespace: minikube-mysql\ntype: Opaque\n```\n
can be revealed as follows:
```shell\nkapitan refs --reveal -f compiled/minikube-mysql/manifests/mysql_secret.yml\n```\n
This will substitute the referenced secrets with the actual decrypted secrets stored at the referenced paths and display the file content.
You can also use:
```shell\nkapitan refs --reveal --ref-file refs/targets/all-glob/mysql/password\n```\n
or
```shell\nkapitan refs --reveal --tag \"?{base64:targets/all-glob/mysql/password}\"\n# or\nkapitan refs --reveal --tag \"?{base64:targets/all-glob/mysql/password:3192c15c}\"\n```\n
for more convenience.
"},{"location":"references/#embedded-refs","title":"Embedded refs","text":"Please refer to the CLI reference
"},{"location":"references/#yaml-subvars-references","title":"YAML SubVars References","text":"Kapitan is also able to use access specific keys in YAML content by using subvars.
For instance given a reference plain:larder with content:
```yaml\nfood:\n apples: 1\n```\n
I could now have an inventory variable like:
```yaml\nparameters:\n number_of_apples: ?{plain:larder@food.apple}\n```\n
"},{"location":"references/#using-subvars-to-ingest-yaml-from-command-line-tools","title":"Using subvars to ingest yaml from command line tools","text":"Subvars can have a very practical use for storing YAML outputs coming straight from other tools. For instance, I could use the GCP gcloud command to get all the information about a cluster, and write it into a reference
```shell\ngcloud container clusters describe \\\n --project ${TARGET_NAME}-project \\\n gke-cluster --zone europe-west1 --format yaml \\\n | kapitan refs --write plain:clusters/${TARGET_NAME}/cluster -t ${TARGET_NAME} -f -\n```\n
knowing the output of gcloud to produce yaml that contain the following values:
```yaml\n...\nname: gke-cluster\nreleaseChannel:\n channel: REGULAR\nselfLink: https://container.googleapis.com/v1/projects/kapicorp/locations/europe-west1/clusters/gke-cluster\n...\n```\n
I can not reference the link to the cluster in the inventory using:
```yaml\nparameters:\n cluster:\n name: ?{plain:clusters/${target_name}/cluster@name}\n release_channel: ?{plain:clusters/${target_name}/cluster@releaseChannel.channel}\n link: ?{plain:clusters/${target_name}/cluster@selfLink}\n```\n
Combined with a Jinja template, I could write automatically documentation containing the details of the clusters I use.
```text\n{% set p = inventory.parameters %}\n# Documentation for {{p.target_name}}\n\nCluster [{{p.cluster.name}}]({{p.cluster.link}}) has release channel {{p.cluster.release_channel}}\n```\n
"},{"location":"references/#hashicorp-vault","title":"Hashicorp Vault","text":""},{"location":"references/#vaultkv","title":"vaultkv","text":"Considering a key-value pair like my_key:my_secret in the path secret/foo/bar in a kv-v2(KV version 2) secret engine on the vault server, to use this as a secret use:
```shell\necho \"foo/bar:my_key\" | kapitan refs --write vaultkv:path/to/secret_inside_kapitan -t <target_name> -f -\n```\n
To write a secret in the vault with kapitan use a ref tag with following structure:
```yaml\nparameters:\n ...\n secret:\n my_secret: ?{vaultkv:targets/${target_name}/mypath:mount:path/in/vault:mykey||<functions>}\n ...\n```\n
Leave mount empty to use the specified mount from vault params from the inventory (see below). Same applies to the path/in/vault where the ref path in kapitan gets taken as default value.
Parameters in the secret file are collected from the inventory of the target we gave from CLI -t <target_name>. If target isn't provided then kapitan will identify the variables from the environment when revealing secret.
The environment variables which can also be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Note that providing these variables through the inventory in envvar style is deprecated. Users should update their inventory to set these values in keys without the VAULT_ prefix and in all lowercase. For example VAULT_ADDR: https://127.0.0.1:8200 should be given as addr: https://127.0.0.1:8200 in the inventory. Please note that configuring one of these values in both kapitan.secrets.vaultkv in the inventory and in the environment will cause a validation error.
Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=alpha-secret/foo/bar then mount: alpha-secret (default secret)engine: secret engine used, either kv-v2 or kv (default kv-v2)
The environment variables which cannot be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID.
```yaml\n parameters:\n kapitan:\n secrets:\n vaultkv:\n auth: userpass\n engine: kv-v2\n mount: team-alpha-secret\n addr: http://127.0.0.1:8200\n namespace: CICD-alpha\n skip_verify: false\n client_key: /path/to/key\n client_cert: /path/to/cert\n ```\n
"},{"location":"references/#vaulttransit","title":"vaulttransit","text":"Considering a key-value pair like my_key:my_secret in the path secret/foo/bar in a transit secret engine on the vault server, to use this as a secret use:
```shell\necho \"any.value:whatever-you_may*like\" | kapitan refs --write vaulttransit:my_target/to/secret_inside_kapitan -t <target_name> -f -\n```\n
Parameters in the secret file are collected from the inventory of the target we gave from CLI -t <target_name>. If target isn't provided then kapitan will identify the variables from the environment when revealing secret.
Environment variables that can be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=my_mount (default transit)crypto_key: Name of the encryption key defined in vault-
always_latest: Always rewrap ciphertext to latest rotated crypto_key version Environment variables cannot be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID.
parameters:\n kapitan:\n vars:\n target: my_target\n namespace: my_namespace\n secrets:\n vaulttransit:\n VAULT_ADDR: http://vault.example.com:8200\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n VAULT_SKIP_VERIFY: \"True\"\n auth: token\n mount: transit\n crypto_key: new_key\n always_latest: False\nparameters:\n target_name: secrets\n kapitan:\n secrets:\n vaulttransit:\n VAULT_ADDR: http://127.0.0.1:8200\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n VAULT_SKIP_VERIFY: \"True\"\n auth: token\n mount: transit\n crypto_key: key\n always_latest: False\n
"},{"location":"references/#azure-kms-secret-backend","title":"Azure KMS Secret Backend","text":"To encrypt secrets using keys stored in Azure's Key Vault, a key_id is required to identify an Azure key object uniquely. It should be of the form https://{keyvault-name}.vault.azure.net/{object-type}/{object-name}/{object-version}.
"},{"location":"references/#defining-the-kms-key","title":"Defining the KMS key","text":"This is done in the inventory under parameters.kapitan.secrets.
```yaml\nparameters:\n kapitan:\n vars:\n target: ${target_name}\n namespace: ${target_name}\n secrets:\n azkms:\n key: 'https://<keyvault-name>.vault.azure.net/keys/<object-name>/<object-version>'\n```\n
The key can also be specified using the --key flag
"},{"location":"references/#creating-a-secret","title":"Creating a secret","text":"Secrets can be created using any of the methods described in the \"creating your secret\" section.
For example, if the key is defined in the prod target file
```shell\necho \"my_encrypted_secret\" | kapitan refs --write azkms:path/to/secret_inside_kapitan -t prod -f -\n```\n
Using the --key flag and a key_id
```shell\necho \"my_encrypted_secret\" | kapitan refs --write azkms:path/to/secret_inside_kapitan --key=<key_id> -f -\n```\n
"},{"location":"references/#referencing-and-revealing-a-secret","title":"Referencing and revealing a secret","text":"Secrets can be referenced and revealed in any of the ways described above.
For example, to reveal the secret stored at path/to/secret_inside_kapitan
```shell\nkapitan refs --reveal --tag \"?{azkms:path/to/secret_inside_kapitan}\"\n```\n
Note: Cryptographic algorithm used for encryption is rsa-oaep-256.
"},{"location":"related/","title":"Related projects","text":" - Tesoro - Kubernetes Admission Controller for Kapitan Secrets
- Kapitan Reference - Reference repository to get started
- sublime-jsonnet-syntax - Jsonnet syntax highlighting for Sublime Text
- language-jsonnet - Jsonnet syntax highlighting for Atom
- vim-jsonnet - Jsonnet plugin for Vim (requires a vim plugin manager)
"},{"location":"support/","title":"Get support with Kapitan","text":""},{"location":"support/#community","title":"Community","text":" - Join us on kubernetes.slack.com
#kapitan(Get invited) - Follow us on Twitter @kapitandev.
- Website
https://kapitan.dev - Mailing List kapitan-discuss@googlegroups.com(Subscribe)
"},{"location":"support/#resources","title":"Resources","text":" - Main Blog, articles and tutorials: Kapitan Blog
- Generators and reference kapitan repository: Kapitan Reference
- Kapitan Reference: our reference repository to get started with Kapitan.
"},{"location":"kap_proposals/kap_0_kadet/","title":"Kadet","text":"This introduces a new experimental input type called Kadet.
Kadet is essentially a Python module offering a set of classes and functions to define objects which will compile to JSON or YAML. A complete example is available in examples/kubernetes/components/nginx.
Author: @ramaro
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#overview","title":"Overview","text":"","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#baseobj","title":"BaseObj","text":"BaseObj implements the basic object implementation that compiles into JSON or YAML. Setting keys in self.root means they will be in the compiled output. Keys can be set as an hierarchy of attributes (courtesy of addict) The self.body() method is reserved for setting self.root on instantiation:
The example below:
class MyApp(BaseObj):\n def body(self):\n self.root.name = \"myapp\"\n self.root.inner.foo = \"bar\"\n self.root.list = [1, 2, 3]\n
compiles into:
---\nname: myapp\ninner:\n foo: bar\nlist:\n - 1\n - 2\n - 3\n
The self.new() method can be used to define a basic constructor. self.need() checks if a key is set and errors if it isn't (with an optional custom error message). kwargs that are passed onto a new instance of BaseObj are always accessible via self.kwargs In this example, MyApp needs name and foo to be passed as kwargs.
class MyApp(BaseObj):\n def new(self):\n self.need(\"name\")\n self.need(\"foo\", msg=\"please provide a value for foo\")\n\n def body(self):\n self.root.name = self.kwargs.name\n self.root.inner.foo = self.kwargs.foo\n self.root.list = [1, 2, 3]\n\nobj = MyApp(name=\"myapp\", foo=\"bar\")\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#setting-a-skeleton","title":"Setting a skeleton","text":"Defining a large body with Python can be quite hard and repetitive to read and write. The self.update_root() method allows importing a YAML/JSON file to set the skeleton of self.root.
MyApp's skeleton can be set instead like this:
#skel.yml\n---\nname: myapp\ninner:\n foo: bar\nlist:\n - 1\n - 2\n - 3\n
class MyApp(BaseObj):\n def new(self):\n self.need(\"name\")\n self.need(\"foo\", msg=\"please provide a value for foo\")\n self.update_root(\"path/to/skel.yml\")\n
Extending a skeleton'd MyApp is possible just by implementing self.body():
class MyApp(BaseObj):\n def new(self):\n self.need(\"name\")\n self.need(\"foo\", msg=\"please provide a value for foo\")\n self.update_root(\"path/to/skel.yml\")\n\n def body(self):\n self.set_replicas()\n self.root.metadata.labels = {\"app\": \"mylabel\"}\n\ndef set_replicas(self):\n self.root.spec.replicas = 5\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#inheritance","title":"Inheritance","text":"Python inheritance will work as expected:
class MyOtherApp(MyApp):\n def new(self):\n super().new() # MyApp's new()\n self.need(\"size\")\n\ndef body(self):\n super().body() # we want to extend MyApp's body\n self.root.size = self.kwargs.size\n del self.root.list # get rid of \"list\"\n\nobj = MyOtherApp(name=\"otherapp1\", foo=\"bar2\", size=3)\n
compiles to:
---\nname: otherapp1\ninner:\n foo: bar2\nreplicas: 5\nsize: 3\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#components","title":"Components","text":"A component in Kadet is a python module that must implement a main() function returning an instance ofBaseObj. The inventory is also available via the inventory() function.
For example, a tinyapp component:
# components/tinyapp/__init__.py\nfrom kapitan.inputs.kadet import BaseOBj, inventory\ninv = inventory() # returns inventory for target being compiled\n\nclass TinyApp(BaseObj):\n def body(self):\n self.root.foo = \"bar\"\n self.root.replicas = inv.parameters.tinyapp.replicas\n\ndef main():\n obj = BaseOb()\n obj.root.deployment = TinyApp() # will compile into deployment.yml\n return obj\n
An inventory class must be created for tinyapp:
# inventory/classes/components/tinyapp.yml\n\nparameters:\n tinyapp:\n replicas: 1\n kapitan:\n compile:\n - output_path: manifests\n input_type: kadet\n output_type: yaml\n input_paths:\n - components/tinyapp\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_0_kadet/#common-components","title":"Common components","text":"A library in --search-paths (which now defaults to . and lib/) can also be a module that kadet components import. It is loaded using the load_from_search_paths():
kubelib = load_from_search_paths(\"kubelib\") # lib/kubelib/__init__.py\n\ndef main():\n obj = BaseObj()\n obj.root.example_app_deployment = kubelib.Deployment(name=\"example-app\")\n return obj\n
","tags":["kubernetes","kadet"]},{"location":"kap_proposals/kap_10_azure_key_vault/","title":"Support for Azure Key Management","text":"This feature will enable users to encrypt secrets using keys stored in Azure's Key Vault. The azkms keyword will be used to access the azure key management backend.
"},{"location":"kap_proposals/kap_10_azure_key_vault/#specification","title":"Specification","text":"key_id uniquely identifies an Azure key object and it's version stored in Key Vault. It is of the form https://{keyvault-name}.vault.azure.net/{object-type}/{object-name}/{object-version}. It needs to be made accessible to kapitan in one of the following ways:
parameters:\n kapitan:\n secrets:\n azkms:\n key: key_id #eg https://kapitanbackend.vault.azure.net/keys/myKey/deadbeef\n
kapitan refs --key=<key_id> --write azkms:/path/to/secret -f file_with_secret_data.txt\n
"},{"location":"kap_proposals/kap_10_azure_key_vault/#using-a-key-to-encrypt-a-secret","title":"Using a key to encrypt a secret","text":"The following command will be used to encrypt a secret (using the specified key from Key Vault) and save it in the refs-path along with it's metadata
echo \"my_treasured_secret\" | kapitan refs --write azkms:path/to/secret_inside_kapitan -t <target_name> -f -\n
The -t <target_name> is used to get the information about key_id.
Once the secret is Base64 encoded and encrypted using the key, it will be stored in path/to/secret_inside_kapitan as
data: bXlfdHJlYXN1cmVkX3NlY3JldAo=\nencoding: original\nkey: https://kapitanbackend.vault.azure.net/keys/myKey/deadbeef\ntype: azkms\n
note Cryptographic algorithm used for encryption would be rsa-oaep-256. Optimal Asymmetric Encryption Padding (OAEP) is a padding scheme often used together with RSA encryption.
"},{"location":"kap_proposals/kap_10_azure_key_vault/#referencing-a-secret","title":"referencing a secret","text":"Secrets can be refered using ?{azkms:path/to/secret_id} e.g.
parameter:\n mysql:\n storage: 10G\n storage_class: standard\n image: mysql:latest\n users:\n root:\n password: ?{azkms:path/to/secret}\n
"},{"location":"kap_proposals/kap_10_azure_key_vault/#revealing-a-secret","title":"Revealing a secret","text":"After compilation, the secret reference will be postfixed with 8 characters from the sha256 hash of the retrieved password/secret
apiVersion: v1\ndata:\n MYSQL_ROOT_PASSWORD: ?{azkms:path/to/secret:deadbeef}\nkind: Secret\nmetadata:\n labels:\n name: example-mysql\n name: example-mysql\n namespace: minikube-mysql\ntype: Opaque\n
To reveal the secret, the following command will be used $ kapitan ref --reveal -f compiled/file/containing/secret
"},{"location":"kap_proposals/kap_10_azure_key_vault/#dependencies","title":"Dependencies","text":" - azure-keyvault-keys
- azure-identity
note Kapitan will not be responsible for authentication or access management to Azure
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/","title":"Hashicorp Vault Transit","text":"This feature allows the user to fetch secrets from Hashicorp Vault, with the new secret backend keyword 'vaulttransit'.
Author: @xqp @Moep90
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/#specification","title":"Specification","text":"The following variables need to be exported to the environment(depending on authentication used) where you will run kapitan refs --reveal in order to authenticate to your HashiCorp Vault instance:
- VAULT_ADDR: URL for vault
- VAULT_SKIP_VERIFY=true: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing
- VAULT_TOKEN: token for vault or file (~/.vault-tokens)
- VAULT_ROLE_ID: required by approle
- VAULT_SECRET_ID: required by approle
- VAULT_USERNAME: username to login to vault
- VAULT_PASSWORD: password to login to vault
- VAULT_CLIENT_KEY: the path to an unencrypted PEM-encoded private key matching the client certificate
- VAULT_CLIENT_CERT: the path to a PEM-encoded client certificate for TLS authentication to the Vault server
- VAULT_CACERT: the path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate
- VAULT_CAPATH: the path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate
- VAULT_NAMESPACE: specify the Vault Namespace, if you have one
Considering any stringdata like any.value:whatever-you_may*like ( in our case let\u2019s encrypt any.value:whatever-you_may*like with vault transit ) using the key 2022-02-13-test in a transit secret engine with mount mytransit on the vault server, to use this as a secret either follow:
echo \"any.value:whatever-you_may*like\" > somefile.txt\nkapitan refs --write vaulttransit:<target_name>/to/secret_inside_kapitan --file somefile.txt --target <target_name>\n
or in a single line
echo \"any.value:whatever-you_may*like\" | kapitan refs --write vaulttransit:<target_name>/to/secret_inside_kapitan -t <target_name> -f -\n
The entire string \"any.value:whatever-you_may*like\" will be encrypted by vault and looks like this in return: vault:v2:Jhn3UzthKcJ2s+sEiO60EUiDmuzqUC4mMBWp2Vjg/DGl+GDFEDIPmAQpc5BdIefkplb6yrJZq63xQ9s=. This then gets base64 encoded and stored in the secret_inside_kapitan. Now secret_inside_kapitan contains the following
data: dmF1bHQ6djI6SmhuM1V6dGhLY0oycytzRWlPNjBFVWlEbXV6cVVDNG1NQldwMlZqZy9ER2wrR0RGRURJUG1BUXBjNUJkSWVma3BsYjZ5ckpacTYzeFE5cz0=\nencoding: original\ntype: vaulttransit\nvault_params:\n VAULT_ADDR: http://127.0.0.1:8200\n VAULT_SKIP_VERIFY: 'True'\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n auth: token\n crypto_key: key\n mount: transit\n always_latest: false\n
Encoding tells the type of data given to kapitan, if it is original then after decoding base64 we'll get the original secret and if it is base64 then after decoding once we still have a base64 encoded secret and have to decode again. Parameters in the secret file are collected from the inventory of the target we gave from CLI --target my_target. If target isn't provided then kapitan will identify the variables from the environment, but providing auth is necessary as a key inside target parameters like the one shown:
parameters:\n kapitan:\n vars:\n target: my_target\n namespace: my_namespace\n secrets:\n vaulttransit:\n VAULT_ADDR: http://vault.example.com:8200\n VAULT_TOKEN: s.i53a1DL83REM61UxlJKLdQDY\n VAULT_SKIP_VERIFY: \"True\"\n auth: token\n mount: transit\n crypto_key: new_key\n always_latest: False\n
Environment variables that can be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=alpha-secret/foo/bar then mount: alpha-secret (default secret)crypto_key: Name of the encryption key defined in vaultalways_latest: Always rewrap ciphertext to latest rotated crypto_key version Environment variables should NOT be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID. This makes the secret_inside_kapitan file accessible throughout the inventory, where we can use the secret whenever necessary like ?{vaulttransit:${target_name}/secret_inside_kapitan}
Following is the example file having a secret and pointing to the vault ?{vaulttransit:${target_name}/secret_inside_kapitan}
parameters:\n releases:\n app_version: latest\n app:\n image: app:app-tag\n release: ${releases:app_version}\n replicas: ${replicas}\n args:\n - --verbose=${verbose}\n - --password=?{vaulttransit:${target_name}/secret_inside_kapitan||random:str}\n
when ?{vaulttransit:${target_name}/secret_inside_kapitan} is compiled, it will look same with an 8 character prefix of sha256 hash added at the end like:
kind: Deployment\nmetadata:\n name: app\n namespace: my_namespace\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: app\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=?{vaulttransit:${target_name}/secret_inside_kapitan||random:str}\n image: app:app-tag\n name: app\n
Only the user with the required tokens/permissions can reveal the secrets. Please note that the roles and permissions will be handled at the Vault level. We need not worry about it within Kapitan. Use the following command to reveal the secrets:
kapitan refs --reveal -f compile/file/containing/secret\n
Following is the result of the app-deployment.md file after Kapitan reveal.
kind: Deployment\nmetadata:\n name: app\n namespace: my_namespace\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: app\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=\"any.value:whatever-you_may*like\"\n image: app:app-tag\n name: app\n
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/#vault-policies","title":"Vault policies","text":"path \"mytransit/encrypt/2022-02-13-test\" {\n capabilities = [ \"create\", \"update\" ]\n}\n\npath \"mytransit/decrypt/2022-02-13-test\" {\n capabilities = [ \"create\", \"update\" ]\n}\n
"},{"location":"kap_proposals/kap_11_hashicorp_vault_transit/#dependencies","title":"Dependencies","text":" - hvac is a python client for Hashicorp Vault
"},{"location":"kap_proposals/kap_1_external_dependencies/","title":"External dependencies","text":"This features allows kapitan to fetch files from online repositories/sources during compile and store in a particular target directory.
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_1_external_dependencies/#specification","title":"Specification","text":"Specify the files to be fetched as follows:
parameters:\n kapitan:\n dependencies:\n - type: git | http[s]\n output_path: <output_path>\n source: <git/http[s]_url>\n
The output path is the path to save the dependency into. For example, it could be /components/external/manifest.jsonnet. Then, the user can specify the fetched file as a kapitan.compile item along with the locally-created files.
Git type may also include ref and subdir parameters as illustrated below:
- type: git\n output_path: <output_path>\n source: <git_url>\n subdir: relative/path/in/repository\n ref: <commit_hash/branch/tag>\n force_fetch: <bool>\n
If the file already exists at output_path, the fetch will be skipped. For fresh fetch of the dependencies, users may add --fetch option as follows:
kapitan compile --fetch\n
Users can also add the force_fetch: true option to the kapitan.dependencies in the inventory in order to force fetch of the dependencies of the target every time.
"},{"location":"kap_proposals/kap_1_external_dependencies/#implementation-details","title":"Implementation details","text":""},{"location":"kap_proposals/kap_1_external_dependencies/#dependencies","title":"Dependencies","text":" - GitPython module (and git executable) for git type
- requests module for http[s]
- (optional) tqdm for reporting download progress
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/","title":"Helm Charts Input Type","text":"This will allow kapitan, during compilation, to overwrite the values in user-specified helm charts using its inventory by calling the Go & Sprig template libraries. The helm charts can be specified via local path, and users may download the helm chart via external-dependency feature (of http[s] type).
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#specification","title":"Specification","text":"This feature basically follows the helm template command available. This will run after the fetching of the external dependencies takes place, such that users can simultaneously specify the fetch as well as the import of a helm chart dependency.
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#semantics","title":"Semantics","text":"kapitan:\n compile:\n - input_type: helm\n input_path: <path_to_chart_dir>\n output_path: <output_path>\n set-file:\n - <optional_file_path>\n - ...\n values_file: <optional_values_file>\n namespace: <optional_namespace>\n
This mostly maps to the options available to helm template command (refer to here).
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#implementation-details","title":"Implementation details","text":"C-binding between Helm (Go) and Kapitan (Python) will be created. Helm makes use of two template libraries, namely, text/template and Sprig. The code for helm template command will be converted into shared object (.so) using CGo, which exposes C interface that kapitan (i.e. CPython) could use. The source code for helm template command is found here. This file will be modified to
- remove redundant options
- expose C-interface for Kapitan
"},{"location":"kap_proposals/kap_2_helm_charts_input_type/#dependencies","title":"Dependencies","text":""},{"location":"kap_proposals/kap_3_schema_validation/","title":"Schema Validation (for k8s)","text":"If a yaml/json output is to be used as k8s manifest, users may specify its kind and have kapitan validate its structure during kapitan compile. The plan is to have this validation feature extendable to other outputs as well, such as terraform.
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_3_schema_validation/#specification","title":"Specification","text":"The following inventory will validate the structure of Kubernetes Service manifest file in .
parameters:\n kapitan:\n validate:\n - output_type: kubernetes.service\n version: 1.6.6\n output_path: relative/path/in/target\n
version parameter is optional: if omitted, the version will be set to the stable release of kubernetes (tbc).
"},{"location":"kap_proposals/kap_3_schema_validation/#implementation","title":"Implementation","text":" - The schemas will be downloaded by requests from this repository.
- Caching of schema will also be implemented.
"},{"location":"kap_proposals/kap_3_schema_validation/#dependencies","title":"Dependencies","text":" - jsonschema to validate the output yaml/json against the correct schema
"},{"location":"kap_proposals/kap_4_standalone_executable/","title":"Standalone Kapitan Executable (Discontinued)","text":"Create a portable (i.e. static) kapitan binary for users. This executable will be made available for each release on Github. The target/tested platform is Debian 9 (possibly Windows to be supported in the future).
Criteria:
- speed of the resulting binary
- size of the resulting binary
- portability of the binary (single-file executable or has an accompanying folder)
- cross-platform
- actively maintained
- supports Python 3.6, 3.7
Author: @yoshi-1224
"},{"location":"kap_proposals/kap_4_standalone_executable/#tools-to-be-explored","title":"Tools to be explored","text":" - (tentative first-choice) Pyinstaller
- (Alternative) nuitka (also part of GSoC 2019. It might soon support single-file executable output).
"},{"location":"kap_proposals/kap_5_ref_types_redesign/","title":"Ref Types Redesign","text":"Redesign Kapitan Secrets and rename them as References or Ref.
Breaking changes:
$ kapitan secrets is replaced with $ kapitan refs- the default secrets directory
./secrets/ changes to ./refs/ - the
--secrets-path flag changes to --refs-path - ref ref type is renamed to base64 e.g.
?{ref:some/ref} into ?{base64:some/ref}
Status: In progress
Author: @ramaro
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#proposal","title":"Proposal","text":"Rename Secrets into Ref (or References) to improve consistency and meaning of the backend types by removing the ref backend and introducting new backends:
Type Description Encrypted? Compiles To gpg GnuPG Yes hashed tag gkms Google KMS Yes hashed tag awskms Amazon KMS Yes hashed tag base64 base64 No hashed tag plain plain text No plain text The type value will now need to be representative of the way a reference is stored via its backend.
A new plain backend type is introduced and will compile into revealed state instead of a hashed tag.
A new base64 backend type will store a base64 encoded value as the backend suggests (replacing the old badly named ref backend).
The command line for secrets will be instead:
kapitan refs --write gpg:my/secret1 ...\nkapitan refs --write base64:my/file ...\nkapitan refs --write plain:my/info ...\n
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#plain-backend","title":"plain backend","text":"The plain backend type will allow referring to external state by updating refs programmatically (e.g. in your pipeline)
For example, one can update the value of an environment variable and use ?{plain:my/user} as a reference in a template:
echo $USER | kapitan refs --write plain:my/user -f -\n
Or update a docker image value as ref ?{plain:images/dev/envoy}:
echo 'envoyproxy/envoy:v1.10.0' | kapitan refs --write plain:images/dev/envoy -f -\n
These references will be compiled into their values instead of hashed tags.
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#base64-backend","title":"base64 backend","text":"The base64 backend type will function as the original ref type. Except that this time, the name is representative of what is actually happening :)
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#refs-path","title":"Refs path","text":"Refs will be stored by default in the ./refs path set by --refs-path replacing the --secrets-path flag.
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#background","title":"Background","text":""},{"location":"kap_proposals/kap_5_ref_types_redesign/#kapitan-secrets","title":"Kapitan Secrets","text":"Kapitan Secrets allow referring to restricted information (passwords, private keys, etc...) in templates while also securely storing them.
On compile, secret tags are updated into hashed tags which validate and instruct Kapitan how to reveal tags into decrypted or encoded information.
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#kapitan-secrets-example","title":"Kapitan Secrets example","text":"The following command creates a GPG encrypted secret with the contents of file.txt for recipient ramaro@google.com to read:
kapitan secrets --write gpg:my/secret1 -f file.txt --recipients ramaro@google.com\n
This secret can be referred to in a jsonnet compoment:
{\n \"type\": \"app\",\n \"name\": \"test_app\",\n \"username\": \"user_one\",\n \"password\": \"?{gpg:my/secret1}\"\n}\n
When this compoment is compiled, it looks like (note the hashed tag):
type: app\nname: test_app\nusername: user_one\npassword: ?{gpg:my/secret1:deadbeef}\n
A user with the required permissions can reveal the compiled component:
$ kapitan secrets --reveal -f compiled/mytarget/manifests/component.yml\n\ntype: app\nname: test_app\nusername: user_one\npassword: secret_content_of_file.txt\n
"},{"location":"kap_proposals/kap_5_ref_types_redesign/#secret-backend-comparison","title":"Secret Backend Comparison","text":"Kapitan today offers multiple secret backends:
Type Description Encrypted? Compiles To gpg GnuPG Yes hashed tag gkms Google KMS Yes hashed tag awskms Amazon KMS Yes hashed tag ref base64 No hashed tag However, not all backends are encrypted - this is not consistent!
The ref type is not encrypted as its purpose is to allow getting started with the Kapitan Secrets workflow without the need of setting up the encryption backends tooling (gpg, gcloud, boto, etc...)
"},{"location":"kap_proposals/kap_6_hashicorp_vault/","title":"Hashicorp Vault","text":"This feature allows the user to fetch secrets from Hashicorp Vault, with the new secret backend keyword 'vaultkv'.
Author: @vaibahvk @daminisatya
"},{"location":"kap_proposals/kap_6_hashicorp_vault/#specification","title":"Specification","text":"The following variables need to be exported to the environment(depending on authentication used) where you will run kapitan refs --reveal in order to authenticate to your HashiCorp Vault instance:
- VAULT_ADDR: URL for vault
- VAULT_SKIP_VERIFY=true: if set, do not verify presented TLS certificate before communicating with Vault server. Setting this variable is not recommended except during testing
- VAULT_TOKEN: token for vault or file (~/.vault-tokens)
- VAULT_ROLE_ID: required by approle
- VAULT_SECRET_ID: required by approle
- VAULT_USERNAME: username to login to vault
- VAULT_PASSWORD: password to login to vault
- VAULT_CLIENT_KEY: the path to an unencrypted PEM-encoded private key matching the client certificate
- VAULT_CLIENT_CERT: the path to a PEM-encoded client certificate for TLS authentication to the Vault server
- VAULT_CACERT: the path to a PEM-encoded CA cert file to use to verify the Vault server TLS certificate
- VAULT_CAPATH: the path to a directory of PEM-encoded CA cert files to verify the Vault server TLS certificate
- VAULT_NAMESPACE: specify the Vault Namespace, if you have one
Considering a key-value pair like my_key:my_secret ( in our case let\u2019s store hello:batman inside the vault ) in the path secret/foo in a kv-v2(KV version 2) secret engine on the vault server, to use this as a secret either follow:
echo \"foo:hello\" > somefile.txt\nkapitan refs --write vaultkv:path/to/secret_inside_kapitan --file somefile.txt --target dev-sea\n
or in a single line
echo \"foo:hello\" | kapitan refs --write vaultkv:path/to/secret_inside_kapitan -t dev-sea -f -\n
The entire string \"foo:hello\" is base64 encoded and stored in the secret_inside_kapitan. Now secret_inside_kapitan contains the following
data: Zm9vOmhlbGxvCg==\nencoding: original\ntype: vaultkv\nvault_params:\n auth: token\n
Encoding tells the type of data given to kapitan, if it is original then after decoding base64 we'll get the original secret and if it is base64 then after decoding once we still have a base64 encoded secret and have to decode again. Parameters in the secret file are collected from the inventory of the target we gave from CLI --target dev-sea. If target isn't provided then kapitan will identify the variables from the environment, but providing auth is necessary as a key inside target parameters like the one shown:
parameters:\n kapitan:\n secrets:\n vaultkv:\n auth: userpass\n engine: kv-v2\n mount: team-alpha-secret\n VAULT_ADDR: http://127.0.0.1:8200\n VAULT_NAMESPACE: CICD-alpha\n VAULT_SKIP_VERIFY: false\n VAULT_CLIENT_KEY: /path/to/key\n VAULT_CLIENT_CERT: /path/to/cert\n
Environment variables that can be defined in kapitan inventory are VAULT_ADDR, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_CLIENT_CERT, VAULT_CLIENT_KEY, VAULT_CAPATH & VAULT_CACERT. Extra parameters that can be defined in inventory are:
auth: specify which authentication method to use like token,userpass,ldap,github & approlemount: specify the mount point of key's path. e.g if path=alpha-secret/foo/bar then mount: alpha-secret (default secret)engine: secret engine used, either kv-v2 or kv (default kv-v2) Environment variables cannot be defined in inventory are VAULT_TOKEN,VAULT_USERNAME,VAULT_PASSWORD,VAULT_ROLE_ID,VAULT_SECRET_ID. This makes the secret_inside_kapitan file accessible throughout the inventory, where we can use the secret whenever necessary like ?{vaultkv:path/to/secret_inside_kapitan}
Following is the example file having a secret and pointing to the vault ?{vaultkv:path/to/secret_inside_kapitan}
parameters:\n releases:\n cod: latest\n cod:\n image: alledm/cod:${cod:release}\n release: ${releases:cod}\n replicas: ${replicas}\n args:\n - --verbose=${verbose}\n - --password=?{vaultkv:path/to/secret_inside_kapitan}\n
when ?{vaultkv:path/to/secret_inside_kapitan} is compiled, it will look same with an 8 character prefix of sha256 hash added at the end like:
kind: Deployment\nmetadata:\n name: cod\n namespace: dev-sea\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: cod\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=?{vaultkv:path/to/secret_inside_kapitan:57d6f9b7}\n image: alledm/cod:v2.0.0\n name: cod\n
Only the user with the required tokens/permissions can reveal the secrets. Please note that the roles and permissions will be handled at the Vault level. We need not worry about it within Kapitan. Use the following command to reveal the secrets:
kapitan refs --reveal -f compile/file/containing/secret\n
Following is the result of the cod-deployment.md file after Kapitan reveal.
kind: Deployment\nmetadata:\n name: cod\n namespace: dev-sea\nspec:\n replicas: 1\n template:\n metadata:\n labels:\n app: cod\n spec:\n containers:\n - args:\n - --verbose=True\n - --password=batman\n image: alledm/cod:v2.0.0\n name: cod\n
"},{"location":"kap_proposals/kap_6_hashicorp_vault/#dependencies","title":"Dependencies","text":" - hvac is a python client for Hashicorp Vault
"},{"location":"kap_proposals/kap_7_remote_inventory/","title":"Remote Inventory Federation","text":"This feature would add the ability to Kapitan to fetch parts of the inventory from remote locations (https/git). This would allow users to combine different inventories from different sources and build modular infrastructure reusable across various repos.
Author: @alpharoy14
"},{"location":"kap_proposals/kap_7_remote_inventory/#specification","title":"Specification","text":"The configuration and declaration of remote inventories would be done in the inventory files.
The file specifications are as follows:
parameters:\n kapitan:\n inventory:\n - type: <inventory_type> #git\\https\n source: <source_of_inventory>\n output_path: <relative_output_path>\n
On executing the $ kapitan compile --fetch command, first the remote inventories will be fetched followed by fetching of external dependencies and finally merge the inventory to compile.
"},{"location":"kap_proposals/kap_7_remote_inventory/#copying-inventory-files-to-the-output-location","title":"Copying inventory files to the output location","text":"The output path is the path to save the inventory items into. The path is relative to the inventory/ directory. For example, it could be /classes/. The contents of the fetched inventory will be recursively copied.
The fetched inventory files will be cached in the .dependency_cache directory if --cache is set. eg. $ kapitan compile --fetch --cache
"},{"location":"kap_proposals/kap_7_remote_inventory/#force-fetching","title":"Force fetching","text":"While fetching, the output path will be recursively checked to see if it contains any file with the same name. If so, kapitan will skip fetching it.
To overwrite the files with the newly downloaded inventory items, we can add the --force-fetch flag to the compile command, as shown below.
$ kapitan compile --force-fetch
"},{"location":"kap_proposals/kap_7_remote_inventory/#url-type","title":"URL type","text":"The URL type can be either git or http(s). Depending on the URL type, the configuration file may have additional arguments.
E.g Git type may also include aditional ref parameter as illustrated below:
inventory:\n - type: git #git\\https\n source: <source_of_inventory>\n output_path: <output_path>\n ref: <commit_hash/branch/tag>\n
"},{"location":"kap_proposals/kap_7_remote_inventory/#implementation-details","title":"Implementation details","text":"TODO
"},{"location":"kap_proposals/kap_7_remote_inventory/#dependencies","title":"Dependencies","text":" - GitPython module (and git executable) for git type
- requests module for http[s]
"},{"location":"kap_proposals/kap_8_google_secret_management/","title":"Support for Google Secret Manager","text":"This feature will enable users to retrieve secrets from Google Secret Manager API using the gsm keyword.
"},{"location":"kap_proposals/kap_8_google_secret_management/#specification","title":"Specification","text":"project_id uniquely identifies GCP projects, and it needs to be made accessible to kapitan in one of the following ways:
parameters:\n kapitan:\n secrets:\n gsm:\n project_id: Project_Id\n
kapitan refs --google-project-id=<Project_Id> --write gsm:/path/to/secret_id -f secret_id_file.txt\n
- As an environment variable
export PROJECT_ID=<Project_Id>\n
"},{"location":"kap_proposals/kap_8_google_secret_management/#using-a-secret","title":"Using a secret","text":"In GCP, a secret contains one or more secret versions, along with its metadata. The actual contents of a secret are stored in a secret version. Each secret is identified by a name. We call that variable secret_id e.g. my_treasured_secret. The URI of the secret becomes projects/<Project_Id>/secrets/my_treasured_secret
The following command will be used to add a secret_id to kapitan.
echo \"my_treasured_secret\" | kapitan refs --write gsm:path/to/secret_inside_kapitan -t <target_name> -f -\n
The -t <target_name> is used to get the information about Project_ID.
The secret_id is Base64 encoded and stored in path/to/secret_inside_kapitan as
data: bXlfdHJlYXN1cmVkX3NlY3JldAo=\nencoding: original\ntype: gsm\ngsm_params:\n project_id: Project_ID\n
"},{"location":"kap_proposals/kap_8_google_secret_management/#referencing-a-secret","title":"referencing a secret","text":"Secrets can be refered using ?{gsm:path/to/secret_id:version_id} e.g.
parameter:\n mysql:\n storage: 10G\n storage_class: standard\n image: mysql:latest\n users:\n root:\n password: ?{gsm:path/to/secret_id:version_id}\n
Here, version_id will be an optional argument. By default it will point to latest.
"},{"location":"kap_proposals/kap_8_google_secret_management/#revealing-a-secret","title":"Revealing a secret","text":"After compilation, the secret reference will be postfixed with 8 characters from the sha256 hash of the retrieved password
apiVersion: v1\ndata:\n MYSQL_ROOT_PASSWORD: ?{gsm:path/to/secret_id:version_id:deadbeef}\nkind: Secret\nmetadata:\n labels:\n name: example-mysql\n name: example-mysql\n namespace: minikube-mysql\ntype: Opaque\n
To reveal the secret, the following command will be used $ kapitan ref --reveal -f compiled/file/containing/secret
"},{"location":"kap_proposals/kap_8_google_secret_management/#dependencies","title":"Dependencies","text":" - google-cloud-secret-manager
note Kapitan will not be responsible for authentication or access management to GCP
"},{"location":"kap_proposals/kap_8_modularize_kapitan/","title":"Modularize Kapitan","text":"Kapitan is packaged in PYPI and as a binary along with all its dependencies. Adding an extra key/security backend means that we need to ship another dependency with that PYPI package, making deploying changes more complicated. This project would modularize kapitan into core dependencies and extra modules.
"},{"location":"kap_proposals/kap_8_modularize_kapitan/#usage","title":"Usage","text":"pip3 install --user kapitan # to install only core dependencies\nPip3 install --user kapitan[gkms] \u200b# gkms is the module\n
"},{"location":"kap_proposals/kap_8_modularize_kapitan/#implementation","title":"Implementation","text":" - The main module includes the essential kapitan dependencies and reclass dependencies, which will be included in the \u200brequirement.txt\u200b file.
- The extra module pypi extras will be defined in the
s\u200betup.py\u200b file. - The extra dependencies are of secret backends like (AWS Key backend, Google KMS Key backend, Vault Key backend etc.) and Helm support.
"},{"location":"kap_proposals/kap_9_bring_your_own_helm/","title":"Bring Your Own Helm Proposal","text":""},{"location":"kap_proposals/kap_9_bring_your_own_helm/#the-problem","title":"The Problem","text":"Currently the helm binding can't be run on Mac OSX. Attempts to fix this have been made on several occasions:
- https://github.com/kapicorp/kapitan/pull/414
- https://github.com/kapicorp/kapitan/pull/547
- https://github.com/kapicorp/kapitan/pull/568
There are some issues with the current bindings besides the lack of Mac OSX support. The golang runtime (1.14) selected will effect older versions helm templates: https://github.com/helm/helm/issues/7711. Users can't select the version of helm they'd like to use for templating.
"},{"location":"kap_proposals/kap_9_bring_your_own_helm/#solution","title":"Solution","text":"Users supply their own helm binary. This allows them to control the version of golang runtime and version of helm they'd like to use.
In Kapitan we could rewrite the interface to use subprocess and perform commands. The cli of helm 2 vs helm 3 is slightly different but shouldn't be difficult to codify.
This would be great to get rid of cffi and golang which will reduce complexity and build time of the project.
Depending on how this goes, this could pave the way for a \"bring your own binary\" input type.
"},{"location":"pages/core_concepts/","title":"Kapitan Overview","text":""},{"location":"pages/core_concepts/#kapitan-at-a-glance","title":"Kapitan at a glance","text":"Kapitan is a powerful configuration management tool designed to help engineers manage complex systems through code. It centralizes and simplifies the management of configurations with a structured approach that revolves around a few core concepts.
Kapitan diagram %%{ init: { securityLevel: 'loose'} }%%\ngraph LR\n classDef pink fill:#f9f,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef blue fill:#00FFFF,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n TARGET1 --> KAPITAN\n TARGET2 --> KAPITAN\n TARGETN --> KAPITAN\n KAPITAN --> EXTERNAL\n KAPITAN --> GENERATORS\n KAPITAN --> HELM\n KAPITAN --> JINJA\n KAPITAN --> JSONNET\n KAPITAN --> KADET\n EXTERNAL --> OUTPUT\n GENERATORS --> OUTPUT\n JINJA --> OUTPUT\n JSONNET --> OUTPUT\n KADET --> OUTPUT\n HELM --> OUTPUT\n GKMS --> REFERENCES\n AWSKMS --> REFERENCES\n VAULT --> REFERENCES\n OTHER --> REFERENCES\n PLAIN --> REFERENCES\n OUTPUT --> TARGETN_OUTPUT\n OUTPUT --> TARGET1_OUTPUT\n OUTPUT --> TARGET2_OUTPUT\n REFERENCES --> KAPITAN\n TARGET1_OUTPUT --> DOCUMENTATION\n TARGET1_OUTPUT --> KUBERNETES\n TARGET1_OUTPUT --> SCRIPTS\n TARGET1_OUTPUT --> TERRAFORM\n CLASSES --> TARGET1\n CLASSES --> TARGET2\n CLASSES --> TARGETN\n\n subgraph \"Inventory\"\n CLASSES[classes]\n TARGET1([\"target 1\"]):::pink\n TARGET2([\"target 2\"])\n TARGETN([\"target N\"])\n end\n\n subgraph \"references\"\n direction TB\n GKMS[\"GCP KMS\"]\n AWSKMS[\"AWS KMS\"]\n VAULT[\"Hashicorp Vault\"]\n OTHER[\"others\"]\n PLAIN[\"plain\"]\n REFERENCES[\"references\"]\n end\n\n KAPITAN((\"<img src='/images/kapitan_logo.png'; width='80'/>\")):::blue\n click EXTERNAL \"/compile#external\"\n\n subgraph \"Input Types\"\n EXTERNAL[\"external\"]\n GENERATORS[\"generators\"]\n HELM[\"helm\"]\n JINJA[\"jinja\"]\n JSONNET[\"jsonnet\"]\n KADET[\"kadet\"]\n end\n\n OUTPUT{{\"compiled output\"}}:::blue\n\n\n\n subgraph \" \"\n TARGET1_OUTPUT([target1]):::pink\n DOCUMENTATION[\"docs\"]\n KUBERNETES[\"manifests\"]\n SCRIPTS[\"scripts\"]\n TERRAFORM[\"terraform\"]\n end\n\n TARGET2_OUTPUT([\"target 2\"])\n TARGETN_OUTPUT([\"target N\"])\n
Let's explore these concepts in a way that's accessible to new users:
"},{"location":"pages/core_concepts/#inventory","title":"Inventory","text":"At the core of Kapitan lies the Inventory, a structured database of variables meticulously organized in YAML files.
This hierarchical setup serves as the single source of truth (SSOT) for your system's configurations, making it easier to manage and reference the essential components of your infrastructure.
Whether you're dealing with Kubernetes configurations, Terraform resources, or even business logic, the Inventory allows you to define and store these elements efficiently. This central repository then feeds into Kapitan's templating engines, enabling seamless reuse across various applications and services.
"},{"location":"pages/core_concepts/#input-types","title":"Input Types","text":"Kapitan takes the information stored in the Inventory and brings it to life through its templating engines upon compilation. Some of the supported input types are: Python, Jinja2, Jsonnet, Helm, and we're adding more soon.
This process transforms static data into dynamic configurations, capable of generating a wide array of outputs like Kubernetes manifests, Terraform plans, documentation, and scripts.
It's about making your configurations work for you, tailored to the specific needs of your projects.
See Input Types for more
"},{"location":"pages/core_concepts/#generators","title":"Generators","text":"Generators offer a straightforward entry point into using Kapitan, requiring minimal to no coding experience.
These are essentially pre-made templates that allow you to generate common configuration files, such as Kubernetes manifests, directly from your Inventory data.
Kapitan provides a wealth of resources, including the Kapitan Reference GitHub repository and various blog posts, to help users get up and running with generators.
"},{"location":"pages/core_concepts/#kadet","title":"Kadet","text":"For those looking to leverage the full power of Kapitan, Kadet introduces a method to define and reuse complex configurations through Python.
This internal library facilitates the creation of JSON and YAML manifests programmatically, offering a higher degree of customization and reuse. Kadet empowers users to craft intricate configurations with the simplicity and flexibility of Python.
"},{"location":"pages/core_concepts/#references","title":"References","text":"Kapitan References provide a secure way to store passwords, settings, and other essential data within your project. Think of them as special code placeholders.
- Flexibility: Update a password once, and Kapitan updates it everywhere automatically.
- Organization: References tidy up your project, especially when you're juggling multiple settings or environments (dev, staging, production). Security: Protect sensitive information like passwords with encryption
Tip
Use Tesoro, our Kubernetes Admission Controller, to complete your integration with Kubernetes for secure secret decryption on-the-fly.
"},{"location":"pages/external_dependencies/","title":"External dependencies","text":"Kapitan has the functionality to fetch external dependencies from remote locations.
Supported dependencies types are:
"},{"location":"pages/external_dependencies/#usage","title":"Usage","text":"Kapitan by default will not attempt to download any dependency, and rely on what is already available.
"},{"location":"pages/external_dependencies/#basic-fetching","title":"Basic fetching","text":"You can use the fetch option to explicitly fetch the dependencies:
clidotfile kapitan compile --fetch\n
.kapitan
to make it default, then simply use kapitan compile
...\ncompile:\n fetch: true\n
This will download the dependencies and store them at their respective output_path.
"},{"location":"pages/external_dependencies/#overwrite-local-changes","title":"Overwrite local changes","text":"When fetching a dependency, Kapitan will refuse to overwrite existing files to preserve your local modifications.
Use the force-fetch option to force overwrite your local files in the output_path.
clidotfile kapitan compile --force-fetch\n
.kapitan
to make it default, then simply use kapitan compile
...\ncompile:\n force-fetch: true\n
"},{"location":"pages/external_dependencies/#caching","title":"Caching","text":"Kapitan also supports caching Use the --cache flag to cache the fetched items in the .dependency_cache directory in the root project directory.
```shell\nkapitan compile --cache --fetch\n```\n
"},{"location":"pages/external_dependencies/#defining-dependencies","title":"Defining dependencies","text":"githttphelm"},{"location":"pages/external_dependencies/#syntax","title":"Syntax","text":"parameters:\n kapitan:\n dependencies:\n - type: git\n output_path: path/to/dir\n source: git_url # mkdocs (1)!\n subdir: relative/path/from/repo/root (optional) # mkdocs (2)!\n ref: tag, commit, branch etc. (optional) # mkdocs (3)!\n submodules: true/false (optional) # mkdocs (4)!\n
- Git types can fetch external
git repositories through either HTTP/HTTPS or SSH URLs. - Optional supports for cloning just a sub-directory
- Optional support for accessing them in specific commits and branches (refs).
- Optional support to disable fetching the submodules of a repo.
Note
This type depends on the git binary installed on your system and available to Kapitan.
"},{"location":"pages/external_dependencies/#example","title":"Example","text":"Say we want to fetch the source code from our kapitan repository, specifically, kapicorp/kapitan/kapitan/version.py. Let's create a very simple target file inventory/targets/kapitan-example.yml.
parameters:\n kapitan:\n vars:\n target: kapitan-example\n dependencies:\n - type: git\n output_path: source/kapitan\n source: git@github.com:kapicorp/kapitan.git\n subdir: kapitan\n ref: master\n submodules: true\n compile:\n - input_paths:\n - source/kapitan/version.py\n input_type: jinja2 # just to copy the file over to target\n output_path: .\n
"},{"location":"pages/external_dependencies/#syntax_1","title":"Syntax","text":"parameters:\n kapitan:\n dependencies:\n - type: http | https # mkdocs (2)!\n output_path: path/to/file # mkdocs (1)!\n source: http[s]://<url> # mkdocs (2)!\n unpack: True | False # mkdocs (3)!\n
output_path must fully specify the file name. For example:- http[s] types can fetch external dependencies available at
http:// or https:// URL. - archive mode: download and unpack
"},{"location":"pages/external_dependencies/#example_1","title":"Example","text":"Single fileArchive Say we want to download kapitan README.md file. Since it's on Github, we can access it as https://raw.githubusercontent.com/kapicorp/kapitan/master/README.md. Using the following inventory, we can copy this to our target folder:
parameters:\n kapitan:\n vars:\n target: kapitan-example\n dependencies:\n - type: https\n output_path: README.md\n source: https://raw.githubusercontent.com/kapicorp/kapitan/master/README.md\n compile:\n - input_paths:\n - README.md\n input_type: jinja2\n output_path: .\n
"},{"location":"pages/external_dependencies/#syntax_2","title":"Syntax","text":"parameters:\n kapitan:\n dependencies:\n - type: helm\n output_path: path/to/chart\n source: http[s]|oci://<helm_chart_repository_url>\n version: <specific chart version>\n chart_name: <name of chart>\n helm_path: <helm binary>\n
Fetches helm charts and any specific subcharts in the requirements.yaml file.
helm_path can be used to specify where the helm binary name or path. It defaults to the value of the KAPITAN_HELM_PATH environment var or simply to helm if neither is set. You should specify only if you don't want the default behavior.
source can be either the URL to a chart repository, or the URL to a chart on an OCI registry (supported since Helm 3.8.0).
"},{"location":"pages/external_dependencies/#example_2","title":"Example","text":"If we want to download the prometheus helm chart we simply add the dependency to the monitoring target. We want a specific version 11.3.0 so we put that in.
parameters:\n kapitan:\n vars:\n target: monitoring\n dependencies:\n - type: helm\n output_path: charts/prometheus\n source: https://kubernetes-charts.storage.googleapis.com\n version: 11.3.0\n chart_name: prometheus\n compile:\n - input_type: helm\n output_path: .\n input_paths:\n - charts/prometheus\n helm_values:\n alertmanager:\n enabled: false\n helm_params:\n namespace: monitoring\n name: prometheus\n
"},{"location":"pages/remote_repositories/","title":"Remote Inventories","text":"Kapitan is capable of recursively fetching inventory items stored in remote locations and copy it to the specified output path. This feature can be used by specifying those inventory items in classes or targets under parameters.kapitan.inventory. Supported types are:
Class items can be specified before they are locally available as long as they are fetched in the same run. Example of this is given below.
"},{"location":"pages/remote_repositories/#git-type","title":"Git type","text":"Git types can fetch external inventories available via HTTP/HTTPS or SSH URLs. This is useful for fetching repositories or their sub-directories, as well as accessing them in specific commits and branches (refs).
Note: git types require git binary on your system.
"},{"location":"pages/remote_repositories/#definition","title":"Definition","text":"parameters:\n kapitan:\n inventory:\n - type: git\n output_path: path/to/dir\n source: git_url\n subdir: relative/path/from/repo/root (optional)\n ref: tag, commit, branch etc. (optional)\n
"},{"location":"pages/remote_repositories/#example","title":"Example","text":"Lets say we want to fetch a class from our kapitan repository, specifically kapicorp/kapitan/tree/master/examples/docker/inventory/classes/dockerfiles.yml.
Lets create a simple target file docker.yml
Note
external dependencies are used to fetch dependency items in this example.
targets/docker.yml
classes:\n - dockerfiles\nparameters:\n kapitan:\n vars:\n target: docker\n inventory:\n - type: git\n source: https://github.com/kapicorp/kapitan\n subdir: examples/docker/inventory/classes/\n output_path: classes/\n dependencies:\n - type: git\n source: https://github.com/kapicorp/kapitan\n subdir: examples/docker/components\n output_path: components/\n - type: git\n source: https://github.com/kapicorp/kapitan\n subdir: examples/docker/templates\n output_path: templates/\n dockerfiles:\n - name: web\n image: amazoncorretto:11\n - name: worker\n image: amazoncorretto:8\n
kapitan compile --fetch\n
click to expand output [WARNING] Reclass class not found: 'dockerfiles'. Skipped!\n[WARNING] Reclass class not found: 'dockerfiles'. Skipped!\nInventory https://github.com/kapicorp/kapitan: fetching now\nInventory https://github.com/kapicorp/kapitan: successfully fetched\nInventory https://github.com/kapicorp/kapitan: saved to inventory/classes\nDependency https://github.com/kapicorp/kapitan: saved to components\nDependency https://github.com/kapicorp/kapitan: saved to templates\nCompiled docker (0.11s)\n
"},{"location":"pages/remote_repositories/#http-type","title":"http type","text":"http[s] types can fetch external inventories available at http:// or https:// URL.
"},{"location":"pages/remote_repositories/#definition_1","title":"Definition","text":"parameters:\n kapitan:\n inventory:\n - type: http | https\n output_path: full/path/to/file.yml\n source: http[s]://<url>\n unpack: True | False # False by default\n
"},{"location":"pages/remote_repositories/#example_1","title":"Example","text":"targets/mysql-generator-fetch.yml
classes:\n - common\n - kapitan.generators.kubernetes\nparameters:\n kapitan:\n inventory:\n - type: https\n source: https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml\n output_path: classes/kapitan/generators/kubernetes.yml\n components:\n mysql:\n image: mysql\n
kapitan compile --fetch\n
click to expand output ./kapitan compile -t mysql-generator-fetch --fetch\nInventory https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml: fetching now\nInventory https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml: successfully fetched\nInventory https://raw.githubusercontent.com/kapicorp/kapitan-reference/master/inventory/classes/kapitan/generators/kubernetes.yml: saved to inventory/classes/kapitan/generators/kubernetes.yml\n\n...\ncut\n...\n\nCompiled mysql-generator-fetch (0.06s)\n
"},{"location":"pages/blog/","title":"Blog","text":""},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/","title":"5 Years of Kapitan","text":"Last October we quietly celebrated 5 years of Kapitan.
In 5 years, we've been able to witness a steady and relentless of Kapitan, which has however never caught the full attention of the majority of the community.
The main issue has always been around an embarassing lack of documentation, and we've worked hard to improve on that, with more updates due soon.
Let this first blog post from a revamped website be a promise to our community of a better effort in explaining what sets Kapitan apart, and makes it the only tool of its kind.
And let's start with a simple question: Why do you even need Kapitan?
Credits
In reality Kapitan's heatbeat started about 9 months earlier at DeepMind Health, created by [**Ricardo Amaro**](https://github.com/ramaro) with the help of some of my amazing team: in no particular order [Adrian Chifor](https://github.com/adrianchifor), [Paul S](https://github.com/uberspot) and [Luis Buriola](https://github.com/gburiola). It was then kindly released to the community by Google/DeepMind and is has so been improved thanks to more than [50 contributors](https://github.com/kapicorp/kapitan/graphs/contributors).\n
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#why-do-i-need-kapitan","title":"Why do I need Kapitan?","text":"Kapitan is a hard sell, but a rewarding one. For these main reasons:
- Kapitan solves problems that some don\u2019t know/think to have.
- Some people by now have probably accepted the Status Quo and think that some suffering is part of their job descriptions.
- Objectively, Kapitan requires an investment of effort to learn how to use a new tool, and this adds friction.
All I can say it is very rewarding once you get to use it, so stick with me while I try to explain the problems that Kapitan is solving
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#the-problems","title":"The problems","text":"It would be reductive to list the problems that Kapitan solves, because sometimes we ourselves are stunned by what Kapitan is being used for, so I will start with some common relatable ones, and perhaps that will give you the right framing to understand how to use it with your setup.
In its most basic explanation, Kapitan solves the problem of avoiding duplication of configuration data: by consolidating it in one place (the Inventory), and making it accessible by all the tools and languages it integrates with (see Input Types).
This configuration data is then used by Kapitan (templates) to configure and operate a number of completely distinct and unaware tools which would normally not be able to share their configurations.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#without-kapitan","title":"Without Kapitan","text":"Let's consider the case where you want to define a new bucket, with a given bucket_name. Without Kapitan you would probably need to:
- Write a PR on your Terraform repository to create the new bucket.
- Which name should I use? Make sure to write it down!
CTRL-C - Write a PR for your
values.yaml file to configure your Helm chart: <CTRL-V> - Write somewhere some documentation to write down the bucket name and why it exists. Another
<CTRL-V> - Another PR to change some
**kustomize** configuration for another service to tell it to use the new bucket <CTRL-V> - Days after, time to upload something to that bucket:
gsutil cp my_file wait_what_was_the_bucket_name_again.. Better check the documentation: CTRL-C + <CTRL-V>
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#with-kapitan","title":"With Kapitan","text":"When using Kapitan, your changes are likely to be contained within one PR, from which you can have a full view of everything that is happening. What happens is explained in this flow
\n%%{ init: { securityLevel: 'loose'} }%%\ngraph LR\n classDef pink fill:#f9f,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef blue fill:#00FFFF,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef bold color:#000,font-weight: bold;\n\n DATA --> KAPITAN\n BUCKET --> DATA\n KAPITAN --> KUBERNETES\n KAPITAN --> TERRAFORM\n KAPITAN --> DOCUMENTATION\n KAPITAN --> SCRIPT\n KAPITAN --> HELM\n KUBERNETES --> BUCKET_K8S\n TERRAFORM --> BUCKET_TF\n DOCUMENTATION --> BUCKET_DOC\n SCRIPT --> BUCKET_SCRIPT\n HELM --> BUCKET_HELM\n\n\n DATA[(\"All your data\")]\n BUCKET(\"bucket_name\")\n KAPITAN((\"<img src='/images/kapitan_logo.png'; width='150'/>\")):::blue\n\n\n subgraph \" \"\n KUBERNETES([\"Kubernetes\"]):::pink\n BUCKET_K8S(\".. a ConfigMap uses bucket_name\"):::bold\n end\n subgraph \" \"\n TERRAFORM([\"Terraform\"]):::pink\n BUCKET_TF(\"..creates the bucket bucket_name\"):::bold\n end\n subgraph \" \"\n DOCUMENTATION([\"Documentation\"]):::pink\n BUCKET_DOC(\"..references a link to bucket_name\"):::bold\n end\n subgraph \" \"\n SCRIPT([\"Canned Script\"]):::pink\n BUCKET_SCRIPT(\"..knows how to upload files to bucket_name\"):::bold\n end\n subgraph \" \"\n HELM([\"Helm\"]):::pink\n BUCKET_HELM(\"..configures a chart to use the bucket_name\"):::bold\n end
Thanks to its flexiblility, you can use Kapitan to generate all sorts of configurations: Kubernetes and Terraform resources, ArgoCD pipelines, Docker Compose files, random configs, scripts, documentations and anything else you find relevant. The trick is obviously on how to drive these changes, but it is not as complicated as it sounds. We'll get there soon enough!
Let's see now another example of things that are so established in the way to do things that become elusivly impossible to see. As a way to highlight the potential issues with this way of doing things, let's ask some questions on your current setup. We pick on Kubernetes this time.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/#kubernetes","title":"Kubernetes","text":"I\u2019ll start with Kubernetes, such a popular and brilliant solution to problems most people should not be concerned with (jokes apart, I adore Kubernetes). To most, Kubernetes is that type of solution that quickly turns into a problem of its own right.
So.. how do you deploy to Kubernetes right now?
Helm comes to mind first, right?
Kapitan + Helm: BFF
In spite of Kapitan being initially considered (even by ourselves) as an alternative to Helm, we\u2019ve actually enjoyed the benefits of integrating with this amazing tool and the ecosystem it gives us access to. So yes, good news: you can use Helm right from within Kapitan!.
Well, let\u2019s put that to a test. How do you manage your Helm charts? I\u2019ll attempt to break these questions down into categories.
Code OrganizationDRYMaintenanceOperationsDocumentationSecrets managementEverything else - Where do you keep your Helm charts?
- In a single repository?
- How many repositories?
- Alongside the code you develop?
- What about the official ones that you didn't create yourself?
- How many
values.yaml files do you have? - How much consistency is there between them? any snowflakes?
- If you change something, like with the
bucket_name example above: - how many places do you need to go and update?
- And how many times do you get it wrong?
- Don't you feel all your charts look the same?
- Yet how many times do you need to deviate from the one you thought captured everything?
- What if you need to make a change to all your charts at once: how do you deal with it?
- What about configuration files, how do you deal with templating those?
- How do you deal with \u201cofficial\u201d charts, do they always cover what you want to do?
- How do you deal with modifications that you need to apply to your own version of a an official chart?
- What if you need to make a change that affects ALL your charts?
- Or if the change is for all the charts for a set of microservices?
- How many times you find yourself seting parameters on the command line of Helm and other tools?
- How many times did you connect to the wrong context in Kubernetes
- How many of your colleagues have the same clean context setup as you have?
- How many things are there that you wish you were tracking?
- How do I connect to the production database? Which user is it again?
- How easy is it for you to create a new environment from scratch?
- Are you sure?
- When was the last time you tried?
- How easy is it to keep your configuration up to date?
- Does your documentation need to be \u201cunderstood\u201d or can be just executed on?
- Would you be able to follow those instructions at 3am on a Sunday morning?
- How do you handle secrets in your repository?
- Do you know how to create your secrets from scratch?
- Do you remember that token you created 4 months ago? How did you do that?
- How long would it take you?
- Is the process of creating them \u201csecure\u201d?
- Or does it leave you with random certificates and tokens unencrypted on your \u201cDownloads\u201d folder?
- The above concerns: do they also apply to other things you manage?
- Terraform?
- Pipelines?
- Random other systems you interact with?
I\u2019ll stop here because I do not want to lose you, and neither do I want to discourage you.
But if you look around it\u2019s true, you do have a very complicated setup. And Kapitan can help you streamline it for you. In fact, Kapitan can leave you with a consistent and uniform way to manage all these concerns at once.
My job here is done: you have awakened and you won't look at your setup in the same way. Keep tuned and learn about how Kapitan can change the way you do things.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-new-kapitan-release--v0310/","title":"New Kapitan release v0.31.0","text":"The Kapicorp team is happy to to announce a new release of Kapitan.
This release is yet another great bundle of features and improvements over the past year, the majority of which have been contributions from our community!
Head over our release page on GitHub for a full list of features and contributors.
If you missed it, have a look at our latest blog post here 5 years of Kapitan
Please help us by visiting our Sponsor Kapitan page.
"},{"location":"pages/blog/04/12/2022/kapitan-logo-new-kapitan-release--v0320/","title":"New Kapitan release v0.32.0","text":"The Kapicorp team is happy to to announce a new release of Kapitan.
This release contains loads of improvements for the past 6 months, the majority of which have been contributions from our community!
Head over our release page on GitHub for a full list of features and contributors.
Please help us by visiting our Sponsor Kapitan page.
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/","title":"Deploying Keda with Kapitan","text":"We have worked hard to bring out a brand new way of experience Kapitan, through something that we call generators
Although the concept is something we've introduced in 2020 with our blog post Keep your ship together with Kapitan, the sheer amount of new capabilities (and frankly, the embarassing lack of documentation and examples) forces me to show you the new capabilities using a practicle example: deploying Keda.
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#objective-of-this-tutorial","title":"Objective of this tutorial","text":"We are going to deploy Keda using the helm chart approach. While Kapitan supports a native way to deploy helm charts using the helm input type, we are going instead to use a generator based approach using the \"charts\" generator.
This tutorial will show you how to configure kapitan to:
- download a helm chart
- compile a helm chart
- modify a helm chart using mutations
The content of this tutorial is already available on the kapitan-reference
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#deploying-keda","title":"Deploying KEDA","text":""},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#define-parameters","title":"Define parameters","text":"## inventory/classes/components/keda.yml\nparameters:\n keda:\n params:\n # Variables to reference from other places\n application_version: 2.11.2\n service_account_name: keda-operator\n chart_name: keda\n chart_version: 2.11.2\n chart_dir: system/sources/charts/${keda:params:chart_name}/${keda:params:chart_name}/${keda:params:chart_version}/${keda:params:application_version}\n namespace: keda\n helm_values: {}\n...\n
Override Helm Values
As an example we could be passing to helm an override to the default values parameters to make the operator deploy 2 replicas.
helm_values:\n operator:\n replicaCount: 2\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#download-the-chart","title":"Download the chart","text":"Kapitan supports downloading dependencies, including helm charts.
When Kapitan is run with the --fetch, it will download the dependency if not already present. Use --force-fetch if you want to download it every time. Learn more about External dependencies
## inventory/classes/components/keda.yml\n...\n kapitan:\n dependencies:\n # Tells kapitan to download the helm chart into the chart_dir directory\n - type: helm\n output_path: ${keda:params:chart_dir}\n source: https://kedacore.github.io/charts\n version: ${keda:params:chart_version}\n chart_name: ${keda:params:chart_name}\n...\n
Parameter interpolation
Notice how we are using parameter interpolation from the previously defined keda.params section. This will make it easier in the future to override some aspects of the configuration on a per-target base.
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#generate-the-chart","title":"Generate the chart","text":"## inventory/classes/components/keda.yml\n...\n charts:\n # Configures a helm generator to compile files for the given chart\n keda:\n chart_dir: ${keda:params:chart_dir}\n helm_params:\n namespace: ${keda:params:namespace}\n name: ${keda:params:chart_name}\n helm_values: ${keda:params:helm_values}\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#compile","title":"Compile","text":"Before we can see any effect, we need to attach the class to a target. We will create a simple target which looks like
# inventory/targets/tutorials/keda.yml\nclasses:\n- common\n- components.keda\n
Now when we run kapitan compile we will see the chart being donwloaded and the manifests being produced.
./kapitan compile -t keda --fetch\nDependency keda: saved to system/sources/charts/keda/keda/2.11.2/2.11.2\nRendered inventory (1.87s)\nCompiled keda (2.09s)\n
kapitan compile breakdown
--fetch tells kapitan to fetch the chart if it is not found locally-t keda tells kapitan to compile only the previously defined keda.yml target
ls -l compiled/keda/manifests/\ntotal 660\n-rw-r--r-- 1 ademaria root 659081 Aug 29 10:25 keda-bundle.yml\n-rw-r--r-- 1 ademaria root 79 Aug 29 10:25 keda-namespace.yml\n-rw-r--r-- 1 ademaria root 7092 Aug 29 10:25 keda-rbac.yml\n-rw-r--r-- 1 ademaria root 1783 Aug 29 10:25 keda-service.yml\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#using-mutations","title":"Using mutations","text":"Now let's do a couple of things that would not be easy to do with helm natively.
You can already notice that the content of the chart is being splitted into multiple files: this is because the Generator is configured to separate different resources types into different files for convenience and consistency. The mechanism behing it is the \"Mutation\" of type \"bundle\" which tells Kapitan which file to save a resource into.
Here are some example \"mutation\" which separates different kinds into different files
mutations:\n bundle:\n - conditions:\n kind: [Ingress]\n filename: '{content.component_name}-ingress'\n ...\n - conditions:\n kind: [HorizontalPodAutoscaler, PodDisruptionBudget, VerticalPodAutoscaler]\n filename: '{content.component_name}-scaling'\n - conditions:\n kind: ['*']\n filename: '{content.component_name}-bundle'\n
Catch-all rule
Notice the catchall rule at the end that puts everything that has not matched into the bundle.yml file
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#bundle-mutation","title":"bundle mutation","text":"Currently most of the keda related resources are bundled into the -bundle.yml file Instead, we want to separate them into their own file.
Let's add this configuration:
charts:\n # Configures a helm generator to compile files for the given chart\n keda:\n chart_dir: ${keda:params:chart_dir}\n ...\n mutations:\n bundle:\n - conditions:\n # CRDs need to be setup separately\n kind: [CustomResourceDefinition]\n filename: '{content.component_name}-crds'\n
Upon compile, you can now see that the CRD are being moved to a different file:
ls -l compiled/keda/manifests/\ntotal 664\n-rw-r--r-- 1 ademaria root 11405 Aug 29 10:56 keda-bundle.yml\n-rw-r--r-- 1 ademaria root 647672 Aug 29 10:56 keda-crds.yml\n-rw-r--r-- 1 ademaria root 79 Aug 29 10:56 keda-namespace.yml\n-rw-r--r-- 1 ademaria root 7092 Aug 29 10:56 keda-rbac.yml\n-rw-r--r-- 1 ademaria root 1783 Aug 29 10:56 keda-service.yml\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#patch-mutation","title":"patch mutation","text":"As we are using Argo, we want to pass a special argocd.argoproj.io/sync-options annotation to the CRD only so that ArgoCD can handle them properly.
For this we are going to use the patch mutation:
...\n mutations:\n...\n patch:\n - conditions:\n kind: [CustomResourceDefinition]\n patch:\n metadata:\n annotations:\n argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true,Replace=true\n
Upon compile, you can now see that the CRDs have been modified as required:
diff --git a/compiled/keda/manifests/keda-crds.yml b/compiled/keda/manifests/keda-crds.yml\nindex 2662bf3..9306c3a 100644\n--- a/compiled/keda/manifests/keda-crds.yml\n+++ b/compiled/keda/manifests/keda-crds.yml\n@@ -2,6 +2,7 @@ apiVersion: apiextensions.k8s.io/v1\n kind: CustomResourceDefinition\n metadata:\n annotations:\n+ argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true,Replace=true\n controller-gen.kubebuilder.io/version: v0.12.0\n
"},{"location":"pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/#summary","title":"Summary","text":"With this tutorial have explored some capabilities of Kapitan to manage and perform changes to helm charts. Next tutorial will show how to make use of Keda and deploy a generator for Keda resources
"},{"location":"pages/blog/12/02/2024/kapitan-logo-new-kapitan-release--v0331/","title":"New Kapitan release v0.33.1","text":"The Kapicorp team is happy to to announce a new release of Kapitan.
This release contains loads of improvements for the past 8 months, the majority of which have been contributions from our community!
Head over our release page on GitHub for a full list of features and contributors.
Please help us by visiting our Sponsor Kapitan page.
"},{"location":"pages/commands/kapitan_compile/","title":"CLI Reference | kapitan compile","text":""},{"location":"pages/commands/kapitan_compile/#kapitan-compile","title":"kapitan compile","text":"Merges inventory and inputs and produces generated files in the output folder (/compiled by default)
"},{"location":"pages/commands/kapitan_compile/#compile-all-targets","title":"Compile all targets","text":"kapitan compile\n
click to expand output Compiled mysql-generator-fetch (0.18s)\nCompiled vault (0.25s)\nCompiled pritunl (0.22s)\nCompiled gke-pvm-killer (0.05s)\nCompiled examples (0.30s)\nCompiled mysql (0.08s)\nCompiled postgres-proxy (0.06s)\nCompiled echo-server (0.06s)\nCompiled global (0.03s)\nCompiled guestbook-argocd (0.08s)\nCompiled tutorial (0.13s)\nCompiled kapicorp-project-123 (0.03s)\nCompiled kapicorp-demo-march (0.03s)\nCompiled kapicorp-terraform-admin (0.03s)\nCompiled sock-shop (0.32s)\nCompiled tesoro (0.09s)\nCompiled dev-sockshop (0.32s)\nCompiled prod-sockshop (0.38s)\nCompiled argocd (2.29s)\n
"},{"location":"pages/commands/kapitan_compile/#selective-compilation","title":"Selective compilation","text":""},{"location":"pages/commands/kapitan_compile/#using-target-names","title":"Using target names","text":"Compiles one or more targets selected by name using --targets or -t
kapitan compile -t mysql tesoro\n
click to expand output Compiled mysql (0.06s)\nCompiled tesoro (0.09s)\n
"},{"location":"pages/commands/kapitan_compile/#using-labels","title":"Using labels","text":"Compiles one or more targets selected matching labels with --labels or -l
Info
This works if you have labelled your targets using the following syntax:
parameters:\n ...\n kapitan:\n ...\n labels:\n customer: acme\n
see Labels for more details
$ kapitan compile -l customer=acme\nCompiled acme-project (0.14s)\nCompiled acme-pipelines (0.10s)\n
"},{"location":"pages/commands/kapitan_compile/#fetch-on-compile","title":"Fetch on compile","text":"Use the --fetch flag to fetch Remote Inventories and the External Dependencies.
kapitan compile --fetch\n
This will download the dependencies according to their configurations By default, kapitan does not overwrite an existing item with the same name as that of the fetched inventory items.
Use the --force-fetch flag to force fetch (update cache with freshly fetched items) and overwrite inventory items of the same name in the output_path.
kapitan compile --force-fetch\n
Use the --cache flag to cache the fetched items in the .dependency_cache directory in the root project directory.
kapitan compile --cache --fetch\n
"},{"location":"pages/commands/kapitan_compile/#embed-references","title":"Embed references","text":"By default, Kapitan references are stored encrypted (for backends that support encription) in the configuration repository under the /refs directory.
For instance, a reference tag ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de} would point to a phisical file on disk under /refs like:
refs/targets/minikube-mysql/mysql/password
data: hQEMA8uOJKdm07XTAQgAp5i [[ CUT ]] BwqYc3g7PI09HCJZdU=\nencoding: base64\nrecipients:\n- fingerprint: D9234C61F58BEB3ED8552A57E28DC07A3CBFAE7C\ntype: gpg\n
The --embed-refs flags tells Kapitan to embed these references on compile, alongside the generated output. By doing so, compiled output is self-contained and can be revealed by Tesoro or other tools.
kapitan compile --embed-refs\n
See how the compiled output for this specific target changes to embed the actul encrypted content, (marked by ?{gpg: :embedded} to indicate it is a gpg reference) rather than just holding a reference to it (like in this case ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de} which points to ).
click to expand output diff --git a/examples/kubernetes/compiled/minikube-mysql/manifests/mysql_app.yml b/examples/kubernetes/compiled/minikube-mysql/manifests/mysql_app.yml\n[[ CUT ]]\napiVersion: v1\ndata:\n- MYSQL_ROOT_PASSWORD: ?{gpg:targets/minikube-mysql/mysql/password:ec3d54de}\n- MYSQL_ROOT_PASSWORD_SHA256: ?{gpg:targets/minikube-mysql/mysql/password_sha256:122d2732}\n+ MYSQL_ROOT_PASSWORD: ?{gpg:eyJkYXRhIjogImhR [[ CUT ]] gInR5cGUiOiAiZ3BnIn0=:embedded}\n+ MYSQL_ROOT_PASSWORD_SHA256: ?{gpg:eyJkYXRhI [[ CUT ]] eXBlIjogImdwZyJ9:embedded}\n
"},{"location":"pages/commands/kapitan_compile/#help","title":"help","text":"kapitan compile --help\n
click to expand output usage: kapitan compile [-h] [--inventory-backend {reclass}]\n [--search-paths JPATH [JPATH ...]]\n [--jinja2-filters FPATH] [--verbose] [--prune]\n [--quiet] [--output-path PATH] [--fetch]\n [--force-fetch] [--force] [--validate]\n [--parallelism INT] [--indent INT]\n [--refs-path REFS_PATH] [--reveal] [--embed-refs]\n [--inventory-path INVENTORY_PATH] [--cache]\n [--cache-paths PATH [PATH ...]]\n [--ignore-version-check] [--use-go-jsonnet]\n [--compose-target-name] [--schemas-path SCHEMAS_PATH]\n [--yaml-multiline-string-style STYLE]\n [--yaml-dump-null-as-empty]\n [--targets TARGET [TARGET ...] | --labels\n [key=value ...]]\n\noptions:\n -h, --help show this help message and exit\n --inventory-backend {reclass,reclass-rs}\n Select the inventory backend to use (default=reclass)\n --search-paths JPATH [JPATH ...], -J JPATH [JPATH ...]\n set search paths, default is [\".\"]\n --jinja2-filters FPATH, -J2F FPATH\n load custom jinja2 filters from any file, default is\n to put them inside lib/jinja2_filters.py\n --verbose, -v set verbose mode\n --prune prune jsonnet output\n --quiet set quiet mode, only critical output\n --output-path PATH set output path, default is \".\"\n --fetch fetch remote inventories and/or external dependencies\n --force-fetch overwrite existing inventory and/or dependency item\n --force overwrite existing inventory and/or dependency item\n --validate validate compile output against schemas as specified\n in inventory\n --parallelism INT, -p INT\n Number of concurrent compile processes, default is 4\n --indent INT, -i INT Indentation spaces for YAML/JSON, default is 2\n --refs-path REFS_PATH\n set refs path, default is \"./refs\"\n --reveal reveal refs (warning: this will potentially write\n sensitive data)\n --embed-refs embed ref contents\n --inventory-path INVENTORY_PATH\n set inventory path, default is \"./inventory\"\n --cache, -c enable compilation caching to .kapitan_cache and\n dependency caching to .dependency_cache, default is\n False\n --cache-paths PATH [PATH ...]\n cache additional paths to .kapitan_cache, default is\n []\n --ignore-version-check\n ignore the version from .kapitan\n --use-go-jsonnet use go-jsonnet\n --compose-target-name Create same subfolder structure from inventory/targets\n inside compiled folder\n --schemas-path SCHEMAS_PATH\n set schema cache path, default is \"./schemas\"\n --yaml-multiline-string-style STYLE, -L STYLE\n set multiline string style to STYLE, default is\n 'double-quotes'\n --yaml-dump-null-as-empty\n dumps all none-type entries as empty, default is\n dumping as 'null'\n --targets TARGET [TARGET ...], -t TARGET [TARGET ...]\n targets to compile, default is all\n --labels [key=value ...], -l [key=value ...]\n compile targets matching the labels, default is all\n
"},{"location":"pages/commands/kapitan_dotfile/","title":"CLI Reference | .kapitan config file","text":""},{"location":"pages/commands/kapitan_dotfile/#kapitan","title":".kapitan","text":"Kapitan allows you to coveniently override defaults by specifying a local .kapitan file in the root of your repository (relative to the kapitan configuration):
This comes handy to make sure Kapitan runs consistently for your specific setup.
Info
Any Kapitan command can be overridden in the .kapitan dotfile, but here are some of the most common examples.
"},{"location":"pages/commands/kapitan_dotfile/#version","title":"version","text":"To enforce the Kapitan version used for compilation (for consistency and safety), you can add version to .kapitan:
version: 0.30.0\n\n...\n
This constrain can be relaxed to allow minor versions to be also accepted:
version: 0.30 # Allows any 0.30.x release to run\n\n...\n
"},{"location":"pages/commands/kapitan_dotfile/#command-line-flags","title":"Command line flags","text":"You can also permanently define all command line flags in the .kapitan config file. For example:
...\n\ncompile:\n indent: 4\n parallelism: 8\n
would be equivalent to running:
kapitan compile --indent 4 --parallelism 8\n
For flags which are shared by multiple commands, you can either selectively define them for single commmands in a section with the same name as the command, or you can set any flags in section global, in which case they're applied for all commands. If you set a flag in both the global section and a command's section, the value from the command's section takes precedence over the value from the global section.
As an example, you can configure the inventory-path in the global section of the Kapitan dotfile to make sure it's persisted across all Kapitan runs.
...\n\nglobal:\n inventory-path: ./some_path\n
which would be equivalent to running any command with --inventory-path=./some_path.
Another flag that you may want to set in the global section is inventory-backend to select a non-default inventory backend implementation.
global:\n inventory-backend: reclass\n
which would be equivalent to always running Kapitan with --inventory-backend=reclass.
Please note that the inventory-backend flag currently can't be set through the command-specific sections of the Kapitan config file.
"},{"location":"pages/commands/kapitan_inventory/","title":"CLI Reference | kapitan inventory","text":""},{"location":"pages/commands/kapitan_inventory/#kapitan-inventory","title":"kapitan inventory","text":"Renders the resulting inventory values for a specific target.
For example, rendering the inventory for the mysql target:
kapitan inventory -t mysql\n
click to expand output __reclass__:\n environment: base\n name: mysql\n node: mysql\n timestamp: Wed Nov 23 23:19:28 2022\n uri: yaml_fs:///src/inventory/targets/examples/mysql.yml\napplications: []\nclasses:\n - kapitan.kube\n - kapitan.generators.kubernetes\n - kapitan.generators.argocd\n - kapitan.generators.terraform\n - kapitan.generators.rabbitmq\n - kapitan.common\n - common\n - components.mysql\nenvironment: base\nexports: {}\nparameters:\n _reclass_:\n environment: base\n name:\n full: mysql\n short: mysql\n components:\n mysql:\n config_maps:\n config:\n data:\n mysql.cnf:\n value: ignore-db-dir=lost+found\n mytemplate.cnf:\n template: components/mysql/mytemplate.cnf.j2\n values:\n mysql:\n client:\n port: 3306\n socket: /var/run/mysqld/mysqld.sock\n mysqld:\n bind-address: 127.0.0.1\n max_allowed_packet: 64M\n thread_concurrency: 8\n mount: /etc/mysql/conf.d/\n env:\n MYSQL_DATABASE: ''\n MYSQL_PASSWORD:\n secretKeyRef:\n key: mysql-password\n MYSQL_ROOT_PASSWORD:\n secretKeyRef:\n key: mysql-root-password\n MYSQL_USER: ''\n image: mysql:5.7.28\n ports:\n mysql:\n service_port: 3306\n secrets:\n secrets:\n data:\n mysql-password:\n value: ?{plain:targets/mysql/mysql-password||randomstr|base64}\n mysql-root-password:\n value: ?{plain:targets/mysql/mysql-root-password||randomstr:32|base64}\n versioned: true\n type: statefulset\n volume_claims:\n datadir:\n spec:\n accessModes:\n - ReadWriteOnce\n resources:\n requests:\n storage: 10Gi\n storageClassName: standard\n volume_mounts:\n datadir:\n mountPath: /var/lib/mysql\n docs:\n - templates/docs/README.md\n generators:\n manifest:\n default_config:\n annotations:\n manifests.kapicorp.com/generated: 'true'\n service_account:\n create: false\n type: deployment\n kapitan:\n compile:\n - input_paths:\n - components/generators/kubernetes\n input_type: kadet\n output_path: manifests\n output_type: yml\n - input_params:\n function: generate_docs\n template_path: templates/docs/service_component.md.j2\n input_paths:\n - components/generators/kubernetes\n input_type: kadet\n output_path: docs\n output_type: plain\n - input_params:\n function: generate_pre_deploy\n input_paths:\n - components/generators/kubernetes\n input_type: kadet\n output_path: pre-deploy\n output_type: yml\n - input_paths:\n - components/generators/argocd\n input_type: kadet\n output_path: argocd\n output_type: yml\n - input_params:\n generator_root: resources.tf\n input_paths:\n - components/generators/terraform\n input_type: kadet\n output_path: terraform\n output_type: json\n - ignore_missing: true\n input_paths:\n - resources/state/mysql/.terraform.lock.hcl\n input_type: copy\n output_path: terraform/\n - input_paths:\n - components/generators/rabbitmq\n input_type: kadet\n output_path: rabbitmq\n output_type: yml\n - input_paths:\n - templates/docs/README.md\n input_type: jinja2\n output_path: docs\n - input_paths: []\n input_type: jinja2\n output_path: scripts\n - input_paths: []\n input_type: jsonnet\n output_path: manifests\n output_type: yml\n dependencies:\n - output_path: lib/kube.libsonnet\n source: https://raw.githubusercontent.com/bitnami-labs/kube-libsonnet/master/kube.libsonnet\n type: https\n - output_path: lib/kube-platforms.libsonnet\n source: https://raw.githubusercontent.com/bitnami-labs/kube-libsonnet/master/kube-platforms.libsonnet\n type: https\n - output_path: components/generators/kubernetes\n ref: master\n source: https://github.com/kapicorp/kapitan-reference.git\n subdir: components/generators/kubernetes\n type: git\n - output_path: components/generators/terraform\n ref: master\n source: https://github.com/kapicorp/kapitan-reference.git\n subdir: components/generators/terraform\n type: git\n vars:\n target: mysql\n manifests: []\n mysql:\n settings:\n client:\n port: 3306\n socket: /var/run/mysqld/mysqld.sock\n mysqld:\n bind-address: 127.0.0.1\n max_allowed_packet: 64M\n thread_concurrency: 8\n namespace: mysql\n scripts: []\n target_name: mysql\n
"},{"location":"pages/commands/kapitan_lint/","title":"CLI Reference | kapitan lint","text":""},{"location":"pages/commands/kapitan_lint/#kapitan-lint","title":"kapitan lint","text":"Perform a checkup on your inventory or refs.
./kapitan lint\n
click to expand output Running yamllint on all inventory files...\n\n.yamllint not found. Using default values\nFile ./inventory/classes/components/echo-server.yml has the following issues:\n 95:29: forbidden implicit octal value \"0550\" (octal-values)\nFile ./inventory/classes/terraform/gcp/services.yml has the following issues:\n 15:11: duplication of key \"enable_compute_service\" in mapping (key-duplicates)\n\nTotal yamllint issues found: 2\n\nChecking for orphan classes in inventory...\n\nNo usage found for the following 6 classes:\n{'components.argoproj.cd.argocd-server-oidc',\n'components.helm.cert-manager-helm',\n'components.rabbitmq-operator.rabbitmq-configuration',\n'components.rabbitmq-operator.rabbitmq-operator',\n'features.gkms-demo',\n'projects.localhost.kubernetes.katacoda'}\n
"},{"location":"pages/commands/kapitan_searchvar/","title":"CLI Reference | kapitan searchvar","text":""},{"location":"pages/commands/kapitan_searchvar/#kapitan-searchvar","title":"kapitan searchvar","text":"Shows all inventory files where a variable is declared:
./kapitan searchvar parameters.components.*.image\n
click to expand output ./inventory/classes/components/vault.yml ${vault:image}\n./inventory/classes/components/logstash.yml eu.gcr.io/antha-images/logstash:7.5.1\n./inventory/classes/components/gke-pvm-killer.yml estafette/estafette-gke-preemptible-killer:1.2.5\n./inventory/classes/components/mysql.yml mysql:5.7.28\n./inventory/classes/components/postgres-proxy.yml gcr.io/cloudsql-docker/gce-proxy:1.16\n./inventory/classes/components/echo-server.yml jmalloc/echo-server\n./inventory/classes/components/trivy.yml ${trivy:image}\n./inventory/classes/components/filebeat.yml ${filebeat:image}:${filebeat:version}\n./inventory/classes/components/pritunl/pritunl-mongo.yml docker.io/bitnami/mongodb:4.2.6-debian-10-r23\n./inventory/classes/components/pritunl/pritunl.yml alledm/pritunl\n./inventory/classes/components/weaveworks/user-db.yml weaveworksdemos/user-db:0.3.0\n./inventory/classes/components/weaveworks/catalogue.yml weaveworksdemos/catalogue:0.3.5\n./inventory/classes/components/weaveworks/user.yml weaveworksdemos/user:0.4.7\n./inventory/classes/components/weaveworks/session-db.yml redis:alpine\n./inventory/classes/components/weaveworks/catalogue-db.yml weaveworksdemos/catalogue-db:0.3.0\n./inventory/classes/components/weaveworks/carts-db.yml mongo\n./inventory/classes/components/weaveworks/orders-db.yml mongo\n./inventory/classes/components/weaveworks/orders.yml weaveworksdemos/orders:0.4.7\n./inventory/classes/components/weaveworks/shipping.yml weaveworksdemos/shipping:0.4.8\n./inventory/classes/components/weaveworks/queue-master.yml weaveworksdemos/queue-master:0.3.1\n./inventory/classes/components/weaveworks/rabbitmq.yml rabbitmq:3.6.8-management\n./inventory/classes/components/weaveworks/payment.yml weaveworksdemos/payment:0.4.3\n./inventory/classes/components/weaveworks/front-end.yml weaveworksdemos/front-end:0.3.12\n./inventory/classes/components/weaveworks/carts.yml weaveworksdemos/carts:0.4.8\n./inventory/classes/components/kapicorp/tesoro.yml kapicorp/tesoro\n
"},{"location":"pages/commands/kapitan_validate/","title":"CLI Reference | kapitan validate","text":""},{"location":"pages/commands/kapitan_validate/#kapitan-validate","title":"kapitan validate","text":"Validates the schema of compiled output. Validate options are specified in the inventory under parameters.kapitan.validate. Supported types are:
"},{"location":"pages/commands/kapitan_validate/#usage","title":"Usage","text":"standalonemanual with kapitan compileautomatic with .kapitan dotfile kapitan validate\n
click to expand output created schema-cache-path at ./schemas\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_secret.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_jsonnet.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_simple.yml\n
kapitan compile --validate\n
click to expand output Rendered inventory (0.27s)\nCompiled labels (0.23s)\nCompiled removal (0.00s)\nCompiled busybox (0.24s)\nCompiled minikube-nginx-jsonnet (0.49s)\nCompiled minikube-nginx-kadet (0.25s)\nCompiled minikube-mysql (0.59s)\nCompiled minikube-es (1.17s)\nCompiled all-glob (1.55s)\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_secret.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_jsonnet.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_simple.yml\n
You can leverage the .kapitan dotfile to make sure validate runs every time you run compile.
example .kapitan
...\n\ncompile:\n validate: true\n
The validate command will now be implied for every compile run
kapitan compile\n
click to expand output Rendered inventory (0.27s)\nCompiled labels (0.23s)\nCompiled removal (0.00s)\nCompiled busybox (0.24s)\nCompiled minikube-nginx-jsonnet (0.49s)\nCompiled minikube-nginx-kadet (0.25s)\nCompiled minikube-mysql (0.59s)\nCompiled minikube-es (1.17s)\nCompiled all-glob (1.55s)\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_secret.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_jsonnet.yml\nValidation: manifest validation successful for ./compiled/minikube-mysql/manifests/mysql_service_simple.yml\n
"},{"location":"pages/commands/kapitan_validate/#kubernetes-setup","title":"Kubernetes Setup","text":"Kubernetes has different resource kinds, for instance:
servicedeploymentstatefulset
Kapitan has built in support for validation of Kubernetes kinds, and automatically integrates with https://kubernetesjsonschema.dev. See github.com/instrumenta/kubernetes-json-schema for more informations.
Info
Kapitan will automatically download the schemas for Kubernetes Manifests directly from https://kubernetesjsonschema.dev
By default, the schemas are cached into ./schemas/, which can be modified with the --schemas-path option.
override permanently schema-path
Remember to use the .kapitan dotfile configuration to override permanently the schema-path location.
$ cat .kapitan\n# other options abbreviated for clarity\nvalidate:\n schemas-path: custom/schemas/cache/path\n
"},{"location":"pages/commands/kapitan_validate/#example","title":"Example","text":"Refer to the mysql example.
kubernetes/inventory/classes/component/mysql.yml validate: \n - type: kubernetes # mkdocs (1)! \n output_paths: # mkdocs (2)! \n - manifests/mysql_secret.yml\n kind: secret # temporarily replaced with 'deployment' during test # mkdocs (3)! \n version: 1.14.0 # optional, defaults to 1.14.0 # mkdocs (4)! \n - type: kubernetes\n output_paths:\n - manifests/mysql_service_jsonnet.yml\n - manifests/mysql_service_simple.yml\n kind: service\n version: 1.14.0\n
type | currently only Kubernetes is supportedoutput_paths | list of files to validatekind | a Kubernetes resource kindversion | a Kubernetes API version, defaults to 1.14.0
"},{"location":"pages/contribute/code/","title":"Kapitan code","text":"Many of our features come from contributions from external collaborators. Please help us improve Kapitan by extending it with your ideas, or help us squash bugs you discover.
It's simple, just send us a PR with your improvements!
","tags":["community"]},{"location":"pages/contribute/code/#submitting-code","title":"Submitting code","text":"We would like ask you to fork Kapitan project and create a Pull Request targeting master branch. All submissions, including submissions by project members, require review.
","tags":["community"]},{"location":"pages/contribute/code/#setup","title":"Setup","text":"We build kapitan using poetry.
-
Install poetry
pip install poetry\n
-
Install dependencies
poetry install --all-extras --with dev --with test --with docs\n
Poetry creates a virtual environment with the required dependencies installed.
-
Run kapitan with the own compiled code
poetry run kapitan <your command>\n
Because we are using a pinned version of reclass which is added as a submodule into Kapitan's repository, you need to pull it separately by executing the command below:
git submodule update --init\n
","tags":["community"]},{"location":"pages/contribute/code/#troubleshoot","title":"Troubleshoot","text":"Check if gcc is installed:
brew install gcc@5\n
","tags":["community"]},{"location":"pages/contribute/code/#testing","title":"Testing","text":"Run make test to run all tests. If you modify anything in the examples/ folder make sure you replicate the compiled result of that in tests/test_kubernetes_compiled. If you add new features, run make test_coverage && make test_formatting to make sure the test coverage remains at current or better levels and that code formatting is applied.
If you would like to evaluate your changes by running your version of Kapitan, you can do that by running bin/kapitan from this repository or even setting an alias to it.
python3 -m unittest tests/test_vault_transit.py\n
","tags":["community"]},{"location":"pages/contribute/code/#code-style","title":"Code Style","text":"To make sure you adhere to the Style Guide for Python (PEP8) Python Black is used to apply the formatting so make sure you have it installed with pip3 install black.
","tags":["community"]},{"location":"pages/contribute/code/#apply-via-git-hook","title":"Apply via Git hook","text":" - Run
pip3 install pre-commit to install precommit framework. - In the Kapitan root directory, run
pre-commit install - Git add/commit any changed files you want.
","tags":["community"]},{"location":"pages/contribute/code/#apply-manually","title":"Apply manually","text":"Run make format_codestyle before submitting.
","tags":["community"]},{"location":"pages/contribute/code/#release-process","title":"Release process","text":" - Create a branch named
release-v<NUMBER>. Use v0.*.*-rc.* if you want pre-release versions to be uploaded. - Update CHANGELOG.md with the release changes.
- Once reviewed and merged, Github Actions will auto-release.
- The merge has to happen with a merge commit not with squash/rebase so that the commit message still mentions
kapicorp/release-v* inside.
","tags":["community"]},{"location":"pages/contribute/code/#packaging-extra-resources-in-python-package","title":"Packaging extra resources in python package","text":"To package any extra resources/files in the pip package, make sure you modify both MANIFEST.in.
","tags":["community"]},{"location":"pages/contribute/code/#leave-a-comment","title":"Leave a comment","text":"","tags":["community"]},{"location":"pages/contribute/documentation/","title":"Documentation","text":"Our documentation usully prevents new users from adopting Kapitan. Help us improve by contributing with fixes and keeping it up-to-date.
","tags":["community"]},{"location":"pages/contribute/documentation/#articles","title":"Articles","text":"Write articles on Kapitan and share your way of working. Inspire others, and reach out to have your article published / endorsed by us.
","tags":["community"]},{"location":"pages/contribute/documentation/#this-website","title":"This Website","text":"Find something odd? Let us know or change it yourself: you can edit pages of this website on Github by clicking the pencil icon at the top right of this page!
","tags":["community"]},{"location":"pages/contribute/documentation/#update-documentation","title":"Update documentation","text":"We use mkdocs to generate our gh-pages from .md files under docs/ folder.
Updating our gh-pages is therefore a two-step process.
","tags":["community"]},{"location":"pages/contribute/documentation/#update-the-markdown","title":"Update the markdown","text":"Submit a PR for our master branch that updates the .md file(s). Test how the changes would look like when deployed to gh-pages by serving it on localhost:
- Edit the
strict property in mkdocs.yml and set it to false. make local_serve_documentation- Now the documentation site should be available at
localhost:8000.
","tags":["community"]},{"location":"pages/contribute/documentation/#submit-a-pr","title":"Submit a PR","text":"Once the above PR has been merged, our CI will deploy your docs automatically.
","tags":["community"]},{"location":"pages/contribute/sponsor/","title":"Sponsor Kapitan","text":"Do you want to help the project? Great! There are many ways to do it
We accept donations throught GitHubs Sponsors. Alternatively reach out for other ways to support us.
Companies and individuals sponsoring us on a regular base will be recognised and called out on our website
","tags":["community"]},{"location":"pages/contribute/talk/","title":"Talk about Kapitan","text":"Our project needs your support to get noticed! Please let everyone know that you are using Kapitan
- Help us grow: give us a star
- Join us on kubernetes.slack.com
#kapitan(Get invited) - Tweet about us on Twitter . Remember to add @kapitandev to your tweets
- Share our website
https://kapitan.dev - Write tutorials and blog posts and join the many who have done it already! Get published on the Kapitan Blog
- Share what Kapitan does for you and for your company
- Inspire your colleagues and network on LinkedIn
"},{"location":"pages/input_types/copy/","title":"Copy","text":"This input type simply copies the input templates to the output directory without any rendering/processing. For Copy, input_paths can be either a file or a directory: in case of a directory, all the templates in the directory will be copied and outputted to output_path.
Supported output types: N/A (no need to specify output_type)
Example
kapitan:\n compile:\n - input_type: copy\n ignore_missing: true # Do not error if path is missing. Defaults to False\n input_paths:\n - resources/state/${target_name}/.terraform.lock.hcl\n output_path: terraform/\n
"},{"location":"pages/input_types/external/","title":"External","text":"This input type executes an external script or binary. This can be used to manipulate already compiled files or execute binaries outside of kapitan that generate or manipulate files.
For example, ytt is a useful yaml templating tool. It is not built into the kapitan binary, however, with the external input type, we could specify the ytt binary to be executed with specific arguments and environment variables.
In this example, we're removing a label from a k8s manifests in a directory ingresses and placing it into the compiled target directory.
parameters:\n target_name: k8s-manifests\n kapitan:\n vars:\n target: ${target_name}\n compile:\n - input_type: external\n input_paths:\n - /usr/local/bin/ytt # path to ytt on system\n output_path: .\n args:\n - -f\n - ingresses/ # directory with ingresses\n - -f\n - ytt/remove.yaml # custom ytt script\n - \">\"\n - \\${compiled_target_dir}/ingresses/ingresses.yaml # final merged result\n
Supported output types: N/A (no need to specify output_type)
Additionally, the input type supports field env_vars, which can be used to set environment variables for the external command. By default, the external command doesn't inherit any environment variables from Kapitan's environment. However, if environment variables $PATH or $HOME aren't set in env_vars, they will be propagated from Kapitan's environment to the external command's environment.
Finally, Kapitan will substitute ${compiled_target_dir} in both the command's arguments and the environment variables. This variable needs to be escaped in the configuration to ensure that reclass won't interpret it as a reclass reference.
"},{"location":"pages/input_types/helm/","title":"Input Type | Helm","text":"This is a Python binding to helm template command for users with helm charts. This does not require the helm executable, and the templates are rendered without the Tiller server.
Unlike other input types, Helm input types support the following additional parameters under kapitan.compile:
parameters:\n kapitan:\n compile:\n - output_path: <output_path>\n input_type: helm\n input_paths:\n - <chart_path>\n helm_values:\n <object_with_values_to_override>\n helm_values_files:\n - <values_file_path>\n helm_path: <helm binary>\n helm_params:\n name: <chart_release_name>\n namespace: <substitutes_.Release.Namespace>\n output_file: <string>\n validate: true\n \u2026\n
helm_values is an object containing values specified that will override the default values in the input chart. This has exactly the same effect as specifying --values custom_values.yml for helm template command where custom_values.yml structure mirrors that of helm_values.
helm_values_files is an array containing the paths to helm values files used as input for the chart. This has exactly the same effect as specifying --file my_custom_values.yml for the helm template command where my_custom_values.yml is a helm values file. If the same keys exist in helm_values and in multiple specified helm_values_files, the last indexed file in the helm_values_files will take precedence followed by the preceding helm_values_files and at the bottom the helm_values defined in teh compile block. There is an example in the tests. The monitoring-dev(kapitan/tests/test_resources/inventory/targets/monitoring-dev.yml) and monitoring-prd(kapitan/tests/test_resources/inventory/targets/monitoring-prd.yml) targets both use the monitoring(tests/test_resources/inventory/classes/component/monitoring.yml) component. This component has helm chart input and takes a common.yml helm_values file which is \"shared\" by any target that uses the component and it also takes a dynamically defined file based on a kapitan variable defined in the target.
helm_path can be use to provide the helm binary name or path. helm_path defaults to the value of KAPITAN_HELM_PATH env var if it is set, else it defaults to helm
helm_params correspond to the flags for helm template. Most flags that helm supports can be used here by replacing '-' by '_' in the flag name.
Flags without argument must have a boolean value, all other flags require a string value.
Special flags:
name: equivalent of helm template [NAME] parameter. Ignored if name_template is also specified. If neither name_template nor name are specified, the --generate-name flag is used to generate a name.-
output_file: name of the single file used to output all the generated resources. This is equivalent to call helm template without specifing output dir. If not specified, each resource is generated into a distinct file.
-
include_crds and skip_tests: These flags are enabled by default and should be set to false to be removed.
debug: prints the helm debug output in kapitan debug log.-
namespace: note that due to the restriction on helm template command, specifying the namespace does not automatically add metadata.namespace property to the resources. Therefore, users are encouraged to explicitly specify it in all resources:
metadata:\n namespace: {{ .Release.Namespace }} # or any other custom values\n
See the helm doc for further detail.
"},{"location":"pages/input_types/helm/#example","title":"Example","text":"Let's use nginx-ingress helm chart as the input. Using kapitan dependency manager, this chart can be fetched via a URL as listed in https://helm.nginx.com/stable/index.yaml.
On a side note, https://helm.nginx.com/stable/ is the chart repository URL which you would helm repo add, and this repository should contain index.yaml that lists out all the available charts and their URLs. By locating this index.yaml file, you can locate all the charts available in the repository.
We can use version 0.3.3 found at https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz. We can create a simple target file as inventory/targets/nginx-from-chart.yml whose content is as follows:
parameters:\n kapitan:\n vars:\n target: nginx-from-chart\n dependencies:\n - type: https\n source: https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz\n unpack: True\n output_path: components/charts\n compile:\n - output_path: .\n input_type: helm\n input_paths:\n - components/charts/nginx-ingress\n helm_values:\n controller:\n name: my-controller\n image:\n repository: custom_repo\n helm_params:\n name: my-first-release-name\n namespace: my-first-namespace\n
To compile this target, run:
$ kapitan compile --fetch\nDependency https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz : fetching now\nDependency https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz : successfully fetched\nDependency https://helm.nginx.com/stable/nginx-ingress-0.3.3.tgz : extracted to components/charts\nCompiled nginx-from-chart (0.07s)\n
The chart is fetched before compile, which creates components/charts/nginx-ingress folder that is used as the input_paths for the helm input type. To confirm if the helm_values actually has overridden the default values, we can try:
$ grep \"my-controller\" compiled/nginx-from-chart/nginx-ingress/templates/controller-deployment.yaml\n name: my-controller\n app: my-controller\n app: my-controller\n
"},{"location":"pages/input_types/helm/#building-the-binding-from-source","title":"Building the binding from source","text":"Run
cd kapitan/inputs/helm\n./build.sh\n
This requires Go 1.14.
"},{"location":"pages/input_types/helm/#helm-subcharts","title":"Helm subcharts","text":"There is an external dependency manager of type helm which enables you to specify helm charts to download, including subcharts.
"},{"location":"pages/input_types/introduction/","title":"Introduction","text":"Note: make sure to read up on inventory before moving on.
"},{"location":"pages/input_types/introduction/#phases-of-the-compile-command","title":"Phases of the compile command","text":"Now that we have a basic understanding of Kapitan inventory, we can talk about the kapitan compile command.
The command has five distinct phases:
graph LR\n classDef pink fill:#f9f,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n classDef blue fill:#00FFFF,stroke:#333,stroke-width:4px,color:#000,font-weight: bold;\n INVENTORY[\"Inventory\"]:::pink\n\n COMPILE[\"Compile\"]:::pink\n FINISH[\"Finish\"]:::pink\n COPY[\"Copy\"]:::pink\n\n\n subgraph \"fetch\"\n F{\"fetch?\"}\n FETCH[\"fetch dependencies\"]\n end\n\n subgraph \"validate\"\n V{\"validate?\"}\n VALIDATE[\"Validate\"]\n end\n\n subgraph \"reveal\"\n REVEAL[\"Reveal\"]\n R{\"reveal?\"}\n end\n\n INVENTORY --> F\n F --> |yes| FETCH\n FETCH --> COMPILE\n F ==> |no| COMPILE\n COMPILE ==> R\n R ==> |no| COPY\n R --> |yes| REVEAL\n REVEAL --> COPY\n COPY --> V\n V --> |yes| VALIDATE\n V ==> |no| FINISH\n VALIDATE --> FINISH\n\n
Step Flag Description Configuration Inventory Kapitan uses reclass to render a final version of the inventory. Fetch --fetch Kapitan fetches external dependencies parameters.kapitan.dependencies Compile Kapitan compiles the input types for each target parameters.kapitan.compile Reveal --reveal Kapitan reveals the secrets directly in the compiled output parameters.kapitan.secrets Copy Kapitan moves the output files from the tmp directory to /compiled Validate --validate Kapitan validates the schema of compiled output. parameters.kapitan.validate Finish Kapitan has completed all tasks"},{"location":"pages/input_types/introduction/#supported-input-types","title":"Supported input types","text":"Input types can be specified in the inventory under kapitan.compile in the following format:
jinja2jsonnetkadethelmcopy parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: jinja2\n input_params: # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n
- a dict passed to the template
Please see Jinja
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: jsonnet\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: [`yaml` | `json`]\n
- (Default: global --prune)
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: kadet\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: [`yaml` | `json`]\n
- (Default: global --prune)
Please see Kadet
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: helm\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: <output_type_specific_to_input_type>\n
- (Default: global --prune)
parameters:\n kapitan:\n compile:\n - output_path: <output_path_in_target_dir>\n input_type: copy\n prune: false # (1)!\n input_paths:\n - directory/\n - file\n - globbed/path/*\n output_type: <output_type_specific_to_input_type>\n
- (Default: global --prune)
"},{"location":"pages/input_types/jinja/","title":"Input Type | Jinja2","text":"This input type is probably the most simple input type to use: it is very versatile and is commonly used to create scripts and documentation files.
It renders jinja2 templates.
"},{"location":"pages/input_types/jinja/#example-configuration","title":"Example configuration","text":"Here's some configuration from the nginx example
examples/kubernetes/inventory/classes/component/nginx-common.yml
templates: #(1)!\n - docs/nginx/README.md\n - components/nginx-deploy.sh\n\n kapitan:\n compile:\n - output_path: . #(2)!\n input_type: jinja2\n input_paths: ${templates} #(3)!\n
- We define a list with all the templates we want to compile with this input type
- Then input type will render the files a the root of the target compiled folder e.g.
compiled/${target_name} - We pass the list as
input_paths
Notice how make use of variable interpolation to use the convenience of a list to add all the files we want to compile. You can now simply add to that list from any other place in the inventory that calls that class.
input_paths can either be a file, or a directory: in case of a directory, all the templates in the directory will be rendered.input_params (optional) can be used to pass extra parameters, helpful when needing to use a similar template for multiple components in the same target.
"},{"location":"pages/input_types/jinja/#documentation","title":"Documentation","text":"We usually store documentation templates under the templates/docs directory.
examples/kubernetes/docs/nginx/README.md
{% set i = inventory.parameters %}\n\n# Welcome to the README!\n\nTarget *{{ i.target_name }}* is running:\n\n* {{ i.nginx.replicas }} replicas of *nginx* running nginx image {{ i.nginx.image }}\n* on cluster {{ i.cluster.name }}\n
Compiled result
# Welcome to the README!\n\nTarget *minikube-nginx-jsonnet* is running:\n\n* 1 replicas of *nginx* running nginx image nginx:1:15.8\n* on cluster minikube\n
"},{"location":"pages/input_types/jinja/#scripts","title":"Scripts","text":"When we use Jinja to render scripts, we tend to call them \"canned scripts\" to indicate that these scripts have everything needed to run without extra parameters.
We usually store script templates under the templates/scripts directory.
examples/kubernetes/components/nginx-deploy.sh
#!/bin/bash -e\nDIR=$(dirname ${BASH_SOURCE[0]})\n{% set i = inventory.parameters %} #(1)!\n\nKUBECTL=\"kubectl -n {{i.namespace}}\" #(2)!\n\n# Create namespace before anything else\n${KUBECTL} apply -f ${DIR}/pre-deploy/namespace.yml\n\nfor SECTION in manifests\ndo\n echo \"## run kubectl apply for ${SECTION}\"\n ${KUBECTL} apply -f ${DIR}/${SECTION}/ | column -t\ndone\n
- We import the
inventory as a Jinja variable - We use to set the
namespace explicitly
Compiled result
#!/bin/bash -e\nDIR=$(dirname ${BASH_SOURCE[0]})\n #(1)!\n\nKUBECTL=\"kubectl -n minikube-nginx-jsonnet\" #(2)!\n\n# Create namespace before anything else\n${KUBECTL} apply -f ${DIR}/pre-deploy/namespace.yml\n\nfor SECTION in manifests\ndo\n echo \"## run kubectl apply for ${SECTION}\"\n ${KUBECTL} apply -f ${DIR}/${SECTION}/ | column -t\ndone\n
- The script is now a \"canned script\" and ready to be used for this specif target.
- You can see that the namespace has been replaced with the target's one.
"},{"location":"pages/input_types/jinja/#accessing-the-inventory","title":"Accessing the inventory","text":"Templates will be provided at runtime with 3 variables:
inventory: To access the inventory for that specific target.inventory_global: To access the inventory of all targets.input_params: To access the optional dictionary provided to the input type.
Use of inventory_global
inventory_global can be used to generate a \"global\" README.md that contains a link to all generated targets.
| *Target* |\n|------------------------------------------------------------------------|\n{% for target in inventory_global | sort() %}\n{% set p = inventory_global[target].parameters %}\n|[{{target}}](../{{target}}/docs/README.md) |\n{% endfor %}\n
Compiled result
| *Target* |\n|------------------------------------------------------------------------|\n| [argocd](../argocd/docs/README.md) |\n| [dev-sockshop](../dev-sockshop/docs/README.md) |\n| [echo-server](../echo-server/docs/README.md) |\n| [examples](../examples/docs/README.md) |\n| [gke-pvm-killer](../gke-pvm-killer/docs/README.md) |\n| [global](../global/docs/README.md) |\n| [guestbook-argocd](../guestbook-argocd/docs/README.md) |\n| [kapicorp-demo-march](../kapicorp-demo-march/docs/README.md) |\n| [kapicorp-project-123](../kapicorp-project-123/docs/README.md) |\n| [kapicorp-terraform-admin](../kapicorp-terraform-admin/docs/README.md) |\n| [mysql](../mysql/docs/README.md) |\n| [postgres-proxy](../postgres-proxy/docs/README.md) |\n| [pritunl](../pritunl/docs/README.md) |\n| [prod-sockshop](../prod-sockshop/docs/README.md) |\n| [sock-shop](../sock-shop/docs/README.md) |\n| [tesoro](../tesoro/docs/README.md) |\n| [tutorial](../tutorial/docs/README.md) |\n| [vault](../vault/docs/README.md) |\n
"},{"location":"pages/input_types/jinja/#jinja2-custom-filters","title":"Jinja2 custom filters","text":"We support the following custom filters for use in Jinja2 templates:
EncodingTimeRegexpfileglobboolternaryshufflereveal_maybe sha256yamltomlb64encodeb64decode SHA256 hashing of text
{{ text | sha256 }}
Dump text as YAML
{{ text | yaml }}
Dump text as TOML
{{ text | toml }}
base64 encode text
{{ text | b64encode }}
base64 decode text
{{ text | b64decode }}
to_datetimestrftime return datetime object for string
{{ \"2019-03-07 13:37:00\" | to_datetime }}
return current date string for format
{{ \"%a, %d %b %Y %H:%M\" | strftime }}
regex_replaceregex_escaperegex_searchregex_findall perform a re.sub returning a string
{{ hello world | regex_replace(pattern=\"world\", replacement=\"kapitan\")}}
escape all regular expressions special characters from string
{{ \"+s[a-z].*\" | regex_escape}}
perform re.search and return the list of matches or a backref
{{ hello world | regex_search(\"world.*\")}}
perform re.findall and return the list of matches as array
{{ hello world | regex_findall(\"world.*\")}}
return list of matched regular files for glob
{{ ./path/file* | fileglob }}
return the bool for value
{{ yes | bool }}
value ? true_val : false_val
{{ condition | ternary(\"yes\", \"no\")}}
randomly shuffle elements of a list
{{ [1, 2, 3, 4, 5] | shuffle }}
reveal ref/secret tag only if compile --reveal flag is set
{{ \"?{base64:my_ref}\" | reveal_maybe}}
Tip
You can also provide path to your custom filter modules in CLI. By default, you can put your filters in lib/jinja2_filters.py and they will automatically get loaded.
"},{"location":"pages/input_types/jsonnet/","title":"Input Type | Jsonnet","text":"Jsonnet is a superset of json format that includes features such as conditionals, variables and imports. Refer to jsonnet docs to understand how it works.
Note: unlike jinja2 templates, one jsonnet template can output multiple files (one per object declared in the file).
"},{"location":"pages/input_types/jsonnet/#accessing-the-inventory","title":"Accessing the inventory","text":"Typical jsonnet files would start as follows:
local kap = import \"lib/kapitan.libjsonnet\"; #(1)!\nlocal inv = kap.inventory(); #(2)!\nlocal p = inv.parameters; #(3)!\n\n{\n \"data_java_opts\": p.elasticsearch.roles.data.java_opts, #(4)!\n}\n
- Import the Kapitan inventory library.
- Assign the content of the full inventory for this specific target to the
inv variable. - Assign the content of the
inventory.parameters to a variable p for convenience. - Use the
p variable fo access a specific intentory value
Note: The dictionary keys of the jsonnet object are used as filenames for the generated output files. If your jsonnet is not a dictionary, but is a valid json(net) object, then the output filename will be the same as the input filename. E.g. 'my_string' is inside templates/input_file.jsonnet so the generated output file will be named input_file.json for example and will contain \"my_string\".
"},{"location":"pages/input_types/jsonnet/#jinja2-templating","title":"Jinja2 templating","text":"Kapitan allows you to compile a Jinja template from within Jsonnet:
local kap = import \"lib/kapitan.libjsonnet\";\n\n{\n \"jon_snow\": kap.jinja2_template(\"templates/got.j2\", { is_dead: false }),\n}\n
"},{"location":"pages/input_types/jsonnet/#callback-functions","title":"Callback functions","text":"In addition, importing kapitan.libjsonnet makes available the following native_callback functions gluing reclass to jsonnet (amongst others):
inventoryjinja2_templateyamlfile I/Osha256_stringgzip_b64jsonschema returns a dictionary with the inventory for target
renders the jinja2 file with context specified
yaml_loadyaml_load_streamyaml_dumpyaml_dump_stream returns a json string of the specified yaml file
returns a list of json strings of the specified yaml file
returns a string yaml from a json string
returns a string yaml stream from a json string
file_readfile_existsdir_files_listdir_files_read reads the file specified
returns informative object if a file exists
returns a list of file in a dir
returns an object with keys - file_name and values - file contents
returns sha256 of string
returns base64 encoded gzip of obj
validates obj with schema, returns object with 'valid' and 'reason' keys
"},{"location":"pages/input_types/jsonnet/#jsonschema-validation","title":"Jsonschema validation","text":"Given the follow example inventory:
mysql:\n storage: 10G\n storage_class: standard\n image: mysql:latest\n
The yaml inventory structure can be validated with the new jsonschema() function:
local schema = {\n type: \"object\",\n properties: {\n storage: { type: \"string\", pattern: \"^[0-9]+[MGT]{1}$\"},\n image: { type: \"string\" },\n }\n};\n// run jsonschema validation\nlocal validation = kap.jsonschema(inv.parameters.mysql, schema);\n// assert valid, otherwise error with validation.reason\nassert validation.valid: validation.reason;\n
If validation.valid is not true, it will then fail compilation and display validation.reason.
Fails validation because storage has an invalid pattern (10Z)
Jsonnet error: failed to compile /code/components/mysql/main.jsonnet:\nRUNTIME ERROR: '10Z' does not match '^[0-9]+[MGT]{1}$'\n\nFailed validating 'pattern' in schema['properties']['storage']:\n {'pattern': '^[0-9]+[MGT]{1}$', 'type': 'string'}\n\nOn instance['storage']:\n '10Z'\n\n/code/mysql/main.jsonnet:(19:1)-(43:2)\n\nCompile error: failed to compile target: minikube-mysql\n
"},{"location":"pages/input_types/kadet/","title":"Input Type | Kadet","text":"Kadet is an extensible input type for Kapitan that enables you to generate templates using Python.
The key benefit being the ability to utilize familiar programing principles while having access to Kapitan's powerful inventory system.
A library that defines resources as classes using the Base Object class is required. These can then be utilized within components to render output.
The following functions are provided by the class BaseObj().
Method definitions:
new(): Provides parameter checking capabilitiesbody(): Enables in-depth parameter configuration
Method functions:
root(): Defines values that will be compiled into the outputneed(): Ability to check & define input parametersupdate_root(): Updates the template file associated with the class
A class can be a resource such as a Kubernetes Deployment as shown here:
class Deployment(BaseObj): # (1)!\n def new(self): # (2)!\n self.need(\"name\", \"name string needed\")\n self.need(\"labels\", \"labels dict needed\")\n self.need(\"containers\", \"containers dict needed\")\n self.update_root(\"lib/kubelib/deployment.yml\")\n\n def body(self): # (3)!\n self.root.metadata.name = self.kwargs.name # (4)!\n self.root.metadata.namespace = inv.parameters.target_name\n self.root.spec.template.metadata.labels = self.kwargs.labels\n self.root.spec.template.spec.containers = self.kwargs.containers\n
- The deployment is an
BaseObj() which has two main functions. new(self) is used to perform parameter validation & template compilationbody(self) is utilized to set those parameters to be rendered.self.root.metadata.name is a direct reference to a key in the corresponding yaml.
Kadet supports importing libraries as you would normally do with Python. These libraries can then be used by the components to generate the required output.
...\nkubelib = kadet.load_from_search_paths(\"kubelib\") #(1)!\n...\nname = \"nginx\"\nlabels = kadet.BaseObj.from_dict({\"app\": name})\nnginx_container = kubelib.Container( #(2)!\n name=name, image=inv.parameters.nginx.image, ports=[{\"containerPort\": 80}]\n)\n...\ndef main():\n output = kadet.BaseObj() #(3)!\n output.root.nginx_deployment = kubelib.Deployment(name=name, labels=labels, containers=[nginx_container]) #(4)!\n output.root.nginx_service = kubelib.Service( #(5)!\n name=name, labels=labels, ports=[svc_port], selector=svc_selector\n )\n return output #(6)!\n
- We import a library called
kubelib using load_from_search_paths() - We use
kubelib to create a Container - We create an output of type
BaseObj and we will be updating the root element of this output. - We use
kubelib to create a Deployment kind. The Deployment makes use of the Container created. - We use
kubelib to create a Service kind. - We return the object. Kapitan will render everything under
output.root
Kadet uses a library called addict to organise the parameters inline with the yaml templates. As shown above we create a BaseObject() named output. We update the root of this output with the data structure returned from kubelib. This output is what is then returned to kapitan to be compiled into the desired output type.
For a deeper understanding please refer to github.com/kapicorp/kadet
Supported output types:
"},{"location":"pages/input_types/remove/","title":"Remove","text":"This input type simply removes files or directories. This can be helpful if you can't control particular files generated during other compile inputs.
For example, to remove a file named copy_target, specify an entry to input_paths, compiled/${kapitan:vars:target}/copy_target.
parameters:\n target_name: removal\n kapitan:\n vars:\n target: ${target_name}\n compile:\n - input_type: copy\n input_paths:\n - copy_target\n output_path: .\n # test removal of a file\n - input_type: remove\n input_paths:\n - compiled/${kapitan:vars:target}/copy_target\n output_path: .\n
As a reminder, each input block within the compile array is run sequentially for a target in Kapitan. If we reversed the order of the inputs above like so:
parameters:\n target_name: removal\n kapitan:\n vars:\n target: ${target_name}\n compile:\n - input_type: remove\n input_paths:\n - compiled/${kapitan:vars:target}/copy_target\n output_path: .\n - input_type: copy\n input_paths:\n - copy_target\n output_path: .\n
The first input block would throw an error because the copy input command hasn't run yet to produce the file being removed by the remove input block.
Supported output types: N/A (no need to specify output_type)
"},{"location":"pages/inventory/advanced/","title":"Advanced Inventory Features","text":""},{"location":"pages/inventory/advanced/#target-labels","title":"Target labels","text":"Kapitan allows you to define labels in your inventory, which can then be used to group together targets with similar labels.
For instance you could define the following:
Defines a class to add the customer label to selected targets
inventory/classes/type/customer_project.yml
parameters:\n customer_name: ${target_name} # Defaults to the target_name\n kapitan:\n labels:\n customer: ${customer_name}\n
Apply the class to the target for customer acme
inventory/targets/customers/acme.yml
classes:\n...\n- type.customer_project\n\nparameters:\n...\n
You can now selectively compile targets for customer acme using the following (see see Labels for more details )
kapitan compile -l customer=acme\nCompiled acme (0.06s)\nCompiled acme-documentation (0.09s)\n
"},{"location":"pages/inventory/backends/","title":"Alternative Inventory Backends","text":"We provide pluggable backend alternatives for the Inventory:
reclass(default): The original kapitan inventory (see reclass)reclass-rs: The Rust drop in replacement (see reclass-rs)omegaconf: An alternative inventory solution based on Omegaconf
You can switch the backend to by:
- (recommended) Define the backend in the .kapitan config file.
- Passing the
--inventory-backend=backend command line option when calling kapitan.
"},{"location":"pages/inventory/classes/","title":"Classes","text":""},{"location":"pages/inventory/classes/#usage","title":"Usage","text":"The next thing you want to learn about the inventory are classes. A class is a yaml file containing a fragment of yaml that we want to import and merge into the inventory.
Classes are fragments of yaml: feature sets, commonalities between targets. Classes let you compose your Inventory from smaller bits, eliminating duplication and exposing all important parameters from a single, logically organised place. As the Inventory lets you reference other parameters in the hierarchy, classes become places where you can define something that will then get referenced from another section of the inventory, allowing for composition.
Classes are organised under the inventory/classes directory substructure. They are organised hierarchically in subfolders, and the way they can be imported into a target or other classes depends on their location relative to the inventory/classes directory.
"},{"location":"pages/inventory/classes/#importing-classes","title":"Importing classes","text":"To import a class from within another file of the Inventory, you can follow these instructions:
- take the file path relative to the
inventory/classes/ directory - remove the
.yml file extension - replace
/ with .
For example, this will import the class inventory/classes/applications/sock-shop.yaml
classes:\n- applications.sock-shop\n
"},{"location":"pages/inventory/classes/#definition","title":"Definition","text":"Let's take a look at the common class which appears in the example above:
As explained, because the common.yaml is directly under the inventory/classes subdirectory, it can be imported directly into a target with:
classes:\n- common\n
If we open the file, we find another familiar yaml fragment.
inventory/classes/common.yml
classes:\n- kapitan.common\n\nparameters:\n namespace: ${target_name}\n target_name: ${_reclass_:name:short}\n
Notice that this class includes an import definition for another class, kapitan.common. We've already learned this means that kapitan will import a file on disk called inventory/classes/kapitan/common.yml
You can also see that in the parameters section we now encounter a new syntax which unlocks another powerful inventory feature: parameters interpolation!
"},{"location":"pages/inventory/introduction/","title":"What is the inventory?","text":"The Inventory is a core component of Kapitan: this section aims to explain how it works and how to best take advantage of it.
The Inventory is a hierarchical YAML based structure which you use to capture anything that you want to make available to Kapitan, so that it can be passed on to its templating engines.
inventory/\n\u251c\u2500\u2500 classes/\n\u2502 \u251c\u2500\u2500 common/\n\u2502 \u2502 \u2514\u2500\u2500 base.yml # Foundational configurations\n\u2502 \u251c\u2500\u2500 components/\n\u2502 \u2502 \u2514\u2500\u2500 web/ # Component-specific settings\n\u2502 \u2514\u2500\u2500 environments/\n\u2502 \u2514\u2500\u2500 production.yml # Environment-specific configurations\n\u2514\u2500\u2500 targets/\n \u2514\u2500\u2500 production/\n \u2514\u2500\u2500 web.yml # Specific deployment configurations\n
The Kapitan inventory is divided between targets and classes.
Both classes and targets are yaml file with the same structure
classes:\n - common\n - my.other.class\n\nparameters:\n key: value\n key2:\n subkey: value\n
Classes are found by default under the inventory/classes directory and define common settings and data that you define once and can be included in other files. This promotes consistency and reduces duplication.
Classes are identified with a name that maps to the directory structure they are nested under. In this example, the kapicorp.common class represented by the file classes/kapicorp/common.yml
# classes/kapicorp/common.yml\nclasses:\n - common\n\nparameters:\n namespace: ${target_name}\n base_docker_repository: quay.io/kapicorp\n base_domain: kapicorp.com\n
Targets are found by default under the inventory/targets directory and represent the different environments or components you want to manage. Each target is a YAML file that defines a set of configurations.
For example, you might have targets for production, staging, and development environments.
# targets/production.yml\nclasses:\n - kapicorp.common\n - components.web\n - environments.production\n\nparameters:\n target_name: web\n
By combining target and classes, the Inventory becomes the SSOT for your whole configuration, and learning how to use it will unleash the real power of Kapitan.
"},{"location":"pages/inventory/parameters_interpolation/","title":"Parameters Interpolation","text":"Note
as a shorthand, when we encounter deep yaml structures like the following:
parameters:\n components:\n nginx:\n image: nginx:latest\n
Usually when we want to talk about the image subkey, we normally use either of the following:
parameters.components.nginx.imagecomponents.nginx.image
However, when used in parameter expansion, remember to:
- replace the
. with : - omit the
parameters initial key which is implied - wrap it into the
${} variable interpolation syntax
The correct way to reference parameters.nginx.image then becomes ${components:nginx:image}.
The Inventory allows you to refer to other values defined elsewhere in the structure, using parameter interpolation.
Given the example:
parameters:\n cluster:\n location: europe\n\n application:\n location: ${cluster:location}\n\n namespace: ${target_name}\n target_name: dev\n
Here we tell Kapitan that:
namespace should take the same value defined in target_nametarget_name should take the literal string devapplication.location should take the same value as defined in cluster.location
It is important to notice that the inventory can refer to values defined in other classes, as long as they are imported by the target. So for instance with the following example
classes:\n - project.production\n\n parameters:\n application:\n location: ${cluster.location}\n
Here in this case application.location refers to a value location which has been defined elsewhere, perhaps (but not necessarily) in the project.production class.
Also notice that the class name (project.production) is not in any ways influencing the name or the structed of the yaml it imports into the file
"},{"location":"pages/inventory/reclass-rs/","title":"The reclass-rs inventory backend","text":""},{"location":"pages/inventory/reclass-rs/#overview","title":"Overview","text":"Reclass-rs is a reimplementation of Kapitan's Reclass fork in Rust. Please note that the Rust implementation doesn't support all the features of Kapitan's Reclass fork yet.
However, reclass-rs improves rendering time for the inventory significantly, especially if you're making heavy use of parameter references in class includes. If some of the Reclass features or options that you're using are missing in reclass-rs, don't hesitate to open an issue in the reclass-rs project.
"},{"location":"pages/inventory/reclass-rs/#installation","title":"Installation","text":"The reclass-rs Python package is an optional dependency of Kapitan. You can install it as follows:
pip install kapitan[reclass-rs]\n
"},{"location":"pages/inventory/reclass-rs/#usage","title":"Usage","text":"To use the reclass-rs inventory backend, you need to pass --inventory-backend=reclass-rs on the command line. If you want to permanently switch to the reclass-rs inventory backend, you can select the inventory backend in the .kapitan config file:
global:\n inventory-backend: reclass-rs\n
"},{"location":"pages/inventory/reclass-rs/#performance-comparison","title":"Performance comparison","text":"For the performance comparison, a real Kapitan inventory which makes heavy use of parameter interpolation in class includes was rendered with both Reclass and reclass-rs. The example inventory that was used for the performance comparison contains 325 classes and 56 targets. The example inventory renders to a total of 25MB of YAML.
"},{"location":"pages/inventory/reclass-rs/#reclass","title":"Reclass","text":"$ time kapitan inventory -v --inventory-backend=reclass > inv.yml\n[ ... some output omitted ... ]\nkapitan.resources DEBUG Using reclass as inventory backend\nkapitan.inventory.backends.reclass DEBUG Inventory reclass: No config file found. Using reclass inventory config defaults\nkapitan.inventory.backends.reclass DEBUG Inventory rendering with reclass took 0:01:06.037057\n\nreal 1m23.840s\nuser 1m23.520s\nsys 0m0.287s\n
Reclass takes 1 minute and 6 seconds to render the example inventory. The rest of the runtime (roughly 18 seconds) is spent in writing the resulting 25MB of YAML to the output file.
"},{"location":"pages/inventory/reclass-rs/#reclass-rs","title":"reclass-rs","text":"$ time kapitan inventory -v --inventory-backend=reclass-rs > inv-rs.yml\n[ ... some output omitted ... ]\nkapitan.resources DEBUG Using reclass-rs as inventory backend\nkapitan.inventory.backends.reclass DEBUG Inventory reclass: No config file found. Using reclass inventory config defaults\nreclass-config.yml entry 'storage_type=yaml_fs' not implemented yet, ignoring...\nreclass-config.yml entry 'inventory_base_uri=./inventory' not implemented yet, ignoring...\nreclass-config.yml entry 'allow_none_override=true' not implemented yet, ignoring...\nkapitan.inventory.backends.reclass_rs DEBUG Inventory rendering with reclass-rs took 0:00:01.717107\n\nreal 0m19.921s\nuser 0m35.586s\nsys 0m1.066s\n
reclass-rs takes 1.7 seconds to render the example inventory. The rest of the runtime (roughly 18 seconds) is spent in writing the resulting 25MB of YAML to the output file.
"},{"location":"pages/inventory/targets/","title":"Targets","text":"In the world of Kapitan, a target represents a specific environment or deployment scenario where you want to apply your configurations.
Think of it as a blueprint that defines all the necessary settings and parameters for a particular deployment.
For instance, you might have separate targets for production, staging, and development environments, each with its own unique configurations.
"},{"location":"pages/inventory/targets/#defining-targets","title":"Defining targets","text":"Targets are defined as YAML files within the inventory/targets/ directory of your Kapitan project. Each target file typically includes:
classes:\n # A list of classes to inherit configurations from.\n # This allows you to reuse common settings and avoid repetition\n -\n\nparameters:\n # file parameters that override or extend the parameters inherited from previously loaded classes\n
Example:
# inventory/targets/production/web.yml\nclasses:\n - common\n - components.nginx\n\nparameters:\n environment: production\n description: ${environment} environment\n nginx:\n replicas: 3\n namespace: ${environment}\n
In this example, the production.web target:
- Inherits configurations from the common and components.nginx classes.
- Sets the
environment parameter to production. - Overrides (if defined) the
replicas for the nginx component to 3. - Defines the namespace as
production using variable interpolation. - Creates a dynamic description based on the content of the environment variable.
"},{"location":"pages/inventory/targets/#compiling-targets","title":"Compiling targets","text":"When you run kapitan compile -t , Kapitan:
- Reads the target file: Kapitan parses the YAML file for the specified target.
- Merges configurations: It merges the parameters from the included classes with the target-specific parameters, giving priority to the target's values.
- Generates output in
compiled/target/path/targetname: It uses this merged configuration data, along with the input types and generators, to create the final configuration files for the target environment.
When you run kapitan without the selector, it will run compile for all targets it discovers under the inventory/targets subdirectory.
"},{"location":"pages/inventory/targets/#target-directory-structure","title":"Target directory structure","text":"Targets are not limited to living directly within the inventory/targets directory.
They can be organized into subdirectories to create a more structured and hierarchical inventory. This is particularly useful for managing large and complex projects.
When targets are organized in subdirectories, Kapitan uses the full path from the targets/ directory to create the target name. This name is then used to identify the target during compilation and in the generated output.
Example for target clusters.production.my-cluster
inventory/\n\u2514\u2500\u2500 targets/\n \u2514\u2500\u2500 clusters/\n \u2514\u2500\u2500 production/\n \u2514\u2500\u2500 my-cluster.yml\n
In this example, the my-cluster.yml target file is located within the clusters/production/ subdirectory, and can be identified with clusters.production.my-cluster.
"},{"location":"tags/","title":"Tags","text":""},{"location":"tags/#community","title":"community","text":" - Proposals
- Kapitan Code
- Documentation
- Sponsor Us
"},{"location":"tags/#kadet","title":"kadet","text":""},{"location":"tags/#kubernetes","title":"kubernetes","text":""}]}
\ No newline at end of file
diff --git a/dev/sitemap.xml b/dev/sitemap.xml
index f0d593cc0..528f1ac24 100644
--- a/dev/sitemap.xml
+++ b/dev/sitemap.xml
@@ -2,222 +2,226 @@
https://kapitan.dev/dev/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/ADOPTERS/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/FAQ/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/getting_started/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/proposals/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/references/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/related/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/support/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_0_kadet/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_10_azure_key_vault/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_11_hashicorp_vault_transit/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_1_external_dependencies/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_2_helm_charts_input_type/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_3_schema_validation/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_4_standalone_executable/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_5_ref_types_redesign/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_6_hashicorp_vault/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_7_remote_inventory/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_8_google_secret_management/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_8_modularize_kapitan/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/kap_proposals/kap_9_bring_your_own_helm/
- 2024-11-08
+ 2024-12-11
- https://kapitan.dev/dev/pages/external_dependencies/
- 2024-11-08
+ https://kapitan.dev/dev/pages/core_concepts/
+ 2024-12-11
- https://kapitan.dev/dev/pages/kapitan_overview/
- 2024-11-08
+ https://kapitan.dev/dev/pages/external_dependencies/
+ 2024-12-11
https://kapitan.dev/dev/pages/remote_repositories/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/blog/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/blog/04/12/2022/kapitan-logo-5-years-of-kapitan/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/blog/04/12/2022/kapitan-logo-new-kapitan-release--v0310/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/blog/04/12/2022/kapitan-logo-new-kapitan-release--v0320/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/blog/27/08/2023/kapitan-logo-deploying-keda-with-kapitan/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/blog/12/02/2024/kapitan-logo-new-kapitan-release--v0331/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/commands/kapitan_compile/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/commands/kapitan_dotfile/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/commands/kapitan_inventory/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/commands/kapitan_lint/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/commands/kapitan_searchvar/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/commands/kapitan_validate/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/contribute/code/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/contribute/documentation/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/contribute/sponsor/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/contribute/talk/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/copy/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/external/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/helm/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/introduction/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/jinja/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/jsonnet/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/kadet/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/input_types/remove/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/inventory/advanced/
- 2024-11-08
+ 2024-12-11
+
+
+ https://kapitan.dev/dev/pages/inventory/backends/
+ 2024-12-11
https://kapitan.dev/dev/pages/inventory/classes/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/inventory/introduction/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/inventory/parameters_interpolation/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/inventory/reclass-rs/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/pages/inventory/targets/
- 2024-11-08
+ 2024-12-11
https://kapitan.dev/dev/tags/
- 2024-11-08
+ 2024-12-11
\ No newline at end of file
diff --git a/dev/sitemap.xml.gz b/dev/sitemap.xml.gz
index 01e197368..e7810050a 100644
Binary files a/dev/sitemap.xml.gz and b/dev/sitemap.xml.gz differ
diff --git a/dev/support/index.html b/dev/support/index.html
index b8fc66fb5..d2c40f300 100644
--- a/dev/support/index.html
+++ b/dev/support/index.html
@@ -18,7 +18,7 @@
-
+
@@ -26,7 +26,7 @@
-
+
@@ -213,7 +213,7 @@