diff --git a/proposals/22-03-value-in-the-structured-data.md b/proposals/22-03-value-in-the-structured-data.md new file mode 100644 index 0000000000..eed7fe6ff6 --- /dev/null +++ b/proposals/22-03-value-in-the-structured-data.md @@ -0,0 +1,556 @@ + + +# Replacements and Patch value in the structured data. + +**Authors**: +- koba1t + +**Reviewers**: +- natasha41575 +- knverey + +**Status**: implementable + + +## Summary + +This proposal decides the interfaces to change values in the structured data (like json,yaml) inside a Kubernetes objects' field value and implements changing function target a few formats (mainly json). + + +## Motivation + + + +kustomize can apply structured edits to Kubernetes objects defined in yaml files.\ +Sometimes structured multi-line or long single line string (ex. json,yaml, and other structured format data) is injected in Kubernetes objects' string literal field. From the kustomize perspective, these "structured" multiline strings are just arbitrary unstructured strings.\ +So, kustomize can't manipulate one value on structured, formatted data in the Kubernetes object's string literal field. This function is expected behavior, but kustomize will be very helpful if it can change the value of structured data like json and yaml substrings in a string literal.\ +This proposal allows the user to identify strings literals containing JSON/YAML data so that Kustomize can make structured edits to the data they contain. This allows the requested functionality to be added without violating Kustomize's core principle of supporting structured edits exclusively.\ + +For example, kustomize can't change the value `"REPLACE_TARGET_HOSTNAME"` in this yaml file straightforwardly. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: target-configmap +data: + config.json: |- + {"config": { + "id": "42", + "hostname": "REPLACE_TARGET_HOSTNAME" + }} +``` + +This function has been wanted for a long time. +- https://github.com/kubernetes-sigs/kustomize/issues/680 +- https://github.com/kubernetes-sigs/kustomize/issues/3787 +- https://github.com/kubernetes-sigs/kustomize/issues/4517 + +This function can be an alternative deprecated [vars](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/vars/) function in many use cases. + +**Goals:** + + +1. Provide the way to update values in the structured data like kubernetes objects with JSON/YAML formats. + + +**Non-goals:** + + +1. Do not provide [Unstructured edits, because kustomize eschews parameterization](https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/#unstructured-edits). + +1. Do not provide a way for targeting/merging values within strings from the [patches](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/patches/) field. + +1. Do not provide a way for customizing field merge strategies for the embedded data. + +## Proposal + + + +### Replacement the value in structured data + +I propose to extend the `fieldPath` and `fieldPaths` fields to replace the value in structured data with the replacements function.\ +This idea is extending how to select any field in replacement [TargetSelector](https://github.com/kubernetes-sigs/kustomize/blob/8668691ade05bc17b3c6f44bcd4723735033196e/api/types/replacement.go#L52-L64). If the `source.fieldPath` and `targets.fieldPaths` had extra values after a specific string literal in Yaml, Kustomize tries to parse that string as structure data and tries to drill down using that additional values. + +#### Example. + +```yaml +## replacement +replacements: +- source: + kind: ConfigMap + name: source-configmap + fieldPath: data.HOSTNAME + targets: + - select: + kind: ConfigMap + name: target-configmap + fieldPaths: + - data.config\.json.config.hostname # A path after `config\.json` is pointing one place in the structured data. +``` + +Please check [Story 1](#Story-1). + +### Disciplined merge the value in structured data with configMapGenerator and secretGenerator + +This Proposal is to add parameters for [configMapGenerator](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/configmapgenerator/) and [secretGenerator](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/secretgenerator/) to allow the merging of two string literals when the behavior option is set to merge and string literals value is structured.\ +This idea is to add one parameter, `mergeValues` to [GeneratorArgs](https://github.com/kubernetes-sigs/kustomize/blob/672c751715be7dd0b43b4a2fce956c84452e0db9/api/types/generatorargs.go#L7-L27) used from configMapGenerator and secretGenerator. +The `mergeValues` option is a list that contains two parameters, `key` and `format`. One element corresponds to one string literal, `key` is used to select a string literal to merge, and `format` is used to designate a format for string literal which is JSON or YAML.\ +This merge operation will be implemented for a part of [Overriding Base ConfigMap Values](https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/configmapgenerator/#overriding-base-configmap-values). It will execute to merge two string literal having same [key](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#define-the-key-to-use-when-creating-a-configmap-from-a-file) name when merging two configMap or secret.\ + +#### Example + +```yaml +configMapGenerator: +- name: demo-settings + behavior: merge # This function requires `behavior: merge`. + mergeValues: + - key: config.json # Key with a target to merge. + format: json # Setting structured data format MUST be YAML/JSON. + literals: + - config.json: |- + { + "config": { + "hostname": "REPLACE_TARGET_HOSTNAME", + "value": { + "foo": "bar" + } + } + } +``` + +Please check [Story 2](#Story-2). + +#### Appendix +- https://github.com/kubernetes-sigs/kustomize/issues/680#issuecomment-458834785 + +## User Stories + + +#### Story 1 + +Scenario summary: Replacement the value inside for structured data(json) in the configMap. + + +kustomize patching overlay is very strong to manage common yaml when using many cluster.\ +But, if you want to set cluster specific change value in the json with configMap data field, you have to replacement whole json file. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: target-configmap-dev +data: + config.json: |- + {"config": { + "id": "42", + "hostname": "dev.example.com + }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: target-configmap-prod +data: + config.json: |- + {"config": { + "id": "42", + "hostname": "prod.example.com" + }} +``` + +So if we can replacement this value in the substring formatted with json, we can easy to overlay this difference. + +```yaml +## source +apiVersion: v1 +kind: ConfigMap +metadata: + name: source-configmap +data: + HOSTNAME: www.example.com +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: target-configmap +data: + config.json: |- + {"config": { + "id": "42", + "hostname": "REPLACE_TARGET_HOSTNAME" + }} +``` + +```yaml +## replacement +replacements: +- source: + kind: ConfigMap + name: source-configmap + fieldPath: data.HOSTNAME + targets: + - select: + kind: ConfigMap + name: target-configmap + fieldPaths: + - data.config\.json.config.hostname +``` + +```yaml +## expected +apiVersion: v1 +kind: ConfigMap +metadata: + name: source-configmap +data: + HOSTNAME: www.example.com +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: target-configmap +data: + config.json: '{"config":{"hostname":"www.example.com","id":"42"}}' +``` + +#### Story 2 + +Scenario summary: Merge the two values formatted by structured data(json) with configMapGenerator. + + + +Many application needs to be set with json format file. And, That file is handled with configMap when the application is running on kubernetes.\ +So, If kustomize configMapGenerator can merge for json inside a configMap data value, A json format file will be simple and easy to handle. + +#### Source +```yaml +# base/kustomization.yaml +configMapGenerator: +- name: demo + literals: + - config.json: |- + { + "config": { + "loglevel": debug, + "parameter": { + "foo": "bar" + } + } + } +``` + +```yaml +# overlay/kustomization.yaml +resources: +- ../base +configMapGenerator: +- name: demo + behavior: merge + mergeValues: + - key: config.json # Key with a target to merge. + format: json # Setting structured data format MUST be YAML/JSON. + literals: + - config.json: |- + { + "config": { + "hostname": "www.example.com", + "parameter": { + "baz": "qux" + } + } + } +``` + +#### Result +```yaml +apiVersion: v1 +data: + config.json: |- + { + "config": { + "loglevel": debug, + "hostname": "www.example.com", + "parameter": { + "foo": "bar", + "baz": "qux" + } + } + } +kind: ConfigMap +metadata: + name: demo-xxxxxxxxxx # name suffix hash +``` + + +#### Story 3 + +Scenario summary: Replacement the value inside for yaml in the configMap. + + +A few applications require to use yaml format config file, and some major cloudnative applications are using that.(ex, Prometheus,AlertManager)\ +A value you want to overlay in yaml is usually into a nested yaml structure. So if you can overlay value inside yaml, you won't need to copy a whole yaml file. + +```yaml +## source +apiVersion: v1 +kind: ConfigMap +metadata: + name: environment-config +data: + env: dev +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config +data: + prometheus.yml: |- + global: + external_labels: + prometheus_env: TARGET_ENVIROMENT + scrape_configs: + - job_name: "prometheus" + static_configs: + - targets: ["localhost:9090"] +``` + +```yaml +## replacement +replacements: +- source: + kind: ConfigMap + name: environment-config + fieldPath: data.env + targets: + - select: + kind: ConfigMap + name: prometheus-config + fieldPaths: + - data.prometheus\.yml.global.external_labels?prometheus_env +``` + +```yaml +## expected +apiVersion: v1 +kind: ConfigMap +metadata: + name: environment-config +data: + env: dev +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config +data: + prometheus.yml: |- + global: + external_labels: + prometheus_env: dev + scrape_configs: + - job_name: "prometheus" + static_configs: + - targets: ["localhost:9090"] +``` + +#### Story 4 + +Scenario summary: Replacement the value inside for json in the annotations. + + +A few times, an application on your cluster requires to set json format config for the *Annotations* on kubernetes yaml resources. We need to overlay in this position value. + +##### Example +```yaml +## source +apiVersion: cloud.google.com/v1 +kind: BackendConfig +metadata: + name: debug-backend-config +spec: + securityPolicy: + name: "debug-security-policy" +--- +apiVersion: v1 +kind: Service +metadata: + name: appA-svc + annotations: + cloud-provider/backend-config: '{"ports": {"appA":"gke-default-backend-config"}}' +spec: + ports: + - name: appA + port: 1234 + protocol: TCP + targetPort: 8080 +``` + +```yaml +## replacement +replacements: +- source: + kind: BackendConfig + name: debug-backend-config + fieldPath: metadata.name + targets: + - select: + kind: ConServicefigMap + name: appA-svc + fieldPaths: + - metadata.annotations.cloud-provider/backend-config.ports.appA +``` + +```yaml +## expected +apiVersion: cloud.google.com/v1 +kind: BackendConfig +metadata: + name: debug-backend-config +spec: + securityPolicy: + name: "debug-security-policy" +--- +apiVersion: v1 +kind: Service +metadata: + name: appA-svc + annotations: + cloud-provider/backend-config: '{"ports": {"appA":"debug-backend-config"}}' +spec: + ports: + - name: appA + port: 1234 + protocol: TCP + targetPort: 8080 +``` + +- https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#unique_backendconfig_per_service_port +- https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/guide/ingress/annotations/#listen-ports + + +### Risks and Mitigations + + +### Dependencies + + +### Scalability + + +## Drawbacks + + +## Alternatives + + +## Rollout Plan + + +### Alpha + + +- Will the feature be gated by an "alpha" flag? Which one? +- Will the feature be available in `kubectl kustomize` during alpha? Why or why not? + +### Beta + + +### GA +