Skip to content

Latest commit

 

History

History
43 lines (23 loc) · 2.71 KB

demo.md

File metadata and controls

43 lines (23 loc) · 2.71 KB
Raw

Backstage template generation

Using the CNOE CLI, you can generate Backstage template input fields from Terraform modules, Kubernetes CRDs, and Crossplane XRDs.

Terraform (imperative)

For example, to generate input field from a terraform module in the Data on EKS repository:

git clone https://github.com/awslabs/data-on-eks.git /tmp/data-on-eks

cnoe template tf -i /tmp/data-on-eks/analytics/terraform/spark-k8s-operator -o examples/template-generation -t examples/template-generation/data-on-eks.yaml -p '.spec.parameters[0].properties.tfVars'

The above command takes the terraform module available at /tmp/data-on-eks/analytics/terraform/spark-k8s-operator then inserts generated backstage fields into a partially configured template at examples/template-generation/data-on-eks.yaml. This partially configure template contains fields that are common to all modules in the Data on EKS repository.

You can view what fields are generated by using diff and yq:

diff <(yq -P 'sort_keys(..)' -o=props examples/template-generation/spark-k8s-operator.yaml) <(yq -P 'sort_keys(..)' -o=props examples/template-generation/data-on-eks.yaml)

The rest of this document assumes you've used the spark-k8s-operator module to generate input fields.

Once the template is hydrated, it is ready to use. You can push it to a private git repository within your organization or push it to a public repository. You can then register the template in the Backstage UI (Create... > Register Existing Component) or update your Backstage configuration. The main configuration file is a ConfigMap in the backstage namespace with the name backstage-config.

You can use the template to create a EKS cluster with optimization for data workload using terraform. The example uses a custom Argo Workflows template to run the install script, add a IAM role to the admin group in the cluster, create a Backstage service account, then registers the cluster to Backstage for use.

The Argo Workflows run is associated with the Backstage entity. So you can check on its status within Backstage. If you'd like to see detailed information about the run, you can click on it to go to the Argo Workflows UI.

Once the new cluster is ready, you can submit a Spark Job to the new cluster through Backstage.

Terraform (declarative)

You can also use the terraform controller or Crossplane terraform provider to deploy terraform modules. Examples of this will be included in the future.

Crossplane and ACK

When using Crossplane, the bulk of work is done by Crossplane providers. You can use GitOps tools like ArgoCD to deploy these manifests to the cluster using Backstage scaffolder templates. Examples of this will be included in the future.