Add Helm and CRD override variables for full customization #121

jubrad · 2025-12-06T04:21:45Z

- Add `materialize_spec_override` to materialize-instance module for arbitrary CRD fields
- Expose `helm_values_override` and `materialize_spec_override` in all cloud examples
- Change default authenticator_kind to Password for all deployments
- Add deepmerge provider for spec merging
- Add Python script to generate Terraform types from upstream CRD/Helm schemas
- Set up pyproject.toml with uv for dependency management
- Add GitHub Action to verify schema sync with upstream Materialize
- Update CONTRIBUTING.md with development setup and type generation docs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Auto generating terraform variables for our CRD and Helm chart is something I've been thinking about for a while and this is a proof of concept for it.

There are many benefits to having an object or set of vars that work with LSP or AI in order to help write code (as opposed to object(any)). And it's also extremely beneficial to be able to set any viable attribute for the helm chart or the materialize CR.

PR has been updated to be a bit more reviewable.
Commit 1 - adds generation script
Commit 2 - adds GH action and docs
Commit 3 - Adds the auto-generated files and adds deprecations

jshiwamV · 2025-12-06T13:29:39Z

this looks promising will take a look in detail on monday

alex-hunt-materialize

This is a great idea, but I really don't want us manually parsing and building HCL. Terraform supports json, or we could use a proper HCL parser. I lean toward just converting it all to json, as that opens the door for customers to use their own tools for this too.

I didn't review most of this PR, as it is just a draft. I only did a quick skim of the generator code.

scripts/generate_terraform_types.py

alex-hunt-materialize · 2025-12-08T16:43:10Z

aws/examples/simple/main.tf


-  # Rollout configuration
-  force_rollout   = var.force_rollout
-  request_rollout = var.request_rollout


We should get consistent about how we handle this stuff. These are removed, while most of the others are marked as deprecated.

yeah, marking this as deprecated is probably for the best.

jubrad · 2025-12-08T16:53:32Z

This is a great idea, but I really don't want us manually parsing and building HCL. Terraform supports json, or we could use a proper HCL parser. I lean toward just converting it all to json, as that opens the door for customers to use their own tools for this too.

Should we use an HCL parser instead?
https://pypi.org/project/python-hcl2/
We could also rewrite the variables file into json, which both python and terraform can parse easily.

Great ideas I'll give both a shot.

jubrad · 2025-12-13T04:29:54Z

@alex-hunt-materialize I went down a rabbit hole with variables in json.tf... the variable type ends up being a string with horrific formatting. We could also use python-hcl2 to output hcl, but it also has shit formatting for complex variables. Hand rolling hcl gen and formatting is potentially the ugliest option to code up, but it generates the cleanest output which I think is worth prioritizing.

Introduces generate_terraform_types.py which dynamically generates Terraform variable type definitions from Materialize's upstream schemas: - CRD types from materialize_crd_descriptions.json - Helm parameter types from materialize_operator_chart_parameter.yml The script reads the version from the environmentd_version variable default in the source code and outputs native HCL format (.gen.tf) with proper formatting via terraform fmt. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add check_schema_sync.py to verify generated types match upstream schemas - Add GitHub Actions workflow to run sync check on PRs - Update CONTRIBUTING.md with instructions for regenerating types 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Generated type definitions: - kubernetes/modules/materialize-instance/crd_variables.gen.tf - {aws,azure,gcp}/modules/operator/helm_variables.gen.tf - {aws,azure,gcp}/examples/simple/override_variables.gen.tf Additional changes: - Add helm_values_override and materialize_spec_override variables to operator modules and examples for full customization - Add deprecated force_rollout and request_rollout variables to examples for backwards compatibility, merged into materialize_spec_override - Update materialize-instance module to use the new override variable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

alex-hunt-materialize

Overall this looks pretty good. I do worry a bit about how we will test this, as it is a pretty significant change.

alex-hunt-materialize · 2025-12-17T17:14:55Z

scripts/generate_terraform_types.py

+    if crd_type in type_lookup:
+        # Detect cycles to prevent infinite recursion
+        if crd_type in visited:
+            return "any"


Are we actually hitting cycles? Should we log something here, so we can fix that?

alex-hunt-materialize · 2025-12-17T17:28:17Z

scripts/generate_terraform_types.py

+        if len(parts) == 2:
+            value_type = crd_type_to_terraform(parts[1], type_lookup, visited)
+            if value_type == "string":
+                return "map(string)"


These two lines are useless, as they evaluate to the same as the line below.

alex-hunt-materialize · 2025-12-18T12:42:27Z

aws/examples/simple/override_variables.gen.tf

+      affinity         = optional(any)
+      defaultResources = optional(any)
+      enabled          = optional(bool)
+      nodeSelector     = optional(any)


Many of these are labeled any. We should at least be able to indicate object(any) for the ones that are objects, even if we can't easily say they are a map of string to string.

I have an update to the auto-generated helm values documentation in MaterializeInc/materialize#34563 which may help with this, if you decide to use the helm-docs generated file.

alex-hunt-materialize · 2025-12-18T15:09:31Z

kubernetes/modules/materialize-instance/main.tf

+    environmentdExtraEnv = length(var.environmentd_extra_env) > 0 ? [{
+      name = "MZ_SYSTEM_PARAMETER_DEFAULT"
+      value = join(";", [
+        for item in var.environmentd_extra_env :
+        "${item.name}=${item.value}"
+      ])
+    }] : null


Maybe not something to fix in this PR, but this seems like it is wrong and was wrong before this PR. The environmentd_extra_env should not be assumed to be setting a default parameter, but any environment variable.

alex-hunt-materialize · 2025-12-18T15:16:23Z

scripts/generate_terraform_types.py

+        "string": "any",  # string in helm params often means complex YAML that's stringified
+        "bool": "bool",
+        "int": "number",
+        "object": "any",  # object without more info maps to any


Shouldn't this map to object(any)?

alex-hunt-materialize · 2025-12-18T15:22:57Z

scripts/generate_terraform_types.py

+    # Run terraform fmt to ensure proper formatting
+    run_terraform_fmt(file_path)
+
+    return True  # We always write, so always return True for simplicity


Why even have these return a bool?

jubrad force-pushed the auto-generate-tfvars branch 2 times, most recently from f6c168a to e83a36e Compare December 6, 2025 04:45

alex-hunt-materialize reviewed Dec 8, 2025

View reviewed changes

jubrad force-pushed the auto-generate-tfvars branch 3 times, most recently from 6a3990f to 63cb495 Compare December 13, 2025 04:24

jubrad force-pushed the auto-generate-tfvars branch 2 times, most recently from ec06ba6 to 83e2a04 Compare December 13, 2025 05:26

jubrad and others added 3 commits December 12, 2025 23:32

jubrad force-pushed the auto-generate-tfvars branch from 83e2a04 to a1e37fb Compare December 13, 2025 05:35

jshiwamV mentioned this pull request Dec 16, 2025

variables.tf file in azure and gcp simple/examples are missing force_rollout and request_rollout #120

Open

alex-hunt-materialize reviewed Dec 18, 2025

View reviewed changes

Add Helm and CRD override variables for full customization #121

Are you sure you want to change the base?

Add Helm and CRD override variables for full customization #121

Uh oh!

Conversation

jubrad commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jshiwamV commented Dec 6, 2025

Uh oh!

alex-hunt-materialize left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jubrad commented Dec 8, 2025

Uh oh!

jubrad commented Dec 13, 2025

Uh oh!

alex-hunt-materialize left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jubrad commented Dec 6, 2025 •

edited

Loading