Cloud Resources Refactor (#351)

* WIP: cloud resources Terraform Docs WIP: manage frontegg users [WIP]: Cloud Resources Terraform Docs Add tests for app password Terraform Docs Fix failing tests Add region datasource Add integration tests for user resource Minor changes Terraform Docs Minor changes Add token refresh func * Port the IDs state migraton * Fix failing tests * Update the index tmpl file * Terraform Docs * Add unit tests for the region datasource * Fix failing tests and add region data schema * Terraform Docs * Add more tests * Remove some comments * Minor improvements * Update token refresh func * Add link to SO for deep clone * Prepare the changelog --------- Co-authored-by: bobbyiliev <bobbyiliev@users.noreply.github.com>
MaterializeInc · Jan 10, 2024 · c496aea · c496aea
1 parent e7eb92b
commit c496aea
Show file tree

Hide file tree

Showing 259 changed files with 4,951 additions and 898 deletions.
diff --git a/.github/workflows/acceptance.yml b/.github/workflows/acceptance.yml
@@ -22,13 +22,16 @@ jobs:
       - name: Docker Compose Up
         run: docker compose up --build -d
 
+      - name: Configure hosts file
+        run: echo "127.0.0.1 materialized frontegg cloud" | sudo tee -a /etc/hosts
+
       - name: make testacc
         run: make testacc
         env:
-          MZ_HOST: localhost
-          MZ_USER: mz_system
+          MZ_ENDPOINT: http://localhost:3000
+          MZ_CLOUD_ENDPOINT: http://localhost:3001
+          MZ_PASSWORD: mzp_1b2a3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0b1c2d3e4f5a6b7c8d9e0f1a2b
           MZ_SSLMODE: disable
-          MZ_PORT: 6877
 
       - name: Docker Compose Down
         run: docker compose down
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,48 @@
 
 ## Unreleased
 
+### Features
+* Introduced a unified interface for managing both global and regional resources.
+* Implemented single authentication using an app password for all operations.
+* Added dynamic client allocation for managing different resource types.
+* Enhanced provider configuration with parameters for default settings and optional endpoint overrides.
+* New resources:
+  * App passwords: `materialize_app_password`.
+  * User management `materialize_user`.
+* Added data sources for fetching region details (`materialize_region`).
+* Implemented support for establishing SQL connections across multiple regions.
+* Introduced a new `region` parameter in all resource and data source configurations. This allows users to specify the region for resource creation and data retrieval.
+
+### Breaking Changes
+* **Provider Configuration Changes**:
+  * Deprecated the `host`, `port`, and `user` parameters in the provider configuration. These details are now derived from the app password.
+  * Retained only the `password` definition in the provider configuration. This password is used to fetch all necessary connection information.
+* **New `region` Configuration**:
+  * Introduced a new `default_region` parameter in the provider configuration. This allows users to specify the default region for resource creation.
+  * The `default_region` parameter can be overridden in specific resource configurations if a particular resource needs to be created in a non-default region.
+
+  ```hcl
+  provider "materialize" {
+    password       = var.materialize_app_password
+    default_region = "aws/us-east-1"
+  }
+
+  resource "materialize_cluster" "cluster" {
+    name   = "cluster"
+    region = "aws/us-west-2"
+  }
+  ```
+
+### Misc
+* Mock Services for Testing:
+  * Added a new mocks directory, which includes mock services for the Cloud API and the FrontEgg API.
+  * These mocks are intended for local testing and CI, facilitating development and testing without the need for a live backend.
+
+### Migration Guide
+* Before upgrading to `v0.5.0`, users should ensure that they have upgraded to `v0.4.x` which introduced the Terraform state migration necessary for `v0.5.0`. After upgrading to `v0.4.x`, users should run `terraform plan` to ensure that the state migration has completed successfully.
+* Users upgrading to `v0.5.0` should update their provider configurations to remove the `host`, `port`, and `user` parameters and ensure that the `password` parameter is set with the app password.
+* For managing resources across multiple regions, users should specify the `default_region` parameter in their provider configuration or override it in specific resource blocks as needed using the `region` parameter.
+
 ## 0.4.3 - 2024-01-08
 
 ### Breaking Changes

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -40,15 +40,16 @@ make test
 To run the acceptance tests which will simulate running Terraform commands you will need to set the necessary envrionment variables and start the docker compose:
 
 ```bash
-export MZ_HOST=localhost
-export MZ_USER=mz_system
-export MZ_SSLMODE=disable
-export MZ_PORT=6877
-
 # Start all containers
 docker-compose up -d --build
 ```
 
+Add the following to your `hosts` file so that the provider can connect to the mock services:
+
+```
+127.0.0.1 materialized frontegg cloud
+```
+
 You can then run the acceptance tests:
 
 ```bash
@@ -125,6 +126,7 @@ var clusterSchema = map[string]*schema.Schema{
 		Description: "The size of the cluster.",
 		Optional:    true,
 	},
+	"region": RegionSchema(),
 }
 ```
 
@@ -147,8 +149,12 @@ If the resource can be updated we would also have to change the update context `
 
 ```go
 if d.HasChange("size") {
+		metaDb, region, err := utils.GetDBClientFromMeta(meta, d)
+		if err != nil {
+			return diag.FromErr(err)
+		}
 		_, newSize := d.GetChange("size")
-		b := materialize.NewClusterBuilder(meta.(*sqlx.DB), o)
+		b := materialize.NewClusterBuilder(metaDb, o)
 		if err := b.Resize(newSize.(string)); err != nil {
 				return diag.FromErr(err)
 		}
@@ -182,6 +188,10 @@ Schema: map[string]*schema.Schema{
 			},
 		},
 	},
+	"region": {
+		Type:     schema.TypeString,
+		Computed: true,
+	},
 },
 ```
 
@@ -201,7 +211,7 @@ for _, p := range dataSource {
 
 ## Cutting a release
 
-To cut a new release of the provider, create a new tag and push that tag. This will trigger a Github Action to generate the artifacts necessary for the Terraform Registry.
+To cut a new release of the provider, create a new tag and push that tag. This will trigger a GitHub Action to generate the artifacts necessary for the Terraform Registry.
 
 ```bash
 git tag -a vX.Y.Z -m vX.Y.Z

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -87,3 +87,15 @@ services:
       TF_LOG: INFO
     command: >
       sh -c "tail -F /dev/null"
+
+  frontegg:
+    container_name: frontegg
+    build: mocks/frontegg
+    ports:
+      - "3000:3000"
+
+  cloud:
+    container_name: cloud
+    build: mocks/cloud
+    ports:
+      - "3001:3001"
diff --git a/docs/data-sources/cluster.md b/docs/data-sources/cluster.md
@@ -23,6 +23,7 @@ data "materialize_cluster" "all" {}
 
 - `clusters` (List of Object) The clusters in the account (see [below for nested schema](#nestedatt--clusters))
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 
 <a id="nestedatt--clusters"></a>
 ### Nested Schema for `clusters`

diff --git a/docs/data-sources/cluster_replica.md b/docs/data-sources/cluster_replica.md
@@ -23,6 +23,7 @@ data "materialize_cluster_replica" "all" {}
 
 - `cluster_replicas` (List of Object) The cluster replicas in the account (see [below for nested schema](#nestedatt--cluster_replicas))
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 
 <a id="nestedatt--cluster_replicas"></a>
 ### Nested Schema for `cluster_replicas`

diff --git a/docs/data-sources/connection.md b/docs/data-sources/connection.md
@@ -37,6 +37,7 @@ data "materialize_connection" "materialize_schema" {
 
 - `connections` (List of Object) The connections in the account (see [below for nested schema](#nestedatt--connections))
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 
 <a id="nestedatt--connections"></a>
 ### Nested Schema for `connections`

diff --git a/docs/data-sources/current_cluster.md b/docs/data-sources/current_cluster.md
@@ -27,3 +27,4 @@ output "cluster_name" {
 
 - `id` (String) The ID of this resource.
 - `name` (String)
+- `region` (String) The region in which the resource is located.
diff --git a/docs/data-sources/current_database.md b/docs/data-sources/current_database.md
@@ -27,3 +27,4 @@ output "database_name" {
 
 - `id` (String) The ID of this resource.
 - `name` (String)
+- `region` (String) The region in which the resource is located.
diff --git a/docs/data-sources/database.md b/docs/data-sources/database.md
@@ -23,6 +23,7 @@ data "materialize_database" "all" {}
 
 - `databases` (List of Object) The databases in the account (see [below for nested schema](#nestedatt--databases))
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 
 <a id="nestedatt--databases"></a>
 ### Nested Schema for `databases`

diff --git a/docs/data-sources/egress_ips.md b/docs/data-sources/egress_ips.md
@@ -27,3 +27,4 @@ output "ips" {
 
 - `egress_ips` (List of String) The egress IPs in the account
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
diff --git a/docs/data-sources/index.md b/docs/data-sources/index.md
@@ -37,6 +37,7 @@ data "materialize_index" "materialize_schema" {
 
 - `id` (String) The ID of this resource.
 - `indexes` (List of Object) The indexes in the account (see [below for nested schema](#nestedatt--indexes))
+- `region` (String) The region in which the resource is located.
 
 <a id="nestedatt--indexes"></a>
 ### Nested Schema for `indexes`

diff --git a/docs/data-sources/materialized_view.md b/docs/data-sources/materialized_view.md
@@ -37,6 +37,7 @@ data "materialize_materialized_view" "materialize_schema" {
 
 - `id` (String) The ID of this resource.
 - `materialized_views` (List of Object) The materialized views in the account (see [below for nested schema](#nestedatt--materialized_views))
+- `region` (String) The region in which the resource is located.
 
 <a id="nestedatt--materialized_views"></a>
 ### Nested Schema for `materialized_views`

diff --git a/docs/data-sources/region.md b/docs/data-sources/region.md
@@ -0,0 +1,40 @@
+---
+# generated by https://github.com/hashicorp/terraform-plugin-docs
+page_title: "materialize_region Data Source - terraform-provider-materialize"
+subcategory: ""
+description: |-
+  
+---
+
+# materialize_region (Data Source)
+
+
+
+## Example Usage
+
+```terraform
+data "materialize_region" "all" {}
+
+output "region" {
+  value = data.materialize_region.all
+}
+```
+
+<!-- schema generated by tfplugindocs -->
+## Schema
+
+### Read-Only
+
+- `id` (String) The ID of this resource.
+- `regions` (List of Object) (see [below for nested schema](#nestedatt--regions))
+
+<a id="nestedatt--regions"></a>
+### Nested Schema for `regions`
+
+Read-Only:
+
+- `cloud_provider` (String)
+- `host` (String)
+- `id` (String)
+- `name` (String)
+- `url` (String)
diff --git a/docs/data-sources/role.md b/docs/data-sources/role.md
@@ -22,6 +22,7 @@ data "materialize_role" "all" {}
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `roles` (List of Object) The roles in the account (see [below for nested schema](#nestedatt--roles))
 
 <a id="nestedatt--roles"></a>

diff --git a/docs/data-sources/schema.md b/docs/data-sources/schema.md
@@ -30,6 +30,7 @@ data "materialize_schema" "materialize" {
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `schemas` (List of Object) The schemas in the account (see [below for nested schema](#nestedatt--schemas))
 
 <a id="nestedatt--schemas"></a>

diff --git a/docs/data-sources/secret.md b/docs/data-sources/secret.md
@@ -36,6 +36,7 @@ data "materialize_secret" "materialize_schema" {
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `secrets` (List of Object) The secrets in the account (see [below for nested schema](#nestedatt--secrets))
 
 <a id="nestedatt--secrets"></a>

diff --git a/docs/data-sources/sink.md b/docs/data-sources/sink.md
@@ -36,6 +36,7 @@ data "materialize_sink" "materialize_schema" {
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `sinks` (List of Object) The sinks in the account (see [below for nested schema](#nestedatt--sinks))
 
 <a id="nestedatt--sinks"></a>

diff --git a/docs/data-sources/source.md b/docs/data-sources/source.md
@@ -36,6 +36,7 @@ data "materialize_source" "materialize_schema" {
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `sources` (List of Object) The sources in the account (see [below for nested schema](#nestedatt--sources))
 
 <a id="nestedatt--sources"></a>

diff --git a/docs/data-sources/table.md b/docs/data-sources/table.md
@@ -23,6 +23,7 @@ description: |-
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `tables` (List of Object) The tables in the account (see [below for nested schema](#nestedatt--tables))
 
 <a id="nestedatt--tables"></a>

diff --git a/docs/data-sources/type.md b/docs/data-sources/type.md
@@ -23,6 +23,7 @@ description: |-
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `types` (List of Object) The types in the account (see [below for nested schema](#nestedatt--types))
 
 <a id="nestedatt--types"></a>

diff --git a/docs/data-sources/view.md b/docs/data-sources/view.md
@@ -36,6 +36,7 @@ data "materialize_view" "materialize_schema" {
 ### Read-Only
 
 - `id` (String) The ID of this resource.
+- `region` (String) The region in which the resource is located.
 - `views` (List of Object) The views in the account (see [below for nested schema](#nestedatt--views))
 
 <a id="nestedatt--views"></a>
Original file line number	Diff line number	Diff line change
Expand Up		@@ -27,3 +27,4 @@ output "cluster_name" {

		- `id` (String) The ID of this resource.
		- `name` (String)
		- `region` (String) The region in which the resource is located.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -27,3 +27,4 @@ output "database_name" {

		- `id` (String) The ID of this resource.
		- `name` (String)
		- `region` (String) The region in which the resource is located.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -27,3 +27,4 @@ output "ips" {

		- `egress_ips` (List of String) The egress IPs in the account
		- `id` (String) The ID of this resource.
		- `region` (String) The region in which the resource is located.