terraform-aws-arc-dms

Overview

SourceFuse AWS Reference Architecture (ARC) Terraform module for creating Database Migration Service.

AWS DMS facilitates seamless, secure database migrations to AWS, enabling both homogeneous and heterogeneous data transfers with minimal downtime. DMS supports continuous data replication through Change Data Capture (CDC) and integrates with a broad range of source and target databases, providing a robust, scalable solution for reliable data migration.

For more information about this repository and its usage, please see Terraform AWS ARC CloudFront Usage Guide.

Usage

To see a DMS example, check out the main.tf file in the example folder.

module "aws_dms" {
  source = "../modules/dms"

  # Subnet
  subnet_group_id          = "dms-poc-public-subnet-group"
  subnet_group_description = "Subnet for DMS POC"
  subnet_group_subnet_ids  = ["subnet-1", "subnet-2"] #List of Subnet IDs

  # Instance
  instance_allocated_storage = 5
  instance_apply_immediately = true
  instance_network_type      = "IPV4"
  instance_class             = "dms.t2.micro"
  instance_id                = "DMS-POC"
  instance_subnet_group_id   = "dms-poc-public-subnet-group"
  instance_publicly_accessible = true
  instance_vpc_security_group_ids = ["<sg-id>"] #Security Group ID

  endpoints = {
    db1 = {
      endpoint_id         = "dms-poc-endpoint-1"
      endpoint_type       = "source"
      engine_name         = "postgres"
      database_name       = "poc"
      secrets_manager_arn = "<secret-arn>" #Source endpoint secret arn
      ssl_mode            = "require"

      postgres_settings = {
        execute_timeout = 60
      }
    }

    db2 = {
      endpoint_id         = "dms-poc-endpoint-2"
      endpoint_type       = "target"
      engine_name         = "postgres"
      database_name       = "poc_target"
      secrets_manager_arn = "<secret-arn>" #Target endpoint secret arn
      ssl_mode            = "require"
    }
  }

  replication_tasks = {
    task1 = {
      replication_task_id = "replication-task-1"
      migration_type      = "full-load"         # Full load
      source_endpoint_key = "db1"               # References key in endpoints map
      target_endpoint_key = "db2"               # References key in endpoints map
      table_mappings      = "{\"rules\":[{\"rule-type\":\"selection\",\"rule-id\":\"1\",\"rule-name\":\"1\",\"object-locator\":{\"schema-name\":\"public\",\"table-name\":\"%\"},\"rule-action\":\"include\"}]}"

    }
  }
}

Requirements

Name	Version
terraform	>= 1.4, < 2.0.0
aws	>= 4.0, < 6.0

Providers

No providers.

Modules

Name	Source	Version
aws_dms	./modules/dms	n/a

Resources

No resources.

Inputs

Name	Description	Type	Default	Required
create_subnet_group	Determines whether the replication subnet group will be created	bool	true	no
endpoints	Map of endpoints used in the system	map(object({ endpoint_id = string endpoint_type = string engine_name = string extra_connection_attributes = optional(string) database_name = optional(string) kms_key_arn = string port = optional(number) server_name = optional(string) ssl_mode = optional(string) secrets_manager_arn = optional(string) secrets_manager_access_role_arn = optional(string) service_access_role = optional(string) username = optional(string) postgres_settings = optional(object({ after_connect_script = optional(string) babelfish_database_name = optional(string) capture_ddls = optional(bool) database_mode = optional(string) ddl_artifacts_schema = optional(string) execute_timeout = optional(number) fail_tasks_on_lob_truncation = optional(bool) heartbeat_enable = optional(bool) heartbeat_frequency = optional(number) heartbeat_schema = optional(string) map_boolean_as_boolean = optional(bool) map_jsonb_as_clob = optional(bool) map_long_varchar_as = optional(string) max_file_size = optional(number) plugin_name = optional(string) slot_name = optional(string) })) }))	n/a	yes
instance_allocated_storage	The amount of storage (in gigabytes) to be initially allocated for the replication instance. Min: 5, Max: 6144, Default: 50	number	null	no
instance_allow_major_version_upgrade	Indicates that major version upgrades are allowed	bool	true	no
instance_apply_immediately	Indicates whether the changes should be applied immediately or during the next maintenance window	bool	null	no
instance_auto_minor_version_upgrade	Indicates that minor engine upgrades will be applied automatically to the replication instance during the maintenance window	bool	true	no
instance_availability_zone	The EC2 Availability Zone that the replication instance will be created in	string	null	no
instance_class	The compute and memory capacity of the replication instance as specified by the replication instance class	string	"dms.t2.micro"	no
instance_engine_version	The engine version number of the replication instance	string	null	no
instance_id	The replication instance identifier. This parameter is stored as a lowercase string	string	"dms-instance"	no
instance_kms_key_arn	The Amazon Resource Name (ARN) for the KMS key that will be used to encrypt the connection parameters	string	null	no
instance_multi_az	Specifies if the replication instance is a multi-az deployment. You cannot set the availability_zone parameter if the multi_az parameter is set to true	bool	null	no
instance_network_type	The type of IP address protocol used by a replication instance. Valid values: IPV4, DUAL	string	null	no
instance_preferred_maintenance_window	The weekly time range during which system maintenance can occur, in Universal Coordinated Time (UTC)	string	null	no
instance_publicly_accessible	Specifies the accessibility options for the replication instance	bool	null	no
instance_subnet_group_id	An existing subnet group to associate with the replication instance	string	null	no
instance_vpc_security_group_ids	A list of VPC security group IDs to be used with the replication instance	list(string)	null	no
replication_tasks	n/a	map(object({ replication_task_id = string migration_type = string cdc_start_position = optional(string) cdc_start_time = optional(string) source_endpoint_key = string # Key to reference source endpoint target_endpoint_key = string # Key to reference target endpoint replication_task_settings = optional(string) start_replication_task = optional(bool) table_mappings = string tags = optional(map(string)) }))	{}	no
replication_tasks_serverless	Map of serverless replication tasks	map(object({ migration_type = string replication_task_id = string replication_task_settings = optional(map(string)) supplemental_task_settings = optional(map(string)) start_replication_task = optional(bool) source_endpoint_arn = optional(string) target_endpoint_arn = optional(string) table_mappings = optional(string) serverless_config = optional(object({ availability_zone = optional(string) dns_name_servers = optional(list(string)) kms_key_id = optional(string) max_capacity_units = number min_capacity_units = optional(number) multi_az = optional(bool) preferred_maintenance_window = optional(string) vpc_security_group_ids = optional(list(string)) })) }))	n/a	yes
s3_endpoints	n/a	map(object({ endpoint_id = string endpoint_type = string kms_key_arn = optional(string) ssl_mode = optional(string) add_column_name = optional(bool) add_trailing_padding_character = optional(bool) bucket_folder = optional(string) bucket_name = string canned_acl_for_objects = optional(string) cdc_inserts_and_updates = optional(bool) cdc_inserts_only = optional(bool) cdc_max_batch_interval = optional(number) cdc_min_file_size = optional(number) cdc_path = optional(string) compression_type = optional(string) csv_delimiter = optional(string) csv_no_sup_value = optional(bool) csv_null_value = optional(string) csv_row_delimiter = optional(string) data_format = optional(string) data_page_size = optional(number) date_partition_delimiter = optional(string) date_partition_enabled = optional(bool) date_partition_sequence = optional(string) date_partition_timezone = optional(string) detach_target_on_lob_lookup_failure_parquet = optional(bool) dict_page_size_limit = optional(number) enable_statistics = optional(bool) encoding_type = optional(string) encryption_mode = optional(string) expected_bucket_owner = optional(string) external_table_definition = optional(string) glue_catalog_generation = optional(bool) ignore_header_rows = optional(bool) include_op_for_full_load = optional(bool) max_file_size = optional(number) parquet_timestamp_in_millisecond = optional(bool) parquet_version = optional(string) preserve_transactions = optional(bool) rfc_4180 = optional(bool) row_group_length = optional(number) server_side_encryption_kms_key_id = optional(string) service_access_role_arn = string timestamp_column_name = optional(string) use_csv_no_sup_value = optional(bool) use_task_start_time_for_full_load_timestamp = optional(bool) tags = optional(map(string)) }))	n/a	yes
subnet_group_description	The description for the subnet group	string	"DMS Replication subnet group"	no
subnet_group_id	The name for the replication subnet group. Stored as a lowercase string, must contain no more than 255 alphanumeric characters, periods, spaces, underscores, or hyphens	string	"DMS_replication_subnet_group"	no
subnet_group_subnet_ids	A list of the EC2 subnet IDs for the subnet group	list(string)	[]	no
subnet_group_tags	A map of additional tags to apply to the replication subnet group	map(string)	{}	no

Outputs

Name	Description
certificates	A map of maps containing the certificates created and their full output of attributes and values
dms_access_for_endpoint_iam_role_arn	ARN specifying the role
dms_access_for_endpoint_iam_role_id	Name of the IAM role
dms_access_for_endpoint_iam_role_unique_id	Stable and unique string identifying the role
endpoints	A map of maps containing the endpoints created and their full output of attributes and values
event_subscriptions	A map of maps containing the event subscriptions created and their full output of attributes and values
replication_instance_arn	The Amazon Resource Name (ARN) of the replication instance
replication_instance_tags_all	A map of tags assigned to the resource, including those inherited from the provider default_tags configuration block
replication_subnet_group_id	The ID of the subnet group
replication_tasks	A map of maps containing the replication tasks created and their full output of attributes and values
s3_endpoints	A map of maps containing the S3 endpoints created and their full output of attributes and values
serverless_replication_tasks	A map of maps containing the serverless replication tasks (replication_config) created and their full output of attributes and values

Versioning

while Contributing or doing git commit please specify the breaking change in your commit message whether its major,minor or patch

For Example

git commit -m "your commit message #major"

By specifying this , it will bump the version and if you don't specify this in your commit message then by default it will consider patch and will bump that accordingly

Development

Prerequisites

• terraform
• terraform-docs
• pre-commit
• golang
• golint

Configurations

• Configure pre-commit hooks

  pre-commit install

Tests

• Tests are available in test directory
• Configure the dependencies

  cd test/
  go mod init github.com/sourcefuse/terraform-aws-refarch-vpn
  go get github.com/gruntwork-io/terratest/modules/terraform

• Now execute the test

  go test -timeout  30m

Authors

This project is authored by:

• SourceFuse