GCP MySQL onboarding module (#24)

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 ### Features
 - Aurora PostgreSQL CloudWatch with slow query auditing example
+- Google Cloud SQL for MySQL module
 
 ### Bug Fixes
 - Modified Server Host Name of AWS RDS MS SQL SERVER Dsfhub assets

DSF_VERSION_COMPATABILITY.md

@@ -103,5 +103,9 @@ The following table lists the DSF versions that each module is tested and mainta
       <td>onboard-gcp-bigquery</td>
       <td>4.17+</td>
    </tr>
+   <tr>
+      <td>onboard-gcp-mysql</td>
+      <td>4.17+</td>
+   </tr>
 
 </table>

examples/onboard-gcp-bigquery/README.md

@@ -1,5 +1,5 @@
 # Onboard Google Cloud BigQuery example
-This example includes additional prerequisites that will need to be completed to fully utilize the module. More details can be found in the [onboarding documentation](hhttps://docs.imperva.com/bundle/onboarding-databases-to-sonar-reference-guide/page/BigQuery-Onboarding-Steps_48367536.html).
+This example includes additional prerequisites that will need to be completed to fully utilize the module. More details can be found in the [onboarding documentation](https://docs.imperva.com/bundle/onboarding-databases-to-sonar-reference-guide/page/BigQuery-Onboarding-Steps_48367536.html).
 
 This example creates 'dsfhub' and 'google' resources. More information regarding authentication to each can be found in the relevant provider documentation:
 - [dsfhub](https://registry.terraform.io/providers/imperva/dsfhub/latest/docs)

examples/onboard-gcp-mysql/README.md

@@ -0,0 +1,48 @@
+# Onboard Google Cloud SQL for MySQL example
+This example includes additional prerequisites that will need to be completed to fully utilize the module. More details can be found in the [onboarding documentation](https://docs.imperva.com/bundle/onboarding-databases-to-sonar-reference-guide/page/Cloud-SQL-for-MySQL-Onboarding-Steps_48367584.html).
+
+This example creates 'dsfhub' and 'google' resources. More information regarding authentication to each can be found in the relevant provider documentation:
+- [dsfhub](https://registry.terraform.io/providers/imperva/dsfhub/latest/docs)
+- [google](https://registry.terraform.io/providers/hashicorp/google/latest/docs)
+
+## Prerequisites
+### Service Account
+A Google Service Account will need to be created with permissions to read from PubSub subscriptions. This can be done via the ``google-service-account-dsf`` module. Depending on the authentication mechanism chosen, the service account will either need to be attached to a GCP Compute Engine instance or the service account's credentials file will need to be copied to your Agentless Gateway.
+
+### Google PubSub Subscription
+A Google logging sink, PubSub topic, and PubSub subscription in addition to a GCP PUBSUB asset in DSF will need to be created in advance. This prerequisite is handled by the ``onboard-gcp-pubsub`` module.
+
+<!-- BEGIN_TF_DOCS -->
+## Requirements
+
+No requirements.
+
+## Providers
+
+No providers.
+
+## Modules
+
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_gcp-mysql-1"></a> [gcp-mysql-1](#module\_gcp-mysql-1) | ../../modules/onboard-gcp-mysql | n/a |
+| <a name="module_gcp-mysql-2"></a> [gcp-mysql-2](#module\_gcp-mysql-2) | ../../modules/onboard-gcp-mysql | n/a |
+| <a name="module_gcp-mysql-3"></a> [gcp-mysql-3](#module\_gcp-mysql-3) | ../../modules/onboard-gcp-mysql | n/a |
+| <a name="module_gcp-pubsub-1"></a> [gcp-pubsub-1](#module\_gcp-pubsub-1) | ../../modules/onboard-gcp-pubsub | n/a |
+| <a name="module_gcp-pubsub-2-audit"></a> [gcp-pubsub-2-audit](#module\_gcp-pubsub-2-audit) | ../../modules/onboard-gcp-pubsub | n/a |
+| <a name="module_gcp-pubsub-2-slow-query"></a> [gcp-pubsub-2-slow-query](#module\_gcp-pubsub-2-slow-query) | ../../modules/onboard-gcp-pubsub | n/a |
+| <a name="module_gcp-pubsub-3"></a> [gcp-pubsub-3](#module\_gcp-pubsub-3) | ../../modules/onboard-gcp-pubsub | n/a |
+| <a name="module_service-account"></a> [service-account](#module\_service-account) | ../../modules/google-service-account-dsf | n/a |
+
+## Resources
+
+No resources.
+
+## Inputs
+
+No inputs.
+
+## Outputs
+
+No outputs.
+<!-- END_TF_DOCS -->

examples/onboard-gcp-mysql/main.tf

@@ -0,0 +1,274 @@
+locals {
+  admin_email           = "test@example.com"
+  gateway_id            = "a1b2c3d4-e5f6-g8h9-wxyz-123456790"
+  pubsub_auth_mechanism = "default"
+
+  gcp_mysql_instance_authorized_networks = [
+    {
+      name  = "local"
+      value = "127.0.0.1"
+    }
+  ]
+  gcp_project_id           = "my-gcp-project"
+  gcp_service_account_name = "dsf-service-account"
+
+  excluded_traffic_filter = [
+    {
+      name   = "exclude-internal-traffic"
+      filter = "textPayload:\"[root]\" OR \"__google_connectivity_prober\""
+    }
+  ]
+}
+
+################################################################################
+# Providers
+################################################################################
+terraform {
+  required_providers {
+    dsfhub = {
+      source = "imperva/dsfhub"
+    }
+  }
+}
+
+provider "google" {
+  # Authenticated via "gcloud" CLI
+  project = local.gcp_project_id
+}
+
+provider "dsfhub" {}
+
+################################################################################
+# Prerequisites
+# 1. A service account with permissions to read from the PubSub subscription
+# 2. A Google sink router, PubSub topic and subscription (handled below)
+################################################################################
+module "service-account" {
+  source = "../../modules/google-service-account-dsf"
+
+  account_id     = local.gcp_service_account_name
+  auth_mechanism = local.pubsub_auth_mechanism
+  description    = "MySQL audit pull service account"
+  project        = local.gcp_project_id
+  project_roles = [
+    "roles/pubsub.subscriber",
+    "roles/pubsub.viewer"
+  ]
+}
+
+################################################################################
+# GCP MySQL 8.0
+################################################################################
+locals {
+  gcp_mysql_1_instance_name = "tf-mysql-8"
+}
+
+module "gcp-pubsub-1" {
+  source = "../../modules/onboard-gcp-pubsub"
+
+  gcp_pubsub_admin_email    = local.admin_email
+  gcp_pubsub_audit_type     = "MYSQL"
+  gcp_pubsub_auth_mechanism = local.pubsub_auth_mechanism
+  gcp_pubsub_gateway_id     = local.gateway_id
+
+  project = local.gcp_project_id
+
+  pubsub_subscription_name = "${local.gcp_mysql_1_instance_name}-sub"
+
+  pubsub_topic_name = "${local.gcp_mysql_1_instance_name}-topic"
+
+  sink_router_description = "MySQL 8.0 sink"
+  sink_router_exclusions  = local.excluded_traffic_filter
+  sink_router_filter      = <<EOF
+    resource.type="cloudsql_database"
+    resource.labels.database_id="${local.gcp_project_id}:${local.gcp_mysql_1_instance_name}"
+    logName="projects/${local.gcp_project_id}/logs/cloudsql.googleapis.com%2Fmysql-general.log"
+  EOF
+  sink_router_name        = "${local.gcp_mysql_1_instance_name}-sink"
+}
+
+module "gcp-mysql-1" {
+  source = "../../modules/onboard-gcp-mysql"
+
+  gcp_mysql_admin_email               = local.admin_email
+  gcp_mysql_audit_pull_enabled        = true
+  gcp_mysql_gateway_id                = local.gateway_id
+  gcp_mysql_logs_destination_asset_id = module.gcp-pubsub-1.gcp-pubsub-asset.asset_id
+
+  instance_authorized_networks = local.gcp_mysql_instance_authorized_networks
+  instance_database_version    = "MYSQL_8_0"
+  instance_name                = local.gcp_mysql_1_instance_name
+  instance_region              = "us-west1"
+}
+
+################################################################################
+# GCP MySQL 5.7 with slow query monitoring
+#
+# Notes:
+# The Pubsub containing audit data will be associated with the GCP MySQL asset
+# via the "logs_destination_asset_id" and will get connected to gateway when the
+# database asset does.
+# 
+# The Pubsub containing slow query data will be treated as a "standalone" pubsub
+# and will get connected to gateway by setting the "audit_pull_enabled" field to
+# true directly on the asset. It utilizes the "content_type" field to indicate
+# that the data within the pubsub is coming from a GCP MySQL instance.
+################################################################################
+locals {
+  gcp_mysql_2_instance_name = "tf-mysql-5-7-slow-query"
+}
+
+module "gcp-pubsub-2-audit" {
+  source = "../../modules/onboard-gcp-pubsub"
+
+  gcp_pubsub_admin_email    = local.admin_email
+  gcp_pubsub_audit_type     = "MYSQL"
+  gcp_pubsub_auth_mechanism = local.pubsub_auth_mechanism
+  gcp_pubsub_gateway_id     = local.gateway_id
+
+  project = local.gcp_project_id
+
+  pubsub_subscription_name = "${local.gcp_mysql_2_instance_name}-audit-sub"
+
+  pubsub_topic_name = "${local.gcp_mysql_2_instance_name}-audit-topic"
+
+  sink_router_description = "MySQL 5.7 audit log sink"
+  sink_router_exclusions  = local.excluded_traffic_filter
+  sink_router_filter      = <<EOF
+    resource.type="cloudsql_database"
+    resource.labels.database_id="${local.gcp_project_id}:${local.gcp_mysql_2_instance_name}"
+    logName="projects/${local.gcp_project_id}/logs/cloudsql.googleapis.com%2Fmysql-general.log"
+  EOF
+  sink_router_name        = "${local.gcp_mysql_2_instance_name}-audit-sink"
+}
+
+module "gcp-pubsub-2-slow-query" {
+  source = "../../modules/onboard-gcp-pubsub"
+
+  gcp_pubsub_admin_email        = local.admin_email
+  gcp_pubsub_audit_pull_enabled = true
+  gcp_pubsub_audit_type         = "GCP_MYSQL_SLOW"
+  gcp_pubsub_auth_mechanism     = local.pubsub_auth_mechanism
+  gcp_pubsub_content_type       = "GCP MYSQL"
+  gcp_pubsub_gateway_id         = local.gateway_id
+
+  project = local.gcp_project_id
+
+  pubsub_subscription_name = "${local.gcp_mysql_2_instance_name}-slow-query-sub"
+
+  pubsub_topic_name = "${local.gcp_mysql_2_instance_name}-slow-query-topic"
+
+  sink_router_description = "MySQL 5.7 slow query sink"
+  sink_router_exclusions  = local.excluded_traffic_filter
+  sink_router_filter      = <<EOF
+    resource.type="cloudsql_database"
+    resource.labels.database_id="${local.gcp_project_id}:${local.gcp_mysql_2_instance_name}"
+    logName="projects/${local.gcp_project_id}/logs/cloudsql.googleapis.com%2Fmysql-slow.log"
+  EOF
+  sink_router_name        = "${local.gcp_mysql_2_instance_name}-slow-query-sink"
+}
+
+module "gcp-mysql-2" {
+  source = "../../modules/onboard-gcp-mysql"
+
+  gcp_mysql_admin_email               = local.admin_email
+  gcp_mysql_audit_pull_enabled        = true
+  gcp_mysql_gateway_id                = local.gateway_id
+  gcp_mysql_logs_destination_asset_id = module.gcp-pubsub-2-audit.gcp-pubsub-asset.asset_id
+
+  instance_authorized_networks = local.gcp_mysql_instance_authorized_networks
+  instance_database_version    = "MYSQL_5_7"
+  instance_database_flags = [
+    {
+      "name" : "log_output",
+      "value" : "FILE"
+    },
+    {
+      "name" : "general_log",
+      "value" : "on"
+    },
+    {
+      "name" : "slow_query_log",
+      "value" : "on"
+    },
+    {
+      "name" : "long_query_time",
+      "value" : "10"
+    }
+  ]
+  instance_name   = local.gcp_mysql_2_instance_name
+  instance_region = "us-west1"
+}
+
+################################################################################
+# GCP MySQL 8.0 Many-to-one
+################################################################################
+locals {
+  gcp_mysql_3_instance_name_prefix = "tf-mysql-multi"
+  gcp_mysql_3_instance_types = toset([
+    "dev",
+    "prod",
+    "uat",
+  ])
+
+  # Build instance names
+  gcp_mysql_3_instance_names = toset([
+    for i, type in local.gcp_mysql_3_instance_types : "${local.gcp_mysql_3_instance_name_prefix}-${type}"
+  ])
+
+  # Build database IDs
+  gcp_mysql_3_database_ids = [
+    for i, name in local.gcp_mysql_3_instance_names : "\"${local.gcp_project_id}:${name}\""
+  ]
+
+  # Build sink filter
+  gcp_mysql_3_filter = join(
+    "",
+    [
+      "resource.type=\"cloudsql_database\"\n",
+      "logName=\"projects/${local.gcp_project_id}/logs/cloudsql.googleapis.com%2Fmysql-general.log\"\n",
+      "resource.labels.database_id=(",
+      join(
+        " OR ",
+        local.gcp_mysql_3_database_ids
+      ),
+      ")",
+    ]
+  )
+}
+
+module "gcp-pubsub-3" {
+  source = "../../modules/onboard-gcp-pubsub"
+
+  gcp_pubsub_admin_email    = local.admin_email
+  gcp_pubsub_audit_type     = "MYSQL"
+  gcp_pubsub_auth_mechanism = local.pubsub_auth_mechanism
+  gcp_pubsub_gateway_id     = local.gateway_id
+
+  project = local.gcp_project_id
+
+  pubsub_subscription_name = "${local.gcp_mysql_3_instance_name_prefix}-sub"
+
+  pubsub_topic_name = "${local.gcp_mysql_3_instance_name_prefix}-topic"
+
+  sink_router_description = "MySQL 8.0 many-to-one sink"
+  sink_router_exclusions  = local.excluded_traffic_filter
+  sink_router_filter      = local.gcp_mysql_3_filter
+  sink_router_name        = "${local.gcp_mysql_3_instance_name_prefix}-sink"
+}
+
+module "gcp-mysql-3" {
+  source = "../../modules/onboard-gcp-mysql"
+
+  for_each = local.gcp_mysql_3_instance_names
+
+  gcp_mysql_admin_email               = local.admin_email
+  gcp_mysql_audit_pull_enabled        = true
+  gcp_mysql_gateway_id                = local.gateway_id
+  gcp_mysql_logs_destination_asset_id = module.gcp-pubsub-3.gcp-pubsub-asset.asset_id
+
+  instance_authorized_networks = local.gcp_mysql_instance_authorized_networks
+  instance_database_version    = "MYSQL_8_0"
+  instance_name                = each.key
+  instance_region              = "us-west1"
+}

modules/dsfhub-gcp-mysql/README.md

@@ -0,0 +1,41 @@
+<!-- BEGIN_TF_DOCS -->
+## Requirements
+
+No requirements.
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_dsfhub"></a> [dsfhub](#provider\_dsfhub) | n/a |
+
+## Modules
+
+No modules.
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [dsfhub_data_source.this](https://registry.terraform.io/providers/imperva/dsfhub/latest/docs/resources/data_source) | resource |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_admin_email"></a> [admin\_email](#input\_admin\_email) | The email address to notify about the asset. | `string` | n/a | yes |
+| <a name="input_asset_display_name"></a> [asset\_display\_name](#input\_asset\_display\_name) | User-friendly name of the asset, defined by user | `string` | n/a | yes |
+| <a name="input_asset_id"></a> [asset\_id](#input\_asset\_id) | Unique identifier for the MySQL instance in the form '{project-id}:{instance-region}:{instance-name}'. | `string` | n/a | yes |
+| <a name="input_audit_pull_enabled"></a> [audit\_pull\_enabled](#input\_audit\_pull\_enabled) | If true, sonargateway will collect the audit logs for this system if it can. | `bool` | `false` | no |
+| <a name="input_gateway_id"></a> [gateway\_id](#input\_gateway\_id) | Unique identifier (UID) attached to the jSonar machine controlling the asset | `string` | n/a | yes |
+| <a name="input_logs_destination_asset_id"></a> [logs\_destination\_asset\_id](#input\_logs\_destination\_asset\_id) | The asset\_id of the GCP PUSUB asset that this asset is sending its audit logs to. | `string` | `null` | no |
+| <a name="input_parent_asset_id"></a> [parent\_asset\_id](#input\_parent\_asset\_id) | The asset\_id of the GCP asset representing the GCP account where this data source is located. | `string` | `null` | no |
+| <a name="input_server_host_name"></a> [server\_host\_name](#input\_server\_host\_name) | Hostname (or IP if host is unknown) of the GCP MySQL instance | `string` | n/a | yes |
+| <a name="input_server_ip"></a> [server\_ip](#input\_server\_ip) | IP address (or hostname if IP is unknown) of the GCP MySQL instance | `string` | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| <a name="output_this"></a> [this](#output\_this) | GCP MYSQL asset |
+<!-- END_TF_DOCS -->