Skip to content

Azure SQL Server: split eventhub creation into separate module #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Sep 14, 2024
Merged

Azure SQL Server: split eventhub creation into separate module #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
af96144
split eventhub creation into separate module
mattJsonar Sep 13, 2024
b475bb4
Apply automatic changes
mattJsonar Sep 13, 2024
060b1ed
terraform-docs: automated action
github-actions[bot] Sep 13, 2024
271d0e5
resolve comments
mattJsonar Sep 13, 2024
13eb1ea
wording
mattJsonar Sep 13, 2024
e550e15
Apply automatic changes
mattJsonar Sep 13, 2024
4dce236
terraform-docs: automated action
github-actions[bot] Sep 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions examples/onboard-azure-ms-sql-server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ This example creates both 'azurerm' and 'dsfhub' resources. More information reg
- [dsfhub](https://registry.terraform.io/providers/imperva/dsfhub/latest/docs)

## Prerequisites
Both of the following prerequisites are handled within the ``onboard-azure-eventhub`` module.

### Azure Event Hub Namespace and Event Hub
SQL Server audit logs are sent to an Azure Event Hub and are retrieved by DSF. The Event Hubs are created inside of an Event Hub Namespace, which can contain one or more Event Hubs. Audit logs of multiple SQL Server instances can be sent to a single Event Hub. This module expects an Event Hub to have been created in advance, and will create a corresponding AZURE EVENTHUB asset for it.
SQL Server audit logs are sent to an Azure Event Hub and are retrieved by DSF. The Event Hubs are created within an Event Hub Namespace, which can contain one or more Event Hubs. Audit logs of multiple SQL Server instances can be sent to a single Event Hub. This module expects an Event Hub to have been created in advance, and with a corresponding AZURE EVENTHUB asset onboarded for it.

### Azure Storage Account and Container
Storage Containers are used to store transactional data for the Event Hub import processes, and one Storage Container is required for each Event Hub. These Storage Containers exists inside a Storage Account, which may contain multiple Storage Containers.
Expand All @@ -26,12 +28,8 @@ No providers.
| Name | Source | Version |
|------|--------|---------|
| <a name="module_azure-ms-sql-server-1"></a> [azure-ms-sql-server-1](#module\_azure-ms-sql-server-1) | ../../modules/onboard-azure-ms-sql-server | n/a |
| <a name="module_eventhub"></a> [eventhub](#module\_eventhub) | ../../modules/azurerm-eventhub | n/a |
| <a name="module_eventhub-namespace"></a> [eventhub-namespace](#module\_eventhub-namespace) | ../../modules/azurerm-eventhub-namespace | n/a |
| <a name="module_eventhub-read-authorization"></a> [eventhub-read-authorization](#module\_eventhub-read-authorization) | ../../modules/azurerm-eventhub-namespace-authorization-rule | n/a |
| <a name="module_eventhub-write-authorization"></a> [eventhub-write-authorization](#module\_eventhub-write-authorization) | ../../modules/azurerm-eventhub-namespace-authorization-rule | n/a |
| <a name="module_storage-account"></a> [storage-account](#module\_storage-account) | ../../modules/azurerm-storage-account | n/a |
| <a name="module_storage-container"></a> [storage-container](#module\_storage-container) | ../../modules/azurerm-storage-container | n/a |
| <a name="module_azure-ms-sql-server-2"></a> [azure-ms-sql-server-2](#module\_azure-ms-sql-server-2) | ../../modules/onboard-azure-ms-sql-server | n/a |
| <a name="module_onboard-azure-sql-server-eventhub-1"></a> [onboard-azure-sql-server-eventhub-1](#module\_onboard-azure-sql-server-eventhub-1) | ../../modules/onboard-azure-eventhub | n/a |

## Resources

Expand Down
125 changes: 57 additions & 68 deletions examples/onboard-azure-ms-sql-server/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
locals {
azure_eventhub_name = "sqlservereventhub"
azure_location = "East US"
azure_resource_group_name = "My_Resource_Group"
azure_subscription_id = "123456790-wxyz-g8h9-e5f6-a1b2c3d4"
Expand Down Expand Up @@ -38,94 +37,84 @@ provider "dsfhub" {
# reading and writing to the Event Hub.
# 2. Storage Account and Container
################################################################################
# 1. Azure Event Hub Namespace and Event Hub
module "eventhub-namespace" {
source = "../../modules/azurerm-eventhub-namespace"
module "onboard-azure-sql-server-eventhub-1" {
source = "../../modules/onboard-azure-eventhub"

location = local.azure_location
name = "${local.azure_eventhub_name}ns"
resource_group_name = local.azure_resource_group_name
}

module "eventhub" {
source = "../../modules/azurerm-eventhub"
azure_eventhub_admin_email = local.admin_email
azure_eventhub_format = "Sql"
azure_eventhub_gateway_id = local.gateway_id

name = local.azure_eventhub_name
namespace_name = module.eventhub-namespace.this.name
resource_group_name = local.azure_resource_group_name
}
eventhub_name = "sqlservereventhub"
eventhub_namespace_location = local.azure_location
eventhub_namespace_name = "sqlservereventhubns"
eventhub_namespace_resource_group_name = local.azure_resource_group_name

module "eventhub-write-authorization" {
source = "../../modules/azurerm-eventhub-namespace-authorization-rule"
eventhub_resource_group_name = local.azure_resource_group_name

listen = false
manage = false
name = "${local.azure_eventhub_name}write"
namespace_name = module.eventhub-namespace.this.name
resource_group_name = local.azure_resource_group_name
send = true
storage_account_location = local.azure_location
storage_account_name = "sqlserverstorageacc"
storage_account_resource_group_name = local.azure_resource_group_name
storage_container_name = "sqlserverstoragecon"
}

module "eventhub-read-authorization" {
source = "../../modules/azurerm-eventhub-namespace-authorization-rule"

listen = true
manage = false
name = "${local.azure_eventhub_name}read"
namespace_name = module.eventhub-namespace.this.name
resource_group_name = local.azure_resource_group_name
send = false
}
################################################################################
# Azure SQL Server
################################################################################
module "azure-ms-sql-server-1" {
source = "../../modules/onboard-azure-ms-sql-server"

# 2. Storage Account and Container
module "storage-account" {
source = "../../modules/azurerm-storage-account"
depends_on = [module.onboard-azure-sql-server-eventhub-1]

location = local.azure_location
name = "sqlserverstorageacc"
resource_group_name = local.azure_resource_group_name
}
azure_ms_sql_server_admin_email = local.admin_email
azure_ms_sql_server_audit_pull_enabled = true
azure_ms_sql_server_gateway_id = local.gateway_id
azure_ms_sql_server_location = local.azure_location
azure_ms_sql_server_logs_destination_asset_id = module.onboard-azure-sql-server-eventhub-1.azure-eventhub-asset.asset_id

module "storage-container" {
source = "../../modules/azurerm-storage-container"
diagnostic_setting_eventhub_authorization_rule_id = module.onboard-azure-sql-server-eventhub-1.eventhub-write-authorization.id
diagnostic_setting_eventhub_name = module.onboard-azure-sql-server-eventhub-1.eventhub.name
diagnostic_setting_name = "dsfhubdiagnostic"

name = "sqlserverstoragecon"
storage_account_name = module.storage-account.this.name
server_administrator_login = "exampleadmin"
server_administrator_login_password = "Abcd1234"
server_location = local.azure_location
server_name = "example-azure-sql-server"
server_public_network_access_enabled = true
server_resource_group_name = local.azure_resource_group_name
}

################################################################################
# Azure SQL Server
# Azure SQL Server Many-to-One
################################################################################
module "azure-ms-sql-server-1" {
locals {
sql_server_types = toset([
"dev",
"prod",
"uat"
])
}

module "azure-ms-sql-server-2" {
source = "../../modules/onboard-azure-ms-sql-server"

azure_eventhub_admin_email = local.admin_email
azure_eventhub_asset_display_name = module.eventhub.this.name
azure_eventhub_asset_id = module.eventhub.this.id
azure_eventhub_audit_pull_enabled = true
azure_eventhub_azure_storage_account = module.storage-account.this.name
azure_eventhub_azure_storage_container = module.storage-container.this.name
azure_eventhub_azure_storage_secret_key = module.storage-account.this.primary_access_key
azure_eventhub_eventhub_access_key = module.eventhub-read-authorization.this.primary_key
azure_eventhub_eventhub_access_policy = module.eventhub-read-authorization.this.name
azure_eventhub_eventhub_name = module.eventhub.this.name
azure_eventhub_eventhub_namespace = module.eventhub.this.namespace_name
azure_eventhub_gateway_id = local.gateway_id
azure_eventhub_reason = "default"

azure_ms_sql_server_admin_email = local.admin_email
azure_ms_sql_server_audit_pull_enabled = true
azure_ms_sql_server_gateway_id = local.gateway_id
azure_ms_sql_server_location = local.azure_location

diagnostic_setting_eventhub_authorization_rule_id = module.eventhub-write-authorization.this.id
diagnostic_setting_eventhub_name = module.eventhub.this.name
depends_on = [module.onboard-azure-sql-server-eventhub-1]

for_each = local.sql_server_types

azure_ms_sql_server_admin_email = local.admin_email
azure_ms_sql_server_audit_pull_enabled = true
azure_ms_sql_server_gateway_id = local.gateway_id
azure_ms_sql_server_location = local.azure_location
azure_ms_sql_server_logs_destination_asset_id = module.onboard-azure-sql-server-eventhub-1.azure-eventhub-asset.asset_id

diagnostic_setting_eventhub_authorization_rule_id = module.onboard-azure-sql-server-eventhub-1.eventhub-write-authorization.id
diagnostic_setting_eventhub_name = module.onboard-azure-sql-server-eventhub-1.eventhub.name
diagnostic_setting_name = "dsfhubdiagnostic"

server_administrator_login = "exampleadmin"
server_administrator_login_password = "Abcd1234"
server_location = local.azure_location
server_name = "example-azure-sql-server"
server_name = "example-azure-sql-server-${each.key}"
server_public_network_access_enabled = true
server_resource_group_name = local.azure_resource_group_name
}
68 changes: 68 additions & 0 deletions modules/onboard-azure-eventhub/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# onboard-azure-eventhub
Creates and onboards an Azure Event Hub to DSF Hub, alongside creating the Storage Account used by DSF to store a marker when pulling data from the Event Hub.


<!-- BEGIN_TF_DOCS -->
## Requirements

No requirements.

## Providers

No providers.

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_azure-eventhub-asset"></a> [azure-eventhub-asset](#module\_azure-eventhub-asset) | ../dsfhub-azure-eventhub | n/a |
| <a name="module_eventhub"></a> [eventhub](#module\_eventhub) | ../azurerm-eventhub | n/a |
| <a name="module_eventhub-authorizations"></a> [eventhub-authorizations](#module\_eventhub-authorizations) | ../azurerm-eventhub-namespace-authorization-rule | n/a |
| <a name="module_eventhub-namespace"></a> [eventhub-namespace](#module\_eventhub-namespace) | ../azurerm-eventhub-namespace | n/a |
| <a name="module_storage-account"></a> [storage-account](#module\_storage-account) | ../azurerm-storage-account | n/a |
| <a name="module_storage-container"></a> [storage-container](#module\_storage-container) | ../azurerm-storage-container | n/a |

## Resources

No resources.

## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| <a name="input_azure_eventhub_admin_email"></a> [azure\_eventhub\_admin\_email](#input\_azure\_eventhub\_admin\_email) | The email address to notify about the asset. | `string` | n/a | yes |
| <a name="input_azure_eventhub_format"></a> [azure\_eventhub\_format](#input\_azure\_eventhub\_format) | The type of audit data being sent to the Event Hub. Possible values are: AzureSQL\_Managed, Blob, Cosmos\_Mongo, Cosmos\_SQL, Data\_Explorer, Databricks\_Workspace, File, Mariadb, Mysql, Postgresql, Postgresql\_Flexible, Queue, Sql, Synapse, Table. Defaults to Sql. | `string` | `"Sql"` | no |
| <a name="input_azure_eventhub_gateway_id"></a> [azure\_eventhub\_gateway\_id](#input\_azure\_eventhub\_gateway\_id) | Unique identifier (UID) attached to the jSonar machine controlling the asset | `string` | n/a | yes |
| <a name="input_azure_eventhub_parent_asset_id"></a> [azure\_eventhub\_parent\_asset\_id](#input\_azure\_eventhub\_parent\_asset\_id) | The asset\_id of the azure asset that is sending its audit logs to this AZURE EVENTHUB asset. | `string` | `null` | no |
| <a name="input_azure_eventhub_region"></a> [azure\_eventhub\_region](#input\_azure\_eventhub\_region) | Azure region containing the Event Hub. | `string` | `null` | no |
| <a name="input_eventhub_message_retention"></a> [eventhub\_message\_retention](#input\_eventhub\_message\_retention) | Specifies the number of days to retain the events for this Event Hub. Maximum value is 7 days. Defaults to 1. | `number` | `1` | no |
| <a name="input_eventhub_name"></a> [eventhub\_name](#input\_eventhub\_name) | Specifies the name of the Event Hub resource. Changing this forces a new resource to be created. | `string` | n/a | yes |
| <a name="input_eventhub_namespace_capacity"></a> [eventhub\_namespace\_capacity](#input\_eventhub\_namespace\_capacity) | Specifies the Capacity / Throughput Units for a Standard SKU namespace. Defaults to 1. | `number` | `1` | no |
| <a name="input_eventhub_namespace_location"></a> [eventhub\_namespace\_location](#input\_eventhub\_namespace\_location) | Specifies the supported Azure location where the resource exists. Changing this forces a new resource to be created. | `string` | n/a | yes |
| <a name="input_eventhub_namespace_name"></a> [eventhub\_namespace\_name](#input\_eventhub\_namespace\_name) | Specifies the name of the Event Hub Namespace resource. Changing this forces a new resource to be created. | `string` | n/a | yes |
| <a name="input_eventhub_namespace_public_network_access_enabled"></a> [eventhub\_namespace\_public\_network\_access\_enabled](#input\_eventhub\_namespace\_public\_network\_access\_enabled) | Is public network access enabled for the Event Hub Namespace? Defaults to true. | `bool` | `true` | no |
| <a name="input_eventhub_namespace_resource_group_name"></a> [eventhub\_namespace\_resource\_group\_name](#input\_eventhub\_namespace\_resource\_group\_name) | The name of the resource group in which to create the namespace. Changing this forces a new resource to be created. | `string` | n/a | yes |
| <a name="input_eventhub_namespace_sku"></a> [eventhub\_namespace\_sku](#input\_eventhub\_namespace\_sku) | Defines which tier to use. Valid options are Basic, Standard, and Premium. Please note that setting this field to Premium will force the creation of a new resource. Defaults to Basic. | `string` | `"Basic"` | no |
| <a name="input_eventhub_namespace_tags"></a> [eventhub\_namespace\_tags](#input\_eventhub\_namespace\_tags) | A mapping of tags to assign to the resource. | `map(string)` | `null` | no |
| <a name="input_eventhub_partition_count"></a> [eventhub\_partition\_count](#input\_eventhub\_partition\_count) | Specifies the current number of shards on the Event Hub. Note: partition\_count cannot be changed unless Eventhub Namespace SKU is Premium and cannot be decreased. Maximum value is 32. Defaults to 1. | `number` | `1` | no |
| <a name="input_eventhub_resource_group_name"></a> [eventhub\_resource\_group\_name](#input\_eventhub\_resource\_group\_name) | The name of the resource group in which the Event Hub's parent Namespace exists. Changing this forces a new resource to be created. | `string` | n/a | yes |
| <a name="input_eventhub_status"></a> [eventhub\_status](#input\_eventhub\_status) | Specifies the status of the Event Hub resource. Possible values are Active, Disabled and SendDisabled. Defaults to Active. | `string` | `"Active"` | no |
| <a name="input_storage_account_location"></a> [storage\_account\_location](#input\_storage\_account\_location) | Specifies the supported Azure location where the resource exists. Changing this forces a new resource to be created. | `string` | n/a | yes |
| <a name="input_storage_account_name"></a> [storage\_account\_name](#input\_storage\_account\_name) | Specifies the name of the storage account. Only lowercase Alphanumeric characters allowed. Changing this forces a new resource to be created. This must be unique across the entire Azure service, not just within the resource group. | `string` | n/a | yes |
| <a name="input_storage_account_replication_type"></a> [storage\_account\_replication\_type](#input\_storage\_account\_replication\_type) | Defines the type of replication to use for this storage account. Valid options are LRS, GRS, RAGRS, ZRS, GZRS and RAGZRS. Changing this forces a new resource to be created when types LRS, GRS and RAGRS are changed to ZRS, GZRS or RAGZRS and vice versa. Defaults to GRS. | `string` | `"GRS"` | no |
| <a name="input_storage_account_resource_group_name"></a> [storage\_account\_resource\_group\_name](#input\_storage\_account\_resource\_group\_name) | The name of the resource group in which to create the storage account. Changing this forces a new resource to be created. | `string` | n/a | yes |
| <a name="input_storage_account_tier"></a> [storage\_account\_tier](#input\_storage\_account\_tier) | Defines the Tier to use for this storage account. Valid options are Standard and Premium. | `string` | `"Standard"` | no |
| <a name="input_storage_container_name"></a> [storage\_container\_name](#input\_storage\_container\_name) | The name of the Container which should be created within the Storage Account. Changing this forces a new resource to be created. | `string` | n/a | yes |

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_azure-eventhub-asset"></a> [azure-eventhub-asset](#output\_azure-eventhub-asset) | AZURE EVENTHUB asset. |
| <a name="output_eventhub"></a> [eventhub](#output\_eventhub) | Azure Event Hub. |
| <a name="output_eventhub-namespace"></a> [eventhub-namespace](#output\_eventhub-namespace) | Azure Event Hub Namespace. |
| <a name="output_eventhub-read-authorization"></a> [eventhub-read-authorization](#output\_eventhub-read-authorization) | Read authorization for the Event Hub Namespace. |
| <a name="output_eventhub-write-authorization"></a> [eventhub-write-authorization](#output\_eventhub-write-authorization) | Write authorization for the Event Hub Namespace. |
| <a name="output_storage-account"></a> [storage-account](#output\_storage-account) | Azure Storage Account. |
| <a name="output_storage-container"></a> [storage-container](#output\_storage-container) | Azure Storage Container. |
<!-- END_TF_DOCS -->
Loading