feat: firehose metrics - include linked accounts metrics (#183)

* wrench: include_linked_accounts_metrics

* 🖊️ changelog updated

* 🖊️ version updated

---------

Co-authored-by: jakeskyaws <jakesky@amazon.com>

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## v2.1.0
+### 💡 Enhancements 
+- Added an option to include metrics from source accounts linked to the monitoring account in the Firehose CloudWatch metric stream.
+- Introduced the `include_linked_accounts_metrics` variable to control the inclusion of linked account metrics for Firehose.
+- Updated example configurations to demonstrate usage of the `include_linked_accounts_metrics` variable in Firehose metric streams.
+
 ## v2.0.1
 ### 🧰 Bug fixes 🧰
 - Fixed ecs-ec2 module, adjusted cdot image command to `--config env:OTEL_CONFIG`

examples/firehose-metrics/README.md

@@ -92,5 +92,6 @@ Since the private_key is private and we cant put it hardcoded, it can be exporte
 export TF_VAR_private_key="your-coralogix-private-key"
 * `include_metric_stream_namespaces` --> The list of the the desired namespaces, for example: ["EC2", "DynamoDB"]. For the full list of the available namespaces and how they need to be mentioned, please see [namespaces list](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html)'
 * `include_metric_stream_filter` --> List of inclusive metric filters for namespace and metric_names. For the full list of the available namespaces, please see [namespaces list](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html). To view available metric names of selected namespace, please see [view available metric names](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html)
+* `include_linked_accounts_metrics` --> Include metrics from source accounts that are linked to this monitoring account, please see [CloudWatch cross-account observability](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html).
 * `additional_metric_statistics_enable` --> Enable additional metric statistics for CloudWatch metric streams
 * `additional_metric_statistics` --> List of additional metric statistics for namespace, metric_name and additional_statistics. For the full list of the available namespaces, please see [namespaces list](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html). To view available metric names of selected namespace, please see [view available metric names](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html). For the full list of the available additional statistics, please see [statistics that can be streamed](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-statistics.html)

examples/firehose-metrics/main.tf

@@ -6,6 +6,7 @@ module "cloudwatch_firehose_metrics_coralogix" {
   integration_type_metrics            = var.integration_type_metrics
   include_metric_stream_namespaces    = var.include_metric_stream_namespaces
   include_metric_stream_filter        = var.include_metric_stream_filter
+  include_linked_accounts_metrics     = var.include_linked_accounts_metrics 
   additional_metric_statistics_enable = var.additional_metric_statistics_enable
   additional_metric_statistics        = var.additional_metric_statistics
   output_format                       = var.output_format

examples/firehose-metrics/variables.tf

@@ -52,6 +52,12 @@ variable "include_metric_stream_filter" {
   ]
 }
 
+variable "include_linked_accounts_metrics" {
+  description = "include_linked_accounts_metrics (Optional) If you are creating a metric stream in a monitoring account, specify true to include metrics from source accounts that are linked to this monitoring account, in the metric stream. The default is false."
+  type        = bool
+  default     = false
+}
+
 variable "additional_metric_statistics_enable" {
   description = "To enable the inclusion of additional statistics to the streaming metrics"
   type        = bool

modules/firehose-metrics/README.md

@@ -206,20 +206,21 @@ then the CloudWatch metric stream must be configured with the same format, confi
 | <a name="input_cloudwatch_metric_stream_custom_name"></a> [cloudwatch\_metric\_stream\_custom\_name](variables.tf#L63) | Set the name of the CloudWatch Metric Stream resource, otherwise variable firehose_stream will be used | `string` | `null` | no |
 | <a name="input_include_metric_stream_namespaces"></a> [include\_metric\_stream\_namespaces](variables.tf#L69) | List of specific namespaces to include in the CloudWatch metric stream, see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html | `list(string)` | `[]` | no |
 | <a name="input_include_metric_stream_filter"></a> [include\_metric\_stream\_filter](variables.tf#L75) | Guide to view specific metric names of namespaces, see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html | `list(object({namespace=string, metric_names=list(string)})` | `[]` | no |
-| <a name="input_additional_metric_statistics_enable"></a> [additional\_metric\_statistics\_enable](variables.tf#L85) | To enable the inclusion of additional statistics to the streaming metrics | `bool` | `true` | no |
-| <a name="input_additional_metric_statistics"></a> [additional\_metric\_statistics](variables.tf#L91) | For each entry, specify one or more metrics (metric_name and namespace) and the list of additional statistics to stream for those metrics. Each configuration of metric name and namespace can have a list of additional_statistics included into the AWS CloudWatch Metric Stream. | `list(object({additional_statistics=list(string), metric_name=string, namespace=string}))` | See variables.tf | no |
-| <a name="input_s3_backup_custom_name"></a> [s3\_backup\_custom\_name](variables.tf#L137) | Set the name of the S3 backup bucket, otherwise variable '{firehose_stream}-backup-metrics' will be used. | `string` | n/a | no |
-| <a name="input_existing_s3_backup"></a> [existing\_s3\_backup](variables.tf#L143) | Use an existing S3 bucket to use as a backup bucket. | `string` | n/a | no |
-| <a name="input_lambda_processor_enable"></a> [lambda\_processor\_enable](variables.tf#L149) | Enable the lambda processor function. Set to false to remove the lambda and all associated resources. | `bool` | `true` | no |
-| <a name="input_lambda_processor_custom_name"></a> [lambda\_processor\_custom\_name](variables.tf#L155) | Set the name of the lambda processor function, otherwise variable '{firehose_stream}-metrics-transform' will be used | `string` | `null` | no |
-| <a name="input_lambda_processor_iam_custom_name"></a> [lambda\_processor\_iam\_custom\_name](variables.tf#L161) | Set the name of the lambda processor IAM role, otherwise variable '{firehose_stream}-lambda-processor-iam' will be used. | `string` | n/a | no |
-| <a name="input_existing_lambda_processor_iam"></a> [existing\_lambda\_processor\_iam](variables.tf#L167) | Use an existing lambda processor IAM role. | `string` | n/a | no |
-| <a name="input_firehose_iam_custom_name"></a> [firehose\_iam\_custom\_name](variables.tf#L173) | Set the name of the IAM role & policy, otherwise variable '{firehose_stream}-firehose-metrics-iam' will be used. | `string` | n/a | no |
-| <a name="input_existing_firehose_iam"></a> [existing\_firehose\_iam](variables.tf#L179) | Use an existing IAM role to use as a firehose role. | `string` | n/a | no |
-| <a name="input_metric_streams_iam_custom_name"></a> [metric\_streams\_iam\_custom\_name](variables.tf#L185) | Set the name of the cloudwatch metric streams IAM role & policy, otherwise variable '{firehose_stream}-cw-iam. | `string` | n/a | no |
-| <a name="input_existing_metric_streams_iam"></a> [existing\_metric\_streams\_iam](variables.tf#L191) | Use an existing IAM role to use as a metric streams role. | `string` | n/a | no |
-| <a name="input_user_supplied_tags"></a> [user\_supplied\_tags](variables.tf#L197) | Tags supplied by the user to populate to all generated resources | `map(string)` | n/a | no |
-| <a name="input_override_default_tags"></a> [override\_default\_tags](variables.tf#L203) | Override and remove the default tags by setting to true | `bool` | `false` | no |
+| <a name="include_linked_accounts_metrics"></a> [include\_linked\_accounts\_metric](variables.tf#L85) | Guide to view specific metric names of namespaces, see https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/viewing_metrics_with_cloudwatch.html | `list(object({namespace=string, metric_names=list(string)})` | `[]` | no |
+| <a name="input_additional_metric_statistics_enable"></a> [additional\_metric\_statistics\_enable](variables.tf#L91) | To enable the inclusion of additional statistics to the streaming metrics | `bool` | `true` | no |
+| <a name="input_additional_metric_statistics"></a> [additional\_metric\_statistics](variables.tf#L97) | For each entry, specify one or more metrics (metric_name and namespace) and the list of additional statistics to stream for those metrics. Each configuration of metric name and namespace can have a list of additional_statistics included into the AWS CloudWatch Metric Stream. | `list(object({additional_statistics=list(string), metric_name=string, namespace=string}))` | See variables.tf | no |
+| <a name="input_s3_backup_custom_name"></a> [s3\_backup\_custom\_name](variables.tf#L143) | Set the name of the S3 backup bucket, otherwise variable '{firehose_stream}-backup-metrics' will be used. | `string` | n/a | no |
+| <a name="input_existing_s3_backup"></a> [existing\_s3\_backup](variables.tf#L149) | Use an existing S3 bucket to use as a backup bucket. | `string` | n/a | no |
+| <a name="input_lambda_processor_enable"></a> [lambda\_processor\_enable](variables.tf#L155) | Enable the lambda processor function. Set to false to remove the lambda and all associated resources. | `bool` | `true` | no |
+| <a name="input_lambda_processor_custom_name"></a> [lambda\_processor\_custom\_name](variables.tf#L161) | Set the name of the lambda processor function, otherwise variable '{firehose_stream}-metrics-transform' will be used | `string` | `null` | no |
+| <a name="input_lambda_processor_iam_custom_name"></a> [lambda\_processor\_iam\_custom\_name](variables.tf#L167) | Set the name of the lambda processor IAM role, otherwise variable '{firehose_stream}-lambda-processor-iam' will be used. | `string` | n/a | no |
+| <a name="input_existing_lambda_processor_iam"></a> [existing\_lambda\_processor\_iam](variables.tf#L173) | Use an existing lambda processor IAM role. | `string` | n/a | no |
+| <a name="input_firehose_iam_custom_name"></a> [firehose\_iam\_custom\_name](variables.tf#L179) | Set the name of the IAM role & policy, otherwise variable '{firehose_stream}-firehose-metrics-iam' will be used. | `string` | n/a | no |
+| <a name="input_existing_firehose_iam"></a> [existing\_firehose\_iam](variables.tf#L185) | Use an existing IAM role to use as a firehose role. | `string` | n/a | no |
+| <a name="input_metric_streams_iam_custom_name"></a> [metric\_streams\_iam\_custom\_name](variables.tf#L191) | Set the name of the cloudwatch metric streams IAM role & policy, otherwise variable '{firehose_stream}-cw-iam. | `string` | n/a | no |
+| <a name="input_existing_metric_streams_iam"></a> [existing\_metric\_streams\_iam](variables.tf#L197) | Use an existing IAM role to use as a metric streams role. | `string` | n/a | no |
+| <a name="input_user_supplied_tags"></a> [user\_supplied\_tags](variables.tf#L203) | Tags supplied by the user to populate to all generated resources | `map(string)` | n/a | no |
+| <a name="input_override_default_tags"></a> [override\_default\_tags](variables.tf#L209) | Override and remove the default tags by setting to true | `bool` | `false` | no |
 
 ## Coralgoix regions
 

modules/firehose-metrics/main.tf

@@ -443,6 +443,7 @@ resource "aws_cloudwatch_metric_stream" "cloudwatch_metric_stream" {
   name          = local.cloud_watch_metric_stream_name
   role_arn      = local.metrics_stream_iam_role_arn
   firehose_arn  = aws_kinesis_firehose_delivery_stream.coralogix_stream_metrics.arn
+  include_linked_accounts_metrics = var.include_linked_accounts_metrics
   output_format = var.output_format
 
   dynamic "include_filter" {

modules/firehose-metrics/variables.tf

@@ -82,6 +82,12 @@ variable "include_metric_stream_filter" {
   default = []
 }
 
+  variable "include_linked_accounts_metrics" {
+  description = "include_linked_accounts_metrics (Optional) If you are creating a metric stream in a monitoring account, specify true to include metrics from source accounts that are linked to this monitoring account, in the metric stream. The default is false."
+  type        = bool
+  default     = false
+}
+
 variable "additional_metric_statistics_enable" {
   description = "To enable the inclusion of additional statistics to the streaming metrics"
   type        = bool