Skip to content
/ terraform-aws-firehose-forwarder Public

Terraform module which creates a kinesis stream and a lambda function (with access to the stream) triggered by an sns topic subscription

registry.terraform.io/modules/justtrackio/firehose-forwarder/aws

License

Apache-2.0 license
0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

justtrackio/terraform-aws-firehose-forwarder

BranchesTags

Repository files navigation

terraform-aws-firehose-forwarder

Terraform module which creates a kinesis stream and a lambda function (with access to the stream) triggered by an sns topic subscription

Requirements

Name Version
terraform >= 1.3.0
aws >= 4.67

Providers

Name Version
aws >= 4.67

Modules

Name Source Version
iam_label justtrackio/label/null 0.26.0
kinesis_stream justtrackio/kinesis/aws 1.0.2
this justtrackio/label/null 0.26.0

Resources

Name Type
aws_cloudwatch_log_group.main resource
aws_cloudwatch_log_stream.main resource
aws_iam_policy.firehose resource
aws_iam_policy.firehose_transformer resource
aws_iam_policy.kinesis_forwarder resource
aws_iam_role.firehose resource
aws_iam_role_policy_attachment.firehose_transformer resource
aws_iam_role_policy_attachment.kinesis_forwarder resource
aws_iam_role_policy_attachment.main resource
aws_kinesis_firehose_delivery_stream.main resource
aws_lambda_permission.sns resource
aws_sns_topic_subscription.kinesis_forwarder resource
aws_iam_policy_document.firehose_assume_role_policy data source
aws_iam_policy_document.firehose_policy data source
aws_iam_policy_document.firehose_transformer data source
aws_iam_policy_document.kinesis_forwarder data source

Inputs

Name Description Type Default Required
additional_tag_map Additional key-value pairs to add to each map in tags_as_list_of_maps. Not added to tags or id.
This is for some rare cases where resources want additional configuration of tags
and therefore take a list of maps with tag key, value, and additional configuration.		 map(string) {} no
alarm_enabled Whether or not to create alarms bool false no
alarm_get_records Settings for the get records alarm 
object({
    create               = optional(bool, true)
    period               = optional(number, 60)
    evaluation_periods   = optional(number, 15)
    alarm_description    = optional(string, "This metric monitors kinesis get records successful records (percent)")
    datapoints_to_alarm  = optional(number, 10)
    threshold_percentage = optional(number, 99)
  })
 
{
  "alarm_description": "This metric monitors kinesis get records successful records (percent)",
  "create": true,
  "datapoints_to_alarm": 10,
  "evaluation_periods": 15,
  "period": 60,
  "threshold_percentage": 99
}
 no
alarm_iterator_age_high Settings for the iterator age high alarm 
object({
    create              = optional(bool, true)
    period              = optional(number, 60)
    evaluation_periods  = optional(number, 15)
    alarm_description   = optional(string, "This metric monitors kinesis iterator age")
    datapoints_to_alarm = optional(number, 10)
    threshold_seconds   = optional(number, 60)
  })
 
{
  "alarm_description": "This metric monitors kinesis iterator age",
  "create": true,
  "datapoints_to_alarm": 10,
  "evaluation_periods": 15,
  "period": 60,
  "threshold_seconds": 60
}
 no
alarm_put_records Settings for the put records alarm 
object({
    create               = optional(bool, true)
    period               = optional(number, 60)
    evaluation_periods   = optional(number, 15)
    alarm_description    = optional(string, "This metric monitors kinesis put records successful records (percent)")
    datapoints_to_alarm  = optional(number, 10)
    threshold_percentage = optional(number, 99)
  })
 
{
  "alarm_description": "This metric monitors kinesis put records successful records (percent)",
  "create": true,
  "datapoints_to_alarm": 10,
  "evaluation_periods": 15,
  "period": 60,
  "threshold_percentage": 99
}
 no
alarm_read_bytes_high Settings for the read bytes high alarm 
object({
    create               = optional(bool, true)
    period               = optional(number, 60)
    evaluation_periods   = optional(number, 15)
    alarm_description    = optional(string, "This metric monitors kinesis read bytes utilization")
    datapoints_to_alarm  = optional(number, 10)
    threshold_percentage = optional(number, 80)
  })
 
{
  "alarm_description": "This metric monitors kinesis read bytes utilization",
  "create": true,
  "datapoints_to_alarm": 10,
  "evaluation_periods": 15,
  "period": 60,
  "threshold_percentage": 80
}
 no
alarm_topic_arn ARN of the AWS SNS topic for the CloudWatch alarms string null no
alarm_write_bytes_high Settings for the write bytes high alarm 
object({
    create               = optional(bool, true)
    period               = optional(number, 60)
    evaluation_periods   = optional(number, 15)
    alarm_description    = optional(string, "This metric monitors kinesis write bytes utilization")
    datapoints_to_alarm  = optional(number, 10)
    threshold_percentage = optional(number, 80)
  })
 
{
  "alarm_description": "This metric monitors kinesis write bytes utilization",
  "create": true,
  "datapoints_to_alarm": 10,
  "evaluation_periods": 15,
  "period": 60,
  "threshold_percentage": 80
}
 no
alarm_write_records_high Settings for the write records high alarm 
object({
    create               = optional(bool, true)
    period               = optional(number, 60)
    evaluation_periods   = optional(number, 15)
    alarm_description    = optional(string, "This metric monitors kinesis write records utilization")
    datapoints_to_alarm  = optional(number, 10)
    threshold_percentage = optional(number, 80)
  })
 
{
  "alarm_description": "This metric monitors kinesis write records utilization",
  "create": true,
  "datapoints_to_alarm": 10,
  "evaluation_periods": 15,
  "period": 60,
  "threshold_percentage": 80
}
 no
attributes ID element. Additional attributes (e.g. workers or cluster) to add to id,
in the order they appear in the list. New attributes are appended to the
end of the list. The elements of the list are joined by the delimiter
and treated as a single ID element.		 list(string) [] no
aws_account_id AWS account id string null no
aws_region AWS region string null no
buffer_interval (Optional) Buffer incoming data for the specified period of time, in seconds, before delivering it to the destination. The default value is 300. number 300 no
context Single object for setting entire context at once.
See description of individual variables for details.
Leave string and numeric variables as null to use default value.
Individual variable settings (non-null) override settings in context object,
except for attributes, tags, and additional_tag_map, which are merged.		 any 
{
  "additional_tag_map": {},
  "attributes": [],
  "delimiter": null,
  "descriptor_formats": {},
  "enabled": true,
  "environment": null,
  "id_length_limit": null,
  "label_key_case": null,
  "label_order": [],
  "label_value_case": null,
  "labels_as_tags": [
    "unset"
  ],
  "name": null,
  "namespace": null,
  "regex_replace_chars": null,
  "stage": null,
  "tags": {},
  "tenant": null
}
 no
delimiter Delimiter to be used between ID elements.
Defaults to - (hyphen). Set to "" to use no delimiter at all.		 string null no
descriptor_formats Describe additional descriptors to be output in the descriptors output map.
Map of maps. Keys are names of descriptors. Values are maps of the form
{<br> format = string<br> labels = list(string)<br>}
(Type is any so the map values can later be enhanced to provide additional options.)
format is a Terraform format string to be passed to the format() function.
labels is a list of labels, in order, to pass to format() function.
Label values will be normalized before being passed to format() so they will be
identical to how they appear in id.
Default is {} (descriptors output will be empty).		 any {} no
dynamic_partitioning_enabled Whether or not to use dynamic partitioning bool false no
dynamic_partitioning_lambda_transformer_function_arn ARN of the Lambda function to transform the data for dynamic partitioning string null no
dynamic_partitioning_lambda_transformer_function_name Name of the Lambda function to transform the data for dynamic partitioning string null no
dynamic_partitioning_processor_parameters Set of parameter for the dynamic partitioning processor 
set(object({
    parameter_name  = string
    parameter_value = any
  }))
 [] no
enabled Set to false to prevent the module from creating any resources bool null no
environment ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT' string null no
glue_database_name Name of the Glue database to be used for your schema configuration string n/a yes
glue_table_name Name of the Glue table to be used for your schema configuration string n/a yes
id_length_limit Limit id to this many characters (minimum 6).
Set to 0 for unlimited length.
Set to null for keep the existing setting, which defaults to 0.
Does not affect id_full.		 number null no
kinesis_forwarder_enabled Whether to subscribe to an sns topic to allow a kinesis forwarder lambda to put data into the stream bool true no
kinesis_forwarder_lambda_function_arn ARN of the Lambda function to be used to forward data to the firehose stream string null no
kinesis_forwarder_lambda_function_name Name of the Lambda function to be used to forward data to the firehose stream string null no
kinesis_forwarder_source_arn ARN of the SNS topic to be used as data source by your lambda function string null no
kinesis_retention_period How many hours to keep data of the stream number 24 no
kinesis_shard_count How many shards to create for the input kinesis data stream number 1 no
label_key_case Controls the letter case of the tags keys (label names) for tags generated by this module.
Does not affect keys of tags passed in via the tags input.
Possible values: lower, title, upper.
Default value: title.		 string null no
label_order The order in which the labels (ID elements) appear in the id.
Defaults to ["namespace", "environment", "stage", "name", "attributes"].
You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.		 list(string) null no
label_orders Overrides the labels_order for the different labels to modify ID elements appear in the id 
object({
    cloudwatch = optional(list(string)),
    iam        = optional(list(string)),
    kinesis    = optional(list(string)),
  })
 {} no
label_value_case Controls the letter case of ID elements (labels) as included in id,
set as tag values, and output by this module individually.
Does not affect values of tags passed in via the tags input.
Possible values: lower, title, upper and none (no transformation).
Set this to title and set delimiter to "" to yield Pascal Case IDs.
Default value: lower.		 string null no
labels_as_tags Set of labels (ID elements) to include as tags in the tags output.
Default is to include all labels.
Tags with empty values will not be included in the tags output.
Set to [] to suppress all generated tags.
Notes:
The value of the name tag, if included, will be the id, not the name.
Unlike other null-label inputs, the initial setting of labels_as_tags cannot be
changed in later chained modules. Attempts to change it will be silently ignored.		 set(string) 
[
  "default"
]
 no
log_group_retention_in_days (Optional) Specifies the number of days you want to retain log events in the specified log group. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653, and 0. If you select 0, the events in the log group are always retained and never expire. Defaults to 30 Days. number 30 no
name ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'.
This is the only ID element not also included as a tag.
The "name" tag is set to the full id string. There is no tag with the value of the name input.		 string null no
namespace ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique string null no
organizational_unit Usually used to indicate the AWS organizational unit, e.g. 'prod', 'sdlc' string null no
regex_replace_chars Terraform regular expression (regex) string.
Characters matching the regex will be removed from the ID elements.
If not set, "/[^a-zA-Z0-9-]/" is used to remove all characters other than hyphens, letters and digits.		 string null no
s3_bucket_arn ARN of the S3 Bucket to put the kinesis datastream data into string n/a yes
s3_model_suffix Suffix to use for datalake key string "" no
s3_write_temp_year_partition Whether or not to write data to a temporary Year 9999 partition bool false no
stage ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release' string null no
tags Additional tags (e.g. {'BusinessUnit': 'XYZ'}).
Neither the tag keys nor the tag values will be modified by this module.		 map(string) {} no
tenant ID element _(Rarely used, not included by default)_. A customer identifier, indicating who this instance of a resource is for string null no

Outputs

Name Description
kinesis_firehose_delivery_stream_arn ARN for the created kinesis firehose delivery stream
kinesis_stream_arn ARN for the created kinesis stream

About

Terraform module which creates a kinesis stream and a lambda function (with access to the stream) triggered by an sns topic subscription

registry.terraform.io/modules/justtrackio/firehose-forwarder/aws

Topics

aws lambda kinesis forwarder

Resources

Readme

License

Apache-2.0 license

Code of conduct

Code of conduct

Security policy

Security policy
Activity
Custom properties

Stars

0 stars

Watchers

5 watching

Forks

0 forks
Report repository

Releases 5

v1.2.1 Latest
May 22, 2023
+ 4 releases

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages