Merge branch 'release/4.13.0'

CHANGELOG.md

@@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## Unreleased
 
+## 4.13.0 - 2020-03-26
+- Add: variables `cache_lifecycle_prefix` and `cache_lifecycle_clear` to increase flexibility of the cache usages.
+- Add: Parametrize the AWS ARN for policies (#203) @ericamador
+- Add: Allow ping to runners and agent from cidr range (#201 @fliphess
+- Change: Refactor templatefile (#199) 
+- Change: Types of `runners_volumes_tmpfs`, and  `runners_services_volumes_tmpfs` are changed, check README or default example for details.
+
 ## 4.12.0 - 2020-03-16
 - Add: Option for permissions boundary (#195) @mhulscher
 - Bugfix: Fix cancel spot instance script from destroy provisioners are deprecate

README.md

@@ -2,7 +2,7 @@
 
 # Terraform module for GitLab auto scaling runners on AWS spot instances
 
-> *NEW*: Terraform 0.12 is supported.
+> "Type changes": The types of variable `runners_volumes_tmpfs`, and `runners_services_volumes_tmpfs` are changed to support the Terraform 12 `templatefile` function. Check the [default example](examples/runner-pre-registered/main.tf) for an usages example.
 
 ## Terraform versions
 
@@ -238,7 +238,6 @@ terraform destroy
 |------|---------|
 | aws | n/a |
 | null | n/a |
-| template | n/a |
 
 ## Inputs
 
@@ -248,6 +247,7 @@ terraform destroy
 | allow\_iam\_service\_linked\_role\_creation | Boolean used to control attaching the policy to a runner instance to create service linked roles. | `bool` | `true` | no |
 | ami\_filter | List of maps used to create the AMI filter for the Gitlab runner agent AMI. Must resolve to an Amazon Linux 1 or 2 image. | `map(list(string))` | <pre>{<br>  "name": [<br>    "amzn2-ami-hvm-2.*-x86_64-ebs"<br>  ]<br>}</pre> | no |
 | ami\_owners | The list of owners used to select the AMI of Gitlab runner agent instances. | `list(string)` | <pre>[<br>  "amazon"<br>]</pre> | no |
+| arn\_format | ARN format to be used. May be changed to support deployment in GovCloud/China regions. | `string` | `"arn:aws"` | no |
 | aws\_region | AWS region. | `string` | n/a | yes |
 | aws\_zone | AWS availability zone (typically 'a', 'b', or 'c'). | `string` | `"a"` | no |
 | cache\_bucket | Configuration to control the creation of the cache bucket. By default the bucket will be created and used as shared cache. To use the same cache across multiple runners disable the creation of the cache and provide a policy and bucket name. See the public runner example for more details. | `map` | <pre>{<br>  "bucket": "",<br>  "create": true,<br>  "policy": ""<br>}</pre> | no |
@@ -268,6 +268,7 @@ terraform destroy
 | enable\_gitlab\_runner\_ssh\_access | Enables SSH Access to the gitlab runner instance. | `bool` | `false` | no |
 | enable\_kms | Let the module manage a KMS key, logs will be encrypted via KMS. Be-aware of the costs of an custom key. | `bool` | `false` | no |
 | enable\_manage\_gitlab\_token | Boolean to enable the management of the GitLab token in SSM. If `true` the token will be stored in SSM, which means the SSM property is a terraform managed resource. If `false` the Gitlab token will be stored in the SSM by the user-data script during creation of the the instance. However the SSM parameter is not managed by terraform and will remain in SSM after a `terraform destroy`. | `bool` | `true` | no |
+| enable\_ping | Allow ICMP Ping to the ec2 instances. | `bool` | `false` | no |
 | enable\_runner\_ssm\_access | Add IAM policies to the runner agent instance to connect via the Session Manager. | `bool` | `false` | no |
 | enable\_runner\_user\_data\_trace\_log | Enable bash xtrace for the user data script that creates the EC2 instance for the runner agent. Be aware this could log sensitive data such as you GitLab runner token. | `bool` | `false` | no |
 | enable\_schedule | Flag used to enable/disable auto scaling group schedule for the runner instance. | `bool` | `false` | no |
@@ -279,6 +280,7 @@ terraform destroy
 | instance\_type | Instance type used for the GitLab runner. | `string` | `"t3.micro"` | no |
 | kms\_deletion\_window\_in\_days | Key rotation window, set to 0 for no rotation. Only used when `enable_kms` is set to `true`. | `number` | `7` | no |
 | kms\_key\_id | KMS key id to encrypted the CloudWatch logs. Ensure CloudWatch has access to the provided KMS key. | `string` | `""` | no |
+| log\_group\_name | Option to override the default name (`environment`) of the log group, requires `enable_cloudwatch_logging = true`. | `string` | n/a | yes |
 | overrides | This maps provides the possibility to override some defaults. The following attributes are supported: `name_sg` overwrite the `Name` tag for all security groups created by this module. `name_runner_agent_instance` override the `Name` tag for the ec2 instance defined in the auto launch configuration. `name_docker_machine_runners` ovverrid the `Name` tag spot instances created by the runner agent. | `map(string)` | <pre>{<br>  "name_docker_machine_runners": "",<br>  "name_runner_agent_instance": "",<br>  "name_sg": ""<br>}</pre> | no |
 | permissions\_boundary | Name of permissions boundary policy to attach to AWS IAM roles | `string` | `""` | no |
 | runner\_ami\_filter | List of maps used to create the AMI filter for the Gitlab runner docker-machine AMI. | `map(list(string))` | <pre>{<br>  "name": [<br>    "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*"<br>  ]<br>}</pre> | no |
@@ -314,11 +316,11 @@ terraform destroy
 | runners\_request\_concurrency | Limit number of concurrent requests for new jobs from GitLab (default 1) | `number` | `1` | no |
 | runners\_request\_spot\_instance | Whether or not to request spot instances via docker-machine | `bool` | `true` | no |
 | runners\_root\_size | Runner instance root size in GB. | `number` | `16` | no |
-| runners\_services\_volumes\_tmpfs | Mount temporary file systems to service containers. Must consist of pairs of strings e.g. "/var/lib/mysql" = "rw,noexec", see example | `list` | `[]` | no |
+| runners\_services\_volumes\_tmpfs | n/a | <pre>list(object({<br>    volume  = string<br>    options = string<br>  }))</pre> | `[]` | no |
 | runners\_shm\_size | shm\_size for the runners, will be used in the runner config.toml | `number` | `0` | no |
 | runners\_token | Token for the runner, will be used in the runner config.toml. | `string` | `"__REPLACED_BY_USER_DATA__"` | no |
 | runners\_use\_private\_address | Restrict runners to the use of a private IP address | `bool` | `true` | no |
-| runners\_volumes\_tmpfs | Mount temporary file systems to the main containers. Must consist of pairs of strings e.g. "/var/lib/mysql" = "rw,noexec", see example | `list` | `[]` | no |
+| runners\_volumes\_tmpfs | n/a | <pre>list(object({<br>    volume  = string<br>    options = string<br>  }))</pre> | `[]` | no |
 | schedule\_config | Map containing the configuration of the ASG scale-in and scale-up for the runner instance. Will only be used if enable\_schedule is set to true. | `map` | <pre>{<br>  "scale_in_count": 0,<br>  "scale_in_recurrence": "0 18 * * 1-5",<br>  "scale_out_count": 1,<br>  "scale_out_recurrence": "0 8 * * 1-5"<br>}</pre> | no |
 | secure\_parameter\_store\_runner\_token\_key | The key name used store the Gitlab runner token in Secure Parameter Store | `string` | `"runner-token"` | no |
 | ssh\_key\_pair | Set this to use existing AWS key pair | `string` | `""` | no |

_docs/README.md

@@ -3,7 +3,7 @@
 
 # Terraform module for GitLab auto scaling runners on AWS spot instances
 
-> *NEW*: Terraform 0.12 is supported.
+> "Type changes": The types of variable `runners_volumes_tmpfs`, and  `runners_services_volumes_tmpfs` are changed to support the Terraform 12 `templatefile` function. Check the [default example](examples/runner-pre-registered/main.tf) for an usages example.
 
 ## Terraform versions
 

_docs/TF_MODULE.md

@@ -4,7 +4,6 @@
 |------|---------|
 | aws | n/a |
 | null | n/a |
-| template | n/a |
 
 ## Inputs
 
@@ -14,6 +13,7 @@
 | allow\_iam\_service\_linked\_role\_creation | Boolean used to control attaching the policy to a runner instance to create service linked roles. | `bool` | `true` | no |
 | ami\_filter | List of maps used to create the AMI filter for the Gitlab runner agent AMI. Must resolve to an Amazon Linux 1 or 2 image. | `map(list(string))` | <pre>{<br>  "name": [<br>    "amzn2-ami-hvm-2.*-x86_64-ebs"<br>  ]<br>}</pre> | no |
 | ami\_owners | The list of owners used to select the AMI of Gitlab runner agent instances. | `list(string)` | <pre>[<br>  "amazon"<br>]</pre> | no |
+| arn\_format | ARN format to be used. May be changed to support deployment in GovCloud/China regions. | `string` | `"arn:aws"` | no |
 | aws\_region | AWS region. | `string` | n/a | yes |
 | aws\_zone | AWS availability zone (typically 'a', 'b', or 'c'). | `string` | `"a"` | no |
 | cache\_bucket | Configuration to control the creation of the cache bucket. By default the bucket will be created and used as shared cache. To use the same cache across multiple runners disable the creation of the cache and provide a policy and bucket name. See the public runner example for more details. | `map` | <pre>{<br>  "bucket": "",<br>  "create": true,<br>  "policy": ""<br>}</pre> | no |
@@ -34,6 +34,7 @@
 | enable\_gitlab\_runner\_ssh\_access | Enables SSH Access to the gitlab runner instance. | `bool` | `false` | no |
 | enable\_kms | Let the module manage a KMS key, logs will be encrypted via KMS. Be-aware of the costs of an custom key. | `bool` | `false` | no |
 | enable\_manage\_gitlab\_token | Boolean to enable the management of the GitLab token in SSM. If `true` the token will be stored in SSM, which means the SSM property is a terraform managed resource. If `false` the Gitlab token will be stored in the SSM by the user-data script during creation of the the instance. However the SSM parameter is not managed by terraform and will remain in SSM after a `terraform destroy`. | `bool` | `true` | no |
+| enable\_ping | Allow ICMP Ping to the ec2 instances. | `bool` | `false` | no |
 | enable\_runner\_ssm\_access | Add IAM policies to the runner agent instance to connect via the Session Manager. | `bool` | `false` | no |
 | enable\_runner\_user\_data\_trace\_log | Enable bash xtrace for the user data script that creates the EC2 instance for the runner agent. Be aware this could log sensitive data such as you GitLab runner token. | `bool` | `false` | no |
 | enable\_schedule | Flag used to enable/disable auto scaling group schedule for the runner instance. | `bool` | `false` | no |
@@ -45,6 +46,7 @@
 | instance\_type | Instance type used for the GitLab runner. | `string` | `"t3.micro"` | no |
 | kms\_deletion\_window\_in\_days | Key rotation window, set to 0 for no rotation. Only used when `enable_kms` is set to `true`. | `number` | `7` | no |
 | kms\_key\_id | KMS key id to encrypted the CloudWatch logs. Ensure CloudWatch has access to the provided KMS key. | `string` | `""` | no |
+| log\_group\_name | Option to override the default name (`environment`) of the log group, requires `enable_cloudwatch_logging = true`. | `string` | n/a | yes |
 | overrides | This maps provides the possibility to override some defaults. The following attributes are supported: `name_sg` overwrite the `Name` tag for all security groups created by this module. `name_runner_agent_instance` override the `Name` tag for the ec2 instance defined in the auto launch configuration. `name_docker_machine_runners` ovverrid the `Name` tag spot instances created by the runner agent. | `map(string)` | <pre>{<br>  "name_docker_machine_runners": "",<br>  "name_runner_agent_instance": "",<br>  "name_sg": ""<br>}</pre> | no |
 | permissions\_boundary | Name of permissions boundary policy to attach to AWS IAM roles | `string` | `""` | no |
 | runner\_ami\_filter | List of maps used to create the AMI filter for the Gitlab runner docker-machine AMI. | `map(list(string))` | <pre>{<br>  "name": [<br>    "ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*"<br>  ]<br>}</pre> | no |
@@ -80,11 +82,11 @@
 | runners\_request\_concurrency | Limit number of concurrent requests for new jobs from GitLab (default 1) | `number` | `1` | no |
 | runners\_request\_spot\_instance | Whether or not to request spot instances via docker-machine | `bool` | `true` | no |
 | runners\_root\_size | Runner instance root size in GB. | `number` | `16` | no |
-| runners\_services\_volumes\_tmpfs | Mount temporary file systems to service containers. Must consist of pairs of strings e.g. "/var/lib/mysql" = "rw,noexec", see example | `list` | `[]` | no |
+| runners\_services\_volumes\_tmpfs | n/a | <pre>list(object({<br>    volume  = string<br>    options = string<br>  }))</pre> | `[]` | no |
 | runners\_shm\_size | shm\_size for the runners, will be used in the runner config.toml | `number` | `0` | no |
 | runners\_token | Token for the runner, will be used in the runner config.toml. | `string` | `"__REPLACED_BY_USER_DATA__"` | no |
 | runners\_use\_private\_address | Restrict runners to the use of a private IP address | `bool` | `true` | no |
-| runners\_volumes\_tmpfs | Mount temporary file systems to the main containers. Must consist of pairs of strings e.g. "/var/lib/mysql" = "rw,noexec", see example | `list` | `[]` | no |
+| runners\_volumes\_tmpfs | n/a | <pre>list(object({<br>    volume  = string<br>    options = string<br>  }))</pre> | `[]` | no |
 | schedule\_config | Map containing the configuration of the ASG scale-in and scale-up for the runner instance. Will only be used if enable\_schedule is set to true. | `map` | <pre>{<br>  "scale_in_count": 0,<br>  "scale_in_recurrence": "0 18 * * 1-5",<br>  "scale_out_count": 1,<br>  "scale_out_recurrence": "0 8 * * 1-5"<br>}</pre> | no |
 | secure\_parameter\_store\_runner\_token\_key | The key name used store the Gitlab runner token in Secure Parameter Store | `string` | `"runner-token"` | no |
 | ssh\_key\_pair | Set this to use existing AWS key pair | `string` | `""` | no |

cache/README.md

@@ -30,16 +30,18 @@ module "runner" {
 | Name | Version |
 |------|---------|
 | aws | n/a |
-| template | n/a |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:-----:|
+| arn\_format | ARN format to be used. May be changed to support deployment in GovCloud/China regions. | `string` | `"arn:aws"` | no |
 | cache\_bucket\_name\_include\_account\_id | Boolean to add current account ID to cache bucket name. | `bool` | `true` | no |
 | cache\_bucket\_prefix | Prefix for s3 cache bucket name. | `string` | `""` | no |
 | cache\_bucket\_versioning | Boolean used to enable versioning on the cache bucket, false by default. | `string` | `"false"` | no |
 | cache\_expiration\_days | Number of days before cache objects expires. | `number` | `1` | no |
+| cache\_lifecycle\_clear | Enable the rule to cleanup the cache for expired objects. | `bool` | `true` | no |
+| cache\_lifecycle\_prefix | Object key prefix identifying one or more objects to which the clean up rule applies. | `string` | `"runner/"` | no |
 | create\_cache\_bucket | This module is by default included in the runner module. To disable the creation of the bucket this parameter can be disabled. | `bool` | `true` | no |
 | environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
 | tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |

cache/_docs/TF_MODULE.md

@@ -3,16 +3,18 @@
 | Name | Version |
 |------|---------|
 | aws | n/a |
-| template | n/a |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:-----:|
+| arn\_format | ARN format to be used. May be changed to support deployment in GovCloud/China regions. | `string` | `"arn:aws"` | no |
 | cache\_bucket\_name\_include\_account\_id | Boolean to add current account ID to cache bucket name. | `bool` | `true` | no |
 | cache\_bucket\_prefix | Prefix for s3 cache bucket name. | `string` | `""` | no |
 | cache\_bucket\_versioning | Boolean used to enable versioning on the cache bucket, false by default. | `string` | `"false"` | no |
 | cache\_expiration\_days | Number of days before cache objects expires. | `number` | `1` | no |
+| cache\_lifecycle\_clear | Enable the rule to cleanup the cache for expired objects. | `bool` | `true` | no |
+| cache\_lifecycle\_prefix | Object key prefix identifying one or more objects to which the clean up rule applies. | `string` | `"runner/"` | no |
 | create\_cache\_bucket | This module is by default included in the runner module. To disable the creation of the bucket this parameter can be disabled. | `bool` | `true` | no |
 | environment | A name that identifies the environment, used as prefix and for tagging. | `string` | n/a | yes |
 | tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | `map(string)` | `{}` | no |

cache/main.tf

@@ -30,9 +30,9 @@ resource "aws_s3_bucket" "build_cache" {
 
   lifecycle_rule {
     id      = "clear"
-    enabled = true
+    enabled = var.cache_lifecycle_clear
 
-    prefix = "runner/"
+    prefix = var.cache_lifecycle_prefix
 
     expiration {
       days = var.cache_expiration_days
@@ -52,20 +52,16 @@ resource "aws_s3_bucket" "build_cache" {
   }
 }
 
-data "template_file" "docker_machine_cache_policy" {
-  template = file("${path.module}/policies/cache.json")
-
-  vars = {
-    s3_cache_arn = var.create_cache_bucket == false || length(aws_s3_bucket.build_cache) == 0 ? "arn:aws:s3:::fake_bucket_doesnt_exist" : aws_s3_bucket.build_cache[0].arn
-  }
-}
-
 resource "aws_iam_policy" "docker_machine_cache" {
   count = var.create_cache_bucket ? 1 : 0
 
   name        = "${var.environment}-docker-machine-cache"
   path        = "/"
   description = "Policy for docker machine instance to access cache"
 
-  policy = data.template_file.docker_machine_cache_policy.rendered
+  policy = templatefile("${path.module}/policies/cache.json",
+    {
+      s3_cache_arn = var.create_cache_bucket == false || length(aws_s3_bucket.build_cache) == 0 ? "${var.arn_format}:s3:::fake_bucket_doesnt_exist" : aws_s3_bucket.build_cache[0].arn
+    }
+  )
 }

cache/variables.tf

@@ -38,3 +38,21 @@ variable "create_cache_bucket" {
   type        = bool
   default     = true
 }
+
+variable "cache_lifecycle_clear" {
+  description = "Enable the rule to cleanup the cache for expired objects."
+  type        = bool
+  default     = true
+}
+
+variable "cache_lifecycle_prefix" {
+  description = "Object key prefix identifying one or more objects to which the clean up rule applies."
+  type        = string
+  default     = "runner/"
+}
+
+variable "arn_format" {
+  type        = string
+  default     = "arn:aws"
+  description = "ARN format to be used. May be changed to support deployment in GovCloud/China regions."
+}

examples/runner-default/.terraform-version

@@ -1 +1 @@
-0.12.18
+0.12.23

examples/runner-default/main.tf

@@ -61,12 +61,19 @@ module "runner" {
   runners_additional_volumes = ["/certs/client"]
 
   runners_volumes_tmpfs = [
-    { "/var/opt/cache" = "rw,noexec" },
+    {
+      volume  = "/var/opt/cache",
+      options = "rw,noexec"
+    }
   ]
 
   runners_services_volumes_tmpfs = [
-    { "/var/lib/mysql" = "rw,noexec" },
+    {
+      volume  = "/var/lib/mysql",
+      options = "rw,noexec"
+    }
   ]
+
   # working 9 to 5 :)
   runners_off_peak_periods = "[\"* * 0-9,17-23 * * mon-fri *\", \"* * * * * sat,sun *\"]"
 }

examples/runner-default/providers.tf

@@ -3,10 +3,6 @@ provider "aws" {
   version = "2.41"
 }
 
-provider "template" {
-  version = "2.1.2"
-}
-
 provider "local" {
   version = "1.4"
 }

examples/runner-docker/.terraform-version

@@ -1 +1 @@
-0.12.18
+0.12.23

examples/runner-docker/providers.tf

@@ -3,10 +3,6 @@ provider "aws" {
   version = "2.41"
 }
 
-provider "template" {
-  version = "2.1.2"
-}
-
 provider "local" {
   version = "1.4"
 }

examples/runner-pre-registered/.terraform-version

@@ -1 +1 @@
-0.12.18
+0.12.23

examples/runner-pre-registered/providers.tf

@@ -3,10 +3,6 @@ provider "aws" {
   version = "2.41"
 }
 
-provider "template" {
-  version = "2.1.2"
-}
-
 provider "local" {
   version = "1.4"
 }

examples/runner-public/.terraform-version

@@ -1 +1 @@
-0.12.18
+0.12.23

examples/runner-public/providers.tf

@@ -3,10 +3,6 @@ provider "aws" {
   version = "2.41"
 }
 
-provider "template" {
-  version = "2.1.2"
-}
-
 provider "local" {
   version = "1.4"
 }