Merge branch 'release/3.1.0'

CHANGELOG.md

@@ -5,6 +5,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## [Unreleased]
+## [3.1.0] - 2019-03-09
+- Added: Option to set environment variables for the runners, see the variable `runners_environment_vars`. An example added to the `public-runner` example. 
 
 ## [3.0.0] - 2019-03-29
 - Changed: The runner will register itself based on the registration token. No need to preregister the runner before running terraform. See the [README](README.md) for configuration and migration. #33
@@ -105,7 +107,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - Update default AMI's to The latest Amazon Linux AMI 2017.09.1 - released on 2018-01-17.
 - Minor updates in the example
 
-[Unreleased]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/3.0.0...HEAD
+[Unreleased]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/3.1.0...HEAD
+[3.1.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/3.0.0...3.1.0
 [3.0.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.3.0...3.0.0
 [2.3.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.2.1...2.3.0
 [2.2.1]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.2.0...2.2.1

README.md

@@ -9,17 +9,17 @@ This repo contains a Terraform module and examples to run a [GitLab CI multi run
 
 ![GitLab Runners](https://github.com/npalm/assets/raw/master/images/2017-12-06_gitlab-multi-runner-aws.png)
 
-The setup is based on the blog post: [Auto scale GitLab CI runners and save 90% on EC2 costs](https://about.gitlab.com/2017/11/23/autoscale-ci-runners/) The created runner will have by default a shared cache in S3 and logging is streamed to CloudWatch. The cache in S3 will expire in X days, see configuration. The logging can be disabled. The post mentioned that you have to register the the runner before running the Terraform scripts. Since version 3+ this is not needed anymore you can pass the runner configuration including the runner registration token.
+The setup is based on the blog post: [Auto scale GitLab CI runners and save 90% on EC2 costs](https://about.gitlab.com/2017/11/23/autoscale-ci-runners/) The gitlab-ci runners that this project creates will be configured to use a shared cache via S3 by default. Additionally their logs will be streamed to CloudWatch. The s3 stored cache expiration is configurable and is set to expire in X days by default. Logging can be disabled. The accompanying post mentions that you have to register the the runner before running the Terraform scripts. Since version 3+ this is no longer required. You can simply define the runner configuration, including the runner registration token, via terraform.
 
-Besides the auto scaling option (docker+machine executor) the docker executor is supported as well for a single node.
+In addition to the auto scaling option (docker+machine executor) the docker executor is supported for a single node.
 
 ## Prerequisites
 
 ### Terraform
 
 Ensure you have Terraform installed, see `.terraform-version` for the used version. A handy tool to mange your Terraform version is [tfenv](https://github.com/kamatama41/tfenv).
 
-On mac simple install tfenv using brew.
+On macOS it is simple to install `tfenv` using brew.
 
 ```sh
 brew install tfenv
@@ -33,8 +33,7 @@ tfenv install <version>
 
 ### AWS
 
-To run the Terraform scripts you need to have AWS keys.
-Example file:
+Export your AWS Security Credentials:
 
 ```sh
 export AWS_ACCESS_KEY_ID=...
@@ -43,12 +42,12 @@ export AWS_SECRET_ACCESS_KEY=...
 
 ### Service linked roles
 
-The gitlab runner EC2 instance needs the following service linked roles:
+The gitlab runner EC2 instance requires the following service linked roles:
 
 - AWSServiceRoleForAutoScaling
 - AWSServiceRoleForEC2Spot
 
-By default the EC2 instance is allowed to create the roles, by setting the option `allow_iam_service_linked_role_creation` to `false` you can deny the creation of roles by the instance. In that case you have to ensure the roles exists. You can create them manually or via terraform.
+By default the EC2 instance is allowed to create the required roles, but this can be disabled by setting the option `allow_iam_service_linked_role_creation` to `false`. If disabled you must ensure the roles exist. You can create them manually or via Terraform.
 
 ```hcl
 resource "aws_iam_service_linked_role" "spot" {
@@ -60,26 +59,26 @@ resource "aws_iam_service_linked_role" "autoscaling" {
 }
 ```
 
-### Configuration GitLab runner token
+### GitLab runner token configuration
 
-By default the runner is registered the first time. In previous version this was a manual process. The manual process is still supported but will be removed in future releases. The runner token will be stored in the parameter store. See [example](examples/runner-pre-registered/) for more details.
+By default the runner is registered on initial deployment. In previous versions of this module this was a manual process. The manual process is still supported but will be removed in future releases. The runner token will be stored in the parameter store. See [example](examples/runner-pre-registered/) for more details.
 
-To register the runner automatically set the variable `gitlab_runner_registration_config["token"]` which you can find in your GitLab project, group or global settings. For a generic runner you can find the token in the admin section. By default the runner will be locked to project, not run untagged. Below an example of the configuration map.
+To register the runner automatically set the variable `gitlab_runner_registration_config["token"]`. This token value can be found in your GitLab project, group, or global settings. For a generic runner you can find the token in the admin section. By default the runner will be locked to the target project, not run untagged. Below is an example of the configuration map.
 
-```
-  gitlab_runner_registration_config = {
-    registration_token = "<registration token>"
-    tag_list           = "<your tags, comma separated"
-    description        = "<some description>"
-    locked_to_project  = "true"
-    run_untagged       = "false"
-    maximum_timeout    = "3600"
-  }
+```hcl
+gitlab_runner_registration_config = {
+  registration_token = "<registration token>"
+  tag_list           = "<your tags, comma separated>"
+  description        = "<some description>"
+  locked_to_project  = "true"
+  run_untagged       = "false"
+  maximum_timeout    = "3600"
+}
 ```
 
-For migration to the new setup simply add the runner token to the parameter store. Once the runner is started it will lookup required values in the parameter store. If the value is null a new runner will be created.
+For migration to the new setup simply add the runner token to the parameter store. Once the runner is started it will lookup the required values via the parameter store. If the value is `null` a new runner will be created.
 
-```
+```sh
 # set the following variables, look up the variables in your Terraform config.
 # see your Terraform variables to fill in the vars below.
 aws-region=<${var.aws_region}>
@@ -88,27 +87,26 @@ parameter-name=<${var.environment}>-<${var.secure_parameter_store_runner_token_k
 
 aws ssm put-parameter --overwrite --type SecureString  --name "${parameter-name}" --value ${token} --region "${aws-region}"
 ```
-Once you have created the parameter, you have to remove the variable `runners_token` from your config. Then next time your gitlab runner instance is created it look up the token from the parameter store.
 
-Finally the runner still support the manual runner creation, no changes are required. Please keep in mind that this setup will be removed in future releases.
+Once you have created the parameter, you must remove the variable `runners_token` from your config. The next time your gitlab runner instance is created it will look up the token from the SSM parameter store.
 
+Finally, the runner still supports the manual runner creation. No changes are required. Please keep in mind that this setup will be removed in future releases.
 
 ## Usage
 
 ### Configuration
 
-Update the variables in `terraform.tfvars` to your needs and add the following variables, see previous step for how to obtain the token.
+Update the variables in `terraform.tfvars` according to your needs and add the following variables. See the previous step for instructions on how to obtain the token.
 
 ```hcl
 runner_name  = "NAME_OF_YOUR_RUNNER"
 gitlab_url   = "GITLAB_URL"
 runner_token = "RUNNER_TOKEN"
 ```
 
-The base image used to host the GitLab Runner agent is the latest available Amazon Linux HVM EBS AMI. In previous version of the module an hard coded list of AMI per region was available. This list is replaced by a search filter to find the latest AMI. By setting the filter for example to `amzn-ami-hvm-2018.03.0.20180622-x86_64-ebs` you can lock the version of the AMI.
-
+The base image used to host the GitLab Runner agent is the latest available Amazon Linux HVM EBS AMI. In previous versions of this module a hard coded list of AMIs per region was provided. This list has been replaced by a search filter to find the latest AMI. Setting the filter to `amzn-ami-hvm-2018.03.0.20180622-x86_64-ebs` will allow you to version lock the target AMI.
 
-### Usage module.
+### Usage module
 
 ```hcl
 module "runner" {
@@ -164,14 +162,15 @@ module "runner" {
 | docker_machine_user | Username of the user used to create the spot instances that host docker-machine. | string | `docker-machine` | no |
 | docker_machine_version | Version of docker-machine. | string | `0.16.1` | no |
 | enable_cloudwatch_logging | Boolean used to enable or disable the CloudWatch logging. | string | `true` | no |
-| enable_manage_gitlab_token | Manage the GitLab token in SSM, if `true` the token SSM parameter will be manged by terraform, a destroy removes the parameter for the runner token. When `false` the token will not be manged by terraform. The runner process will still store the token in SSM but a terraform destroy will not remove the token. | string | `true` | no |
+| enable_manage_gitlab_token | Boolean to enable the management of the GitLab token in SSM. If `true` the Gitlab token will be managed via terraform state. If `false` the token will still be stored in SSM however, it will not be managed via terraform. | string | `true` | no |
 | environment | A name that identifies the environment, used as prefix and for tagging. | string | - | yes |
 | gitlab_runner_registration_config | Configuration used to register the runner. See the README for an example, or reference the examples in the examples directory of this repo. | map | `<map>` | no |
-| gitlab_runner_version | Version of the GitLab runner. | string | `11.8.0` | no |
+| gitlab_runner_version | Version of the GitLab runner. | string | `11.9.1` | no |
 | instance_role_json | Docker machine runner instance override policy, expected to be in JSON format. | string | `` | no |
 | instance_role_runner_json | Instance role json for the docker machine runners to override the default. | string | `` | no |
 | instance_type | Instance type used for the GitLab runner. | string | `t2.micro` | no |
 | runners_concurrent | Concurrent value for the runners, will be used in the runner config.toml. | string | `10` | no |
+| runners_environment_vars | Environment variables during build execution, e.g. KEY=Value, see runner-public example. Will be used in the runner config.toml | list | `<list>` | no |
 | runners_executor | The executor to use. Currently supports `docker+machine` or `docker`. | string | `docker+machine` | no |
 | runners_gitlab_url | URL of the GitLab instance to connect to. | string | - | yes |
 | runners_iam_instance_profile_name | IAM instance profile name of the runners, will be used in the runner config.toml | string | `` | no |
@@ -189,16 +188,16 @@ module "runner" {
 | runners_post_build_script | Commands to be executed on the Runner just after executing the build, but before executing after_script. | string | `` | no |
 | runners_pre_build_script | Script to execute in the pipeline just before the build, will be used in the runner config.toml | string | `` | no |
 | runners_pre_clone_script | Commands to be executed on the Runner before cloning the Git repository. this can be used to adjust the Git client configuration first, for example. | string | `` | no |
-| runners_privilled | Runners will run in privileged mode, will be used in the runner config.toml | string | `true` | no |
+| runners_privileged | Runners will run in privileged mode, will be used in the runner config.toml | string | `true` | no |
 | runners_request_concurrency | Limit number of concurrent requests for new jobs from GitLab (default 1) | string | `1` | no |
 | runners_root_size | Runner instance root size in GB. | string | `16` | no |
 | runners_token | Token for the runner, will be used in the runner config.toml. | string | `__REPLACED_BY_USER_DATA__` | no |
 | runners_use_private_address | Restrict runners to the use of a private IP address | string | `true` | no |
-| secure_parameter_store_runner_token_key | The key name used store the Gitlab runner token in Secure Paramater Store | string | `runner-token` | no |
+| secure_parameter_store_runner_token_key | The key name used store the Gitlab runner token in Secure Parameter Store | string | `runner-token` | no |
 | ssh_public_key | Public SSH key used for the GitLab runner EC2 instance. | string | - | yes |
 | subnet_id_runners | List of subnets used for hosting the gitlab-runners. | string | - | yes |
 | subnet_ids_gitlab_runner | Subnet used for hosting the GitLab runner. | list | - | yes |
-| tags | Map of tags that will be added to created resources. By default resources will be taggen with name and environemnt. | map | `<map>` | no |
+| tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environment. | map | `<map>` | no |
 | userdata_post_install | User-data script snippet to insert after GitLab runner install | string | `` | no |
 | userdata_pre_install | User-data script snippet to insert before GitLab runner install | string | `` | no |
 | vpc_id | The target VPC for the docker-machine and runner instances. | string | - | yes |
@@ -214,14 +213,16 @@ module "runner" {
 
 ## Example
 
-An example is provided, execute the following steps to run the sample. Ensure your AWS and Terraform environment is set up correctly. All commands below are supposed to be run inside the directory `example`.
+A few [examples](examples) are provided. Use the following steps to deploy. Ensure your AWS and Terraform environment is set up correctly. All commands below should be run from the `terraform-aws-gitlab-runner/examples` directory.
 
 ### AWS keys
 
-Keys are generated by Terraform and stored in a directory `generated` in the example directory.
+SSH keys are generated by Terraform and stored in the `generated` directory of each example directory.
 
 ### Configure GitLab
 
+*This step is not needed anymore* Configure you runner via `gitlab_runner_registration_config`. Configuring GitLab via the step below is only needed when you choose to create the token manually and set the `runners_token` variable.
+
 Register a new runner:
 
 ```sh
@@ -232,7 +233,7 @@ Once done, lookup the token in GitLab and update the `terraform.tfvars` file.
 
 ## Create runner
 
-Run `terraform init` to initialise Terraform. Next you can run `terraform plan` to inspect the resources that will be created.
+Run `terraform init` to initialize Terraform. Next you can run `terraform plan` to inspect the resources that will be created.
 
 To create the runner run:
 

examples/runner-default/README.md

@@ -1,6 +1,7 @@
 # Example - Spot Runner - Private subnets
 
-Example how to run builds on spot instnaces in a private subent.
+Example how to run builds on spot instances in a private subnet.
 
 ## Prerequisite
-The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using tfenv please check `.terraform-version` for the tested version.
+
+The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using `tfenv` please check `.terraform-version` for the tested version.

examples/runner-default/variables.tf

@@ -5,7 +5,7 @@ variable "aws_region" {
 }
 
 variable "environment" {
-  description = "A name that indentifies the environment, will used as prefix and for taggin."
+  description = "A name that identifies the environment, will used as prefix and for tagging."
   type        = "string"
   default     = "runners-default"
 }

examples/runner-docker/README.md

@@ -3,4 +3,5 @@
 Example how create a gitlab runner using the docker executor on a single node, running in a private subnet.
 
 ## Prerequisite
-The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using tfenv please check `.terraform-version` for the tested version.
+
+The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using `tfenv` please check `.terraform-version` for the tested version.

examples/runner-docker/variables.tf

@@ -5,7 +5,7 @@ variable "aws_region" {
 }
 
 variable "environment" {
-  description = "A name that indentifies the environment, will used as prefix and for taggin."
+  description = "A name that identifies the environment, will used as prefix and for tagging."
   default     = "runners-docker"
   type        = "string"
 }

examples/runner-pre-registered/README.md

@@ -1,6 +1,7 @@
 # Example - Spot Runner - Private subnets
 
-This is the previous default example. For this example you need to register the runner before running terraform and provide the runner token. Since version 3+ the runner can register itself by providing the registration token. This examples is to show case backwards compatibility.
+This is the previous default example. For this example you need to register the runner before running terraform and provide the runner token. Since version 3+ the runner can register itself by providing the registration token. This example is provided to showcase backwards compatibility.
 
 ## Prerequisite
-The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using tfenv please check `.terraform-version` for the tested version.
+
+The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using `tfenv` please check `.terraform-version` for the tested version.

examples/runner-pre-registered/variables.tf

@@ -5,7 +5,7 @@ variable "aws_region" {
 }
 
 variable "environment" {
-  description = "A name that indentifies the environment, will used as prefix and for taggin."
+  description = "A name that identifies the environment, will used as prefix and for tagging."
   default     = "ci-runners"
   type        = "string"
 }

examples/runner-public/README.md

@@ -3,4 +3,5 @@
 Example how create a gitlab runner, running in a public subnet.
 
 ## Prerequisite
-The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using tfenv please check `.terraform-version` for the tested version.
+
+The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using `tfenv` please check `.terraform-version` for the tested version.

examples/runner-public/main.tf

@@ -28,8 +28,9 @@ module "runner" {
   subnet_id_runners        = "${element(module.vpc.public_subnets, 0)}"
   aws_zone                 = "b"
 
-  runners_name       = "${var.runner_name}"
-  runners_gitlab_url = "${var.gitlab_url}"
+  runners_name             = "${var.runner_name}"
+  runners_gitlab_url       = "${var.gitlab_url}"
+  runners_environment_vars = ["KEY=Value", "FOO=bar"]
 
   gitlab_runner_registration_config = {
     registration_token = "${var.registration_token}"

examples/runner-public/variables.tf

@@ -5,7 +5,7 @@ variable "aws_region" {
 }
 
 variable "environment" {
-  description = "A name that indentifies the environment, will used as prefix and for taggin."
+  description = "A name that identifies the environment, will used as prefix and for tagging."
   default     = "runner-public"
   type        = "string"
 }