Skip to content
This repository has been archived by the owner on Jan 25, 2023. It is now read-only.

Commit

Permalink
Merge pull request #57 from lawliet89/aws_autoscaling_attachment
Browse files Browse the repository at this point in the history 
Using `aws_autoscaling_attachment` instead
  • Loading branch information
@brikis98
brikis98 authored Mar 19, 2018
2 parents 0e5a7b1 + ad36eab commit 0a6c168
Show file tree
Hide file tree
Showing 3 changed files with 108 additions and 76 deletions.
170 changes: 108 additions & 62 deletions modules/consul-cluster/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Consul Cluster

This folder contains a [Terraform](https://www.terraform.io/) module to deploy a
[Consul](https://www.consul.io/) cluster in [AWS](https://aws.amazon.com/) on top of an Auto Scaling Group. This module
is designed to deploy an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
This folder contains a [Terraform](https://www.terraform.io/) module to deploy a
[Consul](https://www.consul.io/) cluster in [AWS](https://aws.amazon.com/) on top of an Auto Scaling Group. This module
is designed to deploy an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html)
that has Consul installed via the [install-consul](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul) module in this Module.


Expand All @@ -20,42 +20,42 @@ module "consul_cluster" {
# Specify the ID of the Consul AMI. You should build this using the scripts in the install-consul module.
ami_id = "ami-abcd1234"
# Add this tag to each node in the cluster
cluster_tag_key = "consul-cluster"
cluster_tag_value = "consul-cluster-example"
# Configure and start Consul during boot. It will automatically form a cluster with all nodes that have that same tag.
# Configure and start Consul during boot. It will automatically form a cluster with all nodes that have that same tag.
user_data = <<-EOF
#!/bin/bash
/opt/consul/bin/run-consul --server --cluster-tag-key consul-cluster
EOF
# ... See variables.tf for the other parameters you must define for the consul-cluster module
}
```

Note the following parameters:

* `source`: Use this parameter to specify the URL of the consul-cluster module. The double slash (`//`) is intentional
and required. Terraform uses it to specify subfolders within a Git repo (see [module
sources](https://www.terraform.io/docs/modules/sources.html)). The `ref` parameter specifies a specific Git tag in
this repo. That way, instead of using the latest version of this module from the `master` branch, which
* `source`: Use this parameter to specify the URL of the consul-cluster module. The double slash (`//`) is intentional
and required. Terraform uses it to specify subfolders within a Git repo (see [module
sources](https://www.terraform.io/docs/modules/sources.html)). The `ref` parameter specifies a specific Git tag in
this repo. That way, instead of using the latest version of this module from the `master` branch, which
will change every time you run Terraform, you're using a fixed version of the repo.

* `ami_id`: Use this parameter to specify the ID of a Consul [Amazon Machine Image
* `ami_id`: Use this parameter to specify the ID of a Consul [Amazon Machine Image
(AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) to deploy on each server in the cluster. You
should install Consul in this AMI using the scripts in the [install-consul](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul) module.
* `user_data`: Use this parameter to specify a [User

* `user_data`: Use this parameter to specify a [User
Data](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) script that each
server will run during boot. This is where you can use the [run-consul script](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/run-consul) to configure and
run Consul. The `run-consul` script is one of the scripts installed by the [install-consul](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul)
module.
server will run during boot. This is where you can use the [run-consul script](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/run-consul) to configure and
run Consul. The `run-consul` script is one of the scripts installed by the [install-consul](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul)
module.

You can find the other parameters in [variables.tf](variables.tf).

Check out the [consul-cluster example](https://github.com/hashicorp/terraform-aws-consul/tree/master/MAIN.md) for fully-working sample code.
Check out the [consul-cluster example](https://github.com/hashicorp/terraform-aws-consul/tree/master/MAIN.md) for fully-working sample code.



Expand All @@ -64,15 +64,15 @@ Check out the [consul-cluster example](https://github.com/hashicorp/terraform-aw

### Using the HTTP API from your own computer

If you want to connect to the cluster from your own computer, the easiest way is to use the [HTTP
API](https://www.consul.io/docs/agent/http.html). Note that this only works if the Consul cluster is running in public
If you want to connect to the cluster from your own computer, the easiest way is to use the [HTTP
API](https://www.consul.io/docs/agent/http.html). Note that this only works if the Consul cluster is running in public
subnets and/or your default VPC (as in the [consul-cluster example](https://github.com/hashicorp/terraform-aws-consul/tree/master/MAIN.md)), which is OK for testing
and experimentation, but NOT recommended for production usage.

To use the HTTP API, you first need to get the public IP address of one of the Consul Servers. You can find Consul
servers by using AWS tags. If you're running the [consul-cluster example](https://github.com/hashicorp/terraform-aws-consul/tree/master/MAIN.md), the
[consul-examples-helper.sh script](https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/consul-examples-helper/consul-examples-helper.sh) will do the tag lookup
for you automatically (note, you must have the [AWS CLI](https://aws.amazon.com/cli/),
To use the HTTP API, you first need to get the public IP address of one of the Consul Servers. You can find Consul
servers by using AWS tags. If you're running the [consul-cluster example](https://github.com/hashicorp/terraform-aws-consul/tree/master/MAIN.md), the
[consul-examples-helper.sh script](https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/consul-examples-helper/consul-examples-helper.sh) will do the tag lookup
for you automatically (note, you must have the [AWS CLI](https://aws.amazon.com/cli/),
[jq](https://stedolan.github.io/jq/), and the [Consul agent](https://www.consul.io/) installed locally):

```
Expand Down Expand Up @@ -111,7 +111,7 @@ Success! Data written to: foo
```

And reading that value back:

```
> consul kv get -http-addr=11.22.33.44:8500 foo
Expand All @@ -125,8 +125,8 @@ Finally, you can try opening up the Consul UI in your browser at the URL `http:/

### Using the Consul agent on another EC2 Instance

The easiest way to run [Consul agent](https://www.consul.io/docs/agent/basics.html) and have it connect to the Consul
cluster is to use the same EC2 tags the Consul servers use to discover each other during bootstrapping.
The easiest way to run [Consul agent](https://www.consul.io/docs/agent/basics.html) and have it connect to the Consul
cluster is to use the same EC2 tags the Consul servers use to discover each other during bootstrapping.

For example, imagine you deployed a Consul cluster in `us-east-1` as follows:

Expand All @@ -139,26 +139,72 @@ module "consul_cluster" {
# Add this tag to each node in the cluster
cluster_tag_key = "consul-cluster"
cluster_tag_value = "consul-cluster-example"
# ... Other params omitted ...
# ... Other params omitted ...
}
```

Using the `retry-join-ec2-xxx` params, you can connect run a Consul agent on an EC2 Instance as follows:
Using the `retry-join-ec2-xxx` params, you can connect run a Consul agent on an EC2 Instance as follows:

```
consul agent -retry-join-ec2-tag-key=consul-cluster -retry-join-ec2-tag-value=consul-cluster-example -data-dir=/tmp/consul
```

Two important notes about this command:

1. By default, the Consul cluster nodes advertise their *private* IP addresses, so the command above only works from
1. By default, the Consul cluster nodes advertise their *private* IP addresses, so the command above only works from
EC2 Instances inside the same VPC (or any VPC with proper peering connections and route table entries).
1. In order to look up the EC2 tags, the EC2 Instance where you're running this command must have an IAM role with
the `ec2:DescribeInstances` permission.



## How do you connect load balancers to the Auto Scaling Group (ASG)?

You can use the [`aws_autoscaling_attachment`](https://www.terraform.io/docs/providers/aws/r/autoscaling_attachment.html) resource.

For example, if you are using the new application or network load balancers:

```hcl
resource "aws_lb_target_group" "test" {
// ...
}
# Create a new Consul Cluster
module "consul" {
source ="..."
// ...
}
# Create a new load balancer attachment
resource "aws_autoscaling_attachment" "asg_attachment_bar" {
autoscaling_group_name = "${module.consul.asg_name}"
alb_target_group_arn = "${aws_alb_target_group.test.arn}"
}
```

If you are using a "classic" load balancer:

```hcl
# Create a new load balancer
resource "aws_elb" "bar" {
// ...
}
# Create a new Consul Cluster
module "consul" {
source ="..."
// ...
}
# Create a new load balancer attachment
resource "aws_autoscaling_attachment" "asg_attachment_bar" {
autoscaling_group_name = "${module.consul.asg_name}"
elb = "${aws_elb.bar.id}"
}
```



## What's included in this module?

Expand All @@ -177,7 +223,7 @@ This architecture consists of the following resources:
### Auto Scaling Group

This module runs Consul on top of an [Auto Scaling Group (ASG)](https://aws.amazon.com/autoscaling/). Typically, you
should run the ASG with 3 or 5 EC2 Instances spread across multiple [Availability
should run the ASG with 3 or 5 EC2 Instances spread across multiple [Availability
Zones](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). Each of the EC2
Instances should be running an AMI that has Consul installed via the [install-consul](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul)
module. You pass in the ID of the AMI to run using the `ami_id` input parameter.
Expand All @@ -186,30 +232,30 @@ module. You pass in the ID of the AMI to run using the `ami_id` input parameter.
### EC2 Instance Tags

This module allows you to specify a tag to add to each EC2 instance in the ASG. We recommend using this tag with the
[retry_join_ec2](https://www.consul.io/docs/agent/options.html?#retry_join_ec2) configuration to allow the EC2
Instances to find each other and automatically form a cluster.
[retry_join_ec2](https://www.consul.io/docs/agent/options.html?#retry_join_ec2) configuration to allow the EC2
Instances to find each other and automatically form a cluster.


### Security Group

Each EC2 Instance in the ASG has a Security Group that allows:

* All outbound requests
* All the inbound ports specified in the [Consul documentation](https://www.consul.io/docs/agent/options.html?#ports-used)

The Security Group ID is exported as an output variable if you need to add additional rules.
The Security Group ID is exported as an output variable if you need to add additional rules.

Check out the [Security section](#security) for more details.
Check out the [Security section](#security) for more details.


### IAM Role and Permissions

Each EC2 Instance in the ASG has an [IAM Role](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) attached.
We give this IAM role a small set of IAM permissions that each EC2 Instance can use to automatically discover the other
Instances in its ASG and form a cluster with them. See the [run-consul required permissions
Each EC2 Instance in the ASG has an [IAM Role](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) attached.
We give this IAM role a small set of IAM permissions that each EC2 Instance can use to automatically discover the other
Instances in its ASG and form a cluster with them. See the [run-consul required permissions
docs](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/run-consul#required-permissions) for details.

The IAM Role ARN is exported as an output variable if you need to add additional permissions.
The IAM Role ARN is exported as an output variable if you need to add additional permissions.



Expand All @@ -229,9 +275,9 @@ NOT actually deploy those new instances. To make that happen, you should do the
```
curl -X PUT <OLD_INSTANCE_IP>:8500/v1/agent/leave
```
1. Once the instance has left the cluster, terminate it:
```
aws ec2 terminate-instances --instance-ids <OLD_INSTANCE_ID>
```
Expand All @@ -241,7 +287,7 @@ NOT actually deploy those new instances. To make that happen, you should do the
1. Wait for the new Instance to boot and join the cluster.
1. Repeat these steps for each of the other old Instances in the ASG.
We will add a script in the future to automate this process (PRs are welcome!).
Expand All @@ -250,13 +296,13 @@ We will add a script in the future to automate this process (PRs are welcome!).
## What happens if a node crashes?
There are two ways a Consul node may go down:
1. The Consul process may crash. In that case, `supervisor` should restart it automatically.
1. The EC2 Instance running Consul dies. In that case, the Auto Scaling Group should launch a replacement automatically.
1. The EC2 Instance running Consul dies. In that case, the Auto Scaling Group should launch a replacement automatically.
Note that in this case, since the Consul agent did not exit gracefully, and the replacement will have a different ID,
you may have to manually clean out the old nodes using the [force-leave
command](https://www.consul.io/docs/commands/force-leave.html). We may add a script to do this
automatically in the future. For more info, see the [Consul Outage
command](https://www.consul.io/docs/commands/force-leave.html). We may add a script to do this
automatically in the future. For more info, see the [Consul Outage
documentation](https://www.consul.io/docs/guides/outage.html).
Expand All @@ -283,30 +329,30 @@ Consul can encrypt all of its network traffic. For instructions on enabling netw
The EC2 Instances in the cluster store all their data on the root EBS Volume. To enable encryption for the data at
rest, you must enable encryption in your Consul AMI. If you're creating the AMI using Packer (e.g. as shown in
the [consul-ami example](https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/consul-ami)), you need to set the [encrypt_boot
parameter](https://www.packer.io/docs/builders/amazon-ebs.html#encrypt_boot) to `true`.
the [consul-ami example](https://github.com/hashicorp/terraform-aws-consul/tree/master/examples/consul-ami)), you need to set the [encrypt_boot
parameter](https://www.packer.io/docs/builders/amazon-ebs.html#encrypt_boot) to `true`.
### Dedicated instances
If you wish to use dedicated instances, you can set the `tenancy` parameter to `"dedicated"` in this module.
If you wish to use dedicated instances, you can set the `tenancy` parameter to `"dedicated"` in this module.
### Security groups
This module attaches a security group to each EC2 Instance that allows inbound requests as follows:
* **Consul**: For all the [ports used by Consul](https://www.consul.io/docs/agent/options.html#ports), you can
use the `allowed_inbound_cidr_blocks` parameter to control the list of
* **Consul**: For all the [ports used by Consul](https://www.consul.io/docs/agent/options.html#ports), you can
use the `allowed_inbound_cidr_blocks` parameter to control the list of
[CIDR blocks](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) that will be allowed access and the `allowed_inbound_security_group_ids` parameter to control the security groups that will be allowed access.
* **SSH**: For the SSH port (default: 22), you can use the `allowed_ssh_cidr_blocks` parameter to control the list of
* **SSH**: For the SSH port (default: 22), you can use the `allowed_ssh_cidr_blocks` parameter to control the list of
[CIDR blocks](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) that will be allowed access. You can use the `allowed_inbound_ssh_security_group_ids` parameter to control the list of source Security Groups that will be allowed access.
Note that all the ports mentioned above are configurable via the `xxx_port` variables (e.g. `server_rpc_port`). See
[variables.tf](variables.tf) for the full list.
[variables.tf](variables.tf) for the full list.
### SSH access
Expand All @@ -329,15 +375,15 @@ This module does NOT handle the following items, which you may want to provide o
### Monitoring, alerting, log aggregation
This module does not include anything for monitoring, alerting, or log aggregation. All ASGs and EC2 Instances come
with limited [CloudWatch](https://aws.amazon.com/cloudwatch/) metrics built-in, but beyond that, you will have to
This module does not include anything for monitoring, alerting, or log aggregation. All ASGs and EC2 Instances come
with limited [CloudWatch](https://aws.amazon.com/cloudwatch/) metrics built-in, but beyond that, you will have to
provide your own solutions.
### VPCs, subnets, route tables
This module assumes you've already created your network topology (VPC, subnets, route tables, etc). You will need to
pass in the the relevant info about your network topology (e.g. `vpc_id`, `subnet_ids`) as input variables to this
This module assumes you've already created your network topology (VPC, subnets, route tables, etc). You will need to
pass in the the relevant info about your network topology (e.g. `vpc_id`, `subnet_ids`) as input variables to this
module.
Expand Down
2 changes: 0 additions & 2 deletions modules/consul-cluster/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,6 @@ resource "aws_autoscaling_group" "autoscaling_group" {
desired_capacity = "${var.cluster_size}"
termination_policies = ["${var.termination_policies}"]

target_group_arns = ["${var.target_group_arns}"]
load_balancers = ["${var.load_balancers}"]
health_check_type = "${var.health_check_type}"
health_check_grace_period = "${var.health_check_grace_period}"
wait_for_capacity_timeout = "${var.wait_for_capacity_timeout}"
Expand Down
12 changes: 0 additions & 12 deletions modules/consul-cluster/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -123,18 +123,6 @@ variable "root_volume_delete_on_termination" {
default = true
}

variable "target_group_arns" {
description = "A list of target group ARNs of Application Load Balanacer (ALB) targets to associate with this ASG. If you're using a Elastic Load Balancer (AKA ELB Classic), use the load_balancers variable instead."
type = "list"
default = []
}

variable "load_balancers" {
description = "A list of Elastic Load Balancer (ELB) names to associate with this ASG. If you're using an Application Load Balancer (ALB), use the target_group_arns variable instead."
type = "list"
default = []
}

variable "wait_for_capacity_timeout" {
description = "A maximum duration that Terraform should wait for ASG instances to be healthy before timing out. Setting this to '0' causes Terraform to skip all Capacity Waiting behavior."
default = "10m"
Expand Down

0 comments on commit 0a6c168

Please sign in to comment.