The infrastructure for running WCA Live.
First, install terraform and initialize with:
terraform init
terraform workspace select prod || terraform workspace new prod
To apply changes run:
terraform apply -var-file prod.tfvars
Note that you also need the prod.tfvars
file with configuration and secrets.
WCA Live is encapsulated in a single Docker image (with packaged Elixir release inside). Most of the time a single container is enough, however we also have autoscaling in place that may spawn more containers. Each container runs a single node (Erlang distribution) and has its own IP that is registered in a private DNS by ECS. In case multiple containers are running the nodes form a cluster by querying the DNS and discovering peer IPs (using the libcluster
library).
We use ECS to run the app container(s). We use EC2 capacity provider instead of Fargate, because:
-
At the time of writing EC2 vCPU is significantly cheaper:
- EC2 (2 vCPU + 1 GB) ~ 0.0104$/h
- Fargate (1 vCPU + 1 GB) ~ 0.0449$/h
-
The app workload is fairly burstable (entering a result results in broadcasting the update to all listening clients). Fargate doesn't support bursting at the moment (aws/containers-roadmap#163) and overprovisioning would be costy. With EC2 overprovisioning is less of a concern, which gives the app a fair amount of vCPU for bursts.
We use ECS Service Auto Scaling to spawn more app tasks when CPU usage crosses certain threshold. To run the tasks we use EC2 Auto Scaling Group that is automatically managed by ECS (Cluster Auto Scaling).
We configure task resource requirements to guarantee only one task per EC2 instance, which is optimal in our case (we want to give the Erlang node all resources to manage on its own). The only downside of this is that during Blue/Green deployment we need to spawn a new EC2 instance for the new task, which makes for an additional delay, but it is a fair trade off in our case.
To automate deployment we use a CodePipeline that is triggered by pushing a new app image to ECR. The pipeline:
- Sources the new ECR image arn.
- Prepares files necessary for CodeDeploy (
appspec.yaml
,taskdef.json
) - Runs the migration task.
- Triggers CodeDeploy that does Blue/Green deployment.
An arbitrary set of useful resources: