This repository contains the Ansible scripts for deploying a Kubernetes cluster into DBG ProductDev Amazon AWS account. This account has severe limitations in what the users can do and that is why regular tools such as Kops don't work. This repository is based on the AWS QuickStart guide from Heptio. But the CloudFormation stack is adapted. It will probably not work in any other AWS accounts.
- Prerequisites
- Installation
- Uninstallation
- Configuration
- IAM roles
- Create cluster
- Use cluster
- Delete cluster
- Add-ons
- Ingress
- Tagging Lambda
- Next steps
- Install Ansible
- Install kubectl (see below)
- Install AWS CLI
- Login to AWS, place credentials in env vars
- Install jq (json processor):
sudo yum install jq
- create ~/.kube
- Change the configuration (cluster name, tags, etc.)
- Create the IAM roles
- Create the cluster
- Install tagging lambda function
- Install the addons
- Install ingress (optional)
- Delete all namespaces from the cluster (apart from kube-system)
- Delete the cluster
- Delete the tagging lambda
- Check manually in AWS that all auxilary resources has been properly deleted (loadbalancers, volumes etc.)
The main configuration of the cluster is in the variables in group_vars/all/vars.yaml. Following table shows the different options.
| Option | Explanation | Example |
|---|---|---|
cluster_name |
Name of the cluster which should be created. The name should be DNS compatible. It will be used to name different resources etc. | my-k8s-cluster |
kubeconfig_file |
Where should the kubectl configuration downloaded. |
~/.kube/my-k8s-cluster-config |
tags |
A dictionary with different tags which should be used for resources. The tags have to full fill the DBG requirements. | |
ssh_public_key |
Path to the public part of the SSH key, which should be used for the SSH access to the Kubernetes hosts | ~/.ssh/id_rsa.pub |
ssh_private_key |
Path to the private part of the SSH key, which should be used for the SSH access to the Kubernetes hosts | ~/.ssh/id_rsa |
ssh_cidr |
A CIDR from which SSH access is allowed | 172.35.0.0/16 |
api_cidr |
A CIDR from which Kubernetes API access is allowed | 172.35.0.0/16 |
aws_region |
AWS region where the cluster should be installed. | eu-west-1 |
aws_zones |
List of availability zones in which the cluster should be installed. | eu-west-1a |
vpc_id |
ID of the existing Amazon AWS VPC where the cluster should be placed. | vpc-9694d3fe |
subnet_id |
ID of the existing Amazon AWS subnet where the cluster should be placed. | subnet-ca9cdca2 |
dns_zone |
Hosted DNS zone which has to exist in Route53 | dev.dbgcloud.io |
elb_hosted_zone |
The hosted zone in which the aliased ELB load balancers are hosted (should be dependent on the AWS region) | Z32O12XQLNTSW2 |
ec2_instance_type |
EC2 size of the nodes used as workers. | t2.medium |
ec2_disk_size |
Disk size for the EC2 instances (in GB). | 40 |
kubernetes_node_count |
Number of EC2 worker hosts. | 2 |
kubernetes_network |
Defines which networking plugin should be used in Kubernetes. Tested with Calico only. | calico |
template_url |
DBG Role template URL used for the Cloud Formation stack | |
iam_prefix |
DBG role / policy name prefix | DBG-DEV- |
The IAM roles needed for the setup of this Kubernetes cluster can be created by running:
ansible-playbook create-roles.yaml
Because due to some useless security restrictions, we cannot delete the IAM roles and that makes the life complicated, this is a separate playbook instead of being part of the main playbook as would be more appropriate.
The Kubernetes cluster can be created by running
ansible-playbook create-cluster.yaml
Cluster creation will generate a CloudFormation template. IT will upload this template to a newly created bucket. IT will create a new EC2 KeyPair based on the specified SSH key. Afterwards it will create the Cloud Formation stack and execute it.
The cluster can be accessed via kubectl. A config file is created during the setup. It can be advertised to kubectl via environment variable:
export KUBECONFIG=~/.kube/config-wk359-k8s-qs
kubectl cluster-info
To delete the cluster run
ansible-playbook delete-cluster.yaml
This will delete the CloudFormation stack, the resources it created, the SSH key and the S3 bucket.
Currently, the supported add-ons are:
- Kubernetes dashboard
- Heapster for resource monitoring
- Storage class for automatic provisioning of persisitent volumes
To install the add-ons run
ansible-playbook addons.yaml
Ingress can be used route inbound traffic from the outside of the Kubernetes cluster. It can be used for SSL termination, virtual hosts, load balancing etc. For more details about ingress, go to Kubernetes website.
To install ingress controller based on Nginx, run
ansible-playbook ingress.yaml
After installation, the Ingress will be available under the hostname ingress.<ClusterName>.<DNSZone>.
The AWS Lambda function for tagging of resources (the related IAM and CloudWatch objects) can be also installed and uninstalled separately. To install it run:
ansible-playbook install-lambda.yaml
To uninstall it run:
ansible-playbook uninstall-lambda.yaml
The guide how to add new user has been moved to my blog.
The Deutsche Boerse ProductDev account has no connection to the internet. Therefore all ELB load balancers have to be created as internal. To create the load balancer as internal, use following annotation for your service:
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: '0.0.0.0/0'
The addons.yaml playbook will install several addons. One of them is the Route53 mapper. Route53 mapper allows you to create DNS records for your services directly from Kubernetes. To use it, you have to tag you service with tag dns and value route53:
labels:
dns: route53
...
And specify the desired domain name in annotation:
annotations:
domainName: "ghost.dev.dbgcloud.io"
The Route53 mapper is periodically checking the services and creating the Route53 DNS records for them. The domain you can use in ProductDev AWS account is dev.dbgcloud.io. Once you delete the service, the Route53 mapper doesn't delete the Route53 records automatically. You have to do it on your own.