Create Kubernetes The Hard-Way HA Cluster Operations on Amazon AWS using kops by Terraform Code (*HashiCorp)
Written by Kerem ÇELİKER
- Twitter:
@CloudRss - Linkedin:
linkedin.com/in/keremceliker - Blog:
www.keremceliker.com
Tanzu by VMware, Kubernetes and Docker,OpenShift Container Infrastructures will make a definite change to come into our lives as Cloud or Hybrid-Cloud. This time, I prepared a different Container Mini DevOps Project. It can be easily used to start in an infrastructure project as an example.
This is a Terraform module which provides AWS EC2 (without EKS) and S3,Route53,VPC for publishing services via NGINX Ingress on Kubernetes Kops.
It provides the following example Kubernetes Infra-Stack
HTTPS requests are transfered to a pod via the external VPC and the node port of NGINX (e.g. tcp/31504).
- Terraform v12 (w/aws provider v2.57)
- Visual Studio Code
- Ubuntu AMI 16.04 or newer LTS
- Git
- Kops for Kubernetes Operations
- Kubectl for Kubernetes CLI
- Python3 Tool
- Pip3 Tool
- AWS CLI for AWS Operations
- PuttyGen
- Create Route53 hosted zone of the domain
kubernetes_ingress_domain - Nice to have certificate of the wildcard domain
*.kubernetes_ingress_domain - VPC for the cluster
kops_cluster_name - Subnets for the cluster
kops_cluster_name - Use S3 bucket storage to needed create some storage space
kops_S3state_name - Kubernetes on Kops also needs
ssh key pairto be able to connect toEC2 Kubernetesinstances. - Build Docker image if you want to upload images push it to dockerhub.com.
(Optionally) - Auto scaling group for the nodes of the cluster
kops_cluster_name - Security group for the masters of the cluster
kops_cluster_name - Security group for the nodes of the cluster
kops_cluster_name
1.1) Assumptions:
- Development machine is a Windows 10 Pro.
- User has admin privileges on the machine
- At least 15GB of free RAM is available on the development machine. Otherwise, Visual Code Studio will need editing to adjust available memory:
v.customize ["modifyvm", :id, "--memory", <MEMORY_ALLOCATION>]
1.2) Prerequisites:
- Install latest version of Git bash for Windows Setup (As like me)
- Install latest version of VMware Workstation
- Install latest version of Visual Studio Code
- Install latest version of Terraform CLI by HashiCorp
- Install latest version of AWS CLI
- Install latest version of Kubernetes KubeCtl - Kubernetes CLI
- Install latest version of Kubernetes Kops - Kubernetes CLI
- Amazon AWS Cloud EC2 preparation
- Amazon AWS Cloud Route53 preperation
- Amazon AWS Cloud S3 preperation
- Amazon AWS Cloud VPC preparation
- Amazon AWS Cloud EBS preparation
- Amazon AWS Cloud Elastic IPs preparation
- Amazon AWS Cloud EC2 Auto Scaling preparation
- Amazon AWS Cloud IAM preparation
IAC allows infrastructure to be expressed as code in a simple, human readable language called and automate provisioning by Terraform/GitLab or Ansible/Packer
Install Terraform for Linux :
curl -LO https://releases.hashicorp.com/terraform/0.11.7/terraform_0.12_linux_amd64.zip
unzip terraform_0.12_linux_amd64.zip
chmod +x terraform_0.12_linux_amd64
sudo mv terraform_0.12_linux_amd64 /usr/local/bin/terraform
Install Terraform for Windows :
Install Terraform Exe in Windows 64-bit
Set the PATH to use terraform binaries "set PATH=%PATH%;C:\terraform"
Check Terraform version "Terraform --version"
Install K8s Kubectl and Kops cli tool:
After you install and run "kubectl.exe" into the C:\Kerem\* folder, you will receive output when you type "kubectl -- help" on Windows PowerShell.
After you install and run "kops.exe" into the C:\Kerem\*folder, you will receive output when you type "kubectl -- help" on Windows PowerShell.
Install AWS cli tool:
Download AWS CLI, execute the command line or PowerShell command below:
"aws --version"
- You just need to install the another tools mentioned above in the same way. I'm going through these steps not to repeat again.
Kops_s3.tf
cat kops_s3.tf
resource "aws_s3_bucket" "kops-kerem-S3" {
bucket = "kops-kops-kerem-S3"
acl = "private"
tags {
Name = "Kops kops-kerem-S3"
}
}
Provider.tf
provider "aws" {
profile = "default"
region = "eu-west-2"
}
Variables.tf
variable "region" {
default = "eu-west-2"
}
resource "aws_instance" "KeremKubernetesCls" {
ami = "ami-7ad7c21e"
instance_type = "t2.micro"
tags = { Name = "KubeKerem" }
provisioner "local-exec" {
command = "echo ${aws_instance.KeremKubernetesCls.private_ip} >> private_ips.txt"
}
}
output "ip" {
value = "${aws_instance.KeremKubernetesCls.public_ip}"
}
variable "SET_TO_PRIVATE_KEY" {
default = "InputYourKey"
}
variable "SET_TO_PUBLIC_KEY" {
default = "InputYourKey.pub"
}
In order to determine the closest your data center follows page can be used http://www.cloudping.info/
The easiest way to find EC2 AMI ID for Ubuntu Linux is by following the following site https://cloud-images.ubuntu.com/locator/ec2/
Kubernetes.tf
-The IDs you need to prepare for Kubernetes in terraform below. Accordingly, you just need to build your own structure with Resource and Output routings in Terraform"
locals = {
cluster_name = "kops.keremceliker.com"
master_autoscaling_group_ids = ["${aws_autoscaling_group.master-eu-west-2a-masters-kops-keremceliker-com.id}"]
master_security_group_ids = ["${aws_security_group.masters-kops-keremceliker-com.id}"]
masters_role_arn = "${aws_iam_role.masters-kops-keremceliker-com.arn}"
masters_role_name = "${aws_iam_role.masters-kops-keremceliker-com.name}"
node_autoscaling_group_ids = ["${aws_autoscaling_group.nodes-kops-keremceliker-com.id}"]
node_security_group_ids = ["${aws_security_group.nodes-kops-keremceliker-com.id}"]
node_subnet_ids = ["${aws_subnet.eu-west-2a-kops-keremceliker-com.id}"]
nodes_role_arn = "${aws_iam_role.nodes-kops-keremceliker-com.arn}"
nodes_role_name = "${aws_iam_role.nodes-kops-keremceliker-com.name}"
region = "eu-west-2"
route_table_public_id = "${aws_route_table.kops-keremceliker-com.id}"
subnet_eu-west-2a_id = "${aws_subnet.eu-west-2a-kops-keremceliker-com.id}"
vpc_cidr_block = "${aws_vpc.kops-keremceliker-com.cidr_block}"
vpc_id = "${aws_vpc.kops-keremceliker-com.id}"
}
*Example*
kops create cluster \
--name=$DNS_ZONE \
--state=s3://SOME_BUCKET \
--node-count=2 \
--zones=eu-west-2a,eu-west-2b \
--node-size=t2.medium \
--master-size=t2.small \
--master-size m4.large \
--dns-zone $DOMAIN \
--topology=private \
--networking=flannel \
--cloud=aws \
--ssh-public-key=./some_key \
Don't forget to look into the S3 Bucket after Kops Creation Custer, Terraform and Kops will automatically read all Codes from Cloud S3 Bucket and prepare your Kubernetes infrastructure for the AWS environment.
Kerem@Container:~terraform$ terraform init
Kerem@Container:~terraform$ terraform plan (Or use Validate)
Kerem@Container:~terraform$ terraform apply
When your POC or Project work is complete, you can remove the entire environment with the following command with the following awesome command ;-)
Kerem@Container:~terraform$ terraform destroy
kubectl get pods -o wide --all-namespaces
Once you have a Kubernetes cluster / infrastructure up and running, ssh on to the master node in gitbash. Then create secrets to be used with Web-Site and then deploy Hello Kubernetes Web Site stack using a simple cli. Follow these instructions:
kubectl get podsCreate a Yaml Script for Serverless Web-Site and make upload to Terraform Stack Sourcekubectl apply -f hello-k8s-deploy.yamlkubectl get svc. Wait a couple second or minutes until all Pods and Replicas are reported as deployed.
You must be see as the follows information
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
NGINX Public Web-Site LoadBalancer 100.64.0.1 kops.keremceliker.com 80:31958/TCP 2m26s
kubernetes ClusterIP 100.65.103.179 <none> 443/TCP 59m
Kerem@TanzuPrivateRelease$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-20-43-42.eu-west-2.compute.internal Ready node 18h v1.18
ip-172-20-46-91.eu-west-2.compute.internal Ready node 18h v1.18
ip-172-20-53-78.eu-west-2.compute.internal Ready master 18h v1.18
Kerem@TanzuPrivateRelease$ kubectl cluster-info
Kubernetes master is running at https://api.kops.keremceliker.com
KubeDNS is running at https://api.kops.keremceliker.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Kerem@TanzuPrivateRelease$ kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/my-nginx-deployment-7ddd8b97f8-225gr 1/1 Running 0 74m
kube-system pod/dns-controller-7979bc486-g7vfc 1/1 Running 0 93m
kube-system pod/etcd-manager-events-ip-172-20-53-78.eu-west-2.compute.internal 1/1 Running 0 91m
kube-system pod/etcd-manager-main-ip-172-20-53-78.eu-west-2.compute.internal 1/1 Running 0 92m
kube-system pod/kops-controller-v8g26 1/1 Running 0 92m
kube-system pod/kube-apiserver-ip-172-20-53-78.eu-west-2.compute.internal 1/1 Running 2 92m
kube-system pod/kube-controller-manager-ip-172-20-53-78.eu-west-2.compute.internal 1/1 Running 0 92m
kube-system pod/kube-dns-autoscaler-594dcb44b5-q6z67 1/1 Running 0 93m
kube-system pod/kube-dns-b84c667f4-6gb9v 3/3 Running 0 90m
kube-system pod/kube-dns-b84c667f4-94qj7 3/3 Running 0 93m
kube-system pod/kube-proxy-ip-172-20-43-42.eu-west-2.compute.internal 1/1 Running 0 82m
kube-system pod/kube-proxy-ip-172-20-46-91.eu-west-2.compute.internal 1/1 Running 0 89m
kube-system pod/kube-proxy-ip-172-20-53-78.eu-west-2.compute.internal 1/1 Running 0 91m
kube-system pod/kube-scheduler-ip-172-20-53-78.eu-west-2.compute.internal 1/1 Running 0 91m
kubernetes-dashboard pod/dashboard-metrics-scraper-c79c65bb7-ngqdj 1/1 Running 0 38m
kubernetes-dashboard pod/kubernetes-dashboard-56484d4c5-nxnff 1/1 Running 0 38m
NAMESPACE NAME TYPE CLUSTER-IP PORT(S) AGE
default service/kubernetes ClusterIP 100.64.0.1 443/TCP 93m
default service/my-nginx-service NodePort 100.65.103.179 80:31504/TCP 71m
kube-system service/kube-dns ClusterIP 100.64.0.10 53/UDP,53/TCP 93m
kubernetes-dashboard service/dashboard-metrics-scraper ClusterIP 100.65.82.6 8000/TCP 38m
kubernetes-dashboard service/kubernetes-dashboard ClusterIP 100.68.175.154 443/TCP 38m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kops-controller 1 1 1 1 1 node-role.kubernetes.io/master= 93m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default deployment.apps/my-nginx-deployment 1/1 1 1 74m
kube-system deployment.apps/dns-controller 1/1 1 1 93m
kube-system deployment.apps/kube-dns 2/2 2 2 93m
kube-system deployment.apps/kube-dns-autoscaler 1/1 1 1 93m
kubernetes-dashboard deployment.apps/dashboard-metrics-scraper 1/1 1 1 38m
kubernetes-dashboard deployment.apps/kubernetes-dashboard 1/1 1 1 38m
NAMESPACE NAME DESIRED CURRENT READY AGE
default replicaset.apps/my-nginx-deployment-7ddd8b97f8 1 1 1 74m
kube-system replicaset.apps/dns-controller-7979bc486 1 1 1 93m
kube-system replicaset.apps/kube-dns-autoscaler-594dcb44b5 1 1 1 93m
kube-system replicaset.apps/kube-dns-b84c667f4 2 2 2 93m
kubernetes-dashboard replicaset.apps/dashboard-metrics-scraper-c79c65bb7 1 1 1 38m
kubernetes-dashboard replicaset.apps/kubernetes-dashboard-56484d4c5 1 1 1 38m
If you want to easily manage your Kubernetes Infrastructure as a VMware Tanzu via vSphere GUI Interface, you can install and use the Kubernetes Dashboard as follows.
Depending if you are using a config file, via command-line run
Kerem@TanzuPrivateRelease$ kubectl proxy
or
Kerem@TanzuPrivateRelease$ kubectl --kubeconfig=kubeconfig proxy
You should get a similar response
Starting to serve on 127.0.0.1:8001
Access the Kubernetes Web Administrator Dashboard with external URL or IP address as follows
***Automatization with Tanzu/Pivotal or OpenShift, Kubernetes and DevOps in Containers processes makes your life really absolutely easier and looking cool !