This project create a Kubernetes cluster using terraform on Oracle Cloud, also enable the seamless synchronization of secrets among interconnected projects. This synchronization allows these projects to deploy their applications on the cluster effectively.
You’ll want to ensure you have the following already installed on your local machine before getting started:
- Clone/Fork this repository
- Click here to create Github classic tokens and store them as
PERSONAL_GITHUB_TOKEN_CLASSIC
in repository secret under Setting -> Secrets and variables -> Actions - Login to Oracle Cloud, Click on Profile Icon and your username e.g. "oracleidentitycloudservice/{{email}}"
- Click on API Keys under "Resources" section
- Click "Add Key" and supply an API Key (You can use Generate API Key Pair option if you do not have a key)
- Click on "three dots" and view Configuration file
- Paste the content of the file with correct key_file location (keys generated in step 5) in "~/.oci/config" as instructed
# To verify oci authentication
oci iam compartment list --compartment-id-in-subtree=true
- Setup following secrets in your repository by visiting Setting -> Secrets and variables -> Actions
- Add following secrets in the repository based on configuration file under Setting -> Secrets and variables -> Actions
OCI_CLI_USER
OCI_CLI_TENANCY
OCI_CLI_FINGERPRINT
OCI_CLI_KEY_CONTENT
OCI_CLI_REGION
- Click on "Auth Tokens" in the left hand side under "Resources" tab
- Generate token and store as
OCI_AUTH_TOKEN
in Repository secrets - Click here to find the OCID of a Compartment - ((Direct Link)[https://cloud.oracle.com/identity/compartments?region=uk-london-1])
- Store
OCI_COMPARTMENT_OCID
in Repository secrets - Visit Oracle Cloud and search "Buckets" OR Click Burger Menu -> Storage -> Object Storage & Archive Storage -> Buckets
- Click "Create Bucket" and name it appropriately because it will save terraform states. I usually call them "terraform-backend"
- Update following files
- Replace bucket name
- Replace endpoint, it is usually "https://$OCI_TENANCY_NAMESPACE.compat.objectstorage.$OCI_REGION.oraclecloud.com"
- Name key appropriately or leave it as it is.
terraform/01-infrastructure/terraform.tf
terraform/03-network-loadbalancer/terraform.tf
- Click on Profile Icon and your username in Oracle Cloud Dashboard, click "Customer Secret Keys" in the Resources sidebar.
- Click "Generate Secret Key", name appropriately e.g. "Terraform Backend State"
- Copy secret key and store as
AWS_SECRET_ACCESS_KEY
in Repository secrets - Copy access key against "Terraform Backend State" and store as
AWS_ACCESS_KEY_ID
in Repository secrets - Push all changes, Visit Actions tab in your repository and manually run " 01 - Deploy Core Infrastructure Changes"
- Create kubeconfig file locally
oci ce cluster create-kubeconfig --cluster-id $OCI_COMPARTMENT_OCID --file ~/.kube/oci --region $OCI_CLI_REGION --token-version 2.0.0 --kube-endpoint PUBLIC_ENDPOINT --overwrite
export KUBECONFIG=~/.kube/oci
- Run the following command to see the nodes and their labels. It will display NodeId - Label - Display Name OCID
kubectl get nodes --output=jsonpath='{range .items[*]}{@.metadata.name}{"\t"}{@.metadata.labels.kubernetes\.io/hostname}{"\t"}{@.metadata.labels.displayName}{"\t"}{@.spec.providerID}{"\n"}{end}'
# In case you want to see node pool id
kubectl get nodes --output=jsonpath='{range .items[*]}{@.metadata.name}{"\t"}{@.metadata.labels.kubernetes\.io/hostname}{"\t"}{@.metadata.labels.displayName}{"\t"}{@.spec.providerID}{"\t"}{@.metadata.annotations.oci\.oraclecloud\.com/node-pool-id}{"\n"}{end}'
- Label each node so that we can target them for pod placement later on
kubectl label node NODE_ID_1 "kubernetes.io/hostname"="main" --overwrite
kubectl label node NODE_ID_2 "kubernetes.io/hostname"="node1" --overwrite
...
kubectl label node NODE_ID_4 "kubernetes.io/hostname"="node4" --overwrite
- Copy the OCID of main node and store it as Repository Secret
OCI_MAIN_INSTANCE_OCID
in Github.
cd terraform/02-traefik
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm upgrade --install traefik traefik/traefik \
--namespace kube-system --create-namespace -f values.yaml
kubectl apply -f http-to-https-redirect.yaml
helm upgrade -i clustersecret-operator oci://ghcr.io/sap/clustersecret-operator-helm/clustersecret-operator
- Goto Repository -> Actions -> 03 - Deploy OCI Network Loadbalancer
- Run the action manually
- You can use public ip address in DNS setup
kubectl get pods ---selector app=nginx --all-namespaces
kubectl port-forward service/traefik 32080:80
kubectl port-forward deployment/nginx-deployment 32080:80
kubectl port-forward service/traefik 9000:9000
kubectl port-forward $(kubectl get pods --selector "app.kubernetes.io/name=traefik" --output=name -n kube-system) -n kube-system 9000:9000
kubectl exec --stdin --tty $(kubectl get pods --selector "app=nginx" --output=name) -- /bin/bash
kubectl logs $(kubectl get pods --selector "app.kubernetes.io/instance=traefik-kube-system" --output=name -n kube-system) -n kube-system -f
kubectl logs $(kubectl get pods --selector "app=nginx" --output=name) -f
kubectl cp $(kubectl get pods --selector "app=nginx" --output=name | head -n 1 | awk -F/ '{print $2}'):/usr/share/nginx/html/index.html ./index.html
kubectl cp ./index.html $(kubectl get pods --selector "app=nginx" --output=name | head -n 1 | awk -F/ '{print $2}'):/usr/share/nginx/html/index.html
kubectl get pods --all-namespaces -o=json | jq -c '.items[] | {name: .metadata.name, namespace: .metadata.namespace, claimName:.spec.volumes[] | select( has ("persistentVolumeClaim") ).persistentVolumeClaim.claimName }'
kubectl patch pv <volume-name> -p '{"spec":{"claimRef": null}}'
cd mysql-database/
kubectl create namespace database
kubectl apply -f mysql.yaml --namespace database
kubectl port-forward service/mysql-service 3306:3306 --namespace database
cd mysql-database/
kubectl create namespace phpmyadmin
# replace <<usernamehere>> and <<passwordhere>> in phpmyadmin.yaml
kubectl apply -f phpmyadmin.yaml --namespace phpmyadmin
CREATE DATABASE mydatabase;
CREATE USER 'myuser'@'%' IDENTIFIED BY 'mypassword';
GRANT ALL PRIVILEGES ON mydatabase.* TO 'chatapp'@'%';
kubectl create secret generic database-secret --from-literal=DB_PASS=$(echo -n 'mypassword' | base64) --namespace projectnamespace
- Click on Profile icon in Oracle Cloud and visit "Tenancy: {{yourid}}" link
- Copy "Object storage namespace" and record as
OCI_TENANCY_NAMESPACE
- Find Region ID from Availability Zones. For example UK South London id is "lhr" and store the following host as
DOCKER_HOST
in secrets
{region-id}.ocir.io
- Store
DOCKER_USERNAME
as following, whereOCI_USERNME
is same as appeared in Oracle Cloud under Profile icon "{{OCI_TENANCY_NAMESPACE}}/oracleidentitycloudservice/{{email}}"
echo $OCI_AUTH_TOKEN | docker login $DOCKER_HOST --username=$DOCKER_USERNAME --password-stdin