This document describes the steps necessary to set up an EdgeNet cluster of your own. It assumes Ubuntu 16.04 (or later) as the base machine for the head node and worker nodes.
The EdgeNet Portal that grants you access to nodes must be set up separately, see https://github.com/EdgeNet-project/portal/
This step is identical for the head node and worker nodes.
Note: Please see the latest Kubernetes setup instructions for the most up-to-date information.
apt-get update && apt-get install -y apt-transport-https -y
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat << EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install docker.io kubelet kubeadm kubectl kubernetes-cni -y
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab
This installs the kubeadm cluster bootstrap tool, the web-based Kubernetes
Dashboard, and the configs to instantiate the flannel
container networking fabric on the head node.
kubeadm init --pod-network-cidr=192.168.0.0/16
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
sysctl net.bridge.bridge-nf-call-iptables=1
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
These need to be run to allow users to view namespaces and nodes at clusters scope.
kubectl create -f https://raw.githubusercontent.com/EdgeNet-project/edgenet/master/user_files/yml/namespace-viewer.yml
kubectl create -f https://raw.githubusercontent.com/EdgeNet-project/edgenet/master/user_files/yml/namespace-viewer-crb.yml
kubectl create -f https://raw.githubusercontent.com/EdgeNet-project/edgenet/master/user_files/yml/node-viewer.yml
kubectl create -f https://raw.githubusercontent.com/EdgeNet-project/edgenet/master/user_files/yml/node-viewer-crb.yml
This command will make Kubernetes Dashboard available at http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/.
kubectl proxy
Finally, set up reboot recovery for control plane, as per these instructions.
Run this on all Ubuntu worker nodes to add them to your cluster.
sudo kubeadm join --token <token><IP of Head Node>:6443 --discovery-token-ca-cert-hash <cert hash from head>
(much of that will be generated by the head node during its setup, see above)
NB: The node will be added with the name that matches its hostname on the
master, so if its address is not routable, you need to change this in the
kubelet config file using sudo vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
and add the flag --node-name to match in the join command with the routable
address.
At this point, you should (on the head node) be able to run:
$ kubectl get nodes
# The response looks like this:
(headnode IP) Ready master 23m v1.9.6
(Node-1 IP) Ready <none> 18m v1.9.6
(Node-2 IP) Ready <none> 14m v1.9.6
(Node-3 IP) Ready <none> 10m v1.9.6
This is what a basic cluster looks like, with nothing yet running on it. Next, we set up some users.
We need to enable HTTPS on our head node, and there we can get the benefit of EFF's Certbot by deploying Let's Encrypt certificates. The request calls made to default server at port 80 needs to be passed to HTTPS endpoints.
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name headnode.example.com;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
if ($host = headnode.example.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
return 404; # managed by Certbot
}
Then we need to listen port 8080 ssl and port 443 ssl to proxy pass to EdgeNet Head Node and Kubernetes Dashboard. The calls made to port 8080 passes through the EdgeNet Head Node while the calls made to port 443 passes to Kubernetes Dashboard as you could see below.
server {
listen 8080 ssl;
listen [::]:8080 ssl;
server_name headnode.example.com; # managed by Certbot
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $remote_addr;
proxy_pass http://127.0.0.1:8181;
}
ssl_certificate /etc/letsencrypt/live/headnode.example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/headnode.example.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
server_name headnode.example.com;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
location / {
#try_files $uri $uri/ =404;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Host $remote_addr;
proxy_pass http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/;
}
ssl_certificate /etc/letsencrypt/live/headnode.example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/headnode.example.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
After we clone the head node repository, we just need to be inside of the build directory and run the command below. The command will handle the deployment process.
docker-compose up --build
Then the EdgeNet Head Node Application is ready to use as containerized application.