OpenShift 4.4 RC2 User Provisioned Infrastructure (UPI) Installation on KVM/Libvirt Based on Khizer Naeem's instructions for OCP 4.2 at https://kxr.me/2019/08/17/openshift-4-upi-install-libvirt-kvm/ (updates include new HAProxy syntax, checks for DNS configuration, 4.4 download links, tmux instead of screen)
cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 Beta
virsh version Compiled against library: libvirt 4.5.0 Using library: libvirt 4.5.0 Using API: QEMU 4.5.0 Running hypervisor: QEMU 2.12.0
virsh version --daemon Compiled against library: libvirt 4.5.0 Using library: libvirt 4.5.0 Using API: QEMU 4.5.0 Running hypervisor: QEMU 2.12.0 Running against daemon: 4.5.0
subscription-manager register subscription-manager list --available subscription-manager attach --pool <pool>
Disable IPv6 vi /etc/sysctl.d/70-ipv6.conf Next, add the following lines and save the file. net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 sysctl --load /etc/sysctl.d/70-ipv6.conf
dnf -y update && dnf -y upgrade dnf -y install libvirt* bind-utils wget tar python3 virt-install virt-viewer libguestfs-tools-c tmux httpd-tools git systemctl start libvirtd.service systemctl enable libvirtd
echo -e "[main]\ndns=dnsmasq" > /etc/NetworkManager/conf.d/nm-dns.conf systemctl restart NetworkManager
Check DHCP enabled
virsh net-dumpxml default
<network>
<name>default</name>
<uuid>cc1ad354-d794-441f-a434-4f8215ddce91</uuid>
<forward mode='nat'>
<nat>
<port start='1024' end='65535'/>
</nat>
</forward>
<bridge name='virbr0' stp='on' delay='0'/>
<mac address='52:54:00:4c:22:83'/>
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254'/>
</dhcp>
</ip>
</network>
mkdir ocp4 && cd ocp4
VIR_NET="default"
HOST_NET=$(ip -4 a s $(virsh net-info $VIR_NET | awk '/Bridge:/{print $2}') | awk '/inet /{print $2}')
HOST_IP=$(echo $HOST_NET | cut -d '/' -f1)
[root@dell-r730-001 ocp4]# echo $HOST_NET
192.168.122.1/24
[root@dell-r730-001 ocp4]# echo $HOST_IP
192.168.122.1
DNS_DIR="/etc/NetworkManager/dnsmasq.d"
The first entry in /etc/resolv.conf should be "nameserver 127.0.0.1".
Make sure that any entry in /etc/hosts is forward and reverse resolvable by libvirt/kvm.
You can test this by adding a test record in /etc/hosts:
echo "1.2.3.4 test.local" >> /etc/hosts
and then restart libvirtd so it picks up the hosts file:
systemctl restart libvirtd
and finally check if the forward and reverse lookup works:
dig test.local @${HOST_IP}
dig -x 1.2.3.4 @${HOST_IP}
Verify that you get answers in both the above dig queries.
Make sure that any entry in the dnsmasq.d is also picked up by libvirt/kvm.
You can test this by adding a test srv record:
echo "srv-host=test.local,yayyy.local,2380,0,10" > ${DNS_DIR}/temp-test.conf
systemctl reload NetworkManager
test that both libvirt and your host can resolve the srv record:
dig srv test.local
dig srv test.local @${HOST_IP}
verify that you get the srv answer (yayy.local) in both the above dig queries.
ssh-keygen BASE_DOM="local" CLUSTER_NAME="ocp44" SSH_KEY="/root/.ssh/id_rsa.pub" PULL_SEC='<your pull secret from https://cloud.redhat.com/openshift/install/metal/user-provisioned>'
mkdir rhcos-install wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.4/latest/rhcos-4.4.0-rc.1-x86_64-installer-kernel-x86_64 -O rhcos-install/vmlinuz wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.4/latest/rhcos-4.4.0-rc.1-x86_64-installer-initramfs.x86_64.img -O rhcos-install/initramfs.img
cat <<EOF > rhcos-install/.treeinfo [general] arch = x86_64 family = Red Hat CoreOS platforms = x86_64 version = 4.4.0 [images-x86_64] initrd = initramfs.img kernel = vmlinuz EOF
wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.4/latest/rhcos-4.4.0-rc.1-x86_64-metal.x86_64.raw.gz
Go to
https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.1/x86_64/product-software
and copy the download link of Red Hat Enterprise Linux KVM Guest Image
(right-click on "Download Now" and copy link location)
wget "https://access.cdn.redhat.com/content/.../rhel-8.1-x86_64-kvm.qcow2?user=...&_auth_=..." -O /var/lib/libvirt/images/${CLUSTER_NAME}-lb.qcow2
RHNUSER='your Red Hat Customer Portal username'
RHNPASS='<your Red Hat Customer Portal password'
wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.4.0-rc.2/openshift-install-linux-4.4.0-rc.2.tar.gz wget https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.4.0-rc.2/openshift-client-linux-4.4.0-rc.2.tar.gz tar xf openshift-client-linux-4.4.0-rc.2.tar.gz tar xf openshift-install-linux-4.4.0-rc.2.tar.gz rm -f README.md
mkdir install_dir
cat <<EOF > install_dir/install-config.yaml
apiVersion: v1
baseDomain: ${BASE_DOM}
compute:
- hyperthreading: Disabled
name: worker
replicas: 0
controlPlane:
hyperthreading: Disabled
name: master
replicas: 6
metadata:
name: ${CLUSTER_NAME}
networking:
clusterNetworks:
- cidr: 10.128.0.0/14
hostPrefix: 23
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
pullSecret: '${PULL_SEC}'
sshKey: '$(cat $SSH_KEY)'
EOF
./openshift-install create ignition-configs --dir=./install_dir
Start python3 webserver
WEB_PORT=8000 tmux new -s webserver bash -c "python3 -m http.server" curl http://localhost:8000 Useful tmux commands: ctrl-b d to detach tmux ls tmux attach-session -t webserver
If using firewalld
firewall-cmd --add-source=${HOST_NET}
firewall-cmd --add-port=${WEB_PORT}/tcp
If using iptables
iptables -I INPUT -p tcp -m tcp --dport ${WEB_PORT} -s ${HOST_NET} -j ACCEPT
virt-install --name ${CLUSTER_NAME}-bootstrap \
--disk size=50 --ram 24000 --cpu host --vcpus 4 \
--os-type linux --os-variant rhel7.0 \
--network network=${VIR_NET} --noreboot --noautoconsole \
--location rhcos-install/ \
--extra-args "nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.inst.image_url=http://${HOST_IP}:${WEB_PORT}/rhcos-4.4.0-rc.1-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://${HOST_IP}:${WEB_PORT}/install_dir/bootstrap.ign"
for i in {1..6}
do
virt-install --name ${CLUSTER_NAME}-master-${i} \
--disk size=50 --ram 24000 --cpu host --vcpus 4 \
--os-type linux --os-variant rhel7.0 \
--network network=${VIR_NET} --noreboot --noautoconsole \
--location rhcos-install/ \
--extra-args "nomodeset rd.neednet=1 coreos.inst=yes coreos.inst.install_dev=vda coreos.inst.image_url=http://${HOST_IP}:${WEB_PORT}/rhcos-4.4.0-rc.1-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://${HOST_IP}:${WEB_PORT}/install_dir/master.ign"
done
virsh list --all
virt-customize -a /var/lib/libvirt/images/${CLUSTER_NAME}-lb.qcow2 \
--uninstall cloud-init \
--ssh-inject root:file:$SSH_KEY --selinux-relabel \
--sm-credentials "${RHNUSER}:password:${RHNPASS}" \
--sm-register --sm-attach auto --install haproxy
virt-install --import --name ${CLUSTER_NAME}-lb \
--disk /var/lib/libvirt/images/${CLUSTER_NAME}-lb.qcow2 --memory 24000 --cpu host --vcpus 2 \
--network network=${VIR_NET} --noreboot --noautoconsole
echo "local=/${CLUSTER_NAME}.${BASE_DOM}/" > ${DNS_DIR}/${CLUSTER_NAME}.conf
for x in lb bootstrap master-1 master-2 master-3 master-4 master-5 master-6
do
virsh start ${CLUSTER_NAME}-$x
done
for x in lb bootstrap master-1 master-2 master-3 master-4 master-5 master-6
do
virsh start ${CLUSTER_NAME}-$x
done
Domain ocp44-lb started
Domain ocp44-bootstrap started
Domain ocp44-master-1 started
Domain ocp44-master-2 started
Domain ocp44-master-3 started
Domain ocp44-master-4 started
Domain ocp44-master-5 started
Domain ocp44-master-6 started
virsh list --all
Ensure that all machines received an IP and MAC:
for x in lb bootstrap master-1 master-2 master-3 master-4 master-5 master-6
do
virsh --connect qemu:///system reboot "${CLUSTER_NAME}-$x"
echo `virsh domifaddr "${CLUSTER_NAME}-$x"`
done
for x in lb bootstrap master-1 master-2 master-3 master-4 master-5 master-6
> do
> virsh --connect qemu:///system reboot "${CLUSTER_NAME}-$x"
> echo `virsh domifaddr "${CLUSTER_NAME}-$x"`
> done
Domain ocp44-lb is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet0 52:54:00:01:03:0f ipv4 192.168.122.47/24
Domain ocp44-bootstrap is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet1 52:54:00:72:55:b2 ipv4 192.168.122.94/24
Domain ocp44-master-1 is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet2 52:54:00:67:b2:9d ipv4 192.168.122.224/24
Domain ocp44-master-2 is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet3 52:54:00:56:e4:89 ipv4 192.168.122.37/24
Domain ocp44-master-3 is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet4 52:54:00:21:4d:38 ipv4 192.168.122.214/24
Domain ocp44-master-4 is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet5 52:54:00:88:7a:3e ipv4 192.168.122.46/24
Domain ocp44-master-5 is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet6 52:54:00:f2:f5:64 ipv4 192.168.122.89/24
Domain ocp44-master-6 is being rebooted
Name MAC address Protocol Address ------------------------------------------------------------------------------- vnet7 52:54:00:14:c9:e4 ipv4 192.168.122.153/24
IP=$(virsh domifaddr "${CLUSTER_NAME}-bootstrap" | grep ipv4 | head -n1 | awk '{print $4}' | cut -d'/' -f1)
MAC=$(virsh domifaddr "${CLUSTER_NAME}-bootstrap" | grep ipv4 | head -n1 | awk '{print $2}')
virsh net-update ${VIR_NET} add-last ip-dhcp-host --xml "<host mac='$MAC' ip='$IP'/>" --live --config
Updated network default persistent config and live state
echo "$IP bootstrap.${CLUSTER_NAME}.${BASE_DOM}" >> /etc/hosts
for i in {1..6}
do
IP=$(virsh domifaddr "${CLUSTER_NAME}-master-${i}" | grep ipv4 | head -n1 | awk '{print $4}' | cut -d'/' -f1)
MAC=$(virsh domifaddr "${CLUSTER_NAME}-master-${i}" | grep ipv4 | head -n1 | awk '{print $2}')
virsh net-update ${VIR_NET} add-last ip-dhcp-host --xml "<host mac='$MAC' ip='$IP'/>" --live --config
echo "$IP master-${i}.${CLUSTER_NAME}.${BASE_DOM}" \
"etcd-$((i-1)).${CLUSTER_NAME}.${BASE_DOM}" >> /etc/hosts
echo "srv-host=_etcd-server-ssl._tcp.${CLUSTER_NAME}.${BASE_DOM},etcd-$((i-1)).${CLUSTER_NAME}.${BASE_DOM},2380,0,10" >> ${DNS_DIR}/${CLUSTER_NAME}.conf
done
LBIP=$(virsh domifaddr "${CLUSTER_NAME}-lb" | grep ipv4 | head -n1 | awk '{print $4}' | cut -d'/' -f1)
MAC=$(virsh domifaddr "${CLUSTER_NAME}-lb" | grep ipv4 | head -n1 | awk '{print $2}')
virsh net-update ${VIR_NET} add-last ip-dhcp-host --xml "<host mac='$MAC' ip='$LBIP'/>" --live --config
echo "$LBIP lb.${CLUSTER_NAME}.${BASE_DOM}" \
"api.${CLUSTER_NAME}.${BASE_DOM}" \
"api-int.${CLUSTER_NAME}.${BASE_DOM}" >> /etc/hosts
cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.122.94 bootstrap.ocp44.local 192.168.122.224 master-1.ocp44.local etcd-0.ocp44.local 192.168.122.37 master-2.ocp44.local etcd-1.ocp44.local 192.168.122.214 master-3.ocp44.local etcd-2.ocp44.local 192.168.122.46 master-4.ocp44.local etcd-3.ocp44.local 192.168.122.89 master-5.ocp44.local etcd-4.ocp44.local 192.168.122.153 master-6.ocp44.local etcd-5.ocp44.local 192.168.122.47 lb.ocp44.local api.ocp44.local api-int.ocp44.local
echo "address=/apps.${CLUSTER_NAME}.${BASE_DOM}/${LBIP}" >> ${DNS_DIR}/${CLUSTER_NAME}.conf
echo "server=/${CLUSTER_NAME}.${BASE_DOM}/${LBIP}" >> ${DNS_DIR}/${CLUSTER_NAME}.conf
cat ${DNS_DIR}/${CLUSTER_NAME}.conf
local=/ocp44.local/
srv-host=_etcd-server-ssl._tcp.ocp44.local,etcd-0.ocp44.local,2380,0,10
srv-host=_etcd-server-ssl._tcp.ocp44.local,etcd-1.ocp44.local,2380,0,10
srv-host=_etcd-server-ssl._tcp.ocp44.local,etcd-2.ocp44.local,2380,0,10
srv-host=_etcd-server-ssl._tcp.ocp44.local,etcd-3.ocp44.local,2380,0,10
srv-host=_etcd-server-ssl._tcp.ocp44.local,etcd-4.ocp44.local,2380,0,10
srv-host=_etcd-server-ssl._tcp.ocp44.local,etcd-5.ocp44.local,2380,0,10
address=/apps.ocp44.local/192.168.122.47
server=/ocp44.local/192.168.122.47
ssh lb.${CLUSTER_NAME}.${BASE_DOM} <<EOF
# Allow haproxy to listen on custom ports
semanage port -a -t http_port_t -p tcp 6443
semanage port -a -t http_port_t -p tcp 22623
semanage port -a -t http_port_t -p tcp 443
semanage port -a -t http_port_t -p tcp 80
semanage port -a -t http_port_t -p tcp 8443
echo '
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats
defaults
mode tcp
log global
option tcplog
option dontlognull
option redispatch
retries 3
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s
maxconn 3000
# 6443 points to control plan
frontend ${CLUSTER_NAME}-api
bind *:6443
default_backend master-api
backend master-api
balance source
server bootstrap bootstrap.${CLUSTER_NAME}.${BASE_DOM}:6443 check
server master-1 master-1.${CLUSTER_NAME}.${BASE_DOM}:6443 check
server master-2 master-2.${CLUSTER_NAME}.${BASE_DOM}:6443 check
server master-3 master-3.${CLUSTER_NAME}.${BASE_DOM}:6443 check
server master-4 master-4.${CLUSTER_NAME}.${BASE_DOM}:6443 check
server master-5 master-5.${CLUSTER_NAME}.${BASE_DOM}:6443 check
server master-6 master-6.${CLUSTER_NAME}.${BASE_DOM}:6443 check
# 22623 points to control plane
frontend ${CLUSTER_NAME}-mapi
bind *:22623
default_backend master-mapi
backend master-mapi
balance source
server bootstrap bootstrap.${CLUSTER_NAME}.${BASE_DOM}:22623 check
server master-1 master-1.${CLUSTER_NAME}.${BASE_DOM}:22623 check
server master-2 master-2.${CLUSTER_NAME}.${BASE_DOM}:22623 check
server master-3 master-3.${CLUSTER_NAME}.${BASE_DOM}:22623 check
server master-4 master-4.${CLUSTER_NAME}.${BASE_DOM}:22623 check
server master-5 master-5.${CLUSTER_NAME}.${BASE_DOM}:22623 check
server master-6 master-6.${CLUSTER_NAME}.${BASE_DOM}:22623 check
# 80 points to worker nodes
frontend ${CLUSTER_NAME}-http
bind *:80
default_backend ingress-http
backend ingress-http
balance source
server master-1 master-1.${CLUSTER_NAME}.${BASE_DOM}:80 check
server master-2 master-2.${CLUSTER_NAME}.${BASE_DOM}:80 check
server master-3 master-3.${CLUSTER_NAME}.${BASE_DOM}:80 check
server master-4 master-4.${CLUSTER_NAME}.${BASE_DOM}:80 check
server master-5 master-5.${CLUSTER_NAME}.${BASE_DOM}:80 check
server master-6 master-6.${CLUSTER_NAME}.${BASE_DOM}:80 check
# 443 points to worker nodes
frontend ${CLUSTER_NAME}-https
bind *:443
default_backend infra-https
backend infra-https
balance source
server master-1 master-1.${CLUSTER_NAME}.${BASE_DOM}:443 check
server master-2 master-2.${CLUSTER_NAME}.${BASE_DOM}:443 check
server master-3 master-3.${CLUSTER_NAME}.${BASE_DOM}:443 check
server master-4 master-4.${CLUSTER_NAME}.${BASE_DOM}:443 check
server master-5 master-5.${CLUSTER_NAME}.${BASE_DOM}:443 check
server master-6 master-6.${CLUSTER_NAME}.${BASE_DOM}:443 check
' > /etc/haproxy/haproxy.cfg
systemctl start haproxy
systemctl enable haproxy
EOF
ssh lb.${CLUSTER_NAME}.${BASE_DOM} dnf -y install net-tools
ssh lb.${CLUSTER_NAME}.${BASE_DOM} systemctl start haproxy
ssh lb.${CLUSTER_NAME}.${BASE_DOM} systemctl status haproxy
ssh lb.${CLUSTER_NAME}.${BASE_DOM} netstat -nltupe | grep ':6443\|:22623\|:80\|:443'
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 0 32495 4113/haproxy
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 0 32496 4113/haproxy
tcp 0 0 0.0.0.0:22623 0.0.0.0:* LISTEN 0 32494 4113/haproxy
tcp 0 0 0.0.0.0:6443 0.0.0.0:* LISTEN 0 32492 4113/haproxy
systemctl reload NetworkManager systemctl restart libvirtd
for x in lb bootstrap master-1 master-2 master-3 master-4 master-5 master-6
do
virsh --connect qemu:///system reboot "${CLUSTER_NAME}-$x"
echo `virsh domifaddr "${CLUSTER_NAME}-$x"`
done
ping -c 1 master-1.ocp44.local ping -c 1 master-2.ocp44.local ping -c 1 master-3.ocp44.local ping -c 1 master-4.ocp44.local ping -c 1 master-5.ocp44.local ping -c 1 master-6.ocp44.local ping -c 1 bootstrap.ocp44.local ping -c 1 etcd-0.ocp44.local ping -c 1 etcd-1.ocp44.local ping -c 1 etcd-2.ocp44.local ping -c 1 etcd-3.ocp44.local ping -c 1 etcd-4.ocp44.local ping -c 1 etcd-5.ocp44.local ping -c 1 api.ocp44.local ping -c 1 api-int.ocp44.local
The DNS lookup for the API endpoints also needs to be in place. OpenShift 4 expects api.$CLUSTERDOMAIN and api-int.$CLUSTERDOMAIN to be configured, they can both be set to the same IP address – which will be the IP of the Load Balancer. ping -c 1 api.ocp44.local ping -c 1 api-int.ocp44.local
A wildcard DNS entry needs to be in place for the OpenShift 4 ingress router, which is also a load balanced endpoint. dig *.apps.ocp44.local +short 192.168.122.205
In addition to the mentioned entries, you’ll also need to add SRV records. These records are needed for the masters to find the etcd servers. This needs to be in the form of _etcd-server-ssl._tcp.$CLUSTERDOMMAIN in your DNS server.
dig _etcd-server-ssl._tcp.ocp44.local SRV +short 0 10 2380 etcd-0.ocp44.local. 0 10 2380 etcd-1.ocp44.local. 0 10 2380 etcd-2.ocp44.local. 0 10 2380 etcd-3.ocp44.local. 0 10 2380 etcd-4.ocp44.local. 0 10 2380 etcd-5.ocp44.local.
ssh lb.${CLUSTER_NAME}.${BASE_DOM} systemctl start haproxy
ssh lb.${CLUSTER_NAME}.${BASE_DOM} systemctl status haproxy
./openshift-install --dir=install_dir wait-for bootstrap-complete
ssh core@bootstrap.ocp44.local journalctl -b -f -u bootkube.service
You'll see journal entries like 9_openshift-machineconfig_99-master-ssh.yaml": unable to get REST mapping for "99_openshift-machineconfig_99-master-ssh.yaml": no matches for kind "MachineConfig" in version "machineconfiguration.openshift.io/v1 These messages go away as soon as CRD definition is created (by machine-config operator).
./openshift-install --dir=install_dir wait-for bootstrap-complete INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp43.local:6443... INFO API v1.16.2 up INFO Waiting up to 30m0s for bootstrapping to complete... INFO It is now safe to remove the bootstrap resources
cd ocp4 export KUBECONFIG=install_dir/auth/kubeconfig cp ./oc /usr/bin
BASE_DOMAIN="$(oc get dns.config/cluster -o 'jsonpath={.spec.baseDomain}')"
INGRESS_DOMAIN="$(oc get ingress.config/cluster -o 'jsonpath={.spec.domain}')"
echo $BASE_DOMAIN
ocp44.local
echo $INGRESS_DOMAIN
apps.ocp44.local
openssl genrsa -out marc-example-ca.key 2048
openssl req -x509 -new -key marc-example-ca.key -out marc-example-ca.crt -days 1000 -subj "/C=US/ST=NC/L=Chocowinity/O=OS3/OU=Eng/CN=$BASE_DOMAIN"
openssl genrsa -out marc-example.key 2048
openssl req -new -key marc-example.key -out marc-example.csr -subj "/C=US/ST=NC/L=Chocowinity/O=OS3/OU=Eng/CN=*.$INGRESS_DOMAIN"
openssl x509 -req -in marc-example.csr -CA marc-example-ca.crt -CAkey marc-example-ca.key -CAcreateserial -out marc-example.crt -days 1000
2. Configure the CA as the cluster proxy CA:
oc -n openshift-config create configmap marc-custom-ca --from-file=ca-bundle.crt=marc-example-ca.crt
oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"marc-custom-ca"}}}'
3. Configure the certificate as the ingresscontroller's default certificate:
oc -n openshift-ingress create secret tls marc-custom-default-cert --cert=marc-example.crt --key=marc-example.key
oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"defaultCertificate":{"name":"marc-custom-default-cert"}}}'
oc -n openshift-config create configmap marc-custom-ca --from-file=ca-bundle.crt=marc-example-ca.crt
oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"marc-custom-ca"}}}'
oc -n openshift-config create configmap custom-ca --from-file=ca-bundle.crt=marc-example-ca.crt
oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'
Follow the steps at https://github.com/marcredhat/workshop/blob/master/imageregistry.adoc
Follow the steps at https://github.com/marcredhat/workshop/blob/master/userauth_htpasswd.adoc
Follow the steps at https://blog.openshift.com/openshift-4-3-alertmanager-configuration/
./oc get nodes master-1.ocp44.local Ready master,worker 9m26s v1.16.2 master-2.ocp44.local Ready master,worker 9m20s v1.16.2 master-3.ocp44.local Ready master,worker 9m6s v1.16.2 master-4.ocp44.local Ready master,worker 9m26s v1.16.2 master-5.ocp44.local Ready master,worker 9m20s v1.16.2 master-6.ocp44.local Ready master,worker 9m6s v1.16.2
./oc get csr -o name | xargs oc adm certificate approve
./oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
watch "./oc get clusterversion; echo; ./oc get clusteroperators"
Get kubeadmin's password cat install_dir/auth/kubeadmin-password
On your local machine, add console-openshift-console.apps.ocp44.local and oauth-openshift.apps.ocp43.local to /etc/hosts, pointing to 127.0.0.1
sudo ssh root@<your KVM host> -L 443:console-openshift-console.apps.ocp44.local:443
You can now connect to https://console-openshift-console.apps.ocp44.local as kubeadmin with the password you got from install_dir/auth/kubeadmin-password.
You can also connect to the console as the user configured in the "Create user / authentication using htpasswd" paragraph
Let's deploy a DaemonSet so that we get a container running on each worker node.
oc create sa samarc oc adm policy add-scc-to-user privileged -z samarc oc adm policy add-scc-to-user anyuid -z default -n `oc project -q` --as=system:admin
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: tcpdump
labels:
marc: tcpdump
spec:
selector:
matchLabels:
marc: tcpdump
template:
metadata:
labels:
marc: tcpdump
spec:
nodeSelector:
marc: tcpdumpnode
containers:
- name: tcpdump
image: corfr/tcpdump
securityContext:
#runAsUser: 0
allowPrivilegeEscalation: true
privileged: true
command:
- bin/sleep
- infinity
serviceAccount: samarc
serviceAccountName: samarc
oc label node master-1.ocp44.local marc=tcpdumpnode oc label node master-2.ocp44.local marc=tcpdumpnode oc label node master-3.ocp44.local marc=tcpdumpnode oc label node master-4.ocp44.local marc=tcpdumpnode oc label node master-5.ocp44.local marc=tcpdumpnode oc label node master-6.ocp44.local marc=tcpdumpnode
oc get pods -o wide --all-namespaces | grep tcpdump ovntests tcpdump-2qnl6 1/1 Running 0 14m 10.130.0.46 master-6.ocp44.local <none> <none> ovntests tcpdump-7b8mn 1/1 Running 0 14m 10.131.0.45 master-1.ocp44.local <none> <none> ovntests tcpdump-98hch 1/1 Running 0 14m 10.128.0.62 master-5.ocp44.local <none> <none> ovntests tcpdump-flcv6 1/1 Running 0 14m 10.129.2.63 master-4.ocp44.local <none> <none> ovntests tcpdump-frvjm 1/1 Running 0 14m 10.129.0.70 master-2.ocp44.local <none> <none> ovntests tcpdump-xwjgp 1/1 Running 0 14m 10.128.2.28 master-3.ocp44.local <none> <none>
oc rsh tcpdump-2qnl6 while true ; do echo -e "HTTP/1.1 200 OK\n\n $(date)" | nc -l -p 3306 ; done
oc rsh tcpdump-7b8mn wget 10.130.0.46:3306
oc new-app https://github.com/IBM/deploy-python-openshift-tutorial oc rsh deploy-python-openshift-tutorial-1-kgf9h wget 10.130.0.46:3306
oc rsh ovnkube-node-md96m ovn-nbctl set-connection ptcp:6641
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: allow3306
spec:
podSelector:
matchLabels:
marc: tcpdump
ingress:
- from:
- podSelector:
matchLabels:
marc: tcpdump
ports:
- protocol: TCP
port: 3306
In my case, the IP address seen from inside each container are 10.128.2.17 and 10.129.0.17
oc rsh tcpdump-9prsr
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
link/ether 0a:58:0a:81:00:11 brd ff:ff:ff:ff:ff:ff
inet 10.129.0.17/23 brd 10.129.1.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::c45d:ddff:febc:8a53/64 scope link
valid_lft forever preferred_lft forever
/ # ping 10.128.2.17
PING 10.128.2.17 (10.128.2.17): 56 data bytes
64 bytes from 10.128.2.17: seq=0 ttl=64 time=1.876 ms
64 bytes from 10.128.2.17: seq=1 ttl=64 time=0.468 ms
64 bytes from 10.128.2.17: seq=2 ttl=64 time=0.389 ms
^C
--- 10.128.2.17 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.389/0.911/1.876 ms
In OpenShift we only have a switch (br0) although OpenvSwitch allows to have more. Each switch contains multiple ports. The vxlan0 is responsible for encapsulating packets. The tun0 is the node IP address on the overlay, the rest of ports are pods.
ssh core@worker-1.ocp43.local [core@worker-1 ~]$ sudo ovs-ofctl show -O OpenFlow13 br0 | grep tun0 2(tun0): addr:8e:89:0a:37:b8:64
[core@worker-1 ~]$ sudo ovs-appctl ofproto/trace br0 in_port=2,tcp,nw_src=10.129.0.17,nw_dst=10.128.2.17
Flow: tcp,in_port=2,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,nw_src=10.129.0.17,nw_dst=10.128.2.17,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=0,tcp_flags=0
bridge("br0")
-------------
0. ct_state=-trk,ip, priority 300
ct(table=0)
drop
-> A clone of the packet is forked to recirculate. The forked pipeline will be resumed at table 0.
-> Sets the packet to an untracked state, and clears all the conntrack fields.
Final flow: unchanged
Megaflow: recirc_id=0,ct_state=-trk,eth,ip,in_port=2,nw_src=10.129.0.16/28,nw_frag=no
Datapath actions: ct,recirc(0x696a5)
===============================================================================
recirc(0x696a5) - resume conntrack with default ct_state=trk|new (use --ct-next to customize)
===============================================================================
Flow: recirc_id=0x696a5,ct_state=new|trk,eth,tcp,in_port=2,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,nw_src=10.129.0.17,nw_dst=10.128.2.17,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=0,tcp_flags=0
bridge("br0")
-------------
thaw
Resuming from table 0
0. ip,in_port=2,nw_src=10.129.0.0/23,nw_dst=10.128.0.0/14, priority 300
goto_table:25
25. ip,nw_src=10.129.0.17, priority 100
load:0->NXM_NX_REG0[]
goto_table:30
30. ip,nw_dst=10.128.0.0/14, priority 100
goto_table:90
90. ip,nw_dst=10.128.2.0/23, priority 100, cookie 0x57bbdba6
move:NXM_NX_REG0[]->NXM_NX_TUN_ID[0..31]
-> NXM_NX_TUN_ID[0..31] is now 0
set_field:192.168.122.73->tun_dst
output:1
-> output to kernel tunnel
Final flow: recirc_id=0x696a5,ct_state=new|trk,eth,tcp,tun_src=0.0.0.0,tun_dst=192.168.122.73,tun_ipv6_src=::,tun_ipv6_dst=::,tun_gbp_id=0,tun_gbp_flags=0,tun_tos=0,tun_ttl=0,tun_erspan_ver=0,tun_flags=0,in_port=2,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=00:00:00:00:00:00,nw_src=10.129.0.17,nw_dst=10.128.2.17,nw_tos=0,nw_ecn=0,nw_ttl=0,tp_src=0,tp_dst=0,tcp_flags=0
Megaflow: recirc_id=0x696a5,ct_state=-rpl+trk,eth,ip,tun_id=0/0xffffffff,tun_dst=0.0.0.0,in_port=2,nw_src=10.129.0.17,nw_dst=10.128.2.0/23,nw_ecn=0,nw_frag=no
Datapath actions: set(tunnel(tun_id=0x0,dst=192.168.122.73,ttl=64,tp_dst=4789,flags(df|key))),2
NOTE: 192.168.122.73 is worker-2 (where the destination pod runs)
Sources: https://developers.redhat.com/blog/2016/10/12/tracing-packets-inside-open-vswitch/ http://docs.openvswitch.org/en/latest/topics/tracing/ https://developers.redhat.com/blog/2019/02/27/sidecars-analyze-debug-network-traffic-kubernetes-pod/ https://developers.redhat.com/blog/2019/08/22/troubleshooting-red-hat-openshift-applications-with-throwaway-containers/ https://medium.com/@NTTICT/vxlan-explained-930cc825a51
Gathering logs from failed installations:
./openshift-install gather bootstrap --dir=./install_dir --bootstrap bootstrap.${CLUSTER_NAME}.${BASE_DOM} --master master-1.${CLUSTER_NAME}.${BASE_DOM} --master master-2.${CLUSTER_NAME}.${BASE_DOM} --master master-3.${CLUSTER_NAME}.${BASE_DOM}