Skip to content

Install

Jump to bottom
Jim Culbert edited this page Mar 24, 2022 · 54 revisions

Registration Application Installation

The following describes development and producton installations.

Development

The following details installation of regapp in a development environment using minikube running on kvm2.

The general steps are

  • Create a clean minikube environment
  • Setup local vault (vault/safe)
  • Install operators (kubectl)
  • Install keycloak (kubectl)
  • Install/run regapp (skaffold)

Minikube Clean Install

  • Stop and delete minikube

    • minikube stop
    • minikube delete

  • Restart minikube with olm, ingress and dashboard addons

    • Linux - minikube start --addons olm --addons dashboard --addons ingress --driver=kvm2 --memory=6G
      • minikube addons enable olm --images "UpstreamCommunityOperators=operator-framework/upstream-community-operators:latest"
    • osx - minikube start --addons olm --addons dashboard --addons ingress --driver=virtualbox

  • Configure devemopment machine hostname resolution

    • Your local host and your minikube cluster VM need to be able to resolve keycloak.mss.mghpcc.org and regapp.mss.mghcc.org.
    • osx
      • Get your cluster IP - minikube ip
      • Edit your /etc/hosts file and add entries for keycloak and regapp pointing to the cluster IP
    • Linux - Run setip.sh command (requires sudo to bounce dnsmasq instances)
      • Note: there is a bug in the config for coredns which does not handle large dns responses so we have to pin down email-smtp.us-east-1.amazonaws.com in /etc/hosts - this is a hack and potentially confuses us later but only way to make email work locally in kube-dns...

  • Validate that the minkube VM resolves the addresses correctly

    • minikube ssh
    • ping regapp.mss.mghpcc.org
    • ping keycloak.mss.mghpcc.org

  • Spin the dashboard back up

    • minikube dashboard

Configure and run local vault

  • Install vault and safe software
  • Start vault
    • vault server -dev -dev-listen-address=0.0.0.0:8200
      • 0.0.0.0 is important so that vault binds to all local addresses
  • Export production vault data
    • Logon NERC VPN
    • Login to open shift console
    • get oc logon token and logon
    • Retrieve vault root token (yuk...)
      • oc get -n default secret vault-unseal-keys -o jsonpath='{.data.vault-root}' | base64 -d
    • Validate token by visiting UI production vault
      • logon with token
    • Export
      • port forward to vault pod from local machine
        • oc port-forward -n vault vault-0 8300:8200
      • create safe target
      • Authenticate
        • safe auth token
          • will be prompted for token - provide root token
      • Test
        • safe get -k secret/stf
          • (or something similar...)
      • Export
        • safe export -k secret | openssl enc -e -aes256 -out 20220128_vault_export.enc
          • Note the "-k" option - cert is not trusted, uses same option as curl...
          • "secret" is the name of the path that we are exporting
          • Use a "good" encryption password when prompted by openssl

DISCONNECT FROM NERC VPN

  • Switch kc/oc context to minikube
  • verify you are off vpn and pointed at minikube
  • Import vault data
    • Switch vault target for safe
    • Authenticate safe to dev instance
      • safe auth token
        • Note that the root token is available in the printout in the terminal where you started vault
    • decrypt export into vault
      • openssl enc -d -aes256 -in 20220128_vault_export.enc | safe import -
        • Note: had to specify -md sha256 on mac (older version of openssl on mac defaults md5)
    • MANUAL EDITS
      • coldfront/coldfront-plugin-nese
        • NESE_ENDPOINT
        • NESE_ENDPOINT_ACCESS_KEY
        • NESE_ENDPOINT_SCHEME
        • NESE_ENDPOINT_SECRET_KEY
        • NESE_ENDPOINT_TYPE
        • NESE_ENDPOINT_UID
      • coldfront/django
        • SECRET_KEY

Clone repos

  • Clone nerc-project/nerc-k8s-operators
  • Clone nerc-project/mss-kecloak
  • Clone nerc-project/regapp

DID WE MENTION TO DISCONNECT FROM NERC VPN???

Install operators

  • Configure ExternalSecrets for local vault
    • Edit - nerc-k8s-operators/k8s/overlays/local/external-secrets/nerc-shift-secrets.yaml
    • Use root token copied from vault startup
  • kc config use-context minikube
  • kc apply -k nerc-k8s-operators/k8s/overlays/local/
    • Sometimes need to run twice as ClusterSecretStore creation fails because CRD is not ready
  • Wait for stabilization
    • make sure external secrets and keycloak operators are running/green

Install mss-keycloak

  • kc apply -k mss-keycloak/k8s/overlays/nerc-shift-1/
  • wait for stabilization
    • make sure keycloak-0 is running/green
    • check that logs are showing complete
      • kc logs -f -n keycloak keycloak-0
  • Visually verify client script mods
    • Find keycloak login credentials by visiting the dashboard and looking for the credential-mss-keycloak secret in the keycloak namespace
    • Visit https://keycloak.mss.mghpcc.org and login using credentials from secret
    • Check
      • cilogon idp mapping
      • service account roles for realm-management
      • browser flow idp redirector execution config

Install Regapp

  • Common steps

    • Install regapp namespace (tldr; kept out of base kustonization because of default delete behavior in skaffold)
      • kc apply -f regapp/kubernetes/regapp/overlays/prod/namespace.yml
    • Copy regapp KeycloakClient autogenerated secret to regapp namespace
      • regapp/kubernetes/regapp/base/regapp/copysecrets.sh

  • Non-skaffold deploy

    • kc apply -k regapp/kubernetes/regapp/overlays/dev

  • Skaffold deploy

    • Set docker redirection for minikube correctly
      • eval $(minikube docker-env)
    • Move to the regapp/kubernetes/regapp directory and execute "skaffold dev"

Install Coldfront

  • Update client key in the vault
    • Visit keycloak and retrieve the client key for the coldfront client
    • Login to the vault and find the key secret -> coldfront -> oidc
    • Set the key value OIDC_RP_CLIENT_SECRET to the secret retrieved from keycloak
  • Update keycloak admin creds in the vault
    • Get keycloak admin creds
      • kc get secret -n keycloak credential-mss-keycloak -o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d
    • Update secret -> coldfront -> keycloak-creds
      • Change value KEYCLOAK_PASS to the retrieved secret
  • Install codlfront
    • kc apply -k coldfront-nerc/k8s/overlays/prod/

PRODUCTION

Same steps as above but no minkube or localvault steps

MAGIC

  • Crunchydata restart hung cluster

  • Remote minikube

    • NB, after setting up, edits with new secrets and cluster ip info needs to be done every time you delete and then restart the cluster
    • Things you need
      • Secrets from remote minikube
      • New kubeconfig file
      • Ssh port forward for API server port (8443)
    • Secrets
      • Need to copy the following from remote in $MINIKUBE_HOME (usually ~/.minikube)
        • ca.crt
        • client.crt
        • client.key
      • Kubeconfig file
        • Create new config file; use existing one (e.g. .kube/config) as guidance
          • create cluster, context and user entries
        • Add one additional field to new cluster entry (i.e. this entry is not in "normal" .kube/config)
          • tls-server-name -> this needs to be the same as the node IP for certificate validation
        • We're going to port forward so put local end as server address
          • Note the remote end of the forward needs to be the cluster IP not remote's localhost
        • Update local KUBECONFIG envar to make life happy (colon separated list of files to merge)
          • E.g. KUBECONFIG=/home/jculbert/.kube/config:/home/jculbert/development/nerc/nerc-minikube/kube/config
Clone this wiki locally