Releases · CluedIn-io/Charts

Chart 2.0.0 upgrade

Stage 1: Prepare custom values

First of all migrate your existing configuration to be compatible with v2. Be aware of the following known breaking changes.

Breaking changes

Neo4j

Before	After	Comment

infrastructure.neo4j.authEnabled	infrastructure.neo4j.config.dbms.security.auth_enabled
infrastructure.neo4j.password	infrastructure.neo4j.neo4j.password
infrastructure.neo4j.image	infrastructure.neo4j.cluedinExtensions.image.[registry/repository]	Now it's split into two fields
infrastructure.neo4j.imageTag	infrastructure.neo4j.cluedinExtensions.image.tag
infrastructure.neo4j.imagePullSecret	infrastructure.neo4j.image.imagePullSecrets	Notice, it's a list now
infrastructure.neo4j.priorityClassName	infrastructure.neo4j.podSpec.priorityClassName
infrastructure.neo4j.tolerations	infrastructure.neo4j.podSpec.tolerations
infrastructure.neo4j.serviceAccount.name	infrastructure.neo4j.podSpec.serviceAccountName
infrastructure.neo4j.dbms.memory.use_memrec	-	not used anymore, as Neo4j does it OOTB
infrastructure.neo4j.core.standalone	-	not used anymore
infrastructure.neo4j.core.numberOfServers	-	not used anymore
infrastructure.neo4j.core.discoveryService	-	not used anymore
infrastructure.neo4j.core.persistentVolume.enabled	infrastructure.neo4j.volumes.data.mode	equivalent. Can disable persistence using custom template
infrastructure.neo4j.core.persistentVolume.size	infrastructure.neo4j.volumes.data.defaultStorageClass.requests.storage
-	infrastructure.neo4j.cluedinExtensions.podCommand
infrastructure.neo4j.core.pluginInstallers	infrastructure.neo4j.cluedinExtensions.pluginInstallers
infrastructure.neo4j.core.resources.requests	infrastructure.neo4j.neo4j.resources.requests

Note that for some settings you have neo4j.neo4j - it's not a mistake, you have to nest same key again.

Stage 2: Apply upgrade values

Run helm upgrade with your custom values using the latest v2 HELM chart. On top of your values you have to apply platform-upgrade-v2-stage1.yml values file.

platform-upgrade-v2-stage1.yml

global:
  containerImages:
    initSql:
      enabled: false
    initNeo4J:
      enabled: false
    initCluedIn:
      enabled: false

application:
  annotation:
    replicas: 0
  cluedin:
    roles:
      crawling:
        count: 0
      main:
        count: 0
      processing:
        count: 0
  cluedincontroller:
    enabled: false
  datasource:
    replicas: 0
  gql:
    replicas: 0
  libpostal:
    enabled: false
  monitoring:
    enabled: false
  oauth2:
    enabled: false
  openrefine:
    enabled: false
  prepare:
    replicas: 0
  strategy:
    replicas: 0
  submitter:
    replicas: 0
  ui:
    replicas: 0

infrastructure:
  cert-manager:
    enabled: false
  haproxy-ingress:
    enabled: false
  elasticsearch:
    enabled: false
  monitoring:
    enabled: false
  mssql:
    enabled: false
  neo4j:
    enabled: false
  rabbitmq:
    enabled: false
  redis:
    enabled: false

Example:

helm upgrade -n cluedin my-name cluedin/cluedin-platform
    --values my-custom-values.yaml
    --values platform-upgrade-v2-stage1.yml

⚠ Upgrade values should be applied at the end to override any other values

❗ Wait until deployment finishes. Make sure that all pods are healthy and all jobs are completed.

As result most of the resources should be undeployed, but your volumes and claims should remain.

Stage 3: Apply upgrade-2 values for data-upgrade

Run helm upgrade with your custom values using the latest v2 HELM chart. On top of your values you have to apply platform-upgrade-v2-stage2 values file.

platform-upgrade-v2-stage2.yml

global:
  containerImages:
    initSql:
      enabled: true
    initNeo4J:
      enabled: true
    initCluedIn:
      enabled: false # Don't run nuget installers for now, focus on infrastructure upgrade

# Disable services which should not run during upgrade
application:
  annotation:
    replicas: 0
  cluedin:
    roles:
      crawling:
        count: 0
      main:
        count: 0
      processing:
        count: 0
  cluedincontroller:
    enabled: false
  datasource:
    replicas: 0
  gql:
    replicas: 0
  libpostal:
    enabled: false
  monitoring:
    enabled: false
  oauth2:
    enabled: false
  openrefine:
    enabled: false
  prepare:
    replicas: 0
  strategy:
    replicas: 0
  submitter:
    replicas: 0
  ui:
    replicas: 0

  system:
    runDatabaseJobsOnUpgrade: true # Will run the init jobs to migrate databases

infrastructure:
  cert-manager:
    enabled: false
  haproxy-ingress:
    enabled: false
  elasticsearch:
    enabled: false
  monitoring:
    enabled: false
  redis:
    enabled: false

  mssql:
    command:
    - /usr/bin/stdbuf
    - -i0
    - -o0
    - -e0
    - /bin/bash
    - -c
    - |
      ### LOGGING SETUP
      UPGRADE_LOG_FILE_PATH=$MSSQL_DATA_DIR/upgrade_2017-2022.upgrade.log

      counter=1
      UPGRADE_LOG_FILE_PATH_CURRENT="${UPGRADE_LOG_FILE_PATH%.upgrade.log}.$((counter++)).upgrade.log"
      while [ -e "$UPGRADE_LOG_FILE_PATH_CURRENT" ]; do
        UPGRADE_LOG_FILE_PATH_CURRENT="${UPGRADE_LOG_FILE_PATH%.upgrade.log}.$((counter++)).upgrade.log"
      done

      exec > >(tee $${UPGRADE_LOG_FILE_PATH_CURRENT}) 2>&1
      echo "✍ Upgrade log file path: $UPGRADE_LOG_FILE_PATH_CURRENT"
      ### END LOGGING SETUP

      echo "⌛ [UPGRADE] Changing database file permissions to mssql"

      for env_var in MSSQL_DATA_DIR MSSQL_LOG_DIR MSSQL_BACKUP_DIR MSSQL_MASTER_DATA_FILE MSSQL_MASTER_LOG_FILE; do
          if [ "${!env_var}" ]; then
              chown -R mssql -v ${!env_var}
              echo "[UPGRADE] ✔ Changed owner to mssql for ${env_var}"
          fi
      done

      for env_var in MSSQL_MASTER_DATA_FILE MSSQL_MASTER_LOG_FILE; do
          if [ "${!env_var}" ] && [ -f "${!env_var}" ]; then
              parentdir="$(dirname "${!env_var}")"
              chown -R mssql -v ${parentdir}
              echo "[UPGRADE] ✔ Changed owner to mssql for ${env_var} parent directory"
          fi
      done

      echo "[UPGRADE] ✅ Changed database file permissions to mssql"

      echo "[UPGRADE] 🎁 - Starting MS SQL as mssql user.."
      exec su -c "/opt/mssql/bin/sqlservr --accept-eula" mssql
    # Run as root, as we need to change database file owner
    securityContext:
      runAsNonRoot: false
      runAsUser: 0
      runAsGroup: 0
  
  rabbitmq:
    initContainers:
      - name: rabbit-3-11-enable-feature-flags
        image: "{{ printf \"%s/%s:%s\"  $.Values.image.registry $.Values.image.repository \"3.11.20-debian-11-r25\" }}"
        command:
        - /usr/bin/stdbuf
        - -i0
        - -o0
        - -e0
        - /bin/bash
        - -c
        - |
          ### LOGGING SETUP
          counter=1
          UPGRADE_LOG_FILE_PATH_CURRENT="${UPGRADE_LOG_FILE_PATH%.upgrade.log}.$((counter++)).upgrade.log"
          while [ -e "$UPGRADE_LOG_FILE_PATH_CURRENT" ]; do
            UPGRADE_LOG_FILE_PATH_CURRENT="${UPGRADE_LOG_FILE_PATH%.upgrade.log}.$((counter++)).upgrade.log"
          done

          exec > >(tee $${UPGRADE_LOG_FILE_PATH_CURRENT}) 2>&1
          echo "✍ Upgrade log file path: $UPGRADE_LOG_FILE_PATH_CURRENT"
          ### END LOGGING SETUP

          # Function to handle the interrupt signal and exit the script gracefully
          exit_upon_signal() {
              echo "❌ Received interrupt signal. Exiting..."
              exit 1
          }
          trap exit_upon_signal SIGINT SIGTERM

          echo "[UPGRADE] 🟢 Running RabbitMQ 3.11 Upgrade script"

          if [[ -z "${UPGRADE_LOCK_FILE_PATH}" ]]; then
              echo "[UPGRADE] ⛔ - Env variable LOCK_FILE_PATH is not set"
 ...

Features

[cluedin-application] Removed webapi deployment from helm charts
[cluedin-application] Added priority classes for the deployments
[cluedin-application] Added pod anti-affinity for application high-availability

Production resource values:

                   Requests          Limits

Service	cpu	mem (mb)	cpu	mem (mb)
cluedin-annotation	0.25	512	1	4096
cluedin-controller	0.5	512	0.5	512
cluedin-Datasource	0.5	1024	1	6144
cluedin-openrefine		2045		4096
cluedin-prepare	0.25	128	0.75	512
cluedin-server	0.5	2048	2	12288
cluedin-processing	7	10240	7	12288
cluedin-submitter	1	256	1	512
cluedin-UI	0.5	1024	0.5	1024
cluedin-Gql	0.5	2048	0.5	2048
cluedin-Documentation	0.125	64	0.5	512
prometheus	0.5	512	0.5	1024
Grafana	0.125	64	0.5	512
HA-Proxy	0.125	64	0.5	512
rabbitmq	2	6144	2	6144
Elastic-search	7	26624	7	26624
Neo4j	7	26624	7	26624
Redis	0.5	2048	0.5	2048
cert-manager	0.010	64	0.025	64
cert-manager-cainjector	0.010	64	0.025	64
cert-manager-webhook	0.010	64	0.025	64

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Features

Fixes

Features

Fixes

Features

Fixes

Features

Fixes

Features

Fixes

Features

Fixes

Features

Fixes

Features

Fixes

Chart 2.0.0 upgrade

Stage 1: Prepare custom values

Breaking changes

Stage 2: Apply upgrade values

Stage 3: Apply upgrade-2 values for data-upgrade

Features

Fixes

Releases: CluedIn-io/Charts

Release cluedin-platform-2.3.2

Features

Fixes

Release cluedin-platform-2.4.1

Features

Fixes

Release cluedin-platform-2.4.0

Features

Fixes

Release cluedin-platform-2.4.0-rc.1

Features

Fixes

Release cluedin-platform-2.3.1

Features

Fixes

Release cluedin-platform-2.3.0

Features

Fixes

Release cluedin-platform-2.2.0

Features

Fixes

Release cluedin-platform-2.1.0

Features

Fixes

Release cluedin-platform-2.0.0

Chart 2.0.0 upgrade

Stage 1: Prepare custom values

Breaking changes

Stage 2: Apply upgrade values

Stage 3: Apply upgrade-2 values for data-upgrade

Release cluedin-platform-1.6.0

Features

Fixes