Merge pull request #327 from chaitanya1731/gaudi-fw-upgrade

gaudi: Added Upgrade Gaudi SPI Firmware

gaudi/Gaudi-SPI-Firmware-Upgrade.md

@@ -0,0 +1,51 @@
+# Upgrade Intel Gaudi SPI Firmware
+To upgrade Intel Gaudi SPI Firmware, follow below steps: 
+
+## Prerequisites
+- Make sure Gaudi drivers are unloaded.
+  - On Red Hat OpenShift, delete existing deviceConfig Custom Resource. Verify output on the node using below command: 
+    ```
+    $ lsmod | grep habana
+    
+    ```
+
+## SPI Firmware Upgrade
+Build the container image with `habanalabs-firmware-odm` tool:
+```
+$ oc apply -f https://raw.githubusercontent.com/intel/intel-technology-enabling-for-openshift/gaudi/gaudi_spi_fw_upgrade_build.yaml
+```
+Create service account with required permissions: 
+```
+$ oc create sa gaudi-fw-upgrade-sa -n gaudi-spi-fw-upgrade
+$ oc adm policy add-scc-to-user privileged -z gaudi-fw-upgrade-sa -n gaudi-spi-fw-upgrade
+```
+Deploy and execute the SPI firmware upgrade tool:
+```
+$ oc apply -f https://raw.githubusercontent.com/intel/intel-technology-enabling-for-openshift/gaudi/gaudi_spi_fw_upgrade_job.yaml
+```
+
+Verify Output:
+``` 
+$ oc get pods
+NAME                               READY   STATUS      RESTARTS   AGE
+gaudi-spi-firmware-upgrade-ndmjp   0/1     Completed   0          10m
+```
+```
+$ oc logs gaudi-spi-firmware-upgrade-ndmjp
+.
+.
+####
+#### Finished sending firmware: OK
+```
+Verify using `hl-smi` tool: 
+```
+sh-5.1$ hl-smi -L | grep SPI
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+    Firmware [SPI] Version          : Preboot version hl-gaudi2-1.18.0-fw-53.1.1-sec-9 (Oct 02 2024 - 11:52:39)
+```

gaudi/README.md

@@ -116,4 +116,7 @@ The resources provided are the user interface for customers to claim and consume
 
 | Feature | Resources | Description |
 | ------- | --------- | ----------- |
-| Habana Gaudi | `habana.ai/gaudi` | Number of Habana Gaudi Card resources ready to claim | 
+| Habana Gaudi | `habana.ai/gaudi` | Number of Habana Gaudi Card resources ready to claim | 
+
+## Upgrade Intel Gaudi SPI Firmware
+Refer [Upgrade Intel Gaudi SPI Firmware](/gaudi/Gaudi-SPI-Firmware-Upgrade.md) to upgrade the SPI Firmware on Intel Gaudi.

gaudi/gaudi_spi_fw_upgrade_build.yaml

@@ -0,0 +1,39 @@
+# Copyright (c) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+---
+apiVersion: image.openshift.io/v1
+kind: ImageStream
+metadata:
+  name: gaudi-spi-fw-upgrade
+  namespace: gaudi-spi-fw-upgrade
+---
+kind: BuildConfig
+apiVersion: build.openshift.io/v1
+metadata:
+  name: gaudi-spi-fw-upgrade
+  namespace: gaudi-spi-fw-upgrade
+spec:
+  output:
+    to:
+      kind: ImageStreamTag
+      name: gaudi-spi-fw-upgrade:1.18.0-524
+  strategy:
+    type: Docker
+  source:
+    type: Dockerfile
+    dockerfile: >+
+      ARG
+      BUILDER=vault.habana.ai/gaudi-docker/1.18.0/rhel9.4/habanalabs/pytorch-installer-2.4.0:1.18.0-524
+
+      FROM ${BUILDER} AS builder
+
+      RUN echo "[habanalabs]" > /etc/yum.repos.d/habanalabs.repo && \
+        echo "name=Habana RH9 Linux repo" >> /etc/yum.repos.d/habanalabs.repo && \
+        echo "baseurl=https://vault.habana.ai/artifactory/rhel/9/9.4" >> /etc/yum.repos.d/habanalabs.repo && \
+        echo "gpgkey=https://vault.habana.ai/artifactory/api/v2/repositories/rhel/keyPairs/primary/public" >> /etc/yum.repos.d/habanalabs.repo && \
+        echo "gpgcheck=1" >> /etc/yum.repos.d/habanalabs.repo
+     
+      RUN yum makecache && dnf install -y habanalabs-firmware-odm           
+  triggers:
+    - type: ConfigChange
+  runPolicy: Serial

gaudi/gaudi_spi_fw_upgrade_job.yaml

@@ -0,0 +1,21 @@
+# Copyright (c) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: gaudi-spi-firmware-upgrade
+  namespace: gaudi-spi-fw-upgrade
+spec:
+  template:
+    metadata:
+    spec:
+      restartPolicy: Never
+      serviceAccountName: gaudi-fw-upgrade-sa
+      containers:
+        - name: gaudi-spi-firmware-upgrade
+          securityContext:
+            privileged: true
+          image: image-registry.openshift-image-registry.svc:5000/gaudi-spi-fw-upgrade/gaudi-spi-fw-upgrade:1.18.0-524
+          command: [ "hl-fw-loader", "-y" ]
+          imagePullPolicy: Always