nvidia-operator-validator cannot check host driver installation state but cuda-sample:vectoradd-cuda runs successfully

[davidshen84]

• OS: Gentoo Linux
• k8s: k3s v1.31.5+k3s1
• nvidia driver version: 550.144.03
• libnvidia-container version: 1.17.2
• nvidia-container-toolkit version: 1.17.3

  helm upgrade --install --wait gpu-operator-1739580441 \
       --namespace gpu-operator --create-namespace \
       nvidia/gpu-operator \
       --version=v24.9.2 \
       --set driver.enabled=false,toolkit.enabled=false

In the nvidia-operator-validator pod, the driver-validation container keeps logging the following message while the other containers are stuck in the initializing state.

Attempting to validate a driver container installation
failed to validate the driver, retrying after 5 seconds\n

However, if I run a pod like the following, note I removed the resource request and added runtimeClassName.

  apiVersion: v1
  kind: Pod
  metadata:
    name: cuda-vectoradd
  spec:
    restartPolicy: OnFailure
    runtimeClassName: nvidia
    containers:
      - name: cuda-vectoradd
        image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04

The pod ran successfully with these messages in the log.

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Note, I have set driver.enabled=false for the gpu-operator chart, but the message still says, "Attempting to validate a driver container installation". Maybe the validator should validate the driver in my host OS?

Thanks

[davidshen84]

Oh...Gentoo

On Gentoo, they put vendor-provided binaries to /opt/bin, so the nvidia-smi is at /opt/bin/nvidia-smi, while the validator expects to locate it at /host/usr/bin/nvidia-smi, which maps to /usr/bin/nvidia-smi on my host.

[planetbeing]

I'm running into the same issue on Talos, who had put it in /host/usr/local/bin/nvidia-smi. The attempted chroot /host nvidia-smi works because it's in the path.

I wouldn't blame Gentoo or Talos. Paths should never be hardcoded.

[hzhangxyz]

I'm running into the same issue on Talos, who had put it in /host/usr/local/bin/nvidia-smi. The attempted chroot /host nvidia-smi works because it's in the path.

I wouldn't blame Gentoo or Talos. Paths should never be hardcoded.

Hello, I meet the same situation, how did you resolve this on Talos?

[hzhangxyz]

I noticed there a closed PR in #1050 , it seems something broken again?

[hzhangxyz]

It seems the key point of this issue is located at

gpu-operator/cmd/nvidia-validator/main.go

Lines 769 to 781 in 0dd238e

	func (d *Driver) runValidation(silent bool) (driverInfo, error) {
	err := validateHostDriver(silent)
	if err == nil {
	log.Info("Detected a pre-installed driver on the host")
	return getDriverInfo(true, hostRootFlag, hostRootFlag, "/host"), nil
	}
	err = validateDriverContainer(silent, d.ctx)
	if err != nil {
	return driverInfo{}, err
	}
	return getDriverInfo(false, hostRootFlag, driverInstallDirFlag, driverInstallDirCtrPathFlag), nil
	}

It checks in two step, success of either of two leads to success of the whole check.

• gpu-operator/cmd/nvidia-validator/main.go

  Lines 699 to 717 in 0dd238e

  	func validateHostDriver(silent bool) error {
  	log.Info("Attempting to validate a pre-installed driver on the host")
  	if fileInfo, err := os.Lstat(filepath.Join("/host", wslNvidiaSMIPath)); err == nil && fileInfo.Size() != 0 {
  	log.Infof("WSL2 system detected, assuming driver is pre-installed")
  	disableDevCharSymlinkCreation = true
  	return nil
  	}
  	fileInfo, err := os.Lstat("/host/usr/bin/nvidia-smi")
  	if err != nil {
  	return fmt.Errorf("no 'nvidia-smi' file present on the host: %w", err)
  	}
  	if fileInfo.Size() == 0 {
  	return fmt.Errorf("empty 'nvidia-smi' file found on the host")
  	}
  	command := "chroot"
  	args := []string{"/host", "nvidia-smi"}
  	return runCommand(command, args, silent)
  	}

• gpu-operator/cmd/nvidia-validator/main.go

  Lines 732 to 752 in 0dd238e

  	driverRoot := root(driverInstallDirCtrPathFlag)
  	validateDriver := func(silent bool) error {
  	driverLibraryPath, err := driverRoot.getDriverLibraryPath()
  	if err != nil {
  	return fmt.Errorf("failed to locate driver libraries: %w", err)
  	}
  	nvidiaSMIPath, err := driverRoot.getNvidiaSMIPath()
  	if err != nil {
  	return fmt.Errorf("failed to locate nvidia-smi: %w", err)
  	}
  	cmd := exec.Command(nvidiaSMIPath)
  	// In order for nvidia-smi to run, we need to update LD_PRELOAD to include the path to libnvidia-ml.so.1.
  	cmd.Env = setEnvVar(os.Environ(), "LD_PRELOAD", prependPathListEnvvar("LD_PRELOAD", driverLibraryPath))
  	if !silent {
  	cmd.Stdout = os.Stdout
  	cmd.Stderr = os.Stderr
  	}
  	return cmd.Run()
  	}

[hzhangxyz]

According to the git blame, version before 7d3a8a2 this commit will success since it only check chroot /host nvidia-smi . but that commit add a check by the absolute path, which failed, so the first check failed since that commit.

[hzhangxyz]

The second check always fails since it stuck into finding libnvidia-ml.so.1 in several hard coded library path as

gpu-operator/cmd/nvidia-validator/find.go

Lines 29 to 45 in 0dd238e

	func (r root) getDriverLibraryPath() (string, error) {
	librarySearchPaths := []string{
	"/usr/lib64",
	"/usr/lib/x86_64-linux-gnu",
	"/usr/lib/aarch64-linux-gnu",
	"/lib64",
	"/lib/x86_64-linux-gnu",
	"/lib/aarch64-linux-gnu",
	}
	libraryPath, err := r.findFile("libnvidia-ml.so.1", librarySearchPaths...)
	if err != nil {
	return "", err
	}
	return libraryPath, nil
	}

[hzhangxyz]

OK, so why we use it well until this week?

[hzhangxyz]

Is it because fix in Talos side in siderolabs/extensions#473 ?

[planetbeing]

I'm running into the same issue on Talos, who had put it in /host/usr/local/bin/nvidia-smi. The attempted chroot /host nvidia-smi works because it's in the path.
I wouldn't blame Gentoo or Talos. Paths should never be hardcoded.

Hello, I meet the same situation, how did you resolve this on Talos?

I was unable to work around this so I was unable to get gpu-operator to work. Instead, I gave up and just used this: https://www.talos.dev/v1.11/talos-guides/configuration/nvidia-gpu-proprietary/. This is not relevant to this issue, but I'll just put it here to spare you the grief I experienced trying to get this to work. One issue my linked document didn't cover is that Node Feature Discovery must also be working, and labelling the nodes with the GPUs, or else nothing will ever get scheduled. Here is my Argo CD app for that part. You can discern that I had to set some custom arguments in order for the node to be labelled properly to satisfy NVIDIA device plugin's requirements (or I believe you can also label the node manually).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: node-feature-discovery
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io/node-feature-discovery
spec:
  project: charts
  destination:
    server: https://kubernetes.default.svc
    namespace: node-feature-discovery
  sources:
    - repoURL: https://github.com/planetbeing/home.git
      path: infrastructure/node-feature-discovery
      targetRevision: HEAD
    - repoURL: https://kubernetes-sigs.github.io/node-feature-discovery/charts
      chart: node-feature-discovery
      targetRevision: 0.17.3
      helm:
        values: |
          worker:
            config:
              sources:
                pci:
                  deviceLabelFields:
                    - vendor

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
  revisionHistoryLimit: 10

[davidshen84]

Hi there, Thank you for your detailed input on this issue. I would suggest, however, that it might be more effective to open a new thread for further discussion. This thread is quite old and marked as closed, and creating a new one would likely help in getting the attention of the developers for continued investigation and resolution of the problem. Thanks, David

…

On Wed, 10 Sept 2025, 07:45 planetbeing, ***@***.***> wrote: *planetbeing* left a comment (NVIDIA/gpu-operator#1276) <#1276 (comment)> I'm running into the same issue on Talos, who had put it in /host/usr/local/bin/nvidia-smi. The attempted chroot /host nvidia-smi works because it's in the path. I wouldn't blame Gentoo or Talos. Paths should never be hardcoded. Hello, I meet the same situation, how did you resolve this on Talos? I was unable to work around this so I was unable to get gpu-operator to work. Instead, I gave up and just used this: https://www.talos.dev/v1.11/talos-guides/configuration/nvidia-gpu-proprietary/. This is not relevant to this issue, but I'll just put it here to spare you the grief I experienced trying to get this to work. One issue my linked document didn't cover is that Node Feature Discovery must also be working, and labelling the nodes with the GPUs, or else nothing will ever get scheduled. Here is my Argo CD app for that part. You can discern that I had to set some custom arguments in order for the node to be labelled properly to satisfy NVIDIA device plugin's requirements (or I believe you can also label the node manually). apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: node-feature-discovery namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io/node-feature-discovery spec: project: charts destination: server: https://kubernetes.default.svc namespace: node-feature-discovery sources: - repoURL: https://github.com/planetbeing/home.git path: infrastructure/node-feature-discovery targetRevision: HEAD - repoURL: https://kubernetes-sigs.github.io/node-feature-discovery/charts chart: node-feature-discovery targetRevision: 0.17.3 helm: values: | worker: config: sources: pci: deviceLabelFields: - vendor syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true - PrunePropagationPolicy=foreground - PruneLast=true revisionHistoryLimit: 10 — Reply to this email directly, view it on GitHub <#1276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQBTL3MPQ2XJ5DJ4GGFA33R5C7XAVCNFSM6AAAAABXF3NBOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENZSGM4DINRQHE> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>