Misc GPU updates #2172

tkatila · 2025-11-19T11:41:57Z

Some cleanup (remove unused NFD consts)
Updates to levelzero components
More temperature limit options
Container size optimization for gpu_levelzero
Add 'by-path' mode option

eero-t

Some nits / comments.

eero-t · 2025-11-19T18:52:32Z

build/docker/templates/intel-gpu-levelzero.Dockerfile.in

-        apt-get update && apt-get install --no-install-recommends -y ocl-icd-libopencl1 && \N
-        rm /runtime/level-zero-devel_*.deb && \N
-        cd /runtime && dpkg -i *.deb && rm -rf /runtime && \N
+        apt-get update && apt-get install --no-install-recommends -y ocl-icd-libopencl1 wget ca-certificates && \N


I suspect the main size reduction from this duplication comes actually from removing the accidentally left (large) downloaded deb files, not from dropping l0-dev, wget, certs & their deps. Did you check that?

It mainly comes from this:
COPY --from=builder /runtime /runtime
The runtime deb packages are copied from the build to the final phase, and while they are removed after the install the copy creates a large unnecessary layer. This is evident if you open the container in dive.

Would be nice if one could install packages directly from the build phase.

docker does not support host volumes for builds (needs extension), but podman does. What if packages were on a host tmp volume (-v $(mktemp -d):/temporary:rw), I don't think those go to the final image?

cmd/gpu_levelzero/zes.c

cmd/gpu_plugin/gpu_plugin.go

cmd/gpu_levelzero/zes.c

cmd/gpu_plugin/gpu_plugin.go

eero-t

Approved. Additional container improvements don't need to be part of this PR.

Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

mythi · 2025-11-24T11:45:54Z

cmd/gpu_plugin/gpu_plugin.go

+	globalTempLimit := float64(dp.options.globalTempLimit)
+	memoryTempLimit := float64(dp.options.memoryTempLimit)
+	gpuTempLimit := float64(dp.options.gpuTempLimit)


I can see this follows the existing functionality but I wonder if we could hide this float stuff befind the levelzero API and deal with (u)ints here. or is the cgo api used elsewhere too?

Temp is in Celsius, not in Kelvin, so (in some contrived corner-case) it could be also negative. :-)

I think there are two options:

Move to integer in the CGo portion. It would change the protobuf definition and would break compatibility between older and newer plugin&levelzero containers. Probably not a big thing, but something worth thinking.

Move to integers in the levelzero-bridge golang code (inside the plugin).

I'll see how the changes would look like.

I was thinking something like 2. but it's not a blocker for this PR.

I made the change.

Use the existing "temp-limit" as the global limit, and introduce GPU and memory thresholds. Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

Fix uninitialized variable that caused random behaviour. Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

By re-downloading the components, we save on the overall container size. While the build time increases slightly, the container size drops by around 100M (520->420). Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

default behaviour stays the same. Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

tkatila · 2025-11-28T10:08:15Z

@mythi @eero-t please re-review

eero-t

IMHO by-path/ stuff docs/comments could be be clarified a bit.

eero-t · 2025-11-28T12:15:40Z

cmd/gpu_plugin/README.md

+* `single` - Symlinks are individually mounted per device. Default.
+  * Mostly Works, but is known to have issues with some pytorch workloads. See [issue](https://github.com/intel/intel-device-plugins-for-kubernetes/issues/2158).
+* `none` - No symlinks are mounted.
+  * Aligned with docker use where devices are included with privileged mode.


Suggested change

* Aligned with docker use where devices are included with privileged mode.

* Aligned with Docker `privileged` mode devices usage.

eero-t · 2025-11-28T12:17:32Z

cmd/gpu_plugin/README.md

+  * Mostly Works, but is known to have issues with some pytorch workloads. See [issue](https://github.com/intel/intel-device-plugins-for-kubernetes/issues/2158).
+* `none` - No symlinks are mounted.
+  * Aligned with docker use where devices are included with privileged mode.
+* `all` - All symlinks are mounted even if only one is allocated by the container.


Maybe something like:

Suggested change

* `all` - All symlinks are mounted even if only one is allocated by the container.

* `all` - Mounts whole DRM `by-path` directory. Pro: symlink file types are preserved. Con: symlinks are present for all devices.

eero-t · 2025-12-01T14:09:13Z

cmd/gpu_plugin/gpu_plugin.go

 	flag.StringVar(&prefix, "prefix", "", "Prefix for devfs & sysfs paths")
 	flag.BoolVar(&opts.enableMonitoring, "enable-monitoring", false, "whether to enable '*_monitoring' (= all GPUs) resource")
 	flag.BoolVar(&opts.healthManagement, "health-management", false, "enable GPU health management")
+	flag.StringVar(&opts.bypathMount, "bypath", bypathOptionSingle, "bypath mounting options: single, none, all. Default: single")


Maybe:

Suggested change

flag.StringVar(&opts.bypathMount, "bypath", bypathOptionSingle, "bypath mounting options: single, none, all. Default: single")

flag.StringVar(&opts.bypathMount, "bypath", bypathOptionSingle, "DRI device 'by-path/' directory mounting options: single, none, all. Default: single")

eero-t · 2025-12-01T14:10:08Z

deployments/operator/crd/bases/deviceplugin.intel.com_gpudeviceplugins.yaml

                type: string
+              bypathMode:
+                description: |-
+                  ByPathMode changes how plugin handles the DRM by-path mounting for GPU devices.


Suggested change

ByPathMode changes how plugin handles the DRM by-path mounting for GPU devices.

ByPathMode changes how plugin handles the DRM by-path/ directory mounting for GPU devices.

eero-t · 2025-12-01T14:10:33Z

pkg/apis/deviceplugin/v1/gpudeviceplugin_types.go

 	// +kubebuilder:validation:Enum=balanced;packed;none
 	PreferredAllocationPolicy string `json:"preferredAllocationPolicy,omitempty"`

+	// ByPathMode changes how plugin handles the DRM by-path mounting for GPU devices.


Suggested change

// ByPathMode changes how plugin handles the DRM by-path mounting for GPU devices.

// ByPathMode changes how plugin handles the DRM by-path/-dir mounting for GPU devices.

tkatila requested review from bart0sh, kad and mythi as code owners November 19, 2025 11:41

tkatila requested a review from eero-t November 19, 2025 11:42

eero-t reviewed Nov 19, 2025

View reviewed changes

tkatila force-pushed the gpu-misc-updates branch 2 times, most recently from 426558c to f270956 Compare November 20, 2025 09:30

eero-t approved these changes Nov 20, 2025

View reviewed changes

gpu: remove unused variables

3c7c5f5

Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

tkatila force-pushed the gpu-misc-updates branch from f270956 to 51cf816 Compare November 24, 2025 07:20

mythi approved these changes Nov 24, 2025

View reviewed changes

tkatila added 3 commits November 25, 2025 12:06

gpu: add support for additional temperature limits

cdb4bf3

Use the existing "temp-limit" as the global limit, and introduce GPU and memory thresholds. Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

gpu: levelzero: update compute-runtime components

c0e8e4f

Fix uninitialized variable that caused random behaviour. Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

build: levelzero: re-download compute-runtime and others

d2e3c4b

By re-downloading the components, we save on the overall container size. While the build time increases slightly, the container size drops by around 100M (520->420). Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

tkatila force-pushed the gpu-misc-updates branch from 51cf816 to d2e3c4b Compare November 25, 2025 10:15

mythi approved these changes Nov 25, 2025

View reviewed changes

tkatila added 2 commits November 28, 2025 12:02

gpu plugin: add by-path mount options

2ffbe4b

default behaviour stays the same. Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

operator: add gpu plugin's by-path option

881d9a8

Signed-off-by: Tuomas Katila <tuomas.katila@intel.com>

eero-t reviewed Dec 1, 2025

View reviewed changes

	* Aligned with docker use where devices are included with privileged mode.
	* Aligned with Docker `privileged` mode devices usage.

	* `all` - All symlinks are mounted even if only one is allocated by the container.
	* `all` - Mounts whole DRM `by-path` directory. Pro: symlink file types are preserved. Con: symlinks are present for all devices.

	flag.StringVar(&opts.bypathMount, "bypath", bypathOptionSingle, "bypath mounting options: single, none, all. Default: single")
	flag.StringVar(&opts.bypathMount, "bypath", bypathOptionSingle, "DRI device 'by-path/' directory mounting options: single, none, all. Default: single")

	ByPathMode changes how plugin handles the DRM by-path mounting for GPU devices.
	ByPathMode changes how plugin handles the DRM by-path/ directory mounting for GPU devices.

	// ByPathMode changes how plugin handles the DRM by-path mounting for GPU devices.
	// ByPathMode changes how plugin handles the DRM by-path/-dir mounting for GPU devices.

Misc GPU updates #2172

Are you sure you want to change the base?

Misc GPU updates #2172

Uh oh!

Conversation

tkatila commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eero-t left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eero-t left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tkatila commented Nov 28, 2025

Uh oh!

eero-t left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tkatila commented Nov 19, 2025 •

edited

Loading