feat(ci): wanda ray image builds, uploads to Dockerhub

[andrew-anyscale]

Adds Dockerfile and Wanda files for Ray images, pulling artifacts from previous steps. Buildkite steps also updated to use these, including both build and upload steps.

Topic: ray-image
Relative: push-script

Signed-off-by: andrew andrew@anyscale.com

[andrew-anyscale]

Reviews in this chain:
└#59936 feat(ci): wanda ray image builds, uploads to Dockerhub
　└#59937 Add wanda anyscale image builds for release tests

[andrew-anyscale]

#	head	base	diff	date	summary
0	02a8921d	f104bf59	diff	Jan 7 7:57 AM	9 files changed, 593 insertions(+), 49 deletions(-)
1	fe687866	288cadb6	diff	Jan 7 7:58 AM	0 files changed
2	46cc8625	288cadb6	diff	Jan 7 8:38 AM	4 files changed, 440 insertions(+), 31 deletions(-)
3	70df7561	164e2e4c	rebase	Jan 7 11:29 AM	0 files changed
4	ae32d45a	b4fb3e06	rebase	Jan 7 12:21 PM	0 files changed
5	04cfd26a	cd792539	rebase	Jan 7 17:29 PM	0 files changed
6	55d1473f	7d257e47	rebase	Jan 7 17:31 PM	0 files changed
7	84d74797	e93a821d	diff	Jan 8 8:27 AM	1 file changed, 6 insertions(+), 14 deletions(-)
8	c275fe24	8a4bdf5b	rebase	Jan 8 8:32 AM	0 files changed
9	7c78fa2d	a0b03e94	rebase	Jan 8 12:01 PM	0 files changed
10	29cf5abd	32347fa5	rebase	Jan 8 12:51 PM	0 files changed
11	d566c8a8	8d1057a9	diff	Jan 8 13:16 PM	0 files changed
12	91900297	d8901f73	rebase	Jan 8 13:33 PM	0 files changed
13	73822f53	9e438258	rebase	Jan 8 14:30 PM	0 files changed
14	94805e95	6801f72f	rebase	Jan 9 8:45 AM	0 files changed
15	fc79d67d	9834ea38	rebase	Jan 9 10:01 AM	0 files changed
16	db94f935	25e0d03c	rebase	Jan 9 10:59 AM	0 files changed
17	05c06707	2a361721	rebase	Jan 9 12:36 PM	0 files changed
18	a6718508	b81d4cb4	rebase	Jan 9 13:32 PM	0 files changed
19	51fe33a5	4933e1ce	rebase	Jan 13 8:55 AM	0 files changed
20	ed74c8c7	19d388a4	rebase	Jan 13 10:55 AM	0 files changed
21	6242ab25	7c1782d4	rebase	Jan 13 15:50 PM	0 files changed
22	a941fd54	09f88fb0	rebase	Jan 13 15:50 PM	0 files changed
23	9db9513a	c7f89bff	rebase	Jan 13 17:03 PM	0 files changed
24	6d4a0beb	e2ebabb6	rebase	Jan 13 17:09 PM	0 files changed
25	dd5133b8	7b8a44a4	rebase	Jan 13 17:09 PM	0 files changed
26	ae00964a	1410defe	rebase	Jan 13 17:12 PM	0 files changed
27	021b2e13	64ac2fd8	rebase	Jan 13 17:14 PM	0 files changed
28	52b7b213	d5b11ffd	rebase	Jan 13 17:33 PM	0 files changed
29	8cfa977c	524d02c4	diff	Jan 13 21:52 PM	0 files changed
30	924d46f1	d1f94807	diff	Jan 15 13:10 PM	10 files changed, 476 insertions(+), 432 deletions(-)
31	275827a2	d1f94807	diff	Jan 15 13:13 PM	0 files changed
32	04d68065	545eefc9	diff	Jan 15 13:14 PM	1 file changed, 11 insertions(+)
33	6e985e93	545eefc9	diff	Jan 15 13:40 PM	3 files changed, 13 insertions(+), 15 deletions(-)
34	17d793f4	545eefc9	diff	Jan 15 13:47 PM	1 file changed, 3 insertions(+), 10 deletions(-)
35	716cc651	545eefc9	diff	Jan 15 16:04 PM	9 files changed, 47 insertions(+), 14 deletions(-)
36	976be9fb	545eefc9	diff	Jan 15 16:48 PM	7 files changed, 2 insertions(+), 14 deletions(-)
37	f4e1306c	a0e25707	diff	Jan 15 17:41 PM	0 files changed
38	73da8387	06d411f4	rebase	Jan 15 18:02 PM	0 files changed
39	c547d3f5	06d411f4	diff	Jan 16 9:22 AM	1 file changed, 4 insertions(+), 1 deletion(-)
40	1f64272b	c97824d8	diff	Jan 16 13:28 PM	0 files changed
41	a6bdfece	dfa84607	rebase	Jan 16 14:05 PM	0 files changed
42	6a668ed7	6efc0092	diff	Jan 20 10:30 AM	7 files changed, 23 insertions(+), 20 deletions(-)

[gemini-code-assist]

Code Review

This pull request successfully migrates the Ray Docker image builds to use wanda, which simplifies the CI configuration. The new wanda.yaml files and the shared Dockerfile are well-structured. However, I've identified a few areas for improvement, primarily in the new push_ray_image.py script. There's a critical issue where the script only supports pushing the main ray images, which is a regression from the previous system that also handled ray-extra and ray-llm variants. Additionally, the script duplicates significant portions of the image tagging logic from another file, which will create maintenance challenges. I've also included a suggestion to reduce duplication in the .buildkite/build.rayci.yml file using YAML anchors.

[gemini-code-assist]

The GPU_PLATFORM constant is hardcoded here and is also defined in ci/ray_ci/docker_container.py. Duplicating constants can lead to inconsistencies if one is updated and the other is not.

This constant should be defined in a single, shared location and imported where needed. This could be done as part of the larger refactoring of the tag generation logic I mentioned in another comment.

[andrew-anyscale]

Results from #60186:

=== Comparing images ===
Image 1: 029272617770.dkr.ecr.us-west-2.amazonaws.com/rayproject/citemp@sha256:5f324c6e299db38c60f1361847ad351788ba9808e0cf5462924bc0469ee01ee7
Image 2: rayproject/ray:nightly.260115.f18213

Fetching image configs...
=== Config differences (excluding history/rootfs) ===
--- /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/config1.json	2026-01-15 16:32:23
+++ /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/config2.json	2026-01-15 16:32:23
@@ -15,18 +15,17 @@
     ],
     "WorkingDir": "/home/ray",
     "Labels": {
-      "io.ray.ray-commit": "0d911cd230fbcf90963348136f3693df658cd8f2",
+      "io.ray.ray-commit": "f182131f846c8de5eef21ad6a0621aa30b6005cb",
       "io.ray.ray-version": "3.0.0.dev0",
       "org.opencontainers.image.ref.name": "ubuntu",
       "org.opencontainers.image.version": "22.04"
     },
-    "ArgsEscaped": true,
     "OnBuild": null,
     "Shell": [
       "/bin/bash",
       "-c"
     ]
   },
-  "created": "2026-01-16T00:20:02.976635141Z",
+  "created": "2026-01-15T07:05:55.735088841Z",
   "os": "linux"
 }

=== Environment variables ===
--- Image 1 ---
--- Image 2 ---
--- Diff ---

=== Labels ===
--- Image 1 ---
{
  "io.ray.ray-commit": "0d911cd230fbcf90963348136f3693df658cd8f2",
  "io.ray.ray-version": "3.0.0.dev0",
  "org.opencontainers.image.ref.name": "ubuntu",
  "org.opencontainers.image.version": "22.04"
}
--- Image 2 ---
{
  "io.ray.ray-commit": "f182131f846c8de5eef21ad6a0621aa30b6005cb",
  "io.ray.ray-version": "3.0.0.dev0",
  "org.opencontainers.image.ref.name": "ubuntu",
  "org.opencontainers.image.version": "22.04"
}
--- Diff ---
--- /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/labels1.json	2026-01-15 16:32:25
+++ /var/folders/dg/jkv044t9525bdy3bq2ksxwjh0000gp/T/tmp.5mtTUMytHA/labels2.json	2026-01-15 16:32:26
@@ -1,5 +1,5 @@
 {
-  "io.ray.ray-commit": "0d911cd230fbcf90963348136f3693df658cd8f2",
+  "io.ray.ray-commit": "f182131f846c8de5eef21ad6a0621aa30b6005cb",
   "io.ray.ray-version": "3.0.0.dev0",
   "org.opencontainers.image.ref.name": "ubuntu",
   "org.opencontainers.image.version": "22.04"

=== Entrypoint/Cmd ===
--- Image 1 ---
{
  "Entrypoint": null,
  "Cmd": [
    "/bin/bash"
  ]
}
--- Image 2 ---
{
  "Entrypoint": null,
  "Cmd": [
    "/bin/bash"
  ]
}

=== Working Dir ===
--- Image 1 ---
/home/ray
--- Image 2 ---
/home/ray

=== User ===
--- Image 1 ---
1000
--- Image 2 ---
1000

=== Exposed Ports ===
--- Image 1 ---
{}
--- Image 2 ---
{}

=== Architecture/OS ===
--- Image 1 ---
{
  "architecture": "amd64",
  "os": "linux"
}
--- Image 2 ---
{
  "architecture": "amd64",
  "os": "linux"
}

=== Important Files ===
Pip freeze diff summary (packages that differ):
< ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=bc4f590d0b8248746ffd20b5da7348f294fa508dadac8188221df0318d9551bd
> ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=680cc52fb4116ef3565a1d1df5975228672965d57d355792cac767c48473ffef
(no differences)

--- Ray wheel location in pip-freeze ---
Image 1:
ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=bc4f590d0b8248746ffd20b5da7348f294fa508dadac8188221df0318d9551bd
Image 2:
ray @ file:///home/ray/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#sha256=680cc52fb4116ef3565a1d1df5975228672965d57d355792cac767c48473ffef

[andrew-anyscale]

This will be deprecated in favor of an envfile--

leaning towards having an env_file support in wanda spec/config that can read in build args to set, so that we can do labeling in the dockerfile that are based on file contents

[cursor]

Glob validation fails to detect zero wheel files

Low Severity

The wheel file validation logic uses a bash glob pattern without setting nullglob, so if no files match /home/ray/ray-*.whl, the array contains the unexpanded literal pattern string as one element. The check ${#WHEEL_FILES[@]} -ne 1 then incorrectly passes, and the error message would claim "found 1" wheel file when actually there are zero. The pip install would still fail eventually, but with a confusing "file not found" error rather than the intended descriptive message.

 

[aslonnie]

can we inline these args' default values in the Dockerfile? and remove them from here?

[andrew-anyscale]

The challenge is that the same ci/docker/ray-image.Dockerfile is used across all of ray-extra-image-cpu.wanda.yaml, ray-image-cpu.wanda.yaml, etc.

I can see if we can still keep the same reduced set of Dockerfiles, while also pushing the default values into the Dockerfile itself

[andrew-anyscale]

Options are to either inline some degree of default values:

# wanda.yaml
build_args:
  - PYTHON_VERSION
  - ARCH_SUFFIX
  - IMAGE_TYPE=ray-llm
  - PLATFORM=cu$CUDA_VERSION
  - BASE_VARIANT=base-extra
  - RAY_COMMIT=$BUILDKITE_COMMIT
  - RAY_VERSION
 
# Dockerfile
ARG PYTHON_VERSION=3.10
ARG PLATFORM=cpu
ARG ARCH_SUFFIX=
ARG IMAGE_TYPE=ray
ARG BASE_VARIANT=base
ARG BASE_IMAGE=cr.ray.io/rayproject/${IMAGE_TYPE}-py${PYTHON_VERSION}-${PLATFORM}-${BASE_VARIANT}${ARCH_SUFFIX}
ARG RAY_WHEEL_IMAGE=cr.ray.io/rayproject/ray-wheel-py${PYTHON_VERSION}${ARCH_SUFFIX}

Or split out into N different Dockerfiles with defaults backed into the Dockerfile itself

Longer term, I lean more towards keeping a low subset of Dockerfiles. Out of curiosity, what would benefits be of pushing the args' default values into the Dockerfile if it comes at this cost?

[andrew-anyscale]

Resolved offline. Benefit is removing the re-use of the same declared build_args that are redundantly passed to create the image

[aslonnie]

can we inline these args' default values in the Dockerfile? and remove them from here?

in the dockerfile, you would need to write:

ARG PYTHON_VERSION
ARG ARCH_SUFFIX
ARG CUDA_VERSION
ARG BASE_IMAGE=cr.ray.io/rayproject/ray-llm-py$PYTHON_VERSION-cu$CUDA_VERSION-base-extra$ARCH_SUFFIX

FROM $BASE_IMAGE