Migrate wheel builds to wanda (x86_64) #59935

andrew-anyscale · 2026-01-07T15:57:48Z

This migrates ray wheel builds from CLI-based approach to wanda-based
container builds for x86_64.

Changes:

Add ray-wheel.wanda.yaml and Dockerfile for wheel builds
Update build.rayci.yml wheel steps to use wanda
Add wheel upload steps that extract from wanda cache

Topic: ray-wheel

andrew-anyscale · 2026-01-07T15:57:52Z

Reviews in this chain:
└#59935 Migrate wheel builds to wanda (x86_64)
　└#59969 Add wanda cpp wheel builds (x86_64)
　　└#59936 Add wanda ray image builds for Docker Hub
　　　└#59937 Add wanda anyscale image builds for release tests

andrew-anyscale · 2026-01-07T15:57:52Z

#	head	base	diff	date	summary
0	f104bf59	be77f0aa	diff	Jan 7 7:57 AM	8 files changed, 369 insertions(+), 7 deletions(-)
1	288cadb6	be77f0aa	diff	Jan 7 7:58 AM	0 files changed
2	164e2e4c	be77f0aa	diff	Jan 7 11:29 AM	1 file changed, 22 insertions(+)
3	b4fb3e06	be77f0aa	diff	Jan 7 12:21 PM	1 file changed, 1 insertion(+), 1 deletion(-)
4	cd792539	be77f0aa	diff	Jan 7 17:29 PM	1 file changed, 1 insertion(+), 1 deletion(-)
5	7d257e47	be77f0aa	diff	Jan 7 17:31 PM	1 file changed, 20 deletions(-)
6	e93a821d	9700991f	diff	Jan 8 8:27 AM	5 files changed, 53 insertions(+), 179 deletions(-)
7	8a4bdf5b	9700991f	diff	Jan 8 8:32 AM	2 files changed, 6 insertions(+), 17 deletions(-)
8	a0b03e94	8c3a7135	diff	Jan 8 12:01 PM	1 file changed, 14 insertions(+), 9 deletions(-)
9	32347fa5	8c3a7135	diff	Jan 8 12:51 PM	1 file changed, 12 insertions(+), 1 deletion(-)
10	9074421e	8c3a7135	diff	Jan 8 13:33 PM	1 file changed, 9 insertions(+), 41 deletions(-)
11	1b2c341f	8c3a7135	diff	Jan 8 14:30 PM	0 files changed
12	05785cb4	8c3a7135	diff	Jan 9 8:45 AM	2 files changed, 15 insertions(+), 32 deletions(-)
13	37410f9a	f580a273	diff	Jan 9 10:01 AM	1 file changed, 6 insertions(+), 3 deletions(-)

gemini-code-assist

Code Review

This pull request migrates the x86_64 wheel builds from a CLI-based approach to wanda-based container builds. This involves adding new wanda configuration files, Dockerfiles, and updating the Buildkite CI configuration. The changes look good and are a solid step towards modernizing the build process. I have a couple of suggestions to improve maintainability and build efficiency.

gemini-code-assist · 2026-01-07T16:03:18Z

.buildkite/build.rayci.yml

    commands:
-      - bazel run //ci/ray_ci:build_in_docker -- wheel --python-version {{matrix}} --architecture x86_64 --upload
+      # Extract wheel from wanda image (Wanda cache in ECR)
+      - wanda_image="$RAYCI_WORK_REPO:$RAYCI_BUILD_ID-ray-wheel-py{{matrix}}"
+      - container_id=$(docker create $wanda_image)
+      - mkdir -p /tmp/wheels
+      - docker cp ${container_id}:/ /tmp/wheels/
+      - docker rm ${container_id}
+      - mv /tmp/wheels/*.whl .whl/
+      - ./ci/build/copy_build_artifacts.sh wheel


This block of commands is nearly identical to the one in the linux_cpp_wheels_upload step (lines 130-138). To improve maintainability and reduce duplication, consider extracting this logic into a reusable script.

For example, you could create a script ci/build/extract_and_upload_wheel.sh that takes the image name suffix as an argument:

#!/bin/bash set -euo pipefail IMAGE_NAME_SUFFIX=$1 WANDA_IMAGE="$RAYCI_WORK_REPO:$RAYCI_BUILD_ID-$IMAGE_NAME_SUFFIX" echo "--- Extracting wheel from $WANDA_IMAGE" CONTAINER_ID=$(docker create "$WANDA_IMAGE") # Use trap to ensure the container is removed even if the script fails trap 'docker rm -f "$CONTAINER_ID"' EXIT # Clean up previous runs and prepare directories rm -rf /tmp/wheels mkdir -p /tmp/wheels .whl # Copy all files from the root of the container. docker cp "${CONTAINER_ID}:/." /tmp/wheels/ # Move the wheel to the .whl directory for the upload script mv /tmp/wheels/*.whl .whl/ # Upload the artifact ./ci/build/copy_build_artifacts.sh wheel

Then you could simplify the commands in both steps to a single line:

For linux_wheels_upload: commands: - ./ci/build/extract_and_upload_wheel.sh ray-wheel-py{{matrix}}

For linux_cpp_wheels_upload: commands: - ./ci/build/extract_and_upload_wheel.sh ray-cpp-wheel-py{{matrix}}

ci/docker/ray-wheel.Dockerfile

andrew-anyscale · 2026-01-07T17:49:28Z

.buildkite/build.rayci.yml

+  # Gating: only on releases/* OR (master AND nightly)
+  - label: ":s3: upload: wheel py{{matrix}} (x86_64)"
+    key: linux_wheels_upload
+    if: build.branch =~ /^releases\// || (build.branch == "master" && build.env("RAYCI_SCHEDULE") == "nightly")


Following logic on ci/build/copy_build_artifacts.sh, it seemed solid to just skip this step entirely when we're not in a postmerge state. I can also remove this if it's unnecessary

https://github.com/ray-project/ray/blob/master/ci/build/copy_build_artifacts.sh#L49-L57

it already has skip-on-premerge tag, so this if statement is not really needed.

there are cases where people sometimes want to build wheels and upload to the s3 from random branches, and we do allow that.

andrew-anyscale · 2026-01-07T18:35:29Z

.rayciversion

@@ -1 +1 @@
-0.21.0
+0.22.0


This bump is necessary to get Wanda's symlink functionality. Covered by #59744

understood, let's bump up wanda version in another PR first.

andrew-anyscale · 2026-01-07T18:35:38Z

ci/pipeline/test_rules.txt

see https://github.com/ray-project/ray/pull/59935/files#r2669650812

andrew-anyscale · 2026-01-07T18:36:13Z

End stack release test running here: https://buildkite.com/ray-project/release/builds/74229/steps/canvas

Mostly passing, with failures possibly related to flakes. Re-running those

.buildkite/build.rayci.yml

ci/docker/ray-cpp-core.Dockerfile

aslonnie

high-level looks really promising!

maybe split the ray-cpp wheel parts out in a follow up PR? although we are installing ray-cpp wheel in the release image, actually no release tests need ray-cpp wheel in there, so we might be able to just skip it when doing release tests, and we can maybe consider release another ray-cpp image, just to reduce the size of our released ray image.

aslonnie · 2026-01-08T07:07:48Z

.buildkite/build.rayci.yml

+    matrix:
+      - "3.10"
+      - "3.11"
+      - "3.12"


does ray-cpp does not yet have python 3.13?

one thing is that ray-cpp wheel's content is actually exactly the same for different python versions.. we might actually make it python version agnostic since we are here..

aslonnie · 2026-01-08T07:09:11Z

.buildkite/build.rayci.yml

+      - wanda_image="$RAYCI_WORK_REPO:$RAYCI_BUILD_ID-ray-wheel-py{{matrix}}"
+      - container_id=$(docker create $wanda_image)
+      - mkdir -p /tmp/wheels
+      - docker cp ${container_id}:/ /tmp/wheels/
+      - docker rm ${container_id}
+      - mkdir -p .whl
+      - mv /tmp/wheels/*.whl .whl/


could you make these a bash script in ./ci/build/ and call that with bash?

same with the ray-cpp one.

we try not to write many shell command lines on buildkite test step definitions. this makes the scripts easier to run and debug locally.

aslonnie · 2026-01-08T07:09:25Z

.buildkite/build.rayci.yml

+      - docker rm ${container_id}
+      - mkdir -p .whl
+      - mv /tmp/wheels/*.whl .whl/
+      - ./ci/build/copy_build_artifacts.sh wheel


maybe add explicit bash here too?

aslonnie · 2026-01-08T07:10:57Z

.buildkite/build.rayci.yml

+  # Gating: only on releases/* OR (master AND nightly)
+  - label: ":s3: upload: wheel py{{matrix}} (x86_64)"
+    key: linux_wheels_upload
+    if: build.branch =~ /^releases\// || (build.branch == "master" && build.env("RAYCI_SCHEDULE") == "nightly")


it already has skip-on-premerge tag, so this if statement is not really needed.

there are cases where people sometimes want to build wheels and upload to the s3 from random branches, and we do allow that.

.buildkite/build.rayci.yml

aslonnie · 2026-01-08T07:18:58Z

ci/docker/ray-cpp-wheel.wanda.yaml

+  - pyproject.toml
+  - README.rst
+  - ci/build/build-manylinux-wheel.sh
+  - python/


I actually don't think ray-cpp wheel building needs most of the files under python/.. because the wheel does not really contain python files, but mostly just cpp headers.

which means this ray-cpp wheel build might be cache-able.

aslonnie · 2026-01-08T07:22:38Z

also, could you verify that the wheel being built in this way, if from the same source code, has exactly the same content as the wheels that are built in the current/old way?

andrew-anyscale · 2026-01-08T17:45:34Z

I've made updates to split out the ray-cpp to a separate change--

also, could you verify that the wheel being built in this way, if from the same source code, has exactly the same content as the wheels that are built in the current/old way?

Let me get some validation going to confirm the full structure. Then I'll re-mark this as ready for review!

andrew-anyscale · 2026-01-08T20:10:39Z

Ran verifications against commit f6c2b5f on branch #59971

Results

Script used: https://github.com/ray-project/ray/pull/59971/files#diff-7752e49bebcb555eb3a21b7db9b83036898d923c00383f567acb2214c32d8e81

Raw print: https://github.com/ray-project/ray/pull/59971/files#diff-9e6772480096d2377f1e89b9b25f51d8fb7d6781878896b864f57c0d6fdb5c04

Compared wanda-built wheel against S3 reference wheel:

Postsubmit (old flow): ray-3.0.0.dev0-cp312-cp312-manylinux2014_x86_64.whl (S3, commit f6c2b5f7...)
- 029272617770.dkr.ecr.us-west-2.amazonaws.com/rayproject/citemp@sha256:48f4e1c9c8fd4492fb4e612a45b90b2deebdf545ce06ef4405f6bce634967795
Wanda (new flow): ray-3.0.0.dev0-cp312-cp312-manylinux2014_x86_64.whl (wanda build)
- https://buildkite.com/ray-project/premerge/builds/57208/steps/canvas?jid=019b9ee6-d747-4264-ad84-d8dd8d4ef649#019b9ee6-d747-4264-ad84-d8dd8d4ef649/L8673

Files Compared

Category	Result
File list	Identical (no missing/extra files)
Python source files	Identical
Thirdparty `.so` files (aiohttp, etc.)	Identical
`ray/_raylet.so`	Binary differs (see below)
`ray/_version.py`	Expected diff (commit hash)
`METADATA`	Ordering diff only (cosmetic)

`_raylet.so` Binary Analysis

Property	Remote	Local	Status
Size	42,215,040 bytes	42,215,040 bytes	Match
Symbols	-	-	No differences
Embedded paths	none	none	Clean

Conclusion: Binary difference is due to non-deterministic compilation (linker ordering, etc.), not code differences. Functionally equivalent.

Expected Differences

_version.py: Different commit hash due to cherry-pick (f6c2b5f7... vs d70dfe31...)
METADATA: Requires-Dist entries in different order (Python dict iteration order variance) - same dependencies, different serialization order

ci/pipeline/test_rules.txt

ci/build/extract_wanda_wheels.sh

aslonnie · 2026-01-09T00:21:45Z

ci/docker/ray-wheel.Dockerfile

+# Verify required artifacts exist before unpacking
+for f in /tmp/ray_pkg.zip /tmp/ray_py_proto.zip /tmp/ray_java_pkg.zip /tmp/dashboard.tar.gz; do
+  [[ -f "$f" ]] || { echo "ERROR: missing artifact: $f"; exit 1; }
+done


these are copied? how will they possibly be missing?

Good point. Removing

aslonnie · 2026-01-09T05:17:31Z

ci/docker/ray-wheel.Dockerfile

+./ci/build/build-manylinux-wheel.sh "$PY_BIN"
+
+# Sanity check: ensure wheels exist
+shopt -s nullglob


what does this line do?

It changes how loops operate when globs don't match anything

Without nullglob:

for f in *.log; do echo "processing $f" done

If there are no .log files, the loop runs once with f literally equal to *.log.

With nullglob:

If there are no matches, the loop runs zero times

ok, where that is a nice trick, it is changing the global settings. one can append some other script without knowing about this detail.

maybe just use wheels = ($(find . -maxdepth 0 -name '*.whl'))

aslonnie · 2026-01-09T05:20:43Z

ci/docker/ray-wheel.Dockerfile

+ENV BUILDKITE_COMMIT=${BUILDKITE_COMMIT:-unknown} \
+    PYTHON_VERSION=${PYTHON_VERSION} \
+    SKIP_BAZEL_BUILD=1 \
+    RAY_DISABLE_EXTRA_CPP=1


could you move SKIP_BAZEL_BUILD RAY_DISABLE_EXTRA_CPP into the inlined bash script rather than making it image-wide env, and make it closer to the wheel building script's calling point? the other ones are also not necessary to be defined here I think.

Done https://github.com/ray-project/ray/compare/1b2c341fd206b05cd29344f244ac8d7524884d91..05785cb42e44f52e04542fae73c7f4e71cb767ed#diff-1bb18f8e95635ad35397ce4fd116adb842c8d1163e6faff2d02c3411ca74717aR44-R46

aslonnie · 2026-01-09T05:22:08Z

ci/docker/ray-wheel.Dockerfile

+COPY --chown=2000:100 ci/build/build-manylinux-wheel.sh ci/build/
+COPY --chown=2000:100 README.rst pyproject.toml ./
+COPY --chown=2000:100 rllib/ rllib/
+COPY --chown=2000:100 python/ python/


what does 2000:100 mean? is it also used somewhere else?

I was shooting for passing ownership as non-root to the forge user (2000) . There's a similar pattern here, though it may not be necessary after all

ray/ci/ray_ci/builder_container.py

Line 39 in dad1efb

"chown -R 2000:100 /artifact-mount",

User: https://github.com/ray-project/ray/blob/master/.buildkite/release-automation/forge_x86_64.Dockerfile?utm_source=chatgpt.com#L56

I'll try running a build without this to see if it succeeds!

Tried running without the ownership passing, and since we're running as USER forge, it runs into issues since it cannot create the necessary directory structure

https://buildkite.com/ray-project/premerge/builds/57282/steps/canvas?jid=019ba375-2b30-45be-a272-ba97bc9fb48e#019ba375-2b30-45be-a272-ba97bc9fb48e/L298

I can however swap up to just chowning to the user. Let me try that for clarity

Done https://github.com/ray-project/ray/compare/1b2c341fd206b05cd29344f244ac8d7524884d91..05785cb42e44f52e04542fae73c7f4e71cb767ed#diff-1bb18f8e95635ad35397ce4fd116adb842c8d1163e6faff2d02c3411ca74717aR38-R41

aslonnie · 2026-01-09T07:05:22Z

ci/build/extract_wanda_wheels.sh

+
+# Image has no default CMD, so provide a dummy command.
+container_id="$(docker create "${WANDA_IMAGE}" /no-such-cmd)"
+


you can actually use crane export to directly get a tarball of the files, especially for container images that only have one layer with some files that is stored in a container registry, it works pretty reliably.

and you can avoid doing all the docker pull/cp/rm dance.

Sweet! So long as the uploader workers have crane, this'll be awesome. Added https://github.com/ray-project/ray/compare/1b2c341fd206b05cd29344f244ac8d7524884d91..05785cb42e44f52e04542fae73c7f4e71cb767ed#diff-236c8a31e5fa459f5409c9ba6f36706170109b0d377f98ba38b3b0d4943a2c97R28

aslonnie · 2026-01-09T17:25:15Z

ci/docker/ray-wheel.Dockerfile

+./ci/build/build-manylinux-wheel.sh "$PY_BIN"
+
+# Sanity check: ensure wheels exist
+shopt -s nullglob


ok, where that is a nice trick, it is changing the global settings. one can append some other script without knowing about this detail.

maybe just use wheels = ($(find . -maxdepth 0 -name '*.whl'))

aslonnie · 2026-01-09T17:25:35Z

ci/docker/ray-wheel.Dockerfile

+wheels=(.whl/*.whl)
+if (( ${#wheels[@]} == 0 )); then
+  echo "ERROR: No wheels produced in .whl/"
+  ls -la .whl || true


nit: why add the || true?

It's for a nit reason. In the case .whl doesn't exist, we return with exit code 2. This makes sure we consistently return exit code 1 on failure

aslonnie · 2026-01-09T17:27:42Z

.rayciversion

@@ -1 +1 @@
-0.21.0
+0.22.0


understood, let's bump up wanda version in another PR first.

This migrates ray wheel builds from CLI-based approach to wanda-based container builds for x86_64. Changes: - Add ray-wheel.wanda.yaml and Dockerfile for wheel builds - Update build.rayci.yml wheel steps to use wanda - Add wheel upload steps that extract from wanda cache Topic: ray-wheel Signed-off-by: andrew <andrew@anyscale.com>

This was referenced Jan 7, 2026

Add wanda ray image builds for Docker Hub #59936

Draft

Add wanda anyscale image builds for release tests #59937

Draft

Add wanda builds for aarch64 architecture #59938

Draft

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from f104bf5 to 288cadb Compare January 7, 2026 15:59

gemini-code-assist bot reviewed Jan 7, 2026

View reviewed changes

andrew-anyscale commented Jan 7, 2026

View reviewed changes

ci/pipeline/test_rules.txt

Copy link

Contributor Author

andrew-anyscale Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see https://github.com/ray-project/ray/pull/59935/files#r2669650812

andrew-anyscale marked this pull request as ready for review January 7, 2026 19:07

andrew-anyscale requested a review from a team as a code owner January 7, 2026 19:07

andrew-anyscale added the go add ONLY when ready to merge, run all tests label Jan 7, 2026

cursor bot reviewed Jan 7, 2026

View reviewed changes

.buildkite/build.rayci.yml Outdated Show resolved Hide resolved

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch 2 times, most recently from 164e2e4 to b4fb3e0 Compare January 7, 2026 20:21

cursor bot reviewed Jan 7, 2026

View reviewed changes

.buildkite/build.rayci.yml Outdated Show resolved Hide resolved

ci/docker/ray-cpp-core.Dockerfile Outdated Show resolved Hide resolved

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch 2 times, most recently from cd79253 to 7d257e4 Compare January 8, 2026 01:31

ray-gardener bot added core Issues that should be addressed in Ray Core devprod labels Jan 8, 2026

aslonnie reviewed Jan 8, 2026

View reviewed changes

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 7d257e4 to e93a821 Compare January 8, 2026 16:27

andrew-anyscale mentioned this pull request Jan 8, 2026

wanda python-agnostic ray-cpp wheel (py3-none tag)(x86_64) #59969

Draft

andrew-anyscale marked this pull request as draft January 8, 2026 16:27

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from e93a821 to 8a4bdf5 Compare January 8, 2026 16:33

andrew-anyscale removed the go add ONLY when ready to merge, run all tests label Jan 8, 2026

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 8a4bdf5 to a0b03e9 Compare January 8, 2026 20:01

andrew-anyscale marked this pull request as ready for review January 8, 2026 20:11

cursor bot reviewed Jan 8, 2026

View reviewed changes

ci/pipeline/test_rules.txt Show resolved Hide resolved

ci/build/extract_wanda_wheels.sh Show resolved Hide resolved

andrew-anyscale marked this pull request as draft January 8, 2026 20:52

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch 2 times, most recently from 32347fa to 9074421 Compare January 8, 2026 21:33

andrew-anyscale marked this pull request as ready for review January 8, 2026 22:28

andrew-anyscale requested a review from aslonnie January 8, 2026 22:28

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 9074421 to 1b2c341 Compare January 8, 2026 22:30

aslonnie reviewed Jan 9, 2026

View reviewed changes

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 1b2c341 to 05785cb Compare January 9, 2026 16:45

aslonnie reviewed Jan 9, 2026

View reviewed changes

andrew-anyscale force-pushed the andrew/revup/master/ray-wheel branch from 05785cb to 37410f9 Compare January 9, 2026 18:02


		# Image has no default CMD, so provide a dummy command.
		container_id="$(docker create "${WANDA_IMAGE}" /no-such-cmd)"

		@@ -1 +1 @@
		0.21.0
		0.22.0

		@@ -1 +1 @@
		0.21.0
		0.22.0

Migrate wheel builds to wanda (x86_64) #59935

Are you sure you want to change the base?

Migrate wheel builds to wanda (x86_64) #59935

Uh oh!

Conversation

andrew-anyscale commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrew-anyscale commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andrew-anyscale commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrew-anyscale commented Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aslonnie left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aslonnie commented Jan 8, 2026

Uh oh!

andrew-anyscale commented Jan 8, 2026

Uh oh!

andrew-anyscale commented Jan 8, 2026

Results

Files Compared

_raylet.so Binary Analysis

Expected Differences

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Without nullglob:

With nullglob:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

andrew-anyscale commented Jan 7, 2026 •

edited

Loading

aslonnie left a comment •

edited

Loading

`_raylet.so` Binary Analysis