Skip to content

Conversation

@andrew-anyscale
Copy link
Contributor

This adds wanda-based builds for aarch64 architecture.

Changes:

  • Update linux_aarch64.rayci.yml with wanda steps for wheel and image builds

Topic: ray-aarch64
Relative: ray-image
Labels: draft

@andrew-anyscale
Copy link
Contributor Author

andrew-anyscale commented Jan 7, 2026

Reviews in this chain:
#59935 Migrate wheel builds to wanda (x86_64)
 ├#59969 Add wanda cpp wheel builds (x86_64)
 └#59936 Add wanda ray image builds for Docker Hub
  └#59937 Add wanda anyscale image builds for release tests
   └#59938 Add wanda builds for aarch64 architecture

@andrew-anyscale
Copy link
Contributor Author

andrew-anyscale commented Jan 7, 2026

# head base diff date summary
0 c1885bff 02a8921d diff Jan 7 7:57 AM 1 file changed, 253 insertions(+), 38 deletions(-)
1 9f766d8b fe687866 diff Jan 7 7:58 AM 0 files changed
2 b5443e9c 46cc8625 rebase Jan 7 8:38 AM 0 files changed
3 fda23332 70df7561 rebase Jan 7 11:29 AM 0 files changed
4 62921a10 ae32d45a rebase Jan 7 12:21 PM 0 files changed
5 eff7e491 04cfd26a rebase Jan 7 17:29 PM 0 files changed
6 f025c58d 55d1473f rebase Jan 7 17:31 PM 0 files changed
7 0980709c 1f049e81 diff Jan 8 8:27 AM 1 file changed, 6 insertions(+), 24 deletions(-)
8 09563a45 d43ccd7f rebase Jan 8 8:32 AM 0 files changed
9 ea805b3f 8980d9ce rebase Jan 8 12:01 PM 0 files changed
10 7e7a73e5 4a143b7a rebase Jan 8 12:51 PM 0 files changed

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request migrates the aarch64 builds in the Buildkite pipeline to use the wanda build system. The changes are a good step towards standardizing the build process. However, I've identified a couple of issues. The commands for extracting wheels from built images are brittle and could fail if the wheel is not in the expected root directory. More importantly, it appears the ray-extra Docker images are no longer being pushed to a public registry, which is a regression from the previous pipeline configuration. My review includes suggestions to address these points.

Comment on lines +261 to 262
- name: ray-extra-image-cpu-build-aarch64
label: "wanda: ray-extra py{{matrix}} cpu (aarch64)"
wanda: ci/docker/ray-extra-image-cpu.wanda.yaml
matrix:
- "3.10"
- "3.11"
- "3.12"
env:
PYTHON_VERSION: "{{matrix}}"
ARCH_SUFFIX: "-aarch64"
tags:
- python_dependencies
- docker
- oss
instance_type: medium-arm64
commands:
- bazel run //ci/ray_ci:build_in_docker -- docker --python-version {{matrix}}
--platform cu11.7.1-cudnn8 --platform cu11.8.0-cudnn8
--platform cu12.1.1-cudnn8 --platform cu12.3.2-cudnn9
--platform cu12.4.1-cudnn --platform cu12.5.1-cudnn
--platform cu12.6.3-cudnn --platform cu12.8.1-cudnn
--platform cu12.9.1-cudnn
--platform cpu
--image-type ray-extra --architecture aarch64 --upload
instance_type: builder-arm64
depends_on:
- manylinux-aarch64
- forge-aarch64
- raycudabaseextra-aarch64
- ray-wheel-build-aarch64
- raycpubaseextra-aarch64
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The previous implementation for building ray-extra images (lines 154-179 in the original file) included an --upload flag, which resulted in the images being pushed. The new wanda-based build steps for ray-extra only build and cache the images in ECR, but they are not pushed to a public registry like Docker Hub. This appears to be a regression. To restore the original behavior, push steps for ray-extra images should be added, similar to the ones implemented for the ray images (e.g., lines 363-385).

- mkdir -p /tmp/wheels
- docker cp ${container_id}:/ /tmp/wheels/
- docker rm ${container_id}
- mv /tmp/wheels/*.whl .whl/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The command mv /tmp/wheels/*.whl .whl/ is brittle as it assumes the wheel file is in the root of the extracted directory and will fail to find wheels in subdirectories. Using find makes the extraction more robust by searching recursively. The mv $(find ...) pattern will also correctly fail if no wheel files are found, preserving the intended failure behavior on missing artifacts.

      - mv $(find /tmp/wheels -name "*.whl") .whl/

- mkdir -p /tmp/wheels
- docker cp ${container_id}:/ /tmp/wheels/
- docker rm ${container_id}
- mv /tmp/wheels/*.whl .whl/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The command mv /tmp/wheels/*.whl .whl/ is brittle as it assumes the wheel file is in the root of the extracted directory and will fail to find wheels in subdirectories. Using find makes the extraction more robust by searching recursively. The mv $(find ...) pattern will also correctly fail if no wheel files are found, preserving the intended failure behavior on missing artifacts.

      - mv $(find /tmp/wheels -name "*.whl") .whl/

@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch from 9f766d8 to b5443e9 Compare January 7, 2026 16:38
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 46cc862 to 70df756 Compare January 7, 2026 19:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch 2 times, most recently from fda2333 to 62921a1 Compare January 7, 2026 20:21
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 70df756 to ae32d45 Compare January 7, 2026 20:21
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch from 62921a1 to eff7e49 Compare January 8, 2026 01:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from ae32d45 to 04cfd26 Compare January 8, 2026 01:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch from eff7e49 to f025c58 Compare January 8, 2026 01:31
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 55d1473 to 84d7479 Compare January 8, 2026 16:27
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch from f025c58 to 0980709 Compare January 8, 2026 16:27
@andrew-anyscale andrew-anyscale changed the base branch from andrew/revup/master/ray-image to andrew/revup/master/anyscale-image January 8, 2026 16:27
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch from 0980709 to 09563a4 Compare January 8, 2026 16:32
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 1f049e8 to d43ccd7 Compare January 8, 2026 16:33
This migrates ray wheel builds from CLI-based approach to wanda-based
container builds for x86_64.

Changes:
- Add ray-wheel.wanda.yaml and Dockerfile for wheel builds
- Update build.rayci.yml wheel steps to use wanda
- Add wheel upload steps that extract from wanda cache

Topic: ray-wheel
Labels: draft
Signed-off-by: andrew <andrew@anyscale.com>
This adds wanda-based builds for ray Docker Hub images.

Changes:
- Add ray-image-cpu/cuda.wanda.yaml for ray images
- Add ray-extra-image-cpu/cuda.wanda.yaml for extra dependencies
- Add ray-llm-image-cuda.wanda.yaml and ray-llm-extra variant
- Add push_ray_image.py for pushing to Docker Hub via crane
- Update build.rayci.yml with wanda image build and push steps

Topic: ray-image
Relative: ray-wheel
Labels: draft
Signed-off-by: andrew <andrew@anyscale.com>
This adds wanda-based builds for anyscale images used in release tests.

Changes:
- Add ray-anyscale-cpu/cuda.wanda.yaml for ray anyscale images
- Add ray-llm-anyscale-cuda.wanda.yaml for LLM images
- Add ray-ml-anyscale-cuda.wanda.yaml for ML images
- Add push_anyscale_image.py for pushing to ECR/GCP/Azure via crane
- Update release/build.rayci.yml with build and push steps
- Update gcloud_docker_login.sh to v550.0.0 with arch detection

Topic: anyscale-image
Relative: ray-image
Labels: draft
Signed-off-by: andrew <andrew@anyscale.com>
This adds wanda-based builds for aarch64/ARM architecture.

Changes:
- Update linux_aarch64.rayci.yml with wanda aarch64 builds

Topic: ray-aarch64
Relative: anyscale-image
Labels: draft
Signed-off-by: andrew <andrew@anyscale.com>
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from d43ccd7 to 8980d9c Compare January 8, 2026 20:01
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch from 09563a4 to ea805b3 Compare January 8, 2026 20:01
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-aarch64 branch from ea805b3 to 7e7a73e Compare January 8, 2026 20:52
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 9 times, most recently from 5de31a9 to a96c770 Compare January 9, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants