-
Notifications
You must be signed in to change notification settings - Fork 7.1k
Add wanda builds for aarch64 architecture #59938
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: andrew/revup/master/anyscale-image
Are you sure you want to change the base?
Add wanda builds for aarch64 architecture #59938
Conversation
|
02a8921 to
fe68786
Compare
c1885bf to
9f766d8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request migrates the aarch64 builds in the Buildkite pipeline to use the wanda build system. The changes are a good step towards standardizing the build process. However, I've identified a couple of issues. The commands for extracting wheels from built images are brittle and could fail if the wheel is not in the expected root directory. More importantly, it appears the ray-extra Docker images are no longer being pushed to a public registry, which is a regression from the previous pipeline configuration. My review includes suggestions to address these points.
| - name: ray-extra-image-cpu-build-aarch64 | ||
| label: "wanda: ray-extra py{{matrix}} cpu (aarch64)" | ||
| wanda: ci/docker/ray-extra-image-cpu.wanda.yaml | ||
| matrix: | ||
| - "3.10" | ||
| - "3.11" | ||
| - "3.12" | ||
| env: | ||
| PYTHON_VERSION: "{{matrix}}" | ||
| ARCH_SUFFIX: "-aarch64" | ||
| tags: | ||
| - python_dependencies | ||
| - docker | ||
| - oss | ||
| instance_type: medium-arm64 | ||
| commands: | ||
| - bazel run //ci/ray_ci:build_in_docker -- docker --python-version {{matrix}} | ||
| --platform cu11.7.1-cudnn8 --platform cu11.8.0-cudnn8 | ||
| --platform cu12.1.1-cudnn8 --platform cu12.3.2-cudnn9 | ||
| --platform cu12.4.1-cudnn --platform cu12.5.1-cudnn | ||
| --platform cu12.6.3-cudnn --platform cu12.8.1-cudnn | ||
| --platform cu12.9.1-cudnn | ||
| --platform cpu | ||
| --image-type ray-extra --architecture aarch64 --upload | ||
| instance_type: builder-arm64 | ||
| depends_on: | ||
| - manylinux-aarch64 | ||
| - forge-aarch64 | ||
| - raycudabaseextra-aarch64 | ||
| - ray-wheel-build-aarch64 | ||
| - raycpubaseextra-aarch64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The previous implementation for building ray-extra images (lines 154-179 in the original file) included an --upload flag, which resulted in the images being pushed. The new wanda-based build steps for ray-extra only build and cache the images in ECR, but they are not pushed to a public registry like Docker Hub. This appears to be a regression. To restore the original behavior, push steps for ray-extra images should be added, similar to the ones implemented for the ray images (e.g., lines 363-385).
.buildkite/linux_aarch64.rayci.yml
Outdated
| - mkdir -p /tmp/wheels | ||
| - docker cp ${container_id}:/ /tmp/wheels/ | ||
| - docker rm ${container_id} | ||
| - mv /tmp/wheels/*.whl .whl/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The command mv /tmp/wheels/*.whl .whl/ is brittle as it assumes the wheel file is in the root of the extracted directory and will fail to find wheels in subdirectories. Using find makes the extraction more robust by searching recursively. The mv $(find ...) pattern will also correctly fail if no wheel files are found, preserving the intended failure behavior on missing artifacts.
- mv $(find /tmp/wheels -name "*.whl") .whl/
.buildkite/linux_aarch64.rayci.yml
Outdated
| - mkdir -p /tmp/wheels | ||
| - docker cp ${container_id}:/ /tmp/wheels/ | ||
| - docker rm ${container_id} | ||
| - mv /tmp/wheels/*.whl .whl/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The command mv /tmp/wheels/*.whl .whl/ is brittle as it assumes the wheel file is in the root of the extracted directory and will fail to find wheels in subdirectories. Using find makes the extraction more robust by searching recursively. The mv $(find ...) pattern will also correctly fail if no wheel files are found, preserving the intended failure behavior on missing artifacts.
- mv $(find /tmp/wheels -name "*.whl") .whl/9f766d8 to
b5443e9
Compare
46cc862 to
70df756
Compare
fda2333 to
62921a1
Compare
70df756 to
ae32d45
Compare
62921a1 to
eff7e49
Compare
ae32d45 to
04cfd26
Compare
eff7e49 to
f025c58
Compare
55d1473 to
84d7479
Compare
f025c58 to
0980709
Compare
0980709 to
09563a4
Compare
1f049e8 to
d43ccd7
Compare
This migrates ray wheel builds from CLI-based approach to wanda-based container builds for x86_64. Changes: - Add ray-wheel.wanda.yaml and Dockerfile for wheel builds - Update build.rayci.yml wheel steps to use wanda - Add wheel upload steps that extract from wanda cache Topic: ray-wheel Labels: draft Signed-off-by: andrew <andrew@anyscale.com>
This adds wanda-based builds for ray Docker Hub images. Changes: - Add ray-image-cpu/cuda.wanda.yaml for ray images - Add ray-extra-image-cpu/cuda.wanda.yaml for extra dependencies - Add ray-llm-image-cuda.wanda.yaml and ray-llm-extra variant - Add push_ray_image.py for pushing to Docker Hub via crane - Update build.rayci.yml with wanda image build and push steps Topic: ray-image Relative: ray-wheel Labels: draft Signed-off-by: andrew <andrew@anyscale.com>
This adds wanda-based builds for anyscale images used in release tests. Changes: - Add ray-anyscale-cpu/cuda.wanda.yaml for ray anyscale images - Add ray-llm-anyscale-cuda.wanda.yaml for LLM images - Add ray-ml-anyscale-cuda.wanda.yaml for ML images - Add push_anyscale_image.py for pushing to ECR/GCP/Azure via crane - Update release/build.rayci.yml with build and push steps - Update gcloud_docker_login.sh to v550.0.0 with arch detection Topic: anyscale-image Relative: ray-image Labels: draft Signed-off-by: andrew <andrew@anyscale.com>
This adds wanda-based builds for aarch64/ARM architecture. Changes: - Update linux_aarch64.rayci.yml with wanda aarch64 builds Topic: ray-aarch64 Relative: anyscale-image Labels: draft Signed-off-by: andrew <andrew@anyscale.com>
d43ccd7 to
8980d9c
Compare
09563a4 to
ea805b3
Compare
ea805b3 to
7e7a73e
Compare
5de31a9 to
a96c770
Compare
This adds wanda-based builds for aarch64 architecture.
Changes:
Topic: ray-aarch64
Relative: ray-image
Labels: draft