Skip to content

Conversation

@andrew-anyscale
Copy link
Contributor

This adds wanda-based builds for anyscale images used in release tests.

Changes:

  • Add ray-anyscale-cpu/cuda.wanda.yaml for ray anyscale images
  • Add ray-llm-anyscale-cuda.wanda.yaml for LLM images
  • Add ray-ml-anyscale-cuda.wanda.yaml for ML images
  • Add push_anyscale_image.py for pushing to ECR/GCP/Azure via crane
  • Update release/build.rayci.yml with build and push steps
  • Update gcloud_docker_login.sh to v550.0.0 with arch detection

Topic: anyscale-image
Relative: ray-wheel
Labels: draft

@andrew-anyscale
Copy link
Contributor Author

andrew-anyscale commented Jan 7, 2026

Reviews in this chain:
#59935 Migrate wheel builds to wanda (x86_64)
 └#59969 Add wanda cpp wheel builds (x86_64)
  └#59936 Add wanda ray image builds for Docker Hub
   └#59937 Add wanda anyscale image builds for release tests

@andrew-anyscale
Copy link
Contributor Author

andrew-anyscale commented Jan 7, 2026

# head base diff date summary
0 f963b425 f104bf59 diff Jan 7 7:57 AM 8 files changed, 553 insertions(+), 24 deletions(-)
1 07109714 288cadb6 diff Jan 7 7:58 AM 0 files changed
2 4cdb2dd9 46cc8625 diff Jan 7 8:38 AM 2 files changed, 238 insertions(+), 5 deletions(-)
3 f65669a5 70df7561 rebase Jan 7 11:29 AM 0 files changed
4 11418977 ae32d45a rebase Jan 7 12:21 PM 0 files changed
5 af3b600e 04cfd26a rebase Jan 7 17:29 PM 0 files changed
6 6bbeb9e4 55d1473f rebase Jan 7 17:31 PM 0 files changed
7 1f049e81 84d74797 diff Jan 8 8:27 AM 2 files changed, 12 insertions(+), 12 deletions(-)
8 d43ccd7f c275fe24 rebase Jan 8 8:32 AM 0 files changed
9 8980d9ce 7c78fa2d rebase Jan 8 12:01 PM 0 files changed
10 4a143b7a 29cf5abd rebase Jan 8 12:51 PM 0 files changed
11 9243672a d566c8a8 rebase Jan 8 13:16 PM 0 files changed
12 8c684c8e 91900297 rebase Jan 8 13:33 PM 0 files changed
13 826bac0f 73822f53 rebase Jan 8 14:30 PM 0 files changed
14 ebdca80e 94805e95 rebase Jan 9 8:45 AM 0 files changed
15 35249bf7 fc79d67d rebase Jan 9 10:01 AM 0 files changed
16 0e1bd8b1 db94f935 rebase Jan 9 10:59 AM 0 files changed
17 5de31a90 05c06707 rebase Jan 9 12:36 PM 0 files changed
18 a96c770e a6718508 rebase Jan 9 13:32 PM 0 files changed

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Anyscale image build process to use Wanda for building and a new Python script for pushing images to multiple registries. The changes include new Wanda configuration files, the push_anyscale_image.py script, and updates to the Buildkite pipeline. My review identifies a few critical issues that will break the build, including a missing file in a Bazel target and an incomplete matrix configuration in the pipeline. I've also provided several suggestions to improve maintainability by reducing code duplication and making parts of the new script less brittle.

Comment on lines 321 to 344
py_binary(
name = "push_ray_image",
srcs = ["push_ray_image.py"],
exec_compatible_with = ["//bazel:py3"],
deps = [
":crane_lib",
"//ci/ray_ci:ray_ci_lib",
ci_require("click"),
],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This py_binary target for push_ray_image references a source file push_ray_image.py that is not included in this pull request. This will cause the Bazel build to fail. Please either add the missing file or remove this target if it's not intended to be part of this change.

Comment on lines +171 to +172
- bash release/gcloud_docker_login.sh release/aws2gce_iam.json
- bash release/azure_docker_login.sh
- az acr login --name rayreleasetest
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These authentication commands are duplicated across the anyscalebuild, anyscalellmbuild, and anyscalemlbuild push jobs. To improve maintainability and reduce redundancy, consider using YAML anchors and aliases to define these commands once and reuse them in each job. This will make future updates to the authentication process much easier.


# GPU_PLATFORM is the default GPU platform that gets aliased as "gpu"
# This must match the definition in ci/ray_ci/docker_container.py
GPU_PLATFORM = "cu12.1.1-cudnn8"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The GPU_PLATFORM is hardcoded here. The comment mentions it must match a definition in another file, which makes this brittle. If the default GPU platform changes, this script will require a manual update, which can be easily missed. Consider sourcing this value from a shared configuration file or passing it as a command-line argument to make the script more robust and maintainable.

Comment on lines +69 to +75
def _format_platform_tag(platform: str) -> str:
"""Format platform as -cpu or shortened CUDA version."""
if platform == "cpu":
return "-cpu"
# cu12.3.2-cudnn9 -> -cu123
versions = platform.split(".")
return f"-{versions[0]}{versions[1]}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic for parsing the CUDA version string using platform.split('.') is fragile. It assumes a specific format with dots as separators and might fail with an IndexError if the platform string format changes (e.g., cu12-cudnn8). Using a regular expression would be more robust for extracting the version parts. For example: re.match(r"cu(\\d+)\\.(\\d+)", platform).

Comment on lines +78 to +127
def _get_image_tags(python_version: str, platform: str) -> List[str]:
"""
Generate image tags matching the original docker_container.py format.
Returns multiple tags for the image (canonical + aliases).
For GPU_PLATFORM, also generates -gpu alias tags to match release test expectations.
"""
branch = os.environ.get("BUILDKITE_BRANCH", "")
commit = os.environ.get("BUILDKITE_COMMIT", "")[:6]
rayci_build_id = os.environ.get("RAYCI_BUILD_ID", "")

py_tag = _format_python_version_tag(python_version)
platform_tag = _format_platform_tag(platform)

# For GPU_PLATFORM, also create -gpu alias (release tests use type: gpu)
platform_tags = [platform_tag]
if platform == GPU_PLATFORM:
platform_tags.append("-gpu")

tags = []

if branch == "master":
# On master, use sha and build_id as tags
for ptag in platform_tags:
tags.append(f"{commit}{py_tag}{ptag}")
if rayci_build_id:
for ptag in platform_tags:
tags.append(f"{rayci_build_id}{py_tag}{ptag}")
elif branch.startswith("releases/"):
# On release branches, use release name
release_name = branch[len("releases/") :]
for ptag in platform_tags:
tags.append(f"{release_name}.{commit}{py_tag}{ptag}")
if rayci_build_id:
for ptag in platform_tags:
tags.append(f"{rayci_build_id}{py_tag}{ptag}")
else:
# For other branches (PRs, etc.)
pr = os.environ.get("BUILDKITE_PULL_REQUEST", "false")
if pr != "false":
for ptag in platform_tags:
tags.append(f"pr-{pr}.{commit}{py_tag}{ptag}")
else:
for ptag in platform_tags:
tags.append(f"{commit}{py_tag}{ptag}")
if rayci_build_id:
for ptag in platform_tags:
tags.append(f"{rayci_build_id}{py_tag}{ptag}")

return tags
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This function has a lot of repeated code, especially the block for appending rayci_build_id tags, which is identical in all if/elif/else branches. This can be refactored to be more DRY (Don't Repeat Yourself), which improves readability and maintainability. You could determine the tag prefix first, then generate the tags in a single block of code.

def _get_image_tags(python_version: str, platform: str) -> List[str]:
    """
    Generate image tags matching the original docker_container.py format.

    Returns multiple tags for the image (canonical + aliases).
    For GPU_PLATFORM, also generates -gpu alias tags to match release test expectations.
    """
    branch = os.environ.get("BUILDKITE_BRANCH", "")
    commit = os.environ.get("BUILDKITE_COMMIT", "")[:6]
    rayci_build_id = os.environ.get("RAYCI_BUILD_ID", "")

    py_tag = _format_python_version_tag(python_version)
    platform_tag = _format_platform_tag(platform)

    # For GPU_PLATFORM, also create -gpu alias (release tests use type: gpu)
    platform_tags = [platform_tag]
    if platform == GPU_PLATFORM:
        platform_tags.append("-gpu")

    tags = []
    if branch == "master":
        prefix = commit
    elif branch.startswith("releases/"):
        release_name = branch[len("releases/") :]
        prefix = f"{release_name}.{commit}"
    else:
        pr = os.environ.get("BUILDKITE_PULL_REQUEST", "false")
        if pr != "false":
            prefix = f"pr-{pr}.{commit}"
        else:
            prefix = commit

    for ptag in platform_tags:
        tags.append(f"{prefix}{py_tag}{ptag}")

    if rayci_build_id:
        for ptag in platform_tags:
            tags.append(f"{rayci_build_id}{py_tag}{ptag}")

    return tags

@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 0710971 to 4cdb2dd Compare January 7, 2026 16:38
@andrew-anyscale andrew-anyscale changed the base branch from andrew/revup/master/ray-wheel to andrew/revup/master/ray-image January 7, 2026 16:38
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 46cc862 to 70df756 Compare January 7, 2026 19:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 4cdb2dd to f65669a Compare January 7, 2026 19:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 70df756 to ae32d45 Compare January 7, 2026 20:21
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from f65669a to 1141897 Compare January 7, 2026 20:21
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from ae32d45 to 04cfd26 Compare January 8, 2026 01:29
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 2 times, most recently from af3b600 to 6bbeb9e Compare January 8, 2026 01:31
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 04cfd26 to 55d1473 Compare January 8, 2026 01:31
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 6bbeb9e to 1f049e8 Compare January 8, 2026 16:27
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 55d1473 to 84d7479 Compare January 8, 2026 16:27
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 84d7479 to c275fe2 Compare January 8, 2026 16:32
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 2 times, most recently from d43ccd7 to 8980d9c Compare January 8, 2026 20:01
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch 2 times, most recently from 7c78fa2 to 29cf5ab Compare January 8, 2026 20:52
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 8980d9c to 4a143b7 Compare January 8, 2026 20:52
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 29cf5ab to d566c8a Compare January 8, 2026 21:16
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 4a143b7 to 9243672 Compare January 8, 2026 21:16
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from d566c8a to 9190029 Compare January 8, 2026 21:33
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 2 times, most recently from 8c684c8 to 826bac0 Compare January 8, 2026 22:30
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 9190029 to 73822f5 Compare January 8, 2026 22:30
@andrew-anyscale andrew-anyscale added the go add ONLY when ready to merge, run all tests label Jan 9, 2026
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 826bac0 to ebdca80 Compare January 9, 2026 16:45
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 73822f5 to 94805e9 Compare January 9, 2026 16:45
This adds wanda-based builds for anyscale images used in release tests.

Changes:
- Add ray-anyscale-cpu/cuda.wanda.yaml for ray anyscale images
- Add ray-llm-anyscale-cuda.wanda.yaml for LLM images
- Add ray-ml-anyscale-cuda.wanda.yaml for ML images
- Add push_anyscale_image.py for pushing to ECR/GCP/Azure via crane
- Update release/build.rayci.yml with build and push steps
- Update gcloud_docker_login.sh to v550.0.0 with arch detection

Topic: anyscale-image
Relative: ray-image
Labels: draft
Signed-off-by: andrew <andrew@anyscale.com>
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 94805e9 to fc79d67 Compare January 9, 2026 18:01
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch 2 times, most recently from 35249bf to 0e1bd8b Compare January 9, 2026 18:59
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from fc79d67 to db94f93 Compare January 9, 2026 18:59
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 0e1bd8b to 5de31a9 Compare January 9, 2026 20:36
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from db94f93 to 05c0670 Compare January 9, 2026 20:36
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/anyscale-image branch from 5de31a9 to a96c770 Compare January 9, 2026 21:33
@andrew-anyscale andrew-anyscale force-pushed the andrew/revup/master/ray-image branch from 05c0670 to a671850 Compare January 9, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants