wanda python-agnostic ray-cpp wheel (py3-none tag)(x86_64)

[andrew-anyscale]

The ray-cpp wheel contains only C++ headers, libraries, and executables
with no Python-specific code. Previously we built 4 identical wheels
(one per Python version: cp310, cp311, cp312, cp313), wasting CI time
and storage.

This change produces a single wheel tagged py3-none-manylinux2014_*
that works with any Python 3.x version.

Changes:

• Add ray-cpp-core.wanda.yaml and Dockerfile for cpp core
• Add ray-cpp-wheel.wanda.yaml for cpp wheel builds
• Add ci/build/build-ray-cpp-wheel.sh for Python-agnostic wheel builds
• Add RayCppBdistWheel class to setup.py that forces py3-none tags
  (necessary because BinaryDistribution.has_ext_modules() causes
  bdist_wheel to use interpreter-specific ABI tags by default)
• Update ray-cpp-wheel.wanda.yaml to build single wheel per architecture
• Update .buildkite/build.rayci.yml to remove Python version matrix
  for cpp wheel build/upload steps

Topic: ray-cpp-wheel
Relative: ray-wheel

Signed-off-by: andrew andrew@anyscale.com

[andrew-anyscale]

Reviews in this chain:
└#60125 ci: install crane in forge image
　└#59969 wanda python-agnostic ray-cpp wheel (py3-none tag)(x86_64)
　　└#59936 Add wanda ray image builds for Docker Hub
　　　└#59937 Add wanda anyscale image builds for release tests

[andrew-anyscale]

#	head	base	diff	date	summary
0	bf5b228f	e93a821d	diff	Jan 8 8:27 AM	4 files changed, 160 insertions(+)
1	ed903b74	8a4bdf5b	diff	Jan 8 8:32 AM	1 file changed, 2 insertions(+), 2 deletions(-)
2	21500ccb	a0b03e94	rebase	Jan 8 12:01 PM	0 files changed
3	cf7cab60	32347fa5	diff	Jan 8 12:51 PM	4 files changed, 129 insertions(+), 22 deletions(-)
4	8d1057a9	32347fa5	diff	Jan 8 13:13 PM	1 file changed, 2 insertions(+)
5	d8901f73	9074421e	diff	Jan 8 13:33 PM	3 files changed, 19 insertions(+), 29 deletions(-)
6	9e438258	1b2c341f	diff	Jan 8 14:30 PM	1 file changed, 1 deletion(-)
7	6801f72f	05785cb4	diff	Jan 9 8:45 AM	1 file changed, 12 insertions(+), 27 deletions(-)
8	9834ea38	37410f9a	diff	Jan 9 10:01 AM	1 file changed, 6 insertions(+), 3 deletions(-)
9	25e0d03c	37410f9a	diff	Jan 9 10:59 AM	4 files changed, 66 insertions(+), 28 deletions(-)
10	2a361721	37410f9a	diff	Jan 9 12:36 PM	4 files changed, 13 insertions(+), 22 deletions(-)
11	b81d4cb4	37410f9a	diff	Jan 9 13:32 PM	1 file changed, 54 insertions(+), 1 deletion(-)
12	4933e1ce	d9144e57	diff	Jan 13 8:55 AM	0 files changed
13	19d388a4	7df471c9	rebase	Jan 13 10:55 AM	0 files changed
14	7c1782d4	8d27704a	diff	Jan 13 15:50 PM	0 files changed
15	09f88fb0	8d27704a	diff	Jan 13 15:50 PM	0 files changed
16	c7f89bff	be2c5c4e	diff	Jan 13 17:03 PM	0 files changed
17	e2ebabb6	9cb14c92	rebase	Jan 13 17:09 PM	0 files changed
18	7b8a44a4	c9d30e15	rebase	Jan 13 17:09 PM	0 files changed
19	1410defe	daea687d	rebase	Jan 13 17:12 PM	0 files changed
20	64ac2fd8	19505953	rebase	Jan 13 17:14 PM	0 files changed
21	d5b11ffd	19505953	diff	Jan 13 17:33 PM	1 file changed, 2 insertions(+), 7 deletions(-)

[gemini-code-assist]

Code Review

This pull request introduces Wanda-based builds for Ray C++ wheels on x86_64, adding new CI configurations and Dockerfiles. The changes are well-structured, but there are opportunities to improve maintainability and ensure correctness across different Python versions. My review includes suggestions to refactor the Buildkite configuration to reduce duplication and to address a potential issue with hardcoded Python dependency files in the Wanda configuration.

[gemini-code-assist]

There's significant duplication in the newly added build steps. The matrix for Python versions and the env configuration are repeated. To improve maintainability and reduce redundancy, consider using YAML anchors and aliases.

For example, you could define:

x-python-versions: &python-versions
  - "3.10"
  - "3.11"
  - "3.12"

x-cpp-build-env: &cpp-build-env
  PYTHON_VERSION: "{{matrix}}"
  ARCH_SUFFIX: ""
  HOSTTYPE: "x86_64"
  MANYLINUX_VERSION: "260103.868e54c"

And then reference them in your build steps:

- name: ray-cpp-core-build
  ...
  matrix: *python-versions
  env: *cpp-build-env

- name: ray-cpp-wheel-build
  ...
  matrix: *python-versions
  env: *cpp-build-env

- label: ":s3: upload: cpp wheel py{{matrix}} (x86_64)"
  ...
  matrix: *python-versions

This would make the configuration cleaner and easier to update in the future.

[andrew-anyscale]

Verifying cpp wheel found no diffs between the old and this method:

Remote URL : f6c2b5f

• https://s3-us-west-2.amazonaws.com/ray-wheels/master/f6c2b5f7c48f295e1f73ca73ba7541f86bbeb5ef/ray_cpp-3.0.0.dev0-cp312-cp312-manylinux2014_x86_64.whl

Local wheel: 029272617770.dkr.ecr.us-west-2.amazonaws.com/rayproject/citemp@sha256:922a6d572dc2508d00cf247487facb2cbe918d3c924bc00ff53fd99726af5eb3

• https://buildkite.com/ray-project/premerge/builds/57234/steps/canvas?jid=019b9fa4-a426-4411-8687-fd8f878f68d3#019b9fa4-a426-4411-8687-fd8f878f68d3/L11184

https://github.com/ray-project/ray/pull/59971/files

[aslonnie]

could you resolve merge conflict? probably due to the oss tag thing

[aslonnie]

well.. the oss tag thing has not merged yet..

[andrew-anyscale]

could you resolve merge conflict?

I’ve learned this is an expected artifact of stacked PRs with the tooling I use once ancestor PRs merge. Ray (correctly) squash-merges PRs, which rewrites the commit history. These changes land on master as the new squashed commit, but the original commits my downstream PRs were based on no longer exist. After the base PR merges, this creates the merge conflict. The rebase is usually straightforward because the content is the same, but it does lead to this behavior

Hopefully GitHub can support stacked PRs more natively in the future 🙏

[aslonnie]

looks good in general. just some small things on styling.

[aslonnie]

while this analysis is great, can we remove it from the source code? or make it more succinct? this should just briefly explain that ray-cpp wheel is python version agnostic, so based on python wheel standard, we return these value, etc, etc..

this paragraph feels a bit too AI-ish.. it is okay for leaving it in the comments, but a bit too much for having it in the docstring..

[aslonnie]

I am merging this in as things are passing. could you follow up with a PR that cleans up this docstring a bit?

[andrew-anyscale]

I'll create a followup on this

[andrew-anyscale]

#60132