Create Conda CI test env in one step #144
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -57,6 +57,7 @@ DEPENDENCIES=( | |
| libraft | ||
| libraft-headers | ||
| librmm | ||
| libwholegraph | ||
| pylibcugraph | ||
| pylibwholegraph | ||
| rmm | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,17 @@ | ||
| #!/bin/bash | ||
| # Copyright (c) 2022-2025, NVIDIA CORPORATION. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # Support invoking test_python.sh outside the script directory | ||
| cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")"/../ | ||
|
|
||
| if [[ "${RAPIDS_CUDA_VERSION%%.*}" == "11" ]]; then | ||
| DGL_CHANNEL="dglteam/label/th23_cu118" | ||
| else | ||
| DGL_CHANNEL="dglteam/label/th23_cu121" | ||
| fi | ||
|
|
||
| . /opt/conda/etc/profile.d/conda.sh | ||
|
|
||
| RAPIDS_VERSION="$(rapids-version)" | ||
|
|
@@ -14,15 +20,6 @@ rapids-logger "Downloading artifacts from previous jobs" | |
| CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp) | ||
| PYTHON_CHANNEL=$(rapids-download-conda-from-s3 python) | ||
|
|
||
| RAPIDS_TESTS_DIR=${RAPIDS_TESTS_DIR:-"${PWD}/test-results"} | ||
| RAPIDS_COVERAGE_DIR=${RAPIDS_COVERAGE_DIR:-"${PWD}/coverage-results"} | ||
| mkdir -p "${RAPIDS_TESTS_DIR}" "${RAPIDS_COVERAGE_DIR}" | ||
|
|
@@ -50,31 +47,26 @@ set +e | |
| # bulk sampler IO tests (hangs in CI) | ||
|
|
||
| if [[ "${RUNNER_ARCH}" != "ARM64" ]]; then | ||
| rapids-logger "(cugraph-dgl) Generate Python testing dependencies" | ||
| rapids-dependency-file-generator \ | ||
| --output conda \ | ||
| --file-key test_cugraph_dgl \ | ||
| --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \ | ||
| --prepend-channel "${CPP_CHANNEL}" \ | ||
| --prepend-channel "${PYTHON_CHANNEL}" \ | ||
| --prepend-channel pytorch \ | ||
| --prepend-channel conda-forge \ | ||
| --prepend-channel "${DGL_CHANNEL}" \ | ||
KyleFromNVIDIA marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| --prepend-channel nvidia \ | ||
| | tee env.yaml | ||
|
|
||
| rapids-mamba-retry env create --yes -f env.yaml -n test_cugraph_dgl | ||
|
|
||
| # activate test_cugraph_dgl environment for dgl | ||
| set +u | ||
| conda activate test_cugraph_dgl | ||
| set -u | ||
|
|
||
|
|
||
| rapids-print-env | ||
|
|
||
|
|
@@ -98,22 +90,23 @@ else | |
| fi | ||
|
|
||
| if [[ "${RUNNER_ARCH}" != "ARM64" ]]; then | ||
| rapids-logger "(cugraph-pyg) Generate Python testing dependencies" | ||
| rapids-dependency-file-generator \ | ||
| --output conda \ | ||
| --file-key test_cugraph_pyg \ | ||
| --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \ | ||
| --prepend-channel "${CPP_CHANNEL}" \ | ||
| --prepend-channel "${PYTHON_CHANNEL}" \ | ||
| --prepend-channel pytorch \ | ||
| | tee env.yaml | ||
|
|
||
| rapids-mamba-retry env create --yes -f env.yaml -n test_cugraph_pyg | ||
|
|
||
| # Temporarily allow unbound variables for conda activation. | ||
| set +u | ||
| conda activate test_cugraph_pyg | ||
| set -u | ||
|
|
||
| rapids-print-env | ||
|
|
||
| rapids-logger "Check GPU usage" | ||
|
|
@@ -136,23 +129,23 @@ else | |
| fi | ||
|
|
||
| if [[ "${RUNNER_ARCH}" != "ARM64" ]]; then | ||
| rapids-logger "(pylibwholegraph) Generate Python testing dependencies" | ||
| rapids-dependency-file-generator \ | ||
| --output conda \ | ||
| --file-key test_pylibwholegraph \ | ||
| --matrix "cuda=${RAPIDS_CUDA_VERSION%.*};arch=$(arch);py=${RAPIDS_PY_VERSION}" \ | ||
| --prepend-channel "${CPP_CHANNEL}" \ | ||
| --prepend-channel "${PYTHON_CHANNEL}" \ | ||
| --prepend-channel pytorch \ | ||
|
There was a problem hiding this comment. Not a blocking comment for this PR, but @tingyu66 @alexbarghi-nv are we planning to drop use of the Linking this related conversation: #99 (comment) There was a problem hiding this comment. Thank you! I can get that started in a follow-up PR if you'd like. |
||
| | tee env.yaml | ||
|
|
||
| rapids-mamba-retry env create --yes -f env.yaml -n test_pylibwholegraph | ||
|
|
||
| # Temporarily allow unbound variables for conda activation. | ||
| set +u | ||
| conda activate test_pylibwholegraph | ||
| set -u | ||
|
|
||
| rapids-print-env | ||
|
|
||
| rapids-logger "Check GPU usage" | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -43,11 +43,14 @@ files: | |
| - cuda_version | ||
| - docs | ||
| - py_version | ||
| - depends_on_libwholegraph | ||
| test_cpp: | ||
| output: none | ||
| includes: | ||
| - cuda_version | ||
| - test_cpp | ||
| - depends_on_libwholegraph | ||
| - depends_on_libwholegraph_tests | ||
| test_notebooks: | ||
| output: none | ||
| includes: | ||
|
|
@@ -68,6 +71,46 @@ files: | |
| - depends_on_ogb | ||
| - py_version | ||
| - test_python_common | ||
| test_cugraph_dgl: | ||
| output: none | ||
| includes: | ||
| - cuda_version | ||
| - depends_on_cugraph | ||
| - depends_on_cudf | ||
| - depends_on_dgl | ||
| - depends_on_pytorch | ||
| - depends_on_ogb | ||
| - py_version | ||
| - test_python_common | ||
| - depends_on_pylibwholegraph | ||
| - depends_on_cugraph_dgl | ||
| test_cugraph_pyg: | ||
| output: none | ||
| includes: | ||
| - cuda_version | ||
| - depends_on_cugraph | ||
| - depends_on_cudf | ||
| - depends_on_dgl | ||
| - depends_on_pytorch | ||
| - depends_on_ogb | ||
| - py_version | ||
| - test_python_common | ||
| - depends_on_pylibwholegraph | ||
| - depends_on_cugraph_pyg | ||
| test_pylibwholegraph: | ||
| output: none | ||
| includes: | ||
| - cuda_version | ||
| - depends_on_cugraph | ||
| - depends_on_cudf | ||
| - depends_on_dgl | ||
| - depends_on_pytorch | ||
| - depends_on_ogb | ||
|
There was a problem hiding this comment. I don't think this is what you want, to replace
Lines 438 to 464 in 2f8afde That looks like a mistake, not sure where it happened. @tingyu66 could you please take a look? Maybe that was the result of a bad merge conflict resolution or something. There was a problem hiding this comment. This is a temporary constraint of PyTorch that only applies to test environments where ogb is being used. There was a problem hiding this comment. Bradley and I discussed this here: #104 There was a problem hiding this comment. A better name for this might be We typically use There was a problem hiding this comment. Got it, thanks! The comment directly above this, about this being there for the benefit of It still would be better to name it There was a problem hiding this comment. Ok, also devcontainer builds will fail until #148 gets merged. I expect that to pass CI now that the cudf fix is merged. There was a problem hiding this comment. Now that that's merged, I've merged in latest |
||
| - py_version | ||
| - test_python_common | ||
| - depends_on_mkl | ||
| - depends_on_pylibwholegraph | ||
| - test_python_pylibwholegraph | ||
|
|
||
| py_build_pylibwholegraph: | ||
| output: pyproject | ||
|
|
@@ -509,6 +552,18 @@ dependencies: | |
| - pylibwholegraph-cu11==25.4.*,>=0.0.0a0 | ||
| - {matrix: null, packages: [*pylibwholegraph_unsuffixed]} | ||
|
|
||
| depends_on_libwholegraph: | ||
| common: | ||
| - output_types: conda | ||
| packages: | ||
| - libwholegraph==25.4.*,>=0.0.0a0 | ||
|
|
||
| depends_on_libwholegraph_tests: | ||
| common: | ||
| - output_types: conda | ||
| packages: | ||
| - libwholegraph-tests==25.4.*,>=0.0.0a0 | ||
|
|
||
| depends_on_rmm: | ||
| common: | ||
| - output_types: conda | ||
|
|
@@ -658,3 +713,13 @@ dependencies: | |
| packages: &cupy_packages_cu11 | ||
| - cupy-cuda11x>=13.2.0 | ||
| - {matrix: null, packages: *cupy_packages_cu11} | ||
| depends_on_cugraph_pyg: | ||
| common: | ||
| - output_types: conda | ||
| packages: | ||
| - cugraph-pyg==25.4.*,>=0.0.0a0 | ||
| depends_on_mkl: | ||
| common: | ||
| - output_types: conda | ||
| packages: | ||
| - mkl<2024.1.0 | ||
jameslamb marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are ARM tests doing anything here? Maybe they should exit earlier (or be skipped in the CI matrix) if not.
There are a few comments of the form
# Reactivate the test environment back. I’m not sure if we have a test environment to go back to, with the new single-solve environments.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can do that in a follow-up PR. I'd like to limit the scope of this one to consolidating the environment creation.