Windows build w/ CUDA failing

[carsonswope]

I'm trying to build locally on windows with CUDA enabled. It fails with this error:

CMake Error at CMakeLists.txt:750 (enable_language):
  The CMAKE_CUDA_COMPILER:

    Z:/code/bfx/onnxruntime/build/cuda-sdk-win-v11.8/bin/nvcc

  is not a full path to an existing compiler tool.

  Tell CMake where to find the compiler by setting either the environment
  variable "CUDACXX" or the CMake cache entry CMAKE_CUDA_COMPILER to the full
  path to the compiler, or to the compiler name if it is in the PATH.

(Note I downloaded my own copy of the CUDA SDK and put it in the build directory, and passed that path to build.bat via --cuda_home argument, so the path shown above is as expected)

I see that the CMAKE_CUDA_COMPILER variable is set in build.py here:

onnxruntime/tools/ci_build/build.py

Line 1290 in da6b950

cmake_args.append(f"-DCMAKE_CUDA_COMPILER={cuda_home}/bin/nvcc")

Which I see was changed from this commit: #20987

This change seems reasonable enough on linux, but on windows executables are .exe files. If I change build.py to add a '.exe' to the end of the nvcc path, then it works fine on windows, as the provided path now is the full path to nvcc.

But clearly the CI windows build haven't been broken for 6 months.. So what am I missing?

[aboccag]

Hi, are you putting the root directory of cuda ?
For example $CUDA_HOME = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6" Then build using --cuda_home "$CUDA_HOME"

[carsonswope]

Yes, of course. I put the CUDA SDK (11.8) at Z:/code/bfx/onnxruntime/build/cuda-sdk-win-v11.8, which I pass to the build script via --cuda_home. It's clearly getting passed correctly because I see that path in the error message coming from cmake.

What I really want to know is how it is that setting CMAKE_CUDA_COMPILER variable in build.py is supposed to work without the .exe suffix on windows - I can easily work around it by modifying build.py to add the suffix if building on windows.

[amarin16]

These are the commands that I successfully used today to build with CUDA on Windows (using the latest changes on the main branch):

> git clean -xdf
> .\build.bat --config Debug --build_shared_lib --parallel --skip_submodule_sync  --skip_tests --use_cuda  --cudnn_home "C:\Program Files\NVIDIA\CUDNN\v9.4" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6"

See https://onnxruntime.ai/docs/build/eps.html#cuda

[aboccag]

I manage to build it as well from the main branch but not from the latest releases (1.20.1 and 1.20.2) I don't have any explicit error, it just fail. Can you try ?
Best regards

[amarin16]

I successfully built from the rel-1.20.0 branch using the same command.