Bug: ggml-rocm.so not found in llamafile 0.8.13

[winstonma]

Contact Details

No response

What happened?

I just downloaded Meta-Llama-3.1-8B-Instruct.Q5_K_M.llamafile which comes with llamafile v0.8.13. And I started with argument -ngl 9999. And I got the following error:

# v0.8.13
$ ./Meta-Llama-3.1-8B-Instruct.Q5_K_M.llamafile -ngl 9999
import_cuda_impl: initializing gpu module...
extracting /zip/llama.cpp/ggml.h to ~/.llamafile/v/0.8.13/ggml.h
extracting /zip/llamafile/compcap.cu to ~/.llamafile/v/0.8.13/compcap.cu
extracting /zip/llamafile/llamafile.h to ~/.llamafile/v/0.8.13/llamafile.h
extracting /zip/llamafile/tinyblas.h to ~/.llamafile/v/0.8.13/tinyblas.h
extracting /zip/llamafile/tinyblas.cu to ~/.llamafile/v/0.8.13/tinyblas.cu
extracting /zip/llama.cpp/ggml-impl.h to ~/.llamafile/v/0.8.13/ggml-impl.h
extracting /zip/llama.cpp/ggml-cuda.h to ~/.llamafile/v/0.8.13/ggml-cuda.h
extracting /zip/llama.cpp/ggml-alloc.h to ~/.llamafile/v/0.8.13/ggml-alloc.h
extracting /zip/llama.cpp/ggml-common.h to ~/.llamafile/v/0.8.13/ggml-common.h
extracting /zip/llama.cpp/ggml-backend.h to ~/.llamafile/v/0.8.13/ggml-backend.h
extracting /zip/llama.cpp/ggml-backend-impl.h to ~/.llamafile/v/0.8.13/ggml-backend-impl.h
extracting /zip/llama.cpp/ggml-cuda.cu to ~/.llamafile/v/0.8.13/ggml-cuda.cu
extract_cuda_dso: note: prebuilt binary /zip/ggml-rocm.so not found
get_rocm_bin_path: note: hipInfo not found on $PATH
get_rocm_bin_path: note: $HIP_PATH/bin/hipInfo does not exist
get_rocm_bin_path: note: /opt/rocm/bin/hipInfo does not exist
llamafile_log_command: /usr/bin/rocminfo
llamafile_log_command: hipcc -O3 -fPIC -shared --offload-arch=gfx1030 -march=native -mtune=native -DGGML_USE_HIPBLAS -Wno-return-type -Wno-unused-result -Wno-unused-function -Wno-expansion-to-defined -DIGNORE0 -DNDEBUG -DGGML_BUILD=1 -DGGML_SHARED=1 -DGGML_MULTIPLATFORM -DGGML_CUDA_DMMV_X=32 -DK_QUANTS_PER_ITERATION=2 -DGGML_CUDA_PEER_MAX_BATCH_SIZE=128 -DGGML_CUDA_MMV_Y=1 -DGGML_USE_CUBLAS -DGGML_MINIMIZE_CODE_SIZE -o ~/.llamafile/v/0.8.13/ggml-rocm.so.4fwh4p ~/.llamafile/v/0.8.13/ggml-cuda.cu -lhipblas -lrocblas
...

On the same laptop I downloaded v0.8.12 and it seems fine

# v0.8.12
$ ./Meta-Llama-3.1-8B-Instruct.Q5_K_M.llamafile.1 -ngl 9999
import_cuda_impl: initializing gpu module...
link_cuda_dso: note: dynamically linking ~/.llamafile/v/0.8.12/ggml-rocm.so
ggml_cuda_link: welcome to ROCm SDK with hipBLAS
link_cuda_dso: GPU support loaded
...

After further checking it shows that after this commit I would not be build llamafile that can run on my GPU.

By the way I am using AMD 6800U with 680M graphics card.

Just wonder if I miss anything on v0.8.13. Thanks

Version

llamafile v0.8.13

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

[sporteman]

Same issue in a Ubuntu 20.4 with a NVIDIA Corporation TU104GL [Tesla T4]