Building SPECFEM3D with GPU support on Ada Lovelace architecture

[denis-anuprienko]

Dear SPECFEM community,

I am trying to build SPECFEM3D on an Ada Lovelace GPU.
I've seen several "Using the GPU version of the code" sections in manuals for various SPECFEM codes, so I go like

./configure --with-cuda=cuda12 ,

but I get

Error in setConst_hprime_xx: invalid device symbol The problem is maybe that the architecture specified in the Makefile (-gencode=arch=compute_**,code="sm_**,compute_**") doesn't fit your card, please doublecheck

I examined Makefile to see the "gencode=..." section, but even after I change 90 to 89 (which should supposedly work for Ada Lovelace), the error still stays.

Since I see no mentions of Ada Lovelace (after Ampere just goes Hopper) in the "Using the GPU version of the code" sections, perhaps there are some inherent problems with running SPECFEM there? Or is there any experience?
I would greatly appreciate any guidance.

Sincerely,
Denis

[danielpeter]

the step with changing the gencode section looks like the right thing to do. so, in Makefile.in, you would change:

GENCODE_80 = -gencode=arch=compute_80,code=\"sm_80,compute_80\"

to

GENCODE_89 = -gencode=arch=compute_89,code=\"sm_89,compute_89\"

and then run the configuration with ./configure --with-cuda=cuda11 ... or similar with GENCODE_90 = .. and flag --with-cuda=cuda12.

it looks like the Ada Lovelace architecture was primarily for gaming, while Grace Hopper adopted more by HPC centers. anyway, it's difficult to debug without an actual card. maybe you can add more error output to see how the actual compilation line looks like in your case. and double-check with the details of your GPU card to see which CUDA SM version it should support.

there's also a small utility program in ./utils/GPU_tools/check_cuda_device.cu that you can try to compile and run on your card to get some device information:

cd ./utils/GPU_tools/
nvcc --gpu-architecture=sm_89 -o check_cuda_device check_cuda_device.cu
./check_cuda_device

it already helps to see if that compiles or not.

[denis-anuprienko]

Dear Daniel,
thank you for the reply. It actually worked as you suggested, I only had to change 89 to 86 for some weird reason.