-
Notifications
You must be signed in to change notification settings - Fork 100
QUDA Debugging
The CUDA toolkit include a valgrind-like tool cuda-memcheck which can be used to isolate memory errors. However, due to QUDA's large number of kernels, the default patching behavior of the tool can result in the host system running out of CPU memory, resulting in virtual memory swapping causing a huge slowdown. To rectify this issue, set the environment variable CUDA_MEMCHECK_PATCH_MODULE=1. Note this setting requires an NVIDIA driver 418 or greater.
QUDA supports address and undefined sanitizers to aid in finding subtle, difficult to find memory bugs. To enable the use of sanitizers, ensure that you build with g++ or clang++, and set -DCMAKE_BUILD_TYPE=SANITIZE. This will build QUDA with both the address and undefined sanitizers enabled (-fsanitize=address,undefined).
Due to conflicts with running the CUDA driver with address sanitizer, it is necessary to set the run-time environment variable ASAN_OPTIONS="protect_shadow_gap=0" when running. Memory leak checking can be disable if desired with detect_leaks=0 (default is enabled).
To run with maximum checking enabled use
ASAN_OPTIONS=protect_shadow_gap=0,strict_string_checks=1:detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1
The use of these run-time sanitizers has been critical in finding bugs in QUDA, and should be complementary to running with valgrind (but not at the same time).
One can configure cmake to use clang-tidy which will run on the non-CUDA files with this command
cmake "-DCMAKE_CXX_CLANG_TIDY=/usr/bin/clang-tidy-8;-checks=*" $PATH_TO_QUDA
When building, you be greeted with a slew of warnings, that may or may not be important. At present this will do nothing for CUDA source code files (*.cu) though we may find a solution to this in the future.