RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

[HaoxuZhang]

Hi, I was running test.py on single GPU but got this error. Do you know what is wrong? Thank you so much.

Ubuntu 16.04
python 2.7
pytorch 0.4.0
CUDA 9.0
CUDNN 7.0

Traceback (most recent call last):
  File "test.py", line 185, in <module>
    main()
  File "test.py", line 54, in main
    test(file_list[0], model_path)
  File "test.py", line 111, in test
    pred_map = net.test_forward(img)
  File "/home/work/zhanghaoxu/GCC-SFCN/models/CC.py", line 43, in test_forward
    density_map = self.CCN(img)            
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/work/zhanghaoxu/GCC-SFCN/models/resSFCN.py", line 44, in forward
    x = self.frontend(x)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torchvision/models/resnet.py", line 83, in forward
    out = self.conv2(out)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/software/conda/envs/zhanghaoxu/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

[HaoxuZhang]

@gjy3035

[DenisTrukhin]

@HaoxuZhang I had the same error due to incorrect version of pytorch (it was compiled for CUDA 8.0). When I installed version compatible with CUDA 9.0, it works well.
Desired version (torch-0.4.0-cp27-cp27mu-linux_x86_64.whl) here:
https://download.pytorch.org/whl/cu90/torch_stable.html