Error when training in StyleGAN mode using a database of npy files

[albusdemens]

Hi, I am trying to train StyleGAN using a database of npy files. When I launch python train.py PGAN -c maps.json things run fine, but when instead I use python train.py StyleGAN -c maps.json I get a CUDNN error (see below). Do you have suggestions on how to fix this? Thanks!

Traceback (most recent call last):
  File "train.py", line 137, in <module>
    GANTrainer.train()
  File "/home/alberto/Documents/StyleGAN_reimplementation/pytorch_GAN_zoo/models/trainer/progressive_gan_trainer.py", line 237, in train
    maxIter=self.modelConfig.maxIterAtScale[scale])
  File "/home/alberto/Documents/StyleGAN_reimplementation/pytorch_GAN_zoo/models/trainer/gan_trainer.py", line 487, in trainOnEpoch
    inputLabels=labels)
  File "/home/alberto/Documents/StyleGAN_reimplementation/pytorch_GAN_zoo/models/base_GAN.py", line 223, in optimizeParameters
    lossD.backward(retain_graph=True)
  File "/home/alberto/anaconda3/envs/SG_pytorch/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/alberto/anaconda3/envs/SG_pytorch/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

[Molugan]

Hi,

Sorry for the delay, can I have a sample of your file maps.json ?

[ds2268]

Hi @Molugan

I experience the same error with StyleGAN:

python train.py StyleGAN -c some_custom_config.json --restart -n some_custom_name

-> RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input

This happens right at the beginning.
Might be something with the PyTorch version (pytorch/pytorch#32564)

[JanineCHEN]

Hi everyone. has this issue been resolved?

[JanineCHEN]

What I have found:
The issue exits with pytorch1.4 cuda-toolkit 10.0
Yet the issue is gone when I shift to pytorch1.3 cuda-toolkit 10.0
With exactly the same code.