NaN loss issue during training

[abrar-aw]

Hi there, I really appreciate your work and I am trying to implement the model on a custom dataset consisting of just 100 words in native sign language. Each word contains about 8 video samples. However, when I try to train your custom model, I keep getting NaN loss and NaN validation loss. I have so far tried tweaking it with decreasing the learning rate, batch size and even some parameters in the model, like the dropout rate and different RNN like LSTM, but I just can't seem to get it out of NaN values. The dataset I am using is similar to WLASL dataset, 256x256 res @ 25 FPS. Could you please advise on what I could do to get it to work? I need some expert opinion on this please, thanks!!

Added info: This only happens when I am trying to train more 20 classes/glosses.

[simonefinelli]

I think the number of videos may have an impact. In fact, by increasing the number of classes, if there are few videos for each case, then it could be that the network has a loss that is too large to be represented (the network has a lot of difficulty in distinguishing cases). Also make sure that the number of output neurons of the last layer coincides with the number of classes.