loss nan error

[AppleJoker94]

Hi, really thank you for your work.
I try to do this, but in step 2 Train the Siamese-Shift Encoder, my loss is nan.
Is there any solution?

model.x = (None, 576, 1)
model.y = (None, 5)
C:\Users\CAIKVM3\SJ_WS\Woosung\FaultNet-master\FaultNet-master\models\FaultNet_hard--1e-07a_0.0001lr_5P_5K_50f_15k_2s_1x2x4d_1n_0.2m_2win_0.075sft_5od_10pat0.h5
Model: "encoder_model"

---

Layer (type) Output Shape Param # Connected to

input_layer (InputLayer) [(None, 576, 1)] 0

---

initial_conv (Conv1D) (None, 576, 50) 800 input_layer[0][0]

---

dilated_conv_2_tanh_s0 (Conv1D) (None, 576, 50) 37550 initial_conv[0][0]

---

activation (Activation) (None, 576, 50) 0 dilated_conv_2_tanh_s0[0][0]

---

lambda (Lambda) (None, 576, 50) 0 activation[0][0]

---

spatial_dropout1d (SpatialDropo (None, 576, 50) 0 lambda[0][0]

---

conv1d (Conv1D) (None, 576, 50) 2550 spatial_dropout1d[0][0]

---

add (Add) (None, 576, 50) 0 initial_conv[0][0]
conv1d[0][0]

---

dilated_conv_4_tanh_s0 (Conv1D) (None, 576, 50) 37550 add[0][0]

---

activation_1 (Activation) (None, 576, 50) 0 dilated_conv_4_tanh_s0[0][0]

---

lambda_1 (Lambda) (None, 576, 50) 0 activation_1[0][0]

---

spatial_dropout1d_1 (SpatialDro (None, 576, 50) 0 lambda_1[0][0]

---

conv1d_1 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_1[0][0]

---

add_1 (Add) (None, 576, 50) 0 add[0][0]
conv1d_1[0][0]

---

dilated_conv_16_tanh_s0 (Conv1D (None, 576, 50) 37550 add_1[0][0]

---

activation_2 (Activation) (None, 576, 50) 0 dilated_conv_16_tanh_s0[0][0]

---

lambda_2 (Lambda) (None, 576, 50) 0 activation_2[0][0]

---

spatial_dropout1d_2 (SpatialDro (None, 576, 50) 0 lambda_2[0][0]

---

conv1d_2 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_2[0][0]

---

add_2 (Add) (None, 576, 50) 0 add_1[0][0]
conv1d_2[0][0]

---

dilated_conv_2_tanh_s1 (Conv1D) (None, 576, 50) 37550 add_2[0][0]

---

activation_3 (Activation) (None, 576, 50) 0 dilated_conv_2_tanh_s1[0][0]

---

lambda_3 (Lambda) (None, 576, 50) 0 activation_3[0][0]

---

spatial_dropout1d_3 (SpatialDro (None, 576, 50) 0 lambda_3[0][0]

---

conv1d_3 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_3[0][0]

---

add_3 (Add) (None, 576, 50) 0 add_2[0][0]
conv1d_3[0][0]

---

dilated_conv_4_tanh_s1 (Conv1D) (None, 576, 50) 37550 add_3[0][0]

---

activation_4 (Activation) (None, 576, 50) 0 dilated_conv_4_tanh_s1[0][0]

---

lambda_4 (Lambda) (None, 576, 50) 0 activation_4[0][0]

---

spatial_dropout1d_4 (SpatialDro (None, 576, 50) 0 lambda_4[0][0]

---

conv1d_4 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_4[0][0]

---

add_4 (Add) (None, 576, 50) 0 add_3[0][0]
conv1d_4[0][0]

---

dilated_conv_16_tanh_s1 (Conv1D (None, 576, 50) 37550 add_4[0][0]

---

activation_5 (Activation) (None, 576, 50) 0 dilated_conv_16_tanh_s1[0][0]

---

lambda_5 (Lambda) (None, 576, 50) 0 activation_5[0][0]

---

spatial_dropout1d_5 (SpatialDro (None, 576, 50) 0 lambda_5[0][0]

---

conv1d_5 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_5[0][0]

---

add_6 (Add) (None, 576, 50) 0 conv1d[0][0]
conv1d_1[0][0]
conv1d_2[0][0]
conv1d_3[0][0]
conv1d_4[0][0]
conv1d_5[0][0]

---

activation_6 (Activation) (None, 576, 50) 0 add_6[0][0]

---

conv1d_6 (Conv1D) (None, 562, 1) 751 activation_6[0][0]

---

flatten (Flatten) (None, 562) 0 conv1d_6[0][0]

---

dense (Dense) (None, 5) 2815 flatten[0][0]

---

output_dense (Activation) (None, 5) 0 dense[0][0]

---

lambda_6 (Lambda) (None, 5) 0 output_dense[0][0]

Total params: 244,966
Trainable params: 244,966
Non-trainable params: 0

---

C:\Users\CAIKVM3\Anaconda3\lib\site-packages\keras\optimizer_v2\optimizer_v2.py:355: UserWarning: The lr argument is deprecated, use learning_rate instead.
warnings.warn(
C:\Users\CAIKVM3\Anaconda3\lib\site-packages\keras\engine\training.py:1972: UserWarning: Model.fit_generator is deprecated and will be removed in a future version. Please use Model.fit, which supports generators.
warnings.warn('Model.fit_generator is deprecated and '
C:\Users\CAIKVM3\Anaconda3\lib\site-packages\keras\utils\generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
warnings.warn('Custom mask layers require a config and must override '
Epoch 1/1000
401/401 [==============================] - 30s 55ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 2/1000
401/401 [==============================] - 21s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 3/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 4/1000
401/401 [==============================] - 21s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 5/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 6/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 7/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 8/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 9/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 10/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 11/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
C:\Users\CAIKVM3\SJ_WS\Woosung\FaultNet-master\FaultNet-master\models\FaultNet_hard--1e-07a_0.0001lr_5P_5K_50f_15k_2s_1x2x4d_1n_0.2m_2win_0.1sft_5od_10pat0.h5
Model: "encoder_model"

---

Layer (type) Output Shape Param # Connected to

input_layer (InputLayer) [(None, 576, 1)] 0

---

initial_conv (Conv1D) (None, 576, 50) 800 input_layer[0][0]

---

dilated_conv_2_tanh_s0 (Conv1D) (None, 576, 50) 37550 initial_conv[0][0]

---

activation (Activation) (None, 576, 50) 0 dilated_conv_2_tanh_s0[0][0]

---

lambda (Lambda) (None, 576, 50) 0 activation[0][0]

---

spatial_dropout1d (SpatialDropo (None, 576, 50) 0 lambda[0][0]

---

conv1d (Conv1D) (None, 576, 50) 2550 spatial_dropout1d[0][0]

---

add (Add) (None, 576, 50) 0 initial_conv[0][0]
conv1d[0][0]

---

dilated_conv_4_tanh_s0 (Conv1D) (None, 576, 50) 37550 add[0][0]

---

activation_1 (Activation) (None, 576, 50) 0 dilated_conv_4_tanh_s0[0][0]

---

lambda_1 (Lambda) (None, 576, 50) 0 activation_1[0][0]

---

spatial_dropout1d_1 (SpatialDro (None, 576, 50) 0 lambda_1[0][0]

---

conv1d_1 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_1[0][0]

---

add_1 (Add) (None, 576, 50) 0 add[0][0]
conv1d_1[0][0]

---

dilated_conv_16_tanh_s0 (Conv1D (None, 576, 50) 37550 add_1[0][0]

---

activation_2 (Activation) (None, 576, 50) 0 dilated_conv_16_tanh_s0[0][0]

---

lambda_2 (Lambda) (None, 576, 50) 0 activation_2[0][0]

---

spatial_dropout1d_2 (SpatialDro (None, 576, 50) 0 lambda_2[0][0]

---

conv1d_2 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_2[0][0]

---

add_2 (Add) (None, 576, 50) 0 add_1[0][0]
conv1d_2[0][0]

---

dilated_conv_2_tanh_s1 (Conv1D) (None, 576, 50) 37550 add_2[0][0]

---

activation_3 (Activation) (None, 576, 50) 0 dilated_conv_2_tanh_s1[0][0]

---

lambda_3 (Lambda) (None, 576, 50) 0 activation_3[0][0]

---

spatial_dropout1d_3 (SpatialDro (None, 576, 50) 0 lambda_3[0][0]

---

conv1d_3 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_3[0][0]

---

add_3 (Add) (None, 576, 50) 0 add_2[0][0]
conv1d_3[0][0]

---

dilated_conv_4_tanh_s1 (Conv1D) (None, 576, 50) 37550 add_3[0][0]

---

activation_4 (Activation) (None, 576, 50) 0 dilated_conv_4_tanh_s1[0][0]

---

lambda_4 (Lambda) (None, 576, 50) 0 activation_4[0][0]

---

spatial_dropout1d_4 (SpatialDro (None, 576, 50) 0 lambda_4[0][0]

---

conv1d_4 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_4[0][0]

---

add_4 (Add) (None, 576, 50) 0 add_3[0][0]
conv1d_4[0][0]

---

dilated_conv_16_tanh_s1 (Conv1D (None, 576, 50) 37550 add_4[0][0]

---

activation_5 (Activation) (None, 576, 50) 0 dilated_conv_16_tanh_s1[0][0]

---

lambda_5 (Lambda) (None, 576, 50) 0 activation_5[0][0]

---

spatial_dropout1d_5 (SpatialDro (None, 576, 50) 0 lambda_5[0][0]

---

conv1d_5 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_5[0][0]

---

add_6 (Add) (None, 576, 50) 0 conv1d[0][0]
conv1d_1[0][0]
conv1d_2[0][0]
conv1d_3[0][0]
conv1d_4[0][0]
conv1d_5[0][0]

---

activation_6 (Activation) (None, 576, 50) 0 add_6[0][0]

---

conv1d_6 (Conv1D) (None, 562, 1) 751 activation_6[0][0]

---

flatten (Flatten) (None, 562) 0 conv1d_6[0][0]

---

dense (Dense) (None, 5) 2815 flatten[0][0]

---

output_dense (Activation) (None, 5) 0 dense[0][0]

---

lambda_6 (Lambda) (None, 5) 0 output_dense[0][0]

Total params: 244,966
Trainable params: 244,966
Non-trainable params: 0

---

Epoch 1/1000
401/401 [==============================] - 22s 55ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 2/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 3/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 4/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 5/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 6/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 7/1000
401/401 [==============================] - 21s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 8/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 9/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 10/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 11/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
C:\Users\CAIKVM3\SJ_WS\Woosung\FaultNet-master\FaultNet-master\models\FaultNet_hard--1e-07a_0.0001lr_5P_5K_50f_15k_2s_1x2x4d_1n_0.2m_2win_0.2sft_5od_10pat0.h5
Model: "encoder_model"

---

Layer (type) Output Shape Param # Connected to

input_layer (InputLayer) [(None, 576, 1)] 0

---

initial_conv (Conv1D) (None, 576, 50) 800 input_layer[0][0]

---

dilated_conv_2_tanh_s0 (Conv1D) (None, 576, 50) 37550 initial_conv[0][0]

---

activation (Activation) (None, 576, 50) 0 dilated_conv_2_tanh_s0[0][0]

---

lambda (Lambda) (None, 576, 50) 0 activation[0][0]

---

spatial_dropout1d (SpatialDropo (None, 576, 50) 0 lambda[0][0]

---

conv1d (Conv1D) (None, 576, 50) 2550 spatial_dropout1d[0][0]

---

add (Add) (None, 576, 50) 0 initial_conv[0][0]
conv1d[0][0]

---

dilated_conv_4_tanh_s0 (Conv1D) (None, 576, 50) 37550 add[0][0]

---

activation_1 (Activation) (None, 576, 50) 0 dilated_conv_4_tanh_s0[0][0]

---

lambda_1 (Lambda) (None, 576, 50) 0 activation_1[0][0]

---

spatial_dropout1d_1 (SpatialDro (None, 576, 50) 0 lambda_1[0][0]

---

conv1d_1 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_1[0][0]

---

add_1 (Add) (None, 576, 50) 0 add[0][0]
conv1d_1[0][0]

---

dilated_conv_16_tanh_s0 (Conv1D (None, 576, 50) 37550 add_1[0][0]

---

activation_2 (Activation) (None, 576, 50) 0 dilated_conv_16_tanh_s0[0][0]

---

lambda_2 (Lambda) (None, 576, 50) 0 activation_2[0][0]

---

spatial_dropout1d_2 (SpatialDro (None, 576, 50) 0 lambda_2[0][0]

---

conv1d_2 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_2[0][0]

---

add_2 (Add) (None, 576, 50) 0 add_1[0][0]
conv1d_2[0][0]

---

dilated_conv_2_tanh_s1 (Conv1D) (None, 576, 50) 37550 add_2[0][0]

---

activation_3 (Activation) (None, 576, 50) 0 dilated_conv_2_tanh_s1[0][0]

---

lambda_3 (Lambda) (None, 576, 50) 0 activation_3[0][0]

---

spatial_dropout1d_3 (SpatialDro (None, 576, 50) 0 lambda_3[0][0]

---

conv1d_3 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_3[0][0]

---

add_3 (Add) (None, 576, 50) 0 add_2[0][0]
conv1d_3[0][0]

---

dilated_conv_4_tanh_s1 (Conv1D) (None, 576, 50) 37550 add_3[0][0]

---

activation_4 (Activation) (None, 576, 50) 0 dilated_conv_4_tanh_s1[0][0]

---

lambda_4 (Lambda) (None, 576, 50) 0 activation_4[0][0]

---

spatial_dropout1d_4 (SpatialDro (None, 576, 50) 0 lambda_4[0][0]

---

conv1d_4 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_4[0][0]

---

add_4 (Add) (None, 576, 50) 0 add_3[0][0]
conv1d_4[0][0]

---

dilated_conv_16_tanh_s1 (Conv1D (None, 576, 50) 37550 add_4[0][0]

---

activation_5 (Activation) (None, 576, 50) 0 dilated_conv_16_tanh_s1[0][0]

---

lambda_5 (Lambda) (None, 576, 50) 0 activation_5[0][0]

---

spatial_dropout1d_5 (SpatialDro (None, 576, 50) 0 lambda_5[0][0]

---

conv1d_5 (Conv1D) (None, 576, 50) 2550 spatial_dropout1d_5[0][0]

---

add_6 (Add) (None, 576, 50) 0 conv1d[0][0]
conv1d_1[0][0]
conv1d_2[0][0]
conv1d_3[0][0]
conv1d_4[0][0]
conv1d_5[0][0]

---

activation_6 (Activation) (None, 576, 50) 0 add_6[0][0]

---

conv1d_6 (Conv1D) (None, 562, 1) 751 activation_6[0][0]

---

flatten (Flatten) (None, 562) 0 conv1d_6[0][0]

---

dense (Dense) (None, 5) 2815 flatten[0][0]

---

output_dense (Activation) (None, 5) 0 dense[0][0]

---

lambda_6 (Lambda) (None, 5) 0 output_dense[0][0]

Total params: 244,966
Trainable params: 244,966
Non-trainable params: 0

---

Epoch 1/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 2/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 3/1000
401/401 [==============================] - 21s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 4/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 5/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 6/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 7/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 8/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 9/1000
401/401 [==============================] - 21s 53ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 10/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00
Epoch 11/1000
401/401 [==============================] - 22s 54ms/step - loss: nan - triplet_acc: 0.0000e+00 - val_loss: nan - val_triplet_acc: 0.0000e+00