Discontinuous values in TensorBoard when restarting the training

[NaAMue]

Hi,
I have a question regarding the training values in TensorBoard. When restarting training a network where training has been stopped (e.g. here at E:123 and E:447), values like r.m.s.e., efficiency, Jaccard index, loss… make a jump and seem to converge to a different value. Does this have any influence on the network performance and should I rather train the the network in one go without breaks?

[Haydnspass]

Hey @NaAMue ,
I think this is simply due to slightly different validation sets that are re-generated upon warm starting. i would not worry, however you could evaluate both models on the same validation to double-check.

[Haydnspass]

@NaAMue Do you always observe the performance worsens, or is it sometimes the other way around (which I would expect)?

[NaAMue]

Hi Lucas,

Thank you for your answers. Yes, the jumps happen in both directions.

Best,
Nadine