Training so slow

[hoyden]

I trained the model with my own data, but it so slow! It took me nearly 5 days to train 30k steps. BTW, I used 4 Tesla P40. Do you know how to solve it?

[ivanvovk]

@hoyden what RTF do you have at Tesla P40?

[hoyden]

@ivanvovk RTF 0.618 from the training log

[ivanvovk]

@hoyden I guess there might be some problems with distributed training. Maybe try to restart training on a single GPU, it won't differ too much.