Multispeaker fine-tuning conditions

[152334H]

So far, I have failed to adequately train a fine-tuned model on a large variety of speakers. I notice that, even when I undersample common speakers to make the dataset more roughly diverse, I still notice a general failure of the resultant model to handle most speakers well -- even on the speakers with the maximum of 1k lines included!

Possible avenues for investigation:

• The issue of cheater latents
• Fine-tuning other parts of the model? But then why would it work well enough for a single speaker, but not multiple speakers?
• Maybe a higher learning rate is needed? This sounds dumb, because the model was already hovering around a specific loss after 1k steps on 1e-5, but maybe it's stuck in some local minima?
• To check: is the dataset actually randomised in order during training? I noticed that certain voices were better than others, maybe I just accidentally grouped all of them?
• Batch sizes? Maybe it'll train better if I just crank up grad accum + bs.
• Just train longer?

[Brugio96]

Hi @152334H and thanks for your work, have you found a solution to this problem?

[152334H]

I am quitting text-to-speech; blogpost coming Soon:tm: