Alltalk Multi-GPU training

[Darian461]

Hey! I just wanted to see if there was any consideration for training on a multi-GPU setup in the V2 Beta.
I was able to run training by doing: "python -m trainer.distribute --gpus '0,1' " with a seperate training script in the coqui repo. So training is technically possible with XTTS2.
Not exactly sure how it would be implemented in the alltalk setup, but it would be nice to see.
Outside of that, V2 beta looks great! Thank you for putting the effort in a project like this.

[erew123]

Hi @Darian461

Thanks on V2, hope to get it fully out of BETA soon! :)

I'm 99% sure that command line is actually a call/setup direct into the Huggingface Transformers and not actually part of anything within a Coqui script.

Coqui References

https://github.com/coqui-ai/Trainer?tab=readme-ov-file#training-with-accelerate
https://docs.coqui.ai/en/latest/training_a_model.html (also Step 4 on this link)

Huggingface Trainer References

https://huggingface.co/docs/transformers/en/main_classes/trainer

and as I understand this is a mix of setting the available/visible devices in a system for acceleration, which typically all devices are visible, unless you restrict them down (see the "I have multiple GPU's" here https://github.com/erew123/alltalk_tts?tab=readme-ov-file#startup-performance-and-compatibility-issues)

So the actual command line:

$ CUDA_VISIBLE_DEVICES="0, 1, 2" python -m trainer.distribute --script <path_to_your_script>/script.py

This bit CUDA_VISIBLE_DEVICES="0, 1, 2 is a system variable you need to set on a per system basis, pre-loading any script, though this is set to All devices are available for use as standard.

And this bit python -m trainer.distribute --script is telling Python to load in huggingface transformers with distributed processing. In other words, rather than automatically pick just one available device, use them all (or whatever you specified in CUDA_VISIBLE_DEVICES).

Finally <path_to_your_script>/script.py could be any script you wanted. The script doesn't have to be modified or altered in the slightest, Python and the Huggingface transformers are doing all the management/handling of all the devices.

Putting all this information together aka TLDR

In short, there would be no code changes to anything in AllTalk, or any Coqui scripts. You would always have to kick off the finetuning script with python -m trainer.distribute --script finetune.py so you could either do that manually OR simply alter the start_finetuning.bat (or sh file) and it should just always run distributed (add in the -m trainer.distribute --script)

So it may be better I document how people can do this, rather than force it into the start_finetuning batch/shell script, just because I cant say what other impacts this could have on some systems that may only be single GPU or who knows with AI trainers or across the multiple types of platforms I'm trying to support.

Does that cover off everything you need to know/suggest?

Thanks

[Darian461]

Wow thats a great explanation! That makes sense, I noticed that some of the other tts models I looked at can more or less automatically "see" multiple gpus. Whereas the coqui script complains and doesn't let you start training if it sees multiple cards. I was curious as to how that all worked.

Adding CUDA_VISIBLE_DEVICES=0,1 python -m trainer.distribute --script to the start script ended up sending an error. It looks like the module adds some arguments when it starts and that causes finetune.py to not start.

Wasn't a problem though, as I ended up creating a script and ran the fine-tuning manually. Then put the model into alltalk and continued to step 3.

Everything works great! The only other thing I would suggest is the option to remove specific entries in the TTS Generator. The sentence splitting is great but sometimes I end up wanting to change how the sentence reads or combine some to make it sound more natural. So being able to clear specific entries will help with that.
Again thank you for the response and all the work that you've put in!