Load Trained Model

[krishankantmehra]

Greetings everyone!
I'm trying to fine-tune this model using fine-tune.py from this repo. I am using seamlessM4T_medium as base for this. I'm able to train and generate checkpoints.pt file.
But I'm not able to find any instructions on how to use this file to evaluate fine-tuning. Can someone please share any link if any instructions are available or give any suggestion on how to do it? any help is much appreciated. Thanks.

[developeranalyser]

me too..
you find any way ?

[krishankantmehra]

Nope...

[developeranalyser]

you can use Translator in

translator = Translator(
model_name_or_card="seamlessM4T_v2_large",
vocoder_name_or_card="vocoder_v2",
device=device,
dtype=dtype,
apply_mintox=True,
)

translator.from_pre_tained .(torch.load(...))

ok ?

[RRThivyan]

Hello, I am working on TTS project. I need to convert text to speech for indic languages like Kannada, Tamil, Marathi etc which don't have support in seamlessm4t. I am trying to finetune using google/fleur audio dataset. But I am getting assertion error. Would anyone kindly help me what is the mistake I did? This error is causing severe lag in project completion.

From my understanding, the issue arise due to units field is Null; but the fleur dataset has all records have Null for units.
I downloaded the dataset using seamlessm4t m4t/dataset.py only and its said that the fleur dataset is compatible with finetune.py, but the code is throwing this error.

This is the command I gave and the following error I received.

thivyan@thivyan-tts:/home/jupyter/myfiles$ m4t_finetune
--train_dataset /home/jupyter/myfiles/fleurs/train/train_manifest.json
--eval_dataset /home/jupyter/myfiles/fleurs/validation/validation_manifest.json
--batch_size 10
--eval_steps 1000
--max_epochs 100
--learning_rate 0.00005
--patience 10
--save_model_to /home/jupyter/myfiles/checkpoints/tam_tts_m4tL.pt
--model_name seamlessM4T_large
--mode TEXT_TO_SPEECH
2024-09-03 09:19:07,679 ERROR -- seamless_communication.cli.m4t.finetune.dist_utils.67686: Cannot init disributed context, as environment varaibles are not set.
Using the cached tokenizer of seamlessM4T_large. Set force to True to download again.
2024-09-03 09:19:07,932 INFO -- finetune.67686: Finetune Params: FinetuneParams(model_name='seamlessM4T_large', save_model_path=PosixPath('/home/jupyter/myfiles/checkpoints/tam_tts_m4tL.pt'), finetune_mode=<FinetuneMode.TEXT_TO_SPEECH: 'TEXT_TO_SPEECH'>, float_dtype=torch.float16, max_epochs=100, label_smoothing=0.2, warmup_steps=100, log_steps=10, eval_steps=1000, patience=10, learning_rate=5e-05, train_batch_size=10, eval_batch_size=10, device=device(type='cuda'))
Using the cached checkpoint of seamlessM4T_large. Set force to True to download again.
2024-09-03 09:19:42,256 INFO -- seamless_communication.cli.m4t.finetune.trainer.67686: Freeze s2t: True, freeze t2u: False
/home/jupyter/myfiles/seamless_env/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:28: UserWarning: The verbose parameter is deprecated. Please use get_last_lr() to access the learning rate.
warnings.warn("The verbose parameter is deprecated. Please use get_last_lr() "
2024-09-03 09:19:42,267 INFO -- seamless_communication.cli.m4t.finetune.trainer.67686: Start Finetuning
2024-09-03 09:19:42,267 INFO -- seamless_communication.cli.m4t.finetune.trainer.67686: Evaluation Step 0...
Traceback (most recent call last):
File "/home/jupyter/myfiles/seamless_env/bin/m4t_finetune", line 8, in
sys.exit(main())
File "/home/jupyter/myfiles/seamless_communication/src/seamless_communication/cli/m4t/finetune/finetune.py", line 217, in main
finetune.run()
File "/home/jupyter/myfiles/seamless_communication/src/seamless_communication/cli/m4t/finetune/trainer.py", line 411, in run
self._eval_model(n_batches=100)
File "/home/jupyter/myfiles/seamless_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/jupyter/myfiles/seamless_communication/src/seamless_communication/cli/m4t/finetune/trainer.py", line 351, in _eval_model
loss = self.calc_loss(batch, *self.model(batch))
File "/home/jupyter/myfiles/seamless_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/jupyter/myfiles/seamless_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jupyter/myfiles/seamless_communication/src/seamless_communication/cli/m4t/finetune/trainer.py", line 127, in forward
assert batch.text_to_units.prev_output_tokens is not None
AssertionError

Here is a sample from train_manifest.json file
{"source": {"id": 878, "lang": "tam", "text": "\u0ba4\u0bca\u0bb1\u0bcd\u0bb1\u0bc1 \u0ba8\u0bcb\u0baf\u0bbe\u0ba9\u0ba4\u0bc1 \u0b92\u0bb0\u0bc1 \u0bb5\u0bc8\u0bb0\u0bb8\u0bcd \u0baa\u0bc7\u0b95\u0bcd\u0b9f\u0bc0\u0bb0\u0bbf\u0baf\u0bae\u0bcd \u0baa\u0bc2\u0b9e\u0bcd\u0b9a\u0bc8 \u0b85\u0bb2\u0bcd\u0bb2\u0ba4\u0bc1 \u0baa\u0bbf\u0bb1 \u0ba4\u0bca\u0bb1\u0bcd\u0bb1\u0bc1\u0b95\u0bb3\u0bcd \u0baa\u0bcb\u0ba9\u0bcd\u0bb1 \u0ba8\u0bc1\u0ba3\u0bcd\u0ba3\u0bc1\u0baf\u0bbf\u0bb0\u0bcd\u0b95\u0bb3\u0bbe\u0bb2\u0bcd \u0b8f\u0bb1\u0bcd\u0baa\u0b9f\u0bc1\u0bb5\u0ba4\u0bc1", "audio_local_path": "/home/jupyter/myfiles/fleurs/train/downloads/extracted/dd294ecfb28c8d9c95dda2bf0dc971d94422c064ec4e3d2220ff7fb1e7081688/train/10004352805657441150.wav", "waveform": null, "sampling_rate": 16000, "units": null}, "target": {"id": 878, "lang": "tam", "text": "\u0ba4\u0bca\u0bb1\u0bcd\u0bb1\u0bc1 \u0ba8\u0bcb\u0baf\u0bbe\u0ba9\u0ba4\u0bc1 \u0b92\u0bb0\u0bc1 \u0bb5\u0bc8\u0bb0\u0bb8\u0bcd \u0baa\u0bc7\u0b95\u0bcd\u0b9f\u0bc0\u0bb0\u0bbf\u0baf\u0bae\u0bcd \u0baa\u0bc2\u0b9e\u0bcd\u0b9a\u0bc8 \u0b85\u0bb2\u0bcd\u0bb2\u0ba4\u0bc1 \u0baa\u0bbf\u0bb1 \u0ba4\u0bca\u0bb1\u0bcd\u0bb1\u0bc1\u0b95\u0bb3\u0bcd \u0baa\u0bcb\u0ba9\u0bcd\u0bb1 \u0ba8\u0bc1\u0ba3\u0bcd\u0ba3\u0bc1\u0baf\u0bbf\u0bb0\u0bcd\u0b95\u0bb3\u0bbe\u0bb2\u0bcd \u0b8f\u0bb1\u0bcd\u0baa\u0b9f\u0bc1\u0bb5\u0ba4\u0bc1", "audio_local_path": "/home/jupyter/myfiles/fleurs/train/downloads/extracted/dd294ecfb28c8d9c95dda2bf0dc971d94422c064ec4e3d2220ff7fb1e7081688/train/13289636858665595157.wav", "waveform": null, "sampling_rate": 16000, "units": null}}

[deepak-kumar-98]

@RRThivyan how much dataset you prepared for training?
can u pls share 1 instance of it, so that i can pepare for hindi as well as bengali and trying to fine tune seamlessm4t for speech to speech in hindi to bengali,
i am new here, any help would greatly appriciated

[deepak-kumar-98]

@RRThivyan same error i am encountering, is it resolved?