mixed languages sentence.

[shuanshuan]

Is there any way or model support mixed languages sentence。

  我真的很喜欢 AllTalkTTS

It mainly speaks Chinese and pronounces English correctly.

[erew123]

Hi @shuanshuan

Just so that we are on the same page and please correct me if I am wrong, but you want to be able to send over a single block of text to a TTS engine that is mixed languages and have it generate the speech as a 1x process.

So, assuming that is the correct interpretation, the answer is typically a no.... BUT it may be possible with the XTTS model (I cant speak for other models atm).

If you want to try this, you will need to be on AllTalk v2 and please understand this is my best guess after a quick little look at how the XTTS tokenizer/phonemizer is working. Please read all the below before you make any changes as it may not be exactly what you want.

We would need to update the allowed API passthrough to allow square brackets [ and ]

So in Global Settings > AllTalk API Defaults and at the bottom of the page, update the API Allowed Text Filtering/Passthrough Settings to the following:

[^a-zA-Z0-9\s.,;:!?\-\'"$\[\]\u0400-\u04FF\u00C0-\u017F\u0150\u0151\u0170\u0171\u011E\u011F\u0130\u0131\u0900-\u097F\u2018\u2019\u201C\u201D\u3001\u3002\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF\u0600-\u06FF\u0750-\u077F\uFB50-\uFDFF\uFE70-\uFEFF\uAC00-\uD7A3\u1100-\u11FF\u3130-\u318F\uFF01\uFF0c\uFF1A\uFF1B\uFF1F]

Once you have done that and saved the settings, reload the XTTS model.

Once that is loaded, you can try something like:

[zh]我真的很喜欢[en]AllTalkTTS

and you should see the brackets/language shown [zh] [en], but not pronounced, only the following blocks of text.

As you will see, we are specifying the language in brackets e.g. [zh] which should force the tokenizer to use that language for that section of the speech....... please note the use of "should".

To explain the tokenizers https://github.com/erew123/alltalk_tts/blob/main/system/ft_tokenizer/tokenizer.py or actually phonemization process, these are used by all TTS engines to break down specific languages and handle the language unique specific quirks. So many TTS engines use espeak-ng or gruut or something like that. When you tell it which language to use, it can then deal with the languages unique things like expanding abbreviations (Dr to Doctor), numbers (12 to twelve), symbols (% to percent), vocabulary mapping etc, basically things that are unique/special about that language. Hence, you always (currently) have to tell the TTS engine what language the text is in, so that it can correctly process the language and its unique characteristics, quirks and symbols then convert those to sound representations that match the language.

So, in theory, the way I have shown above, to use the [2x digit language character code] should work with XTTS and it should see that code as being the language for the next block of text.

As far as I am aware, there is no TTS engine/model that will just identify a language without a character code being sent. So currently, you would always have to tag each section of language within the text.

Hope that helps clarify.

Thanks

[erew123]

Hi @shuanshuan

I used the XTTS 2.0.3 model.

[en]Hello, how are you? [fr]Bonjour, comment allez-vous? [it]Ciao, come stai? [zh]你好，你好吗？

Are you certain the API Allowed Text Filtering/Passthrough Settings has been updated correctly and the brackets are passing through?

[^a-zA-Z0-9\s.,;:!?\-\'"$\[\]\u0400-\u04FF\u00C0-\u017F\u0150\u0151\u0170\u0171\u011E\u011F\u0130\u0131\u0900-\u097F\u2018\u2019\u201C\u201D\u3001\u3002\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF\uF900-\uFAFF\u0600-\u06FF\u0750-\u077F\uFB50-\uFDFF\uFE70-\uFEFF\uAC00-\uD7A3\u1100-\u11FF\u3130-\u318F\uFF01\uFF0c\uFF1A\uFF1B\uFF1F]

bracketsAREpassed.mp4

bracketsareNOTpassed.mp4

[shuanshuan]

These are correct
[en]Hello, how are you? [fr]Bonjour, comment allez-vous? [it]Ciao, come stai?
but the [zh] 你好，你好吗？ is not

There is a bug,which may be related to this. When you have chosen zh from languages dropdown list , then generateTTS , the console how error:

[AllTalk API] Error with API request: language: language needs to be one of the following: ar, zh-cn, cs, nl, en, fr, de, hu, hi, it, ja, ko, pl, pt, ru, es, tr.

[erew123]

Hi @shuanshuan

No probs. I've set it to allow both zh and zh-cn a40c7a7

In Coqui's tokenizer, it should work with both

alltalk_tts/system/ft_tokenizer/tokenizer.py

Line 22 in 1a8a20f

if lang == "zh":

and

alltalk_tts/system/ft_tokenizer/tokenizer.py

Line 676 in 1a8a20f

lang = "zh-cn" if lang == "zh" else lang

Though you may wish to try both in your text line to be generated or the API to see what you get. Unfortunately I do not speak Chinese so I cannot hear what is or isnt being produced by the Coqui AI model. All I can say for sure is that AllTalk is passing over the text to Coqui's script and the AI model.

Thanks

[shuanshuan]

You can hear the difference, if you the languages to zh, and genrate the voice in Chinese sentence only.

Even you remove the language breackets, set the language to en, the follow sentence are still be pronounceed correctly.

Hello, how are you? Bonjour, comment allez-vous? Ciao, come stai?

[erew123]

Hi @shuanshuan As mentioned you may want to try [zh-cn] and see if that makes any difference. If not, the only suggestion I have is to address this to the person maintaining the Coqui scripts. The backend Coqui scripts are being maintained here https://github.com/idiap/coqui-ai-TTS FYI, this person is NOT a member of Coqui nor worked for Coqui and as I understand, they do this in their free time, so I am unable to comment as to how quickly they may be able to respond.

They would be able to look at the tokenizer, however, like myself, Im not sure how deep their actual understanding of the AI model will be, being that it was not created by either of us.

Thanks

[shuanshuan]

[zh]我真的很喜欢[en]AllTalkTTS
Cool , I will try the V2
Thank you