Helping French TTS to pronounce numbers by notkaramel · Pull Request #739 · Shared-Reality-Lab/IMAGE-server

notkaramel · 2023-07-19T23:06:13Z

Changes Made

Issue: French TTS (espnet-tts-fr) struggles with numerical numbers #723
Added num2words library to help convert numbers in numeric form to text (e.g. "42" to "quarante deux")
Added debugging logs indicating cases of numeric conversion: ("segment" as a part of the input string that's separated by a space)
- Case 0: segment is numeric or float number (e.g., "2012", "23.12", "2021."). This includes case where a number is at the end of a sentence with "." as a part of the segment.
- Case 1: segment has ",". This is due to multilang-support tranlated to Metropolitan French where they have a different notation system.
- Case 2: segment is a range, containing a hyphen '-'. This is often seen in charts with age or year range. It also requires extra syntax to be added. E.g., "34-59" would be "de trente quatre à cinquante neuf".
- NOTE: This might conflict with words in French that has hyphen, like "gratte-ciel", "arc-en-ciel", or names like "Lionel-Groulx".

Testings

Tested with large numbers with demimal point, year: https://etherscan.io/chart/ethersupplygrowth
Tested with range: https://www.inspq.qc.ca/covid-19/donnees/age-sexe
Tested with regular numbers: any graphics with countable quantity of object(s).

Please ensure you've followed the checklist and provide all the required information before requesting a review.
If you do not have everything applicable to your PR, it will not be reviewed!
If you don't know what something is or if it applies to you, ask!

Don't delete below this line.

Required Information

I referenced the issue addressed in this PR.
I described the changes made and how these address the issue.
I described how I tested these changes.

Coding/Commit Requirements

I followed applicable coding standards (e.g., PEP8)
I have not committed any models or other large files.

New Component Checklist (mandatory for new microservices)

I added an entry to docker-compose.yml and build.yml.
I created A CI workflow under .github/workflows.

…TTS to pronounce numbers of all kinds

notkaramel · 2023-07-24T18:43:29Z

services/espnet-tts-fr/src/app.py

+def isfloat(num):
+    try:
+        float(num)
+        return True
+    except ValueError:
+        return False


I added this because there's (supposedly) no built-in function in Python to check if a number is a float.

JRegimbal

Looking great!

JRegimbal · 2023-07-25T14:37:10Z

services/espnet-tts-fr/src/app.py

+            logger.debug(segment.split())
+            for s in segment.split():
+                try:
+                    logger.debug(f'Performing on: "{s}"')


This is going to get very chatty very quickly...

Not merging right now, but do you anticipate this being useful going forward? I'm a little worried about the slim possibility of this leaking some personal info.

I find this helpful to debug what component the TTS couldn't convert to numbers sometimes, but it can definitely be case specific (log in the except statement) rather than logging out everything.

You might be right... In that case, I would have to change the logs of multilang-support as well because it has been logging the whole segment out everytime, both in English and French 🫤

Ah, yeah please do that too then!

notkaramel · 2023-07-26T17:39:55Z

@JRegimbal I just reformat the processSegment so that it would let normal text through + added edge case handling for hyphen cases: it can now handle number range and also words/names like "gratte-ciel" (skyscrapper), "Lionel-Groulx", etc., without trying to convert them to number :)

JRegimbal · 2023-07-26T21:26:40Z

services/espnet-tts-fr/src/app.py

+            s = num2words(float(tempns), lang='fr')
    elif "-" in s:
-        logger.debug("case 3: -")
+        logger.debug("Case: has a - as separator")


Hmm, this would likely fail for cases where two separators are used. E.g., "39,5-40,0". We could probably do something with regexps with the - separator as well. Get the fields we need excluding text with ([\d,.]+)-([\d,.]+), replace with de $1 à $2 and then apply another find and replace on $1 and $2 to replace occurrences of , with .. Does this make sense?

Yes, it does. I will try them out.

JRegimbal · 2023-07-26T21:28:07Z

services/espnet-tts-fr/src/app.py

+        s = num2words(s, lang='fr')
    elif "," in s:
-        logger.debug(f"Case: '{s}' has , as separator")
+        logger.debug("Case: has a , as separator")


Hmm, does this still get run for text uses of ,?

Could also catch with a regexp find replace ((\d+),(\d+) to $1.$2).

It catches text with ,, but it won't perform any changes since on the next line, it couldn't cast the text to float.

I will look into regex and string processing more, thanks for the suggestion! I genuinely forgot about regex and went straight for the simpliest implementation.

notkaramel · 2023-08-04T18:52:52Z

@JRegimbal I added regex detection & processing and it's now working great on the examples from French TTS (espnet-tts-fr) struggles with numerical numbers #723. Please take a look at the changes!
One thing I'm concerning is that the TTS doesn't have pauses, user might feel like it's too fast. Should I create a new work item for that?

JRegimbal

Looks good!

JRegimbal · 2023-08-07T14:59:05Z

@notkaramel you can make a new issue, but I'd put it in backlog for right now and bring it up tomorrow if you're still looking for something to do. There may be a higher priority issue.

Adding num2words to requirements and first attempt on helping French …

f5776f2

…TTS to pronounce numbers of all kinds

notkaramel added the enhancement New feature or request label Jul 19, 2023

notkaramel self-assigned this Jul 19, 2023

notkaramel linked an issue Jul 19, 2023 that may be closed by this pull request

French TTS (espnet-tts-fr) struggles with numerical numbers #723

Closed

notkaramel mentioned this pull request Jul 20, 2023

EPIC: As a blind or low vision Francophone, I want to experience IMAGE in French, both for the UI and the generated interpretations, so that I can use IMAGE in my preferred language. #663

Closed

notkaramel marked this pull request as ready for review July 24, 2023 18:32

notkaramel requested a review from JRegimbal July 24, 2023 18:32

Finalize changes and removed unnecessary cases

18cd12c

notkaramel assigned JRegimbal and unassigned notkaramel Jul 24, 2023

notkaramel commented Jul 24, 2023

View reviewed changes

JRegimbal approved these changes Jul 25, 2023

View reviewed changes

JRegimbal assigned notkaramel and unassigned JRegimbal Jul 25, 2023

notkaramel added 3 commits July 26, 2023 12:55

Commented out translation logging

9704030

Commented out TTS logging + added edge case handling for case hyphen

06aeb96

Deleted new segment logging

4799560

notkaramel assigned JRegimbal and unassigned notkaramel Jul 26, 2023

JRegimbal requested changes Jul 26, 2023

View reviewed changes

JRegimbal assigned notkaramel and unassigned JRegimbal Jul 26, 2023

notkaramel mentioned this pull request Jul 31, 2023

STORY: As a Francophone IMAGE user, I want the extension to be thoroughly tested and validated, so that I can have a seamless experience in my preferred language. #736

Closed

17 tasks

notkaramel added 3 commits August 4, 2023 14:37

Using regex to process numbers and ranges

4093b98

Commented out error logging

33331cb

Removed tts logging the new segment for privacy matters

8154967

notkaramel assigned JRegimbal and unassigned notkaramel Aug 4, 2023

JRegimbal approved these changes Aug 7, 2023

View reviewed changes

JRegimbal merged commit a7c79ad into main Aug 7, 2023

JRegimbal deleted the french-tts-vs-numbers branch August 7, 2023 14:59

Conversation

notkaramel commented Jul 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made

Testings

Required Information

Coding/Commit Requirements

New Component Checklist (mandatory for new microservices)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JRegimbal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notkaramel commented Jul 26, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notkaramel commented Aug 4, 2023

Uh oh!

JRegimbal left a comment

Choose a reason for hiding this comment

Uh oh!

JRegimbal commented Aug 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

notkaramel commented Jul 19, 2023 •

edited

Loading