srt AI?

[xdax1]

Hello, would it be possible to add an option that creates a voiceover for .srt subtitles? So that it also takes the time from the subtitles so that the audio at the same time has synchronization.

[Sharrnah]

Hi. That is something i have thought about.

But it is some complex problem to handle if you want to do it well.

• You need to have exact timings (should be doable already)
• You need to detect speakers and generate a different voice for each of them so not everyone speaks in the same voice. (Speaker diarization model is already included, but i think needs some more work to work better).
• separate speech from the rest.
• You have to decide which TTS you want to use for this which sounds good enough.
• Generate the translated speech and make sure it fits in the timeframe. (difficult because different languages often have different lengths.) There might be multiple ways to solve this like:
  • Change the Audio speed while maintaining the pitch. (might work but could sound weird)
  • Change the Video speed and keep the audio as it is. (might work but could look weird. Also what with the background audio?)
  • just ignore the issue and hope for the best. (not really a good idea i think).
• Try to sync mouth movement with the speech. (There is AI for that and i even have an internal plugin for this, but it has many issues and i have stopped working on it. There might be newer solutions out there already, haven't looked for it again.)

So right now i think this is out of scope for this project.
If you have other ideas or want to help on such a Plugin please let me know.

[xdax1]

Hi,
You can create some simple first and then improve it. For now, I myself created a simple script in python that uses eleven labs to create a voiceover from subtitles, but it is unprofitable because they offer few credits :/.
How does it work?

• Each snippet from the .srt subtitle file are split and generated individually as separate audio files
• The script from the subtitles also takes the start time of the passage, thus determining when the voiceover should begin (synchronization)
• At the end it combines everything using ffmpeg, creates “silence” in the audio until the time of the voiceover from the subtitles

Maybe you can use any of the ideas. If I come up with anything else, I'll let you know.