The russian lyrics don't fit the music

[tankist-git-2]

When I create music with Russian language lyrics I hear that lyrics don't fit the music. One word can be stretched to several musical beat.

How to reproduce:

1. Write lyrics in russian
2. Select "ru" in drop-down selector under lyrics field.
3. Listen to the result

I used Windows bundle to install Ace 1.5 on my PC. Windows 10, 32 Gb RAM, RTX5080.

There are two examples. One of this was generated by ACE 1.5 and rated by my as absolutely bad vocal to music fit. The next one was generated in Udio and is used as reference of good vocal fit.

ace_15_bad.mp3

udio_good.mp3

[Kwisss]

I trained a lora on rank 64 and rank 256 for 50 epochs on Dutch music, 100 songs and the absolute first thing it seems to pick up is better arrangements. I recon this is a perfect use case for training a lora.

[koda-dernet]

I trained a lora on rank 64 and rank 256 for 50 epochs on Dutch music, 100 songs and the absolute first thing it seems to pick up is better arrangements. I recon this is a perfect use case for training a lora.

Hello, i wanted to ask how your training pipeline goes, did you train on the Gradio GUI and on base? May i ask your settings?

EDIT: Better questions, on your dataset preparation, do you use 5Hz or do you caption manually? Do you train on full songs?

[ChuxiJ]

Russian should be well-supported. If you encounter issues with generation, check if the duration matches the lyrics, and if you've reset the metadata (BPM, key, time signature, duration, etc.) to let LM generate it automatically. You can click the dice icon to find examples in Russian and see how to generate correctly.

[Kwisss]

I trained a lora on rank 64 and rank 256 for 50 epochs on Dutch music, 100 songs and the absolute first thing it seems to pick up is better arrangements. I recon this is a perfect use case for training a lora.

Hello, i wanted to ask how your training pipeline goes, did you train on the Gradio GUI and on base? May i ask your settings?

EDIT: Better questions, on your dataset preparation, do you use 5Hz or do you caption manually? Do you train on full songs?

My tests so far has been unsatisfactory, I used 5Hz to caption the music (full songs) but edited the song description of every song one by one.
had to replace the occasional guitar to piano, refine the genres it mentioned and the wrong guesses about the language.

Right now I am in the process of curating the database I build even further by double checking the bpm and key of every song.
And reversing stupid mistake I made while merging the database (wordpad text encoding broke the flat sign and accented e, it warned me during saving but I'm naive haha)

I tried training rank 64 and 256, both with batch 1 and 8 and once with 1/10th the learning rate. loss did not seem to drop much after 50 epochs, hoovering around 65 loss. the resulting lora when tested does not seem to show the result I would've liked yet.

[ChuxiJ]

There are a lot of training results: https://discord.com/channels/1369256267645849741/1372633614172553276