Training Error: [WinError 32] The process cannot access the file because it is being used by another process

[MarianaMendanha]

Hi everyone,

I'm encountering an issue while trying to fine-tune a voice model using XTTS. I previously fine-tuned a voice with the XTTS model and wanted to create new fine-tunes. I moved the old model to the models directory as suggested in the "🔜 What to do next" tab, deleted the old model and dataset, and generated a new dataset. However, when I attempt to train either with XTTS or the first version of my already trained model using the following parameters:

• Select the Model to train: xttsv2_2.0.3
• Learning Rate: 1e-5
• Learning Rate Scheduler(s): Cosine Annealing Warm Restarts
• Optimizer: AdamW
• Workers/Threads: 4
• Number of epochs: 50
• Batch size: 128
• Grad accumulation steps: 4
• Max permitted audio size in seconds: 15

I receive the following error:

 > EPOCH: 0/50
 --> C:\Users\mariana.cruz\Desktop\XTTS-WebUI\alltalk_tts\finetune\Jorge-v2\training\XTTS_FT-September-11-2024_11+51AM-3419843
[FINETUNE] Sampling by language: dict_keys(['en'])

 > TRAINING (2024-09-11 11:51:32)
C:\Users\mariana.cruz\Desktop\XTTS-WebUI\alltalk_tts\alltalk_environment\env\Lib\site-packages\transformers\models\gpt2\modeling_gpt2.py:544: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
[FINETUNE] Error: An unexpected error occurred: [WinError 32] The process cannot access the file because it is being used by another process: 'C:/Users/mariana.cruz/Desktop/XTTS-WebUI/alltalk_tts/finetune/Jorge-v2/training/XTTS_FT-September-11-2024_11+51AM-3419843\\trainer_0_log.txt'

I have tried restarting my PC and using the handle tool to identify and kill the process using the file, as shown below, but it didn't work:

handle:
python.exe         pid: 13452  type: File          17DC: C:\Users\mariana.cruz\Desktop\XTTS-WebUI\alltalk_tts\finetune\Jorge-v2\training\XTTS_FT-September-11-2024_11+51AM-3419843\trainer_0_log.txt

What should I do to resolve this issue? Any help would be greatly appreciated!

[MarianaMendanha]

I resolved my problem reducing the batch size and grad accumulation. It works with:

• Workers/Threads: 10
• Batch size: 20
• Grad accumulation steps: 2
  It does not work with below setup or larger batch sizes (I receive the same Error):
• Workers/Threads: 10
• Batch size: 24
• Grad accumulation steps: 4

** I think the problem is my GPU memory.

[erew123]

Sorry I didnt get back to you in a timely way. I believe your conclusion is correct! Thanks

[Eleksar387]

I also get this error. No matter how low the settings. I have a 3060 with 12 GB RAM. Can't get finetuning to work. It uses around 3GB when it stops with the error message.

[FINETUNE] The text length exceeds the character limit of 250 for language 'en', this might cause truncated audio.
[FINETUNE] Error: An unexpected error occurred: [WinError 32] The process cannot access the file because it is being used by another process: 'F:/AI/alltalk_tts/finetune/character/training/XTTS_FT-September-11-2024_11+51AM-3419843\trainer_0_log.txt'

[erew123]

This has been reported by one other person. Here is what they said:

And here are the instructions I gave them

1. Download the Windows Process Explorer (ProcessXP) utility and extract it https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
2. After restarting your computer to ensure any file locks are closed, clear out the finetuning folder, bar the put-voice-samples-in-here folder (along with your original samples). Feel free to move your current files elsewhere for if you want to restore them later. It should look like the following with no other folders:
   
3. Start the finetuning and run Step 1 to create your dataset (no need to validate or anything, we are just checking why/how/if the file gets locked). I would keep all the base default settings for everything.
4. Once you have run Step 1, move to step 2 and run it, again keeping the base default settings. Assumedly it wont reach the Evaluation process and you will get the error about the file being in use.
5. If you do get that error message, leave Finetuning running and open ProcessXP and select Find > Find Handle or DLL and then search for trainer_0_log and click the search button. In theory, the file should only have 1x lock, that is by the Python process. Nothing else should be accessing or locking that file. It should look like this:
   

If something else has a lock on the file OR you python process has crashed in the past, that is your issue. I would always suggest restarting the machine as a first course of action to ensure that all process locks are released.

[Eleksar387]

@erew123 Thank you for your reply. I tried it with no success. Restarted my computer and also checked it with process explorer. There is just this single pocess.
However what I have found, it depends on the file. I tried a different file and then it worked. I just can't exactly say what the issue is. Maybe file size. The file that worked has about 7MB (mp3). Two other files with 10 and 15 MB didnt' work (mp3 and wav). All tested with the standard settings.
It's still unclear to me what causes this problem.