Save a checkpoint when we press on Stop Training despite the Save Every N Epochs setting & Training optimization

[MrGTAmodsgerman]

Is your feature request related to a problem? Please describe.
I have an old GPU (GTX 1080Ti 11GB) + 32GB of RAM and i tested how it performs on training a LoRa. It estimated 167h and 50 min for the training get finished. I didn't expected this this slow, so at some point i stopped training and it haven't reached the first checkpoint save setting yet. So i can't resume on what i already trained. The 1 hour is lost.

Describe the solution you'd like
When i stop training it should safe a checkpoint. You never know when or why you wanna stop a training. So it would be very helpful to not having to rely on a predicted value setting for Save Every N Epochs.

Describe alternatives you've considered
Buying a new GPU

Additional context
I set 600 Epochs for 70 instrumental song files

I trained a Flux LoRA before which took a night. So there is room for optimization i think.

[ChuxiJ]

You can stop the training by simply closing the program in the backend. You’ll have to restart it anyway once training is finished. The stop button works intermittently, but this isn’t that critical.

[MrGTAmodsgerman]

You can stop the training by simply closing the program in the backend. You’ll have to restart it anyway once training is finished. The stop button works intermittently, but this isn’t that critical.

But did you understood that my post is not about stopping the training, rather to create a checkpoint on stopping?

[ChuxiJ]

yes, I see

we recommend to use side step training. Please see the tutorial

[MrGTAmodsgerman]

yes, I see

we recommend to use side step training. Please see the tutorial

You mean this?
https://github.com/ace-step/ACE-Step-1.5/blob/main/docs/sidestep/VRAM%20Optimization%20Guide.md

[ChuxiJ]

Yes, have you tried?

[MrGTAmodsgerman]

Yes, have you tried?

No, not yet. I have to figure out how to run it trough Pinokio

[MrGTAmodsgerman]

Yes, have you tried?

I managed to run it now with Pinokio after some troubleshooting. Now i am able to train. But why not merge it into the original Ace Step version? 😊

[ChuxiJ]

I'm not sure which version of folk Pinokio is. What needs to be done if we want to merge it?