[Auto3DSeg] Continuing Training for Folds

[imruljubair]

Hi,

I am using Auto3DSeg for my dataset. I used the following code -

from monai.apps.auto3dseg import AutoRunner
import os
runner = AutoRunner(input="/mnt/datastore/jubair/datasets/For_MONAI/tasks/HVSMR48_raw/input.yaml",  work_dir="/mnt/datastore/jubair/datasets/For_MONAI/tasks/HVSMR48_raw/work_dir")
runner.run()

While the training was working for dints_1, I had to stop it before completing it. Now if I run the above code, it does not continue from the previously unfinished dints_1, rather it starts training for dints_2. Is there anyway I can continue from dints_1?

[KumoLiu]

Hi @imruljubair, thanks for your interest here.
You can remove the cache dir under work dir.

MONAI/monai/apps/auto3dseg/auto_runner.py

Line 186 in ff43028

work_dir/

[imruljubair]

Thank you so much @KumoLiu

[pwrightkcl]

@imruljubair note that the current options for resuming are at the level of folds. That is, you can either skip the partially-trained fold or restart it from scratch. I do not believe it is possible to resume training a partially-trained fold from a checkpoint, at least at the level of the AutoRunner or BundleGen objects, but I am looking into how to implement that locally. For more info, please review discussion #5754, issue #5756, and PR #6310. The latter explains where the trained / not trained flag is.

[imruljubair]

@pwrightkcl Thanks for your response. Auto3dSeg is great. It'd nicer if you could introduce checkpoint-based resuming.

[pwrightkcl]

@imruljubair I'm not a dev, just a user like you. I'm trying to figure out how to add the feature locally by modifying the generated code, but I'm pretty new at this.

[YazdanSalimi]

I wish we had the option to load partially trained models at each load, it saves a lot of time by not training the partially trained fold from scracth.
Thank you

[pwrightkcl]

I shared my personal code to resume partially-trained folds in segresnet here: #7506. The ideas came from the earlier discussions I linked in my previous reply. You would have to implement your own solution for dints and swin unetr, but I would imagine they save checkpoints and track current epoch number in a similar way.

[YazdanSalimi]

I shared my personal code to resume partially-trained folds in segresnet here: #7506. The ideas came from the earlier discussions I linked in my previous reply. You would have to implement your own solution for dints and swin unetr, but I would imagine they save checkpoints and track current epoch number in a similar way.

Thank you!