Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the bug that MAISI ckpt cannot be loaded after finetune. #654

Merged
merged 2 commits into from
Sep 18, 2024

Conversation

guopengf
Copy link
Contributor

Fixes # .

Description

MAISI output checkpoint after finetuning cannot be used as trained_controlnet_path.
This problem came from the CheckpointSaver. When a single key is provided for save_dict such as:
"save_dict": {
"controlnet_state_dict": "@ControlNet"
},
The saved dict does contain the key " "controlnet_state_dict". However, it directly saves the state_dict of controlnet as the checkpoint.

The workaround is that we also save the optimizer state. For example,
"save_dict": {
"controlnet_state_dict": "@ControlNet",
"optimizer": "@optimizer"
}. Then, the MAISI output checkpoint after fine-tuning can be properly loaded.

Status

Ready/Work in progress/Hold

Please ensure all the checkboxes:

  • Codeformat tests passed locally by running ./runtests.sh --codeformat.
  • In-line docstrings updated.
  • Update version and changelog in metadata.json if changing an existing bundle.
  • Please ensure the naming rules in config files meet our requirements (please refer to: CONTRIBUTING.md).
  • Ensure versions of packages such as monai, pytorch and numpy are correct in metadata.json.
  • Descriptions should be consistent with the content, such as eval_metrics of the provided weights and TorchScript modules.
  • Files larger than 25MB are excluded and replaced by providing download links in large_file.yml.
  • Avoid using path that contains personal information within config files (such as use /home/your_name/ for "bundle_root").

Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>
@guopengf guopengf self-assigned this Sep 16, 2024
@guopengf guopengf requested a review from KumoLiu September 16, 2024 22:31
@guopengf guopengf changed the title Fix the bug ckpt cannot be loaded after finetune. Fix the bug that ckpt cannot be loaded after finetune. Sep 16, 2024
@guopengf guopengf changed the title Fix the bug that ckpt cannot be loaded after finetune. Fix the bug that MAISI ckpt cannot be loaded after finetune. Sep 16, 2024
Signed-off-by: Pengfei Guo <pengfeig@nvidia.com>
Copy link
Collaborator

@KumoLiu KumoLiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix, looks good to me.

@KumoLiu
Copy link
Collaborator

KumoLiu commented Sep 18, 2024

/build

@KumoLiu KumoLiu merged commit 6db359d into Project-MONAI:dev Sep 18, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants