issue resuming unfinished model training cont'd

[helizabeth1103]

Hello,

I am following up on issue 1042. I apologize for the very delayed reply, I have been waiting for server time and did not get to test the suggested action until now.

As suggested, I changed my script to direct to what I thought was the latest checkpoint, and in the output file of the resubmission, it does say "restarted from checkpoint." However, before the first run timed out, it was about 90% completed, and one the resubmission began, it appeared to start again from zero and has an ETA comparable to the first submission. Am I directing to the wrong checkpoint? Is there additional information I need to include?

Thank you!

Here is the resubmission script:

apptainer exec --bind /90daydata,$TMPDIR --nv /90daydata/pbarc/haley.arnold/AI_ModelTraining/Samples/deepvariant_1.6.0.sif train
--config="/90daydata/pbarc/haley.arnold/AI_ModelTraining/Samples/dv_config.py:base"
--config.train_dataset_pbtxt="shuffledFM2s_chr5.with_label_channelsize.shuffled.pbtxt"
--config.tune_dataset_pbtxt="shuffledFM1s_chr5.with_label_channelsize.shuffled.pbtxt"
--config.init_checkpoint=/90daydata/pbarc/haley.arnold/AI_ModelTraining/Illumina_Tephritids/zcuc10x_modeltraining/model_chr5trainandval_newertruth_out/checkpoints
/ckpt-200000
--config.num_epochs=10
--config.learning_rate=0.02
--config.num_validation_examples=0
--experiment_dir="model_chr5trainandval_newertruth_out/"
--strategy=mirrored
--config.batch_size=32

Here is the "checkpoint" file in its current state, with the resubmission still running. At the time of resubmission, the model_checkpoint_path was "ckpt-200000" which I understood as being the best/most recent.

model_checkpoint_path: "ckpt-110032"
all_model_checkpoint_paths: "ckpt-27508"
all_model_checkpoint_paths: "ckpt-55016"
all_model_checkpoint_paths: "ckpt-200000"
all_model_checkpoint_paths: "ckpt-100000"
all_model_checkpoint_paths: "ckpt-110032"
all_model_checkpoint_timestamps: 1768907198.731081
all_model_checkpoint_timestamps: 1768963401.6893659
all_model_checkpoint_timestamps: 1769269562.6065133
all_model_checkpoint_timestamps: 1770005859.406302
all_model_checkpoint_timestamps: 1770038952.1190722
last_preserved_timestamp: 1768836072.2734745

Here is the log file from the first submission:
deepvariant_modeltrain-19001099-atlas-0245.err.txt

And here is the current log file from the ongoing resubmission:
deepvariant_modeltrain-19062375-atlas-0244.err.txt

[pichuan]

OK, what about this as an answer:

From "deepvariant_modeltrain-19062375-atlas-0244.err.txt", you have:

I0129 20:23:33.609190 139658065700672 keras_modeling.py:362] inceptionv3: load_weights from checkpoint: /90daydata/pbarc/haley.arnold/AI_ModelTraining/Illumina_Tephritids/zcuc10x_modeltraining/model_chr5trainandval_newertruth_out/checkpoints/ckpt-200000

This means that the training will initialize the weights from this file.

Looking at your checkpoint file, it's a bit unexpected to me that the top line is:

model_checkpoint_path: "ckpt-110032"

while one of the lines below has a larger ckpt:

all_model_checkpoint_paths: "ckpt-200000"

This could be your current resubmission started from step 0 and has now reached step 110,032, overwriting the "latest" pointer in that file.

If you want to resume by initializing the weight of a previously trained model, but don't want to mess up previous experiment directories, you can start with a brand new --experiment_dir. That way, the training will be initialized by the previous weight, but the checkpoint counting will start from 0 (and won't override previous ckpts). And you can reduce num_epochs or simply stop the experiment when the learning curve looks like it's converged.

If you want to resume from a previous, existing --experiment_dir, in theory if you specify the --experiment_dir and leave out --init_checkpoint, you can see if TF can pick from from where it left off. However, like I said, it's a bit unexpected to me that your model_checkpoint_path has a lower number than 200000. So, if you know ckpt-200000 is a good place to start, the approach of warm-starting + a brand-new --experiment_dir + running for fewer epochs might be a better choice.

[helizabeth1103]

Right now I am concerned with getting the training to resume from where it left off after exceeding the time limit on my GPU server, meaning getting it to a point where it has a finished and testable model.

Here are the contents of my checkpoint folder when model_train reaches the time limit before the job finishes:

As compared to my checkpoints folder for a different model-train run with unrelated data, that did finish within the time limit, yielding a checkpoint folder I can direct to in run_deepvariant to test it (script example below):

apptainer exec --bind /90daydata,$TMPDIR /90daydata/pbarc/haley.arnold/AI_ModelTraining/Samples/deepvariant_1.6.0.sif run_deepvariant
--model_type WGS
--customized_model "/90daydata/pbarc/haley.arnold/AI_ModelTraining/Illumina_Tephritids/zcuc5x_modeltraining/model_chr5trainandval_newertruth_out/checkpoints/ckpt-247572"
--ref "${reference}"
--reads "${inputdir}/${sample}.bam"
--sample_name "${sample}"
--output_vcf "${outputdir}/${sample}.vcf.gz"
--output_gvcf "${outputdir}/${sample}.g.vcf.gz"
--intermediate_results_dir "${outputdir}/${sample}"

It was my understanding that "warmstarting" refers to further training a model on an already-completed customized training model (so for in my second image above, running train_model while directing to the starting point of "directory/ckpt-247572." Or as you said in your reply:

If you want to resume by initializing the weight of a previously trained model, but don't want to mess up previous experiment directories, you can start with a brand new --experiment_dir. That way, the training will be initialized by the previous weight, but the checkpoint counting will start from 0 (and won't override previous ckpts). And you can reduce num_epochs or simply stop the experiment when the learning curve looks like it's converged

Is that the case, or does warmstarting refer to what I am trying to do, i.e., resume an unfinished process?

Since I do not have a model folder output usable in testing in the case of the first image, I am more asking if it is possible to resume from one of the checkpoints cataloged by only .index and .data-00000-of-00001 files. Many of my models hit the server time limit when the ETA shows they only needed a short time more to finish the run and produce a "finished" checkpoint folder as in the second image shown above. I attempted one of your suggestions, by altering the script only by removing the --init_checkpoint line and attempting to allow TF to pick up where it left off, but it did not resume from any of the checkpoints and began training from 0 again.

The first image I show above is the result of the run in progress at the time of my last post, which since again timed out. The end of the activity log file looks like this:

The contents of its checkpoint file are now:

model_checkpoint_path: "ckpt-110032"
all_model_checkpoint_paths: "ckpt-27508"
all_model_checkpoint_paths: "ckpt-55016"
all_model_checkpoint_paths: "ckpt-200000"
all_model_checkpoint_paths: "ckpt-100000"
all_model_checkpoint_paths: "ckpt-110032"
all_model_checkpoint_timestamps: 1768907198.731081
all_model_checkpoint_timestamps: 1768963401.6893659
all_model_checkpoint_timestamps: 1769269562.6065133
all_model_checkpoint_timestamps: 1770005859.406302
all_model_checkpoint_timestamps: 1770038952.1190722
last_preserved_timestamp: 1768836072.2734745

Is it possible to resume this run at the point it left off, 71.3%, and if so, how do I direct the script to the appropriate checkpoint so it can finish producing the new model for testing? If this question reveals that I have a fundamental knowledge gap in how this works, would you please kindly direct me to where I can correct that? :)