Evaluation code

[o0t1ng0o]

Hi @jwmao1 ,

Thank you for releasing your code.
Can you provide the evaluation code for your model?

I meet several errors when loading your model in MeDiM/evaluation/inference_t2i.py, such as missing config.json. Even i include the config file, it meets the weights mismatch problems as follows:

RuntimeError: Error(s) in loading state_dict for Embedding:
        size mismatch for weight: copying a param with shape torch.Size([40196, 3072]) from checkpoint, the shape in current model is torch.Size([264192, 3072]).

After i revising the config file to address this error, I still meet the missing parameters error like:

Some weights of the model checkpoint at /raid/home/CAMCA/hj880/wt/ckpts/huggingface/hub/models--JiaMao--medim_ckpts/snapshots/dcca20779d09e154a25d10b39eefb4f86d6f82e9 were not used when initializing GemmaForCausalLM: ['model.layers.0.adaLN_modulation.bias', 'model.layers.0.adaLN_modulation.weight', 'model.layers.1.adaLN_modulation.bias', 'model.layers.1.adaLN_modulation.weight', 'model.layers.2.adaLN_modulation.bias', 'model.layers.2.adaLN_modulation.weight', 'model.layers.3.adaLN_modulation.bias', 'model.layers.3.adaLN_modulation.weight', 'model.layers.4.adaLN_modulation.bias', 'model.layers.4.adaLN_modulation.weight', 'model.layers.5.adaLN_modulation.bias', 'model.layers.5.adaLN_modulation.weight', 'model.layers.6.adaLN_modulation.bias', 'model.layers.6.adaLN_modulation.weight', 'model.layers.7.adaLN_modulation.bias', 'model.layers.7.adaLN_modulation.weight', 'model.layers.8.adaLN_modulation.bias', 'model.layers.8.adaLN_modulation.weight', 'model.layers.9.adaLN_modulation.bias', 'model.layers.9.adaLN_modulation.weight', 'model.time_embedder.mlp.0.bias', 'model.time_embedder.mlp.0.weight', 'model.time_embedder.mlp.2.bias', 'model.time_embedder.mlp.2.weight']
- This IS expected if you are initializing GemmaForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing GemmaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Could you please provide the evaluation code?

Thanks.

[o0t1ng0o]

@nsjia @supexp

[jwmao1]

Hi @o0t1ng0o,

Thanks a lot for your interest in our work and for trying out the released code. Sorry for the confusion caused by the checkpoint/config loading behavior you ran into.

For evaluation, we have already open-sourced the inference and metric scripts:

• Inference entry: MeDiM/inference.py
• Metric scripts: fid_metric.sh, is_score.py, report_metric.py, pair_metric.py

In practice, you can run inference with a command like the following:

python inference.py \
  +experiments='[large_scale_train]' \
  debug=true \
  loader.batch_size=1 \
  data_path_dir_train=./dataset/pathgen/train \
  data_path_dir_val=./dataset/pathgen/test \
  data_mimic_dir_train=./dataset/mimic-cxr/data \
  data_mimic_dir_val=./dataset/mimic-cxr/test \
  model.vqgan_config=./models/chameleon/vqgan.yaml \
  model.vqgan_ckpt=./models/vqgan_ckpt \
  model.llama_ckpt=./models/Llama-2-7b-hf \
  model.liquid_ckpt=./models/Liquid_V1_7B \
  trainer.load_from_state_dict=./checkpoint_1000000/ \
  eval.task='conditional_mask'

For different evaluation settings, you can switch eval.task among:
mask_all
mask_text_only
mask_image_only
conditional_mask

Thanks again for your focus!