Learning to Correction: Explainable Feedback Generation for Visual Commonsense Reasoning Distractor

Jiali Chen¹, Xusen Hei¹, Yuqi Xue¹, Yuancheng Wei¹, Jiayuan Xie², Yi Cai^*,1 Qing Li²

¹South China University of Technology   ²The Hong Kong Polytechnic University   
^*Corresponding author   

Overview of Pedagogical Expert Instructed Feedback Generation (PEIFG) model

🏗️ Run PEIFG

Installation

1. Clone this repository and navigate to the PEIFG folder

gt clone https://github.com/Gary-code/PEIFG.git
cd PEIFG

2. Install Package

• Install Anaconda or Miniconda distribution based on Python3.10 from their downloads' site.
• Main packages: PyTorch = 1.13, transformers = 4.43

3. Download model weights:

🚀 Instructblip (Q-Former), Qwen1.8B, OPT-350M

Data Preparation

Our VCR-DF dataset reuse the images from the official VCR dataset. You can download in VCR.

The annotation of our VCR-DF dataset is in ./vcr_df_dataset folder.

Train

cd train
bash lora_train_feedback.sh

📑 Citation

@inproceedings{peifg,
  author       = {Jiali Chen and
                  Xusen Hei and
                  Yuqi Xue and
                  Yuancheng Wei and
                  Jiayuan Xie and
                  Yi Cai and
                  Qing Li},
  editor       = {Jianfei Cai and
                  Mohan S. Kankanhalli and
                  Balakrishnan Prabhakaran and
                  Susanne Boll and
                  Ramanathan Subramanian and
                  Liang Zheng and
                  Vivek K. Singh and
                  Pablo C{\'{e}}sar and
                  Lexing Xie and
                  Dong Xu},
  title        = {Learning to Correction: Explainable Feedback Generation for Visual
                  Commonsense Reasoning Distractor},
  booktitle    = {Proceedings of the 32nd {ACM} International Conference on Multimedia,
                  {MM} 2024, Melbourne, VIC, Australia, 28 October 2024 - 1 November
                  2024},
  pages        = {8209--8218},
  publisher    = {{ACM}},
  year         = {2024},
}