Skip to content

Latest commit

 

History

History
85 lines (59 loc) · 2.82 KB

README.md

File metadata and controls

85 lines (59 loc) · 2.82 KB

Learning to Correction: Explainable Feedback Generation for Visual Commonsense Reasoning Distractor

Jiali Chen1, Xusen Hei1, Yuqi Xue1, Yuancheng Wei1, Jiayuan Xie2, Yi Cai*,1 Qing Li2

1South China University of Technology   2The Hong Kong Polytechnic University   
*Corresponding author   

arXiv

figure1

Overview of Pedagogical Expert Instructed Feedback Generation (PEIFG) model

🏗️ Run PEIFG

Installation

  1. Clone this repository and navigate to the PEIFG folder
gt clone https://github.com/Gary-code/PEIFG.git
cd PEIFG
  1. Install Package
  • Install Anaconda or Miniconda distribution based on Python3.10 from their downloads' site.
  • Main packages: PyTorch = 1.13, transformers = 4.43
  1. Download model weights:

🚀 Instructblip (Q-Former), Qwen1.8B, OPT-350M

Data Preparation

Our VCR-DF dataset reuse the images from the official VCR dataset. You can download in VCR.

The annotation of our VCR-DF dataset is in ./vcr_df_dataset folder.

Train

cd train
bash lora_train_feedback.sh

📑 Citation

@inproceedings{peifg,
  author       = {Jiali Chen and
                  Xusen Hei and
                  Yuqi Xue and
                  Yuancheng Wei and
                  Jiayuan Xie and
                  Yi Cai and
                  Qing Li},
  editor       = {Jianfei Cai and
                  Mohan S. Kankanhalli and
                  Balakrishnan Prabhakaran and
                  Susanne Boll and
                  Ramanathan Subramanian and
                  Liang Zheng and
                  Vivek K. Singh and
                  Pablo C{\'{e}}sar and
                  Lexing Xie and
                  Dong Xu},
  title        = {Learning to Correction: Explainable Feedback Generation for Visual
                  Commonsense Reasoning Distractor},
  booktitle    = {Proceedings of the 32nd {ACM} International Conference on Multimedia,
                  {MM} 2024, Melbourne, VIC, Australia, 28 October 2024 - 1 November
                  2024},
  pages        = {8209--8218},
  publisher    = {{ACM}},
  year         = {2024},
}