Skip to content

Medical Visual Question Answering via Conditional Reasoning [ACM MM 2020]

License

Notifications You must be signed in to change notification settings

Awenbocc/med-vqa

Repository files navigation

Med-VQA

The implementation of Medical Visual Question Answering via Conditional Reasoning [ACM MM 2020]
We evaluate our proposal on VQA-RAD dataset.

Conditional Reasoning Framework

We propose QCR [Question-Conditioned Reasoning Module] and TCR [Type-Conditioned Reasoning] modules, which guide the importance selection over multimodal fusion features and learn different reasoning skills for different types of tasks separately. The detailed architecture is shown in the figure below.
Image text

Results

Overall Open-ended Closed-ended
Base 66.1 49.2 77.2
Our proposal 71.6 60.0 79.3

Our proposal achieves significantly increased accuracy in predicting answers to both closed-ended and open-ended questions, especially for open-ended questions.

Training

1. Dependence

pip install -r requirements.txt

2. Preparing data

All pre-prepared data can be found in data package

3. Running

git clone https://github.com/Awenbocc/Med-VQA.git
cd ./Med-VQA
python main.py --gpu 0 --seed 88

License

MIT License

Citation

Please cite following in your publications if they help you.

@inproceedings{zhan2020medical,
  title={Medical visual question answering via conditional reasoning},
  author={Zhan, Li-Ming and Liu, Bo and Fan, Lu and Chen, Jiaxin and Wu, Xiao-Ming},
  booktitle={Proceedings of the 28th ACM International Conference on Multimedia},
  pages={2345--2354},
  year={2020}
}

About

Medical Visual Question Answering via Conditional Reasoning [ACM MM 2020]

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published