Reproducibility questions

[LinZichuan]

Hi @Yangsenqiao

I have some questions about the code.

1. How many hours are required to train models using bash scripts/run_efficient_gpt4o_judge.sh？Are you using 32 A100 GPUs with 80GB memory each for this training?

2. The code saves checkpoints every 5 steps. How should we select the checkpoint for final evaluation to reproduce the results in Table 2 of the paper? Do we choose the checkpoints with the highest validation accuracy reward?

3. For the visionthink results presented in the paper, were they obtained using gpt4o-as-judge or qwen-as-judge?

Looking forward to your reply. Thank you!

[Yangsenqiao]

Hi @LinZichuan,

Thanks for your interest in VisionThink.

A1: We train it on 32 GPUs, but it can also be trained on 8 GPUs.

A2: We use the checkpoint where the validation set accuracy no longer shows significant improvement.

A3: We use GPT-4o as the judge.

In addition, we are currently organizing the rule-based code to make it more compute- and API-friendly.

[Solus-sano]

Hi @LinZichuan,

Thanks for your interest in VisionThink.

A1: We train it on 32 GPUs, but it can also be trained on 8 GPUs.

A2: We use the checkpoint where the validation set accuracy no longer shows significant improvement.

A3: We use GPT-4o as the judge.

In addition, we are currently organizing the rule-based code to make it more compute- and API-friendly.

How many steps will it take to see val acc no longer show improvement on your 32 GPUs experiment?

I'm trying to reproduce the efficient_gpt4o_judge on 8*A800 80GB, any suggestions for changing the configuration in .sh file?