Question about Reproducibility on LLM-Judged Benchmarks

[yic20]

Hi @Yangsenqiao ,
Thank you for your excellent work!

I am trying to reproduce the results reported in your paper using the released model weights VisionThink-Efficient. Some of my results are very close to yours, but for a few benchmarks, my scores are lower and seem a bit random.

My Results

Benchmark	Paper's Score	My Score
MMBench	80.0	79.59
RealWorldQA	68.5	68.5
POPE	86.0	86.69
MME	2400	2403.3
MathVista	67.5	65.7
MathVerse	48.0	45.9
MMVet	67.1	61.8

The scores for MMBench, RealWorldQA, POPE, and MME look great and match your paper.
However, my scores for MathVista, MathVerse, and MMVet are lower and I also noticed the scores for these three change each time I run the test.

Questions

1. I think the difference might be because these benchmarks use LLM-as-judge. Could you please share some details about your evaluation setup?

2. I noticed that samples in MMMU can contain multiple images. So how did you handle multiple images input for evaluation?

Thanks!

[pspdada]

As far as I know, MM-Vet is a benchmark with a relatively small number of test samples and uses an LLM judge, so the results might fluctuate. I hope this finding is helpful to you.

[Yangsenqiao]

Thanks for your interest in VisionThink. We have updated the evaluation scripts last week. Please feel free to take a look.

Btw, if I miss the issue or reply late, feel free to email me directly. In most cases, I will respond within 48 hours. Thanks!

[yic20]

Thank you for your reply. I was wondering if all the benchmarks were evaluated using LLM-as-judge? @Yangsenqiao