Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

38 Open 176 Closed
38 Open 176 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

MMBench-Video Dataset Download Bug
#589 opened Nov 9, 2024 by Noctis-SC
Evaluations get tuck on the last 4 questions
#588 opened Nov 9, 2024 by XuGW-Kevin
ChartQA eval bug
#584 opened Nov 7, 2024 by lemonliu1992
Reproducing Qwen2-VL-72B-Instruct Evaluation Results Fails
#580 opened Nov 6, 2024 by ChuanyangZheng
4
Corrupted MathVista testmini data WIP
#578 opened Nov 5, 2024 by LeoDu0314
1
AI2d gpt和claude3.5官方分数非常高 awaiting confirm
#577 opened Nov 5, 2024 by Violettttee
1
Qwen-VL-Max-0809 MME中celebrity测出来和榜单结果差距有10分左右 awaiting confirm
#571 opened Nov 2, 2024 by lihua8848
7
[Bug] RuntimeError: world_size (8) is not equal to tensor_model_parallel_size (1) x pipeline_model_parallel_size (1) awaiting confirm
#561 opened Oct 31, 2024 by Wiselnn570
2
LLaVA v1.5 7b very low performance on TextVQA
#554 opened Oct 28, 2024 by wufeim
The result of gpt4o on MathVista is 61.6, which is lower than the 62.7 on the list. awaiting confirm
#553 opened Oct 28, 2024 by kkk123698745
3
评测MVBench时报错
#551 opened Oct 25, 2024 by helloworld01001
2
评测InternVL2-8B出现OOM awaiting confirm
#533 opened Oct 21, 2024 by AACengineer
2
评测mmbench_video会报错 no response
#530 opened Oct 19, 2024 by helloworld01001
14
How to eval with multigpu when using device_map='auto'
#528 opened Oct 18, 2024 by qianwangn
@FangXinyu-0913
1
Different evaluate result between different VLMEvalKit version
#523 opened Oct 16, 2024 by terry-for-github
@kennymckormick
1
Are images compressed in tsv files?
#517 opened Oct 12, 2024 by ywwynm
3
textonly benchmarks
#497 opened Sep 30, 2024 by JcWang20
@PhoenixZ810
1
[Help Wanted] the alignment with official accuracy in llama3.2-vision help wanted Extra attention is needed
#493 opened Sep 29, 2024 by droidXrobot
8
Reproducing QWen2VL Results on Video Benchmarks with VLMEvalKit
#484 opened Sep 23, 2024 by aniki-ly
@FangXinyu-0913
4
【提问】关于语音模型的评测 help wanted Extra attention is needed
#473 opened Sep 19, 2024 by xx0412
1
Input length of input_ids is 0, but max_length is set to -2009. awaiting confirm
#455 opened Sep 10, 2024 by wangli68
1
ChatAPI - ERROR - HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url: /v1/chat/completions (Caused by ProxyError('Unable to connect to proxy', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f481233f640>: Failed to establish a new connection: [Errno 111] Connection refused'))) network issue
#406 opened Aug 25, 2024 by yansuoyuli
3
CMMMU数据集支持 help wanted Extra attention is needed
#398 opened Aug 21, 2024 by DietDietDiet
5
关于llava-v1.5-7b复现
#391 opened Aug 19, 2024 by helloworld01001
13
关于幻觉benchmark的支持 help wanted Extra attention is needed
#384 opened Aug 15, 2024 by shufangxun
1
ProTip! Mix and match filters to narrow down what you’re looking for.