AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

MMBENCH: Is Your Multi-Modal Model an All-around Player?

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin

ECCV 2024 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 1,494 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use CircularEval instead of single-pass VanillaEval (ablation)VLM evaluation on MMBench · MMBench devaccuracy: OpenFlamingo v2 36.7% -> 2.6%; GPT-4v -10.8%Hurt
CircularEval widens the gap between model sizes (ablation)LLaVA-v1.5-13B vs 7B · MMBench devTop-1 accuracy gap: 2.1% -> 4.7%Improved
add LLM-based choice extraction on top of heuristic matching (ablation)VisualGLM-6B · MMBench devaccuracy: +23.2Improved
add LLM-based choice extraction (ablation)Qwen-VL-Plus · MMBench devaccuracy: +15.0Improved
add LLM-based choice extraction (ablation)Gemini-Pro-V · MMBench devaccuracy: +0.8Improved
swap choice extractor among GPT-4, GPT-3.5 and InternLM2-7B (ablation)choice extraction · MMBench devaccuracy: gap at most 0.3% for top proprietary VLMsNo clear effect
switch LLM from Vicuna-v1.5 to InternLM2-20B (ablation)LLaVA series · MMBench testoverall accuracy: 63.4 -> 72.3 (7B -> 20B InternLM2)Improved
scale Vicuna from 7B to 13B (ablation)MiniGPT4, InstructBLIP, LLaVA v1.5 · MMBench testoverall Top-1 accuracy: +8.3%, +1.5%, +3.5%Improved
assume perfect answers on content-moderation rejected questions (ablation)proprietary VLMs · MMBench testaccuracy: GPT-4v 74.3 -> 76.2; Gemini-Pro-V 70.2 -> 72.6Improved
few-shot and chain-of-thought prompting instead of zero-shot (ablation)VLMs · MMBenchaccuracyNo clear effect
evaluate in Chinese instead of Englishmost VLMs · MMBench-CNaccuracy: InternLM-XComposer2 drops less than 1%Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy