Research map / Vision-language and multimodal / Multimodal LLMs
MMBENCH: Is Your Multi-Modal Model an All-around Player?
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CircularEval instead of single-pass VanillaEval (ablation) | VLM evaluation on MMBench · MMBench dev | accuracy: OpenFlamingo v2 36.7% -> 2.6%; GPT-4v -10.8% | Hurt |
| CircularEval widens the gap between model sizes (ablation) | LLaVA-v1.5-13B vs 7B · MMBench dev | Top-1 accuracy gap: 2.1% -> 4.7% | Improved |
| add LLM-based choice extraction on top of heuristic matching (ablation) | VisualGLM-6B · MMBench dev | accuracy: +23.2 | Improved |
| add LLM-based choice extraction (ablation) | Qwen-VL-Plus · MMBench dev | accuracy: +15.0 | Improved |
| add LLM-based choice extraction (ablation) | Gemini-Pro-V · MMBench dev | accuracy: +0.8 | Improved |
| swap choice extractor among GPT-4, GPT-3.5 and InternLM2-7B (ablation) | choice extraction · MMBench dev | accuracy: gap at most 0.3% for top proprietary VLMs | No clear effect |
| switch LLM from Vicuna-v1.5 to InternLM2-20B (ablation) | LLaVA series · MMBench test | overall accuracy: 63.4 -> 72.3 (7B -> 20B InternLM2) | Improved |
| scale Vicuna from 7B to 13B (ablation) | MiniGPT4, InstructBLIP, LLaVA v1.5 · MMBench test | overall Top-1 accuracy: +8.3%, +1.5%, +3.5% | Improved |
| assume perfect answers on content-moderation rejected questions (ablation) | proprietary VLMs · MMBench test | accuracy: GPT-4v 74.3 -> 76.2; Gemini-Pro-V 70.2 -> 72.6 | Improved |
| few-shot and chain-of-thought prompting instead of zero-shot (ablation) | VLMs · MMBench | accuracy | No clear effect |
| evaluate in Chinese instead of English | most VLMs · MMBench-CN | accuracy: InternLM-XComposer2 drops less than 1% | Hurt |
Closest papers by meaning
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.94)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.92)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.92)
- VHELM: A Holistic Evaluation of Vision Language Models (NeurIPS 2024 · similarity 0.92)
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.91)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
