Research map / Vision-language and multimodal / Multimodal LLMs
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMT-Bench, a 31,325-question multi-choice benchmark spanning 32 meta-tasks and 162 subtasks to evaluate LVLMs on multitask multimodal understanding, and uses a LoRA-based task map for taxonomy analysis.
Problem
Existing LVLM benchmarks cover few multimodal tasks and rudimentary capabilities, so they cannot track progress toward multitask AGI breadth.
Approach
Builds a broad hierarchical benchmark by repurposing many public datasets into uniform metadata and generating multi-choice questions (manually or via ChatGPT), then evaluates 30 LVLMs and builds a task map from LoRA-derived task vectors to find in- and out-of-domain tasks.
Key results
- InternVL-Chat-V1.2-34B achieves the top overall score of 63.4%, above GPT-4V (62.0%) and GeminiProVision (61.6%).
- GPT-4V scores 62.0 overall and 55.6 excluding visual recognition, showing substantial room for improvement.
- BLIP2 without instruction tuning outperforms most instruction-tuned LVLMs such as LLaVA variants on the closed-set benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multiple images as the prompt instead of a single image | GeminiProVision · Face Retrieval (FR) | accuracy: 30.5 to 92.5 | Helped |
| use multiple-image prompts rather than a single-image prompt | Multi-Images LVLMs · 28 tasks requiring multiple images | accuracy | Helped |
| use multiple-image prompts rather than a single-image prompt | Single-Image LVLMs, except Monkey · 28 tasks requiring multiple images | accuracy | Helped |
| use visual referring prompts | most LVLMs · 14 visual referring prompting tasks | accuracy | Hurt |
| increase model size from 7B to 13B | LLaVA-v1.5 · MMT-Bench, all subtasks | accuracy: 49.5 to 51.7 | Helped |
| increase model size from 7B to 13B | LLaVA-v1.5-Xtuner · MMT-Bench, all subtasks | accuracy: 50.2 to 51.1 | Helped |
| upgrade the LLM from InternLM to InternLM2 | LLaVA · MMT-Bench, all subtasks | accuracy: 49.7 to 50.8 | Helped |
Surprising findings
- BLIP2, which lacks instruction tuning, outperforms most LVLMs trained on millions of instruction examples.
- Most LVLMs show poor generalization on visual referring (visual prompting) tasks, with visual prompting lagging text-coordinate settings.
Limitations
- Meta-task averages over subtasks with different subtask counts may bias assessment.
- Benchmark cannot include all multimodal tasks, so it is not a full test of multitask AGI.
- Data is curated from existing public datasets, which may introduce source and demographic biases.
Experimental setup
Datasets: SEED-Bench MMBench MM-Vet MMMU Tiny LVLM-eHub WhoopsVQA DocVQA ChartQA
Benchmarks: MMT-Bench
Metrics: accuracy Kendall's tau
Baselines: GPT-4V GeminiProVision Qwen-VL-Plus LLaVA-NEXT-34B BLIP2 Random Guess Frequent Choice
Builds on: OpenCompass evaluation protocol Taskonomy Task2Vec LoRA
Closest papers by meaning
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.91)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.91)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.91)
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · similarity 0.90)
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.90)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.