Research map / Vision-language and multimodal / Multimodal LLMs
SEED-Bench: Benchmarking Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SEED-Bench-2, a 24K-question human-verified multiple-choice benchmark across 27 dimensions that evaluates MLLMs' hierarchical capabilities (L1–L3) including image generation, scored by answer likelihood ranking and CLIP similarity.
Problem
Existing MLLM benchmarks only assess comprehension of single image-text inputs with open-form answers judged by humans or GPT, so they fail to measure interleaved comprehension and image/text generation capabilities.
Approach
Organizes MLLM capabilities into hierarchical levels L0–L4 by input/output modality and builds a large multiple-choice benchmark covering comprehension and generation, scored objectively without human or GPT judges (answer-ranking by likelihood for text, CLIP similarity for images).
Key results
- InternLM-Xcomposer-VL achieves the top L1 averaged accuracy of 59.2% (about 60% ceiling), the best among 23 evaluated models at L1 and L2.
- Only a few open-source MLLMs reach L3; SEED-LLaMA ranks first at L3 with averaged accuracy 44.8%, followed by Emu at 42.3%.
- Most MLLMs score lower on L2 interleaved image-text comprehension than on L1 fixed-format inputs; all models score around 30% or below on chart understanding and visual mathematics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a model trained on interleaved image-text data rather than one trained only on structured image-caption pairs | MLLMs evaluated on SEED-Bench-2 Part-2 · Interleaved Image-Text Comprehension | accuracy | Helped |
| Evaluate on interleaved image-text inputs instead of fixed-form image-text inputs | Most evaluated MLLMs · SEED-Bench-2 Part-2 vs L1 | average accuracy | Hurt |
| Add image-generation ability while retaining text output | NExt-GPT · SEED-Bench-2 L1 | average accuracy | Hurt |
| Train video-language models on video data | VideoChatGPT and Valley · SEED-Bench-2 temporal understanding | accuracy | Hurt |
| Apply MLLMs to chart understanding and visual mathematics | Evaluated MLLMs · Chart Understanding and Visual Mathematics | accuracy | Hurt |
Surprising findings
- Video-specialized models such as Video-ChatGPT and Valley underperform image-pretrained MLLMs on temporal understanding.
- NExt-GPT, despite reaching L3 generation capability, performs poorly on L1 comprehension tasks.
Limitations
- L4 interleaved image-text output generation is not evaluated.
- Only open-source MLLMs are evaluated; some L3-capable models are closed.
- Image generation is scored by CLIP similarity, which is an indirect proxy for correctness.
Experimental setup
Datasets: CC3M Tag2Text captions IC03 IC13 IIIT5k SVT MME MMBench Google Landmark v2 PlotQA VCR ScienceQA
Benchmarks: SEED-Bench-2
Metrics: accuracy averaged accuracy per capability level rank
Baselines: BLIP-2 InstructBLIP LLaVA-1.5 Kosmos-2 Qwen-VL-Chat InternLM-Xcomposer-VL Emu NExt-GPT SEED-LLaMA
Builds on: SEED-Bench-1 MMBench MME
Closest papers by meaning
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.93)
- MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models (AAAI 2026 · similarity 0.92)
- II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models (NeurIPS 2024 · similarity 0.92)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.92)
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark (ICML 2024 · similarity 0.91)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.