Research map / Vision-language and multimodal / Multimodal LLMs

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds MMBench-Live, a continuously updatable multimodal benchmark generated by a multi-agent pipeline (task-guided real-time image acquisition with feedback control, QA generation, and executable tool-based verification), yielding 5.9K new VLM evaluation instances at about $30 per update.

Problem

Static VLM benchmarks suffer from temporal staleness, data contamination, and high maintenance cost, while prior update strategies either drift semantically or depend on fixed sources and task formats.

Approach

Treats benchmark evolution as task-guided dataset construction: task-related visual patterns extracted from the original benchmark guide feedback-controlled real-time web image retrieval, and generated QA pairs are accepted only if an executable tool-based reasoning chain, run by a vision-blind text verifier, reproduces the answer.

Key results

• MMBench-Live contains 5.9K new QA pairs with 96.06% manual answer correctness and costs about $30 and 1-2 hours per update.
• Cross-version model-averaged correlation with MMBench is Pearson 0.8876 and Spearman 0.8286; Qwen2.5-VL-7B scores 83.10% overall on MMBench-Live.
• PaCoST contamination signal is weaker on MMBench-Live (mean delta 0.0060, Cohen's d 0.059) than on MMBench (mean delta 0.0118, d 0.097).

Surprising findings

• Qwen3-VL-8B-Instruct underperforms its predecessor Qwen2.5-VL-7B on MMBench-Live in the GPT-based version (80.41% vs 83.10% overall).
• The task-level correlation with the original benchmark is weaker (Pearson 0.52) than the model-averaged correlation, indicating instability at fine-grained capability level.

Limitations

• Dynamic updates cannot fully remove implicit memorization of high-frequency visual concepts.
• QA quality is bounded by the underlying agents and foundation models; subtle noise persists under visual ambiguity (e.g., spatial-relation failures in rotated layouts).
• The task-grounded paradigm limits task-space expansion and new task formulations.

Experimental setup

Datasets: MMBench dev set Google Image API retrieved images Flickr images

Benchmarks: MMBench-Live MMBench

Metrics: accuracy Pearson correlation Spearman correlation FID PCA-KL divergence PaCoST confidence shift (Cohen's d, paired t-test)

Baselines: DeepSeek-VL-7B-Chat InstructBLIP-Vicuna-7B LLaVA-v1.5-7B mPLUG-Owl2-7B Qwen3-VL-8B-Instruct Qwen2.5-VL-7B-Instruct VLB-style image editing variants (Adding, Elimination, Outpainting)

Builds on: MMBench PaCoST LLaVA

Closest papers by meaning

• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
• MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.90)
• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.90)
• Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · similarity 0.90)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.90)
• SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.90)