Research map / Vision-language and multimodal / Multimodal LLMs
MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMLongBench, a 13,331-example benchmark across five long-context vision-language task categories at standardized 8K-128K lengths with unified cross-modal token counting, and evaluates 46 LCVLMs.
Problem
Existing benchmarks for long-context vision-language models cover only one task type (mostly needle-in-a-haystack or long-document VQA), use inconsistent image-token length control, and rarely provide multiple standardized context lengths.
Approach
A broad benchmark spanning VRAG, NIAH, many-shot ICL, summarization and long-document VQA, with image and text tokens counted jointly (patches plus 2x2 pixel unshuffle) and each example provided at five standardized lengths from 8K to 128K, plus analysis of cross-category correlations.
Key results
- Even GPT-4o averages only 62.9 at 128K; Gemini-2.5-Pro outperforms open-source models by about 20 absolute points on most tasks except ICL.
- Qwen2-VL-72B, trained with a 32K window, still reaches an average of 51.9 at 128K input.
- Reasoning helps: Gemini-2.0-Flash-Thinking improves summarization by 25.3% and DocVQA by 10.1% over Gemini-2.0-Flash.
- NIAH tasks correlate weakly with other categories; long-document VQA has the highest average correlation with other categories.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the thinking variant instead of the base model | Gemini-2.0-Flash · MMLongBench summarization | score: +25.3% | Helped |
| Use the thinking variant instead of the base model | Gemini-2.0-Flash · MMLongBench DocVQA | score: +10.1% | Helped |
| Replace PDF-formatted ViQuAE images with the corresponding entity names | LCVLMs on ViQuAE · ViQuAE 128K | score: +26.4 points | Helped |
| Replace PDF-formatted documents with OCR-extracted plain text | LCVLMs on MMLongBench-Doc · MMLongBench-Doc | score | Mixed |
| Use PDF documents instead of OCR-extracted plain text | LCVLMs on MMLongBench-Doc vision-needed cases · MMLongBench-Doc vision-needed cases | score | Helped |
| Use OCR-extracted plain text instead of PDF documents | LCVLMs on MMLongBench-Doc text-pure cases · MMLongBench-Doc text-pure cases | score | Helped |
| Use the corresponding LLM instead of the LCVLM on OCR-extracted text | Qwen2.5 models on DocVQA text-pure cases · DocVQA text-pure cases | score | Helped |
Surprising findings
- ROUGE-L is nearly flat across input lengths and model sizes, so it is replaced by GPT-4o-based metrics for summarization.
- Simply predicting a list of zeros scores over 30 on image-needle counting in MM-NIAH, so the soft accuracy metric is exploitable.
Limitations
- Greedy decoding and 100 sampled examples per dataset limit statistical robustness; no error bars are reported.
- Padding documents in DocVQA may be accidentally relevant to the question.
- Summarization relies on GPT-4o as judge, with open-source judges shown to be less reliable.
Experimental setup
Datasets: InfoSeek ViQuAE TriviaQA KILT Visual Haystack MM-NIAH OBELICS Stanford Cars Food101 SUN397 iNat2021 GovReport
Benchmarks: MMLongBench
Metrics: SubEM accuracy Fluency-F1 ROUGE-L ROUGE F1
Baselines: GPT-4o Claude-3.7-Sonnet Gemini-2.5-Pro Gemini-2.0-Flash Qwen2.5-VL-32B InternVL3-38B Ovis2-34B Qwen2-VL-72B Gemma3-27B
Builds on: Visual Haystack MM-NIAH HELMET LongDocURL KILT
Closest papers by meaning
- MMLongCite: A Benchmark for Evaluating Faithfulness of Long-Context Vision-Language Models (NeurIPS 2026 · similarity 0.94)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.93)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.92)
- MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models (ICLR 2025 · similarity 0.92)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.91)
- VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.