Research map / Vision-language and multimodal / Multimodal LLMs
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMDU, a multi-turn multi-image long-context benchmark judged by GPT-4o, plus MMDU-45k instruction-tuning data built from Wikipedia with GPT-4o, showing that fine-tuning open LVLMs on it improves MMDU and several existing benchmarks.
Problem
Open-source LVLMs lag far behind proprietary models in real-world conversations involving long multi-turn, multi-image contexts, and existing benchmarks focus on single-choice or short-answer, few-image settings with little conversational instruction-tuning data.
Approach
Build multi-turn, multi-image dialogues from clustered Wikipedia entries using GPT-4o generation with human review, evaluate open-ended answers with GPT-4o as judge across six dimensions, and show that adding the resulting 45k-conversation dataset to SFT closes part of the open/closed gap.
Key results
- GPT-4o scores 70.2 on MMDU while the best open-source model (LLaVA-1.6-mistral) reaches 42.8 on MMDU.
- Fine-tuning InternLM-XC2 on MMDU-45k raises its MMDU average from 35.6 to 50.1 (+14.5).
- Adding MMDU-45k to InternLM-XC2 SFT improves MMStar by +1.1, MathVista by +1.5, and ChartQA by +1.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune InternLM-XC2 with MMDU-45k | InternLM-XC2 · MMDU | average score: 35.6 -> 50.1; +14.5 | Helped |
| Fine-tune LLaVa1.5 with MMDU-45k | LLaVa1.5 · MMDU | average score: 32.2 -> 37.2; +5.0 | Helped |
| Fine-tune InternLM-XC2 with MMDU-45k | InternLM-XC2 · MathVista | score: +1.5 | Helped |
| Fine-tune InternLM-XC2 with MMDU-45k | InternLM-XC2 · AI2D | score: +0.0 | No clear effect |
| Fine-tune LLaVa1.5 with MMDU-45k | LLaVa1.5 · HallusionBench | score: -0.1 | Hurt |
| Fine-tune LLaVa-1.5 with MMDU-45k | LLaVa-1.5 · Mantis (sequence) | score: +6.9 | Helped |
| Increase the context window from 2k to 4k tokens (ablation) | LLaVa-1.5 · MMDU | overall score: 20.9 -> 30.0 | Helped |
| Increase the context window from 2k to 8k tokens with RoPE interpolation (ablation) | LLaVa-1.5 + MMDU-45k · MMDU | overall score: 22.3 -> 37.1 | Helped |
| Use continue training instead of adding MMDU-45k to the existing pool (ablation) | LLaVa-1.5 · MMDU | overall score: 37.2 vs 37.3 | No clear effect |
| Fine-tune LLaVa-1.5 with MMDU-45k | LLaVa-1.5 · MMStar | score: +1.0 | Helped |
| Fine-tune InternLM-XC2 with MMDU-45k | InternLM-XC2 · MMStar | score: +1.1 | Helped |
Surprising findings
- LLaVA-1.5 fine-tuned on MMDU-45k gains only about +0 on MMB and HallusionBench, showing gains concentrate on multi-image and conversational metrics.
- Continued training and adding to the existing data pool give essentially the same final results.
Limitations
- Benchmark is English-only and does not cover multilingual abilities.
- Focuses on daily scenarios rather than specialized domain expertise such as math.
- Relies on GPT-4o for both data generation and judging; Wikipedia-based data may carry biases and hallucination risk.
Experimental setup
Datasets: Wikipedia MMDU-45k LLaVA-665k
Benchmarks: MMDU MMStar MathVista ChartQA MMBench MMMU AI2D HallusionBench MMVet BLINK Qbench2 Mantis
Metrics: average score over six dimensions (creativity, richness, vis Pearson/Spearman/Kendall correlation with human scores
Baselines: GPT-4o GPT-4-turbo Claude3-Opus Qwen-VL-Max Monkey Idefics2 LLaVA-1.5 LLaVA-1.6 Deepseek-VL MiniCPM-V-2.5 Yi-VL InternVL-Chat-V1.5
Builds on: LLaVA 1.5 InternLM-XComposer2 Wikipedia MMStar MathVista
Closest papers by meaning
- ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Ablation Capability for Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback (CVPR 2024 · similarity 0.89)
- V^2Dial: Unification of Video and Visual Dialog via Multimodal Experts (CVPR 2025 · similarity 0.89)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.89)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.89)
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.