Research map / Vision-language and multimodal / Multimodal LLMs
ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Ablation Capability for Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConvBench, a 577-conversation multi-turn benchmark that progressively tests LVLMs on perception, reasoning, and creation with an LLM-judged, hierarchy-based error attribution protocol.
Problem
Existing LVLM benchmarks evaluate capabilities independently in single-turn dialogues, which makes it hard to attribute errors to perception versus reasoning and ignores multi-turn instruction following.
Approach
A three-level capability hierarchy (perception, reasoning, creation) instantiated as three sequential instructions per image, with human-verified references and ablations that substitute perfect perception or reasoning answers to attribute errors, scored by an LLM judge (ConvEval) in pairwise or direct grading.
Key results
- GPT-4V achieves the top pairwise overall score of 39.51% win rate vs human reference, with 38.47, 39.34, and 37.61 for perception, reasoning, and creation.
- Under perfect perception, pairwise reasoning scores rise by an average of 11.21 points across 19 LVLMs, indicating weak perception undermines reasoning.
- BLIP2-FLAN-T5-XL scores near zero (0.11 pairwise R2) while GPT-4V reaches 40.55 pairwise multi-turn score, showing a large gap to human references.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a human-verified perception response instead of the model's perception response (ablation) | LVLMs evaluated on ConvBench · ConvBench reasoning, pairwise grading | win rate: +11.21 average score | Helped |
| Use a human-verified perception response instead of the model's perception response (ablation) | LVLMs evaluated on ConvBench · ConvBench creation, pairwise grading | win rate: +5.31 average score | Helped |
| Use a human-verified perception response instead of the model's perception response (ablation) | LVLMs evaluated on ConvBench · ConvBench reasoning and creation, direct grading | score: +1.25 average score for each | Helped |
| Use human-verified perception and reasoning responses instead of the model's responses (ablation) | LVLMs evaluated on ConvBench · ConvBench creation, pairwise grading | win rate: +6.96 average score | Helped |
| Use human-verified perception and reasoning responses instead of the model's responses (ablation) | LVLMs evaluated on ConvBench · ConvBench creation, direct grading | score: -0.39 average score | Hurt |
| Include annotated focus points when evaluating creative responses (ablation) | ConvEval · 50 creative instances | comparative win rate of evaluation quality: 56% vs 44% | Helped |
| Include perception, reasoning, and creation judgments in the overall conversation evaluation (ablation) | ConvEval · 50 instances | comparative win rate of evaluation quality: 60% vs 40% | Helped |
| Include perception, reasoning, and creation judgments in the overall conversation evaluation (ablation) | ConvEval · 50 instances | failure to provide an ultimate rating: 8% | Helped |
| Evaluate reasoning or creation instructions in a multi-turn sequence rather than directly in a single turn (ablation) | GPT-4V on ConvBench · ConvBench reasoning and creation | performance | Helped |
Surprising findings
- GeminiProVision, an open-source-labelled model in the paper, underperforms despite strong single-turn results, attributed to weak multi-turn ability.
- Direct grading gives a mean creation shift of -0.39 under perfect perception and reasoning, suggesting the judge struggles with subtle distinctions.
Limitations
- Judge is ChatGPT-3.5, which may have position bias and struggles with subtle pairwise differences in direct grading.
- Pairwise win-rate and direct scores disagree in magnitude; creation shifts are unstable under direct grading.
- Reference answers are generated by GPT-4V and only refined by humans, which may bias references.
Experimental setup
Datasets: VisIT-Bench
Benchmarks: ConvBench
Metrics: win rate vs human reference direct grading score (0-10) per-turn and overall conversation scores
Baselines: GPT-4V Claude Reka Flash InternVL-Chat-V1-2 LLaVA-V1.5-13B ShareGPT4V-13B Qwen-VL-Chat MiniGPT-4
Builds on: VisIT-Bench LLaVA MMBench
Closest papers by meaning
- MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs (NeurIPS 2024 · similarity 0.90)
- DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback (CVPR 2024 · similarity 0.89)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.88)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- VisionArena: 230k Real World User-VLM Conversations with Preference Labels (CVPR 2025 · similarity 0.88)
- HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.