Research map / Vision-language and multimodal / Multimodal LLMs
CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CAPability, a multi-view (6 views, 12 dimensions) human-annotated image/video caption benchmark that scores captions with precision, hit, and a know-but-cannot-tell (KT) gap metric derived from QA.
Problem
Traditional caption benchmarks use short reference sentences and n-gram metrics that fail to evaluate detailed captions from modern MLLMs, and newer benchmarks cover only narrow aspects while ignoring thoroughness.
Approach
Annotate visual elements (not reference sentences) across 12 dimensions and score captions via GPT-4 Turbo judgments of MIS/COR/INC, yielding precision (correctness) and hit (correctness plus coverage); convert annotations to QA to compute a know-but-cannot-tell gap.
Key results
- Gemini-2.0-flash (79.3%) and GPT-4o (79.2%) reach the highest average precision among evaluated models, while Gemini-1.5-pro leads average hit at 60.4%.
- Gemini-1.5-pro has the lowest average KT at 25.2%, whereas LLaVA-OV-72B, Qwen2VL-72B and InternVL2.5-78B exceed 40%.
- Traditional metrics (CIDEr, BLEU-4, METEOR) fail to align with human judgments, while CAPability rankings show high consistency with a human pairwise arena.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini-1.5-pro instead of GPT-4o (0806) for caption generation | MLLM captioning · CAPability | hit (Avg.): 56.5 -> 60.4 | Helped |
| Use Gemini-2.0-flash instead of GPT-4o (0806) for caption generation | MLLM captioning · CAPability | precision (Avg.): 79.2 -> 79.3 | Helped |
| Use Qwen2.5VL-7B instead of other evaluated 7B open-source models | 7B open-source MLLMs · CAPability | hit (Avg.): 44.8 -> 48.1 | Helped |
| Increase the open-source model scale from 7B to 72B | Open-source MLLMs · CAPability | precision (Avg.) | Helped |
| Use Gemini-1.5-pro instead of LLaVA-OneVision-72B for caption generation | MLLM captioning · CAPability-QA | K̄T (Avg.): 41.3 -> 25.2 | Helped |
| Rely on traditional caption similarity metrics instead of CAPability evaluation | Visual caption evaluation · CAPability | consistency with human evaluation | Hurt |
Surprising findings
- Models show high precision but much lower hit on many dimensions, indicating they describe only what they are confident about.
- Commercial and open 72B models have similar QA accuracy yet vary widely in KT, so knowing an answer does not imply stating it in captions.
Limitations
- Cultural bias toward mainland China and East Asian data sources.
- Only one randomly selected element per dimension per sample, so coverage is statistical rather than exhaustive.
- Evaluation relies on GPT-4 Turbo as judge and images for static dimensions, videos only for dynamic dimensions.
Experimental setup
Datasets: SA-1B COYO-700M Wukong Wikipaintings CompreCap Dream-1K VSI-Bench
Benchmarks: CAPability CAPability-QA
Metrics: precision hit KT (know but cannot tell) CIDEr BLEU-4 METEOR human arena ranking consistency
Baselines: CIDEr BLEU-4 METEOR GPT-4-Turbo direct scoring DetailCaps Dream-1K VDC CompreCap
Builds on: VBench GenEval MUC-7 metric definitions
Closest papers by meaning
- CaptionQA: Is Your Caption as Useful as the Image Itself? (CVPR 2026 · similarity 0.91)
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.91)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.91)
- G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o (AAAI 2025 · similarity 0.90)
- Enhancing Descriptive Captions with Visual Attributes for Multimodal Perception (CVPR 2026 · similarity 0.90)
- OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.