Research map / Vision-language and multimodal / Multimodal LLMs

CaptionQA: Is Your Caption as Useful as the Image Itself?

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces CaptionQA, a utility-based benchmark that scores MLLM captions by how well a text-only LLM answers 33k taxonomy-grounded multiple-choice questions using only the caption across four domains.

Problem

Existing caption evaluations (n-gram metrics, VLM-as-judge, sparse image-QA benchmarks) do not measure whether captions can replace images for real downstream tasks, and are often non-deterministic or reward verbosity.

Approach

Score captions by QA-on-caption utility: a fixed text-only LLM (Qwen2.5-72B) answers dense, domain-taxonomy-grounded multiple-choice questions from the caption alone, with an explicit "cannot answer" option and a penalty scheme that favors precision over hallucinated detail.

Key results

• Proprietary models lead overall: GPT-5 scores 90.29% and Gemini 2.5 Pro 90.12% on the Long/Simple prompts, while the best open-source model Qwen3-VL-30B-A3B reaches 87.12%.
• Caption utility lags image utility: proprietary models lose 9.2-16.4% from QA-on-image to QA-on-caption, and open-source models lose 11-32.4%.
• Models close on image-QA can diverge sharply on CaptionQA; Claude Sonnet 4.5 and LLaVA-OneVision-7B differ by only 1.1% on QA-on-image but 17.2% on caption-based scores.

Surprising findings

• Taxonomy-Hinted prompts, which add structure, degrade caption utility relative to Long prompts in 23 of 25 categories (mean -10.8%).
• Simple to Long length increases yield almost no gain (+0.35% mean), indicating diminishing returns from longer captions.

Limitations

• Relies on a single text-only QA reader (Qwen2.5-72B), so scores reflect the reader's ability as well as caption quality.
• Small image set (657 images) across four domains, though the paper argues ranking stability holds.
• Question generation relies on GPT-family generators and GPT-5/Gemini validators, which may introduce model-specific biases.

Experimental setup

Datasets: Ego4D EgoExo4D ScanNet Open X-Embodiment

Benchmarks: CaptionQA

Metrics: accuracy cannot ratio utility score (per-question score with partial credit for ca

Baselines: Qwen3-VL GLM-4.1V Qwen2.5-VL InternVL3.5 InternVL3 NVLM-D LLaVA-OneVision GPT-5 Gemini 2.5 Pro Claude Sonnet 4.5

Builds on: MMBench MMMU CapArena CAPability DeCapBench

Closest papers by meaning

• CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.91)
• CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.88)
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning (ICLR 2026 · similarity 0.88)
• Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models (ICLR 2025 · similarity 0.88)
• Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.88)
• A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.88)