Research map / Vision-language and multimodal / Multimodal LLMs
Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DCScore, a primitive-information-unit metric for detailed image captioning, the DeCapBench benchmark built on it, and FeedQuill, a fine-grained feedback method for PPO-based preference optimization that reduces VLM hallucinations.
Problem
Evaluating detailed image captions is underexplored because existing metrics and coarse reference captions fail to measure hallucination and fine-grained comprehensiveness of modern VLMs.
Approach
Decompose captions into self-sufficient primitive information units, verify each unit against the image with a VLM, and score precision (hallucination) and recall (coverage vs. hyper-detailed references); reuse the same decomposition-verification signals as fine-grained rewards for PPO on LLaVA models.
Key results
- DCScore reaches PCC 0.6605 with human judgments versus 0.4230 for the best prior metric (FLEUR), and DeCapBench reaches Spearman 0.90 with Arena Elo on captioning.
- FeedQuill-7B (on LLaVA-Onevision-7B) scores 55.65 on DeCapBench versus 53.44 for GPT-4o and 43.49 for its base model.
- FeedQuill on LLaVA-1.5-7B reduces mmHal-V hallucination-related score gain to 2.60 from 1.85 and lowers CHAIR_S from 47.8 to 5.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply FeedQuill preference optimization | LLaVA-Onevision-7B · DeCapBench | score: 43.49 -> 55.65 | Helped |
| apply FeedQuill preference optimization | LLaVA-1.5-7B · mmHal-V | score: 1.85 -> 2.60 | Helped |
| increase reward-model preference data from 10k to the larger FeedQuill dataset (ablation) | FeedQuill reward model · held-out preference datasets | average accuracy: 71.3% to 75.2% | Helped |
| use both c_p and c_r rewards instead of c_p alone (ablation) | LLaVA-1.5-7B preference optimization · LLaVA-W | score: 67.3 -> 76.0 | Helped |
| use only the c_r information-unit-count reward, without c_p precision reward (ablation) | LLaVA-1.5-7B preference optimization · DeCapBench | score: 24.50 -> 10.03 | Hurt |
| combine responses from same and other models rather than using either source alone (ablation) | LLaVA-1.5-13B preference data · DeCapBench | score: 24.50 -> 37.73 | Helped |
| remove omission elements from DCScore evaluation (ablation) | DCScore · human caption judgments | PCC (ρ): 0.6605 -> 0.6151 | Hurt |
| remove non-descriptive elements from DCScore evaluation (ablation) | DCScore · human caption judgments | PCC (ρ): 0.6605 -> 0.6213 | Hurt |
| apply FeedQuill preference optimization | LLaVA-Onevision-7B · MMBench | score: 80.8 -> 80.5 | Hurt |
| apply FeedQuill preference optimization | LLaVA-Onevision-7B · SciQA | score: 96.0 -> 95.9 | Hurt |
| apply FeedQuill preference optimization | LLaVA-1.6-13B · WildVision | score: 33.69 -> 49.69 | Helped |
Surprising findings
- Preference optimization using only the richness score (c_r) degrades performance well below the base model, showing that unconstrained length/richness rewards harm quality without a precision term.
- FeedQuill-7B surpasses GPT-4V on LLaVA-W despite being trained only on captioning tasks.
Limitations
- Decomposition and verification rely on GPT-4o as the LLM and judge, which may introduce bias and cost.
- Human consistency study uses only 500 captions and the paper leaves the number of annotators unspecified (X).
- Preference-data verification uses open-source VLMs whose judgments are not manually verified.
Experimental setup
Datasets: ImageInWords MSCOCO OpenImages ShareGPT4V RLHF-V RLAIF-V HA-DPO POVID CSR STIC
Benchmarks: DeCapBench MMBench MMStar VizWiz SciQA mmHal-V LLaVA-W WildVision CHAIR AI2D ChartQA SEEDBench
Metrics: PCC R^2 Kendall tau Sample-wise tau Spearman correlation F1 score (DCScore) accuracy CHAIR_S CHAIR_I
Baselines: BLEU-4 ROUGE METEOR CIDEr SPICE CLIP-Score PAC-Score CAPTURE CLAIR FLEUR GPT4-Eval FaithScore
Builds on: PPO RLAIF-V FaithScore CAPTURE LLaVA
Closest papers by meaning
- Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.90)
- A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · similarity 0.90)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.89)
- Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage (ICML 2025 · similarity 0.89)
- Top-Down Semantic Refinement for Image Captioning (AAAI 2026 · similarity 0.88)
- Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.