Research map / Vision-language and multimodal / Multimodal LLMs

Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CapMAS, a training-free LLM-MLLM multiagent pipeline that decomposes hyper-detailed captions into atomic propositions, verifies them against the image, and revises the caption, plus a GPT-4o-based factuality metric and a 19,899-question coverage VQA benchmark.

Problem

Hallucinations in hyper-detailed image captions from MLLMs are hard to detect with existing methods, which degrade on long outputs, and existing caption metrics fail to measure factuality and coverage of detailed captions.

Approach

Decompose a long caption into atomic propositions, check each against the image with an MLLM via True/False token probabilities (isolated from context), and have an LLM rewrite the caption using only verified units; evaluation separates factuality (GPT-4o decomposition and verification) from coverage (answerability of multiple-choice questions from the caption).

Key results

• CapMAS with LLaMA-3-8B and InternVL-Chat-V1.5 raises factuality from 65.5 to 75.9 (avg 62.9 to 67.1) while coverage drops only from 48.2 to 47.3.
• Factuality-metric Spearman correlation with human preferences is 70.2 for the proposed metric vs 67.9 for FActScore and 62.5 for FaithScore.
• On captioning, CapMAS (avg 54.3) outperforms all compared methods including VCD, OPERA, LURE, Volcano, LRV and SPARC, whose averages range from 35.5 to 53.3.

Measured effects reported in the paper

Surprising findings

• Decoding-based and other factuality methods (e.g., VCD) can reduce factuality of detailed captions while improving coverage.
• Idefics2-8B has high factuality (85.2) but very low coverage (13.4), and VQA rankings do not predict detailed captioning performance.

Limitations

• Coverage drops notably when correcting GPT-4V captions with weaker MLLMs such as LLaVA-NeXT (coverage 53.5 to 48.4-50.8).
• Factuality judgments depend on GPT-4o and the MLLM's own image understanding, so the Clean captions do not reach perfect scores.
• The VFC baseline could not be reproduced and thus was not directly compared.

Experimental setup

Datasets: IIW-400 DOCCI DCI

Benchmarks: DOCCI meta-evaluation IIW-400 coverage VQA OpenCompass MME POPE

Metrics: factuality coverage CLAIR Spearman correlation fluency AUROC FPR95

Baselines: VCD OPERA LURE Volcano LRV SPARC CLAIR ALOHa CIDEr CLIP-S RefCLIP-S FaithScore

Builds on: Self-Refine FActScore FaithScore CLAIR VCD

Closest papers by meaning

• PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training (ICLR 2025 · similarity 0.92)
• Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning (NeurIPS 2025 · similarity 0.90)
• When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding (NeurIPS 2025 · similarity 0.89)
• Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.89)
• Hallucination Augmented Contrastive Learning for Multimodal Large Language Model (CVPR 2024 · similarity 0.88)
• Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.88)