Research map / Vision-language and multimodal / Multimodal LLMs

Top-Down Semantic Refinement for Image Captioning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TDSR, a top-down MCTS-based planning framework that refines VLM image captions coarse-to-fine, using visual-guided parallel expansion and a lightweight value network to cut VLM calls.

Problem

VLM image captioning is myopic under greedy/beam decoding, struggling to keep global narrative coherence while capturing fine-grained details and avoiding hallucination.

Approach

Recasts captioning as a goal-oriented hierarchical planning MDP solved with a tailored MCTS that uses visual-guided parallel expansion, a lightweight transformer value network, adaptive early stopping, and a composite reward with depth incentive and redundancy penalty; applied as a plug-and-play module on top of existing VLMs.

Key results

• On DetailCaps, Qwen2.5-VL-7B+TDSR reaches CAPTURE 72.2 vs 64.7 for the base model; LLaVA-1.5-7B+TDSR reaches 66.7 vs 49.99.
• On COMPOSITIONCAP, Qwen2.5-VL-7B+TDSR reports CIDEr 129.4 and BERTScore 88.9 vs 120.3 and 81.3 for the base model.
• Removing parallel expansion raises latency to 10.56s (4.71x), per the paper's efficiency analysis; the paper does not report a measured VLM call reduction table beyond the abstract's order-of-magnitude claim.

Surprising findings

• The COMPOSITIONCAP BERTScore for the TDSR-enhanced model (88.9) jumps about 7.6 points over its base, and the paper's efficiency section reports 82.5 for the same setting, an internal inconsistency.
• The theoretical sections claim a tighter regret bound and hallucination suppression guarantees, but these are sketched under strong assumptions and not validated empirically.

Limitations

• Several reported numbers are inconsistent across sections (e.g., COMPOSITIONCAP BERTScore 88.9 vs 82.5; ablation CIDEr values differ from tables).
• Hyperparameter list leaves several key values as '...' (branching factor, alpha, c_puct) and the appendix is partly generic.
• Theoretical guarantees rely on strong assumptions (finite state space, Lipschitz value network) that do not match the actual LLM setting.

Experimental setup

Datasets: DetailCaps COMPOSITIONCAP POPE COCO

Benchmarks: DetailCaps COMPOSITIONCAP POPE

Metrics: CAPTURE F1_obj F1_attr F1_rel ROUGE-L BLEU-4 METEOR CIDEr BERTScore accuracy

Baselines: IT Patch Matters FineCaption Shikra-13B MiniGPT-v2 Ferret-13B VisionLLM-H KOSMOS-2 Alpha-CLIP-13B VistaLLM

Builds on: Monte Carlo Tree Search UCT AlphaGo Markov Decision Process

Closest papers by meaning

• MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning (CVPR 2026 · similarity 0.89)
• Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning (ICLR 2025 · similarity 0.88)
• Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning (CVPR 2025 · similarity 0.88)
• Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception (CVPR 2025 · similarity 0.88)
• FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity (CVPR 2025 · similarity 0.88)
• BalCapRL : A Balanced Framework for RL-Based MLLM Image Captioning (NeurIPS 2026 · similarity 0.87)