Research map / Vision-language and multimodal / Multimodal LLMs
Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPARC, a training-free decoding method that selectively amplifies visual token value vectors across generation steps using an EMA-based relative activation score and cumulative selection counts, improving both precision and recall in detailed image captioning.
Problem
Detailed image captioning in MLLMs faces a precision-recall trade-off: existing hallucination-mitigation methods that boost visual attention improve precision but reduce recall, and visual attention weakens and gets noisier as captions lengthen.
Approach
Select visual tokens whose attention jumps relative to an EMA of their past attention (relative activation score), count how often each is selected, and scale their value vectors by alpha^count so that consistently relevant tokens are progressively reinforced rather than all visual tokens being uniformly amplified.
Key results
- On LLaVA-1.5 with IIW-400, CLAIR improves from 56.36 (baseline) to 61.49 (+5.13), the best among compared methods (PAI 56.86, OPERA 51.02).
- On CHAIR, SPARC is the only method that improves recall over the baseline (79.98 vs 79.46) while also raising precision (87.72 vs 84.70) and F1 (83.67 vs 81.99).
- Decoding latency is 31.21 ms/token vs 30.37 for baseline (+2.8%), while OPERA takes 322.28 ms/token (+961%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SPARC instead of the baseline | LLaVA-1.5 · IIW-400 | CLAIR: 56.36 -> 61.49 | Helped |
| Apply SPARC instead of the baseline | LLaVA-1.5 · DOCCI | CLAIR: 59.26 -> 62.70 | Helped |
| Apply SPARC instead of the baseline | LLaVA-1.5 · MS-COCO 2014 val | Recall: +0.52%p | Helped |
| Apply SPARC instead of the baseline | LLaVA-1.5 · MS-COCO 2014 val | Precision: +3.02%p | Helped |
| Apply SPARC instead of the baseline | LLaVA-1.5 · MS-COCO 2014 val | F1: +1.68%p | Helped |
| Apply SPARC instead of the baseline | LLaVA-NeXT · IIW-400 | CLAIR: 58.86 -> 64.94 | Helped |
| Apply SPARC instead of the baseline | Qwen2-VL · IIW-400 | CLAIR: 78.34 -> 79.70 | Helped |
| Apply SPARC instead of the baseline | LLaVA-NeXT · DOCCI | CLAIR: 62.49 -> 66.99 | Helped |
| Apply SPARC instead of the baseline | Qwen2-VL · DOCCI | CLAIR: 79.22 -> 80.64 | Helped |
| Apply SPARC instead of naive attention scaling | LLaVA-1.5 · MS-COCO 2014 val | Attention on Relevant Regions (%): 15.50 -> 19.17 | Helped |
| Apply SPARC instead of the baseline | LLaVA-1.5 · IIW-400 | Generation Time (ms/token): 30.37 -> 31.21 | Hurt |
| Remove token selection and recalibrate all image tokens (ablation) | SPARC on LLaVA-1.5 · IIW-400 | CLAIR: 61.49 -> 59.10 | Hurt |
Surprising findings
- Naive attention amplification reduces recall, and the paper argues this is linked to attention becoming static across decoding steps.
- PAI achieves the highest precision on CHAIR (90.64) but the lowest recall (72.44).
Limitations
- Hyperparameters (layer, tau, alpha, beta) must be tuned per model, with thresholds differing across LLaVA-1.5, LLaVA-NeXT and Qwen2-VL.
- Evaluation relies heavily on GPT-4o-based CLAIR scoring and small human study (100 captions) on one model.
- Gains on CHAIR recall are small (+0.52 points).
Experimental setup
Datasets: DOCCI MS-COCO 2014 validation IIW-400
Benchmarks: IIW-400 DOCCI CHAIR
Metrics: CLAIR score CHAIR precision CHAIR recall CHAIR F1 token generation time (ms/token) human winning ratio
Baselines: OPERA VCD VOLCANO PAI naive attention enhancement
Builds on: OPERA VCD VOLCANO PAI
Closest papers by meaning
- See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.89)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.89)
- Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.89)
- Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference (ICML 2026 · similarity 0.89)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.