Research map / Vision-language and multimodal / Multimodal LLMs
RAR: Reversing Visual Attention Re-Sinking for Unlocking Potential in Multimodal Large Language Models
Zhehan Kan, Xin Li, Yanlin Liu, Xiaochen Yang, Xinghua Jiang, Yinsong Liu, Deqiang Jiang, Xing Sun, Qingmin Liao, Wenming Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a "visual attention re-sinking" phenomenon as the cause of suboptimal final decoder layers in MLLMs and proposes RAR to reverse it and unlock the underused capacity.
Problem
MLLMs often have suboptimal output layers, where intermediate decoder layers outperform the final layers, indicating underutilized model capacity.
Approach
Attributes the suboptimal final layers to a visual attention re-sinking phenomenon, which the abstract says is driven by attention gradient sparsity; RAR is proposed to reverse this re-sinking so the final layers better use visual information. Full method details are not available from the abstract.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune with SADS head sparsification instead of plain SFT | Qwen2.5-VL-3B · OVDEval | accuracy: 39.9 -> 43.8 | Improved |
| fine-tune with SADS instead of SFT on hallucination | Qwen2.5-VL-32B · CHAIR | hallucination rate: 30.1 -> 21.2 | Improved |
| add SADS on top of SFT | Qwen2.5-VL-7B · MME | score: 2230.2 -> 2289.8 | Improved |
| keep all vision heads, sparsify sink heads at inference-time structure | Qwen2.5-VL-3B · OVDEval latency | latency: 1.332 -> 1.195 | Improved |
| add an extra sink_S head (ablation) | Qwen2.5-VL-3B · OVDEval | accuracy: 43.8 -> 43.0 | Hurt |
| remove one vision head (ablation) | Qwen2.5-VL-3B · OVDEval | accuracy: 43.8 -> 42.6 | Hurt |
| dynamic entropy-valley head selection vs fixed 25% sink heads (ablation) | SADS · OVDEval | accuracy: 42.8 -> 43.8 | Improved |
| keep 100% sink heads (no sparsification) (ablation) | SADS · OVDEval | accuracy: 43.8 -> 39.9 | Hurt |
| add shared head designation (ablation) | SADS · benchmarks | accuracy | Improved |
| start SADS from earlier or later layer than selected (ablation) | SADS · OVDEval | accuracy | Hurt |
| replace SADS by KL regularization on head weights (ablation) | SFT · OVDEval | accuracy: 39.9 -> 40.8 (vs 43.8 SADS) | No clear effect |
| replace SADS by attention reweighting (ablation) | SFT · OVDEval | accuracy: 39.9 -> 40.6 (vs 43.8 SADS) | No clear effect |
Surprising findings
- Intermediate decoder layers of MLLMs often outperform the final layers, signaling underutilized capacity in the output layers.
Closest papers by meaning
- See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.93)
- Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models (ICML 2026 · similarity 0.92)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.91)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.91)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
