AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

RAR: Reversing Visual Attention Re-Sinking for Unlocking Potential in Multimodal Large Language Models

Zhehan Kan, Xin Li, Yanlin Liu, Xiaochen Yang, Xinghua Jiang, Yinsong Liu, Deqiang Jiang, Xing Sun, Qingmin Liao, Wenming Yang

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies a "visual attention re-sinking" phenomenon as the cause of suboptimal final decoder layers in MLLMs and proposes RAR to reverse it and unlock the underused capacity.

Problem

MLLMs often have suboptimal output layers, where intermediate decoder layers outperform the final layers, indicating underutilized model capacity.

Approach

Attributes the suboptimal final layers to a visual attention re-sinking phenomenon, which the abstract says is driven by attention gradient sparsity; RAR is proposed to reverse this re-sinking so the final layers better use visual information. Full method details are not available from the abstract.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune with SADS head sparsification instead of plain SFTQwen2.5-VL-3B · OVDEvalaccuracy: 39.9 -> 43.8Improved
fine-tune with SADS instead of SFT on hallucinationQwen2.5-VL-32B · CHAIRhallucination rate: 30.1 -> 21.2Improved
add SADS on top of SFTQwen2.5-VL-7B · MMEscore: 2230.2 -> 2289.8Improved
keep all vision heads, sparsify sink heads at inference-time structureQwen2.5-VL-3B · OVDEval latencylatency: 1.332 -> 1.195Improved
add an extra sink_S head (ablation)Qwen2.5-VL-3B · OVDEvalaccuracy: 43.8 -> 43.0Hurt
remove one vision head (ablation)Qwen2.5-VL-3B · OVDEvalaccuracy: 43.8 -> 42.6Hurt
dynamic entropy-valley head selection vs fixed 25% sink heads (ablation)SADS · OVDEvalaccuracy: 42.8 -> 43.8Improved
keep 100% sink heads (no sparsification) (ablation)SADS · OVDEvalaccuracy: 43.8 -> 39.9Hurt
add shared head designation (ablation)SADS · benchmarksaccuracyImproved
start SADS from earlier or later layer than selected (ablation)SADS · OVDEvalaccuracyHurt
replace SADS by KL regularization on head weights (ablation)SFT · OVDEvalaccuracy: 39.9 -> 40.8 (vs 43.8 SADS)No clear effect
replace SADS by attention reweighting (ablation)SFT · OVDEvalaccuracy: 39.9 -> 40.6 (vs 43.8 SADS)No clear effect

Surprising findings

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy