Research map / Vision-language and multimodal / Multimodal LLMs
When Sinks Help or Hurt: Unified Framework for Attention Sink in MLLMs
Jiho Choi, Jaemin Kim, JINHWI PARK, Seunghoon Hong, Sanghwan Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn per-layer V-sink vs rest key scaling gate (LSG) at layer 10 with NTP loss, frozen backbone | LLaVA-1.5-7B · CVBench | accuracy delta (%p): +2.14 | Improved |
| stack five independently trained LSG gates {L3,L6,L7,L10,L19} (ablation) | LSG · MMStar / CVBench | accuracy delta (%p): +0.72 -> +1.55 MMStar; +2.14 -> +3.08 CVBench | Improved |
| add L19 gate on top of three stacked gates (ablation) | LSG stack · MMStar | accuracy delta (%p): +1.17 -> +0.74 | Hurt |
| apply LSG at early layer L2 (ablation) | LSG · MMStar / CVBench sub-tasks | non-negative sub-tasks of 10: 3 of 10 non-negative vs 8 of 10 at L10 | Hurt |
| replace prior sink methods (VAR, Sink-to-the-front, DIYSink) with LSG L10 | LLaVA-1.5-7B · MMStar / CVBench | accuracy delta (%p): +0.72/+2.14 vs best prior +0.25/+0.75 | Improved |
| rebalance sink vs ordinary attention with oracle gate on fine-grained perception (ablation) | LLaVA-1.5-7B · MMStar fine-grained | accuracy gain (%p): up to +6.49 | Improved |
| rebalance sink vs ordinary attention with oracle gate on coarse perception (ablation) | LLaVA-1.5-7B · MMStar coarse | accuracy gain (%p): up to -4.27 | Hurt |
| strengthen sinks in logical reasoning, early vs late layers (ablation) | LLaVA-1.5-7B · MMStar logical | accuracy gain (%p): +7.07 at L4-L8; negative at L22-L26 | Mixed |
| split rest group into L-sinks and ordinary tokens (3-group) (ablation) | oracle key gating · CVBench Relation | accuracy gain (%p) | No clear effect |
Closest papers by meaning
- To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.92)
- See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.91)
- When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.89)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.89)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.89)
- On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
