Research map / Large language models / Language, safety and interpretability

On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Task vectors and steering · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes attention sinks in Omni-LLMs and proposes OutRo, a training-free inference-time method that rotates non-sink head outputs toward the sink value direction and relaxes the causal mask at sink tokens in one early layer, improving seven video QA benchmarks with about 1.1x decoding overhead.

Problem

Attention sink tokens absorb disproportionate attention in Omni-LLMs that process video, audio and text, and prior work disagrees on whether they are redundant or functionally meaningful; existing sink-aware inference methods add overhead or need attention maps.

Approach

Sink value vectors act as a shared bias added to all token outputs; OutRo exploits this by gated rotation of non-sink head outputs toward the sink value direction and a one-time causal-mask relaxation for sink queries at an early layer, all without attention maps or extra forward passes.

Key results

• On Qwen2.5-Omni-7B, OutRo improves AVHBench from 71.60 to 73.78 (+2.18), the largest gain among the reported benchmarks.
• OutRo adds only about 1.11x decoding latency (0.98 to 1.09 sec/token) with unchanged GPU memory versus Flash-attn, while VCD takes 2.02x and AVCD 8.04x.
• OutRo improves video-SALMONN2+ and Qwen2.5-Omni-3B across most audio-visual and visual-only benchmarks, and also gives gains on VLMs such as VideoLLaMA3 (+1.47 on VideoHolmes).

Surprising findings

• High sink attention does not imply head redundancy: heads with similar high sink scores produce both improvements and degradations when pruned.
• Zeroing the top sink key dimension removes the sink attention pattern but does not improve performance, and consistently degrades it.

Limitations

• Rotation strength gamma must be manually tuned per model and the optimal value varies across architectures.
• Improvements are modest in magnitude; many gains are around 0.1 to 1 point.
• Evaluation is limited to video QA and captioning tasks, mostly on a few Omni-LLMs.

Experimental setup

Datasets: AVHBench OmniBench AVUT DailyOmni VideoHolmes VideoMME ActivityNetQA

Benchmarks: OmniBench AVUT AVHBench DailyOmni VideoHolmes VideoMME (medium) ActivityNetQA

Metrics: accuracy decoding latency (sec/token) GPU memory

Baselines: Flash-attn decoding VCD AVCD VAR Eager attention

Builds on: massive activations visual attention sink video contrastive decoding (VCD) AVCD

Closest papers by meaning

• Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.92)
• Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings (NeurIPS 2025 · similarity 0.90)
• When Attention Sink Emerges in Language Models: An Empirical View (ICLR 2025 · similarity 0.90)
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.90)
• See What You Are Told: Visual Attention Sink in Large Multimodal Models (ICLR 2025 · similarity 0.90)
• Efficient Streaming Language Models with Attention Sinks (ICLR 2024 · similarity 0.88)