Research map / Architectures and efficiency / Attention and transformers

STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes STILL, an intra-layer hybrid linear attention method that routes tokens to softmax attention via a local Self-Saliency Score and preserves pretrained norms with a Norm-Preserved Feature Map, linearizing Llama 3/3.1 8B with about 0.04B tokens.

Problem

Linearizing pretrained LLMs with intra-layer hybrid attention relies on position-based sliding-window routing, which misses distant but important tokens, and learnable feature maps distort pretrained vector norms that softmax attention depends on.

Approach

Score each token's importance with a Self-Saliency Score computed from the KL-style divergence between local sliding-window attention with and without the diagonal term, route the top-scoring tokens to softmax attention and the rest to linear attention, and use an NP-Map that normalizes the MLP output direction and reinjects the original norm; routing is delayed and done per chunk for parallel tra

Key results

• On Llama 3.1 8B, STILL reaches 61.3 MMLU (vs 46.9 for Liger-GLA) and 74.7 average including MMLU, above the original Llama 3.1 8B average of 74.2.
• On S-NIAH at 4K, STILL scores 86.2 while LoLCATs and Liger-GLA largely fail (8.8 and 0.0).
• On the Llama 3.2 1B intra-inter setup at 16K, STILL gets 100 on S-NIAH-1 and a RULER score of 27.20, outperforming Zebra-Llama by 13.83.

Surprising findings

• Diagonal attention alone is nearly as good as Self-Saliency on short-context reasoning (72.0 vs 72.5), but far worse on long-context retrieval (23.1 vs 42.6 average).

Limitations

• Experiments limited to models up to 8B parameters; transfer to 70B+ untested.
• Tailored to text-only autoregressive models; not extended to multimodal architectures.
• Fixed chunk and window configurations; no dynamic task-adaptive window sizing.

Experimental setup

Datasets: Cleaned Alpaca DCLM

Benchmarks: PIQA ARC-E ARC-C HellaSwag WinoGrande MMLU S-NIAH RULER BABILong

Metrics: accuracy normalized accuracy RULER score peak GPU memory latency

Baselines: LoLCATs Liger-GLA Hedgehog-LoLCATs T2R-LoLCATs SUPRA Llamba Mamba2 Zebra-Llama MambaInLlama SWA

Builds on: LoLCATs Hedgehog Liger KL-LS NaLaFormer LoRA

Closest papers by meaning

• Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.89)
• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.88)
• The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.88)
• One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.88)
• Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
• Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.87)