Research map / Architectures and efficiency / Attention and transformers

LinearARD: Linear-Memory Attention Distillation for RoPE Restoration

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LinearARD, a self-distillation method that restores RoPE-scaled LLMs by aligning final-layer Q/Q, K/K, V/V relation distributions with a native-RoPE teacher, using a linear-memory KL kernel, reaching RULER gains with 4.25M tokens.

Problem

RoPE-based context extension (e.g., position interpolation) degrades short-context accuracy and requires data-hungry continued pre-training to recover; dense attention-level distillation is memory-prohibitive at long sequences.

Approach

Distill row-wise forward KL between dense Q/Q, K/K, V/V self-relation distributions of the final layer from a frozen native-RoPE teacher to the RoPE-scaled student, with a tiled kernel that stores only log-sum-exp statistics and recomputes logits in backward to get exact KL gradients in linear memory.

Key results

• With 4.25M tokens, LinearARD recovers 93.1%/94.2%/94.3% of native short-context Avg. on LLaMA2-7B, LLaMA3-8B, and Mistral-7B-v0.1.
• LinearARD reaches RULER 63.0/67.8/60.2 on the three backbones, while CPT and LongReD at 4.25M tokens score near zero (0.01-0.04).
• At 32K RULER, LinearARD improves over LongReD (256M tokens) on all three backbones (41.8->47.4, 50.0->51.9, 36.3->37.0).

Surprising findings

• CPT and LongReD at the 4.25M-token budget score near zero on RULER, while the same budget with LinearARD yields strong long-context scores.
• On Mistral-7B-v0.1, LinearARD trails LongReD on RULER Avg. all (60.2 vs 62.3) despite winning at 32K.

Limitations

• Mistral-7B RULER average trails LongReD at 8K/16K; medium-length robustness may benefit from hidden-state distillation.
• Kernel is about 1.6-1.9x faster than naive dense implementation at short lengths and remains a dominant cost in step time; end-to-end throughput drops at 32K.

Experimental setup

Datasets: PG19 SlimPajama

Benchmarks: MMLU LAMBADA MathQA BoolQ OpenBookQA PIQA SIQA ARC-Challenge RULER

Metrics: accuracy recovery ratio RULER average

Baselines: CPT LongReD PI-scaled model without restoration

Builds on: LongReD FlashAttention MiniLM Position Interpolation LoRA

Closest papers by meaning

• LoLCATs: On Low-Rank Linearizing of Large Language Models (ICLR 2025 · similarity 0.88)
• The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry (ICLR 2024 · similarity 0.88)
• SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024 · similarity 0.88)
• Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.88)
• Liger: Linearizing Large Language Models to Gated Recurrent Structures (ICML 2025 · similarity 0.87)
• Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.87)