Research map / Architectures and efficiency / Attention and transformers
Rope to Nope and Back Again: A New Hybrid Attention Strategy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes RoPE, NoPE and QK-Norm attention for long context, then proposes RNope-SWA, which interleaves full-attention NoPE layers with sliding-window RoPE layers (1:3 ratio) to get better long-context extrapolation, short-context accuracy and training/inference efficiency.
Problem
Long-context LLMs built on RoPE show performance limits at extended lengths, and attention mechanisms such as QK-Norm, NoPE and RoPE have unclear strengths and weaknesses for long-context modeling, while full attention is costly in training and serving.
Approach
Attention analysis reveals a division of labor when NoPE and RoPE layers are interleaved: NoPE layers handle long-range retrieval while RoPE layers capture local, recency-biased information. Restricting RoPE layers to sliding-window attention and keeping NoPE layers full-attention yields a hybrid that extrapolates well without RoPE scaling and cuts KV cache and compute.
Key results
- RNope-SWA scores 59.5 on MMLU vs 57.5 for the RoPE baseline and 42.7 vs 40.9 on GSM8K.
- On Ruler retrieval at 256k, RNope-SWA reaches 74.8 vs 57.1 for the baseline; on Needles-256k it scores 9.97 vs 8.25.
- RNope-SWA trains about 50% faster than the baseline at 64k context and nearly 2x faster at 128k, with up to 75% KV-cache reduction in theory and about 70% latency reduction empirically at 990k input.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interleave NoPE full-attention layers with RoPE sliding-window layers | Baseline · Needles-256k | score: 8.25 -> 9.97 | Helped |
| Interleave NoPE full-attention layers with RoPE sliding-window layers | Baseline · Ruler Retrieval 256k | score: 57.1 -> 74.8 | Helped |
| Interleave NoPE full-attention layers with RoPE sliding-window layers | Baseline · Ruler QA 256k | score: 30.0 -> 42.5 | Helped |
| Interleave NoPE full-attention layers with RoPE sliding-window layers | Baseline · standard benchmarks (MMLU, HellaSwag, ARC-E, ARC-C, SATEn, SATMath, GSM8K, Winogrande, MBPP) | score: +2.0% on MMLU; +1.8% on GSM8k | Mixed |
| Increase SFT RoPE θ from 10,000 to 4 million in interleaved NoPE/RoPE layers (ablation) | RNoPE · Needles-128k | score: 8.036 -> 6.203 | Hurt |
| Add sliding-window attention to RoPE layers (window size 8,192) (ablation) | RNoPE-10k · Needles-128k | score: 8.04 -> 9.56 | Helped |
| Add QK-Norm to RoPE attention (ablation) | RoPE model · Needles 65k | score: 9.82 -> 7.93 | Hurt |
| Remove explicit positional embeddings (NoPE) instead of using RoPE (ablation) | RoPE model · Needles 65k | score: 9.82 -> 9.03 | Hurt |
Surprising findings
- QK-Norm, a widely used stabilizer, performs worst on long-context NIAH and shows a weak attention sink and higher attention entropy.
- Interleaving NoPE and RoPE layers causes RoPE layers to lose retrieval ability and NoPE layers to specialize in retrieval, and increasing RoPE theta hurts rather than helps the hybrid.
Limitations
- Preference training (alignment) was deferred to future work.
- NIAH is acknowledged as a limited benchmark for deeper long-context understanding.
- Analysis of the hybrid's cross-layer mechanism is largely empirical; the underlying reasons are unexplored.
Experimental setup
Datasets: diverse pretraining data (unspecified) RAG datasets multilingual translation datasets with extended passages long code instruction datasets long context retrieval datasets
Benchmarks: MMLU HellaSwag ARC-E ARC-C SAT GSM8K Winogrande MBPP Needle-in-a-haystack Ruler retrieval Ruler QA CommonsenseQA
Metrics: accuracy needles score Ruler retrieval score Ruler QA score training throughput KV cache latency
Baselines: RoPE baseline (Command R+ style) QK-Norm variant NoPE variant RNoPE-10k RNoPE-100k RNoPE-2M RNoPE-4M RNoPE-10k-swa
Builds on: RoPE NoPE QK-Norm Command R+ architecture Ruler
Closest papers by meaning
- Base of RoPE Bounds Context Length (NeurIPS 2024 · similarity 0.90)
- Round and Round We Go! What makes Rotary Positional Encodings useful? (ICLR 2025 · similarity 0.90)
- Scaling Laws of RoPE-based Extrapolation (ICLR 2024 · similarity 0.90)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.90)
- Mixture of In-Context Experts Enhance LLMs' Long Context Awareness (NeurIPS 2024 · similarity 0.89)
- Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.