Research map / Architectures and efficiency / Attention and transformers
Radar: Fast Long-Context Decoding for Any Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Radar is a training-free decoding accelerator that groups cached keys into segments summarized by random-feature (performer-style) averages, selects top segments per query, and attends only to those tokens plus a sliding window, giving O(t^1.5) total decoding time for any pre-trained Transformer.
Problem
Dot-product attention cost grows linearly per decoded token (quadratically overall), and existing fixes either need retraining (kernelized/linear attention) or permanently evict tokens via heuristics (StreamingLLM, H2O, SnapKV), losing information needed later.
Approach
Use random-feature kernel approximations of exp-dot attention to pre-compute per-segment mean embeddings, so segment importance for a query is found in O(1) per segment; restructure segments only at perfect-square lengths and keep a sliding-window buffer, with a high-probability guarantee of recovering the top segment.
Key results
- On LongBench with Llama-2-7b-chat-hf at 1024 middle tokens, Radar reaches average score 33.07 vs SnapKV 32.83 at the same budget (and 2048-token SnapKV 32.83 with 64.38 percentile vs Radar 71.25).
- Radar keeps perplexity within about 0.2 of vanilla attention on PG-19 with 16K prefill while achieving more than 2x speedup, with degradation at most 10%.
- On Mistral-7B-Instruct-v0.2 with 4096 middle tokens (about 13% of tokens), Radar's LongBench average score is 69.38 percentile, close to vanilla's 75.63 score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Radar instead of vanilla attention | Llama-7b · LongBench | Avg. Score: 25.80 -> 25.84 | Helped |
| Use Radar instead of SnapKV with 1024 middle tokens | Llama-7b · LongBench | Avg. Score: 25.76 -> 25.84 | Helped |
| Use Radar instead of SnapKV with 1024 middle tokens | Llama-2-7b-chat-hf · LongBench | Avg. Score: 32.41 -> 32.93 | Helped |
| Use Radar instead of SnapKV with 4096 middle tokens | Mistral-7B-Instruct-v0.2 · LongBench | Avg. Score: 42.25 -> 42.30 | Helped |
| Increase projection dimension n (ablation) | Radar · PG-19 sample | perplexity | Helped |
| Increase the number of selected segments k (ablation) | Radar · PG-19 sample | generation quality | Helped |
| Select segments with the lowest approximate segment-attention scores instead of the highest (ablation) | Radar · PG-19 sample | perplexity | Hurt |
| Randomly select segments instead of using approximate attention scores (ablation) | Radar · PG-19 sample | perplexity | Hurt |
| Use approximate segment search instead of exact segment search (ablation) | Radar · PG-19 sample | perplexity | Hurt |
| Use Radar instead of vanilla attention | Llama and Mistral · PG-19 and The Stack code sample with 16,384-token prompts | perplexity: around 0.2 | Hurt |
| Use Radar instead of StreamingLLM | Llama and Mistral · PG-19 and The Stack code sample with 16,384-token prompts | perplexity | Helped |
| Use Radar instead of H2O | Mistral · PG-19 without prompts | perplexity: 0.05 | Helped |
Surprising findings
- H2O and SnapKV degrade severely on Mistral models in the no-prompt generation and extended-generation settings, while Radar stays stable.
- StreamingLLM with the same budget is often the worst method on LongBench, and Landmark attention underperforms vanilla on most non-QA tasks.
Limitations
- Radar does not reduce runtime memory and uses extra O(sqrt(t)) memory for segment representations.
- Segments are contiguous and the segmenting strategy is fixed; non-contiguous or adaptive segmenting is left to future work.
- Approximation relies on random features with finite projection dimension n, so top-segment recovery is only probabilistic.
Experimental setup
Datasets: PG-19 (first sample) The Stack (NASA CDF Python sample) LongBench (16 English and code subtasks)
Benchmarks: PG-19 LongBench
Metrics: perplexity elapsed time / generation throughput average score average percentile
Baselines: vanilla attention StreamingLLM H2O SnapKV Landmark attention SubGen
Builds on: attention kernelization (random features / performer) StreamingLLM H2O SnapKV
Closest papers by meaning
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.85)
- Predictive Geometry of Hidden Trajectories in Transformers (NeurIPS 2026 · similarity 0.84)
- InfoFlow: A Framework for Multi-Layer Transformer Analysis (NeurIPS 2026 · similarity 0.84)
- TPP-SD: Accelerating Transformer Point Process Sampling with Speculative Decoding (NeurIPS 2025 · similarity 0.84)
- Two Heads are Better than One: Simulating Large Transformers with Small Ones (NeurIPS 2025 · similarity 0.84)
- IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.