Research map / Architectures and efficiency / Attention and transformers
RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAT, a chunk-based layer that applies intra-chunk recurrence and inter-chunk softmax attention, and a hybrid with sliding-window attention, evaluated by pretraining 1.3B models on FineWeb-Edu.
Problem
Softmax attention is compute-costly for long sequences, while RNNs compress the full history into a fixed-size state that degrades in long contexts and limits precise retrieval.
Approach
Partition the sequence into chunks; apply gated linear recurrence within each chunk for local dependencies and softmax attention over chunk-level keys/values for long-range access. Chunk size interpolates between attention (L=1) and RNN (L=T), and the layer is combined with sliding-window attention in a hybrid.
Key results
- RAT with chunk size 16 achieves 7x training speedup at 100K sequence length and 9x generation speedup at position 4K for a single temporal-mixing block versus attention, with pretraining perplexity 7.67 vs 7.61 for attention.
- Hybrid RAT-SWA (L=16) reaches the best commonsense accuracy (58.0 avg vs 57.1 for Attention-SWA) and 28.2 on code completion vs 21.7 for Attention-SWA on LongBench.
- Maximum throughput of the 1.3B model reaches 31170 tokens/sec versus 3052 for attention (about 10x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace full attention with RAT-16 | 1.3B model · LongBench QA | Avg. F1: 18.2 -> 19.6 | Helped |
| Replace full attention with RAT-16 | 1.3B model · LongBench code completion | Avg. EditSum: 23.9 -> 17.4 | Hurt |
| Interleave RAT-16 with sliding-window attention | 1.3B RAT-16 model · LongBench code completion | Avg. EditSum: 21.7 -> 28.2 | Helped |
| Interleave RAT-16 with sliding-window attention | 1.3B RAT-16 model · Commonsense reasoning | Avg. accuracy: 56.7 -> 58.0 | Helped |
| Interleave RAT-16 with sliding-window attention | 1.3B RAT-16 model · NarrativeQA summary-only SFT | F1: 60.8 -> 63.2 | No clear effect |
| Use intra-chunk attention and inter-chunk recurrence instead of the standard RAT arrangement (ablation) | RAT-64 · PG19 pretraining, T=16384 | PPL: 13.34 -> 14.06 | Hurt |
| Combine intra-chunk and inter-chunk outputs by addition instead of hierarchical organization (ablation) | RAT-64 · PG19 pretraining, T=16384 | PPL: 13.34 -> 14.4 | Hurt |
| Replace standard RAT with dilated attention (ablation) | RAT-128 · PG19 pretraining | PPL: 13.44 -> 15.22 | Hurt |
| Allocate more parameters to intra-chunk RNN gates rather than inter-chunk query/key projections (ablation) | RAT · PG19 book pretraining | PPL: 0.4–0.5 | Helped |
| Use chunk-indexed cross-chunk RoPE instead of original RoPE (ablation) | RAT · PG19 book pretraining | PPL: 0.3 | Helped |
| Increase chunk size from 64 to 256 (ablation) | RAT · 200M pretraining | PPL: 13.42 -> 13.72 | Hurt |
| Use NoPE instead of RoPE (ablation) | RAT-16-SWA · FineWeb-Edu validation | PPL: 7.57 -> 7.63 | Hurt |
Surprising findings
- RNN is within about two points of attention on short commonsense reasoning tasks despite compressing the full sequence.
- Reversing the design (attention within chunks, recurrence across chunks) substantially underperforms RAT at matched FLOPs.
Limitations
- Experiments limited to models up to 1.3B parameters; not scaled to 7B or 14B.
- No SFT techniques from industry practice; evaluation focuses on pretraining and train-test splits.
- Length generalization issues remain from positional encoding; none of the variants match RNN stability.
Experimental setup
Datasets: FineWeb-Edu PG19 NarrativeQA QMSum WikiSum RULER
Benchmarks: PIQA ARC-C ARC-E HellaSwag LAMBADA WinoGrande BoolQ LongBench NarrativeQA QMSum WikiSum RULER
Metrics: accuracy perplexity F1 Rouge-L EditSim/EditSum exact match tokens/sec latency
Baselines: Attention RNN Attention-SWA Mamba2 DeltaNet GatedDeltaNet FLASH RNN-Dilated attention
Builds on: flash attention RoPE FLASH dilated attention Mamba2
Closest papers by meaning
- Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access (NeurIPS 2025 · similarity 0.89)
- Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions (ICLR 2025 · similarity 0.89)
- Simple linear attention language models balance the recall-throughput tradeoff (ICML 2024 · similarity 0.89)
- Star Attention: Efficient LLM Inference over Long Sequences (ICML 2025 · similarity 0.89)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.88)
- RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.