Research map / Architectures and efficiency / Attention and transformers
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rodimus, a purely recurrent linear-attention model with data-dependent tempered selection (DDTS), and Rodimus+, which hybridizes it with sliding-window shared-key attention (SW-SKA), achieving strong accuracy-efficiency trade-offs on language modeling and recall benchmarks.
Problem
Softmax attention has O(T) per-token generation cost, while existing linear/recurrent models lose accuracy relative to softmax attention; the paper asks whether LLM complexity can be reduced without sacrificing performance.
Approach
A linear-attention recurrence whose forget/input gates are data-dependent and tempered by an input-dependent temperature gate (DDTS), combined with a sliding-window attention that shares keys across heads (SKA), so that semantic, token, and head compression are combined in one hybrid block.
Key results
- Rodimus+-1.6B trained on 1T tokens achieves 58.61 average accuracy on downstream tasks, above Qwen2-1.5B (58.30, 7T tokens) and RWKV6-1.6B (56.31, 1.42T tokens).
- At 130M scale trained on 300B Pile tokens, Rodimus achieves 42.88 average and Rodimus+ 43.15, versus Mamba2 at 42.65.
- On WikiText-103 at ~44M params, Rodimus+ reaches lower perplexity than Transformer++ and Llama3 with a smaller hidden state (expansion factor 32 vs Mamba2's 128).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Rodimus instead of Mamba2 | 130M models trained on 100B tokens · Downstream zero-shot tasks | AVG acc: 41.22 -> 41.96 | Helped |
| Add SW-SKA to Rodimus | 130M models trained on 300B tokens · Downstream zero-shot tasks | AVG acc: 42.88 -> 43.15 | Helped |
| Add SW-SKA to Rodimus | Pile language model · Pile validation | perplexity: 5.88 -> 5.88 | No clear effect |
| Use Rodimus+ rather than Qwen2 | 1.6B Rodimus+ and 1.5B Qwen2 · Downstream zero-shot tasks | AVG acc: 58.30 -> 58.61 | Helped |
| Include the temperature gate in DDTS (ablation) | Rodimus · Language modeling | validation perplexity | Helped |
| Include the selection gate and input gate in DDTS (ablation) | Rodimus · Language modeling | validation perplexity | Helped |
| Include hat-vbeta_t in DDTS (ablation) | Rodimus · Language modeling | validation perplexity | Helped |
| Include vbeta_t in the state transition (ablation) | Rodimus · Training | training performance | Hurt |
| Use a two-hop residual connection (ablation) | Rodimus+ · Downstream tasks | performance | Helped |
| Use SKA instead of GQA (ablation) | Attention head compression · Benchmarks across backbone models | performance | Helped |
| Use temperature gate tau_t as an input-dependent gate rather than a fixed constant (ablation) | Rodimus DDTS · Historical information filtering | filtering of irrelevant information | Helped |
Surprising findings
- On the Pile 350M 7B-token experiment, Rodimus and Rodimus+ show similar perplexity (5.88 each), contrary to the WikiText-103 trend where Rodimus+ is clearly better, which the authors attribute to limited data scale.
- Rodimus with expansion factor 64 has half the state size of Mamba2 yet outperforms it on various benchmarks.
Limitations
- Rodimus+ is harder to train and needs more data to outperform Rodimus at small scale (similar PPL on the 350M Pile experiment).
- Comparisons use different training corpora and token counts for several baselines, so comparisons are not fully controlled.
Experimental setup
Datasets: WikiText-103 The Pile FineWeb curated pretraining corpus
Benchmarks: WikiText-103 LAMBADA PIQA HellaSwag WinoGrande OpenBookQA ARC-Easy ARC-Challenge MQAR NeedleBench
Metrics: perplexity accuracy acc_norm
Baselines: Mamba Mamba2 GLA HGRN2 RWKV6 Transformer++ Llama3 Qwen2 Pythia Recurrent-Gemma
Builds on: Mamba2 gated linear attention Griffin multi-query attention grouped-query attention
Closest papers by meaning
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.89)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.89)
- ArkVale: Efficient Generative LLM Inference with Recallable Key-Value Eviction (NeurIPS 2024 · similarity 0.89)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.89)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.89)
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.