Research map / Architectures and efficiency / Attention and transformers
Reformer: The Efficient Transformer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LSH attention with more hashing rounds (ablation) | 1-layer Transformer · Duplication task | accuracy: 52.5% -> 94.8% | Helped |
| Train from scratch with LSH-4 rather than full attention (ablation) | 1-layer Transformer · Duplication task | accuracy: 100% -> 99.9% | Hurt |
| Evaluate LSH-trained models with LSH-8 (ablation) | 1-layer Transformer · Duplication task | accuracy: 91.9% -> 100% | Helped |
| Replace full attention with shared-QK attention (ablation) | Transformer · enwik8 and imagenet64 | bits per dim | No clear effect |
| Replace standard residual layers with reversible layers (ablation) | Transformer · enwik8 and imagenet64 | bits per dim | No clear effect |
| Split feed-forward computation into chunks (ablation) | Reformer · Transformer feed-forward layers | numerical equivalence | No clear effect |
| Train longer with further tuning and improvements | 12-layer Reformer · enwik8 test | bits/dim: 1.19 -> 1.05 | Helped |
Closest papers by meaning
- Two Heads are Better than One: Simulating Large Transformers with Small Ones (NeurIPS 2025 · similarity 0.90)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.88)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.88)
- CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs (NeurIPS 2026 · similarity 0.88)
- MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections (ICML 2025 · similarity 0.88)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.