Research map / Architectures and efficiency / Attention and transformers
Block-Recurrent Transformers
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, Behnam Neyshabur
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add one block-recurrent layer instead of a 13th non-recurrent layer | Slide:13L · PG19 test (tokens) | bits per token: 3.58 -> 3.53 | Improved |
| add block recurrence instead of a 13th layer on arXiv | Slide:13L · arXiv | bits per token: 1.42 -> 1.24 | Improved |
| use fixed gate instead of LSTM gate (skip config vs lstm dual) (ablation) | Block-Recurrent Transformer · arXiv | bits per token: 1.26 -> 1.24 | Improved |
| increase recurrent states from 512 to 1024 (ablation) | Rec:fixed:skip · PG19 test (tokens) | bits per token: 3.53 -> 3.51 | Improved |
| increase recurrent states from 1024 to 2048 (ablation) | Rec:fixed:skip · PG19 test (tokens) | bits per token: 3.51 -> 3.55 | Hurt |
| add a second recurrent layer (adjacent layers 9 and 10) (ablation) | Block-Recurrent Transformer · PG19 | perplexity | No clear effect |
| add block feedback (all layers cross-attend to recurrent states) (ablation) | Rec:fixed:skip · PG19 test (tokens) | bits per token: 3.53 -> 3.49 | Improved |
| shrink window size from 512 to 128 (ablation) | Rec:fixed:skip · PG19 test (tokens) | bits per token: 3.53 -> 3.58 | Hurt |
| use recurrence over larger window XL:2048 vs recurrence | XL:2048 · GitHub validation | bits per token: 1.01 -> 0.976 | Improved |
| add recurrence at 1.3B scale | 26-layer Transformer-XL · PG19 test (tokens) | bits per token: 3.31 -> 3.22 | Improved |
| train sliding window over 4096 segment (differentiable) instead of Transformer-XL 512 | XL:512 · PG19 test (tokens) | bits per token: 3.62 -> 3.60 | Improved |
Closest papers by meaning
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.89)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.88)
- Block Transformer: Global-to-Local Language Modeling for Fast Inference (NeurIPS 2024 · similarity 0.87)
- Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.87)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.87)
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
