Research map / Architectures and efficiency / Attention and transformers
Universal Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add recurrent self-attentive refinement across depth | Transformer · Copy, Reverse, and Addition length 400 | char-acc: Copy: 0.53 -> 0.91; Reverse: 0.13 -> 0.96 | Helped |
| Add recurrent self-attentive refinement across depth | Transformer · WMT14 En-De | BLEU: 28.0 -> 28.9 | Helped |
| Add per-position dynamic halting (ablation) | Universal Transformer · bAbI 10K train joint | average error: 0.47 -> 0.29 | Helped |
| Add per-position dynamic halting (ablation) | Universal Transformer · bAbI 1K train joint | average error: 8.50 -> 7.78 | Helped |
| Add per-position dynamic halting (ablation) | Universal Transformer · Subject-verb agreement, 3 attractors | accuracy: 0.940 -> 0.944 | Helped |
| Add per-position dynamic halting (ablation) | Universal Transformer · LAMBADA LM test | perplexity: 319 (0.17) -> 142 (0.19) | Helped |
| Use per-position dynamic halting instead of fixed-depth refinement (ablation) | Universal Transformer · WMT14 En-De | BLEU | Hurt |
| Replace dynamic halting with fixed 8-step depth (ablation) | Universal Transformer · LAMBADA LM test | perplexity: 142 -> 202 | Hurt |
| Replace dynamic halting with fixed 9-step depth (ablation) | Universal Transformer · LAMBADA LM test | perplexity: 142 -> 239 | Hurt |
| Use dynamic halting rather than fixed 6-step depth (ablation) | Universal Transformer · LAMBADA RC test | accuracy: 0.5216 -> 0.5625 | Mixed |
| Use recurrent UT rather than Transformer | Transformer · Subject-verb agreement | accuracy: 0.962 -> 0.992 | Helped |
| Use recurrent UT rather than Transformer | Transformer · LAMBADA RC test | accuracy: 0.3988 -> 0.5625 | Helped |
Closest papers by meaning
- FlashRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models (ICLR 2026 · similarity 0.90)
- On the "Induction Bias" in Sequence Models (ICML 2026 · similarity 0.89)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.88)
- When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspective (NeurIPS 2025 · similarity 0.87)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
- Implicit Language Models are RNNs: Balancing Parallelization and Expressivity (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.