Research map / Architectures and efficiency / Attention and transformers
Character-Level Language Modeling with Deeper Self-Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase transformer depth from 12 to 64 layers | Character Transformer · text8 test | bpc: 1.18 -> 1.13 | Helped |
| Increase context length from 32 to 512 characters | T64 · text8 test | bpc: 1.34 -> 1.13 | Helped |
| Add Multiple Positions auxiliary loss (ablation) | T64 · text8 dev, context 512 | bpc: 2.482 -> 1.062 | Helped |
| Add Intermediate Layer Losses (ablation) | T64 · text8 dev, context 512 | bpc: 1.158 -> 1.062 | Helped |
| Add learned per-layer Positional Embeddings instead of sinusoidal timing signal (ablation) | T64 · text8 dev, context 512 | bpc: 1.069 -> 1.062 | Helped |
| Add Multiple Targets auxiliary loss (ablation) | T64 · text8 dev, context 512 | bpc: 1.068 -> 1.062 | Helped |
| Replace momentum optimizer with SGD (ablation) | T64 · text8 dev, context 512 | bpc: 1.062 -> 1.065 | Hurt |
Closest papers by meaning
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.88)
- RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling (NeurIPS 2025 · similarity 0.88)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.88)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.