Research map / Architectures and efficiency / RNNs and normalization
Mesh-TensorFlow: Deep Learning for Supercomputers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale the Transformer by increasing feed-forward width and attention heads (ablation) | Mesh-TensorFlow Transformer · One Billion Word | word perplexity: 35.0 -> 24.0 | Helped |
| Scale the Transformer by increasing feed-forward width and attention heads (ablation) | Mesh-TensorFlow Transformer · Wikipedia | subword perplexity: 8.74 -> 6.01 | Helped |
| Scale the Transformer by increasing feed-forward width and attention heads (ablation) | Mesh-TensorFlow Transformer · WMT14 EN-FR | BLEU: 41.8 -> 43.9 | Helped |
| Scale the Transformer by increasing feed-forward width and attention heads (ablation) | Mesh-TensorFlow Transformer · WMT14 EN-DE | BLEU: 25.5 -> 27.5; 27.5 -> 26.7 | Mixed |
| Evaluate One Billion Word logits multiplied by 0.9 (ablation) | 4.90B Mesh-TensorFlow Transformer · One Billion Word | word perplexity: 24.0 -> 23.5 | Helped |
| Use a 2D TPU mesh with data parallelism across rows and model parallelism across columns | Mesh-TensorFlow Transformer · Transformer training on TPUv2 | computational efficiency: over 50% | Helped |
Closest papers by meaning
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.88)
- SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization (ICML 2017 · similarity 0.88)
- Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.87)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.87)
- DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing (AAAI 2024 · similarity 0.86)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.