Research map / Architectures and efficiency / RNNs and normalization
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Layerstack within a fixed 6h or 12h training budget | T5-Base pre-training · SNI | ROUGE-L: 34.2 ± 0.2 -> 38.9 ± 0.2; 38.3 ± 0.4 -> 39.2 ± 0.1 | Helped |
| Use Layerstack at a 24h budget | T5-Base pre-training · SNI | ROUGE-L: 39.5 ± 0.1 -> 38.9 ± 0.2 | Hurt |
| Use Layerdrop instead of the baseline | T5-Base pre-training · SNI | ROUGE-L: 34.2 ± 0.2 -> 31.5 ± 0.3; 38.3 ± 0.4 -> 34.4 ± 0.8; 39.5 ± 0.1 -> 38.0 ± 0.5 | Hurt |
| Use batch selection with its selection computation charged to the budget | BERT-Base-like pre-training · C4 validation | validation loss: Baseline 2.42 / 2.20 / 2.10; Selective Backprop 2.73 / 2.44 / 2.18; RHO Loss 2.61 / 2.37 / 2.16 | Hurt |
| Ignore batch-selection costs and train for the same number of iterations (ablation) | RHO Loss · C4, 12h, GLUE | GLUE score: 77.79 ± 0.2 -> 77.16 ± 0.4 | Hurt |
| Ignore batch-selection costs and train for the same number of iterations (ablation) | Selective Backprop · C4, 12h, GLUE | GLUE score: 77.79 ± 0.2 -> 77.92 ± 0.1 | No clear effect |
| Do not count Sophia Hessian update steps and run the same number of iterations (ablation) | Sophia · BERT validation | validation loss | No clear effect |
| Increase the Selective Backprop selectivity scale β (ablation) | Selective Backprop · pre-training validation | validation loss | Hurt |
| Increase the RHO Loss mega-batch size (ablation) | RHO Loss · pre-training validation | validation loss | Hurt |
| Use Layerdrop when the baseline overfits on a truncated, repeatedly seen dataset (ablation) | BERT pre-training · small dataset, roughly 180 epochs | validation loss | Helped |
Closest papers by meaning
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.89)
- Progressive Residual Warmup for Language Model Pretraining (NeurIPS 2026 · similarity 0.88)
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.88)
- Memory-Augmenting Decoder-Only Language Models through Encoders (Student Abstract) (AAAI 2024 · similarity 0.88)
- Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling (ICML 2025 · similarity 0.87)
- The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.