Research map / Reinforcement learning / Visual and representation RL
Accelerating neural network training: An analysis of the AlgoPerf competition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reports results of the inaugural AlgoPerf: Training Algorithms competition, scoring 18 submissions from 10 teams on time-to-result across deep learning workloads under external-tuning and self-tuning rulesets, with engineering lessons on fair JAX/PyTorch comparison.
Problem
Empirical comparisons of neural network training algorithms have been unfair (weak baselines, incomplete hyperparameter tuning, confounding architecture changes), so the state of the art in training algorithms is unclear.
Approach
A competitive, time-to-result benchmark on fixed hardware where submissions must either supply workload-agnostic hyperparameter search spaces (external tuning) or be fully hyperparameter-free (self-tuning), scored with performance profiles across fixed base and held-out workloads.
Key results
- External-tuning winner PyTorch Distributed Shampoo achieved about 28% average speed-up over the NadamW baseline across eight base workloads (benchmark score 0.78).
- Self-tuning winner Schedule Free AdamW achieved about 8% average speed-up over the self-tuning baseline and scored 0.85 in the self-tuning ruleset.
- Across 18 submissions and about 4000 runs, five different submissions were fastest on at least one of the eight base workloads, and no single submission dominated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| non-diagonal preconditioning (Distributed Shampoo) instead of NAdamW baseline | external tuning ruleset · 8 base workloads | speed-up vs baseline: +27.87% vs baseline | Helped |
| hyperparameter-free Schedule Free AdamW instead of self-tuning baseline | self-tuning ruleset · 8 base workloads | speed-up vs baseline: +7.76% vs self-tuning baseline | Helped |
| self-tuning Schedule Free AdamW vs external tuning baseline | self-tuning ruleset · 7 base workloads both trained | training time relative to external baseline: approx 10% faster | Helped |
| removing held-out workloads from scoring (ablation) | external tuning ruleset · benchmark score ranking | ranking: only CASPR Adaptive and Amos swap positions | No clear effect |
| reducing number of tuning trials from 20 to 5 in external tuning | AlgoPerf benchmark protocol · competition configuration | compute cost | No clear effect |
| training on the PyTorch 2.0 stack with torch.compile adoption (vs eager PyTorch) (ablation) | PyTorch workload implementations · JAX vs PyTorch wall-clock gap | performance gap: gap could be as large as 60% without these adjustments | Helped |
| Schedule Free AdamW self-tuning submission evaluated under external tuning rules | external tuning ruleset · 8 base workloads | benchmark score: 0.4804 vs 0.71 for external Schedule Free AdamW | Hurt |
Surprising findings
- The self-tuning Schedule Free AdamW would have ranked eighth under the external tuning ruleset, and was about 10% faster than the external-tuning baseline on seven shared workloads.
- Only one submission (Generalized Adam) reliably trained the ResNet workload, even though its target was set with the same procedure as other workloads.
Limitations
- Held-out workloads add substantial compute and engineering cost while providing limited measured benefit.
- Infinite scores for failed targets make average speed-ups weakly meaningful, since the penalty for missing a target is relatively weak.
- Scoring is limited to validation targets on fixed hardware (8x NVIDIA V100) and a limited set of workloads.
Experimental setup
Datasets: Criteo 1TB fastMRI ImageNet LibriSpeech OGBG WMT
Benchmarks: AlgoPerf benchmark (8 fixed base workloads, 6 held-out workloads)
Metrics: time-to-result benchmark score via performance profiles geometric mean speed-up over baseline workload runtime normalized by budget
Baselines: NadamW baseline (external tuning) NadamW baseline (self-tuning) Generalized Adam CASPR Adaptive Cyclic LR LAWA EMA LAWA Queue NadamP Amos
Builds on: AlgoPerf benchmark Distributed Shampoo Schedule-Free AdamW AdamW performance profiles
Closest papers by meaning
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.87)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.86)
- A Stable Whitening Optimizer for Efficient Neural Network Training (NeurIPS 2025 · similarity 0.85)
- When, Where and Why to Average Weights? (ICML 2025 · similarity 0.85)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.85)
- Principled Architecture-aware Scaling of Hyperparameters (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.