Research map / Optimization / Optimizers
MARS: Unleashing the Power of Variance Reduction for Training Large Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MARS, a unified preconditioned variance-reduction framework that uses scaled STORM-style gradient correction with clipping, instantiated as MARS-AdamW, MARS-Lion and MARS-Shampoo, and shows improved GPT-2 pretraining over AdamW, Lion and Muon.
Problem
Variance reduction has rarely helped training deep networks or LLMs, and adaptive optimizers like AdamW plateau in large-model pretraining; the paper asks whether variance reduction can accelerate large model training.
Approach
Scale the STORM recursive-momentum gradient correction by a factor gamma_t, clip the resulting gradient estimator by norm, and feed it into preconditioned (AdamW-, Lion- or Shampoo-style) updates, so that the variance-reduction strength is tunable and the framework subsumes AdamW/Lion/Muon-like updates.
Key results
- On GPT-2 large, MARS-AdamW reaches a best validation loss of 2.511 versus 2.568 for AdamW; AdamW needs 50B tokens to hit 2.58 while MARS needs 28B.
- On the 5-shot average over nine benchmarks for GPT-2 large, MARS-AdamW scores 50.15 versus 48.64 for AdamW.
- MARS-AdamW reaches 45.66 on HellaSwag versus 42.55 for AdamW in the 5-shot table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add scaled variance-reduced updates to AdamW | GPT-2 large · OpenWebText validation | loss: 2.568 -> 2.511 | Helped |
| Add scaled variance-reduced updates to Lion | GPT-2 large · OpenWebText validation | loss: 2.565 -> 2.534 | Helped |
| Add scaled variance-reduced updates to AdamW | GPT-2 large · HellaSwag, 5-shot | accuracy: 42.55 -> 45.66 | Helped |
| Add scaled variance-reduced updates to Lion | GPT-2 large · HellaSwag, 5-shot | accuracy: 43.41 -> 44.00 | Helped |
| Add scaled variance-reduced updates to AdamW | GPT-2 large · Average, 5-shot | accuracy: 48.64 -> 50.15 | Helped |
| Add scaled variance-reduced updates to Lion | GPT-2 large · Average, 5-shot | accuracy: 48.61 -> 49.80 | Helped |
| Use MARS-approx instead of MARS-exact (ablation) | MARS · GPT-2 training experiments | performance | No clear effect |
| Use flexible variance-reduction strength instead of fixing gamma at 1 (ablation) | MARS · Optimization convergence | convergence | Helped |
Surprising findings
- Muon underperforms AdamW on the 5-shot downstream average (47.31 vs 48.64) despite its reported convergence speed.
- Lion with the variance-reduced correction (MARS-Lion) is only marginally better than MARS-AdamW on some metrics and the approximate MARS variant performs about as well as the exact one.
Limitations
- Each step requires two gradient computations per iteration in the exact version, adding cost; the approximate variant reuses the previous batch to avoid this.
- Theoretical convergence rate O(log T / T^{1/3}) relies on assumptions (bounded variance, smoothness, lower-bounded preconditioner) and a time-varying beta schedule that differs from practice.
Experimental setup
Datasets: OpenWebText
Benchmarks: ARC-E ARC-C BoolQ HellaSwag OBQA PIQA WinoGrande MMLU SciQ
Metrics: validation loss accuracy tokens to target loss wall-clock time
Baselines: AdamW Lion Muon SuperAdam Adan Sophia
Builds on: STORM AdamW Lion Shampoo SPIDER
Closest papers by meaning
- Understanding MARS: When Scaling Momentum Provably Helps (ICML 2026 · similarity 0.89)
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.88)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.87)
- ASGO: Adaptive Structured Gradient Optimization (NeurIPS 2025 · similarity 0.86)
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.86)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.