Research map / Optimization / Optimizers
Seesaw: Accelerating Training by Balancing Batch Size and Learning Rate Scheduling
Alexandru Meterez, Depen Morwani, Jingfeng Wu, Costin-Andrei Oncescu, Cengiz Pehlevan, Sham M. Kakade
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace cosine decay with Seesaw (cut LR by 1/sqrt2 and double batch size) | cosine decay, 150M-600M · C4 Chinchilla scale, batch size at or below CBS | serial wall-clock time: reduces by ≈36% | Improved |
| replace cosine decay with Seesaw | cosine decay, 150M · C4 Chinchilla, B=256 | final validation loss: 3.0353 -> 3.0346 | No clear effect |
| replace cosine decay with Seesaw | cosine decay, 300M · C4 Chinchilla, B=256 | final validation loss: 2.8591 -> 2.8561 | No clear effect |
| replace cosine decay with Seesaw | cosine decay, 600M · C4 Chinchilla, B=256 | final validation loss: 2.6904 -> 2.6883 | No clear effect |
| replace cosine decay with Seesaw at batch size well above critical | cosine decay, 150M · C4 Chinchilla, B=1024 | final validation loss: 3.1214 -> 3.1318 | Hurt |
| replace cosine decay with Seesaw at batch size well above critical | cosine decay, 300M · C4 Chinchilla, B=1024 | final validation loss: 2.9369 -> 2.9490 | Hurt |
| use more aggressive ramp (alpha=1, beta=4) instead of alpha=sqrt2, beta=2 along alpha*sqrt(beta)=2 (ablation) | Seesaw · 150M, B=256 and 512 | validation loss | Hurt |
| double batch size at fixed learning rate instead of Seesaw (ablation) | Seesaw · 150M, B=128 and 256 | validation loss | Hurt |
| constant LR with batch ramp instead of Seesaw past CBS (ablation) | Seesaw · 150M, B=1024 to 8192 | validation loss | Hurt |
| enable z-loss (ablation) | cosine decay · 150M C4, Chinchilla | final validation loss | No clear effect |
| add weight decay 0.0001 (lr 0.003) | Seesaw vs cosine · 150M C4, B=128 | final validation loss: 3.0125 -> 3.0027 | No clear effect |
| train 4x Chinchilla (overtrained) with Seesaw instead of cosine | cosine decay · 150M C4, B=512 | final validation loss: 2.8814 -> 2.8820 | No clear effect |
Closest papers by meaning
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.92)
- Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Training (NeurIPS 2025 · similarity 0.89)
- Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.88)
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.88)
- Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixtures (ICLR 2025 · similarity 0.87)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
