AI Paper Atlas

Research map / Optimization / Optimizers

Seesaw: Accelerating Training by Balancing Batch Size and Learning Rate Scheduling

Alexandru Meterez, Depen Morwani, Jingfeng Wu, Costin-Andrei Oncescu, Cengiz Pehlevan, Sham M. Kakade

ICLR 2026 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace cosine decay with Seesaw (cut LR by 1/sqrt2 and double batch size)cosine decay, 150M-600M · C4 Chinchilla scale, batch size at or below CBSserial wall-clock time: reduces by ≈36%Improved
replace cosine decay with Seesawcosine decay, 150M · C4 Chinchilla, B=256final validation loss: 3.0353 -> 3.0346No clear effect
replace cosine decay with Seesawcosine decay, 300M · C4 Chinchilla, B=256final validation loss: 2.8591 -> 2.8561No clear effect
replace cosine decay with Seesawcosine decay, 600M · C4 Chinchilla, B=256final validation loss: 2.6904 -> 2.6883No clear effect
replace cosine decay with Seesaw at batch size well above criticalcosine decay, 150M · C4 Chinchilla, B=1024final validation loss: 3.1214 -> 3.1318Hurt
replace cosine decay with Seesaw at batch size well above criticalcosine decay, 300M · C4 Chinchilla, B=1024final validation loss: 2.9369 -> 2.9490Hurt
use more aggressive ramp (alpha=1, beta=4) instead of alpha=sqrt2, beta=2 along alpha*sqrt(beta)=2 (ablation)Seesaw · 150M, B=256 and 512validation lossHurt
double batch size at fixed learning rate instead of Seesaw (ablation)Seesaw · 150M, B=128 and 256validation lossHurt
constant LR with batch ramp instead of Seesaw past CBS (ablation)Seesaw · 150M, B=1024 to 8192validation lossHurt
enable z-loss (ablation)cosine decay · 150M C4, Chinchillafinal validation lossNo clear effect
add weight decay 0.0001 (lr 0.003)Seesaw vs cosine · 150M C4, B=128final validation loss: 3.0125 -> 3.0027No clear effect
train 4x Chinchilla (overtrained) with Seesaw instead of cosinecosine decay · 150M C4, B=512final validation loss: 2.8814 -> 2.8820No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy