AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Fantastic Pretraining Optimizers and Where to Find Them

Kaiyue Wen, David Leo Wright Hall, Tengyu Ma, Percy Liang

ICLR 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning · 90 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase AdamW warmup from 1000 to 2000 steps (ablation)AdamW · 130M 1x Chinchilla C4 val lossvalidation loss: 3.298 -> 3.282Improved
decrease AdamW batch size from 256 to 128 (ablation)AdamW · 130M 1x Chinchilla C4 val lossvalidation loss: 3.282 -> 3.263Improved
increase AdamW max grad norm from 1 to 2 (ablation)AdamW · 130M 1x Chinchilla C4 val lossvalidation loss: 3.263 -> 3.263No clear effect
tune the learning rate only in the GPT-3 recipe for AdamW (ablation)AdamW · 100M model pretrainingspeedup: nearly 2x speedupImproved
use Muon instead of MarsMars · C4/EN val loss, 130M 1x Chinchillavalidation loss: 3.537 -> 3.464Improved
use Muon instead of LionLion · C4/EN val loss, 520M 1x Chinchillavalidation loss: 3.108 -> 3.073Improved
use Kron or Muon instead of Mars at 8x ChinchillaMars · C4/EN val loss, 520M 8x Chinchillavalidation loss: 2.906 -> 2.900Improved
use Kron instead of Muon as data-to-model ratio rises to 8xMuon · C4/EN val loss, 130Mvalidation loss: 3.464 -> 3.492 (1x); 3.240 -> 3.239 (8x)Mixed
use Soap or NAdamW instead of Muon at 16x ChinchillaMuon · 130M and 300M models, 16x Chinchillavalidation lossImproved
use Muon instead of AdamW at 1.2B, 8x ChinchillaAdamW · 10-task average, 1.2Baverage accuracy: 67.15 -> 66.98No clear effect
use NAdamW instead of AdamW at 1.2B, 8x ChinchillaAdamW · 10-task average, 1.2Baverage accuracy: 67.15 -> 66.70Hurt
scale model from 0.1B to 1.2B for matrix-based optimizersMuon and Soap · speedup over AdamWspeedup: 1.4x -> 1.1xHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy