Research map / Optimization / Nonconvex and smooth optimization

EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization

NeurIPS 2026 · Optimization › Nonconvex and smooth optimization · approach: Acceleration and Nesterov dynamics · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes EMA-Nesterov, a single-loop Nesterov-style lookahead that extrapolates along an EMA of parameter updates instead of the last one-step difference, wrapping optimizers such as Adam, SOAP, and Muon for LLM pre-training with convergence guarantees on convex problems.

Problem

Classical Nesterov lookahead uses one-step iterate differences, which are noisy and unstable in deep learning training; prior long-horizon lookahead methods are non-adaptive to changing trajectory trends.

Approach

Replace the Nesterov lookahead direction with an exponential moving average of parameter updates (a low-pass filter over the trajectory), with a lookahead step-size schedule, retaining accelerated rates in convex analysis.

Key results

• EMA-Nesterov + Muon on Llama 1B reduces validation perplexity from 12.22 to 12.12 over 5000 iterations.
• On NanoGPT, the method gives about 6% speedup for Muon in reaching the 3.28 validation loss target.
• Across optimizers, perplexity drops range from 0.208 (Muon) to 1.305 (Adam) on NanoGPT.

Surprising findings

• The Muon implementation's 'Nesterov' momentum is actually an EMA of stochastic gradients with no connection to Nesterov lookahead.

Limitations

• Convergence analysis assumes convex, smooth objectives while deep learning is non-convex.
• Requires a lookahead step-size schedule with warm-up and early rest stages tuned to the learning rate schedule.
• Hyperparameters (gamma, max beta) differ across model sizes and batch sizes.

Experimental setup

Datasets: FineWeb C4-en

Benchmarks: NanoGPT FineWeb validation Llama C4 pre-training validation

Metrics: validation perplexity validation loss iterations to reach target loss

Baselines: Muon Adam SOAP NorMuon Aurora Pessimistic lookahead SNOO GPA

Builds on: Nesterov's accelerated gradient Lookahead SNOO GPA Muon

Closest papers by meaning

• Nesterov acceleration in benignly non-convex landscapes (ICLR 2025 · similarity 0.88)
• Nesterov acceleration despite very noisy gradients (NeurIPS 2024 · similarity 0.87)
• Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.86)
• MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.85)
• Improving Sharpness-Aware Minimization by Lookahead (ICML 2024 · similarity 0.85)
• Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning (NeurIPS 2026 · similarity 0.85)