Research map / Architectures and efficiency / Pruning and efficient tuning

Optimistic Dual Averaging Unifies Modern Optimizers

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces SODA, a generalized optimistic dual averaging framework that unifies Muon, Lion, NAdam and others, and a weight-decay-free wrapper using a 1/k decay schedule anchored at initialization.

Problem

Weight decay is hard to tune per optimizer and its horizon/model-size dependence is unclear; existing optimizers are treated as ad hoc combinations of momentum, averaging, and geometry.

Approach

A single optimistic dual averaging template with both dual (gradient averaging and optimism) and primal (iterate averaging and extrapolation) processing; in the modernized regime, the averaging parameter lambda_k=1/(k+2) acts as an anchored, scheduled weight decay that removes the need to tune weight decay.

Key results

• SODA(Muon) outperforms Muon with swept weight decay on 124M NanoGPT at both 1x and 4x Chinchilla without extra tuning.
• SODA-dagger (optimistic variant with Lion-like setup, alpha=bar-alpha=0.05) gives best validation loss, e.g. 3.26596 at 5k steps versus 3.27613 for the no-optimism variant.
• SODA-dagger is competitive with or better than Scion/uScion across model sizes from 64M to 1B under Chinchilla scaling, with the gap growing with scale.

Surprising findings

• The optimal weight decay for Muon decreases with training horizon, contradicting the constant-in-horizon 1/d rule.
• Setting the anchor z^0 to zero instead of the initialization degrades performance.

Limitations

• Theory is restricted to convex functions and single-epoch training; multi-epoch may need a regularization hyperparameter.
• The wrapper as implemented needs extra memory to recover the update delta, unless the base optimizer is modified.
• Convergence analysis relies on strong convexity of the regularizer, handled via a tau-scaled surrogate.

Experimental setup

Datasets: FineWeb100

Benchmarks: FineWeb100 validation loss

Metrics: validation loss

Baselines: Muon AdamW Scion uScion Lion GPA

Builds on: Optimistic Dual Averaging Schedule-Free Generalized Primal Averaging Muon Lion

Closest papers by meaning

• A simpler approach to accelerated optimization: iterative averaging meets optimism (ICML 2020 · similarity 0.85)
• Online mirror descent and dual averaging: keeping pace in the dynamic case (ICML 2020 · similarity 0.84)
• Composite Optimization with Error Feedback: the Dual Averaging Approach (ICLR 2026 · similarity 0.84)
• Adaptive Partitioning Schemes for Optimistic Optimization (ICML 2025 · similarity 0.83)
• A General Framework for User-Guided Bayesian Optimization (ICLR 2024 · similarity 0.83)
• Optimistic Online-to-Batch Conversions for Accelerated Convergence and Universality (NeurIPS 2025 · similarity 0.82)