Research map / Architectures and efficiency / RNNs and normalization

Optimizer-Induced Mode Connectivity: From AdamW to Muon

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that for two-layer ReLU networks, regularized solution sets induced by AdamW/Signum/Muon/normalized-momentum optimizers are connected at sufficient width, constructs a finite-width example where AdamW and Muon solutions lie in disconnected zero-loss components, and proposes a polychain alignment path for GPT-2 pretraining experiments.

Problem

Whether solution sets restricted to an optimizer's implicit bias (norm-constrained regions induced by AdamW, Muon, Lion-K) are connected, and how they are arranged relative to each other in the loss landscape, which classical mode connectivity does not address.

Approach

Defines optimizer-regularized solution sets as interpolating solutions intersected with the dual-norm ball induced by the optimizer's implicit bias, proves intra-optimizer connectivity at large width for two-layer ReLU nets, and gives a finite-width example where AdamW and Muon land in different zero-loss components; empirically adds a learnable bend-point (polychain) path with alignment to GPT-2

Key results

• Polychain-plus-alignment path reduces AdamW same-optimizer loss barrier on BookCorpus from ~0.34 (linear) to 0.05±0.01 and Muon from 0.32 to 0.04±0.01.
• Along AdamW-to-Muon polychain paths, stable rank increases monotonically while eval loss stays relatively stable.
• Theory: intra-optimizer connectivity proven for Normalized GD with momentum and Muon when m ≥ 4P, and for Signum and AdamW for sufficiently large m; finite-width example gives a loss barrier of at least 1/2 between AdamW and Muon solutions.

Surprising findings

• Cross-optimizer (AdamW-Muon) paths keep low loss in the GPT-2 experiments despite the theory showing disconnected zero-loss components at finite width.
• Intermediate models along AdamW-Muon paths show negative loss barrier on out-of-distribution datasets.

Limitations

• Intra-optimizer theory is limited to two-layer ReLU networks in the interpolation regime with large width.
• Empirical results use small GPT-2 models (~30–50M parameters) trained from scratch.
• Connectivity of the regularized set is necessary but not sufficient for limit-point connectivity.

Experimental setup

Datasets: enwik8 WikiText-103 Stories BookCorpus One Billion Word

Benchmarks: enwik8 WikiText-103 Stories BookCorpus One Billion Word

Metrics: validation loss barrier stable rank singular value histograms

Baselines: linear interpolation Git Re-Basin-style alignment path (Transformer mode connectivity)

Builds on: Lion-K mode connectivity Git Re-Basin transformer model merging via canonical alignment AdamW implicit bias (KKT / l-infinity constraint)

Closest papers by meaning

• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.87)
• On the Convergence Analysis of Muon (NeurIPS 2026 · similarity 0.86)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.86)
• Phases of Muon: When Muon Eclipses SignSGD (NeurIPS 2026 · similarity 0.86)
• Tensorion: A Tensor-Aware Generalization of the Muon Optimizer (NeurIPS 2026 · similarity 0.86)
• From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.85)