Research map / Optimization / Optimizers

Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

NeurIPS 2026 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Pion, an optimizer that updates each weight matrix by left and right orthogonal (matrix-exponential of skew-symmetric Lie-algebra gradients) transformations, preserving its singular values without POET-style reparameterization.

Problem

Standard additive optimizers (Adam/AdamW, Muon) let weight spectral norms drift during LLM training, which can destabilize training; POET preserves spectrum via reparameterization but at the cost of training complexity and instability.

Approach

Update the identity factors of W directly on the orthogonal group: W_{t+1}=exp(-eta G_out) W_t exp(-eta G_in), with G_in and G_out skew-symmetric projections of the gradient; adds RMS-controlled scale consistency, Lie-algebra momentum, and a second-order exponential approximation.

Key results

• On LLaMA-1.3B pretraining (54B C4 tokens), Pion reaches average benchmark accuracy 47.69 vs 46.34 for Muon and 44.74 for AdamW, though its validation loss (2.7350) is slightly above Muon (2.7225).
• In normalization-free 60M LLaMA pretraining, AdamW and Muon diverge to NaN while Pion trains stably over 9.6B tokens.
• On RLVR with GRPO, Pion averages 36.12 on Qwen3-1.7B vs 34.82 (AdamW) and 32.08 (Muon), and 38.32 on DeepSeek-R1-Distill-Qwen-1.5B vs 37.32 (Muon).

Surprising findings

• AdamW and Muon diverge to NaN without normalization layers, whereas Pion remains stable.
• Pion's full version uses about 16% more memory than AdamW per GPU and is about 44% slower per step than AdamW.

Limitations

• Pion's persistent optimizer state (Lie-algebra second moments) adds substantial memory overhead, growing with the square of matrix dimensions for rectangular matrices.
• Pion is about 3% slower per step than Muon and 44% slower than AdamW in the reported setup.
• Convergence analysis relies on assumptions such as bounded trajectory and an idealized isospectral manifold; μP compatibility is only shown in a preliminary form.

Experimental setup

Datasets: C4 MetaMathQA Magicoder-Evol-Instruct-110K DeepMath

Benchmarks: ARC-C ARC-E BoolQ HellaSwag PIQA SciQ TriviaQA Winogrande GSM8K HumanEval AIME24 AIME25

Metrics: accuracy validation loss avg@K pass rate

Baselines: AdamW Muon

Builds on: POET Muon AdamW GRPO

Closest papers by meaning

• Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
• GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.86)
• Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.86)
• Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (ICLR 2024 · similarity 0.86)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.86)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.86)