Research map / Optimization / Optimizers

Tensorion: A Tensor-Aware Generalization of the Muon Optimizer

NeurIPS 2026 · Optimization › Optimizers · approach: Adam convergence and adaptive learning rates · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Tensorion, a Muon-style optimizer that performs LMO steps over a relaxed tensor spectral norm defined via adaptively chosen unfoldings, and reports gains on CIFAR, Tiny ImageNet and ViT/Swin fine-tuning.

Problem

Muon applies orthogonalized (spectral-norm LMO) updates only to matrices, while many modern parameters are tensors (e.g., convolution kernels); the exact tensor spectral norm LMO is NP-hard, so a tractable tensor-aware generalization is needed.

Approach

Define a relaxed tensor spectral norm by duality from the maximum nuclear norm over a family of tensor unfoldings; the LMO then reduces to Newton-Schulz orthogonalization on one chosen unfolding (chosen offline by a balanced-shape heuristic), recovering Muon for order-2 tensors.

Key results

• On CIFAR-10 ResNet-18, T-SGD reaches 95.3% accuracy vs 94.1% for SGD-M and 93.7% for AdamW.
• On CIFAR-100 ResNet-34, T-SGD reaches 79.1% vs 78.2% for SGD-M and 74.2% for AdamW.
• On Tiny ImageNet ResNet-50, T-SGD reaches 62.7% vs 61.01% for SGD-M and 57.4% for AdamW; the offline unfolding heuristic (90.44%) is comparable to the online choice (91.10%) on CIFAR-10 ablation.

Surprising findings

• Applying Tensorion with AdamW on non-tensor blocks gave a large test loss increase on CIFAR-100 (4.89 vs 1.33 for AdamW), and T-AdamW was not better than AdamW in accuracy on CIFAR-100.
• Some fixed unfoldings (tau={3}, {4}, {1,2}) performed notably worse than the default, showing sensitivity to the unfolding choice.

Limitations

• Evaluation limited primarily to computer vision classification at modest scale.
• Unclear whether the method scales to large training regimes like Muon.
• Gains are mixed: T-AdamW underperforms on some settings, and differences among high-accuracy unfoldings are not statistically resolved.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet Imagenette

Benchmarks: CIFAR-10 CIFAR-100 Tiny ImageNet Imagenette

Metrics: test accuracy test loss

Baselines: SGD-M AdamW Muon (tau={1} unfolding) T-SGD T-AdamW

Builds on: Muon AdamW SGD with momentum linear minimization oracle framework

Closest papers by meaning

• From SGD to Muon: Adaptive Optimization via Schatten-p Norms (NeurIPS 2026 · similarity 0.90)
• Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds (NeurIPS 2026 · similarity 0.89)
• Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective (NeurIPS 2026 · similarity 0.88)
• AMUSE: Anytime Muon with Stable Gradient Evaluation (NeurIPS 2026 · similarity 0.88)
• On the Convergence Analysis of Muon (NeurIPS 2026 · similarity 0.88)
• Convolutions and More as Einsum: A Tensor Network Perspective with Advances for Second-Order Methods (NeurIPS 2024 · similarity 0.87)