Research map / Optimization / Nonconvex and smooth optimization

Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?

NeurIPS 2026 · Optimization › Nonconvex and smooth optimization · approach: Stochastic gradient last-iterate analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes entry-wise smooth shrinkage clipping (x·exp(-|x|/τ)) as a cheap surrogate for spectral clipping, justified by a localization metric and Bayes-motivated derivation, with O(ε^-4) convergence under Cauchy-contaminated noise and NanoGPT token savings over Adam and Muon.

Problem

Stochastic gradient noise in LLM pretraining is heavy-tailed and causes loss spikes; spectral control of matrix updates (e.g., spectral clipping, Muon) is accurate but expensive, so the question is whether cheap entry-wise clipping can control the spectrum of matrix updates.

Approach

Introduces a normalized localization ratio from a first-order perturbation of σ1 to show heavy-tailed entrywise noise is localized and entry-wise control suffices; derives smooth shrinkage x·e^{-|x|/τ} as a Bayes-motivated closed-form surrogate (unique multiplicative redescending form) for the entrywise MMSE estimator under a Gaussian prior, applied either post-optimizer (Adam) or pre-msign (Muon)

Key results

• Smooth shrinkage improves Adam on NanoGPT (276M) pretraining, saving ~7.6% of training tokens.
• Applying smooth shrinkage before Muon's msign saves ~1.9% of training tokens over Muon.
• Convergence O(ε^-4) for post- and pre-clipping under Cauchy-contaminated entrywise noise; heavy tail only affects constants.

Surprising findings

• Smooth shrinkage advantage over hard clipping grows in heavy-tail-dominated regimes, attributed to rare tokens in language data.

Limitations

• Derivation assumes a Gaussian prior on the clean gradient bulk, a modeling choice not verified empirically.
• Convergence analysis assumes known Cauchy-contaminated noise model.
• Dichotomy between entry-wise contamination and bounded-second-moment subspace perturbation does not cover all noise types.

Experimental setup

Datasets: FineWeb

Benchmarks: FineWeb validation loss (NanoGPT pretraining) random Gaussian feature regression

Metrics: validation loss training tokens saved final-loss speedup

Baselines: Adam Muon hard coordinate-wise clipping global gradient clipping post-spectral clipping SPECTRA

Builds on: Muon SPECTRA clipped SGD Adam modded-nanoGPT

Closest papers by meaning

• Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed (ICML 2025 · similarity 0.89)
• To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions (ICLR 2025 · similarity 0.87)
• Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.87)
• Optimized Gradient Clipping for Noisy Label Learning (AAAI 2025 · similarity 0.86)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.86)
• Sign-Based Optimizers Are Effective Under Heavy-Tailed Noise (NeurIPS 2026 · similarity 0.86)