AI Paper Atlas

Research map / Optimization / Optimizers

Enhancing LLM Training via Spectral Clipping

Xiaowen Jiang, Andrei Semenov, Sebastian U Stich

ICML 2026 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add post-spectral clipping (SPECTRA) to AdamWAdamW · FineWeb-Edu 780Mvalidation loss: 2.608 -> 2.582Improved
add post-spectral clipping (SPECTRA) to SignumSignum · FineWeb-Edu 780Mvalidation loss: 2.600 -> 2.573Improved
add post-spectral clipping (SPECTRA) to AdEMAMixAdEMAMix · FineWeb-Edu 780Mvalidation loss: 2.591 -> 2.574Improved
add spectral clipping to Signum under small batch (16K tokens) and large lrSignum · FineWeb-Edu 160M small batchfinal validation loss: 4.964 -> 3.196Improved
give baselines 20% more training steps instead of using SPECTRA (ablation)AdamW, Signum, MARS, D-Muon · FineWeb-Edu 780Mvalidation loss: AdamW 2.590 vs Spectra-AdamW 2.582No clear effect
set clipping threshold c to 1 instead of 5-50 (ablation)Spectra-AdamW · FineWeb-Edu 160Mvalidation loss: 2.936 -> 2.975Hurt
vary clipping threshold c from 5 to 50 (ablation)Spectra-AdamW · FineWeb-Edu 160Mvalidation loss: 2.940, 2.936, 2.936, 2.940No clear effect
vary Newton-Schulz iterations among 5, 10, 15 (ablation)Spectra-AdamW · FineWeb-Edu 160Mvalidation loss: 2.939, 2.936, 2.937No clear effect
add pre-spectral clipping on top of post-clipping (ablation)Spectra-Signum · FineWeb-Edu 160M small batchfinal validation loss: 3.196 -> 3.191No clear effect
use post-spectral clipping instead of post-orthogonalization (ablation)Spectra · FineWeb-Edu 160Mvalidation lossImproved
add SPECTRA to base optimizersSignum, AdamW, AdEMAMix · FineWeb-Edu 820M 5-shot downstreamaverage accuracy: Signum 49.84 -> 51.62; AdamW 49.76 -> 50.30; AdEMAMix 49.05 -> 50.99Improved
add SPECTRA to base optimizersSignum, AdamW, AdEMAMix · 820M weight normsaggregated l-infinity weight normImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy