Research map / Optimization / Optimizers
Enhancing LLM Training via Spectral Clipping
Xiaowen Jiang, Andrei Semenov, Sebastian U Stich
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add post-spectral clipping (SPECTRA) to AdamW | AdamW · FineWeb-Edu 780M | validation loss: 2.608 -> 2.582 | Improved |
| add post-spectral clipping (SPECTRA) to Signum | Signum · FineWeb-Edu 780M | validation loss: 2.600 -> 2.573 | Improved |
| add post-spectral clipping (SPECTRA) to AdEMAMix | AdEMAMix · FineWeb-Edu 780M | validation loss: 2.591 -> 2.574 | Improved |
| add spectral clipping to Signum under small batch (16K tokens) and large lr | Signum · FineWeb-Edu 160M small batch | final validation loss: 4.964 -> 3.196 | Improved |
| give baselines 20% more training steps instead of using SPECTRA (ablation) | AdamW, Signum, MARS, D-Muon · FineWeb-Edu 780M | validation loss: AdamW 2.590 vs Spectra-AdamW 2.582 | No clear effect |
| set clipping threshold c to 1 instead of 5-50 (ablation) | Spectra-AdamW · FineWeb-Edu 160M | validation loss: 2.936 -> 2.975 | Hurt |
| vary clipping threshold c from 5 to 50 (ablation) | Spectra-AdamW · FineWeb-Edu 160M | validation loss: 2.940, 2.936, 2.936, 2.940 | No clear effect |
| vary Newton-Schulz iterations among 5, 10, 15 (ablation) | Spectra-AdamW · FineWeb-Edu 160M | validation loss: 2.939, 2.936, 2.937 | No clear effect |
| add pre-spectral clipping on top of post-clipping (ablation) | Spectra-Signum · FineWeb-Edu 160M small batch | final validation loss: 3.196 -> 3.191 | No clear effect |
| use post-spectral clipping instead of post-orthogonalization (ablation) | Spectra · FineWeb-Edu 160M | validation loss | Improved |
| add SPECTRA to base optimizers | Signum, AdamW, AdEMAMix · FineWeb-Edu 820M 5-shot downstream | average accuracy: Signum 49.84 -> 51.62; AdamW 49.76 -> 50.30; AdEMAMix 49.05 -> 50.99 | Improved |
| add SPECTRA to base optimizers | Signum, AdamW, AdEMAMix · 820M weight norms | aggregated l-infinity weight norm | Improved |
Closest papers by meaning
- Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients? (NeurIPS 2026 · similarity 0.90)
- Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation (NeurIPS 2026 · similarity 0.87)
- Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory (NeurIPS 2026 · similarity 0.87)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.86)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.86)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
