Research map / Architectures and efficiency / RNNs and normalization
Understanding and improving Shampoo and SOAP via Kullback-Leibler Minimization
Wu Lin, Scott C. Lowe, Felix Dangel, Runa Eschenhagen, Zikun Xu, Roger Baker Grosse
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use KL-Shampoo's EMA eigenvalue estimation with QR eigenbasis instead of step-size grafting with Adam | KL-Shampoo · Language model pre-training | validation loss | Improved |
| replace instantaneous eigenvalue estimation with the EMA scheme (ablation) | Shampoo variants · Language model pre-training | validation loss | Improved |
| replace Frobenius norm or von Neumann divergence with KL divergence (ablation) | Shampoo schemes · Language model pre-training | validation loss | Improved |
| add KL-based two-sided estimation to SOAP (KL-SOAP) instead of Adam in eigenbasis | SOAP · Language model pre-training | validation loss | Improved |
| use KL-Shampoo instead of KL-SOAP | KL-SOAP · Language model pre-training | validation loss | Improved |
| apply EMA eigenvalue scheme to trace-scaling (VN) Shampoo (ablation) | VN-Shampoo · Language model pre-training | validation loss | Improved |
Closest papers by meaning
- Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner (NeurIPS 2025 · similarity 0.89)
- A New Perspective on Shampoo's Preconditioner (ICLR 2025 · similarity 0.89)
- 4-bit Shampoo for Memory-Efficient Network Training (NeurIPS 2024 · similarity 0.84)
- A Stable Whitening Optimizer for Efficient Neural Network Training (NeurIPS 2025 · similarity 0.84)
- KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Products (NeurIPS 2025 · similarity 0.82)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
