Research map / Optimization / Optimizers
Preconditioner on Matrix Lie Group for SGD
Xi-Lin Li
This paper tried a common change: Add batch / layer normalization
Use batch normalization · ImageNet ILSVRC2012 · training performance Hurt
What 431 accepted papers report for this change
- 313 improved
- 39 no clear effect
- 13 mixed
- 66 hurt
- MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers ICML 2026 · use independent layer norm per FFN · training convergence · convergenceImproved
- Angel or Demon: Investigating the Plasticity Interventions' Impact on Backdoor Threats in Deep Reinforcement Learning ICML 2026 · apply Layer Normalization in TM-Scratch · Acrobot and MountainCar · BTP: reduces BTP by over 30%Hurt
- MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning ICML 2026 · add layer normalization · MEAL Level 1 · soup deliveredImproved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Newton type preconditioning instead of ordinary gradient descent | Rosenbrock optimization · Rosenbrock function | iterations to optimal solution: about $200$ iterations | Improved |
| Use Fisher type preconditioning | Rosenbrock optimization · Rosenbrock function | optimization performance | Hurt |
| Use Newton type preconditioned SGD with a scaling-and-normalization preconditioner | AlexNet · ImageNet ILSVRC2012 | top-1 validation accuracy: about $56\%$ | Improved |
| Use a scaling-and-normalization preconditioner | Preconditioned SGD · ImageNet ILSVRC2012 | convergence speed | Improved |
| Use batch normalization | AlexNet · ImageNet ILSVRC2012 | training performance | Hurt |
| Use Newton type preconditioned SGD instead of SGD | LSTM · Wikitext-2, dropout rate 0.2 | test perplexity: $100.7$ -> $103.9$ | Hurt |
| Use Fisher type preconditioned SGD instead of SGD | LSTM · Wikitext-2, dropout rate 0.2 | test perplexity: $100.7$ -> $105.8$ | Hurt |
| Use Newton type preconditioned SGD instead of SGD | LSTM · Wikitext-2, dropout rate 0.35 | test perplexity: $102.3$ -> $100.7$ | Improved |
| Use Fisher type preconditioned SGD instead of SGD | LSTM · Wikitext-2, dropout rate 0.35 | test perplexity: $102.3$ -> $103.7$ | Hurt |
| Use either Newton type or Fisher type preconditioned SGD instead of SGD | LSTM · Wikitext-2, dropout rates 0.2 and 0.35 | training loss | Improved |
| Use diagonal rather than scaling-and-normalization or scaling-and-whitening preconditioning (ablation) | Preconditioned SGD · Wikitext-2 | performance | Hurt |
| Add momentum to the optimizer | LSTM training methods, including Adam and sparse Adam · Wikitext-2 | performance | Hurt |
Closest papers by meaning
- SymMaP: Improving Computational Efficiency in Linear Solvers through Symbolic Preconditioning (NeurIPS 2025 · similarity 0.85)
- Learning Sparse Approximate Inverse Preconditioners for Conjugate Gradient Solvers on GPUs (NeurIPS 2025 · similarity 0.85)
- The Expected Loss of Preconditioned Langevin Dynamics Reveals the Hessian Rank (AAAI 2024 · similarity 0.85)
- Graph Neural Preconditioners for Iterative Solutions of Sparse Linear Systems (ICLR 2025 · similarity 0.84)
- Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions (ICML 2024 · similarity 0.84)
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
