AI Paper Atlas

Research map / Optimization / Optimizers

Preconditioner on Matrix Lie Group for SGD

Xi-Lin Li

ICLR 2019 · Optimization › Optimizers · approach: Natural gradient and Fisher approximations · 0 citations (OpenAlex)

This paper tried a common change: Add batch / layer normalization

Use batch normalization · ImageNet ILSVRC2012 · training performance Hurt

What 431 accepted papers report for this change

See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use Newton type preconditioning instead of ordinary gradient descentRosenbrock optimization · Rosenbrock functioniterations to optimal solution: about $200$ iterationsImproved
Use Fisher type preconditioningRosenbrock optimization · Rosenbrock functionoptimization performanceHurt
Use Newton type preconditioned SGD with a scaling-and-normalization preconditionerAlexNet · ImageNet ILSVRC2012top-1 validation accuracy: about $56\%$Improved
Use a scaling-and-normalization preconditionerPreconditioned SGD · ImageNet ILSVRC2012convergence speedImproved
Use batch normalizationAlexNet · ImageNet ILSVRC2012training performanceHurt
Use Newton type preconditioned SGD instead of SGDLSTM · Wikitext-2, dropout rate 0.2test perplexity: $100.7$ -> $103.9$Hurt
Use Fisher type preconditioned SGD instead of SGDLSTM · Wikitext-2, dropout rate 0.2test perplexity: $100.7$ -> $105.8$Hurt
Use Newton type preconditioned SGD instead of SGDLSTM · Wikitext-2, dropout rate 0.35test perplexity: $102.3$ -> $100.7$Improved
Use Fisher type preconditioned SGD instead of SGDLSTM · Wikitext-2, dropout rate 0.35test perplexity: $102.3$ -> $103.7$Hurt
Use either Newton type or Fisher type preconditioned SGD instead of SGDLSTM · Wikitext-2, dropout rates 0.2 and 0.35training lossImproved
Use diagonal rather than scaling-and-normalization or scaling-and-whitening preconditioning (ablation)Preconditioned SGD · Wikitext-2performanceHurt
Add momentum to the optimizerLSTM training methods, including Adam and sparse Adam · Wikitext-2performanceHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy