AI Paper Atlas

Research map / Optimization / Optimizers

MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates

Mohammad Mozaffari, Sikan Li, Zhao Zhang, Maryam Mehri Dehnavi

NeurIPS 2023 · Optimization › Optimizers · approach: Second-order and structured preconditioning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use hybrid MKOR-H (switch to first order on loss-rate) instead of MKOR (ablation)MKOR · SQuAD v1.1 BERT-Largeiterations to converge: 1,000 -> 600Improved
MKOR-H vs LAMB first-order baselineMKOR-H · SQuAD v1.1 BERT-Largeend-to-end time: 7.97 -> 3.10 h (2.57x speedup)Improved
MKOR vs Eva second-order baselineMKOR-H · SQuAD v1.1speedup: 1.69xImproved
train MKOR for 1500 steps instead of LAMB 1563 stepsMKOR · GLUE averageaverage metric: 0.8023 -> 0.8214Improved
reduce MKOR steps to 600 (ablation)MKOR · GLUE averageaverage metric: 0.8023 -> 0.8078Improved
MKOR-H instead of MKOR at 600 steps (ablation)MKOR · GLUE averageaverage metric: 0.8078 -> 0.811Improved
use KAISA instead of LAMBKAISA · GLUE averageaverage metric: 0.8023 -> 0.796Hurt
use MKOR instead of SGDMKOR · ImageNet ResNet-50 75.9% targetepochs: 88 -> 57Improved
MKOR vs KAISA on ResNet-50 ImageNetMKOR · ImageNet ResNet-50wall-clock speedup: 54 -> 57 epochs, 1.04x fasterMixed
increase factor inversion frequency (ablation)MKOR · Autoencoder CIFAR-100convergence rateImproved
increase factor inversion frequency (ablation)KAISA · Autoencoder CIFAR-100per-iteration costHurt
sweep learning rate from 10 to 0.01 (ablation)MKOR · CIFAR10 ResNet-50epochs to converge: 94, 79, 78, 76 epochs across learning ratesNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy