Research map / Optimization / Optimizers
MKOR: Momentum-Enabled Kronecker-Factor-Based Optimizer Using Rank-1 Updates
Mohammad Mozaffari, Sikan Li, Zhao Zhang, Maryam Mehri Dehnavi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hybrid MKOR-H (switch to first order on loss-rate) instead of MKOR (ablation) | MKOR · SQuAD v1.1 BERT-Large | iterations to converge: 1,000 -> 600 | Improved |
| MKOR-H vs LAMB first-order baseline | MKOR-H · SQuAD v1.1 BERT-Large | end-to-end time: 7.97 -> 3.10 h (2.57x speedup) | Improved |
| MKOR vs Eva second-order baseline | MKOR-H · SQuAD v1.1 | speedup: 1.69x | Improved |
| train MKOR for 1500 steps instead of LAMB 1563 steps | MKOR · GLUE average | average metric: 0.8023 -> 0.8214 | Improved |
| reduce MKOR steps to 600 (ablation) | MKOR · GLUE average | average metric: 0.8023 -> 0.8078 | Improved |
| MKOR-H instead of MKOR at 600 steps (ablation) | MKOR · GLUE average | average metric: 0.8078 -> 0.811 | Improved |
| use KAISA instead of LAMB | KAISA · GLUE average | average metric: 0.8023 -> 0.796 | Hurt |
| use MKOR instead of SGD | MKOR · ImageNet ResNet-50 75.9% target | epochs: 88 -> 57 | Improved |
| MKOR vs KAISA on ResNet-50 ImageNet | MKOR · ImageNet ResNet-50 | wall-clock speedup: 54 -> 57 epochs, 1.04x faster | Mixed |
| increase factor inversion frequency (ablation) | MKOR · Autoencoder CIFAR-100 | convergence rate | Improved |
| increase factor inversion frequency (ablation) | KAISA · Autoencoder CIFAR-100 | per-iteration cost | Hurt |
| sweep learning rate from 10 to 0.01 (ablation) | MKOR · CIFAR10 ResNet-50 | epochs to converge: 94, 79, 78, 76 epochs across learning rates | No clear effect |
Closest papers by meaning
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.87)
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.87)
- SMMF: Square-Matricized Momentum Factorization for Memory-Efficient Optimization (AAAI 2025 · similarity 0.86)
- Layerwise LQR for Geometry-Aware Optimization of Deep Networks (NeurIPS 2026 · similarity 0.86)
- A geometric framework for momentum-based optimizers for low-rank training (NeurIPS 2025 · similarity 0.85)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
