Research map / Optimization / Optimizers
Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension
Wenbo Gong, Meyer Scetbon, Chao Ma, Edward Meeds
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add optimal low-rank compensation (ablation) | Alice · C4 eval ppl, LLaMA-130M | eval perplexity: 25.11 -> 21.95 | Improved |
| add subspace switching on top of tracking (ablation) | Alice · C4 eval ppl, LLaMA-130M | eval perplexity: 27.35 -> 25.11 | Improved |
| add low-rank tracking alone (no switching or compensation) (ablation) | Alice · C4 eval ppl, LLaMA-130M | eval perplexity: 26.96 -> 27.35 | Hurt |
| drop tracking (Alice-0) in the full Alice (ablation) | Alice · C4 eval ppl, LLaMA-130M | eval perplexity: 21.95 vs 21.99 | No clear effect |
| use orthogonal-complement switching instead of Gaussian or Gaussian-mix bases (ablation) | Alice · C4 eval ppl | eval perplexity | Improved |
| use optimal compensation instead of Fira's heuristic compensation (ablation) | Alice · C4 eval ppl | eval perplexity | Improved |
| use Alice instead of Adam | Alice · LLaMA-1.3B C4 pretraining | eval perplexity: 16.44 -> 13.52 | Improved |
| use Alice instead of Fira | Alice · LLaMA-130M C4, last layer not Adam | eval perplexity: 25.21 -> 21.95 | Improved |
| use RACS (SSGD) instead of Apollo-svd | RACS · LLaMA-1.3B C4 | eval perplexity: 14.10 -> 13.52 | Improved |
| train 1B LLaMA with Alice instead of 7B with Apollo | Alice · C4 eval ppl at 150K steps | eval perplexity: 13.02 -> 12.97 | Improved |
Closest papers by meaning
- LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.90)
- A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models (ICML 2025 · similarity 0.90)
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining (NeurIPS 2025 · similarity 0.89)
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.89)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
