Research map / Optimization / Optimizers
MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MicroAdam, a memory-efficient Adam variant that Top-K-compresses gradients into a sliding window and stores the error-feedback buffer itself quantized to 4 bits, with AMSGrad-style convergence guarantees.
Problem
Adam's optimizer states double memory overhead relative to SGD, and existing memory-efficient adaptive methods (Adafactor, 8-bit Adam, GaLore) lack rigorous convergence guarantees or lose accuracy.
Approach
Compress gradients via Top-K sparsification into a sliding window of past sparse gradients from which Adam moments are recomputed, correct the compression error with error feedback, and compress the error-feedback buffer itself via 4-bit randomized quantization so the memory overhead of EF is negligible.
Key results
- On GLUE/MNLI, MicroAdam (m=10) reaches 85.10% on BERT-Base vs 83.53% for Adam and 84.61% for Adam-8b, at 2.55 GB memory.
- On Llama-2 7B GSM-8k fine-tuning, MicroAdam (m=10) reaches 34.72% accuracy with 5.65 GB optimizer state vs 25.1 GB for Adam (34.50%).
- On Open-Platypus with Llama-2 7B, MicroAdam averages 62.36 vs 62.10 for AdamW, using 46.63 GB total vs 67.17 GB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MicroAdam instead of Adam-8b | BERT-Base fine-tuning · GLUE/MNLI, BERT-Base | accuracy: 84.61% -> 85.10% | Helped |
| Use MicroAdam instead of Adam-8b | BERT-Large fine-tuning · GLUE/MNLI, BERT-Large | accuracy: 86.18% -> 86.17% | No clear effect |
| Use MicroAdam instead of Adam | OPT-1.3B fine-tuning · GLUE/MNLI, OPT-1.3B | accuracy: 87.90% -> 88.18% | Helped |
| Increase sliding window from m=10 to m=20 (ablation) | MicroAdam · GSM-8k, LLaMA-2 7B | accuracy: 34.72% -> 35.10% | Helped |
| Use MicroAdam instead of Adam-8b | LLaMA-2 7B fine-tuning · GSM-8k, LLaMA-2 7B | accuracy: 34.34% -> 34.72% | Helped |
| Use MicroAdam instead of Adam | LLaMA-2 7B fine-tuning · GSM-8k, LLaMA-2 7B | accuracy: 34.50% -> 34.72% | Helped |
| Use MicroAdam instead of Adam-8b | LLaMA-2 13B fine-tuning · GSM-8k, LLaMA-2 13B | accuracy: 45.19% -> 44.88 % | Hurt |
| Use MicroAdam instead of AdamW | LLaMA-2 7B instruction fine-tuning · Open-Platypus, Average Accuracy | accuracy: 62.10 -> 62.36 | Helped |
| Use MicroAdam instead of Adam-8b | LLaMA-2 7B instruction fine-tuning · ARC-c 25-shot, Open-Platypus | accuracy: 51.96 -> 53.07 | Helped |
| Use MicroAdam instead of Adam-8b | LLaMA-2 7B instruction fine-tuning · HellaSwag 10-shot, Open-Platypus | accuracy: 77.51 -> 77.46 | Hurt |
| Use MicroAdam instead of AdamW | LLaMA-2 7B instruction fine-tuning · MMLU 5-shot, Open-Platypus | accuracy: 45.53 -> 45.04 | Hurt |
| Use MicroAdam instead of AdamW-8bit | ResNet pre-training · ImageNet val, ResNet-18 | accuracy: 70.13% -> 71.86% | Helped |
Surprising findings
- MicroAdam updates only about 10% of weights per step yet outperforms SGD and AdamW on ResNet-18 ImageNet pre-training, which the authors attribute to implicit regularization.
- The authors found MicroAdam too sparse for LLM pre-training, with difficulties matching AdamW-8bit, so they restrict claims to fine-tuning.
Limitations
- Update is too sparse for LLM pre-training; attention layers seem to need dense updates.
- Only sparsity is studied as gradient projection; low-rank projection is not implemented.
- GaLore could not be run on Llama-2 7B in the fine-tuning setup because SVD was too slow.
Experimental setup
Datasets: GLUE/MNLI GSM-8k Open-Platypus ImageNet
Benchmarks: GLUE/MNLI GSM-8k ARC-c HellaSwag MMLU Winogrande ImageNet
Metrics: accuracy memory usage (GB) training loss runtime
Baselines: Adam Adam-8bit CAME GaLore SGD AdamW
Builds on: Adam AMSGrad Comp-AMS error feedback GaLore
Closest papers by meaning
- Adam-mini: Use Fewer Learning Rates To Gain More (ICLR 2025 · similarity 0.87)
- LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics (ICLR 2025 · similarity 0.87)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.86)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.86)
- FRUGAL: Memory-Efficient Optimization by Reducing State Overhead for Scalable Training (ICML 2025 · similarity 0.86)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.