Research map / Optimization / Optimizers
ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZO-AdaMU, a zeroth-order optimizer that relocates adaptive momentum and a simulated-annealed uncertainty onto the SPSA perturbation, to fine-tune LLMs with forward passes only, improving on MeZO in stability, convergence and generalization.
Problem
Memory-efficient full-parameter fine-tuning of LLMs with zeroth-order optimization suffers from oscillating SPSA gradient estimates, overfitting without momentum regularization, and slow convergence from perturbation-irrelevant momentum.
Approach
Instead of applying momentum to the gradient (as in Adam-style ZO methods like ZO-AdaMM), ZO-AdaMU builds the SPSA perturbation itself from a momentum-centered Gaussian and a zero-centered Gaussian, with the mixing uncertainty and smoothing parameters set by a simulated-annealing schedule.
Key results
- On OPT-13B with 1,000 examples, ZO-AdaMU (2x) reaches 92.1 on SST-2 and 89.0 on COPA, beating MeZO's 91.4 and 88.0.
- On RoBERTa-large, ZO-AdaMU attains 93.8 on SST-2 versus MeZO's 93.3, and 95.2 on TREC with LoRA versus MeZO's 94.3.
- Memory analysis reports ZO-AdaMU fits the same largest OPT models as MeZO on the listed A100 and V100 configurations, with a slight extra memory cost over MeZO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ZO-AdaMU (2×) instead of MeZO | OPT-13B · COPA | accuracy: 88.0 -> 89.0 | Helped |
| Use ZO-AdaMU (2×) instead of MeZO | OPT-13B · RTE | accuracy: 66.1 -> 72.9 | Helped |
| Use ZO-AdaMU (2×) instead of MeZO | OPT-13B · SQuAD | score: 84.7 -> 82.4 | Hurt |
| Use ZO-AdaMU (2×) instead of MeZO | OPT-13B · CB | accuracy: 67.9 -> 67.9 | No clear effect |
| Use ZO-AdaMU instead of MeZO-Adam | RoBERTa-large · SST-2 | accuracy: 93.3 ($\pm$0.6) -> 93.8 ($\pm$0.3) | Helped |
| Use ZO-AdaMU instead of MeZO-Adam | RoBERTa-large · RTE | accuracy: 79.2 ($\pm$1.2) -> 77.9 ($\pm$ 2.0) | Hurt |
| Adapt the momentum schedule on perturbation rather than gradients (ablation) | ZO prompt-tuning · SST2 | accuracy: 90.6 -> 92.1 | Helped |
| Adapt the momentum schedule on perturbation rather than gradients (ablation) | ZO prompt-tuning · ReCoRD | score: 81.2 -> 83.0 | Helped |
| Adapt the momentum schedule on perturbation rather than gradients (ablation) | ZO prompt-tuning · SQuAD | score: 82.1 -> 82.4 | Helped |
Surprising findings
- Adam (FT) remains the strongest on several classification tasks on OPT-13B (e.g., CB 83.9 vs 67.9 for ZO-AdaMU), contrary to the paper's claim of generalization gains.
- Several reported gains are small (often within one standard deviation across 5 runs on RoBERTa-large), and the regret bound comparison is asserted rather than rigorously derived.
Limitations
- Adds a momentum buffer, so memory is slightly higher than MeZO.
- Theoretical analysis relies on strong assumptions (bounded gradients and iterate distances) and the Lemma derivations are loosely justified.
- Gains over MeZO are often small relative to reported standard deviations.
Experimental setup
Datasets: SST-2 RTE CB BoolQ WSC WIC MultiRC COPA ReCoRD SQuAD DROP SST-5
Benchmarks: SuperGLUE BoolQ SQuAD DROP SST-2 SST-5 SNLI MNLI RTE TREC
Metrics: accuracy F1
Baselines: MeZO MeZO (LoRA) MeZO (prefix) MeZO-Adam Adam LOMO ZO-AdaMM AdamW AdaMax
Builds on: MeZO ZO-AdaMM SPSA Adam
Closest papers by meaning
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.90)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.90)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.90)
- MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization (NeurIPS 2025 · similarity 0.89)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.