Research map / Large language models / Instruction and fine-tuning
AdaMeZO: Adam-style Zeroth-Order Optimizer for LLM Fine-tuning Without Maintaining the Moments
Zhijie Cai, Haolong Chen, Guangxu Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add truncated first moment to MeZO (ablation) | MeZO · SQuAD (OPT-1.3B) | F1: 73.3 -> 75.6 | Improved |
| add truncated first moment to MeZO (ablation) | MeZO · SST2 (OPT-1.3B) | accuracy: 90.9 -> 90.9 | No clear effect |
| add truncated first moment to MeZO (ablation) | MeZO · COPA (OPT-1.3B) | accuracy: 74.5 -> 75.3 | Improved |
| add truncated second moment on top of first moment (ablation) | AdaMeZO · SST2 (OPT-1.3B) | accuracy: 90.9 -> 91.6 | Improved |
| add truncated second moment on top of first moment (ablation) | AdaMeZO · SQuAD (OPT-1.3B) | F1: 75.6 -> 76.1 | Improved |
| use beta2 0.9 instead of 0.99 (ablation) | AdaMeZO · SST2 (OPT-1.3B) | accuracy: 90.9 -> 91.6 | Improved |
| use beta1 0.7 instead of 0.6 (ablation) | AdaMeZO · SST2 (OPT-1.3B) | accuracy: 91.1 -> 91.6 | Improved |
| use beta1 0.7 instead of 0.8 (ablation) | AdaMeZO · COPA (OPT-1.3B) | accuracy: 74.3 -> 75.5 | Improved |
| use horizon h larger than 5 (ablation) | AdaMeZO · SST2 eval loss (OPT-1.3B) | evaluation loss | No clear effect |
| replace MeZO with AdaMeZO | MeZO · RoBERTa-large k=16 avg | accuracy: 64.8 -> 66.0 | Improved |
| replace MeZO with AdaMeZO | MeZO · SuperGLUE avg (OPT-1.3B) | accuracy: 61.4 -> 63.1 | Improved |
| replace MeZO with AdaMeZO | MeZO · SuperGLUE avg (LLaMA3-3B) | accuracy: 61.4 -> 64.1 | Improved |
Closest papers by meaning
- Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer (ICLR 2025 · similarity 0.88)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization (AAAI 2024 · similarity 0.86)
- Addax: Utilizing Zeroth-Order Gradients to Improve Memory Efficiency and Performance of SGD for Fine-Tuning Language Models (ICLR 2025 · similarity 0.86)
- Refining Adaptive Zeroth-Order Optimization at Ease (ICML 2025 · similarity 0.86)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
