Research map / Large language models / Instruction and fine-tuning
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks six zeroth-order and forward-gradient optimizers for memory-efficient LLM fine-tuning across multiple models, tasks, and PEFT schemes, and proposes block-wise ZO, hybrid ZO-FO, and sparsity-induced ZO enhancements.
Problem
Full back-propagation fine-tuning of LLMs has large memory overhead; prior work (MeZO) only used basic ZO-SGD, leaving the broader space of ZO optimizers and their accuracy/efficiency trade-offs for LLM fine-tuning unexplored.
Approach
Systematically benchmarks ZO-SGD, ZO-SGD-Sign, ZO-SGD-Momentum, ZO-SGD-Conservative, ZO-Adam and Forward-Gradient against FO-SGD/Adam across models, tasks and PEFT schemes, with a detailed peak-memory analysis, and introduces block-wise ZO updates, hybrid ZO-FO training, and sparse random perturbations to reduce ZO gradient variance.
Key results
- On OPT-13B with LoRA, Forward-Grad and ZO-Adam outperform the best FO method on COPA, but on WinoGrande FO methods lead ZO methods by about 5-6 points.
- ZO-SGD fine-tuning of full OPT-13B on MultiRC uses 64 GB on one A100 versus 148 GB for FO-SGD on two A100s, and cuts per-iteration runtime by about 41.9% versus FO-SGD.
- Block-wise ZO-SGD-Block reaches 93.69% on SST2 and 57.2% on WinoGrande with OPT-1.3B, versus 90.83% and 55.5% for MeZO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use prompt-based task alignment instead of no alignment (ablation) | ZO-SGD and ZO-Adam · SST2 and RTE, Roberta-Large FT | test accuracy: 89.4 vs 79.2; 68.7 vs 60.4 (ZO-SGD); 89.8 vs 79.2; 69.2 vs 58.7 (ZO-Adam) | Helped |
| increase query budget above 1 (ablation) | Forward-Grad and ZO-SGD · LoRA-based SST2, OPT-1.3B | accuracy: 1%~2% | Helped |
| increase query budget above 500 (ablation) | Forward-Grad vs ZO-SGD · LoRA-based SST2, OPT-1.3B | accuracy: 1%~2% | Helped |
| estimate gradients block-wise instead of model-wise (ablation) | ZO-SGD · SST2, OPT-1.3B | accuracy: 90.83 -> 93.69 (1 forward pass vs 26) | Helped |
| estimate gradients block-wise instead of model-wise (ablation) | ZO-SGD · WinoGrande, OPT-1.3B | accuracy: 55.5 -> 57.2 (1 forward pass vs 26) | Helped |
| use 26 model-wise queries instead of one query (ablation) | ZO-SGD · SST2, OPT-1.3B | accuracy: 90.83 -> 91.28 | Helped |
| use 26 model-wise queries instead of one query (ablation) | ZO-SGD · WinoGrande, OPT-1.3B | accuracy: 55.5 -> 55.7 | Helped |
| increase the number of ZO layers from 0 to 4 (ablation) | Hybrid ZO-FO · SST2, OPT-1.3B | accuracy: 91.22 -> 91.12 | Hurt |
| increase the number of ZO layers from 0 to 8 (ablation) | Hybrid ZO-FO · SST2, OPT-1.3B | accuracy: 91.22 -> 90.79 | Hurt |
| increase the number of ZO layers from 0 to 12 (ablation) | Hybrid ZO-FO · SST2, OPT-1.3B | accuracy: 91.22 -> 89.48 | Hurt |
| increase the number of ZO layers from 0 to 16 (ablation) | Hybrid ZO-FO · SST2, OPT-1.3B | accuracy: 91.22 -> 89.42 | Hurt |
| increase the number of ZO layers from 0 to 20 (ablation) | Hybrid ZO-FO · SST2, OPT-1.3B | accuracy: 91.22 -> 89.27 | Hurt |
Surprising findings
- ZO-SGD does not consistently outperform other ZO variants; ZO-Adam or ZO-SGD-MMT is often best, and ZO results have high variance across settings.
- Removing prompt-based task alignment drops ZO-SGD accuracy by about 10 points on SST2, while FO-SGD is nearly unaffected.
Limitations
- ZO methods still lag FO methods on the harder WinoGrande task with larger models, showing a scalability bottleneck.
- Forward-Grad advantage appears only with large query budgets (q>500), which erases its memory benefit.
- Results rely on task-specific prompt templates and grid-searched hyperparameters per method.
Experimental setup
Datasets: SST2 COPA WinoGrande MultiRC
Benchmarks: SST2 COPA WinoGrande MultiRC
Metrics: test accuracy peak memory (GB) GPU count runtime per iteration (s)
Baselines: MeZO FO-SGD FO-Adam Forward-Grad ZO-SGD-Sign ZO-SGD-MMT ZO-SGD-Cons ZO-Adam
Builds on: MeZO ZO-SGD Forward gradient LoRA ZO-Adam
Closest papers by meaning
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.94)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.94)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.93)
- Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training (NeurIPS 2025 · similarity 0.92)
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.92)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.