Research map / Architectures and efficiency / Pruning and efficient tuning
Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeZO-SVRG, a memory-efficient zeroth-order SVRG variant that uses batch-wide single-direction SPSA estimators and in-place updates to fine-tune LMs without prompts, outperforming MeZO on GLUE/SuperGLUE tasks.
Problem
Zeroth-order fine-tuning of language models (MeZO) is unstable at small batch sizes and has a large convergence gap to first-order fine-tuning in the non-prompted setting, while backpropagation is memory-prohibitive for large LMs.
Approach
Adapts ZO-SVRG into a memory-efficient variant (MeZO-SVRG) that combines periodic full-batch ZO gradient estimates with minibatch control-variate updates, uses single-direction batch SPSA estimators for data parallelism, and performs in-place operations with a 3d memory footprint.
Key results
- MeZO-SVRG improves test accuracy over MeZO by up to 20% across masked and autoregressive models (e.g., RoBERTa-large SST-2 full FT 84 vs 56).
- MeZO-SVRG reaches MeZO's peak accuracy with roughly 2x fewer GPU-hours on GPT2 and OPT-2.7B.
- MeZO-SVRG uses about 2x less memory than FO-SGD on autoregressive models at batch size 1 and reaches 76% memory savings vs FO-SGD for RoBERTa-large at bs=64, but uses more memory than MeZO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MeZO-SVRG instead of MeZO | language-model fine-tuning · GLUE and SuperGLUE tasks | test accuracy: up to 20% increase | Helped |
| use MeZO-SVRG instead of MeZO | GPT2 and OPT-2.7B fine-tuning · MNLI, QNLI, SST-2, CoLA | GPU-hours to reach MeZO's peak test accuracy: 0.4 -> 0.3; 5.5 -> 1.9; 19.4 -> 5.6; 2.8 -> 2.2; 2.6 -> 1.1; 5.3 -> 2.7; 48 -> 25; 55 -> 1.4 | Helped |
| replace fullbatch estimator with a large-batch estimator (ablation) | MeZO-SVRG · DistilBERT SST-2 | test accuracy: 70; 71; 69 | No clear effect |
| reduce the perturbation scale below 1e-1 (ablation) | MeZO-SVRG · DistilBERT MNLI | fine-tuning performance | No clear effect |
| increase the perturbation scale above 1e-1 (ablation) | MeZO-SVRG · DistilBERT MNLI | fine-tuning performance | Hurt |
| increase the frequency of fullbatch updates (ablation) | MeZO-SVRG · DistilBERT MNLI | convergence rate | Helped |
| use fullbatch updates exclusively (q=1) rather than combined fullbatch and minibatch updates (ablation) | MeZO-SVRG · DistilBERT MNLI | stability | Hurt |
| use MeZO-SVRG instead of MeZO at smaller batch sizes | RoBERTa-large fine-tuning · SST-2 | robustness to batch size | Helped |
| use MeZO-SVRG instead of FO-SGD | autoregressive model fine-tuning · memory footprint | memory usage: 2× | Helped |
| increase batch size | MeZO-SVRG versus FO-SGD · RoBERTa-large MNLI, context length 128 | memory savings: up to 76% | Helped |
Surprising findings
- MeZO-SVRG's memory overhead vs MeZO grows with batch size, so MeZO-SVRG is more memory-hungry than MeZO (e.g., 4.72 GB vs 2.51 GB at bs=64).
Limitations
- Memory footprint is 3d (higher than MeZO's inference-level footprint), so it is less memory-efficient than MeZO.
- Full-batch gradient estimates are infeasible for large datasets; large-batch approximation is only partially validated.
- Experiments limited to few-shot-style subsets (512 training examples) and models up to 6.7B; PEFT (LoRA, prefix) and black-box settings left for future work.
Experimental setup
Datasets: MNLI QNLI SST-2 CoLA BoolQ MNIST
Benchmarks: MNLI QNLI SST-2 CoLA BoolQ
Metrics: test accuracy fine-tuning loss GPU-hours peak GPU memory (GB)
Baselines: MeZO FO-SGD FO-Adam ZO-SVRG
Builds on: MeZO ZO-SVRG SPSA SVRG
Closest papers by meaning
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.94)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.94)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.92)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.91)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.91)
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.