Research map / Architectures and efficiency / Pruning and efficient tuning
Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sparse-MeZO, a zeroth-order LLM fine-tuning method that perturbs and updates only small-magnitude weights via a dynamic magnitude mask, with a forward-pass mask computation enabling inference-level memory, and reports gains over MeZO on SuperGLUE with LLaMA, Mistral and OPT.
Problem
MeZO zeroth-order fine-tuning of LLMs has noisy gradient estimates that cause slow convergence, sensitivity to learning rate, and suboptimal accuracy, especially when perturbing large-magnitude weights.
Approach
Restrict zeroth-order perturbation and update to a magnitude-based sparse subset of small weights (dynamic mask with per-layer thresholds), which is less harmed by gradient noise, and compute the mask on the fly during the forward pass so memory stays at inference level.
Key results
- On LLaMA-7b, S-MeZO raises RTE accuracy from 71.7% (MeZO) to 80.7% and BoolQ from 75.9% to 80.9%, averaging 80.3% vs 76.6% for MeZO.
- S-MeZO reaches about 3.5x speedup over MeZO on RTE and 3x on BoolQ in convergence steps.
- With the efficient forward-pass mask implementation, S-MeZO uses 14.6 GB memory, matching MeZO and enabling LLaMA-30b fine-tuning on a single A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| selectively perturb and update small-magnitude weights instead of using vanilla MeZO | MeZO on LLaMA-7b · RTE | accuracy: 71.7 -> 80.7 | Helped |
| selectively perturb and update small-magnitude weights instead of using vanilla MeZO | MeZO on LLaMA-7b · BoolQ | accuracy: 75.9 -> 80.9 | Helped |
| use selective magnitude-based masking rather than random masking | MeZO on LLaMA-7b · Average SuperGLUE | accuracy: 76.9 -> 80.3 | Helped |
| selectively perturb and update small-magnitude weights instead of using vanilla MeZO | MeZO on LLaMA2-7b · BoolQ | accuracy: 78.8% -> 82.2% | Helped |
| selectively perturb and update small-magnitude weights instead of using vanilla MeZO | MeZO on Mistral-7B · BoolQ | accuracy: 76.6 -> 79.2 | Helped |
| selectively perturb and update small-magnitude weights instead of using vanilla MeZO | MeZO on Mistral-7B · AQuA | accuracy: +2.6% | Helped |
| increase sparsity from 0.5 to 0.8 (ablation) | S-MeZO on LLaMA-7b · RTE | accuracy: 77.1 -> 82.3 | Helped |
| increase sparsity from 0.5 to 0.8 (ablation) | S-MeZO on LLaMA-7b · BoolQ | accuracy: 79.1 -> 82.5 | Helped |
| increase sparsity from 0.5 to 0.8 (ablation) | S-MeZO on LLaMA-7b · WIC | accuracy: 63.8 -> 64.3 | Mixed |
| compute the sparse mask during the forward pass rather than use vanilla sparse implementation (ablation) | S-MeZO on LLaMA-7b · MultiRC | GPU memory (GB): 28.3 -> 14.6 | Helped |
| use S-MeZO-EI instead of full-parameter fine-tuning | LLaMA-7b · MultiRC | GPU memory (GB): 158.6 -> 14.6 | Helped |
| use S-MeZO instead of MeZO | LLaMA-7b · RTE | steps to reach 70% accuracy: 17,500 -> 5,000 | Helped |
Surprising findings
- Perturbing and updating only small-magnitude weights helps more than random masking (R-MeZO gives little gain over MeZO), and higher sparsity (0.8) often gave the best results.
- Zeroth-order gradient estimates reduce loss on the same batch about 90% of the time but on held-out batches only about 50% of the time.
Limitations
- Remaining performance gap to first-order fine-tuning methods (acknowledged by authors).
- Sparsity hyperparameter and per-task sparsity levels must be tuned; experiments limited to SuperGLUE-style classification tasks and mostly 7B models.
- Convergence theory is based on assumptions (Lipschitz smoothness) and the analysis is loosely argued.
Experimental setup
Datasets: SuperGLUE (BoolQ, RTE, WIC, MultiRC, SST-2, COPA) PIQA SIQA AQuA
Benchmarks: SuperGLUE PIQA SIQA AQuA
Metrics: accuracy memory usage (GB) convergence speed (steps)
Baselines: MeZO R-MeZO MeZO-LoRA LoRA full fine-tuning (Adam) zero-shot in-context learning ZO-SGD-Cons ZO-SGD-Sign ZO-SGD-Adam ZO-AdaMU AdaZeta
Builds on: MeZO lottery ticket hypothesis SPSA LoRA
Closest papers by meaning
- Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.94)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.92)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.92)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.92)
- ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization (AAAI 2024 · similarity 0.91)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.