Research map / Architectures and efficiency / Pruning and efficient tuning
Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOZO, a zeroth-order fine-tuning method using low-rank matrix-wise gradient estimates with lazy subspace sampling (and LOZO-M with momentum), framed as ZO subspace optimization with convergence guarantees.
Problem
Zeroth-order fine-tuning of LLMs performs poorly because random full-rank ZO gradient estimates fail to capture the low-rank structure of first-order LLM gradients, and ZO momentum is memory-expensive.
Approach
Estimate gradients with a low-rank matrix perturbation U V^T (LGE), resample V only every nu steps (lazy sampling) so updates stay in a slowly changing low-rank subspace, and store only the projected momentum N with U, enabling momentum at negligible memory cost.
Key results
- On OPT-13B with 1000 examples, LOZO reaches 84.9 on SQuAD and 91.7 on SST-2, outperforming MeZO (81.8 and 91.3) and MeZO-LoRA on most tasks.
- On RoBERTa-large, LOZO-M reaches 62.7% on MNLI versus 56.7% for MeZO, with memory of 2.84 GB versus MeZO-M's 5.89 GB.
- On OPT-66B, LOZO attains 74.0 on RTE versus 71.5 for MeZO, and LOZO converges in about half the epochs of MeZO on WiC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LOZO instead of MeZO | OPT-30B · SuperGLUE tasks | score | Mixed |
| use LOZO instead of MeZO | OPT-66B · SuperGLUE tasks | score | Mixed |
| add momentum to LOZO | LOZO on RoBERTa-large · MNLI | accuracy (%): 61.6 -> 62.7 | Helped |
| add momentum to LOZO | LOZO on RoBERTa-large · SNLI | accuracy (%): 73.4 -> 74.0 | Helped |
| add momentum to MeZO | MeZO on RoBERTa-large · MNLI | accuracy (%): 56.7 -> 58.9 | Helped |
| add momentum to MeZO | MeZO on RoBERTa-large · SNLI | accuracy (%): 68.5 -> 69.6 | Helped |
| add momentum to LOZO | LOZO on RoBERTa-large · MNLI | memory usage (GB): 2.83 -> 2.84 | Hurt |
| add momentum to MeZO | MeZO on RoBERTa-large · MNLI | memory usage (GB): 3.00 -> 5.89 | Hurt |
| use LOZO instead of MeZO | RoBERTa-large · MNLI | accuracy (%): 56.7 -> 61.6 | Helped |
| use LOZO instead of MeZO | RoBERTa-large · SNLI | accuracy (%): 68.5 -> 73.4 | Helped |
| use LOZO instead of MeZO | OPT-66B · RTE | score: 71.5 -> 74.0 | Helped |
| use LOZO instead of MeZO | OPT-30B · SQuAD | score: 86.1 -> 85.6 | Hurt |
Surprising findings
- MeZO-LoRA underperforms LOZO because it can only optimize low-rank adapters rather than the full parameter matrix.
Limitations
- Integrating LOZO with Adam without extra memory cost is unresolved.
- Minor loss fluctuations appear late in training, possibly due to lazy sampling.
- Random seed regeneration adds computational cost.
Experimental setup
Datasets: SuperGLUE SQuAD DROP SST-2 RTE MNLI SNLI BoolQ WSC WiC MultiRC COPA
Benchmarks: SuperGLUE SQuAD DROP MNLI SNLI RTE SST-2
Metrics: accuracy F1 memory usage (GB) training loss curves
Baselines: MeZO MeZO-LoRA MeZO-M MeZO-Adam zero-shot ICL FT FT-LoRA
Builds on: MeZO ZO-SGD LoRA random coordinate minimization
Closest papers by meaning
- Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.94)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.94)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.94)
- Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.92)
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.92)
- Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.