Research map / Large language models / Instruction and fine-tuning
Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs
Kairun Zhang, Haoyu Li, Yanjun Zhao, Yifan Sun, Huan Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add periodic reset of model parameters (ablation) | ZO Fine-tuner (no normalization) · LLaMA-8B + SQuAD | final accuracy: 0.840 -> 0.856 | Improved |
| add normalization of perturbation variance (ablation) | ZO Fine-tuner · LLaMA-8B + SQuAD | final accuracy: 0.840 -> 0.899 | Improved |
| add both normalization and periodic reset (ablation) | ZO Fine-tuner · Qwen-14B + SST2 | final accuracy: 0.800 -> 0.935 | Improved |
| add both normalization and periodic reset (ablation) | ZO Fine-tuner · LLaMA-8B + SST2 | final loss: 0.3976 -> 0.1789 | Improved |
| use block-wise instead of layer-wise variance sharing (ablation) | ZO Fine-tuner · SST2 LLaMA-8B | accuracy: 0.92 -> 0.94 | Improved |
| use block-wise instead of layer-wise variance sharing (ablation) | ZO Fine-tuner · SQuAD Qwen-14B | accuracy: 0.88 -> 0.91 | Improved |
| replace MeZO by ZO Fine-tuner | MeZO · GSM8K Qwen2.5-14B | accuracy: 81.4 -> 85.6 | Improved |
| replace MeZO by ZO Fine-tuner | MeZO · MATH-500 Qwen2.5-14B | accuracy: 53.0 -> 54.6 | Improved |
| meta-train ZO Fine-tuner on SQuAD instead of COPA (ablation) | ZO Fine-tuner LLaMA-3.2-1B · SQuAD final loss | final loss: 0.4827 -> 0.4903 | No clear effect |
| meta-train ZO Fine-tuner on SST-2 instead of COPA (ablation) | ZO Fine-tuner LLaMA-3.1-8B · SST-2 final loss | final loss: 0.3060 -> 0.1862 | Improved |
| transfer ZO Fine-tuner trained on LLaMA-3.1-8B to Instruct | ZO Fine-tuner · SST2 Llama-3.1-8B-Instruct | accuracy: 0.92 -> 0.95 | Improved |
| use SGD instead of Adam for the first-order trajectory (ablation) | ZO Fine-tuner meta-training · SST-2 LLaMA-3.2-1B | avg training loss [16k,20k): 0.1655 -> 0.1575 | No clear effect |
Closest papers by meaning
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.95)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.94)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.93)
- Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training (NeurIPS 2025 · similarity 0.92)
- Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.92)
- Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
