Research map / Large language models / Instruction and fine-tuning

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiZO, a memory-efficient zeroth-order LLM fine-tuning method that periodically projects per-layer weight distances from the pre-trained anchor, using a ZO-learned projection scalar, to speed convergence and improve accuracy over MeZO and HiZOO.

Problem

Zeroth-order (ZO) fine-tuning of LLMs saves substantial memory but converges much more slowly and is less accurate than first-order (FO) fine-tuning, with the gap attributed to uniform-magnitude per-layer updates.

Approach

Layer-wise divergence analysis shows FO produces diverse per-layer distances from the pre-trained model while ZO produces uniform ones; DiZO periodically rescales each layer's distance from the pre-trained anchor via a projection scalar learned with ZO (re-initialized to the current distance and clipped), keeping the whole process gradient-free.

Key results

• On OPT-2.7B SST-2, DiZO reaches 92.5% accuracy vs 90.0% for MeZO, using 55.7% of MeZO's GPU hours.
• DiZO cuts training GPU hours by up to 48% relative to MeZO across datasets while keeping ZO-level throughput.
• On RoBERTa-large (k=16), DiZO beats full fine-tuning on SST-2 (92.2 vs 91.9) and RTE (67.9 vs 66.4).

Measured effects reported in the paper

Surprising findings

• Learning the projection with first-order Adam gives higher accuracy but only modest GPU-hour savings and requires backward passes, undermining the memory benefit.
• Projecting only Query and Value layers (16.7% of parameters stored) performs as well as projecting all layers, which requires storing the full pre-trained model.

Limitations

• Lacks theoretical justification for using the pre-trained model as anchor and for which layers to project.
• Requires storing a copy of the pre-trained Query and Value weights (about 16.7% of parameters), adding memory overhead.
• Convergence theorem relies on assumptions (bounded parameter ball) that are not fully justified empirically.

Experimental setup

Datasets: SST-2 SST-5 SNLI TREC MNLI RTE SuperGLUE CB BoolQ WIC WSC MultiRC

Benchmarks: SST-2 SST-5 SNLI TREC MNLI RTE SuperGLUE SQuAD DROP MMLU MT-Bench

Metrics: accuracy F1 GPU hours training iterations peak memory throughput

Baselines: MeZO HiZOO LoRA MeZO LoRA HiZOO LoRA full fine-tuning

Builds on: MeZO LoRA HiZOO

Closest papers by meaning

• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.94)
• Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs (NeurIPS 2026 · similarity 0.94)
• Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.94)
• Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.93)
• Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.92)
• Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.91)