AI Paper Atlas

Research map / Large language models / Instruction and fine-tuning

Learning a Zeroth-Order Optimizer for Fine-Tuning LLMs

Kairun Zhang, Haoyu Li, Yanjun Zhao, Yifan Sun, Huan Zhang

ICML 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add periodic reset of model parameters (ablation)ZO Fine-tuner (no normalization) · LLaMA-8B + SQuADfinal accuracy: 0.840 -> 0.856Improved
add normalization of perturbation variance (ablation)ZO Fine-tuner · LLaMA-8B + SQuADfinal accuracy: 0.840 -> 0.899Improved
add both normalization and periodic reset (ablation)ZO Fine-tuner · Qwen-14B + SST2final accuracy: 0.800 -> 0.935Improved
add both normalization and periodic reset (ablation)ZO Fine-tuner · LLaMA-8B + SST2final loss: 0.3976 -> 0.1789Improved
use block-wise instead of layer-wise variance sharing (ablation)ZO Fine-tuner · SST2 LLaMA-8Baccuracy: 0.92 -> 0.94Improved
use block-wise instead of layer-wise variance sharing (ablation)ZO Fine-tuner · SQuAD Qwen-14Baccuracy: 0.88 -> 0.91Improved
replace MeZO by ZO Fine-tunerMeZO · GSM8K Qwen2.5-14Baccuracy: 81.4 -> 85.6Improved
replace MeZO by ZO Fine-tunerMeZO · MATH-500 Qwen2.5-14Baccuracy: 53.0 -> 54.6Improved
meta-train ZO Fine-tuner on SQuAD instead of COPA (ablation)ZO Fine-tuner LLaMA-3.2-1B · SQuAD final lossfinal loss: 0.4827 -> 0.4903No clear effect
meta-train ZO Fine-tuner on SST-2 instead of COPA (ablation)ZO Fine-tuner LLaMA-3.1-8B · SST-2 final lossfinal loss: 0.3060 -> 0.1862Improved
transfer ZO Fine-tuner trained on LLaMA-3.1-8B to InstructZO Fine-tuner · SST2 Llama-3.1-8B-Instructaccuracy: 0.92 -> 0.95Improved
use SGD instead of Adam for the first-order trajectory (ablation)ZO Fine-tuner meta-training · SST-2 LLaMA-3.2-1Bavg training loss [16k,20k): 0.1655 -> 0.1575No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy