Research map / Large language models / Instruction and fine-tuning

Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

NeurIPS 2026 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that ZO (MeZO-style) fine-tuning of LLMs is dominated by a single decoding layer, which aligns with the first activation-outlier layer and can be identified via inference-only analysis; tuning only that layer matches or exceeds full-model MeZO at a fraction of the per-step cost.

Problem

Full-model zeroth-order fine-tuning of LLMs is memory-efficient but it is unclear how useful adaptation is distributed across layers, and which layers matter under forward-only updates.

Approach

Layer-wise ZO fine-tuning reveals a single dominant layer that matches or exceeds full-model MeZO; this layer is task-agnostic, model-specific, and identifiable before training as the first activation-outlier layer, because early residual-stream placement lets its perturbation propagate and accumulate through later layers.

Key results

• On Qwen3-8B, dominant-layer MeZO averages 85.67 vs 84.52 for full-model MeZO FT across the reported tasks.
• On LLaMA2-7B, dominant-layer ZO FT averages 72.44 vs 71.87 for MeZO FT.
• Single-layer ZO achieves up to 4.52x end-to-end per-step speedup on COPA relative to full-model MeZO, with perturb and update steps 27-33x faster.

Surprising findings

• Tuning a single layer (layer 1 in LLaMA2-7B) recovers most or all of full-model MeZO gains, while FO fine-tuning shows no single dominant layer.
• Later layers with large perturbation-induced loss changes (e.g., layer 30) do not yield comparable fine-tuning gains, showing loss sensitivity alone is insufficient.

Limitations

• Remaining performance gap to first-order fine-tuning and many steps needed for good performance.
• Only two model families (LLaMA2-7B and Qwen3-8B) examined.
• Combining with other ZO optimizers such as ZO-AdaMM and FZOO not explored.

Experimental setup

Datasets: SST-2 RTE CB BoolQ WSC MultiRC COPA SQuAD DROP

Benchmarks: SST-2 RTE CB BoolQ WSC MultiRC COPA SQuAD DROP

Metrics: accuracy F1 average score per-step runtime speedup

Baselines: MeZO FT MeZO LoRA Sparse-MeZO First-order AdamW fine-tuning Zero-shot

Builds on: MeZO LoRA Sparse-MeZO LLM.int8()

Closest papers by meaning

• Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.94)
• Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.92)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.92)
• Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models (ICML 2024 · similarity 0.90)
• Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.90)
• Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training (NeurIPS 2025 · similarity 0.90)