Research map / Architectures and efficiency / Pruning and efficient tuning

Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer

ICLR 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HiZOO, a zeroth-order optimizer that estimates a diagonal Hessian with one extra forward pass per step to precondition SPSA-style updates for LLM fine-tuning, plus a low-rank memory-saving variant HiZOO-L.

Problem

Zeroth-order fine-tuning of LLMs (e.g., MeZO) converges slowly or unstably when parameters have heterogeneous curvatures, while first-order and classical second-order methods require prohibitive memory.

Approach

Estimate the diagonal Hessian from zeroth-order function evaluations (one extra forward pass per step) and use it as a preconditioner Σ^{1/2} in the SPSA-style update; compress the diagonal Hessian to rank-1 row/column factors (HiZOO-L, Adafactor-style) to cut memory.

Key results

• On SST-2, HiZOO achieves about 8x speedup over MeZO and 4x speedup on MNLI, with a 2.2% absolute MNLI accuracy gain over MeZO.
• On OPT-66B (prefix tuning), HiZOO reaches 71.7 average vs. 69.0 for MeZO, with up to 5.1% absolute improvement on a single task.
• HiZOO-L keeps Hessian memory overhead to under 10% above MeZO and reports a 4.3% absolute improvement over ZO-Adam while using about 50% of its GPU memory.

Measured effects reported in the paper

Surprising findings

• Using the diagonal Hessian estimate, HiZOO-L (rank-1 compressed) performs nearly identically to full HiZOO, suggesting the Hessian estimate may be sparse and compressible.

Limitations

• Each step requires one extra forward pass, making per-step time about 1.5x MeZO.
• Full HiZOO increases memory versus MeZO by storing a diagonal Hessian of parameter size; the low-rank variant is needed to reduce this.
• Large EMA smoothing coefficient alpha_t can impede convergence or cause training failure due to gradient explosion.

Experimental setup

Datasets: SST-2 SST-5 SNLI MNLI RTE TREC CB WSC WIC COPA MultiRC SQuAD

Benchmarks: SST-2 SST-5 SNLI MNLI RTE TREC CB WSC WIC COPA MultiRC SQuAD

Metrics: accuracy F1 training steps to convergence GPU memory wall-clock time per step

Baselines: MeZO ZO-SGD-MMT ZO-SGD-Cons ZO-SGD-Sign ZO-Adam Adam FT LoRA prefix tuning

Builds on: MeZO SPSA Adafactor MiNES

Closest papers by meaning

• Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.91)
• Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.91)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.91)
• Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning (NeurIPS 2025 · similarity 0.90)
• Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.89)
• Towards Efficient Low-Order Hybrid Optimizer for Language Model Fine-Tuning (AAAI 2025 · similarity 0.89)