Research map / Architectures and efficiency / Pruning and efficient tuning

Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning

NeurIPS 2026 · Architectures and efficiency › Pruning and efficient tuning · approach: Low-rank compression and merging · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ZFO, which takes the update direction from a first-order optimizer (AdamW/Muon) and uses two extra forward evaluations along that direction to fit a Taylor/Padé local model and pick a bounded step size, tested on LLM RLVR fine-tuning with GRPO.

Problem

Choosing the step length (learning rate) along a first-order update direction is sensitive: small steps converge slowly and large steps destabilize training, especially in LLM fine-tuning.

Approach

Decouple direction from step: keep the FO optimizer's direction and use two symmetric fixed-batch forward probes along it to estimate second and third directional derivatives, then maximize a Taylor or Padé local model over a bounded interval; a fixed-budget alternative to line search that generalizes GeN.

Key results

• Padé3 improves over AdamW FO in 14 of 16 LLM model-dataset settings; e.g., Qwen-2.5-Math-1.5B on GSM8K 80.97 vs 78.24 FO (Taylor2 reaches 81.00).
• On Qwen-2.5-Math-1.5B ARC-Challenge, Taylor2 reaches 58.10 vs 49.29 for FO FT (+8.81).
• ZFO adds about 5.2% wall-clock overhead and 11.0%-19.1% peak GPU memory overhead.

Surprising findings

• Taylor3 underperforms FO on several settings (e.g., Qwen MATH 20.18 vs 31.77 FO), so higher-order models are not uniformly better.
• The Padé2 variant, which mostly picks the interval endpoint, sometimes outperforms richer models, suggesting gains are not solely from curvature modeling.

Limitations

• Gains are task- and model-dependent; the best ZFO variant changes across settings and FO seed variance is high.
• Theory assumes smoothness and bounded higher-order derivatives and gives neighborhood-of-stationarity guarantees; Padé and fallback steps do not guarantee improvement.
• Memory overhead is real, unlike MeZO, since the update direction is the FO gradient.

Experimental setup

Datasets: GSM8K SVAMP AsDiv OpenBookQA MATH (512/256 train/eval subset) ARC-Challenge StrategyQA FOLIO CODAH

Benchmarks: GSM8K MATH SVAMP AsDiv OpenBookQA ARC-Challenge StrategyQA FOLIO CODAH

Metrics: accuracy

Baselines: AdamW (FO) MeZO (ZO) Prodigy Armijo line search Barzilai-Borwein Polyak step size PLS

Builds on: GRPO AdamW MeZO GeN

Closest papers by meaning

• Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.89)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.88)
• Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer (ICLR 2025 · similarity 0.87)
• Refining Adaptive Zeroth-Order Optimization at Ease (ICML 2025 · similarity 0.87)
• DeepZero: Scaling Up Zeroth-Order Optimization for Deep Model Training (ICLR 2024 · similarity 0.87)
• Adaptively Incorporating Directional Hints into Zeroth-Order Optimization (NeurIPS 2026 · similarity 0.87)