Research map / Optimization / Nonconvex and smooth optimization

Adaptively Incorporating Directional Hints into Zeroth-Order Optimization

NeurIPS 2026 · Optimization › Nonconvex and smooth optimization · approach: Complexity of nonconvex first-order methods · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CV-ZOD, a zeroth-order descent method that uses control-variate gradient estimates with reference vectors from low-dimensional directional hints, plus a fully zeroth-order adaptive step-size rule achieving interpolated O(1/T)-to-O(d/T) rates.

Problem

Zeroth-order optimization of non-convex functions when auxiliary directional hints (linear subspaces approximating the gradient) are available, with unknown and time-varying hint quality.

Approach

A control-variate zeroth-order gradient estimator that uses a reference vector (projection of gradient onto hint subspace) keeps the estimator unbiased for the smoothed gradient while hints reduce only variance; step size is chosen adaptively from zeroth-order norm estimates.

Key results

• On the fluid inverse problem (d=8192, 15 seeds, 100 steps), CV-ZOD with discount 0.7 and 12/2/2 query allocation reaches mean final objective 0.0136 vs 0.0258 for ZOD.
• Theory: the oracle algorithm's bound interpolates between O(1/T) (aligned hints) and O(d/T) (uninformative hints); the practical adaptive variant matches it up to log factors with O(k+log T) extra queries per iteration.
• On molecular conformations (efavirenz, adenosine, benzylpenicillin), CV-ZOD achieves similar energy reduction to the SurGD reference and outperforms GES.

Surprising findings

• Allocating too many queries to norm estimation (2/12/2 split) severely degrades CV-ZOD, with final objective around 0.08 versus ZOD's 0.026.

Limitations

• Analysis of the adaptive variant relies on a single low-dimensional hint subspace per iteration; multiple surrogates increase k.
• Experiments are limited to simulation-based scientific tasks; large-scale settings such as LLM fine-tuning are not validated.
• Empirical settings use rank-one hints from differentiable surrogates; the adaptive step-size uses exponential discounting heuristic that departs from the theoretical rule.

Experimental setup

Datasets: Wiggle150

Benchmarks: Wiggle150 fluid inverse problem (NVIDIA Warp, n=64)

Metrics: objective value MACE energy change cosine similarity between surrogate and true gradients principal angle

Baselines: ZOD GES SurGD

Builds on: Gaussian smoothing zeroth-order gradient estimator Guided Evolutionary Strategies Prior-Guided Random Gradient-Free

Closest papers by meaning

• Refining Adaptive Zeroth-Order Optimization at Ease (ICML 2025 · similarity 0.90)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations (ICLR 2025 · similarity 0.90)
• Guided Zeroth-Order Methods for Stochastic Non-convex Problems with Decision-Dependent Distributions (ICML 2025 · similarity 0.88)
• On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization (NeurIPS 2025 · similarity 0.88)
• Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient (ICML 2024 · similarity 0.88)
• Stochastic Zeroth-Order Optimization under Strongly Convexity and Lipschitz Hessian: Minimax Sample Complexity (NeurIPS 2024 · similarity 0.87)