Research map / Optimization / Implicit bias and generalization

Zeroth-Order Sharpness-Aware Learning with Exponential Tilting

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Sharpness-aware minimization theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ZEST, a gradient-free zeroth-order method that optimizes an exponentially tilted (t-SAM) objective interpolating between average-loss zeroth-order (MeZO) and max-loss SAM, with theory on sharpness regularizers and RoBERTa/OPT experiments.

Problem

Zeroth-order optimization minimizes the average loss under random perturbations, which only encourages small average loss in a neighborhood rather than flat minima; SAM-style max-loss objectives need gradients. The paper seeks a gradient-free method that targets flatter minima.

Approach

Connects two-point zeroth-order estimators to sharpness-aware minimization via an exponentially tilted objective F_t = (1/t) log E[exp(t f(x+eps))], deriving an unbiased function-value-only gradient for Gaussian and ball perturbations, with naive and bias-corrected ratio estimators, so t interpolates from average-loss (t→0) to max-loss (t→∞).

Key results

• On clean GLUE with RoBERTa-base, ZEST_BC reaches 73.8 on MNLI vs MeZO 71.8 and ZEST_N reaches 83.6 on MRPC vs MeZO 81.9.
• On GLUE with 30% label noise, ZEST_N improves over MeZO by up to 5.9 accuracy points (e.g., QQP 68.3 vs 62.4).
• On OPT-1.3B, ZEST_N gets 78.0 on COPA vs MeZO 74.0 and 79.4 vs 78.8 on SQuAD, with ZEST_BC and ZEST_N performing on par.

Surprising findings

• ZEST_N and ZEST_BC perform on par in experiments, attributed to small k so the O(1/k^2) bias reduction is not noticeable.
• Under ball perturbation, the sharpness regularizer R_t is continuous and non-decreasing in t, with t→∞ giving a max-eigenvalue/slope penalty.

Limitations

• Optimal tilting parameter t is data-dependent and must be tuned on validation data (t in {1,5,20}).
• Experiments limited to small models (RoBERTa-base, OPT-1.3B) and many-shot prompt fine-tuning.
• Ball-perturbation gradient uses a sphere-denominator approximation with bias O(1/sqrt(d)).

Experimental setup

Datasets: SST-2 SST-5 QQP MRPC TREC MNLI SNLI RTE COPA ReCoRD SQuAD DROP

Benchmarks: GLUE COPA ReCoRD SQuAD DROP

Metrics: accuracy F1 average neighborhood loss top-5 Hessian eigenvalues

Baselines: MeZO SGD ESAM SAM TSAM

Builds on: MeZO SAM ESAM tilted SAM (TSAM)

Closest papers by meaning

• Tilted Sharpness-Aware Minimization (ICML 2025 · similarity 0.91)
• Lookbehind-SAM: k steps back, 1 step forward (ICML 2024 · similarity 0.90)
• A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.90)
• Zeroth-Order Optimization Finds Flat Minima (NeurIPS 2025 · similarity 0.89)
• Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization (NeurIPS 2024 · similarity 0.89)
• $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling (NeurIPS 2024 · similarity 0.89)