Research map / Probabilistic and scientific modeling / Bayesian optimization
Sharpness-Aware Black-Box Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SABO, a sharpness-aware black-box optimizer that applies SAM-style min-max over KL-neighborhoods of a Gaussian search distribution, with convergence and generalization bounds, and evaluates on black-box prompt fine-tuning with RoBERTa-large.
Problem
Black-box optimization (e.g., black-box prompt fine-tuning of LLMs via APIs) tends to optimize training loss directly, which yields poor generalization; existing black-box methods lack sharpness-aware generalization control.
Approach
Reparameterize the objective as expected loss under a Gaussian search distribution and perform sharpness-aware minimization in the statistical manifold, using a KL-ball neighborhood over distributions instead of a Euclidean ball over parameters, with stochastic gradient approximation updates for mean and diagonal covariance.
Key results
- On black-box prompt fine-tuning with d=1000, SABO achieves the best results on SST-2 (87.96), AG's News (82.77), MRPC (79.68) and Yelp P. (91.96) among compared black-box methods.
- On synthetic functions with d=500, SABO converges at a rate similar to INGO, while CMA-ES, MMES, and BES fail on several functions (BES fails on all).
- Theory: convergence O(log T / T) for full-batch and O(1/sqrt(T)) for mini-batch under convexity and stated assumptions, plus a generalization bound.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SABO instead of CMA-ES | RoBERTa-large black-box prompt fine-tuning · SST-2, d=1000 | accuracy: 86.85 -> 87.96 | Helped |
| use SABO instead of CMA-ES | RoBERTa-large black-box prompt fine-tuning · AG’s News, d=1000 | accuracy: 82.21 -> 82.77 | Helped |
| use SABO instead of BES | RoBERTa-large black-box prompt fine-tuning · MRPC, d=1000 | F1: 79.09 -> 79.68 | Helped |
| use SABO instead of INGO | RoBERTa-large black-box prompt fine-tuning · RTE, d=1000 | accuracy: 53.07 -> 53.31 | Helped |
| use SABO instead of MMES | RoBERTa-large black-box prompt fine-tuning · SNLI, d=1000 | accuracy: 39.82 -> 40.32 | Helped |
| use SABO instead of MMES | RoBERTa-large black-box prompt fine-tuning · Yelp P., d=1000 | accuracy: 91.63 -> 91.96 | Helped |
| use SABO instead of BES | RoBERTa-large black-box prompt fine-tuning · RTE, d=200 | accuracy: 53.07 -> 53.67 | Helped |
| use SABO instead of MMES | RoBERTa-large black-box prompt fine-tuning · SNLI, d=500 | accuracy: 42.07 -> 42.29 | Helped |
Surprising findings
- BES fails on all synthetic test functions, indicating difficulty without adaptive mean and covariance updates.
Limitations
- Convergence theory assumes convex objective and strongly convex J in mu, with bounded iterates and rho scaling conditions.
- Approximations (first-order Taylor, treating perturbation as constant) mean the update is only approximate to the true min-max solution.
Experimental setup
Datasets: SST-2 Yelp polarity AG's News MRPC RTE SNLI
Benchmarks: SST-2 Yelp polarity AG's News MRPC RTE SNLI synthetic test functions (ellipsoid, l1/2-ellipsoid, different powers, Levy)
Metrics: test accuracy F1 score distance to optimum
Baselines: CMA-ES MMES BES INGO zero-shot
Builds on: sharpness-aware minimization INGO CMA-ES black-box tuning (BBT)
Closest papers by meaning
- MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization (NeurIPS 2025 · similarity 0.86)
- Optimizing the Unknown: Black Box Bayesian Optimization with Energy-Based Model and Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- PABBO: Preferential Amortized Black-Box Optimization (ICLR 2025 · similarity 0.85)
- Posterior Inference with Diffusion Models for High-dimensional Black-box Optimization (ICML 2025 · similarity 0.85)
- A Universal Class of Sharpness-Aware Minimization Algorithms (ICML 2024 · similarity 0.85)
- How Does Black-Box Impact the Learning Guarantee of Stochastic Compositional Optimization? (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.