Research map / Large language models / Language, safety and interpretability
BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces BBOWP, a task where LLMs infer a search space and optimization algorithm from natural-language black-box optimization descriptions, with a 23-task executable benchmark (BBOWP-Bench) and a scale-invariant FRS metric; finds LLMs pick budget-appropriate algorithms but struggle with search-space design.
Problem
Automatically formulating black-box optimization problems (search space design and algorithm selection) from natural-language descriptions, which existing auto-formulation benchmarks, focused on explicit mathematical programs, do not cover.
Approach
Define the BBOWP setting and evaluate meta-solvers by running the chosen optimizer on a fixed executable objective through an identifier-alias adapter that maps generated search spaces to canonical variables, scored with a quantile-based Formulation Ranking Score against random reference configurations.
Key results
- GPT-5 mini achieved the highest average FRS (0.724) under the Small budget, closely followed by Gemini 3 Flash (0.709), while Gemini 3.1 Pro had the lowest (0.542); human baseline average FRS is 0.964.
- LLMs shift toward Bayesian methods (GP, SMAC3, TPE) under the Small budget (N=100) and toward CMA-ES under the Large budget (N=10,000), e.g., Gemini 3.1 Pro selects these at 78.3% vs 30.4%.
- In the Hard setting, Gemini 3.1 Pro scored FRS 0.133 versus 0.728 for Gemini 3 Flash, due to poor search-space design on Tasks 14 and 20.
Surprising findings
- Gemini 3.1 Pro aggressively reduced dimensionality by fixing four variables to constants, collapsing the useful search region on Task 3.
- Larger or more advanced models did not consistently outperform smaller ones; no significant difference across model size was observed.
Limitations
- Generated variables whose identifiers are not in the alias dictionaries are dropped, so variable matching is imprecise.
- Conditional search spaces (common in HPO) are not supported.
- Algorithm choice is restricted to a predefined pool of ten optimizers, and objectives are prebuilt rather than generated per search space.
Experimental setup
Datasets: YAHPO Gym Olympus MECHBench MuJoCo
Benchmarks: BBOWP-Bench
Metrics: best objective value Formulation Ranking Score (FRS) EPDF of objective values
Baselines: Human Baseline
Builds on: COCO OptunaHub NL4Opt
Closest papers by meaning
- Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings (ICML 2025 · similarity 0.88)
- Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.87)
- LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch (ICLR 2025 · similarity 0.87)
- OptMATH: A Scalable Bidirectional Data Synthesis Framework for Optimization Modeling (ICML 2025 · similarity 0.87)
- OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling (ICLR 2025 · similarity 0.87)
- Autoformulation of Mathematical Optimization Models Using LLMs (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.