Research map / Probabilistic and scientific modeling / Bayesian optimization

Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings

ICML 2025 · Probabilistic and scientific modeling › Bayesian optimization · approach: Offline black-box optimization with surrogates · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniSO, a string-based universal offline black-box optimization framework that trains multi-task T5-based regressors (token-targeted UniSO-T and numeric-targeted UniSO-N) with metadata-guided contrastive alignment and Lipschitz smoothness regularization.

Problem

Offline black-box optimization is restricted to single-task, fixed-dimensional settings because heterogeneous numerical search spaces lack a unified representation, preventing cross-task universal optimization.

Approach

Represent designs as JSON-like strings with textual task metadata, train multi-task T5-based regressors (token-level P10 output or embedding-plus-MLP), and regularize latent space with metadata-guided contrastive alignment and a local Lipschitz smoothness loss balanced by loss-value scaling.

Key results

• Improved UniSO-T achieves average rank 2.222 across 9 tasks versus 3.111 for the best single-task numeric expert (BN+BO) in the multi-task comparison table.
• Improved UniSO-T averages rank 9.8 among 21 single-task offline BBO methods under z-score normalization, achieving the best result on TF-Bind-10.
• Zero-shot and few-shot generalization to unseen RobotPush, Rover and LunarLander tasks; zero-shot results significantly exceed the dataset best.

Measured effects reported in the paper

Surprising findings

• Pre-trained T5 embedder attends strongly to grammar/EOS tokens and little to numerical tokens, and training from scratch outperforms pre-trained initialization for UniSO-N.
• Vanilla UniSO-N underperforms numeric-input expert models on several tasks, with Ant and D'Kitty results often below the dataset best.

Limitations

• Training data is limited to about 90K pairs and 9 tasks; authors note future work should train on more data.
• UniSO-N underperforms numeric-input experts; UniSO-T still trails state-of-the-art single-task methods on average.
• Model-inner search relies on black-box optimizers (BO, EA) rather than gradients.

Experimental setup

Datasets: Design-Bench SOO-Bench

Benchmarks: Design-Bench SOO-Bench RobotPush Rover LunarLander

Metrics: un-normalized score normalized score average rank OOD rank correlation

Baselines: BO-qEI CMA-ES REINFORCE BN + BO BN + Grad

Builds on: OmniPred LLAMBO MetaBalance Design-Bench SOO-Bench

Closest papers by meaning

• BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems (NeurIPS 2026 · similarity 0.88)
• Large Language Models to Enhance Bayesian Optimization (ICLR 2024 · similarity 0.86)
• ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.86)
• ROOT: Rethinking Offline Optimization as Distributional Translation via Probabilistic Bridge (NeurIPS 2025 · similarity 0.86)
• Posterior Inference in Latent Space for Scalable Constrained Black-box Optimization (NeurIPS 2026 · similarity 0.86)
• InstructZero: Efficient Instruction Optimization for Black-Box Large Language Models (ICML 2024 · similarity 0.85)