Research map / Probabilistic and scientific modeling / Bayesian optimization
Density Ratio Estimation-based Bayesian Optimization with Semi-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DRE-BO-SSL, a density-ratio-estimation Bayesian optimization method that uses semi-supervised classifiers (label propagation/spreading) on unlabeled points to reduce the over-exploitation of supervised classifiers such as BORE and LFBO.
Problem
Supervised classifiers used in density-ratio-estimation-based Bayesian optimization (BORE, LFBO) are overconfident and over-exploit known good regions, getting stuck in local optima.
Approach
Augment the labeled evaluation set with unlabeled points (sampled from a truncated multivariate normal via minimax tilting, or taken from a fixed pool) and use graph-based semi-supervised classifiers (label propagation and label spreading, with an inductive kernel-weighted extension) to estimate the class probability used as the acquisition function, with the RBF kernel width learned by entropy mi
Key results
- DRE-BO-SSL variants outperform BORE/LFBO with supervised classifiers and GP baselines on synthetic functions, Tabular Benchmarks, NATS-Bench and 64D multi-digit MNIST search, averaged over 20 seeds.
- The paper reports that the fixed-size pool subset of 2,000 points can be used to reduce cost without major performance loss.
- Threshold ratio zeta of 0.33 or 0.5 is generally appropriate, while 0.01 and 0.8 underperform.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DRE-BO-SSL with unlabeled point sampling instead of the tested baseline methods | Bayesian optimization on synthetic benchmarks · Synthetic benchmarks, unlabeled point sampling | simple regret | Helped |
| Use DRE-BO-SSL with a fixed-size pool instead of the tested baseline methods | Bayesian optimization · Synthetic benchmarks, fixed-size pools | simple regret | Helped |
| Use DRE-BO-SSL instead of the tested baseline methods | Bayesian optimization · Tabular Benchmarks | simple regret | Helped |
| Use DRE-BO-SSL instead of the tested baseline methods | Bayesian optimization · NATS-Bench: CIFAR-10, CIFAR-100, and ImageNet-16-120 | simple regret | Helped |
| Use DRE-BO-SSL instead of the tested baseline methods | Bayesian optimization · 64D minimum multi-digit MNIST search | simple regret | Helped |
| Add unlabeled points to semi-supervised learning | DRE-BO-SSL · Synthetic benchmarks | simple regret | Helped |
| Use Gaussian process-based Bayesian optimization instead of DRE-BO-SSL | Bayesian optimization · Tabular Benchmarks | simple regret | Mixed |
| Increase the number of unlabeled points (ablation) | DRE-BO-SSL · Synthetic benchmarks, unlabeled point sampling | simple regret | Mixed |
| Use truncated multivariate normal sampling instead of uniform, Halton, or Sobol' sampling (ablation) | DRE-BO-SSL · Synthetic benchmarks, unlabeled point sampling | simple regret | Mixed |
| Increase the free parameter β (ablation) | Label propagation and label spreading in DRE-BO-SSL · Four synthetic benchmark functions | simple regret | Helped |
| Randomly subsample the fixed-size pool to reduce its subset size (ablation) | DRE-BO-SSL · Fixed-size pools | simple regret | No clear effect |
| Use ζ = 0.01 or ζ = 0.1 instead of the generally recommended threshold ratios (ablation) | DRE-BO-SSL · Synthetic benchmarks | simple regret | Hurt |
Surprising findings
- Varying the number of unlabeled points n_u shows no clear trend; the best setting depends on the benchmark function.
- Smaller threshold ratios (0.01, 0.1) perform worse than larger ones, suggesting over-exploitation is not due to a conservative threshold.
Limitations
- Computational cost of label propagation/spreading grows with pool size, requiring subsampling.
- Performance is sensitive to the number of unlabeled points and sampling distribution, with no universal setting.
- Theoretical guarantees do not directly apply to the sequential Bayesian optimization setting.
Experimental setup
Datasets: Branin Beale Bukin6 Six-Hump Camel Tabular Benchmarks (naval propulsion, protein structure, Parkinson's telemonitoring, slice localization) NATS-Bench (CIFAR-10, CIFAR-100, ImageNet-16-120) MNIST
Benchmarks: Branin Beale Bukin6 Six-Hump Camel Tabular Benchmarks NATS-Bench 64D minimum multi-digit MNIST search
Metrics: simple regret
Baselines: Gaussian process EI Gaussian process UCB BORE with random forest LFBO with random forest BORE/LFBO with gradient boosting BORE/LFBO with XGBoost BORE/LFBO with MLP
Builds on: BORE LFBO label propagation label spreading minimax tilting
Closest papers by meaning
- Stochastic Bayesian Optimization with Unknown Continuous Context Distribution via Kernel Density Estimation (AAAI 2024 · similarity 0.86)
- Density Ratio Estimation with Conditional Probability Paths (ICML 2025 · similarity 0.86)
- High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling (ICML 2024 · similarity 0.86)
- Bayesian Optimisation with Unknown Hyperparameters: Regret Bounds Logarithmically Closer to Optimal (NeurIPS 2024 · similarity 0.85)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.85)
- Binary Losses for Density Ratio Estimation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.