Research map / Learning theory and algorithms / Prediction and decision losses
HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize the deep-kernel parameters using meta-learning on meta-train tasks rather than randomly | Deep Kernel GP (FSBO) · HPO-B-v3 | normalized regret | Helped |
| Use transfer learning instead of non-transfer optimization | Black-box HPO methods · HPO-B-v3 | normalized regret | Helped |
| Use transfer learning instead of non-transfer optimization | Black-box HPO methods · HPO-B-v3 | average rank | Helped |
| Use FSBO rather than the other compared transfer methods | Transfer HPO methods · HPO-B-v3 | normalized regret | Helped |
| Use FSBO rather than the other compared transfer methods | Transfer HPO methods · HPO-B-v3 | average rank | Helped |
| Compare continuous surrogate tasks with discrete benchmark tasks | GP, Deep Kernel GP, and Random Search · HPO-B-v3 continuous versus discrete | comparative performance trend | No clear effect |
| Use Deep Kernel GP rather than GP | Non-transfer HPO methods · HPO-B-v2 | average rank | Helped |
| Use BOHAMIANN rather than Deep GP | Non-transfer HPO methods · HPO-B-v2 | normalized regret | No clear effect |
| Use FSBO rather than Deep Kernel GP without transfer initialization | Deep Kernel GP · HPO-B-v3 | average rank | Helped |
Closest papers by meaning
- Meta-Surrogate Benchmarking for Hyperparameter Optimization (NeurIPS 2019 · similarity 0.92)
- HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO (NeurIPS 2021 · similarity 0.91)
- LAMDA: Two-Phase HPO via Learning Prior from Low-Fidelity Data (AAAI 2026 · similarity 0.91)
- Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning (AAAI 2024 · similarity 0.89)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
- UQ-Guided Hyperparameter Optimization for Iterative Learners (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.