Research map / Learning theory and algorithms / Prediction and decision losses
HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DE instead of random search | HPOBench community benchmarks · HPOBench community | wins/ties/losses vs RS: 18/2/2 (p 0.00043) | Helped |
| use SMAC-RF instead of random search | HPOBench community benchmarks · HPOBench community | wins/ties/losses vs RS: 19/3/0 (p 0.00001) | Helped |
| use TPE instead of random search | HPOBench community benchmarks · HPOBench community | wins/ties/losses vs RS: 13/4/5 (p 0.06690) | No clear effect |
| use DEHB instead of Hyperband | HPOBench community benchmarks · HPOBench community | wins/ties/losses vs HB: 20/2/0 (p 0.00001) | Helped |
| use Dragonfly instead of Hyperband | HPOBench community benchmarks · HPOBench community | wins/ties/losses vs HB: 5/0/17 (p 0.99783) | Hurt |
| use BOHB instead of Hyperband | HPOBench community benchmarks · HPOBench community | wins/ties/losses vs HB: 13/4/5 (p 0.06690) | No clear effect |
| add multi-fidelity (Hyperband) to random search | random search · HPOBench community, full budget | wins/ties/losses: 16/5/1 (p 0.00074) | Helped |
| add multi-fidelity (DEHB) to DE | DE · HPOBench community, 1% budget | wins/ties/losses: 14/3/5 (p 0.03918) | Helped |
| add multi-fidelity (DEHB) to DE | DE · HPOBench community, 100% budget | wins/ties/losses: 6/8/8 (p 0.73827) | No clear effect |
| add multi-fidelity (SMAC-HB) to SMAC-RF | SMAC-RF · HPOBench community, 100% budget | wins/ties/losses: 6/8/8 (p 0.73827) | No clear effect |
| add multi-fidelity (BOHB) to TPE | TPE · HPOBench community, 1% budget | wins/ties/losses: 14/2/6 (p 0.06690) | No clear effect |
Closest papers by meaning
- HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML (NeurIPS 2021 · similarity 0.91)
- Meta-Surrogate Benchmarking for Hyperparameter Optimization (NeurIPS 2019 · similarity 0.90)
- Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning (AAAI 2024 · similarity 0.86)
- OptiBench Meets ReSocratic: Measure and Improve LLMs for Optimization Modeling (ICLR 2025 · similarity 0.85)
- Efficient Hyperparameter Optimization with Adaptive Fidelity Identification (CVPR 2024 · similarity 0.84)
- UQ-Guided Hyperparameter Optimization for Iterative Learners (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.