Research map / Architectures and efficiency / RNNs and normalization
Towards Learning Universal Hyperparameter Optimizers with Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OptFormer instead of a GP for few-shot function prediction | Few-shot response function prediction · RealWorldData test | log-predictive likelihood: 0.83 (0.06) -> 2.12 (0.05) | Helped |
| Use OptFormer instead of a GP for few-shot function prediction | Few-shot response function prediction · HPO-B test | log-predictive likelihood: 4.03 (0.04) -> 6.16 (0.04) | Helped |
| Use OptFormer instead of a GP for few-shot function prediction | Few-shot response function prediction · RealWorldData test | ECE (percent %): 5.34 (0.06) -> 1.11 (0.02) | Helped |
| Use OptFormer instead of a GP for few-shot function prediction | Few-shot response function prediction · HPO-B test | ECE (percent %): 2.39 (0.05) -> 1.89 (0.01) | Helped |
| Add Expected Improvement selection over sampled prior-policy suggestions | OptFormer prior policy · RealWorldData and HPO-B | best normalized function value | Helped |
| Train on in-domain HPO-B data rather than RealWorldData data (ablation) | OptFormer transfer to HPO-B · HPO-B test | best normalized function value | Helped |
| Train on RealWorldData rather than synthetic BBOB data (ablation) | OptFormer transfer to HPO-B · HPO-B test | best normalized function value | Helped |
| Add RealWorldData and BBOB training data to the HPO-B-trained model (ablation) | OptFormer · HPO-B test | best normalized function value | Helped |
| Remove most textual metadata and parameter ranges (ablation) | OptFormer function prediction · HPO-B test | best normalized function value | Hurt |
| Retain necessary metadata but remove other textual metadata and parameter ranges (ablation) | OptFormer prior policy · HPO-B test | best normalized function value | No clear effect |
| Replace the learned prior policy with random search (ablation) | Expected Improvement-augmented OptFormer · HPO-B test | best normalized function value | Hurt |
| Add an acquisition function to the OptFormer prior policy (ablation) | OptFormer prior policy · HPO-B test | best normalized function value | Helped |
Closest papers by meaning
- Meta-Surrogate Benchmarking for Hyperparameter Optimization (NeurIPS 2019 · similarity 0.90)
- HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenML (NeurIPS 2021 · similarity 0.90)
- Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning (AAAI 2024 · similarity 0.88)
- Few-Shot Bayesian Optimization with Deep Kernel Surrogates (ICLR 2021 · similarity 0.88)
- UQ-Guided Hyperparameter Optimization for Iterative Learners (NeurIPS 2024 · similarity 0.87)
- BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RL (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.