Research map / Optimization / Optimizers
Gradient-based Hyperparameter Optimization Over Long Horizons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Share hyperparameters across contiguous time steps and optimize non-greedily | Gradient-based hyperparameter optimization · CIFAR-10, WRN-16-1, 50 epochs | time to zero regret: 3 hours vs 60 hours | Helped |
| Replace non-greedy per-step hyperparameters with shared contiguous windows (ablation) | Non-greedy hypergradient optimization · SVHN, LeNet, 250 steps | hypergradient mean squared error: W=50 | Helped |
| Use greedy rather than non-greedy hypergradients (ablation) | Learning-rate optimization · MNIST and SVHN, LeNet | learning-rate schedule quality | Hurt |
| Use non-greedy optimization without hyperparameter sharing (ablation) | Learning-rate optimization · SVHN, LeNet | learning-rate schedule quality | Hurt |
| Share learning rates over contiguous steps (ablation) | Non-greedy learning-rate optimization · SVHN, LeNet | validation performance: W \sim 50 | Helped |
| Learn momentum and weight decay as schedules rather than single values (ablation) | CIFAR-10 hyperparameter search · CIFAR-10 | test accuracy | No clear effect |
| Increase the number of searched hyperparameters (ablation) | Random Search, BO, Hyperband, and BOHB · CIFAR-10, WRN-16-1 | test accuracy | Hurt |
| Increase the number of searched hyperparameters (ablation) | FDS · CIFAR-10, WRN-16-1 | test accuracy | No clear effect |
| Use smaller inner-loop budgets for some early outer steps (ablation) | FDS · CIFAR-10, WRN-16-1 | time to find good hyperparameters: from ~10 hours to ~3 hours | Helped |
Closest papers by meaning
- Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation (ICLR 2022 · similarity 0.89)
- A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning (AAAI 2025 · similarity 0.88)
- Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization (NeurIPS 2024 · similarity 0.86)
- Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuning (NeurIPS 2025 · similarity 0.86)
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.86)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.