Research map / Optimization / Optimizers
Hyperparameter optimization with approximate gradient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use approximate (inexact) hypergradient with decreasing tolerance (exponential sequence) instead of exact gradient (HOAG) (ablation) | HOAG hyperparameter optimization · 2-regularized logistic regression and kernel Ridge regression (20news, real-sim, Parkinson) | convergence time per iteration / overall convergence speed | Helped |
| Replace implicit-differentiation gradient with iterative differentiation (Domke 2012) using same inner solver | Iterdiff baseline vs HOAG · hyperparameter optimization benchmarks (Figure 3) | suboptimality vs time | Hurt |
| Use a quadratic (k^-2) instead of exponential tolerance decrease sequence (ablation) | HOAG · logistic regression / kernel ridge regression (Figure 2) | convergence stability | Hurt |
| Use exact gradient at every iteration (no tolerance sequence) (ablation) | HOAG · hyperparameter optimization (Figure 2) | convergence time | Hurt |
| Use the logistic loss as validation loss instead of zero-one loss for classification | 2-regularized logistic regression hyperparameter optimization · 20news, real-sim | differentiability requirement (A1) | No clear effect |
| Use backtracking line search for step size in Iterdiff instead of adaptive step size | Iterdiff · hyperparameter optimization benchmarks | performance | Hurt |
Closest papers by meaning
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (ICML 2016 · similarity 0.90)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.86)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · similarity 0.85)
- Parameter-free Clipped Gradient Descent Meets Polyak (NeurIPS 2024 · similarity 0.85)
- Reshuffling Resampling Splits Can Improve Generalization of Hyperparameter Optimization (NeurIPS 2024 · similarity 0.85)
- Sample complexity of data-driven tuning of model hyperparameters in neural networks with structured parameter-dependent dual function (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.