Research map / Optimization / Optimizers
Forward and Reverse Gradient-Based Hyperparameter Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use constrained hyperparameter optimization to downweight or discard noisy training examples | Softmax regression data hyper-cleaner · MNIST test set | accuracy (%): 87.74 -> 90.07 | Helped |
| Use constrained hyperparameter optimization to identify corrupted training examples | Softmax regression data hyper-cleaner · MNIST corrupted-training-example detection | F1: 0.9137 | Helped |
| Set the example-weight L1 radius to 1000, 1500, 2000, or 2500 (ablation) | Data hyper-cleaner · MNIST test set | accuracy (%): 90.07; 90.06; 90.00; 90.09 | No clear effect |
| Learn task relatedness rather than use single-task learning | HMTL · CIFAR-10 | test accuracy (%): 67.47 ± 2.78 -> 70.85 ± 1.87 | Helped |
| Learn task relatedness rather than use single-task learning | HMTL · CIFAR-100 | test accuracy (%): 18.99 ± 1.12 -> 21.15 ± 0.36 | Helped |
| Add an L1 constraint on the learned task-interaction matrix (ablation) | HMTL · CIFAR-10 | test accuracy (%): 70.85 ± 1.87 -> 71.62 ± 1.34 | Helped |
| Add an L1 constraint on the learned task-interaction matrix (ablation) | HMTL · CIFAR-100 | test accuracy (%): 21.15 ± 0.36 -> 22.09 ± 0.29 | Helped |
| Learn the task-interaction matrix as a hyperparameter rather than use the compared MTL baselines | HMTL-S · CIFAR-10 | test accuracy (%): 69.96 ± 1.85; 70.30 ± 1.05; 70.96 ± 1.04 -> 71.62 ± 1.34 | Helped |
| Tune auxiliary-task weight and optimizer hyperparameters with real-time hypergradients | Shared phone-recognition network · TIMIT phone-state test set | accuracy (%): 59.81 -> 61.97 | Helped |
| Use real-time hypergradient optimization instead of random search | Shared phone-recognition network · TIMIT phone-state test set | accuracy (%): 60.36 -> 61.97 | Helped |
| Initialize hyperparameters to zero (null teacher) rather than the informed RTHO initialization (ablation) | RTHO phone-recognition optimization · TIMIT phone-state test set | accuracy (%): 61.97 -> 61.38 | Hurt |
| Use real-time forward hypergradient updates instead of random search | Phone-recognition hyperparameter optimization · TIMIT phone classification validation at 300 min | validation accuracy (%): 61.23 -> 62.88 | Helped |
Closest papers by meaning
- Gradient-based Hyperparameter Optimization Over Long Horizons (NeurIPS 2021 · similarity 0.89)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.86)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.85)
- A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning (AAAI 2025 · similarity 0.85)
- Sample complexity of data-driven tuning of model hyperparameters in neural networks with structured parameter-dependent dual function (NeurIPS 2025 · similarity 0.85)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.