Research map / Architectures and efficiency / RNNs and normalization
Scalable One-Pass Optimisation of High-Dimensional Weight-Update Hyperparameters by Implicit Differentiation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimise weight decay and learning rate in one pass | the proposed hyperparameter optimiser · UCI Energy | test MSE (mean): 24 \pm\,2 -> 2.1 \pm\,0.1 | Helped |
| add momentum optimisation (ablation) | Ours WD+LR · UCI Energy | test MSE (mean): 2.1 \pm\,0.1 -> 0.96 \pm\,0.08 | Helped |
| add independent per-parameter learning rates (ablation) | Ours WD+LR+M · UCI Energy | test MSE (mean): 0.96 \pm\,0.08 -> 0.6 \pm\,0.2 | Helped |
| add independent per-parameter learning rates (ablation) | Ours WD+LR+M · UCI Power | test MSE (mean): 17.19 \pm\,0.01 -> 16.69 \pm\,0.03 | Helped |
| add independent per-parameter learning rates (ablation) | Ours WD+LR+M · Fashion-MNIST | test cross-entropy (mean): 0.369 \pm\,0.002 -> 0.398 \pm\,0.004 | Hurt |
| add independent per-parameter learning rates (ablation) | Ours WD+LR+M · CIFAR-10 | test cross-entropy (mean): 1.13 \pm\,0.02 -> 1.33 \pm\,0.05 | Hurt |
| optimise weight decay, learning rate, and momentum instead of weight decay and learning rate alone (ablation) | Ours WD+LR · Fashion-MNIST | test cross-entropy (mean): 0.373 \pm\,0.002 -> 0.369 \pm\,0.002 | Helped |
| optimise weight decay, learning rate, and momentum instead of weight decay and learning rate alone (ablation) | Ours WD+LR · Penn Treebank | test perplexity (mean): 360 \pm\,20 -> 270 \pm\,30 | Helped |
| optimise weight decay, learning rate, and momentum instead of weight decay and learning rate alone (ablation) | Ours WD+LR · Kin8nm | test MSE (mean): 8.1 \pm\,0.2 -> 8.1 \pm\,0.4 | No clear effect |
| replace approximate hypergradients with exact differentiation through optimisation | Ours WD+LR+M · Fashion-MNIST | test cross-entropy (mean): 0.369 \pm\,0.002 -> 0.386 \pm\,0.001 | Hurt |
| increase the look-back horizon to all training steps (ablation) | Diff-through-Opt · UCI datasets | test loss | No clear effect |
| increase the look-back distance (ablation) | Ours WD+LR+M · UCI Energy | median final test loss | Mixed |
Closest papers by meaning
- Gradient Descent: The Ultimate Optimizer (NeurIPS 2022 · similarity 0.90)
- Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent (ICML 2025 · similarity 0.87)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.87)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.87)
- TFG: Unified Training-Free Guidance for Diffusion Models (NeurIPS 2024 · similarity 0.87)
- A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.