Research map / Architectures and efficiency / RNNs and normalization
A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biases
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a nominal descent term to the blackbox optimizer (ablation) | STAR learned optimizer · Fashion MNIST MLP | meta-training speed and final optimizer performance | Helped |
| Add a nominal descent term to the blackbox optimizer (ablation) | STAR learned optimizer · CIFAR10 ConvNet | training performance | Helped |
| Add Adam-style preconditioning to the blackbox output (ablation) | STAR learned optimizer · in-distribution tasks | meta-loss | Helped |
| Add Adam-style preconditioning to the blackbox output (ablation) | STAR learned optimizer · out-of-distribution tasks | performance | Mixed |
| Use weight decay between 0.1 and 0.5 times the outer learning rate (ablation) | STAR learned optimizer · in-distribution and out-of-distribution tasks | stability and performance tradeoff | Helped |
| Increase weight decay to 1.0 times the outer learning rate (ablation) | STAR learned optimizer · in-distribution and out-of-distribution tasks | optimization performance | Hurt |
| Remove weight decay (ablation) | STAR learned optimizer · meta-training | meta-training performance | No clear effect |
| Remove the blackbox term, leaving the nominal term with magnitude control (ablation) | STAR learned optimizer · in-distribution tasks | optimization performance | Mixed |
| Remove the nominal term (ablation) | STAR learned optimizer · Fashion MNIST MLP | final training performance | No clear effect |
| Remove the nominal term (ablation) | STAR learned optimizer · CIFAR10 ConvNet | training performance | Hurt |
| Apply the STAR design rather than the blackbox optimizer | learned optimizer trained on Fashion MNIST MLP · out-of-distribution tasks | optimization stability | Helped |
| Apply the STAR design rather than the blackbox optimizer | learned optimizer trained on Fashion MNIST MLP · out-of-distribution tasks | optimization performance | Helped |
Closest papers by meaning
- Reverse engineering learned optimizers reveals known and novel mechanisms (NeurIPS 2021 · similarity 0.90)
- Understanding and correcting pathologies in the training of learned optimizers (ICML 2019 · similarity 0.89)
- Investigation into Training Dynamics of Learned Optimizers (Student Abstract) (AAAI 2024 · similarity 0.89)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.87)
- Generalization Analysis of Biased Stochastic Gradient Methods for Minimax Problems (NeurIPS 2026 · similarity 0.87)
- MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.