Research map / Probabilistic and scientific modeling / Dynamical systems
Investigation into Training Dynamics of Learned Optimizers (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically analyzes training dynamics of a learned optimizer (LSTM-based L2O) on MNIST MLPs, examining symmetry-breaking deviations, heavy-tailedness of updates, and update covariance versus SGD, Adam, and Lion.
Problem
Learned optimizers can speed up training, but their optimization dynamics are poorly understood, including how they relate to geometric constraints from network symmetries and to noise in parameter updates.
Approach
Examines learned optimizer trajectories through the lens of rescale/scale symmetry-induced gradient constraints, heavy-tailed noise (alpha-stable tail index), and update covariance, and tests a regularizer that penalizes deviation from these constraints.
Key results
- L2O deviates from rescale and scale symmetry constraints far more than SGD and Adam, with the largest increase early in optimization.
- L2O parameter updates are much less heavy-tailed than gradients (alpha closer to 2) and less heavy-tailed than SGD/Adam updates.
- Adding a regularizer penalizing symmetry-breaking slows L2O optimization substantially as regularization strength increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L2O instead of SGD or Adam | MNIST optimizees with Leaky ReLU · MNIST | update noise tail heaviness | Helped |
| Use L2O instead of baseline optimizers | MNIST optimizees with Leaky ReLU · MNIST | update variation | Mixed |
| Add regularization penalizing update deviations from geometric constraints (ablation) | L2O · MNIST optimizees with Leaky ReLU or batch normalization | optimization speed | Hurt |
| Add symmetry-breaking regularization (ablation) | L2O on optimizee architectures it was not meta-trained on · MNIST optimizees | optimization speed | Hurt |
| Allow rapid symmetry breaking rather than regularizing updates toward geometric constraints (ablation) | L2O · Initial phase of MNIST optimization | optimization performance | Helped |
| Use L2O updates instead of gradients | L2O · Leaky ReLU optimizee on MNIST | noise tail heaviness | Helped |
| Use L2O instead of baseline optimizers | Leaky ReLU optimizee on MNIST · MNIST | maximum eigenvalue of update covariance | Mixed |
Surprising findings
- L2O updates are less heavy-tailed than gradients, suggesting it filters gradient noise, while retaining high update covariance variation.
Limitations
- Experiments limited to small feed-forward networks on MNIST with one hidden layer of 20 neurons.
- Single-paper short abstract; claims about stabilizing effects are interpretive rather than tested on larger settings.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: absolute symmetry deviation training loss tail index alpha largest eigenvalue of update covariance
Baselines: SGD Adam Lion
Builds on: Andrychowicz et al. 2016 learned optimizer Kunin et al. 2021 symmetry geometry Simsekli et al. 2019 heavy-tailed SGD noise
Closest papers by meaning
- Reverse engineering learned optimizers reveals known and novel mechanisms (NeurIPS 2021 · similarity 0.92)
- Understanding and correcting pathologies in the training of learned optimizers (ICML 2019 · similarity 0.90)
- Towards Learned Optimization Free Lunch (ICLR 2026 · similarity 0.89)
- Understanding Optimization in Deep Learning with Central Flows (ICLR 2025 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.