Research map / Optimization / Optimizers
Gradient Descent as Loss Landscape Navigation: a Normative Framework for Deriving Learning Rules
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames learning rules as policies for navigating (partially observable) loss landscapes via a continuous-time optimal control objective, deriving gradient descent, momentum, natural gradient, Adam-like and continual-learning weight-reset rules from different assumptions.
Problem
Learning rules are usually assumed or chosen empirically rather than derived; it is unclear why some rules work better and under what assumptions a rule is optimal.
Approach
Treat learning rules as solutions to a continuous-time optimal control problem over parameter trajectories, with a kinetic-style parameter-change cost plus a discounted loss cost; varying the planning horizon (discount), parameter metric G, drift f, and a Bayesian belief about landscape shape recovers known optimizers including Adam-like rules.
Key results
- Derives a 'ballistic' rule with square-root Hessian scaling that reduces to an Adam-like heuristic implementation performing well on MNIST and CIFAR-10 classifiers.
- Shows that under ballistic parameter dynamics with slowly changing landscape beliefs, the optimal update has effective learning rate scaling as V_t^{-1/2} rather than V_t^{-1}, linking Adam to the framework.
- Recovers gradient descent, momentum, natural gradient and Newton-like rules as limiting cases of one objective, with the discount rate gamma interpolating between momentum and overdamped gradient descent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| optimize a multi-step trajectory instead of a single update | Gradient descent on a double-well loss · Double-well loss | minimum reached | Helped |
| plan over multiple steps, introducing momentum | Learning dynamics · Loss landscape navigation | momentum | Helped |
| use a non-Euclidean parameter-space metric | Gradient descent · Loss landscape navigation | sensitivity to parameter-space geometry | Helped |
| include non-gradient drift in the parameter dynamics | Optimal learning dynamics · Partially controllable parameters | ability to account for background dynamics | Helped |
| infer changing loss-landscape shape from past gradients | Adaptive learning rules · Time-evolving loss landscape | adaptation to inferred landscape shape | Helped |
| use the square root of the curvature estimate in the adaptive learning rate | Adam-like learning rule · Time-evolving loss landscape | optimality under the proposed model | Helped |
| use the heuristic ballistic rule | Ballistic rule · MNIST and CIFAR-10 | classification performance | Helped |
| inject noise or reset unused weights | Continual learning strategies · Continual learning | plasticity | Helped |
Surprising findings
- The square root of the Hessian (rather than its inverse) arises as optimal in the ballistic regime, and the authors argue it is a feature rather than a bug of Adam.
- Natural gradient descent is argued not to be second-order optimization in disguise: the metric G and Hessian H play fundamentally different roles.
Limitations
- Does not account for memory, computational cost, or efficiency in the notion of optimality, and matrix operations may be expensive.
- Heuristic ballistic implementation uses diagonal, uncentered, running-average approximations and was only tested on small MLP and CNN models.
- Empirical link to practice is limited; the authors say linking the framework to specific empirical circumstances is more difficult.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: classification performance
Baselines: gradient descent Newton's method Adam
Builds on: natural gradient descent Adam Wibisono et al. variational accelerated optimization Dohare et al. continual backpropagation
Closest papers by meaning
- Decision-Focused Learning with Directional Gradients (NeurIPS 2024 · similarity 0.86)
- Understanding and Improving Training-free Loss-based Diffusion Guidance (NeurIPS 2024 · similarity 0.86)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.86)
- Exploring Landscapes for Better Minima along Valleys (NeurIPS 2025 · similarity 0.86)
- SLOPE: Optimistic Potential Landscape Shaping for Model-based Reinforcement Learning (NeurIPS 2026 · similarity 0.86)
- LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail) (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.