Research map / Optimization / Optimizers
Derivatives of Stochastic Gradient Descent in parametric optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that derivatives of SGD iterates with respect to a parameter follow an inexact SGD recursion on a quadratic-type auxiliary problem, yielding MSE convergence to the solution derivative under strong convexity with constant, vanishing, and interpolation step-size regimes.
Problem
Understanding how the derivatives (Jacobians) of SGD iterates with respect to a parameter, as used in hyperparameter optimization and meta-learning, converge to the derivative of the solution mapping of a stochastic strongly convex parametric problem.
Approach
Interprets the recursion for SGD derivatives as an inexact (perturbed) SGD on the stochastic problem defining the solution derivative, with the error driven by the convergence of the original SGD iterates, and analyzes this inexact SGD in mean squared error under several step-size regimes.
Key results
- Derivatives of constant step-size SGD stabilize in a noise ball of radius proportional to sqrt(step-size) around the solution derivative.
- With step sizes 2/(mu k + 8 kappa^2) the MSE of derivatives is O(log(k)^2/k).
- In the interpolation regime the derivative MSE converges exponentially fast, O(k(1-1/(8kappa^2))^k).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a step size decreasing as \frac{1}{\mu} \frac{2}{\run + 8\kappa^2} | SGD derivative recursion · Strongly convex parametric optimization | mean squared derivative error: O\left( \frac{\log(k + 8 \kappa^2)^2}{k+8\kappa^2}\right) | Helped |
| Use constant step size \eta_k = \frac{\mu}{4L^2} in the interpolation regime | SGD derivative recursion · Interpolation regime, \sigma = 0 | mean squared derivative error: O\left(k \left(1 - \frac{1}{8 \kappa^2} \right)^k\right) | Helped |
| Use a constant step size | SGD derivative recursion · Strongly convex parametric optimization | mean squared derivative error: \limsup_{k \to \infty} | Mixed |
| Use constant step size in the simple interpolation setting, where interpolation holds only for the iterates | SGD derivative recursion · Ordinary least squares, simple interpolation | derivative error | Mixed |
| Use constant step size in the double interpolation setting | SGD derivative recursion · Ordinary least squares, double interpolation | derivative error | Helped |
Surprising findings
- In simple interpolation (only iterates interpolate), derivatives do not converge exponentially and instead settle in a neighborhood as in the noisy case.
Limitations
- Requires strong convexity and smoothness assumptions; no extension to non-strongly convex or nonsmooth problems.
- Analysis restricted to small step sizes (at most mu/(4L^2)); larger step sizes left to future work.
- Only single-sample SGD analyzed; no minibatch version.
Experimental setup
Datasets: ijcnn1 synthetic regression data
Benchmarks: ijcnn1
Metrics: suboptimality Frobenius norm of derivative error
Builds on: piggyback differentiation implicit function theorem stochastic approximation with biased/inexact oracles
Closest papers by meaning
- Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descent (ICML 2022 · similarity 0.92)
- On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems (NeurIPS 2020 · similarity 0.91)
- Global Convergence and Stability of Stochastic Gradient Descent (NeurIPS 2022 · similarity 0.91)
- Fast Last-Iterate Convergence of SGD in the Smooth Interpolation Regime (NeurIPS 2025 · similarity 0.90)
- Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods (ICLR 2024 · similarity 0.90)
- Nonasymptotic Analysis of Stochastic Gradient Descent with the Richardson–Romberg Extrapolation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.