Research map / Optimization / Optimizers
Automatic Differentiation of Optimization Algorithms with Time-Varying Updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends convergence analysis of automatic differentiation (unrolling) for iteration-dependent (time-varying) fixed-point algorithms, with rate guarantees, and applies it to proximal gradient descent with variable step size and FISTA/APG on partly smooth problems.
Problem
Automatic differentiation of iterative algorithms whose update map changes at each iteration (e.g., variable step size, momentum), where existing convergence results assume a pointwise-converging update map and do not give linear rates for derivative iterates.
Approach
Establishes convergence and linear convergence rates for forward-mode unrolled derivative iterates of time-varying fixed-point maps, without requiring pointwise convergence of the update maps, and transfers the results to PGD with variable step size and APG for partly smooth objectives.
Key results
- Derivative iterates of PGD with variable step size and APG converge linearly, with rate max(q_x, spectral radius of the derivative of the update map), matching the rate of the primal iterates.
- Numerical experiments on l2-regularized logistic regression and l1-regularized linear regression show the derivative error decays as fast as the algorithm's iterate error for PGD (four step-size schemes) and APG.
Limitations
- Requires contraction property near solution and partly smooth/non-degeneracy assumptions for PGD/APG results.
- Experiments limited to toy problems (lasso and l2-regularized logistic regression).
- Reverse-mode memory overhead is not addressed; results are for forward-mode style derivative recursion.
Experimental setup
Metrics: iterate error derivative iterate error (median over 50 instances)
Baselines: Beck (2018) convergence result Mehmood and Ochs (2023)
Builds on: Beck (2018) differentiation of iteration-dependent fixed-point algorithms Mehmood and Ochs (2023) automatic differentiation of proximal gradient methods Liang et al. activity identification of PGD/APG Griewank et al. preconditioned Picard iterations
Closest papers by meaning
- Stochastic Dynamic Barrier Perturbed Gradient Methods for Nonconvex Simple Bilevel Optimization (NeurIPS 2026 · similarity 0.87)
- Select-then-differentiate: Solving Bilevel Optimization with Manifold Lower-level Solution Sets (NeurIPS 2026 · similarity 0.86)
- Dynamic Anisotropic Smoothing for Noisy Derivative-Free Optimization (ICML 2024 · similarity 0.86)
- Derivatives of Stochastic Gradient Descent in parametric optimization (NeurIPS 2024 · similarity 0.86)
- An Accelerated Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness (NeurIPS 2024 · similarity 0.86)
- Accelerating Optimization via Differentiable Stopping Time (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.