Research map / Optimization / Nonconvex and smooth optimization

A Continuous-time Stochastic Gradient Descent Method for Continuous Data

NeurIPS 2024 · Optimization › Nonconvex and smooth optimization · approach: Stochastic gradient last-iterate analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies a continuous-time stochastic gradient process (gradient flow coupled with a Feller index process on a compact space) for minimizing integrals of indexed functions, proving approximation of the full gradient flow, geometric ergodicity under constant learning rate, and convergence under decreasing learning rate.

Problem

Minimizing an expectation (integral over a continuously indexed family) of loss functions, where full gradients are intractable and data are continuous, so standard SGD over finite sums does not directly apply.

Approach

Generalizes the continuous-time SGD of discrete-data gradient processes to continuous index sets by coupling a gradient flow with a general compact-state Feller index process (e.g., reflected Lévy processes, Markov jump processes), with decreasing learning rates implemented by time-rescaling the index process.

Key results

• Shows the constant-learning-rate stochastic gradient process converges weakly to the full gradient flow as the time-scaling parameter epsilon goes to 0.
• Proves geometric ergodicity of the coupled process (theta, index) with a unique stationary measure concentrating around the minimizer as epsilon goes to 0, with Wasserstein rate exp(-kappa t/2) under strong convexity.

Limitations

• Numerical experiments (polynomial regression, PINNs) are only described in the outline; the provided text covers theory only.
• Convergence results require strong convexity of the subsampled target functions uniformly in the index.
• Index process assumptions (Feller, uniform exponential coupling) restrict admissible processes.

Experimental setup

Builds on: stochastic gradient descent stochastic gradient process (discrete index)

Closest papers by meaning

• Communication-Efficient Stochastic Gradient Descent, with Applications to Neural Networks (NeurIPS 2017 · similarity 0.86)
• Distributed Stochastic Gradient Descent: Nonconvexity, Nonsmoothness, and Convergence to Local Minima (ICML 2023 · similarity 0.86)
• SGD without Replacement: Sharper Rates for General Smooth Convex Functions (ICML 2019 · similarity 0.85)
• Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent (NeurIPS 2023 · similarity 0.85)
• Controlling the Flow: Stability and Convergence for Stochastic Gradient Descent with Decaying Regularization (NeurIPS 2025 · similarity 0.83)
• Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification (ICLR 2025 · similarity 0.83)