Research map / Optimization / Nonconvex and smooth optimization

Generalization of noisy SGD in unbounded non-convex settings

ICML 2025 · Optimization › Nonconvex and smooth optimization · approach: Langevin sampling for nonconvex problems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives time-independent KL and Rényi stability bounds for SGLD on smooth, unbounded non-convex losses under dissipativity (uniform LSI) or ergodicity, yielding generalization and DP guarantees that do not diverge with iterations.

Problem

Generalization of noisy iterative gradient methods (SGLD) on unbounded non-convex losses: prior stepwise analyses give bounds that grow with iteration count, leaving vacuous bounds for long training runs.

Approach

Stability analysis of two parallel SGLD chains on neighboring datasets via a per-iteration expansion (gradient step) followed by approximate contraction (split Gaussian heat flow), using a uniform log-Sobolev inequality of the iterates obtained under dissipativity (or Gaussian convolution plus ergodicity) instead of strong convexity.

Key results

• Under dissipativity, KL and Renyi stability of SGLD iterates are bounded uniformly in iteration count, giving O(1/sqrt(n)) generalization bounds that decay to zero as n grows.
• Uniform log-Sobolev constant for SGLD iterates is established under dissipativity (Theorem 5.5), previously known only under strong convexity.
• Without dissipativity, under ergodicity and an LSI on the Gibbs target, KL stability holds with a dimension-polynomial bound (Corollary 6.6).

Surprising findings

• Noisy iterative schemes can run indefinitely with non-vanishing step sizes without early stopping while keeping finite generalization bounds in non-convex settings.

Limitations

• Uniform LSI constant under dissipativity is exponential in dimension.
• The ergodicity-based result includes additional non-stability constants and is not matching the strongly convex rate.
• Requires smoothness and uniform dissipativity or pseudo-Lipschitz/bounded sensitivity assumptions.

Experimental setup

Metrics: KL stability bound Renyi divergence bound generalization gap bound differential privacy epsilon

Baselines: Chourasia et al. 2021 Ye & Shokri 2022 Futami & Fujisawa 2024 Farghly & Rebeschini 2021 Wang et al. 2023 Zhu et al. 2024 Raginsky et al. 2017

Builds on: Xu & Raginsky 2017 information-theoretic generalization Chourasia et al. 2021 SGLD privacy analysis Vempala & Wibisono 2019 Futami & Fujisawa 2024 Chen et al. 2022 contraction along heat flow

Closest papers by meaning

• Time-independent Generalization Bounds for SGLD in Non-convex Settings (NeurIPS 2021 · similarity 0.95)
• Time-Independent Information-Theoretic Generalization Bounds for SGLD (NeurIPS 2023 · similarity 0.94)
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates (NeurIPS 2019 · similarity 0.92)
• On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning (ICLR 2020 · similarity 0.92)
• Optimizing Information-theoretical Generalization Bound via Anisotropic Noise of SGLD (NeurIPS 2021 · similarity 0.92)
• The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.89)