Research map / Optimization / Implicit bias and generalization

Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processes

NeurIPS 2025 · Optimization › Implicit bias and generalization · approach: SGD noise and flat minima · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that any Markov training process with a Gibbs-style stationary distribution has a generalization gap bound, at any time, of order sqrt((beta E L(theta_0) + ln(1/delta))/N), via a second-law divergence bound; applied to Langevin dynamics.

Problem

Generalization gap of over-parameterized models trained with stochastic Markov training dynamics (e.g., Langevin dynamics), where existing bounds depend on training time, mixing, dimensionality, or gradient statistics.

Approach

Bounds the KL and Rényi-infinity divergence of the marginal distribution at any time from a data-independent base measure using a generalized second law (data processing monotonicity relative to a Gibbs stationary distribution), then plugs into PAC-Bayes bounds. The symmetric divergence of a Gibbs distribution removes the partition function, so no mixing, spectral gap or time dependence is needed.

Key results

• For regularized or bounded Langevin dynamics with p_0 matching the stationary base measure, the generalization gap at any time is bounded by sqrt((beta E L(theta_0)+ln(N/delta))/(2N)) with no dependence on training time or dimension.
• Inverse temperature beta = O(N) can be tuned to give non-vacuous bounds (bound plus training error below 0.5) on small binary classification tasks with overparameterized networks.
• In linear regression with CLD, the bound is useful for d << beta << N, giving a vanishing gap since E_{p0}L=O(1).

Measured effects reported in the paper

Surprising findings

• Generalization guarantees hold at every time step without mixing, and the bound does not depend on dimension, gradient norms or training time.
• Symmetrized divergence removes the partition function from the analysis, and the bound is insensitive to whether training converged.

Limitations

• Requires regularization or bounded domain matching the initialization; unregularized training with stationary probability measures is not covered.
• Requires beta = O(N) for non-vacuous bounds, and discrete-time SGD needs an explicit stationary distribution or Metropolis correction.
• Only the generalization gap is bounded, not the absolute error; bound is not very tight at noise levels allowing non-vacuous results.

Experimental setup

Datasets: binary MNIST Fashion MNIST SVHN parity problem synthetic linear regression

Benchmarks: binary MNIST Fashion MNIST SVHN parity problem

Metrics: generalization gap bound training error sum of bound and training error

Baselines: Chen et al. CLD stability bound Farghly et al. bounds for CLD Li et al. SGLD bound Raginsky et al. Gibbs-based bounds

Builds on: PAC-Bayes bounds Langevin dynamics Cover's second law / data processing inequality

Closest papers by meaning

• Time-Independent Information-Theoretic Generalization Bounds for SGLD (NeurIPS 2023 · similarity 0.90)
• Generalization Bounds for Gradient Methods via Discrete and Continuous Prior (NeurIPS 2022 · similarity 0.89)
• Time-independent Generalization Bounds for SGLD in Non-convex Settings (NeurIPS 2021 · similarity 0.89)
• Langevin Dynamics with Continuous Tempering for Training Deep Neural Networks (NeurIPS 2017 · similarity 0.89)
• Generalization of noisy SGD in unbounded non-convex settings (ICML 2025 · similarity 0.87)
• Langevin Monte Carlo for strongly log-concave distributions: Randomized midpoint revisited (ICLR 2024 · similarity 0.87)