Research map / Optimization / Optimizers
Generalization Analysis of Stochastic Weight Averaging with General Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical stability-based generalization bounds for stochastic weight averaging (SWA) in convex and non-convex settings, with and without replacement sampling, showing sharper bounds than SGD, supported by small MNIST/CIFAR10/Adult experiments.
Problem
Generalization of stochastic weight averaging is theoretically understood only for convex losses with sampling with replacement; non-convex losses and sampling without replacement are unanalyzed.
Approach
Uses uniform stability with an induction-based recursive bound on the historical gradient terms of SWA, and combinatorial probability to handle sampling without replacement, yielding bounds sharper than SGD's in each setting.
Key results
- Convex with replacement: SWA stability bound O(T/n) vs SGD O(2T/n) (Theorem 4.1); convex without replacement: SWA O(ln T/n) vs SGD O(T/n) (Theorem 4.2).
- Non-convex: SWA bound O(T^{cβ/(cβ+2)}/n) versus SGD O(T^{cβ/(cβ+1)}/n) with replacement (Theorem 5.1), and the same rate without replacement (Theorem 5.2).
- Experiments on MNIST, CIFAR10 and Adult show SWA gives smaller parameter differences and generalization error than SGD in both sampling regimes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SWA instead of SGD | SWA versus SGD · Convex, sampling with replacement | generalization bound: O(2T/n) -> O(T/n) | Helped |
| Use SWA instead of SGD | SWA versus SGD · Convex, sampling without replacement | generalization bound: O(T/n) -> O(ln T/n) | Helped |
| Use SWA instead of SGD | SWA versus SGD · Non-convex, sampling with replacement | generalization bound: O(T^{cβ/(1+cβ)}/n) -> O(T^{cβ/(2+cβ)}/n) | Helped |
| Use SWA instead of SGD | SWA versus SGD · Non-convex, sampling without replacement | generalization bound: O(T^{cγ/(1+cγ)}/n) -> O(T^{cβ/(2+cβ)}/n) | Mixed |
| Use SWA instead of SGD | SWA versus SGD · MNIST, sampling with replacement | generalization error | Helped |
| Use SWA instead of SGD | SWA versus SGD · CIFAR10, sampling with replacement | generalization error | Helped |
| Use SWA instead of SGD | SWA versus SGD · MNIST, sampling without replacement | generalization error | Helped |
| Use SWA instead of SGD | SWA versus SGD · Adult, sampling without replacement | generalization error | Helped |
Surprising findings
- Sampling without replacement yields sharper bounds than with replacement, but the authors attribute this to the T ≤ n assumption rather than the sampling scheme itself.
Limitations
- Relies on classical assumptions (L-Lipschitz, β-smooth) that may be restrictive in practice.
- Non-convex with-replacement bound uses g in [0,1] for presentation; experiments use small models and only batch-size-128 or batch-size-1 settings differing from theory.
Experimental setup
Datasets: MNIST CIFAR10 Adult
Benchmarks: MNIST CIFAR10 Adult
Metrics: parameter distance generalization error train accuracy test accuracy
Baselines: SGD
Builds on: uniform stability (Hardt et al., 2016) Kuzborskij & Lampert (2018) SGD without replacement stability Stochastic weight averaging (Izmailov et al., 2018)
Closest papers by meaning
- Error Analysis Affected by Heavy-Tailed Gradients for Non-Convex Pairwise Stochastic Gradient Descent (AAAI 2025 · similarity 0.88)
- Generalization Analysis of Biased Stochastic Gradient Methods for Minimax Problems (NeurIPS 2026 · similarity 0.87)
- Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression (NeurIPS 2024 · similarity 0.87)
- Stability and Generalization of Asynchronous SGD: Sharper Bounds Beyond Lipschitz and Smoothness (NeurIPS 2024 · similarity 0.87)
- Uniform Stability and Generalization Error of GD and SGD on Fixed-Point Parameters (NeurIPS 2026 · similarity 0.87)
- Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.