Research map / Optimization / Implicit bias and generalization
Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a high-probability Rademacher-style generalization bound for gradient flow using the data-dependent loss path kernel, combining stability of gradient flow with uniform convergence, and checks it on CIFAR-10 and MNIST experiments.
Problem
Generalization of gradient-based training is only partially understood; existing algorithm-dependent bounds are data-independent, blow up with noise or time, or are hard to compute, and prior loss-path-kernel bounds need a supremum over kernels and held-out data.
Approach
Uses O(1/n) uniform argument stability of gradient flow to show loss path kernels concentrate across datasets, shrinking the function class; this yields a bound Gamma (RKHS-norm times kernel-trace form) computed from the training trajectory alone, with gradient norms along the path.
Key results
- On a two-layer NN trained on 4000 CIFAR-10 cat/dog images, the norm-based bound from prior work is 12557.3, much larger than the proposed bound Gamma, which tracks the true generalization gap.
- For ResNet-18 trained by SGD on full CIFAR-10, Gamma correlates well with the generalization gap, capturing overfitting after long training.
- With random label noise, Gamma increases with the noise fraction, tracking the growing generalization gap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the loss path kernel bound Γ instead of the norm-based bound | Two-layer NN trained by full-batch gradient descent · CIFAR-10 cat and dog (4000 examples) | generalization bound | Helped |
| Track Γ during training | Gradient-flow generalization bound · Real-world datasets | correlation with true generalization gap | Helped |
| Increase the portion of label noise | Two-layer NN trained by full-batch gradient descent · CIFAR-10 cat and dog (4000 examples) | Γ and generalization gap | Hurt |
Surprising findings
- The bound Gamma's leading term can be vanishing in the feature-learning single-index setting, while the NTK-regime bound stays Theta(1).
Limitations
- Analysis is restricted to full-batch gradient flow (infinitesimal step size); GD and SGD with finite learning rates not covered.
- Non-convex stability bound grows exponentially in T, requiring the Rademacher argument to avoid it; a fixed-learning-rate choice needed to keep LPK definition.
- Analysis uses a function class larger than the GF-explored class, which may loosen the bound.
Experimental setup
Datasets: CIFAR-10 MNIST
Benchmarks: CIFAR-10 MNIST
Metrics: generalization gap generalization bound Gamma
Baselines: norm-based bound (Lyu et al. style) NTK-based bound kernel ridge regression bound
Builds on: loss path kernel neural tangent kernel uniform argument stability Rademacher complexity
Closest papers by meaning
- Generalizability of Neural Networks Minimizing Empirical Risk Based on Expressive Power (ICLR 2025 · similarity 0.89)
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.89)
- Multiclass Loss Geometry Matters for Generalization of Gradient Descent in Separable Classification (NeurIPS 2025 · similarity 0.89)
- Fantastic Generalization Measures are Nowhere to be Found (ICLR 2024 · similarity 0.88)
- Generalization in Kernel Regression Under Realistic Assumptions (ICML 2024 · similarity 0.88)
- Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.