Research map / Learning theory and algorithms / Sample complexity
Some Fundamental Aspects about Lipschitz Continuity of Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirically characterises the Lipschitz constant of standard (unregularised) neural networks by tracking a local-Jacobian lower bound and a product-of-layer-norms upper bound, revealing lower-bound fidelity, training-time growth, width-wise double descent, and label-noise effects.
Problem
Lipschitz constants of modern over-parameterised networks are hard to compute tightly and studies are limited to small datasets; it is unclear how the effective Lipschitz constant evolves during training, behaves with width, and responds to label noise.
Approach
Track the simplest lower bound (sup of local Jacobian norms over training data) together with the simplest upper bound (product of per-layer spectral norms), argue the lower bound is the faithful proxy for the effective Lipschitz constant, and use these bounds to study training dynamics, double descent, and label noise at scale (ResNet50/ViT on ImageNet).
Key results
- On a toy 2D example, the lower bound 135.8 is close to the densely sampled effective Lipschitz 144.2, while the upper bound is 389.1.
- For ResNet50 on ImageNet, the upper bound becomes vacuously large, with a gap of about 40 orders of magnitude to the lower bound.
- Lipschitz bounds for CNNs on CIFAR-100 grow until the interpolation threshold and then decrease with width, mirroring the test loss (Lipschitz double descent).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the FCN ReLU classifier to convergence | FCN ReLU · Synthetic 3-class toy dataset | Lipschitz bound: 135.815; 389.097; 144.194 | Mixed |
| Estimate the lower bound using local Jacobian norms rather than pairs of samples | FCN ReLU · Training epochs in the lower-bound comparison | Lipschitz lower-bound estimate: 0.369 vs 0.165; 1.336 vs 0.865; 3.762 vs 2.752; 5.775 vs 4.257; 7.142 vs 5.120; 7.974 vs 5.591; 9.928 vs 6.361; 19.390 v | Helped |
| Continue training from initialization | FCN ReLU networks · Training evolution | C_lower and C_upper | Hurt |
| Increase CNN width beyond the interpolation threshold | CNN · CIFAR-100 | Lipschitz bounds | Helped |
| Increase network width beyond the interpolation threshold | Neural networks · Double Descent settings | Test loss | Helped |
| Add explicit Lipschitz regularization | Neural networks in Double Descent settings · Double Descent | Double Descent trend in Lipschitz bounds | Helped |
| Evaluate lower Lipschitz on adversarially perturbed training samples | CIFAR-10 models · CIFAR-10 | Lower Lipschitz bound | Helped |
| Evaluate lower Lipschitz on out-of-distribution samples | MNIST1D models · MNIST1D | Lower Lipschitz bound | Helped |
| Increase label shuffling while varying network width | CNN · CIFAR-10 | Lower Lipschitz value | Mixed |
| Increase network capacity for fitting random labels | CNN · CIFAR-10 with shuffled labels | Lower Lipschitz value | Helped |
| Increase label-noise strength relative to network capacity | CNN · CIFAR-10 with shuffled labels | Lower Lipschitz value | Mixed |
Surprising findings
- Upper bound at initialisation for ResNet50 is already of order 1e27, attributed to exponential growth with depth.
- Adding label noise shifts the memorisation threshold to higher parameter counts, and heavily over-parameterised networks fit random labels more smoothly than smaller ones.
Limitations
- Upper bound is vacuously loose for deep networks, so conclusions rely mainly on the lower bound, which is not a certified bound.
- Attention layers are not Lipschitz-bounded, so only lower bounds are reported for ViTs.
- ImageNet results use subsampled data and checkpoints due to computational cost.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 MNIST MNIST1D ImageNet
Benchmarks: CIFAR-10 CIFAR-100 MNIST MNIST1D ImageNet
Metrics: lower Lipschitz bound (sup local Jacobian norm) upper Lipschitz bound (product of layer spectral norms) train loss test loss
Baselines: C_alt.lower C_avg_norm (geometric complexity)
Builds on: AutoLip double descent margin-based generalisation bounds
Closest papers by meaning
- Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective (NeurIPS 2022 · similarity 0.92)
- Width Independent Bounds for the Local Lipschitz Constant of Deep Neural Networks at Random Initialization and after Lazy Training (ICML 2026 · similarity 0.92)
- A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness (NeurIPS 2024 · similarity 0.90)
- ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks (NeurIPS 2024 · similarity 0.90)
- Achieving Domain-Independent Certified Robustness via Knowledge Continuity (NeurIPS 2024 · similarity 0.88)
- Approximation theory for 1-Lipschitz ResNets (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.