Research map / Learning theory and algorithms / Kernels and regression theory
Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares deeper vs wider ReLU/super-ReLU networks by optimal generalization error (approximation plus sampling error) under Sobolev H^k losses, and applies the bounds to deep Ritz and PINN for Poisson equations.
Problem
Whether, for a fixed parameter budget and Sobolev-type training loss, deeper (flexible depth, narrow) or wider (shallow, flexible width) networks achieve better generalization error, given finite samples.
Approach
Derives generalization error bounds for deep networks (width O(log L), depth O(L log L)) under H^0, H^1, H^2 losses via uniform covering numbers and pseudo-dimension, without uniform parameter bounds, and compares them with the shallow-network rate to show when depth wins based on sample count and loss regularity.
Key results
- Deep networks with W=O(L(log L)^3) parameters achieve generalization error O((W/(log W)^2)^{-4(n-k)/d} + (W^2/M) log M) for H^k losses, k=0,1,2.
- With W fixed, deep networks beat wide networks when M >= W^{(2n+2d-2k)/d}, per the paper's comparison with shallow-network bounds.
- A small experiment (Table) shows the shallow net (depth 1, width 20) does better in the small-data regime (test mean 1.51e-3 vs 4.96e-3) while the deep net (depth 4, width 10) does better in the large-data regime (3.69e-4 vs 6.18e-4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeNNs instead of WeNNs when sample points are abundant and parameter count is limited | Neural networks under Sobolev loss · Supervised learning; L², H¹, and H² losses | generalization error | Helped |
| Use WeNNs instead of DeNNs when parameter count is large and sample points are limited | Neural networks under Sobolev loss · Supervised learning; L², H¹, and H² losses | generalization error | Helped |
| Increase the derivative order in the loss function | DeNNs versus WeNNs · Sobolev training | generalization error | Mixed |
| Use the deeper, narrower architecture (Depth 4, Width 10) rather than the shallow, wider architecture (Depth 1, Width 20) | Neural networks · Large data regime | mean test performance: Mean: 6.18e-4 -> 3.69e-4 | Helped |
| Use the deeper, narrower architecture (Depth 4, Width 10) rather than the shallow, wider architecture (Depth 1, Width 20) | Neural networks · Small data regime | mean test performance: Mean: 1.51e-3 -> 4.96e-3 | Hurt |
Surprising findings
- Very deep networks without parameter-norm control can have larger sample error, so super-convergence approximation does not automatically translate into better generalization.
- Higher-order Sobolev losses (larger k) shift the region where deep networks are preferable.
Limitations
- Analysis restricted to the underparameterized regime; overparameterized case left to future work.
- Results are upper bounds up to logarithmic factors; the sample-error bound for the deep case depends on the uniform covering number and is not fully tight in parameters.
- Experiments are small and only loosely corroborate the theory.
Experimental setup
Metrics: generalization error bound (approximation + sampling error)
Baselines: shallow neural network generalization bounds
Builds on: Sobolev training deep Ritz method physics-informed neural networks deep super ReLU networks
Closest papers by meaning
- Investigating the Overlooked Hessian Structure: From CNNs to LLMs (ICML 2025 · similarity 0.88)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.88)
- Sensitivity and Generalization in Neural Networks: an Empirical Study (ICLR 2018 · similarity 0.88)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.87)
- Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms (NeurIPS 2024 · similarity 0.86)
- Improving Generalization of Deep Neural Networks by Optimum Shifting (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.