Research map / Architectures and efficiency / CNN design
Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that overparameterized ConvResNeXts trained with weight decay achieve near-minimax nonparametric classification rates for Besov-class functions on low-dimensional manifolds, avoiding the curse of dimensionality.
Problem
Explaining why overparameterized convolutional residual networks generalize well in nonparametric classification, where existing theory requires model size no larger than the sample size and suffers from the curse of dimensionality.
Approach
Analyzes ConvResNeXts (parallel residual blocks with convolutions) trained with weight decay as a nonparametric logistic classifier for Besov functions supported on a low-dimensional manifold; a covering-number bound via chaining shows weight decay enforces block sparsity so the number of blocks can be arbitrarily large without hurting the rate.
Key results
- ConvResNeXt attains the lowest MSE at smaller effective degrees of freedom than kernel ridge regression, GP, XGBoost, decision tree, and Lasso on the synthetic manifold task.
- Under weight decay the theory gives an estimation error of order n^{-(alpha/d)/(2alpha/d+1)} up to log factors and o(1) terms, close to the minimax rate, independent of ambient dimension D except polynomially.
- Kernel ridge regression and NTK-type methods are argued to be suboptimal, with rate lower bounded by (2alpha-d)/(2alpha) for Besov classes with p=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ConvResNeXt instead of the competing methods | Synthetic manifold regression · Synthetic manifold regression | MSE | Helped |
| Use ConvResNeXt instead of kernel ridge regression or Gaussian processes as ambient dimension increases | Synthetic manifold regression · Synthetic manifold regression across ambient dimensions | MSE | Helped |
| Use ConvResNeXt instead of kernel ridge regression or Gaussian processes as sample size increases | Synthetic manifold regression · Synthetic manifold regression across sample sizes | MSE | Helped |
Surprising findings
- Weight decay implicitly enforces sparsity over building blocks, so only finitely many blocks contribute non-trivially even under overparameterization.
- The theory suggests the product MN (paths times depth) matters, not their individual arrangement, so parallel paths may be unnecessary given residual connections.
Limitations
- The analysis decouples optimization from generalization and does not study the training dynamics of gradient descent.
- Numerical experiments are small-scale synthetic 1D manifold tasks with limited architecture settings.
- The rate has constants and exponential dependence on intrinsic dimension d, and some results depend on logarithmic terms and an unspecified o(1) gap.
Experimental setup
Datasets: synthetic 1D manifold in R^3 embedded in higher-dimensional space with irrelevant features
Benchmarks: synthetic 1D manifold regression
Metrics: mean squared error
Baselines: kernel ridge regression Gaussian processes XGBoost decision tree Lasso regression
Builds on: ConvResNeXt ConvResNet Besov space approximation theory neural tangent kernel
Closest papers by meaning
- Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint (ICML 2022 · similarity 0.91)
- Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks (ICML 2021 · similarity 0.91)
- Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive? (ICLR 2023 · similarity 0.87)
- Data Representations' Study of Latent Image Manifolds (ICML 2023 · similarity 0.86)
- Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness (NeurIPS 2025 · similarity 0.84)
- Hardness of Learning Neural Networks under the Manifold Hypothesis (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.