Research map / Learning theory and algorithms / Sample complexity
On the Depth of Deep Neural Networks: A Theoretical View
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add squared margin-gap penalty C1 to cross-entropy | DNN-C · MNIST | test error (%): 0.899\pm0.038 -> 0.734\pm0.046 | Helped |
| Add squared margin-gap penalty C1 to cross-entropy | DNN-C · CIFAR-10 | test error (%): 18.339\pm0.336 -> 17.598\pm0.274 | Helped |
| Add average squared class-margin-gap penalty C2 to cross-entropy (ablation) | DNN-C · MNIST | test error (%): 0.899\pm0.038 -> 0.736\pm0.041 | Helped |
| Add average squared class-margin-gap penalty C2 to cross-entropy (ablation) | DNN-C · CIFAR-10 | test error (%): 18.339\pm0.336 -> 17.728\pm0.283 | Helped |
| Use C2 instead of C1 (ablation) | LMDNN · MNIST and CIFAR-10 | test error (%): MNIST: 0.734\pm0.046 -> 0.736\pm0.041; CIFAR-10: 17.598\pm0.274 -> 17.728\pm0.283 | Mixed |
| Vary margin penalty coefficient lambda (ablation) | LMDNN-C1 and LMDNN-C2 · MNIST and CIFAR-10 | test error | Mixed |
| Use C2 rather than C1 (ablation) | LMDNN · MNIST and CIFAR-10 | test error across lambda settings | Mixed |
| Increase depth while restricting the total number of hidden units (ablation) | Fully connected DNN · MNIST and CIFAR-10 | test error | Mixed |
| Increase network depth | DNN margin bound · Theoretical analysis | Rademacher average upper bound | Hurt |
| Increase network depth | DNN · Multi-class representation complexity | Betti-numbers-based complexity | Helped |
| Add a margin-based penalty term | DNN-C · MNIST and CIFAR-10 | empirical margin error | Helped |
Closest papers by meaning
- Is Deeper Better only when Shallow is Good? (NeurIPS 2019 · similarity 0.89)
- Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer Margin (ICLR 2020 · similarity 0.87)
- Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization) (NeurIPS 2022 · similarity 0.87)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.87)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.85)
- Input Margins Can Predict Generalization Too (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.