Research map / Architectures and efficiency / RNNs and normalization
On the Expressive Power of Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase depth | Fully connected networks with piecewise-linear activations · Activation patterns over input space | activation patterns: O(k^{mn}) | Helped |
| Increase depth | Random ReLU networks · One-dimensional input trajectories | trajectory length: exponentially | Helped |
| Increase depth | Fully connected networks · Empirical transition counts | transitions: exponentially | Helped |
| Increase width | Fully connected networks · Empirical transition counts | transitions | Mixed |
| Add noise of magnitude 0.25 to first-layer weights | Trained CIFAR-10 convolutional network · CIFAR-10 | accuracy: 0.7 drop | Hurt |
| Add noise of magnitude 0.25 to fifth-layer weights | Trained CIFAR-10 convolutional network · CIFAR-10 | accuracy: 0.02 drop | Hurt |
| Train a lower hidden layer instead of a higher hidden layer | Fully connected hard-tanh network with other layers frozen · MNIST | accuracy | Helped |
| Add batch normalization layers | Convolutional network · CIFAR-10 | trajectory length | Hurt |
| Add trajectory regularization in place of batch normalization | Convolutional network · CIFAR-10 | performance | No clear effect |
| Remove batch normalization (ablation) | Convolutional network · CIFAR-10 | performance | Hurt |
| Apply pooling | Convolutional network · CIFAR-10 | trajectory length | Hurt |
| Train the network without batch normalization | Convolutional networks · CIFAR-10 and MNIST | trajectory length | Helped |
Closest papers by meaning
- When Expressivity Meets Trainability: Fewer than $n$ Neurons Can Work (NeurIPS 2021 · similarity 0.88)
- Is Deeper Better only when Shallow is Good? (NeurIPS 2019 · similarity 0.87)
- Characterizing Well-Behaved vs. Pathological Deep Neural Networks (ICML 2019 · similarity 0.87)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.86)
- What Expressivity Theory Misses: Message Passing Complexity for GNNs (NeurIPS 2025 · similarity 0.85)
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.