Research map / Architectures and efficiency / RNNs and normalization
Deep ReLU Networks Have Surprisingly Few Activation Patterns
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on MNIST, including increasingly corrupted labels | Fully connected ReLU networks · MNIST, 2D cross-sections | activation regions | No clear effect |
| Increase the amount of label corruption during MNIST training | Depth 3, width 32 ReLU network · MNIST, lines through input space | activation regions | Mixed |
| Train to memorize randomly labeled 2D points | Depth 3, width 32 ReLU network · Random 2D points with binary labels | activation regions | Mixed |
| Increase memorization difficulty to the maximum tested level | Depth 3, width 32 ReLU network · Random 2D points with binary labels | training accuracy | Hurt |
| Vary the learning rate (ablation) | Depth 3, width 32 ReLU network · Random 2D points with binary labels | activation regions | No clear effect |
| Scale weights at initialization (ablation) | Depth 3, width 32 ReLU network · 5000 random 2D points with independent binary labels | activation regions | No clear effect |
| Use larger weight scale at initialization (ablation) | Depth 3, width 32 ReLU network · Random 2D points with binary labels | activation regions | Mixed |
| Initialize biases to zero (ablation) | ReLU networks · Activation regions | activation regions | Hurt |
Closest papers by meaning
- The Expressive Power of Neural Networks: A View from the Width (NeurIPS 2017 · similarity 0.91)
- On the Expected Complexity of Maxout Networks (NeurIPS 2021 · similarity 0.90)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.89)
- Activation-Descent Regularization for Input Optimization of ReLU Networks (ICML 2024 · similarity 0.88)
- On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth (ICLR 2025 · similarity 0.88)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.