Research map / Architectures and efficiency / RNNs and normalization
Understanding Deep Neural Networks with Rectified Linear Units
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Search activation partitions and top-layer signs, then optimize each candidate with a convex program | Training 2-layer ReLU DNNs by empirical risk minimization · Empirical risk minimization | optimality | Helped |
| Use greater depth to represent constructed hard functions | R→R ReLU DNNs · Depth-size trade-off | minimum size required by shallower networks: at least \frac12k'w^{\frac{k}{k'}}-1 | Helped |
| Compose oscillating piecewise-linear functions across layers | R→R ReLU DNNs · Functions with w^k pieces | minimum size at depth k+1: \frac12kp^{1/k} - 1 | Helped |
| Use a zonotope support function followed by layered piecewise-linear composition | R^n→R ReLU DNNs · Zonotope hard-function family | number of affine pieces: \left ( \sum_{i=0}^{n-1}{m-1 \choose i} \right ) w^k | Helped |
| Represent arbitrary continuous piecewise-linear functions with a bounded-depth ReLU DNN | R^n→R functions · Continuous piecewise-linear functions | representability: \lceil \log_2(n+1) \rceil + 1 | Helped |
Closest papers by meaning
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.91)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.90)
- Characterizing ResNet's Universal Approximation Capability (ICML 2024 · similarity 0.90)
- On the Local Complexity of Linear Regions in Deep ReLU Networks (ICML 2025 · similarity 0.90)
- On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth (ICLR 2025 · similarity 0.89)
- ReLU Network with Width $d+\mathcal{O}(1)$ Can Achieve Optimal Approximation Rate (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.