Research map / Architectures and efficiency / RNNs and normalization
Expressiveness of Rectifier Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn with compressed tanh using the same number of hidden units n as the data-generating ReLU network instead of ReLUs | threshold-like networks on synthetic data · synthetic rectifier-generated data (d=3,10,50; 3 or 10 ReLUs) | test error | Hurt |
| Use 2^n compressed-tanh hidden units (threshold-equivalent construction) instead of n | compressed tanh network · synthetic rectifier-generated data, 3 ReLUs for data generation | test error | Hurt |
| Use 2^n compressed-tanh hidden units instead of n | compressed tanh network · synthetic rectifier-generated data, 10 ReLUs for data generation | test error | Helped |
| Learn with ReLU network of n hidden units (matching the generating architecture) instead of threshold-like networks | rectifier network · synthetic rectifier-generated data (six settings) | test error | Helped |
| Two-layer ReLU networks with all-positive or all-negative hidden units are expressible by threshold networks with 2^n-1 hidden units (upper bound, theoretical) | ReLU network to threshold network conversion · theoretical, d >= n | hidden threshold units required | No clear effect |
| Replace sign units with ReLUs via approximation (each sign unit approximated by 2 ReLUs) | threshold network to ReLU network conversion · theoretical | units required | No clear effect |
Closest papers by meaning
- Understanding Deep Neural Networks with Rectified Linear Units (ICLR 2018 · similarity 0.91)
- Expressivity of ReLU-Networks under Convex Relaxations (ICLR 2024 · similarity 0.90)
- The inductive bias of ReLU networks on orthogonally separable data (ICLR 2021 · similarity 0.89)
- On the Expressiveness of Rational ReLU Neural Networks With Bounded Depth (ICLR 2025 · similarity 0.88)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.88)
- Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.