Research map / Architectures and efficiency / RNNs and normalization
Empirical Phase Diagram for Three-layer Neural Networks with Infinite Width
Hanxu Zhou, Qixu Zhou, Zhenyuan Jin, Tao Luo, Yaoyu Zhang, Zhi-Qin John Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Vary initialization scaling while keeping (γ2, γ3) fixed | Three-layer ReLU NN · Synthetic 4-point dataset, Group 1 (γ2=0, γ3=1.1) | S_W1 and S_W2: $S_{\vW_1}$: $-0.4108$, $-0.4084$, $-0.4231$; $S_{\vW_2}$: $-0.9119$, $-0.9364$, $-0.9310$ | No clear effect |
| Vary initialization scaling while keeping (γ2, γ3) fixed | Three-layer ReLU NN · Synthetic 4-point dataset, Group 2 (γ2=0.7, γ3=2.5) | S_W1 and S_W2: $S_{\vW_1}$: $-0.3130$, $-0.3176$, $-0.3243$; $S_{\vW_2}$: $0.0251$, $0.0263$, $0.0205$ | No clear effect |
| Increase γ3 across initialization regimes | Three-layer ReLU NN, first hidden layer · Synthetic 4-point dataset | Relative change of input weights | Mixed |
| Move into the condensed regime | Three-layer ReLU NN, hidden-layer input weights · Synthetic 4-point dataset and MNIST | Relative change of input weights | Mixed |
| Use different dynamical regimes for different hidden layers | Three-layer ReLU NN · Synthetic 4-point dataset and MNIST | Layerwise dynamical regime | Improved |
| Enter the condensed regime, where active input weights concentrate in isolated orientations | Three-layer ReLU NN · Synthetic 4-point dataset | Weight-orientation condensation | Improved |
Closest papers by meaning
- Phase Diagram for Two-layer ReLU Neural Networks at Infinite-width Limit (JMLR 2021 · similarity 0.94)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.88)
- Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization (ICLR 2024 · similarity 0.87)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.87)
- Towards the Dynamics of a DNN Learning Symbolic Interactions (NeurIPS 2024 · similarity 0.87)
- 4+3 Phases of Compute-Optimal Neural Scaling Laws (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
