Research map / Architectures and efficiency / RNNs and normalization
The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
Mufan Bill Li, Mihai Nica, Daniel M. Roy
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Simulate the covariance SDE instead of the finite-width Markov chain | Neural Covariance SDE · Shaped networks at initialization | simulation cost | Improved |
| Compare SDE predictions with simulated finite networks | Neural Covariance SDE · Shaped ReLU-like and softplus networks | match to output covariance distribution | Improved |
| Shape the activation function with depth-dependent scaling | Deep MLP at initialization · Infinite depth-and-width limit | covariance stability | Improved |
Closest papers by meaning
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.87)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.86)
- Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programs (NeurIPS 2025 · similarity 0.86)
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.85)
- Deep Kernel Posterior Learning under Infinite Variance Prior Weights (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
