Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
When narrower is better: the narrow width limit of Bayesian parallel branching neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically and empirically studies narrow-width limit of Bayesian parallel branching (GCN and residual-MLP) networks, showing kernel renormalization breaks branch symmetry and can lower bias versus wide/GP limit.
Problem
Infinite-width NNGP limits give task-independent kernels and the common belief that wider networks generalize better; the paper asks how narrow width affects generalization in parallel branching networks in the Bayesian setting.
Approach
Derives an exact effective Hamiltonian with order parameters for linear Bayesian branching GCNs via kernel renormalization, proves an equipartition theorem showing student branch readout norms match teacher branch norms in the narrow-width limit (alpha=P/N to infinity), and shows bias can decrease at narrow width.
Key results
- Student branch readout norms at narrow width (N=4) concentrate at the teacher's branch norm values, while at N=1024 they remain near the GP limit value.
- In bias-limited settings the bias decreases and saturates at narrow width, so narrow BPB-NNs can match or beat wide ones in generalization.
- On Cora, the BPB-GCN with L=6 branches shows the last branch has the largest readout norm at narrow width, and first two branches converge robustly across L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use narrow rather than wide hidden width | BPB-GCN student in the student-teacher task · Student-teacher CSBM | branch readout norm matching | Helped |
| Increase hidden width toward N=1024 | BPB-GCN student in the student-teacher task · Student-teacher CSBM | branch readout norm matching | Hurt |
| Use narrower network width | BPB-GCN · Cora | bias | Helped |
| Add more branches that include higher-order convolutions | BPB-GCN · Cora | accuracy | Helped |
| Use narrower network width | Residual-MLP · Student-teacher task | bias | Helped |
| Use narrower network width | Residual-MLP · Cifar10 | bias | Helped |
| Increase network width | Residual-MLP · Cifar10 | test accuracy | Hurt |
| Use narrow rather than wide width | BPB-GCN and residual-MLP · Cora and Cifar10 | generalization error | Mixed |
Surprising findings
- Narrower networks (width far below sample size) can generalize better than wide ones, contrary to the common belief that wider is better.
- On Cifar10 with residual-MLP, the ReLU branch completely dominates at narrow width, and generalization is variance-limited despite bias decreasing.
Limitations
- Theory is for linear networks and the lazy, offline Bayesian (posterior sampling) regime; extension to online training and feature learning is hypothesized, not shown.
- On real datasets (Cora, Cifar10) generalization error is usually variance-limited, and the narrow-width limit is hard to reach within realistic widths on Cora.
- Discrepancies between theory and HMC sampling for Cifar10 bias due to sample correlations under compute budget.
Experimental setup
Datasets: Cora CIFAR-10 contextual stochastic block model (synthetic)
Benchmarks: Cora CIFAR-10 (airplane vs cat) synthetic student-teacher
Metrics: bias variance generalization MSE test accuracy branch readout norms
Baselines: GP / infinite-width limit
Builds on: NNGP kernel renormalization (feature learning in MLPs) linearly decoupled GCN contextual stochastic block model
Closest papers by meaning
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent (NeurIPS 2019 · similarity 0.89)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.88)
- The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis (NeurIPS 2025 · similarity 0.87)
- Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks (NeurIPS 2026 · similarity 0.86)
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.86)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.