Research map / Architectures and efficiency / RNNs and normalization
Make Haste Slowly: A Theory of Emergent Structured Mixed Selectivity in Feature Learning ReLU Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that finite ReLU networks can be exactly imitated by Gated Deep Linear Networks (ReLNs), and uses this to derive learning dynamics that reveal an inductive bias toward structured mixed-selective, node-reusing latent representations in a contextual multi-task setting.
Problem
A theory of feature learning in finite-width ReLU networks is lacking, since existing theories rely on linearity, unstructured data, infinite width, or single hidden layers.
Approach
Establish an equivalence between ReLU networks and Gated Deep Linear Networks by constructing a Rectified Linear Network (ReLN): a GDLN with gates matching the ReLU network's activation patterns at all times, whose linear pathways admit closed-form learning dynamics via the neural race reduction.
Key results
- ReLN loss trajectories closely match the ReLU network loss dynamics on the XoR-style task, with the ReLU network behaving like the faster GDLN and transitioning to a nonlinear boundary at Delta = sqrt(2/3).
- On the contextual task, the ReLU network has no context-specific neurons; the matching ReLN has one common pathway active in all contexts plus contextual pathways active in C-1 contexts, and the pattern persists for 3-5 contexts.
- With a second hidden layer, the first layer is shared across all contexts and linear, and the ReLU network and ReLN both show run-to-run variance in loss trajectories due to delayed singular vector alignment.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of contexts from three to five | ReLU network on the contextual task · Contextual nonlinear task | node reuse and mixed selectivity | Helped |
| Add a second hidden layer with ReLU activation | ReLU network on the contextual task · Contextual nonlinear task | node reuse and mixed selectivity | No clear effect |
| Use contextual pathways active in individual contexts instead of pathways active in two contexts (ablation) | ReLN · Contextual nonlinear task | loss-trajectory match to the ReLU network | Hurt |
| Use pathways active in two contexts rather than pathways active in individual contexts (ablation) | ReLN · Contextual nonlinear task | learning speed | Helped |
| Add a common pathway active for all contexts and contextual pathways active for C-1 contexts | ReLN · Contextual task with three, four, or five contexts | node reuse | Helped |
| Use the fastest implementable GDLN architecture, with pathways reusing nodes across contexts | ReLU network on the contextual task · Contextual nonlinear task | learning speed | Helped |
Surprising findings
- ReLU networks transition to a nonlinear decision boundary well before the problem becomes nonlinearly separable (around Delta = sqrt(2/3)).
- The learned ReLU network contains no context-specific neurons; mixed selectivity emerges as the fastest-learning strategy.
Limitations
- Equivalence is only insightful when the resulting GDLN has few pathways; many pathways become as hard to interpret as neurons.
- Gating structure identification relies on a simple k-means clustering approach and is not trivially obtained.
- Dynamics analysis assumes mutually diagonalizable correlation matrices and weight alignment, which can be violated in deeper networks.
Experimental setup
Datasets: synthetic XoR-style task synthetic hierarchical item-property contextual dataset
Metrics: training loss trajectory learning time singular value trajectories multi-dimensional scaling of latent representations
Baselines: GDLN Single NTK lazy regime
Builds on: Gated Deep Linear Networks deep linear network dynamics (Saxe et al.) neural race reduction silent alignment effect
Closest papers by meaning
- On the Local Complexity of Linear Regions in Deep ReLU Networks (ICML 2025 · similarity 0.88)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.87)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.87)
- Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.87)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.86)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.