Research map / Continual learning / Online and replay-based
How connectivity structure shapes rich and lazy learning in neural circuits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
How the structure (effective rank) of initial connectivity, at fixed weight magnitude, biases whether neural circuits and networks learn in a rich or lazy regime, given that biological connectivity is low-rank unlike the random initializations usually studied.
Approach
Defines effective richness/laziness by post-training change (weight change norm, representation alignment, kernel alignment) rather than initialization scale; proves for two-layer linear networks that high-rank isotropic init maximizes expected NTK alignment across tasks, shows rank-1 task-aligned init gives high alignment, and tests RNNs with SVD-truncated and biologically derived low-rank connec
Key results
- High-rank (SVD-truncated) initial recurrent weights yield smaller weight change, representation change and kernel change than low-rank ones across Neurogym tasks and sMNIST at fixed Frobenius norm (10 runs per setting).
- Biologically derived low-rank connectivity (cell-type statistics, EM data, Dale's law, chain motifs) produces effectively richer learning than random Gaussian initialization.
- In the two-layer linear theory, expected kernel alignment is maximized by isotropic high-rank initialization, while rank-1 init aligned with the task gives alignment approaching 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use high-rank isotropic rather than low-rank initialization | Two-layer linear network · Random tasks with whitened data | tangent kernel alignment | Helped |
| Use low-rank rather than high-rank initial recurrent weights | RNN · 2AF, DMS, CXT, and sMNIST | effective learning richness | Hurt |
| Initialize recurrent weights with biologically motivated low-rank connectivity instead of random Gaussian connectivity | RNN · 2AF | effective learning richness | Helped |
| Align low-rank initialization with task statistics | Two-layer linear network · Student-teacher task with anisotropic features | tangent kernel alignment | Helped |
| Align initialization only with dominant task features rather than all task features | Two-layer linear network · Student-teacher task with anisotropic features | tangent kernel alignment | Helped |
| Replace EM-derived connectivity with shuffled connectivity, removing its low-rank structure (ablation) | RNN · Neuroscience tasks | effective learning laziness | No clear effect |
| Vary initial weight rank from low to high (ablation) | RNN · Neuroscience tasks | effective learning laziness | Helped |
| Use soft rather than sharply truncated initial weight rank (ablation) | RNN · Neuroscience tasks | effective learning laziness | No clear effect |
| Use Uniform rather than Gaussian initialization (ablation) | RNN · Neuroscience tasks | effective learning laziness | No clear effect |
| Constrain Dale's law throughout training rather than only at initialization (ablation) | RNN · Neuroscience tasks | effective learning laziness | No clear effect |
Surprising findings
- Low-rank initialization aligned with task statistics can produce lazier learning than random high-rank initialization, an exception to the general rule.
- Shuffling the EM connectivity removes the low-rank property and the effect on laziness disappears.
Limitations
- Theory restricted to two-layer linear networks with small initialization and whitened data; RNN results are empirical only.
- Focuses only on weight rank; dynamical regime, Dale's law during training, learning speed and generalization not fully explored.
- Task-agnostic theorem is an expectation across tasks, so it does not predict behavior for specific tasks.
Experimental setup
Datasets: Neurogym PerceptualDecisionMaking Neurogym DelayMatchSample Neurogym ContextDecisionMaking sequential MNIST V1 cortical column EM connectivity
Benchmarks: Neurogym 2AF Neurogym DMS Neurogym CXT sequential MNIST
Metrics: weight change norm representation alignment tangent kernel alignment
Baselines: random Gaussian initialization (null)
Builds on: neural tangent kernel rich and lazy learning regimes Neurogym
Closest papers by meaning
- Learning Reveals Invisible Structure in Low-Rank RNNs (NeurIPS 2026 · similarity 0.89)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.89)
- Flexible task abstractions emerge in linear networks with fast and bounded units (NeurIPS 2024 · similarity 0.88)
- Structured flexibility in recurrent neural networks via neuromodulation (NeurIPS 2024 · similarity 0.87)
- Shaping the distribution of neural responses with interneurons in a recurrent circuit model (NeurIPS 2024 · similarity 0.87)
- Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.