Research map / Architectures and efficiency / RNNs and normalization
Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains leaky RNNs with trainable single-neuron timescales on N-parity and N-delayed match-to-sample tasks under single-head vs multi-head curricula, showing multi-head training relies on network-mediated timescales and scales to larger N.
Problem
How single-neuron timescales (membrane-like time constants) and network-mediated timescales contribute to solving memory-demanding tasks with increasing memory requirements, and how the learning curriculum shapes this.
Approach
Compares two curricula: single-head (sequentially train a single N) versus multi-head (keep all earlier N readout heads and add a new one), finding that multi-head networks keep single-neuron timescales near 1 and build long timescales through recurrent connectivity, which trains faster, is more robust, and reaches larger N.
Key results
- Multi-head networks reliably reach N >= 100 on both tasks, while single-head networks reach about N=35 on N-parity and about N=90 on N-DMS.
- Multi-head networks keep mean single-neuron timescale tau near 1 and perform as well as trained-tau networks, whereas single-head networks degrade sharply with fixed tau >= 2.
- Multi-head networks are more robust to ablation of long-timescale neurons and to perturbations of W^R and tau, and retrain faster to larger unseen N.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a multi-head curriculum instead of a single-head curriculum | Leaky RNN · N-parity | maximum N solved: N \approx 35 -> N \geq 100 | Helped |
| Use a multi-head curriculum instead of a single-head curriculum | Leaky RNN · N-DMS | maximum N solved: N \approx 90 -> N \geq 100 | Helped |
| Use a multi-head curriculum instead of a single-head curriculum | Leaky RNN · N-parity and N-DMS | training epochs to reach 98\% accuracy | Helped |
| Use a multi-head curriculum instead of no curriculum | Leaky RNN · N-parity and N-DMS | maximum N solved | Helped |
| Train on multiple N tasks concurrently instead of only the current N | GRU and LSTM · N-parity | training speed | Helped |
| Fix single-neuron timescales to \tau = 1 rather than training \tau (ablation) | Multi-head leaky RNN · N-parity | training performance | No clear effect |
| Fix single-neuron timescales to \tau \geq 2 rather than training \tau (ablation) | Multi-head leaky RNN · N-parity | training performance | Hurt |
| Fix single-neuron timescales instead of training \tau (ablation) | Single-head leaky RNN · N-parity | training performance | Hurt |
| Ablate long-timescale neurons (ablation) | Single-head leaky RNN · large-N memory tasks | relative accuracy | Hurt |
| Ablate short-timescale neurons (ablation) | Multi-head leaky RNN · large-N memory tasks | relative accuracy | Hurt |
| Perturb recurrent connectivity and single-neuron timescales | Multi-head rather than single-head leaky RNN · N-parity, N=30 | relative accuracy | Helped |
| Retrain without a curriculum for a higher N | Multi-head rather than single-head leaky RNN · N-parity, starting from N=16 | relative accuracy after 20 epochs | Helped |
Surprising findings
- Trainable single-neuron timescales offer no advantage over fixed tau=1 in the multi-head curriculum.
- Single-head networks suffer catastrophic forgetting of smaller N, and networks trained directly on multi-N tasks spontaneously learn a multi-head-like curriculum.
Limitations
- Only two relatively simple synthetic tasks with controllable memory requirements were studied.
- Multi-head curriculum applies only to tasks where multiple task versions can be performed on the same data.
- Model is a simplified leaky RNN, not spiking or with other biological timescale mechanisms.
Experimental setup
Datasets: synthetic N-parity sequences synthetic N-delayed match-to-sample sequences
Benchmarks: N-parity N-delayed match-to-sample
Metrics: maximum trained N at 98% accuracy relative accuracy autocorrelation timescale dimensionality (90% variance PCs) training epochs
Baselines: single-head curriculum no curriculum fixed tau RNN GRU LSTM
Builds on: leaky RNN with trainable time constants curriculum learning
Closest papers by meaning
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.89)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.89)
- Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks (NeurIPS 2025 · similarity 0.89)
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
- Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.