Research map / Architectures and efficiency / RNNs and normalization
Dynamical phases of short-term memory mechanisms in RNNs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analytical toy models and ~80,000 trained RNNs show that short-term memory in RNNs is implemented by either slow-point manifolds or limit cycles, with critical learning rates scaling as a power law of delay length.
Problem
How do RNNs implement short-term memory via sequential neural activity, which latent mechanism (slow-point manifolds vs limit cycles) is learned, and how does this depend on task design, learning rate and delay duration?
Approach
Introduces limit cycles as an alternative to slow-point manifolds for generating neural sequences in delayed response tasks, derives toy-model scaling laws for maximum learning rate versus delay (beta_SP in [4,5], beta_LC in [2,3]), and shows that adding a post-reaction window biases RNNs toward limit cycles.
Key results
- Full-rank RNN experiments give critical learning-rate scaling exponents beta_LC = 2.72 ± 0.07 for limit cycles and beta_SP = 4.05 ± 0.10 for slow-point manifolds, matching theoretical ranges.
- Adding a 30 ms post-reaction window causes networks to predominantly learn limit cycles and eliminates slow-point solutions in the experiments.
- A control task without memory maintenance shows a substantially weakened delay dependence (beta = 0.38 ± 0.02).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the delay duration | Full-rank RNNs on delayed cue-discrimination · Delayed cue-discrimination | learned mechanism | Mixed |
| Add a post-reaction window (ablation) | Full-rank RNNs on delayed cue-discrimination · Delayed cue-discrimination | learned mechanism | Helped |
| Add a post-reaction window (ablation) | Rank-2 RNNs on delayed activation · Delayed activation | limit-cycle frequency | Helped |
| Use the modified task with no delayed output (ablation) | Full-rank RNNs · Delayed cue-discrimination without memory maintenance | scaling exponent: β = 0.38 ± 0.02 | Mixed |
| Increase the learning rate | Rank-2 RNNs on delayed activation · Delayed activation | task-solving ability | Hurt |
| Decrease the learning rate from a moderate to the lowest tested rate | Rank-2 RNNs on delayed activation · Delayed activation | learned mechanism | Mixed |
| Train without a post-reaction window (ablation) | Rank-2 RNNs on delayed activation · Delayed activation | learned mechanism | Mixed |
| Use the slow-point mechanism instead of the limit-cycle mechanism | Delayed-response toy models · Delayed activation | maximum learning-rate scaling exponent: β_SP ∈ [4,5]; β_LC ∈ [2,3] | Hurt |
| Remove the requirement to maintain memory (ablation) | Full-rank RNN control · Delayed cue-discrimination without memory maintenance | dependence on task duration: β = 0.38 ± 0.02 | Helped |
Surprising findings
- A brief post-reaction period, a trivial task change, qualitatively switches the learned mechanism from slow-point manifolds to limit cycles.
- Learned mechanisms can evolve during training, with a slow-point manifold later becoming a limit cycle.
Limitations
- Toy models are low-dimensional and may not capture higher-dimensional mechanisms; the paper cautions against over-interpreting them.
- Results are shown for simple delayed activation and cue-discrimination tasks rather than complex memory tasks.
- Compute is described only as several computers with RTX 3090 GPUs, without precise GPU counts.
Experimental setup
Datasets: synthetic delayed activation task synthetic delayed cue-discrimination task
Metrics: critical learning rate scaling exponent beta fraction of networks by mechanism (slow-point vs limit cycle loss
Baselines: slow-point manifold mechanism fixed-point attractor
Builds on: Ghost model saddle-node learning dynamics low-rank RNN latent dynamics
Closest papers by meaning
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.90)
- Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.89)
- Back to the Continuous Attractor (NeurIPS 2024 · similarity 0.89)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.89)
- Predictive Attractor Models (NeurIPS 2024 · similarity 0.89)
- RNNs perform task computations by dynamically warping neural representations (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.