Research map / Architectures and efficiency / RNNs and normalization
Learning Reveals Invisible Structure in Low-Rank RNNs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives closed-form low-dimensional ODEs for gradient-descent learning dynamics in rank-1 low-rank RNNs (exact for linear, Gaussian/DMFT-asymptotic for erf nonlinear), separating loss-visible from loss-invisible overlaps and showing invisible overlaps can encode training history in nonlinear nets.
Problem
Learning in RNNs reshapes connectivity, but the mapping from high-dimensional parameters to function is degenerate, and no analytical low-dimensional description of task-trained nonlinear RNN learning exists within the low-rank overlap framework.
Approach
Express gradient descent directly in scalar overlap space via a Gram (preconditioner) matrix G(theta) computed in closed form; split overlaps into loss-visible (determine function/loss) and loss-invisible (affect only learning metric), showing invariants in linear networks suppress memory while nonlinearity breaks the separation and lets invisible overlaps store training history.
Key results
- For the linear rank-1 RNN, the 10D overlap ODE matches full parameter-space gradient descent loss curves, whereas direct optimization in overlap space gives qualitatively different dynamics.
- In the linear A-B-A protocol with vanilla gradient descent, loss-invisible overlaps fully recover their original values, reflecting an exact conserved matrix K; label noise breaks this invariant and induces drift.
- In the nonlinear twin-network protocol with added overlap noise N(0,0.1), loss-invisible overlaps decode training history while loss-visible overlaps drop to chance level.
Surprising findings
- Under vanilla gradient descent in linear RNNs, loss-invisible overlaps return exactly to original values after A-B-A retraining, so linear networks show no memory.
- In nonlinear networks, sigma_mu and the norms of m and u become loss-visible through the variance Delta(t), breaking the visible-invisible separation.
Limitations
- Analysis restricted to rank-1 connectivity; overlap count grows quadratically with rank.
- Nonlinear analysis assumes parameter components remain approximately Gaussian during training, which is not strictly preserved for larger learning rates or Adam.
- Gradients with respect to overlaps are closed-form only for some tasks (e.g., filter task).
Experimental setup
Datasets: filter task (synthetic white-noise input) flip-flop task stimulus integration task
Benchmarks: filter task flip-flop task
Metrics: training loss overlap dynamics agreement cross-validation decoding accuracy
Baselines: direct gradient descent in overlap space
Builds on: dynamical mean-field theory low-rank RNN framework natural gradient
Closest papers by meaning
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.89)
- Inferring stochastic low-rank recurrent neural networks from neural data (NeurIPS 2024 · similarity 0.89)
- Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.87)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.87)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.87)
- Partial observation can induce mechanistic mismatches in data-constrained models of neural dynamics (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.