Research map / Architectures and efficiency / RNNs and normalization
RNNs perform task computations by dynamically warping neural representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a Riemannian (pullback-metric) framework for dynamical systems driven by inputs on a manifold, showing their state lies on an (m+1)-dimensional manifold, and uses it to show RNNs solve contextual decision-making and working-memory tasks by dynamically warping neural representations.
Problem
Understanding how the geometry of neural representations in dynamical systems (RNNs) reflects task computations over time, beyond fixed-point and static analyses.
Approach
Shows that for time-varying inputs on an m-dimensional manifold the system state lies on an (m+1)-dimensional manifold, and derives the pullback Riemannian metric on this state manifold via an adjoint (sensitivity) differential equation, enabling geometric analysis of computation through dynamics.
Key results
- Theorem: states of a dynamical system driven by inputs on an m-dimensional manifold lie on an (m+1)-dimensional manifold.
- In contextual evidence integration RNN, the metric's time and irrelevant-input eigenvalues decay to zero by decision time, while the relevant-input direction stays large, showing the manifold compresses irrelevant inputs.
- In sequential working memory RNN, the activity lies on a torus with non-zero positive and negative Gaussian curvature that is non-uniform and changes over time, with dynamic warping during retrieval.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the RNN to compress the irrelevant-input direction over time | Contextual decision-making RNN · Contextual evidence integration | irrelevant-input influence on readout | Helped |
| Dynamically warp the manifold to compress irrelevant inputs while stretching near the decision boundary along the relevant input | Contextual decision-making RNN · Contextual evidence integration | representation of relevant and irrelevant inputs | Mixed |
| Warp the torus during retrieval to compress stimuli not immediately retrieved | Sequential working-memory RNN · Sequential working memory | representation of time-specific relevant information | Helped |
| Train the feedforward network on the circular binary-classification task | Three-neuron feedforward network · Circular binary classification | representation near class boundaries | Helped |
Surprising findings
- Trained network's topology matches initialisation for full-rank weights, so topology does not reflect learned computation; geometry does.
Limitations
- Geometry-dynamics-connectivity relationship is degenerate; different architectures can give the same metric.
- Noise treated pathwise under a fixed noise realisation; distribution over noise realisations not characterised.
- Asymptotic and chaotic regimes not fully covered by the topological result.
Experimental setup
Datasets: synthetic circular input manifold synthetic contextual evidence integration inputs synthetic sequential working memory inputs
Metrics: pullback metric entries eigenvalues of metric Gaussian curvature geodesic distance readout sensitivity
Baselines: untrained network linearised fixed-point analysis
Builds on: pullback metric (Riemannian geometry of neural networks) low-rank RNNs line attractor RNN
Closest papers by meaning
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.89)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.88)
- Task structure and nonlinearity jointly determine learned representational geometry (ICLR 2024 · similarity 0.88)
- Geometry-Aware Similarity Metrics for Neural Representations on Riemannian and Statistical Manifolds (NeurIPS 2026 · similarity 0.88)
- Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction (NeurIPS 2024 · similarity 0.87)
- Geometry of naturalistic object representations in recurrent neural network models of working memory (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.