Research map / Architectures and efficiency / RNNs and normalization
Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a traveling-wave working memory model (TWM) for history-dependent dynamical systems, shows RNNs trained on such tasks converge to it, and argues traveling waves ease gradient propagation.
Problem
How working memory variables are stored in RNNs and transformers: prevailing views assume fixed register-like storage updated by interference or decay, and RNNs suffer vanishing/exploding gradients over long histories.
Approach
Model working memory as traveling waves on a neural lattice: each state dimension is a 1-D wave shifted one column per step, with the start boundary computing a function f of the whole substrate. Linear boundary conditions map to an RNN-like matrix, and self-attention boundary conditions map to an autoregressive attention head.
Key results
- RNNs trained on four linear HDS tasks reach test accuracy above 0.94 for most configurations, with low eigenspectrum MAE (about 0.0005 to 0.01) against the theoretical traveling-wave Phi across 20 seeds.
- Basis transformation of trained repeat-copy RNN hidden states reveals input variables stored as waves traveling from index 8 down to index 1.
- The paper argues theoretically that with traveling waves the gradient upper bound no longer has the power-s decay term, though the gradient improvement is shown empirically only qualitatively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive training-horizon scheduling | Elman RNN training on repeat copy and compose copy · Repeat copy and compose copy | training efficiency | Helped |
| Use adaptive training-horizon scheduling | Elman RNN training on repeat copy and compose copy · Repeat copy and compose copy | reported results | No clear effect |
Surprising findings
- Some configurations (e.g., repeat copy with hidden size 64 and no L2) encode variables in bottleneck superposition, giving low-rank nonlinear solutions rather than the linear traveling-wave circuit.
- Gaussian initialization yields learned eigenvalues outside the unit circle, and networks with larger eigenvalues tended to generalize longer in time.
Limitations
- Analysis of learned bases is restricted to linear and binary history-dependent tasks, and the basis-approximation algorithm fails for some tasks due to interference from the operator during input phase.
- The self-attention boundary condition case is a theoretical connection only and is not empirically validated in the paper.
- Experiments use small synthetic tasks (d=s=8) and single-layer Elman RNNs, limiting generality.
Experimental setup
Datasets: synthetic repeat copy task synthetic compose copy task synthetic binary linear HDS tasks
Benchmarks: repeat copy (T1) compose copy (T2) tasks T3 and T4 (linear variable composition)
Metrics: mean absolute error of complex eigenspectrum between predict test accuracy gradient norm
Baselines: Elman RNN with bias Elman RNN without bias
Builds on: WaveRNN Elman RNN H3 state space model backpropagation through time
Closest papers by meaning
- Traveling Waves Encode The Recent Past and Enhance Sequence Learning (ICLR 2024 · similarity 0.93)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.87)
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.86)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
- Inference of Neural Dynamics Using Switching Recurrent Neural Networks (NeurIPS 2024 · similarity 0.85)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.