Research map / Architectures and efficiency / RNNs and normalization
Traveling Waves Encode The Recent Past and Enhance Sequence Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Wave-RNN, a minimal recurrent network whose hidden state is a circular convolutional wave field, and shows it learns faster and reaches lower error than identity-initialized RNNs on synthetic memory and sequential image tasks.
Problem
Whether traveling waves in neural activity serve a computational role for short-term memory, given the lack of simple trainable RNN architectures that exhibit such waves.
Approach
A simple RNN whose recurrent connections are a shift-initialized circular convolution over multi-channel ring-structured hidden state, so that inputs are written into and propagate as traveling waves, compared directly against an identity-initialized wave-free RNN (iRNN) with matched parameters.
Key results
- On the copy task, wRNN reaches more than 5 orders of magnitude lower loss than iRNNs with matched parameters and activations, and stays accurate up to T=480.
- On the adding task, wRNN solves T=1000 sequences (about 2k iterations) where the iRNN fails beyond T=400.
- On psMNIST the wRNN (16 channels) reaches 96.7% versus 92.5% for the iRNN baseline in the paper's table, with far fewer parameters (47k vs 68k) and better parameter efficiency; on nsCIFAR10 wRNN reaches 55.0% vs 51.3% for iRNN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace wave-free iRNN recurrence with traveling-wave wRNN recurrence | iRNN · Adding task T=1000 | Test MSE: 0.16 -> 6×10^{-5} | Helped |
| use wRNN instead of iRNN | sequence model · psMNIST | test accuracy: 92.5 -> 96.7 | Helped |
| use wRNN instead of iRNN | sequence model · sMNIST | test accuracy: 98.5 -> 97.6 | Hurt |
| replace the linear output map with a 2-layer MLP (ablation) | wRNN · psMNIST | test accuracy: 96.7 -> 97.6 | Helped |
| remove V sparse-identity initialization (ablation) | wRNN · Copy task T=80 | MSE: 1× 8^{-11} -> 4× 10^{-11} | No clear effect |
| add u-shift initialization (ablation) | wRNN without u-shift initialization · Copy task T=80 | MSE: 6× 10^{-4} -> 1× 8^{-11} | Helped |
| add V sparse-identity and u-shift initialization (ablation) | wRNN without V and u-shift initialization · Copy task T=80 | MSE: 7× 10^{-4} -> 1× 8^{-11} | Helped |
| add Σ recurrent initialization (ablation) | iRNN · Copy task T=80 | MSE: 1× 10^{-3} -> 2× 10^{-5} | Helped |
| add Σ recurrent and sparse-identity V initialization (ablation) | iRNN · Copy task T=80 | MSE: 1× 10^{-3} -> 8× 10^{-6} | Helped |
| replace convolution with an untied locally connected layer (ablation) | wRNN · sMNIST | accuracy | No clear effect |
Surprising findings
- The iRNN baseline, re-tuned by the authors, outperforms originally reported numbers (98.5% vs 97% on sMNIST, 91% vs 81% on psMNIST), and sometimes beats later state-of-the-art methods.
- A simple linear readout on flattened MNIST reaches 92.4%, suggesting the wRNN's rapid learning may partly reflect a memory-tape-like encoding rather than deep processing.
Limitations
- Experiments use small models with linear encoders and decoders and a single RNN layer, far from state-of-the-art scale.
- On sMNIST the wRNN is slightly below the iRNN (97.6% vs 98.5% baseline), and the CIFAR10 wRNN parameter count is dominated by the input encoder.
Experimental setup
Datasets: MNIST CIFAR10 copy task (synthetic) adding task (synthetic)
Benchmarks: copy task adding task sMNIST psMNIST noisy sequential CIFAR10
Metrics: test MSE test accuracy solved iterations parameter count
Baselines: iRNN LSTM GRU uRNN IndRNN LEM coRNN Lipschitz RNN Neural Wave Machine anti-symmetric RNN
Builds on: iRNN coRNN Neural Wave Machine S4
Closest papers by meaning
- Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks (ICML 2024 · similarity 0.93)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.87)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.86)
- Recurrent neural network dynamical systems for biological vision (NeurIPS 2024 · similarity 0.85)
- Predictive Attractor Models (NeurIPS 2024 · similarity 0.85)
- Complex priors and flexible inference in recurrent circuits with dendritic nonlinearities (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.