Research map / Architectures and efficiency / RNNs and normalization
Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GTF-DEER, combining DEER parallel-in-time Newton solving with generalized teacher forcing to train RNNs for chaotic dynamical systems reconstruction, and uses it to show long training sequences (T>10^4) help capture slow timescales.
Problem
Training sequential models for dynamical systems reconstruction (DSR) is O(T) with BPTT, and DEER-style parallel training diverges on chaotic data; linear SSMs with teacher forcing suffer from exposure bias and low-rank limits for DSR.
Approach
Augment DEER with generalized teacher forcing (with a pseudo-inverse row-space projector of the observation matrix) so the forced RNN is contracting during training, enabling stable O(log T)-style parallel training on chaotic data and on very long sequences.
Key results
- GTF-DEER gives speedups of up to 870x over sequential training on Lorenz-63 at T=32,768 with M=4.
- Training on longer sequences (fixed B*T=32,768) improves long-term D_stsp on forced Lorenz-96 and bursting neuron systems while RMSE(128) saturates early.
- On forced Lorenz-96, LSSMs and rank-6 shPLRNN fail to capture the sinusoidal forcing, while rank-7 shPLRNN trained with GTF-DEER reconstructs it; Mamba-2 performs much worse.
Surprising findings
- Diagonal (quasi-DEER) Jacobian approximations needed about 100x more Newton iterations and diverged in 14/20 partially observed runs.
- Adding a single rank (r=7 vs r=6) to the shPLRNN markedly closes the gap in reconstructing latent forcing dynamics.
Limitations
- Cubic O(M^3 T) work per Newton iteration with full Jacobians limits latent dimension M.
- Convergence guarantee (Proposition) holds strictly only for M <= N; M > N supported empirically.
- GPU memory bandwidth saturates for large B x T x M, reducing scaling gains.
Experimental setup
Datasets: Lorenz-63 Forced Lorenz-96 (simulated) Bursting neuron model (simulated)
Benchmarks: Lorenz-63 forced Lorenz-96 bursting neuron
Metrics: D_stsp (KL divergence via GMM) RMSE(128) wall-clock runtime Newton iterations
Baselines: sequential GTF linear SSM (LSSM) Mamba-2 quasi-DEER low-rank shPLRNN
Builds on: DEER generalized teacher forcing shPLRNN manifold attractor regularization sparse teacher forcing Mamba-2
Closest papers by meaning
- Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction (NeurIPS 2024 · similarity 0.89)
- Parallelizing non-linear sequential models over the sequence length (ICLR 2024 · similarity 0.88)
- Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data (ICLR 2025 · similarity 0.88)
- Out-of-Domain Generalization in Dynamical Systems Reconstruction (ICML 2024 · similarity 0.87)
- Learning long range dependencies through time reversal symmetry breaking (NeurIPS 2025 · similarity 0.87)
- A scalable generative model for dynamical system reconstruction from neuroimaging data (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.