Research map / Probabilistic and scientific modeling / Dynamical systems
Learning long range dependencies through time reversal symmetry breaking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Recurrent Hamiltonian Echo Learning (RHEL), a forward-only gradient estimator for Hamiltonian/symplectic recurrent models that matches adjoint and BPTT gradients via finite differences of echo trajectories.
Problem
Training recurrent state space models with physically realizable, forward-only, Jacobian-free learning rules that scale to long sequences, as an alternative to BPTT and RTRL.
Approach
Computes loss gradients as finite differences between a free Hamiltonian trajectory and a nudged 'echo' trajectory obtained by flipping momenta and replaying inputs backward; shown equivalent to the continuous adjoint method and, for leapfrog-based Hamiltonian Recurrent Units, to BPTT, and extended to stacks (HSSMs) via RHL chaining.
Key results
- On six UEA classification datasets, RHEL averages 62.9% vs 63.4% for BPTT with linear HSSMs, and 57.8% vs 58.8% with nonlinear HSSMs (five seeds).
- On PPG-DaLiA (sequence length ~50k), RHEL reaches MSE 9.5e-2 vs 9.1e-2 for BPTT (linear) and 8.4e-2 vs 7.8e-2 (nonlinear).
- Layer-wise RHEL and BPTT parameter gradients are near-perfectly aligned in cosine similarity and norm ratio on six-block linear and nonlinear HSSMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RHEL instead of BPTT to train linear HSSMs | Linear HSSM · UEA time-series classification test average | accuracy (%): 63.4^{\pm 3.9} -> 62.9^{\pm 4.9} | No clear effect |
| Use RHEL instead of BPTT to train nonlinear HSSMs | Nonlinear HSSM · UEA time-series classification test average | accuracy (%): 58.8^{\pm 4.3} -> 57.8^{\pm 4.0} | No clear effect |
| Use RHEL instead of BPTT to train linear HSSMs | Linear HSSM · PPG-DaLiA test | MSE (×10^-2): 9.1^{\pm 1.1} -> 9.5^{\pm 1.0} | Hurt |
| Use RHEL instead of BPTT to train nonlinear HSSMs | Nonlinear HSSM · PPG-DaLiA test | MSE (×10^-2): 7.8^{\pm 0.5} -> 8.4^{\pm 0.5} | Hurt |
Limitations
- Restricted to non-dissipative Hamiltonian systems, limiting expressivity.
- Finite nudging introduces approximation bias, and RHEL slightly underperforms BPTT, significantly so only for nonlinear HSSMs on regression.
- Still requires explicit Hamiltonian parameter gradients and storing input sequences, so memory gains are not evident within a stack of HRUs; not online.
Experimental setup
Datasets: UEA Worms SCP1 SCP2 Ethanol Heartbeat MotorImagery PPG-DaLiA
Benchmarks: UEA time series classification (6 datasets) PPG-DaLiA
Metrics: accuracy mean-square error cosine similarity of gradients gradient norm ratio
Baselines: BPTT RTRL continuous adjoint state method
Builds on: Hamiltonian Echo Backprop continuous adjoint state method Leapfrog integrator backpropagation through time
Closest papers by meaning
- Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction (NeurIPS 2026 · similarity 0.87)
- Physics-Informed Regularization for Domain-Agnostic Dynamical System Modeling (NeurIPS 2024 · similarity 0.86)
- Stabilizing Backpropagation Through Time to Learn Complex Physics (ICLR 2024 · similarity 0.86)
- Towards Cross Domain Generalization of Hamiltonian Representation via Meta Learning (ICLR 2024 · similarity 0.85)
- Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction (NeurIPS 2024 · similarity 0.85)
- End-to-End Identifiable and Consistent Recurrent Switching Dynamical Systems (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.