Research map / Optimization / Optimizers
Stabilizing Backpropagation Through Time to Learn Complex Physics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a modified backpropagation-through-time vector field for simulator-network training that stops gradients through the network's state input and keeps only components whose sign agrees with the regular gradient, evaluated on three control tasks.
Problem
Backpropagation through time over many unrolled steps of simulator-network systems produces exploding and vanishing gradients, creating an unbalanced optimization landscape that hampers learning of long-horizon control in physics simulations.
Approach
Stop gradient feedback through the neural network's input state (keeping the physics simulator path over the full trajectory) to balance gradient magnitudes, then counteract the resulting rotational vector field by zeroing components whose sign disagrees with the regular gradient (the "combined" update).
Key results
- On the cart pole swing-up with 3 or more poles, the combined update (C) consistently outperforms regular gradient (R), modified (M), and stopped (S) updates, and is the only method solving the task below loss 0.01.
- On quantum control with the 4th excited state target, R and S make no progress while M reaches a decent result and C achieves a loss of about 20.
- In a hyperparameter study on cart pole, 24 M-runs and 37 C-runs reached loss 0.01 versus 7 R-runs; no R-run solved quantum control while M and C had 14 runs below loss 30.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace regular BPTT with the modified update (stop feedback through the network from its output to its input) (ablation) | Recurrent simulator-network training · Guidance-by-Repulsion hyperparameter study | best-run loss: 0.590 -> 0.465 | Helped |
| Use the combined update instead of regular BPTT | Recurrent simulator-network training · Guidance-by-Repulsion hyperparameter study | runs below loss 0.8: 11 -> 38 | Helped |
| Replace regular BPTT with the modified update (stop feedback through the network from its output to its input) (ablation) | Recurrent simulator-network training · Cart Pole Swing-Up hyperparameter study | best-run loss: 0.00194 -> 0.00165 | Helped |
| Use the combined update instead of regular BPTT | Recurrent simulator-network training · Cart Pole Swing-Up hyperparameter study | runs below loss 0.01: 7 -> 37 | Helped |
| Add component-wise sign comparison to the modified update to form the combined update (ablation) | Recurrent simulator-network training · Cart Pole Swing-Up hyperparameter study | best-run loss: 0.00165 -> 0.00145 | Helped |
| Replace regular BPTT with the modified update (stop feedback through the network from its output to its input) (ablation) | Recurrent simulator-network training · Quantum Control hyperparameter study | best-run loss: 85.6 -> 19.0 | Helped |
| Use the combined update instead of regular BPTT | Recurrent simulator-network training · Quantum Control hyperparameter study | runs below loss 30: 0 -> 14 | Helped |
| Add component-wise sign comparison to the modified update to form the combined update (ablation) | Recurrent simulator-network training · Quantum Control hyperparameter study | best-run loss: 19.0 -> 19.0 | No clear effect |
| Use a stopped update that backpropagates only through the last unrolled step instead of regular BPTT | Recurrent simulator-network training · Cart Pole Swing-Up hyperparameter study | runs below loss 0.01: 7 -> 0 | Hurt |
| Use a stopped update that backpropagates only through the last unrolled step instead of regular BPTT | Recurrent simulator-network training · Quantum Control hyperparameter study | runs below loss 30: 0 -> 0 | No clear effect |
Surprising findings
- Stopping gradient feedback through the network input (modified update) can outperform the full regular gradient, and the stopped last-step-only gradient (S) fails on most tasks.
- No regular-gradient run solved the quantum control task in the hyperparameter study, despite the regular gradient norm varying by more than 35 orders of magnitude.
Limitations
- Not evaluated on contact-rich scenarios with discontinuous dynamics, which may violate the well-behaved gradient flow assumption (only a simple wall contact task in appendix).
- Theory is restricted to toy examples and the combined update requires two backpropagation passes, roughly doubling compute in some settings.
- Experiments limited to three control tasks with small fully connected or convolutional controllers.
Experimental setup
Datasets: synthetic initial states for guidance-by-repulsion synthetic initial configurations for cart pole superpositions of ground and first excited state for quantum control
Benchmarks: guidance-by-repulsion cart pole swing-up quantum dipole control cart pole with walls
Metrics: end-of-epoch test loss best-run loss number of runs below loss threshold epoch duration gradient L2 norm
Baselines: regular gradient (BPTT) stopped gradient (last-step only) modified update (ablation)
Builds on: Backpropagation Through Time Adam Crank-Nicolson scheme
Closest papers by meaning
- Learning long range dependencies through time reversal symmetry breaking (NeurIPS 2025 · similarity 0.86)
- Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization (ICLR 2024 · similarity 0.85)
- PRDP: Progressively Refined Differentiable Physics (ICLR 2025 · similarity 0.85)
- How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective (ICML 2024 · similarity 0.85)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.84)
- Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.