Research map / Reinforcement learning / Visual and representation RL
Learning Dynamics of RNNs in Closed-Loop Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops a theory of learning dynamics for linear RNNs trained with policy gradients in closed-loop control of a double integrator, showing three learning stages driven by a trade-off between short-term control and long-term stability.
Problem
Learning dynamics of RNNs trained in closed-loop agent-environment interaction remain poorly understood, unlike open-loop supervised teacher-student settings with i.i.d. inputs.
Approach
Stack environment and RNN states into a coupled linear system whose eigenvalues track learning; under rank-1 connectivity this reduces to a 4D effective system with four order parameters, and a surrogate loss interpolating short-horizon and asymptotic terms explains the stage-wise dynamics and zig-zag trajectories.
Key results
- Closed-loop and open-loop RNNs with identical architecture and initialization follow markedly different trajectories; open-loop shows a sharp closed-loop test loss peak while closed-loop shows a plateau then convergence in three stages.
- The reduced 4D effective system reproduces the spectrum of the full closed-loop matrix P exactly within an episode and tracks full RNN training loss in Stage 3.
- In the 2D multi-frequency tracking task, closed-loop RNN acquisition times of frequencies qualitatively match human participants' trends.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train the RNN in a closed-loop environment instead of open-loop teacher–student imitation | RNN · Double integrator | closed-loop test loss | Mixed |
| train with closed-loop interaction rather than open-loop imitation | RNN · Double integrator | learning dynamics | Mixed |
| train with closed-loop interaction rather than open-loop imitation | RNN · 2D multi-frequency tracking | test loss | Mixed |
| use a continuous-time RNN instead of a discrete-time RNN (ablation) | Closed-loop RNN · Double integrator | learning dynamics | No clear effect |
| add control regularization with β = 0.1 or 1 (ablation) | Closed-loop RNN · Double integrator | final train loss | Hurt |
| add control regularization with β = 0.1 or 1 (ablation) | Closed-loop RNN · Double integrator | learning dynamics | No clear effect |
| increase initial connectivity strength g (ablation) | Open-loop RNN student · Linear-filter imitation | convergence speed | Helped |
| replace the RNN teacher with an optimal LQR teacher (ablation) | Open-loop RNN student · Double integrator | loss trajectory | No clear effect |
| use partial observability rather than full-state feedback (ablation) | Open-loop RNN · k-integrator family | training loss | Hurt |
| use partial observability rather than full-state feedback (ablation) | Closed-loop RNN · k-integrator family | training dynamics | Mixed |
| use full-state feedback instead of partial observability (ablation) | Open-loop RNN · k-integrator family | training dynamics | Helped |
| optimize only the short-term surrogate loss (α = 0) instead of blending short- and long-horizon losses (ablation) | Closed-loop effective model · Double integrator, Stage 2 | dominant eigenvalue trajectory | Hurt |
Surprising findings
- Open-loop RNNs show a transient closed-loop test loss peak even though their open-loop training loss decreases only modestly.
- Optimizing only the short-horizon loss drives the dominant eigenvalue away from stability, so the stage-2 plateau requires a blend of short- and long-term objectives.
Limitations
- Analysis relies on linear activations, rank-1 connectivity, and fixed input weights.
- Reduced effective system is accurate only within an episode; order-parameter learning dynamics are not derived.
- Uses unregularized (beta=0) policy gradient with direct gradients rather than sparse rewards or approximate RL gradients.
Experimental setup
Datasets: simulated double integrator trajectories simulated 2D multi-frequency tracking trajectories
Benchmarks: double integrator control 2D multi-frequency tracking task
Metrics: training loss test loss eigenvalue spectrum of closed-loop matrix effective feedback gain
Baselines: open-loop supervised RNN training
Builds on: policy gradient teacher-student framework linear-quadratic regulator Kalman filter
Closest papers by meaning
- Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.91)
- Inference of Neural Dynamics Using Switching Recurrent Neural Networks (NeurIPS 2024 · similarity 0.90)
- Mechanistic Interpretability of RNNs emulating Hidden Markov Models (NeurIPS 2025 · similarity 0.90)
- Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks (NeurIPS 2025 · similarity 0.89)
- Recurrent neural network dynamical systems for biological vision (NeurIPS 2024 · similarity 0.89)
- Dynamical phases of short-term memory mechanisms in RNNs (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.