Research map / Reinforcement learning / Visual and representation RL
Real-Time Recurrent Learning using Trace Units in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Recurrent Trace Units (RTUs), a real-valued, cosine-parameterized, diagonal-complex recurrent layer that is trained with RTRL in online RL and outperforms GRUs and LRUs at lower computation.
Problem
Training recurrent neural networks online in partially observable RL is expensive: RTRL is exact but quartic in hidden size for dense RNNs, while T-BPTT's cost depends on truncation length and has a bias-compute tradeoff.
Approach
Use a real-valued cosine-representation of complex diagonal recurrence (2x2 rotation blocks with per-unit decay) so RTRL is linear in hidden size, and add a nonlinearity either after (linear RTU) or inside (nonlinear RTU) the recurrence, avoiding complex-valued autodiff.
Key results
- On trace conditioning under a ~15000 FLOP budget, RTUs outperform GRU and LRU trained with T-BPTT across truncation lengths.
- On Mujoco POMDP tasks with ~24k recurrent parameters, RTU agents outperform GRU, LRU and online LRU in all environments.
- RTUs with RTRL scale better than GRU with increasing compute and parameters, with GRU improvement being inconsistent.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RTUs instead of GRUs | online prediction on Trace Conditioning · Trace Conditioning | MSRE | Helped |
| use RTUs instead of LRUs | online prediction on Trace Conditioning · Trace Conditioning | MSRE | Helped |
| add nonlinearity inside the recurrence rather than after it (ablation) | RTU · Reacher POMDP | performance | Helped |
| use nonlinear RTUs instead of linear RTUs (ablation) | memory-based control · POPGym tasks | performance | No clear effect |
| use RTUs instead of online LRUs | PPO recurrent controller · Mujoco POMDP benchmark | performance | Helped |
| use RTUs instead of GRUs | PPO recurrent controller · Mujoco POMDP benchmark | performance | Helped |
| use RTUs instead of GRUs | PPO recurrent controller · Reacher POMDP | performance | No clear effect |
| use stale RTRL gradients rather than recomputing traces (ablation) | RTUs with PPO · PPO policy-gradient experiments | performance | Helped |
| increase the number of RTU parameters as computation increases | RTU · Trace Conditioning | MSRE | Helped |
| increase GRU parameters as computation increases | GRU · Trace Conditioning | MSRE | Mixed |
| increase RTU parameters | RTU · Trace Conditioning | MSRE | Helped |
| increase GRU parameters | GRU · Trace Conditioning | MSRE | Mixed |
Surprising findings
- Complex eigenvalues naturally appear when training a dense vanilla RNN on a simple Three State POMDP, indicating real-valued diagonals are too restrictive.
- Using stale RTRL gradients with PPO yields better performance with RTUs, possibly helping PPO maintain the trust region.
Limitations
- Extension to multilayer recurrence is not principled; traces across layers are costly.
- Linear recurrence allows parallel scans for LRUs, but RTUs with nonlinear recurrence cannot directly use them.
Experimental setup
Datasets: Trace conditioning Mujoco POMDP (Brax) POPGym Reacher POMDP
Benchmarks: Trace conditioning Mujoco POMDP POPGym Reacher POMDP Ant-P
Metrics: mean squared return error episodic return FLOPs runtime per update
Baselines: GRU with T-BPTT LRU with T-BPTT Online LRU GPT2 transformer real-valued diagonal RNN
Builds on: Linear Recurrent Units (LRU) Real-Time Recurrent Learning Proximal Policy Optimization Truncated BPTT Online LRU
Closest papers by meaning
- Exploring the Promise and Limits of Real-Time Recurrent Learning (ICLR 2024 · similarity 0.90)
- Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate (NeurIPS 2024 · similarity 0.87)
- Real-Time Recurrent Reinforcement Learning (AAAI 2025 · similarity 0.86)
- Recurrent Reinforcement Learning with Memoroids (NeurIPS 2024 · similarity 0.85)
- Handling Delay in Real-Time Reinforcement Learning (ICLR 2025 · similarity 0.85)
- Rethinking Transformers in Solving POMDPs (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.