Research map / Reinforcement learning / MDPs and planning
Real-Time Recurrent Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RTRRL, a biologically plausible online actor-critic RL framework for POMDPs that combines a basal-ganglia-inspired Meta-RL recurrent network, TD(λ) eligibility traces, and RFLO or diagonal (LRU) RTRL for online gradient computation.
Problem
Solving partially observable RL tasks with online, biologically plausible learning that avoids BPTT, weight transport, and batched trajectory-based updates.
Approach
Combines a Meta-RL recurrent backbone (CT-RNN or LRU) with linear actor and critic heads trained by TD(λ) eligibility traces, and computes recurrent gradients online with RFLO (random feedback, local) or diagonal RTRL, using feedback alignment instead of weight transport.
Key results
- RTRRL with LRUs achieves the best median reward in almost all tested environments, outperforming PPO with the same number of neurons (32).
- Fully biologically plausible RTRRL-RFLO often achieves comparable performance to PPO and sometimes outperforms it, e.g. on NoisyStatelessCartPole where it performs best.
- On MemoryChain, RTRRL outperformed BPTT-trained networks of the same size; RTRRL-RFLO time per step was 7.49 ms vs 7.58 ms for PPO-LSTM on a single RTX 2070.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Feedback Alignment instead of regular backpropagation for recurrent gradient propagation (ablation) | RTRRL with CT-RNN · many environments | reward | Mixed |
| Use the Meta-RL architecture (ablation) | RTRRL with CT-RNN · RepeatPrevious and other environments | reward | Mixed |
| Vary the entropy-loss weight (ablation) | RTRRL with CT-RNN · MetaMaze | reward | Mixed |
| Train the recurrent network instead of keeping it frozen (ablation) | RTRRL · MemoryChain | reward | Helped |
| Use approximate gradients with RFLO rather than BPTT (ablation) | RTRRL · MemoryChain | reward | Hurt |
| Use RTRRL rather than BPTT-trained PPO | 32-neuron recurrent networks · MemoryChain | reward | Helped |
| Use LRU-based RTRRL rather than PPO | 32-neuron recurrent networks · benchmarks overall | reward | Helped |
Surprising findings
- Feedback alignment often performs as well as regular backpropagation, despite the mismatch in gradient pathways.
- Quite surprisingly, RTRRL outperformed networks of the same size trained with BPTT on the MemoryChain memory task.
Limitations
- High variance in learning due to batch size of 1.
- Requires careful hyperparameter tuning, especially for continuous actions.
- Catastrophic forgetting occurs in some cases without learning rate decay.
Experimental setup
Datasets: gymnax environments popgym environments brax environments (masked to POMDP) bsuite DeepSea Acrobot
Benchmarks: gymnax popgym brax MemoryChain bsuite DeepSea Acrobot
Metrics: best validation reward median reward standard deviation over runs wall-clock time per step
Baselines: PPO with LSTM/CT-RNN (BPTT) Linear TD(λ) PPO CTRNN
Builds on: RFLO real-time recurrent learning (RTRL) TD(λ) Linear Recurrent Units PPO
Closest papers by meaning
- Rethinking Transformers in Solving POMDPs (ICML 2024 · similarity 0.87)
- Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate (NeurIPS 2024 · similarity 0.87)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.86)
- Recurrent Reinforcement Learning with Memoroids (NeurIPS 2024 · similarity 0.86)
- Real-Time Recurrent Learning using Trace Units in Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- DynaMITE-RL: A Dynamic Model for Improved Temporal Meta-Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.