Research map / Architectures and efficiency / RNNs and normalization
Exploring the Promise and Limits of Real-Time Recurrent Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that exact RTRL is tractable for element-wise recurrent architectures (eLSTM, QRNN/SRU-like, linear transformers) and uses it in an actor-critic (R2AC) that beats IMPALA/R2D2 on DMLab memory tasks with fewer frames.
Problem
Real-time recurrent learning (RTRL) is considered impractical due to its quartic/cubic complexity, and prior work evaluates approximations only in diagnostic settings, so its practical promise in realistic tasks is unknown.
Approach
Restrict RNNs to element-wise recurrence (an LSTM variant, eLSTM), which makes exact RTRL tractable (O(N^2) space and per-step time), and combine it with IMPALA-style recurrent actor-critic (R2AC) to evaluate untruncated gradients at scale in RL.
Key results
- On rooms_watermaze with M=100, RTRL reaches 54.8 ± 4.3 vs 45.6 ± 4.7 for TBPTT and 45.9 for R2D2 (10B frames), using under 1.2B frames.
- On rooms_select_nonmatching_object with M=10, RTRL scores 61.8 ± 0.5 vs 54.5 ± 1.1 for TBPTT; at M=50/100 the two are similar.
- RTRL with eLSTM outperforms SnAp-1 applied to feLSTM (52.3 vs 27.7 on watermaze at M=100 comparison table: 54.8 vs 27.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RTRL instead of TBPTT (M=10) | eLSTM actor-critic · DMLab-30 rooms_select_nonmatching_object | score: 54.5 $\pm$ 1.1 -> 61.8 $\pm$ 0.5 | Helped |
| Use RTRL instead of TBPTT (M=50) | eLSTM actor-critic · DMLab-30 rooms_select_nonmatching_object | score: 61.4 $\pm$ 0.5 -> 62.0 $\pm$ 0.4 | No clear effect |
| Use RTRL instead of TBPTT (M=100) | eLSTM actor-critic · DMLab-30 rooms_select_nonmatching_object | score: 61.7 $\pm$ 0.1 -> 62.2 $\pm$ 0.3 | No clear effect |
| Use RTRL instead of TBPTT (M=10) | eLSTM actor-critic · DMLab-30 rooms_watermaze | score: 15.8 $\pm$ 0.9 -> 40.2 $\pm$ 5.6 | Helped |
| Use RTRL instead of TBPTT (M=50) | eLSTM actor-critic · DMLab-30 rooms_watermaze | score: 44.5 $\pm$ 1.5 -> 52.3 $\pm$ 1.9 | Helped |
| Use RTRL instead of TBPTT (M=100) | eLSTM actor-critic · DMLab-30 rooms_watermaze | score: 45.6 $\pm$ 4.7 -> 54.8 $\pm$ 4.3 | Helped |
| Replace LSTM recurrence with element-wise eLSTM recurrence (ablation) | TBPTT actor-critic · DMLab-30 rooms_watermaze | score: 37.6 $\pm$ 5.3 -> 45.6 $\pm$ 4.7 | Helped |
| Replace eLSTM recurrence with full-recurrence feLSTM (ablation) | TBPTT actor-critic · DMLab-30 rooms_select_nonmatching_object | score: 61.7 $\pm$ 0.1 -> 61.5 $\pm$ 0.4 | No clear effect |
| Replace eLSTM recurrence with full-recurrence feLSTM (ablation) | TBPTT actor-critic · DMLab-30 rooms_watermaze | score: 45.6 $\pm$ 4.7 -> 43.2 $\pm$ 4.7 | No clear effect |
| Use exact RTRL with eLSTM instead of SnAp-1 with feLSTM | DMLab-30 actor-critic · DMLab-30 rooms_select_nonmatching_object | score: 53.6 $\pm$ 0.8 -> 62.2 $\pm$ 0.3 | Helped |
| Use exact RTRL with eLSTM instead of SnAp-1 with feLSTM | DMLab-30 actor-critic · DMLab-30 rooms_watermaze | score: 27.7 $\pm$ 3.2 -> 54.8 $\pm$ 4.3 | Helped |
| Use RTRL instead of TBPTT (M=50) | R2AC · ProcGen Chaser | score | No clear effect |
Surprising findings
- With sufficiently large M (50 or 100), RTRL and TBPTT perform essentially the same on several tasks, so the benefit of untruncated gradients is limited to long-credit-assignment tasks.
- Stacking two element-wise recurrent layers reintroduces full recurrence across layers, bringing back the quartic/cubic RTRL complexity.
Limitations
- Exact RTRL is restricted to one-layer element-wise recurrence; multi-layer stacks become intractable.
- Vision stem cannot be trained with RTRL, so it is pretrained with TBPTT or frozen, and most tasks do not truly require RTRL over TBPTT with a large span.
- Element-wise recurrent architectures have limited computational capacity on some synthetic tasks.
Experimental setup
Datasets: DMLab-30 ProcGen Atari 2600
Benchmarks: DMLab-30 rooms_select_nonmatching_object DMLab-30 rooms_watermaze ProcGen Chaser Atari Breakout, Gravitar, MsPacman, Q*bert, Seaquest
Metrics: episode score mean and std over 3 seeds
Baselines: IMPALA R2D2 R2D2+ TBPTT SnAp-1 LSTM
Builds on: IMPALA LSTM Quasi-RNN Simple Recurrent Units SnAp
Closest papers by meaning
- Approximating Real-Time Recurrent Learning with Random Kronecker Factors (NeurIPS 2018 · similarity 0.91)
- Practical Real Time Recurrent Learning with a Sparse Approximation (ICLR 2021 · similarity 0.91)
- Real-Time Recurrent Learning using Trace Units in Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
- Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate (NeurIPS 2024 · similarity 0.87)
- Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.