Research map / Reinforcement learning / Visual and representation RL
When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
Tianwei Ni, Michel Ma, Benjamin Eysenbach, Pierre-Luc Bacon
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Transformer instead of LSTM | memory-based RL agent · Passive T-Maze | return: solves 50 to 1500; LSTM falters at 250 | Improved |
| use Transformer instead of LSTM | memory-based RL agent · Passive Visual Match | success rate: close to Markovian 1/3 | Mixed |
| use Transformer instead of LSTM | memory-based RL agent · Passive Visual Match | return | Improved |
| shift Oracle one step to make task Active T-Maze | Transformer and LSTM · Active T-Maze | success: fail at length 250 | Hurt |
| use Transformer instead of LSTM | memory-based RL agent · Key-to-Door | success rate: medium length around 100 improves; 250 fails | Mixed |
| increase Transformer layers from 1 to 2 or 4 (ablation) | Transformer · Active T-Maze and Key-to-Door | performance: helps up to 100 and 120 | Mixed |
| increase Transformer layers from 2 to 4 (ablation) | Transformer · credit assignment tasks | performance | No clear effect |
| use Transformer instead of LSTM in recurrent TD3 | memory-based RL agent · PyBullet occlusion tasks | sample efficiency over 1.5M steps | Hurt |
| use single layer Transformer or LSTM | memory-based RL agent · Passive T-Maze | return | Improved |
Closest papers by meaning
- Not Only Where, But When: Temporal Scheduling for RLVR (NeurIPS 2026 · similarity 0.86)
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.86)
- What are Key Factors for Updates in RL for LLM Reasoning? (NeurIPS 2026 · similarity 0.86)
- Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.86)
- Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
