Research map / Reinforcement learning / MDPs and planning
Rethinking Transformers in Solving POMDPs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Argues Transformers are theoretically and empirically limited for POMDPs (via regular-language reduction and circuit complexity) and proposes linear RNNs (LRU) as a better sequence model for partially observable RL.
Problem
Whether Transformers are suitable sequence models for solving partially observable RL (POMDPs), given their known difficulty with regular languages and length generalization.
Approach
Reduces regular languages to POMDPs, uses circuit complexity (TC0/NC1) and a Lipschitz argument on softmax attention to show Transformer-based (TF, RL) pipelines fail on certain regular-language POMDPs, and advocates pointwise-recurrent linear RNNs (LRU) as a balance between Transformers and RNNs.
Key results
- On PyBullet occlusion tasks, LRU averages 88.2 normalized score versus 74.6 for LSTM and 39.3 for GPT.
- LRU significantly outperforms LSTM on Ant (P, V) and Walker (V); GPT is lower than both recurrent models on all eight occlusion tasks.
- On pure long-term memory tasks, LRU performs comparably to GPT on Passive T-Maze and beats GPT on Passive Visual Match, while outperforming LSTM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Transformer (GPT) with linear RNN (LRU) as sequence model | PyBullet occlusion tasks · PyBullet Occlusion (8 tasks) | normalized score (average): 88.2 vs 39.3 | Helped |
| Use Transformer (GPT) instead of LSTM or LRU as sequence model under partial observability | PyBullet occlusion tasks · PyBullet Occlusion (8 tasks) | normalized score (average): 39.3 vs 74.6 (LSTM) | Hurt |
| Full observability instead of partial observability (ablation) | GPT vs LRU vs LSTM · PyBullet fully observable tasks | normalized score | No clear effect |
Surprising findings
- With full observability, GPT performs close to LRU and LSTM, suggesting its POMDP gap stems from partial observability.
Limitations
- LRU lacks theoretical guarantees for modeling regular languages.
- Theory focuses on exploitation and ignores exploration in RL.
- Continuous-state POMDPs do not directly map to regular languages, so the complexity argument is less direct there.
Experimental setup
Datasets: D4RL medium-expert PyBullet occlusion tasks Passive T-Maze Passive Visual Match synthetic regular language POMDPs (PARITY, EVEN PAIRS, S5)
Benchmarks: PyBullet occlusion tasks (Ant, Cheetah, Hopper, Walker) Passive T-Maze Passive Visual Match regular language POMDPs (PARITY, EVEN PAIRS, S5)
Metrics: normalized return return MSE ratio t-SNE visualization of hidden states
Baselines: GPT (Transformer) LSTM Passive T-Maze memory baselines
Builds on: Linear Recurrent Unit (LRU) DQN TD3 pomdp-baselines
Closest papers by meaning
- Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate (NeurIPS 2024 · similarity 0.89)
- Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning (ICML 2024 · similarity 0.88)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.88)
- Rethinking Decision Transformer via Hierarchical Reinforcement Learning (ICML 2024 · similarity 0.88)
- Scalable Policy-Based RL Algorithms for POMDPs (NeurIPS 2025 · similarity 0.88)
- Model-based Reinforcement Learning for Confounded POMDPs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.