Research map / Reinforcement learning / Offline RL
Offline Reinforcement Learning as One Big Sequence Modeling Problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| quantile instead of uniform discretization (ablation) | Trajectory Transformer · HalfCheetah Med-Expert | normalized return: 40.8 -> 95.0 | Helped |
| quantile instead of uniform discretization (ablation) | Trajectory Transformer · D4RL locomotion average | normalized return: 72.6 -> 78.9 | Helped |
| quantile instead of uniform discretization (ablation) | Trajectory Transformer · Hopper Med-Replay | normalized return: 99.4 -> 91.5 | No clear effect |
| replace Monte Carlo reward-to-go with IQL Q-function as beam search heuristic | Trajectory Transformer · AntMaze average | normalized return: 63.2 -> 84.0 (vs IQL) | Helped |
| beam search planning with Transformer vs return-conditioning (Decision Transformer) | Trajectory Transformer (+Q) · AntMaze average | normalized return: 11.8 -> 84.0 | Helped |
| limit Transformer context to one timestep (Markovian variant) (ablation) | Trajectory Transformer · fully-observed humanoid long-horizon prediction | prediction log-likelihood | No clear effect |
| long-horizon context under 50% state masking (ablation) | Trajectory Transformer · partially observed humanoid | prediction log-likelihood | Helped |
| Transformer with autoregressive discretized states instead of feedforward Gaussian ensemble dynamics model | dynamics model · humanoid error accumulation | long-horizon accuracy | Helped |
Closest papers by meaning
- Overcoming Slow Decision Frequencies in Continuous Control: Model-Based Sequence Reinforcement Learning for Model-Free Control (ICLR 2025 · similarity 0.91)
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.91)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.90)
- Reinformer: Max-Return Sequence Modeling for Offline RL (ICML 2024 · similarity 0.90)
- Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.90)
- Towards General-Purpose Model-Free Reinforcement Learning (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.