Research map / Reinforcement learning / Offline RL
Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that return-conditioned supervised learning (RCSL) avoids Bellman completeness requirements but cannot stitch trajectories, and proposes MBRCSL, which uses learned dynamics and behavior-policy models with forward sampling to generate stitched high-return trajectories before RCSL training.
Problem
Off-policy DP-based methods like Q-learning diverge under function approximation without Bellman completeness, while RCSL methods lack trajectory stitching ability when datasets are sub-optimal.
Approach
Use learned dynamics model and behavior policy to roll out trajectories by forward sampling, keep those exceeding the dataset's maximum return, and train an RCSL policy on them, obtaining stitching without any dynamic programming backups or Bellman completeness.
Key results
- On Point Maze, MBRCSL achieves 91.5±7.1 return versus 77.9±41.9 for MOPO, 56.6±50.1 for COMBO, and 57.2±4.1 for DT; dataset highest return is 71.8.
- On PickPlace, ClosedDrawer and BlockedDrawer, MBRCSL reaches success rates of 0.48, 0.51 and 0.68, versus 0.22, 0.11 and 0.34 for CQL and 0 for COMBO and DT.
- Theory: on the LinearQ construction, two-layer MLP RCSL needs constant (16) hidden neurons while Q-learning satisfying Bellman completeness needs Omega(|S|) neurons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MBRCSL instead of offline RL baselines | Pointmaze · Pointmaze | return: 91.5$\pm$7.1 vs 77.9$\pm$41.9 | Helped |
| use MBRCSL instead of CQL, COMBO, DT, or BC | simulated robotic tasks · PickPlace | success rate: 0.48$\pm$0.04 vs 0.22$\pm$0.35; 0.48$\pm$0.04 vs 0$\pm$0; 0.48$\pm$0.04 vs 0.07$\pm$ 0.03 | Helped |
| use MBRCSL instead of CQL, COMBO, DT, or BC | simulated robotic tasks · ClosedDrawer | success rate: 0.51$\pm$0.12 vs 0.11$\pm$0.08; 0.51$\pm$0.12 vs 0$\pm$0; 0.51$\pm$0.12 vs 0.38$\pm$0.02 | Helped |
| use MBRCSL instead of CQL, COMBO, DT, or BC | simulated robotic tasks · BlockedDrawer | success rate: 0.68$\pm$0.09 vs 0.34$\pm$0.23; 0.68$\pm$0.09 vs 0$\pm$0; 0.68$\pm$0.09 vs 0.61$\pm$0.02 | Helped |
| replace transformer dynamics model with MLP ensemble (ablation) | MBRCSL · PickPlace | success rate | Hurt |
| replace diffusion behavior policy with MLP behavior policy (ablation) | MBRCSL · PickPlace | success rate | Hurt |
| replace diffusion behavior policy with transformer behavior policy (ablation) | MBRCSL · PickPlace | success rate | Hurt |
| use RCSL instead of CQL across tested model sizes | Pointmaze expert · Pointmaze expert | return: 89.6 vs 11.5; 97.8 vs 11.7; 93.3 vs 14.7; 113 vs 37.8; 105 vs 23 | Helped |
| use RCSL-16 instead of QL-16 | LinearQ · LinearQ (u=16) | difference from optimal return: 1 -> 0 | Helped |
Surprising findings
- Q-learning with hidden width scaling linearly with state space performs no better than a naive always-action-0 policy on LinearQ.
- Decision Transformer achieves 0 success on all three robotic tasks, failing to extract meaningful behavior due to lack of expert trajectories.
Limitations
- Method currently targets deterministic environments; stochastic environments are left as open problem.
- Scaling to more complex, large-scale environments is not demonstrated.
- Robotics results rely on sophisticated transformer dynamics and diffusion behavior policy, with modest absolute success rates (0.48-0.68).
Experimental setup
Datasets: D4RL Maze (Point Maze custom environment) simulated robotics datasets (PickPlace, ClosedDrawer, BlockedDrawer)
Benchmarks: Point Maze (custom, D4RL Maze based) PickPlace ClosedDrawer BlockedDrawer
Metrics: normalized return success rate mean and standard deviation over 4 seeds
Baselines: CQL COMBO MOPO MOReL Decision Transformer %BC BC
Builds on: Decision Transformer MOPO COMBO CQL diffusion policy
Closest papers by meaning
- Critic-Guided Decision Transformer for Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization (NeurIPS 2024 · similarity 0.86)
- Reinformer: Max-Return Sequence Modeling for Offline RL (ICML 2024 · similarity 0.86)
- Generative Trajectory Stitching through Diffusion Composition (NeurIPS 2025 · similarity 0.86)
- Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.85)
- STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.