Research map / Reinforcement learning / Offline RL
Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TempDATA, an offline model-based RL method that learns a temporal-distance-aware latent autoencoder, trains a latent dynamics model, and augments transitions in that latent space for sparse-reward goal-reaching tasks.
Problem
Offline model-based RL methods fail on sparse-reward, long-horizon goal-reaching tasks (e.g., AntMaze), often generating transitions in inaccessible regions beyond obstacles.
Approach
Learn an autoencoder whose latent space captures temporal distance at trajectory and transition levels (via expectile-regression Bellman targets and a transition-consistency loss), then perform model rollouts in this latent space and decode augmented transitions back to state space for any offline RL/skill RL/GCRL policy.
Key results
- On D4RL AntMaze, TempDATA achieves total score 520.2 versus 272.6 for the second-best MBRL (ROMI), and best on 8 of 8 tasks in the MBRL comparison.
- On multi-goal FrankaKitchen and CALVIN, TempDATA gets total score 185.7, the highest among GCRL baselines (HIQL 176.5).
- On pixel-based Visual FrankaKitchen, TempDATA scores 114.6 total, comparable to GC-IQL (116.5), while RepB-SDE scores 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TempDATA instead of the strongest listed baseline | AntMaze umaze · AntMaze umaze | normalized success score: 80.3 ± 18.5 -> 96.3 ±0.0 | Helped |
| Use TempDATA instead of the strongest listed baseline | AntMaze umaze-diverse · AntMaze umaze-diverse | normalized success score: 51.6 ± 23.4 -> 90.4 ±8.6 | Helped |
| Use TempDATA instead of GTA | AntMaze medium-play · AntMaze medium-play | normalized success score: 81.9 ± 8.4 -> 74.8 ±8.3 | Hurt |
| Use TempDATA instead of GTA | AntMaze medium-diverse · AntMaze medium-diverse | normalized success score: 78.1 ± 15.8 -> 69.5 ±10.8 | Hurt |
| Use TempDATA instead of GTA | AntMaze large-play · AntMaze large-play | normalized success score: 44.4 ± 9.3 -> 56.5 ±14.1 | Helped |
| Use TempDATA instead of GTA | AntMaze large-diverse · AntMaze large-diverse | normalized success score: 47.8 ± 13.4 -> 44.2 ±15.3 | Hurt |
| Use TempDATA instead of ROMI | AntMaze ultra-play · AntMaze ultra-play | normalized success score: 4.9 ±2.1 -> 53.2 ±18.2 | Helped |
| Use TempDATA instead of ROMI | AntMaze ultra-diverse · AntMaze ultra-diverse | normalized success score: 8.8 ±6.8 -> 35.3 ±10.9 | Helped |
| Use TempDATA instead of HIQL | FrankaKitchen · kitchen-mixed | average return: 67.7 ±6.8 -> 65.3 ±11.7 | Hurt |
| Use TempDATA instead of HIQL | FrankaKitchen · kitchen-partial | average return: 65.0 ±9.2 -> 70.0 ±12.8 | Helped |
| Use TempDATA instead of HIQL | CALVIN · CALVIN | average return: 43.8 ±39.5 -> 50.4 ± 34.6 | Helped |
| Use TempDATA instead of GC-IQL | Visual FrankaKitchen · kitchen-mixed | average return: 52.9 ±4.7 -> 58.1 ±5.9 | Helped |
Surprising findings
- Naive model-based rollouts consistently reduce returns in the ablation, while the latent temporal-distance augmentation restores or improves performance.
- TempDATA does not use an ensemble of dynamics models, unlike prior MBRL methods, yet reduces training time substantially.
Limitations
- Assumes deterministic environments and a symmetric temporal distance metric; stochastic or partial observations are not addressed.
- Performance on CALVIN and FrankaKitchen has large standard deviations (e.g., 50.4 ± 34.6 on CALVIN).
Experimental setup
Datasets: D4RL AntMaze D4RL Kitchen (partial, mixed) CALVIN D4RL halfcheetah and walker2d (medium-replay, medium-expert)
Benchmarks: D4RL AntMaze FrankaKitchen (D4RL) CALVIN Visual FrankaKitchen D4RL halfcheetah D4RL walker2d
Metrics: normalized success rate total score
Baselines: S4RL SynthER GTA MOPO COMBO RAMBO RepB-SDE ROMI GCSL GC-CQL GC-IQL GC-POR
Builds on: MOPO HILP HIQL IQL hindsight experience relabeling
Closest papers by meaning
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.