Research map / Reinforcement learning / Offline RL
GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GTA, which partially noises offline trajectories with a trajectory-level conditional diffusion model and denoises them under amplified return classifier-free guidance, producing augmented data for any offline RL algorithm.
Problem
Offline RL suffers from limited data coverage and suboptimal datasets; existing augmentation (noise injection, transition-level generative synthesis) either only smooths local regions or is constrained by the offline reward distribution.
Approach
Generate trajectory-level augmentations by partially noising real trajectories with the diffusion forward process and denoising them with classifier-free guidance conditioned on an amplified return (alpha times the original return), so augmented data is both high-return and dynamically plausible, decoupled from policy training.
Key results
- Across TD3BC, CQL, IQL, MCQ on Gym locomotion, GTA raises average normalized score (e.g., TD3BC 76.92 to 84.63 average over 9 locomotion datasets; IQL 78.52 to 86.11).
- On AntMaze with IQL, GTA averages 62.75 vs 52.04 without augmentation and 36.99 for SynthER, where SynthER degrades performance.
- On FrankaKitchen with IQL, GTA averages 53.3 vs 50.23 baseline; on Adroit pen/door-human averages 42.73 vs 36.43.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GTA augmentation | TD3BC · Gym locomotion (Halfcheetah, Hopper, Walker2d) | normalized average score: 78.64 -> 84.63 | Helped |
| Add GTA augmentation | CQL · Gym locomotion (Halfcheetah, Hopper, Walker2d) | normalized average score: 81.20 -> 85.27 | Helped |
| Add GTA augmentation | IQL · Gym locomotion (Halfcheetah, Hopper, Walker2d) | normalized average score: 81.97 -> 86.11 | Helped |
| Add GTA augmentation | MCQ · Gym locomotion (Halfcheetah, Hopper, Walker2d) | normalized average score: 84.52 -> 85.91 | Helped |
| Add GTA augmentation | IQL · Maze2d and AntMaze | normalized average score: 53.99 -> 62.75 | Helped |
| Add GTA augmentation | IQL · Adroit (pen-human, door-human) | normalized average score: 37.95 -> 42.73 | Helped |
| Add GTA augmentation | IQL · FrankaKitchen | normalized average score: 50.85 -> 53.3 | Helped |
| Add GTA augmentation | DrQ+BC · Pixel-based Cheetah-run and Walker-walk | normalized average score: 42.7 -> 45.4 | Helped |
| Generate trajectories with H longer than 1 rather than transitions with H=1 (ablation) | GTA + TD3BC · Halfcheetah-medium-v2 | D4RL score: 46.59 -> 55.85 (H=16); 46.59 -> 57.41 (H=32); 46.59 -> 58.40 (H=64); 46.59 -> 56.93 (H=128) | Helped |
| Use reweighted sampling (ablation) | GTA · Hopper-medium-v2 | D4RL score: 56.95 -> 57.45 | Helped |
| Use reweighted sampling (ablation) | GTA · Walker2d-medium-v2 | D4RL score: 86.43 -> 88.26 | Helped |
| Increase noising ratio (ablation) | GTA + TD3BC · Halfcheetah-medium-v2 | D4RL score | Helped |
Surprising findings
- SynthER substantially degrades performance on AntMaze (IQL average 36.99 vs 52.04 without augmentation) and on sparse expert mixed-quality data (SynthER ~2.0 vs 44.64 baseline).
- Unconditional or fixed-return conditioning fails to improve data optimality, while amplified return conditioning does.
Limitations
- No explicit transition or reward model is trained, so tasks where dynamic violations matter critically may not benefit.
- Hyperparameters mu and alpha must be selected per dataset/environment via search spaces.
- Focused on offline setting; off-to-online and online settings left for future work.
Experimental setup
Datasets: D4RL halfcheetah/hopper/walker2d (medium, medium-replay, medium-expert) D4RL maze2d D4RL antmaze D4RL adroit D4RL franka kitchen VD4RL
Benchmarks: D4RL VD4RL
Metrics: normalized D4RL score oracle reward novelty dynamic MSE
Baselines: None (no augmentation) S4RL SynthER Naive duplication Decision Transformer
Builds on: SynthER S4RL Elucidated Diffusion Model classifier-free guidance Decision Diffuser
Closest papers by meaning
- Trajectory Generation with Conservative Value Guidance for Offline Reinforcement Learning (ICLR 2026 · similarity 0.92)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.89)
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching (ICML 2024 · similarity 0.88)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.88)
- Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR 2024 · similarity 0.88)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.