Research map / Reinforcement learning / Visual and representation RL
Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies when dynamics-invariant data augmentation helps model-free off-policy RL in sparse-reward tasks, isolating state-action coverage, reward density, and augmented replay ratio.
Problem
Data augmentation improves RL data efficiency, but it is unclear which aspects of dynamics-invariant augmentation (state-action coverage, reward density, augmented replay ratio) drive the gains in sparse-reward tasks.
Approach
Proposes a controllable framework that integrates dynamics-invariant augmented transitions into off-policy updates via separate replay buffers, and uses controlled augmentation variants (proximal relabeling with probability p, varying augmentation ratio m) to disentangle coverage, reward density, and augmented replay ratio.
Key results
- In Slide, PickAndPlace, and Flip, TranslateGoalProximal(0) is as data efficient as TranslateGoal, indicating increased state-action coverage alone explains gains there.
- In PickAndPlace, 100% success is reached with augmented replay ratio 0.0625 while no learning occurs at ratio 0.5.
- In Goal2D, TranslateProximal(0.05) outperforms agents with x8 as much policy-generated data in generalization experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Decrease the augmented replay ratio by generating more augmented transitions per observed transition (ablation) | Data augmentation in the Panda tasks · Panda tasks | data efficiency | Helped |
| Decrease the augmented replay ratio to β = 0.0625 instead of β = 0.5 (ablation) | Data augmentation in PandaPickAndPlace-v3 · PandaPickAndPlace-v3 | success rate: no learning occurs with β = 0.5; 100% success rate can be achieved with β = 0.0625 | Helped |
| Decrease the augmented replay ratio (ablation) | Translate and Rotate augmentation · Goal2D-v0 | data efficiency | Helped |
| Increase state-action coverage without adding reward signal (ablation) | TranslateGoalProximal(0) · PandaSlide-v3, PandaPickAndPlace-v3, and PandaFlip-v3 | data efficiency | Helped |
| Increase state-action coverage without adding reward signal (ablation) | TranslateProximal(0) and TranslateGoalProximal(0) · Goal2D-v0 and PandaPush-v3 | data efficiency | Helped |
| Increase reward density by varying p (ablation) | TranslateProximal(p) and TranslateGoalProximal(p) · PandaSlide-v3 and PandaFlip-v3 | data efficiency | No clear effect |
| Increase reward density to p = 0.05 (ablation) | TranslateProximal(p) and TranslateGoalProximal(p) · PandaPush-v3 and PandaPickAndPlace-v3 | data efficiency | Helped |
| Increase reward density to the largest tested p values (ablation) | TranslateProximal(p) and TranslateGoalProximal(p) · Goal2D-v0, PandaPush-v3, and PandaPickAndPlace-v3 | data efficiency | Hurt |
| Increase the update ratio while keeping the augmented replay ratio fixed (ablation) | Data augmentation in the Panda tasks · PandaSlide-v3 | performance | Hurt |
| Increase batch size without increasing the amount of learning data (ablation) | DDPG agents in the Panda tasks · Panda tasks | performance | Hurt |
| Decrease the augmented replay ratio with Reflect augmentation (ablation) | Reflect · Swimmer-v4, Walker2d-v4, Ant-v4, and Humanoid-v4 | data efficiency | Mixed |
| Decrease the augmented replay ratio (ablation) | TranslateGoal augmentation · Panda tasks, generalization evaluation | generalization performance | Helped |
Surprising findings
- Decreasing the augmented replay ratio alone substantially improves data efficiency, and some tasks are unsolvable at high replay ratios.
- Augmentations that generate no reward signal (proximal(0)) can match full DA in several sparse-reward tasks, so reward density is not critical.
Limitations
- Analysis limited to low-dimensional sparse-reward tasks with continuous actions and off-policy model-free algorithms (DDPG, TD3).
- Reward density and state-action coverage cannot be fully disentangled since reward depends on state and action.
- Findings may differ for dense rewards, discrete actions, or visual observations; visual augmentations are out of scope.
Experimental setup
Datasets: panda-gym PandaPush-v3 panda-gym PandaSlide-v3 panda-gym PandaPickAndPlace-v3 panda-gym PandaFlip-v3 Goal2D-v0 (custom 2D navigation) MuJoCo Swimmer-v4, Walker2d-v4, Ant-v4, Humanoid-v4 (appendix)
Benchmarks: panda-gym PandaPush-v3 panda-gym PandaSlide-v3 panda-gym PandaPickAndPlace-v3 panda-gym PandaFlip-v3 Goal2D-v0 MuJoCo Swimmer-v4, Walker2d-v4, Ant-v4, Humanoid-v4
Metrics: interquartile mean success rate interquartile mean return data efficiency (updates to reach performance) generalization success rate
Baselines: No data augmentation x2 policy data TranslateGoal Translate TranslateProximal(0) TranslateGoalProximal(0)
Builds on: DDPG TD3 Hindsight Experience Replay RAD CoDA
Closest papers by meaning
- Revisiting Data Augmentation in Deep Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Control (NeurIPS 2024 · similarity 0.89)
- ACAMDA: Improving Data Efficiency in Reinforcement Learning through Guided Counterfactual Data Augmentation (AAAI 2024 · similarity 0.88)
- GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL (ICLR 2025 · similarity 0.87)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.