Research map / Reinforcement learning / Visual and representation RL
Revisiting Data Augmentation in Deep Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes image-based DRL data augmentation methods (RAD, DrQ, DrAC, SVEA, DrQ-v2) under a unified actor-critic scheme via variance and bias analysis, and proposes a principled method adding KL regularization and tangent prop to the critic.
Problem
Image-based deep RL data augmentation methods are numerous and empirically compared, but it is unclear which technique to prefer and how they relate; their effects on sample efficiency and generalization are not theoretically understood.
Approach
Expresses existing augmentation methods as explicit vs implicit regularization within one off-policy actor-critic scheme, analyzes variance of Q-targets and actor/critic losses, and adds a KL regularization on the actor and a tangent prop regularization on the critic (novel in DRL).
Key results
- Proposed method achieves state-of-the-art sample efficiency in most DMControl environments compared with RAD, DrQ, DrAC, DrQ-v2 and SVEA (aggregated over 9 environments).
- On generalization with video-hard backgrounds after 500k steps, the method improves over SVEA notably in ball in cup catch, finger spin and walker walk.
- KL regularization and tangent prop reduce variance of target Q and critic loss, per the walker-run case study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use more augmented samples (RAD [M=1,K=1] → RAD+ [M=2,K=1] → DrQ [M=2,K=2]) (ablation) | DrQ-style critic training · DeepMind Control Suite, 9 environments, normal background | critic-loss variance | Helped |
| Use more augmented samples (RAD [M=1,K=1] → RAD+ [M=2,K=1] → DrQ [M=2,K=2]) (ablation) | DrQ-style critic training · DeepMind Control Suite, 9 environments, normal background | target Q variance | Helped |
| Add KL regularization between policies on augmented states (ablation) | DrQ · DeepMind Control Suite, 9 environments, normal background | actor invariance | Helped |
| Add KL regularization between policies on augmented states (ablation) | DrQ · DeepMind Control Suite, 9 environments, normal background | target Q variance | Helped |
| Add KL regularization between policies on augmented states (ablation) | DrQ · DeepMind Control Suite, 9 environments, normal background | actor-loss variance | Helped |
| Add tangent prop regularization to the critic (ablation) | DrQ with KL regularization · DeepMind Control Suite, 9 environments, normal background | critic invariance | Helped |
| Use the proposed data-augmentation method instead of the evaluated baselines | RAD, DrQ, DrAC, DrQv2, and SVEA · DeepMind Control Suite, normal background | sample efficiency | Helped |
| Use the proposed method instead of SVEA with random overlay | SVEA · DeepMind Control Suite, video-hard backgrounds | generalization | Mixed |
| Use random conv transformations in target Q-value calculation instead of only random shift (ablation) | Data-augmented critic training · DeepMind Control Suite | critic bias | Hurt |
| Increase updates per iteration when using complex image transformations in target Q-value calculation (ablation) | Data-augmented critic training · DeepMind Control Suite | performance | Helped |
| Use KL regularization with the transformed-state policy as the target rather than the averaged policy (ablation) | Proposed actor loss · Theoretical comparison | KL regularization loss | Hurt |
| Use the reversed KL direction that detaches the original-state policy (ablation) | Policy invariance regularization · DeepMind Control Suite | training stability | Hurt |
Surprising findings
- Using complex random conv in the target does not induce large target variance, though it produces a larger bias in the trained critic.
Limitations
- Theoretical propositions hold under critic invariance assumptions that are only approximately satisfied in practice.
- Tangent prop increases computational cost due to Q differentiation.
Experimental setup
Datasets: DeepMind Control Suite
Benchmarks: DeepMind Control Suite
Metrics: cumulative reward standard deviation of critic loss variance of target Q cosine similarity of augmented features
Baselines: RAD DrQ DrAC DrQ-v2 SVEA
Builds on: DrQ RAD DrAC SVEA DrQ-v2 SAC
Closest papers by meaning
- Automatic Data Augmentation for Generalization in Reinforcement Learning (NeurIPS 2021 · similarity 0.91)
- Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning (NeurIPS 2023 · similarity 0.91)
- Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR 2024 · similarity 0.91)
- Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation (NeurIPS 2021 · similarity 0.90)
- Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Control (NeurIPS 2024 · similarity 0.88)
- Zero Shot Generalization of Vision-Based RL Without Data Augmentation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.