Research map / Reinforcement learning / Visual and representation RL
When Should We Prefer State-to-Visual DAgger over Visual Reinforcement Learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical comparison of a two-stage State-to-Visual DAgger (state SAC teacher then online visual imitation) against visual RL (Asymmetric Actor Critic with SAC) across 16 ManiSkill, DMControl, and Adroit tasks.
Problem
When is a two-stage state-to-visual DAgger pipeline preferable to single-stage visual RL for learning policies from high-dimensional visual inputs, in terms of asymptotic performance, sample efficiency, and wall-clock cost?
Approach
Provides a standardized implementation of State-to-Visual DAgger (off-policy DAgger with replay buffer and loss-threshold early stopping) and benchmarks it against an asymmetric actor-critic visual RL baseline across 16 tasks, distilling practical recommendations on when to use each.
Key results
- State-to-Visual DAgger substantially outperforms visual RL on hard tasks (e.g., OpenDrawer 95.66 vs 43.67 success; Relocate 48.84 vs 0; Humanoid-Walk 296.29 vs 91.58 return), but is similar or slightly worse on easy tasks (e.g., Walker-Run 770.66 vs 812.00).
- State-to-Visual DAgger gives narrower confidence intervals and more stable performance at convergence than visual RL.
- No clear sample-efficiency advantage for State-to-Visual DAgger when both methods solve the task, but it generally reduces wall-clock training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use State-to-Visual DAgger instead of visual RL | Visual policy learning · Hard tasks | asymptotic performance | Helped |
| Use State-to-Visual DAgger instead of visual RL | Visual policy learning · Easy tasks | asymptotic performance | Mixed |
| Use State-to-Visual DAgger instead of visual RL | Visual policy learning · Tasks where both methods solve effectively | environment steps | No clear effect |
| Use State-to-Visual DAgger instead of visual RL | Visual policy learning · Most tasks | wall-clock time | Helped |
| Use State-to-Visual DAgger instead of visual RL | Visual policy learning · All benchmarks and difficulty levels | convergence stability | Helped |
| Use off-policy DAgger with a replay buffer instead of on-policy DAgger (ablation) | State-to-Visual DAgger · Visual imitation training | performance | Helped |
| Add imitation-loss-threshold early stopping (ablation) | State-to-Visual DAgger · Visual imitation training | wall-clock training time | Helped |
| Use Asymmetric Actor Critic instead of DrQ-v2 | Visual RL · Four overlapping DMControl tasks | performance | Helped |
Surprising findings
- Visual RL can unlearn over training (e.g., Adroit Pen), and State-to-Visual DAgger's apparent sample-efficiency gains on hard tasks mainly reflect better asymptotic performance rather than true efficiency.
Limitations
- Easy/hard task split based on a 4M-step threshold is admittedly not rigorous.
- Checkpoint selection effects for the state teacher were not investigated.
- Only three seeds per task; visual RL uses a single representative algorithm and no advanced image feature extraction or augmentation.
Experimental setup
Datasets: ManiSkill2 tasks DMControl tasks Adroit tasks
Benchmarks: ManiSkill2 DMControl Adroit
Metrics: success rate episode return normalized score sample efficiency (environment steps) wall-clock time 95% confidence interval
Baselines: Visual RL (Asymmetric Actor Critic with SAC) DrQ-v2
Builds on: DAgger Soft Actor-Critic Asymmetric Actor Critic DrQ-v2
Closest papers by meaning
- The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages (ICLR 2024 · similarity 0.87)
- DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors (NeurIPS 2024 · similarity 0.87)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.87)
- Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning (ICML 2025 · similarity 0.87)
- DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.