Research map / Reinforcement learning / Visual and representation RL
Accelerating Visual-Policy Learning through Parallel Differentiable Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D.Va, a visual policy learning method that decouples rendering from the computation graph in short-horizon analytical policy gradient (SHAC-style) training, avoiding differentiating through the renderer.
Problem
Visual policy learning with differentiable simulation is costly because backpropagating through rendering requires pixel-space Jacobians, which is memory- and compute-intensive and yields noisy gradients.
Approach
Drop the observation-Jacobian terms (control regularization) from the analytical policy gradient, yielding a decoupled policy gradient that equals a behavior-cloning gradient toward open-loop trajectory-optimized actions; the actor takes stacked rendered frames while the critic is trained on states.
Key results
- On Humanoid, D.Va achieves about 4x improvement in final return over baselines and learns a running policy in 4 hours on a single RTX 4080 GPU.
- D.Va's backward simulation is 2-3x faster than Visual-SHAC with differentiable rendering (e.g., Humanoid 0.68s vs 1.38s).
- Visual-SHAC with differentiable rendering fails on 3D tasks (Ant, Humanoid), with gradient norms exceeding 1e15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decouple visual observations from the computation graph | D.Va versus SHAC with differentiable rendering · 3D visual control tasks | final return | Helped |
| decouple visual observations from the computation graph | D.Va versus SHAC with differentiable rendering · Hopper, Ant, Humanoid, Cartpole | backward simulation time: 0.11(s) vs 0.36(s); 0.19(s) vs 0.51(s); 0.21(s) vs 0.58(s); 0.68(s) vs 1.38(s) | Helped |
| use 1 stacked frame instead of 3 (ablation) | D.Va · Cartpole, Hopper, Ant, Humanoid | final return: $2139 \pm 22.42$ -> $2068 \pm 29.83$; $5067.13 \pm 18.14$ -> $4398.48 \pm 266.80$; $7218 \pm 986.00$ -> $5681 \pm 1344.5 | Mixed |
| use 2 stacked frames instead of 3 (ablation) | D.Va · Cartpole, Hopper, Ant, Humanoid | final return: $2139 \pm 22.42$ -> $2115 \pm 21.20$; $5067.13 \pm 18.14$ -> $5055.23 \pm 6.73$; $7218 \pm 986.00$ -> $7418 \pm 862.39$; | Mixed |
| use 4 stacked frames instead of 3 (ablation) | D.Va · Cartpole, Hopper, Ant, Humanoid | final return: $2139 \pm 22.42$ -> $2155 \pm 25.49$; $5067.13 \pm 18.14$ -> $5055.03 \pm 2.08$; $7218 \pm 986.00$ -> $9680 \pm 222.27$; | Mixed |
| use 5 stacked frames instead of 3 (ablation) | D.Va · Cartpole, Hopper, Ant, Humanoid | final return: $2139 \pm 22.42$ -> $2095 \pm 22.5$; $5067.13 \pm 18.14$ -> $5072.33 \pm 14.62$; $7218 \pm 986.00$ -> $8286 \pm 880.50$; | Mixed |
| use 6 stacked frames instead of 3 (ablation) | D.Va · Cartpole, Hopper, Ant, Humanoid | final return: $2139 \pm 22.42$ -> $2140 \pm 29.84$; $5067.13 \pm 18.14$ -> $5077.63 \pm 23.74$; $7218 \pm 986.00$ -> $8243 \pm 799.45$ | Mixed |
| remove the value function (ablation) | D.Va · most tasks | policy learning effectiveness | Hurt |
| detach observations to use the decoupled policy gradient | SHAC on state space · low-dimensional state-space tasks | training performance | No clear effect |
Surprising findings
- Adding the control regularization (observation Jacobian) term increases gradient norm under visual observations, making optimization noisier, whereas in state space it acts like a residual connection.
Limitations
- Success depends heavily on simulation quality and accuracy; sim-to-real transfer not addressed.
- Discards on-policy samples after each iteration, so sample efficiency is comparable rather than superior to RL baselines.
- Visual-SHAC baseline required smaller horizons and fewer environments due to memory constraints.
Experimental setup
Datasets: simulated control tasks: Cartpole, Hopper, Ant, Humanoid ANYmal locomotion (additional task)
Benchmarks: Cartpole Hopper Ant Humanoid ANYmal locomotion
Metrics: final return wall-clock time environment steps gradient norm memory usage backward simulation time
Baselines: DrQ-v2 CURL DreamerV3 State-to-visual distillation Visual-SHAC with differentiable rendering
Builds on: SHAC DrQ-v2 CURL DreamerV3 PyTorch3D
Closest papers by meaning
- Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulation (ICLR 2025 · similarity 0.87)
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors (NeurIPS 2024 · similarity 0.86)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.86)
- Humanoid Horizon: Extending Task Horizon in Whole-Body Loco-Manipulation via Parallel Training, Dynamic Starting, and Reward Gating (NeurIPS 2026 · similarity 0.86)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.