Research map / Reinforcement learning / Visual and representation RL
DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynaGuide, which steers pretrained diffusion policies at inference by taking gradients of a latent DinoV2 dynamics-model distance metric to guide the action denoising process, without retraining the base policy.
Problem
Steering pretrained robot diffusion policies toward or away from multiple inference-time objectives without retraining the policy, sampling excessively, or anticipating all steering goals during training.
Approach
Separate a latent visual dynamics model (transformer over frozen DinoV2 patch embeddings) from the base policy and use the gradient of a log-sum-exp latent distance metric over positive and negative guidance images to steer the DDIM action denoising process (classifier guidance with stochastic sampling).
Key results
- On CALVIN ArticulatedParts, DynaGuide boosts the target behavior 8.7x over the base policy with an average success of 70%.
- With underspecified goals, goal conditioning falls below 10% success while DynaGuide outperforms it by 5.4x on average.
- On a real robot with an off-the-shelf policy, DynaGuide reaches 72.5% average success in CupPreference and 80% in HiddenCup; it achieves 40% success on Switch-On with only 1% of that behavior in base policy training data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply dynamics guidance during action denoising | CALVIN diffusion policy · CALVIN ArticulatedParts | target behavior success: 70% | Helped |
| Use DynaGuide instead of Goal Conditioning | CALVIN policy with underspecified objectives · CALVIN UnderspecifiedObjectives | target behavior success: 5.4x | Helped |
| Use active diffusion guidance instead of action sampling | CALVIN diffusion policy · CALVIN MultiObjectives | desired behavior success | Helped |
| Guide toward an underrepresented Switch-On behavior | CALVIN diffusion policy trained with reduced Switch-On data · CALVIN UnderrepresentedBehaviors | Switch-On success: 40% success rate with only 1% of the original Switch-On behavior | Helped |
| Use DynaGuide to steer cup preference | Pretrained cup-to-saucer policy · Real robot CupPreference | target behavior success: 72.5% | Helped |
| Use active guidance to select the hidden red cup | Pretrained cup-to-saucer policy · Real robot HiddenCup | red cup selection success: 80% | Helped |
| Use the dynamics model to guide a mug policy toward a mouse | Pretrained mug-arrangement policy · Real robot NovelBehavior | interactions with novel object | Helped |
| Train the dynamics model with noised actions (ablation) | DynaGuide · CALVIN ArticulatedParts | steering success | Helped |
| Train the dynamics model on data from four CALVIN environments instead of only the test environment (ablation) | DynaGuide · CALVIN ArticulatedParts | steering success | Helped |
| Increase the number of stochastic sampling steps (ablation) | DynaGuide · CALVIN Switch-On | guidance performance | Helped |
| Use very low guidance strength (ablation) | DynaGuide · CALVIN Switch-On | success rate | Hurt |
| Use very low sigma (ablation) | DynaGuide · CALVIN Switch-On | guidance success | Hurt |
Surprising findings
- DynaGuide is very robust to the number of guidance conditions, achieving comparable guidance even with one guidance condition.
- Pretraining the dynamics model with noised actions is essential for steering performance.
Limitations
- Guidance conditions are outcome observations, so it is difficult to specify how the objective should be achieved (method specification).
- Requires per-task grid-searched guidance strength and sigma; real-robot results lack error bars due to resource intensity.
- Compute-heavy development; the dynamics model needs substantial training data and per-step gradient computation during inference.
Experimental setup
Datasets: CALVIN play data UMI interface data open-source mouse and mug manipulation datasets
Benchmarks: CALVIN BlockTouch 2D real-robot cup arrangement (CupPreference, HiddenCup, NovelBehavior)
Metrics: target behavior success rate behavior distribution failure rate
Baselines: Base Policy DynaGuide-Sampling (GPC) Goal Conditioning Position Guidance (ITPS) Classifier-Free Guidance GC w/ HER
Builds on: classifier guidance DDIM Dino-WM generative trajectory conditioning (GPC) classifier-free guidance
Closest papers by meaning
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.89)
- One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation (ICML 2025 · similarity 0.88)
- DiRecT: Safe Diffusion-Based Planning via Receding-Horizon Denoising (NeurIPS 2026 · similarity 0.87)
- Two-Steps Diffusion Policy for Robotic Manipulation via Genetic Denoising (NeurIPS 2025 · similarity 0.87)
- Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous and Instruction-guided Driving (CVPR 2024 · similarity 0.86)
- Model-based Diffusion for Trajectory Optimization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.