Research map / Probabilistic and scientific modeling / Dynamical systems
Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Action-Guided Diffusion Policy (DP-AG), which evolves variational observation latents via a VJP of diffusion noise predictions and aligns static and evolved noise predictions with a cycle-consistent InfoNCE loss, evaluated on Robomimic, Franka Kitchen, Push-T, a new Dynamic Push-T, and a UR5 arm.
Problem
Imitation learning policies decouple perception from action: observation features are held fixed while a short action sequence is generated, so intermediate action feedback never refines perception, causing abrupt or discontinuous motions.
Approach
Treat observation features as a variational latent that evolves across diffusion steps under an SDE whose drift is the VJP of the diffusion noise predictor, and tie static and evolved noise predictions together with a cycle-consistent contrastive loss, giving a closed perception-action loop.
Key results
- DP-AG reaches 0.93 target coverage on Push-T (img) vs 0.87 for DP, and 0.80 vs 0.65 on Dynamic Push-T.
- On UR5 tasks, DP-AG achieves 90% success on Candy Push vs 65% for DP and 85% on Peg-in-Hole vs 0% for DP.
- On Robomimic and Franka Kitchen, DP-AG achieves near-perfect success, e.g. 1.00 on most ph tasks and 0.94 on Transport mh.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DP-AG instead of DP | Push-T img · Push-T img | target coverage: 0.87 -> 0.93 | Helped |
| Use DP-AG instead of DP | Push-T kp · Push-T kp | target coverage: 0.95 -> 0.99 | Helped |
| Use DP-AG instead of DP | Dynamic Push-T img · Dynamic Push-T img | target coverage: 0.65 -> 0.80 | Helped |
| Use DP-AG instead of DP | Robomimic and Franka Kitchen · Robomimic and Franka Kitchen tasks | success rate | Mixed |
| Use DP-AG instead of DP | Painting · Painting | avg jerk: 0.083 -> 0.032 | Helped |
| Use DP-AG instead of DP | Painting · Painting | IoU (%): 68.9 -> 92.1 | Helped |
| Use DP-AG instead of DP | Candy Push · Candy Push | success rate (%): 65.0 -> 90.0 | Helped |
| Add VJP-guided evolution to Base Flow | Irregular spiral regression · Irregular spiral | MSE: 0.0095 -> 0.0052 | Helped |
Surprising findings
- Standard DP achieves 0% success on the Peg-in-Hole UR5 task, which DP-AG solves at 85%.
Limitations
- Theoretical continuity bound relies on assumed Lipschitz continuity and a loosely justified VJP-to-noise proportionality step.
- Real-world evaluation uses only 20 trials per task on a single robot arm.
- Flow-matching extension is described only informally without experiments.
Experimental setup
Datasets: Robomimic Franka Kitchen Push-T Dynamic Push-T irregular spiral dataset
Benchmarks: Robomimic Franka Kitchen Push-T Dynamic Push-T UR5 real-world tasks
Metrics: success rate target coverage action MSE IoU average jerk time to complete
Baselines: LSTM-GMM IBC BET DP AdaFlow FlowPolicy
Builds on: Diffusion Policy InfoNCE stochastic adjoint sensitivity DDPM
Closest papers by meaning
- Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution (NeurIPS 2025 · similarity 0.89)
- LAFP: Preserving Latent Action Structure in Latent Policy Learning via Flow Matching (NeurIPS 2026 · similarity 0.89)
- Latent Diffusion Planning for Imitation Learning (ICML 2025 · similarity 0.89)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.88)
- One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation (ICML 2025 · similarity 0.88)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.