Research map / Image generation / Autoregressive and latent generation
Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLOVER, a closed-loop visuomotor framework that pairs a text-conditioned RGB-D video diffusion planner (with optical-flow regularization) with a feedback policy that measures embedding distances to sub-goals to transition sub-goals and replan, reaching state-of-the-art on CALVIN ABC→D.
Problem
Long-horizon robotic manipulation with video-diffusion visual planners is typically open-loop, so execution errors accumulate and sub-goal infeasibility goes undetected; existing feedback signals (pixel differences, pretrained visual features) are poorly suited for error measurement.
Approach
Learn a measurable state-embedding space by modeling the difference between current and goal embeddings as an explicit error that drives an inverse-dynamics action decoder; use cosine distance in this space to adaptively transition sub-goals and to detect unreachable generated plans for replanning.
Key results
- On CALVIN ABC→D, CLOVER reaches 3.53 average length (45.4% five-task success) versus 2.69 for SuSIE and 3.27 for 3D Diffuser Actor.
- On real-world long-horizon tasks, CLOVER achieves 2.1 average length versus 1.1 for RT-1.
- Under visual distraction, CLOVER gets 1.47 average length versus 0.47 for RT-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use closed-loop feedback for adaptive sub-goal selection instead of open-loop execution (ablation) | CLOVER on CALVIN · CALVIN ABC→D | Avg. Len.: +0.44 average completed task length | Helped |
| Add adaptive replanning to detect unreliable visual plans (ablation) | CLOVER on CALVIN · CALVIN ABC→D | Avg. Len. | Helped |
| Add optical flow regularization to the visual planner (ablation) | CLOVER visual planner · Video prediction | SSIM: 0.848 -> 0.858 | Helped |
| Add optical flow regularization to the visual planner (ablation) | CLOVER visual planner · CALVIN ABC→D | Avg. Len.: 3.26 -> 3.53 | Helped |
| Use depth information in the policy's multimodal inputs (ablation) | CLOVER policy on CALVIN · CALVIN ABC→D | Avg. Len. | Helped |
| Use convolution-based multimodal fusion (ablation) | CLOVER policy on CALVIN · CALVIN ABC→D | Avg. Len. | Helped |
| Increase diffusion sampling steps (ablation) | CLOVER visual planner · CALVIN ABC→D | Avg. Len. | Mixed |
| Use learned CLOVER state embeddings rather than CLIP features for error measurement (ablation) | CLOVER feedback mechanism · CALVIN ABC→D | Avg. Len.: 1.63 -> 3.53 | Helped |
| Use learned CLOVER state embeddings rather than LIV representations for error measurement (ablation) | CLOVER feedback mechanism · CALVIN ABC→D | Avg. Len.: 1.77 -> 3.53 | Helped |
| Use VC-1 rather than ViT-S encoder (ablation) | CLOVER inverse-dynamics policy · CALVIN ABC→D | Avg. Len.: 3.35 -> 3.53 | Helped |
| Use VC-1 rather than ViT-B encoder (ablation) | CLOVER inverse-dynamics policy · CALVIN ABC→D | Avg. Len.: 3.46 -> 3.53 | Helped |
| Use VC-1 rather than CLIP encoder (ablation) | CLOVER inverse-dynamics policy · CALVIN ABC→D | Avg. Len.: 3.43 -> 3.53 | Helped |
Surprising findings
- CLIP and LIV embeddings have very narrow distance ranges (within about 6e-2), making them unreliable for closed-loop feedback; unreliable measurements can make closed-loop performance worse than open-loop.
Limitations
- Relies on heavy training on in-domain simulation and real-world data; generalization from web-scale pretraining is untested.
- Task completion assessment in real-world settings is only explored in an appendix and not used for main comparisons.
- Authors hypothesize the performance bottleneck lies in the visual planner.
Experimental setup
Datasets: CALVIN real-world demonstrations (AIRBOT Play)
Benchmarks: CALVIN ABC→D real-world long-horizon tasks visual distraction and dynamic scene generalization
Metrics: task success rate per chain position average completed task length SSIM PSNR LPIPS FID RMSE (depth)
Baselines: MCIL HULC RT-1 RoboFlamingo GR-1 3D Diffuser Actor UniPi SuSIE ACT R3M LIV CLIP
Builds on: UniPi AVDC Imagen RAFT SuSIE
Closest papers by meaning
- FreqPolicy: Efficient Flow-based Visuomotor Policy via Frequency Consistency (NeurIPS 2025 · similarity 0.87)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.85)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.85)
- Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution (NeurIPS 2025 · similarity 0.85)
- Improving Generative Behavior Cloning via Self-Guidance and Adaptive Chunking (NeurIPS 2025 · similarity 0.84)
- FreqPolicy: Frequency Autoregressive Visuomotor Policy with Continuous Tokens (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.