Research map / Vision-language and multimodal / Robot action and navigation
GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GEVRM, a closed-loop VLA that combines a text-guided video diffusion transformer (goal planner) with prototype-contrastive state alignment and a goal-guided diffusion policy, inspired by internal model control, to resist visual perturbations in robot manipulation.
Problem
VLA models for robot manipulation degrade under external perturbations (lighting changes, noise, occlusion, image shifts) that introduce unforeseen state information at deployment.
Approach
Instantiates the internal model control principle in a VLA: a text-conditioned video diffusion planner generates future goal frames, and internal embeddings of current and goal states are aligned via prototypical contrastive learning (SwAV-style Sinkhorn assignments) to estimate perturbations, feeding a goal-guided diffusion policy.
Key results
- On CALVIN ABC->D, GEVRM achieves 0.92/0.70/0.54/0.41/0.26 success across 1-5 chained instructions, above SuSIE (0.87/0.69/0.49/0.38/0.26).
- On the perturbed CALVIN benchmark (five perturbations averaged), GEVRM reaches avg. length 1.62 versus 1.44 for GR-1 and 1.11 for SuSIE.
- Goal generation on CALVIN improves FID to 94.47 from 236.75 (GR-1) and 214.14 (SuSIE).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GEVRM instead of SuSIE for video goal generation | Goal generation · BridgeData | FID: 114.79$\pm${ 21.38} -> \textbf{35.70}$\pm${ 10.77} | Helped |
| Use GEVRM instead of SuSIE for video goal generation | Goal generation · CALVIN | FID: 214.14$\pm${ 45.45} -> \textbf{94.47}$\pm${ 22.54} | Helped |
| Use GEVRM rather than SuSIE | Action execution · CALVIN ABC→D, 1 instruction | success rate: 0.87 -> 0.92 | Helped |
| Use GEVRM rather than GR-1 | Action execution · CALVIN perturbed D average | Avg. Length: 1.44 -> 1.62 | Helped |
| Fine-tune the VAE on robot data (ablation) | GEVRM behavior planner · CALVIN Env. D | task performance | Helped |
| Add state alignment (ablation) | GEVRM policy · CALVIN Env. D | task performance | Helped |
| Vary the state-alignment weight λ (ablation) | GEVRM policy · CALVIN ABC→D | performance | No clear effect |
| Reduce video sampling steps from 50 to 10 (ablation) | GEVRM behavior planner · CALVIN Noise Interference | Avg. Length: 1.76 -> 1.67 | No clear effect |
| Reduce video sampling steps from 50 to 10 (ablation) | GEVRM behavior planner · CALVIN Noise Interference | inference time [s]: 0.598 -> 0.135 | Helped |
| Replace the diffusion policy with an MLP (ablation) | GEVRM action policy · CALVIN Noise Interference | Avg. Length: 1.76 -> 1.40 | Hurt |
| Increase diffusion-policy open-loop control steps from 1 to 4 (ablation) | GEVRM action policy · CALVIN Noise Interference | inference time [s]: 0.077 -> 0.020 | Helped |
| Increase diffusion-policy open-loop control steps from 1 to 4 (ablation) | GEVRM action policy · CALVIN Noise Interference | Avg. Length: 1.76 -> 1.64 | Hurt |
Surprising findings
- Reducing behavior planner sampling steps from 50 to 10 cuts inference time from 0.598 s to 0.135 s with little drop in success rate.
- Under the noise interference perturbation, SuSIE drops to avg. length 0.36 while GEVRM keeps 1.76.
Limitations
- Real-world success rates are moderate (about 0.8 for cups and bowls, 0.6 for soft plush toys) on only 10 trials per task.
- Baselines under perturbation are reproduced by the authors, and the perturbed comparison is limited to SuSIE in the main text.
- Hyperparameter sensitivity and compute resources are not fully detailed.
Experimental setup
Datasets: CALVIN BridgeData real-world UR5 teleoperation data
Benchmarks: CALVIN ABC->D CALVIN perturbed ABC->D BridgeData goal generation
Metrics: success rate per chained instruction average chain length FID FVD LPIPS SSIM PSNR
Baselines: GR-1 SuSIE UniPi HiP RoboFlamingo AVDC HULC MCIL
Builds on: Open-Sora (STDiT3) Stable Diffusion 3 SwAV DDPM GR-1
Closest papers by meaning
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.89)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.89)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.89)
- VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Model (ICML 2026 · similarity 0.88)
- Video Language Planning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.