Research map / Vision-language and multimodal / Robot action and navigation
VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
VLMimic uses VLMs to learn fine-grained, object-centric robot skills from about 5 human videos via hierarchical constraint representations and an iterative comparison skill adapter, without additional training.
Problem
Visual imitation learning with VLMs typically only learns high-level plans and relies on pre-defined motion primitives; learning fine-grained low-level actions directly from limited human videos and generalizing to unseen scenes remains a bottleneck.
Approach
Grounds object-centric interactions from human videos (hand/object segmentation, point clouds, FoundationPose), represents skills as hierarchical semantic (visualized, annotated interactions) plus geometric (keypoint-based, bounded-region and trajectory-code) constraints for VLM reasoning, and adapts them to new scenes via iterative visual and keypoint comparison with failure reasoning.
Key results
- On RLBench with 5 demos, overall success 0.76 vs 0.49 for Demo2Code and 0.44 for CaP.
- Real-world seen-environment success 0.76 and unseen 0.71, vs 0.55 (DP) and 0.43 (Demo2Code) in SE.
- Long-horizon tasks overall success 0.50 vs 0.13 for the best baseline (R3M-DP).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical semantic and geometric constraints rather than geometric constraints alone (ablation) | VLMimic · Real-world manipulation, SE | success rate: 0.61 -> 0.76 | Helped |
| Use hierarchical constraints rather than semantic constraints alone (ablation) | VLMimic · Real-world manipulation, SE | success rate: 0.68 -> 0.76 | Helped |
| Group grasp poses with VLMs instead of direct value prediction or DBSCAN (ablation) | VLMimic · Real-world manipulation, SE | success rate: 0.52 -> 0.76; 0.59 -> 0.76 | Helped |
| Increase the number of human videos from 1 to 3 to 5 (ablation) | VLMimic · Real-world manipulation, SE | success rate: 0.68; 0.72; 0.76 | Helped |
| Vary the human-video count beyond 5 (ablation) | VLMimic · Real-world manipulation, SE | success rate: 7: 0.67; 9: 0.78; 11: 0.78 | Mixed |
| Use both visualized interactions and keypoints for comparison instead of either alone (ablation) | VLMimic skill adapter · Real-world manipulation, UE | success rate: 0.64 -> 0.77; 0.63 -> 0.77 | Helped |
| Increase skill-adaptation iterations from 0 to 4 (ablation) | VLMimic skill adapter · Real-world manipulation, UE | success rate: 0.61 -> 0.77 | Helped |
| Increase skill-adaptation iterations beyond 4 (ablation) | VLMimic skill adapter · Real-world manipulation, UE | success rate: 4: 0.77; 5: 0.77 | No clear effect |
| Add failure reasoning and increase its iterations from 0 to 2 (ablation) | VLMimic · Real-world manipulation, UE | success rate: 0.71 -> 0.77 | Helped |
| Increase failure-reasoning iterations beyond 2 (ablation) | VLMimic · Real-world manipulation, UE | success rate: 2: 0.77; 3: 0.78; 4: 0.77; 5: 0.77 | No clear effect |
Surprising findings
- Demo2Code and CaP with 5 demos outperform diffusion policies trained on 100 robot demonstrations in most tasks.
- Single human video already yields 0.68 SE success, suggesting strong sample efficiency.
Limitations
- Limited by VLM inference latency and computational resource requirements.
- Baselines struggle on long-horizon tasks in unseen environments, so long-horizon evaluation is only in seen settings.
Experimental setup
Datasets: RLBench robot videos (simulation) human demonstration videos (real-world)
Benchmarks: RLBench real-world manipulation tasks real-world long-horizon tasks
Metrics: success rate
Baselines: R3M-DP Diffusion Policy GraphIRL Code as Policies Demo2Code
Builds on: Demo2Code Code as Policies GraphIRL Diffusion Policy R3M
Closest papers by meaning
- Unified Reinforcement and Imitation Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.92)
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.