Research map / Vision-language and multimodal / Robot action and navigation
Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AGNOSTOS, an RLBench-based benchmark of 23 unseen tasks for zero-shot cross-task VLA generalization, and X-ICM, a dynamics-guided in-context LLM method that outperforms evaluated VLA models.
Problem
Existing VLA models are mainly evaluated on within-task generalization; their zero-shot generalization to entirely unseen tasks in simulation is underexplored.
Approach
Builds a benchmark of 23 unseen RLBench tasks split into two difficulty levels, and proposes X-ICM, which prompts an LLM with retrieved seen-task demonstrations (textualized object positions and key-actions) selected by similarity of diffusion-based dynamics features.
Key results
- X-ICM (72B) achieves 30.1% average success on 23 unseen tasks, vs 17.5% for pi0, the best prior VLA model.
- X-ICM (7B) reaches 23.5% overall and 28.6% on Level-1, surpassing pi0 by 6.9 points on Level-1 tasks.
- Dynamics-guided selection raises generalization; the ablation with predicted target-observation features reaches 30.1% vs 25.2% without any dynamic features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use X-ICM (7B) instead of the strongest baseline, π₀ | AGNOSTOS · AGNOSTOS Level-1 | success rate: 21.7 -> 28.6 | Helped |
| Use X-ICM (72B) instead of the strongest baseline, SAM2Act | AGNOSTOS · AGNOSTOS Level-2 | success rate: 15.9 (1.3) -> 20.3 (1.7) | Helped |
| Use X-ICM (72B) instead of the strongest baseline, π₀ | AGNOSTOS · AGNOSTOS all unseen tasks | success rate: 17.5 (0.4) -> 30.1 (1.0) | Helped |
| Add dynamics-guided demonstration selection instead of random sampling (ablation) | X-ICM (72B) · AGNOSTOS Level-1 and Level-2 | success rate | Helped |
| Increase in-context demonstrations from 1 to 12 (ablation) | X-ICM with Qwen2.5-72B-Instruct · AGNOSTOS | success rate | Helped |
| Increase the demonstration count beyond 12 (ablation) | X-ICM with Qwen2.5-72B-Instruct · AGNOSTOS | success rate | No clear effect |
| Scale the backbone from Qwen2.5-7B to Qwen2.5-14B (ablation) | X-ICM · AGNOSTOS all unseen tasks | success rate: 23.5 (1.6) -> 29.7 (0.7) | Helped |
| Scale the backbone from Qwen2.5-14B to Qwen2.5-72B (ablation) | X-ICM · AGNOSTOS all unseen tasks | success rate: 29.7 (0.7) -> 30.1 (1.0) | Helped |
| Select demonstrations using predicted-target visual features (ablation) | X-ICM dynamics-guided sample selection · AGNOSTOS all unseen tasks | success rate: 25.2 (3.2) -> 29.7 (2.2) | Helped |
| Add the initial-observation visual feature to the predicted-target visual feature (ablation) | X-ICM dynamics-guided sample selection · AGNOSTOS all unseen tasks | success rate: 29.7 (2.2) -> 28.6 (1.3) | Hurt |
| Add the language feature to the predicted-target visual feature (ablation) | X-ICM dynamics-guided sample selection · AGNOSTOS all unseen tasks | success rate: 29.7 (2.2) -> 30.1 (1.0) | Helped |
| Add the language feature to the initial-observation visual feature (ablation) | X-ICM dynamics-guided sample selection · AGNOSTOS all unseen tasks | success rate: 27.5 (1.0) -> 27.7 (0.7) | No clear effect |
Surprising findings
- Learned VLA models trained on seen tasks achieve near-zero success on many unseen tasks; all prior models completely fail on at least eight of 23 tasks.
- Using textual language features alone for sample selection did not help, while predicted target-observation features were most effective.
Limitations
- X-ICM performance remains limited on tasks with both novel objects and motion primitives.
- Visual information is used only via textualized object positions, ignoring raw visual context.
- Simulation uses ground-truth object positions for fair comparison with VoxPoser; long-horizon tasks yield only 5% success in real-world tests.
Experimental setup
Datasets: RLBench 18 seen tasks (3600 demonstrations) Open X-Embodiment
Benchmarks: AGNOSTOS
Metrics: success rate
Baselines: PerAct RVT RVT2 Sigma-Agent Instant Policy D4R R3M D4R-Align R3M-Align OpenVLA RDT pi0
Builds on: in-context learning VoxPoser InstructPix2Pix GroundingDINO KAT RoboPrompt
Closest papers by meaning
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.91)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.91)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.91)
- NS-VLA: Towards Neuro-Symbolic Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.