Research map / Vision-language and multimodal / Robot action and navigation
Any2Policy: Learning Visuomotor Policy with Any-Modality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Any2Policy, a visuomotor policy that accepts text, audio, image, video and point-cloud instructions and observations via frozen ImageBind encoders, learnable projections and cross-attention "embodied alignment", and releases a 30-task real-world multimodal robot dataset (RoboAny).
Problem
Current robot learning policies typically accept only one modality for task specification and one for observation, limiting their ability to use rich multi-modal instructions and perception.
Approach
Use a frozen unified multi-modal encoder (ImageBind) with trainable per-modality projections, TokenLearner-based token pruning, and a cross-attention "embodied alignment" module that fuses instruction tokens into observation tokens before a transformer policy trained with behavior cloning MSE.
Key results
- Any2Policy reaches 51% success with text-image vs 39% for a modality-specific model on real-world text-to-image, and 62% for text-video-type combinations with extra modalities (Table 4 reports up to 69% for text+image end-goal with image+point cloud).
- Removing embodied alignment drops real-world success substantially, e.g., text-to-image from 51% to 26%.
- On Franka Kitchen with 10 and 25 demonstrations, Any2Policy outperforms R3M, BLIP-2 and EmbodiedGPT on all tasks except Knobs-left with 25 demos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train across multiple modalities instead of modality-specific training | Any2Policy · RoboAny real-world test, instruction-observation pairs in Table 1 | success rate | Mixed |
| Remove embodied alignment and concatenate projected modality outputs, aligning instruction and observation with an MLP (ablation) | Any2Policy · RoboAny real-world test, instruction-observation pairs in Table 2 | success rate: 51 -> 26; 57 -> 18; 62 -> 31; 49 -> 27; 55 -> 20; 57 -> 16; 56 -> 15; 63 -> 29; 38 -> 28; 46 -> 15; 57 -> 9; 36 -> 11 | Hurt |
| Add audio instruction to text instruction (ablation) | Any2Policy · RoboAny real-world test: instruction Text + Audio, observation Image | success rate: 51 -> 52 | Helped |
| Add image end-goal instruction alongside text and audio (ablation) | Any2Policy · RoboAny real-world test: instruction Text + Audio + Image End-Goal, observation Image | success rate: 51 -> 60 | Helped |
| Add video demonstration instruction to text, audio, and image end-goal instructions (ablation) | Any2Policy · RoboAny real-world test: instruction Text + Audio + Image End-Goal + Video Demonstration, observation Image | success rate: 51 -> 62 | Helped |
| Add video observation to image observation (ablation) | Any2Policy · RoboAny real-world test: instruction Text, observation Image + Video | success rate: 51 -> 57 | Helped |
| Add point cloud observation alongside image and video (ablation) | Any2Policy · RoboAny real-world test: instruction Text, observation Image + Video + Point Cloud | success rate: 51 -> 66 | Helped |
| Add video demonstration instruction and video observation (ablation) | Any2Policy · RoboAny real-world test: instruction Text + Video Demonstration, observation Image + Video | success rate: 51 -> 63 | Helped |
| Add image end-goal instruction and point cloud observation (ablation) | Any2Policy · RoboAny real-world test: instruction Text + Image End-Goal, observation Image + Point Cloud | success rate: 51 -> 69 | Helped |
Surprising findings
- Image-point cloud and video-point cloud pairings score lower than other pairs, indicating point cloud fusion with 2D inputs is still hard.
Limitations
- Point cloud combined with image or video performs worse than other modality pairs.
- Real-world evaluation uses 10 trials per setting and a single workstation setup with 70 objects.
- Simulation evaluation covers only a few modalities (text-image, text-point cloud).
Experimental setup
Datasets: RoboAny (self-collected real-world) Franka Kitchen ManiSkill2
Benchmarks: RoboAny real-world Franka Kitchen ManiSkill2
Metrics: success rate
Baselines: VIMA R3M T5 BLIP-2 EmbodiedGPT modality-specific Any2Policy variants
Builds on: ImageBind RT-1 VIMA TokenLearner
Closest papers by meaning
- GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.88)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.88)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.88)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.87)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.87)
- Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.