Research map / Vision-language and multimodal / Robot action and navigation
SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM-E, a multi-view 3D manipulation policy that LoRA-fine-tunes a SAM ViT-B encoder and predicts a 5-step action sequence in a single pass via multi-channel heatmaps.
Problem
Multi-task 3D robot manipulation from language instructions requires scene understanding and long-horizon action prediction, but existing multi-view policies need many high-quality trajectories, generalize poorly to unseen tasks, and execute inefficiently step by step.
Approach
Use the promptable SAM image encoder (LoRA-fine-tuned on embodied data) as the visual foundation for a multi-view transformer, and add an action-sequence policy head that outputs multi-channel pose heatmaps for a short horizon of future keyframe actions, exploiting temporal smoothness of end-effector actions.
Key results
- SAM-E reaches 70.6% average success over 18 RLBench tasks versus 62.9% for RVT, using about 1130 total inference steps versus RVT's 6158.
- In few-shot adaptation on 6 new tasks, SAM-E (adaptation) achieves 63.2% average success versus 37.1% for RVT (adaptation).
- SAM-E processes a 5-action sequence in 152 ms, while RVT needs 5×103 ms for 5 actions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAM-E with SAM and action-sequence prediction instead of RVT | RLBench multi-task policy · RLBench 18-task multi-task | success rate: 62.9 -> 70.6 | Helped |
| predict action sequences rather than next keypoint poses step by step | SAM-E · RLBench 18-task multi-task | inference steps (sum): 6158±64 -> 1130±12 | Helped |
| initialize the policy from multi-task training and adapt to new tasks | SAM-E · RLBench few-shot adaptation, 6 tasks | success rate: 37.1±1.0 -> 63.2±1.5 | Helped |
| use SAM-E rather than Hiveformer | 10-task comparison · Hiveformer 10-task comparison | success rate: 83.3 -> 88.6±0.7 | Helped |
| replace the SAM encoder with RVT's encoder (ablation) | SAM-E · RLBench 18-task multi-task | success rate: 70.6±0.7 -> 65.3±0.6 | Hurt |
| add LoRA to the key projection (ablation) | SAM-E · RLBench 18-task multi-task | success rate: 70.6±0.7 -> 69.2±0.9 | Hurt |
| remove LoRA fine-tuning (ablation) | SAM-E · RLBench 18-task multi-task | success rate: 70.6±0.7 -> 67.2±1.0 | Hurt |
| fully fine-tune the SAM encoder instead of using LoRA (ablation) | SAM-E · RLBench 18-task multi-task | success rate: 70.6±0.7 -> 65.8±1.0 | Hurt |
| set the action horizon to 1 rather than 5 (ablation) | SAM-E · RLBench 18-task multi-task | success rate: 70.6±0.7 -> 30.6±1.4 | Hurt |
| set the action horizon to 3 rather than 5 (ablation) | SAM-E · RLBench 18-task multi-task | success rate: 70.6±0.7 -> 64.0±0.6 | Hurt |
| set the action horizon to 7 rather than 5 (ablation) | SAM-E · RLBench 18-task multi-task | success rate: 70.6±0.7 -> 66.5±1.2 | Hurt |
| use an action horizon of 7 instead of 5 (ablation) | SAM-E · RLBench 18-task multi-task | inference steps (sum): 1130±12 -> 919±12 | Helped |
Surprising findings
- Full fine-tuning of the SAM encoder underperforms LoRA (65.8% vs 70.6%), which the authors attribute to limited demonstrations.
- Replacing the SAM encoder with R3M, CLIP, or DINO yields lower or similar success than SAM-E, and the RVT-encoder variant still beats RVT, indicating the action-sequence head itself contributes.
Limitations
- Relies on limited robot data; scalability to larger datasets such as Open-X is untested.
- Fixed action horizon may be unsuitable for some tasks (e.g., stack cups).
- Inference time per step is slightly higher than RVT, and some tasks (e.g., place cups, stack cups) show near-zero success.
Experimental setup
Datasets: RLBench demonstrations (RVT/PerAct training set) real-world Franka demonstrations
Benchmarks: RLBench (18 tasks, 249 variations) RLBench few-shot adaptation (6 new tasks) real-world Franka tasks
Metrics: success rate inference steps inference time (ms)
Baselines: RVT PerAct R3M CLIP DINO Hiveformer SAM→RVT
Builds on: Segment Anything Model RVT PerAct LoRA CLIP
Closest papers by meaning
- SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation (ICML 2025 · similarity 0.90)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.89)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.89)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.88)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.88)
- OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.