Research map / Vision-language and multimodal / Robot action and navigation
SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAM2Act, a multi-view robotic transformer policy using SAM2 encoder features with multi-resolution upsampling, and SAM2Act+, which adds SAM2-style memory modules, plus a new MemoryBench benchmark for spatial memory manipulation.
Problem
Robotic manipulation policies struggle with generalization to environmental perturbations, high-precision tasks, and memory-dependent tasks requiring spatial recall.
Approach
Use SAM2 image encoder embeddings (LoRA-tuned) with cascaded multi-resolution upsampling inside the RVT-2 coarse-to-fine multi-view transformer, and add SAM2-inspired memory bank, memory encoder, and memory attention over past translation heatmaps to enable spatial memory.
Key results
- SAM2Act reaches 86.8% average success over 18 RLBench tasks, vs 81.4% for RVT-2.
- SAM2Act+ reaches 94.3% average success on MemoryBench, versus 55.0% for SAM2Act and 54.0% for RVT-2.
- SAM2Act shows an average -4.3% drop across Colosseum perturbations, versus -19.5% for RVT-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SAM2Act instead of RVT-2 | 3D keyframe-based behavior cloning · RLBench 18 tasks | average success rate: 81.4 $\pm$ 3.1 -> 86.8 $\pm$ 0.5 | Helped |
| use SAM2Act instead of RVT-2 | 3D manipulation policy · The Colosseum, 13 perturbation factors | average percentage change: -19.5$\pm$2.8 -> -4.3$\pm$3.6 | Helped |
| add explicit memory components | SAM2Act · MemoryBench | average success rate: 55.0 $\pm$ 24.3 -> 94.3 $\pm$ 9.0 | Helped |
| replace the SAM2 image encoder with the SAM image encoder (ablation) | SAM2Act · RLBench 18 tasks | average success rate: 86.8 $\pm$ 0.5 -> 80.8 $\pm$ 1.9 | Hurt |
| replace the SAM2 image encoder with Depth Anything V2 (ablation) | SAM2Act · RLBench 18 tasks | average success rate: 86.8 $\pm$ 0.5 -> 81.1 $\pm$ 1.2 | Hurt |
| replace the SAM2 image encoder with DINOv2 (ablation) | SAM2Act · RLBench 18 tasks | average success rate: 86.8 $\pm$ 0.5 -> 82.2 $\pm$ 0.5 | Hurt |
| replace multi-resolution upsampling with original convex upsampling (ablation) | SAM2Act · RLBench 18 tasks | average success rate: 86.8 $\pm$ 0.5 -> 84.2 $\pm$ 0.9 | Hurt |
| remove SAM2 multi-resolution image embedding inputs (ablation) | SAM2Act · RLBench 18 tasks | average success rate: 86.8 $\pm$ 0.5 -> 85.7 $\pm$ 0.3 | Hurt |
| replace the SAM2 image encoder with the SAM image encoder (ablation) | SAM2Act · The Colosseum, 13 perturbation factors | average percentage change: -4.3$\pm$3.6 -> -20.7$\pm$1.2 | Hurt |
| remove SAM2 multi-resolution image embedding inputs (ablation) | SAM2Act · The Colosseum, 13 perturbation factors | average percentage change: -4.3$\pm$3.6 -> -19.1$\pm$4.5 | Hurt |
Surprising findings
- SAM2Act with SAM1 encoder or DINOv2 or Depth Anything V2 backbones underperforms SAM2 by up to several points, showing the encoder choice matters.
Limitations
- SAM2Act+ uses a fixed memory window length that does not adapt across tasks.
- Extension to dexterous continuous control remains a challenge.
- Memory mechanism appears to store spatial but not semantic information such as color.
Experimental setup
Datasets: RLBench (18 tasks) The Colosseum MemoryBench
Benchmarks: RLBench The Colosseum MemoryBench real-world Franka tasks
Metrics: success rate average rank percentage change under perturbation
Baselines: RVT-2 RVT PerAct SAM-E Act3D 3D Diffuser Actor 3D-LOTUS ARP+
Builds on: RVT-2 SAM2 PerAct LoRA
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.90)
- SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation (ICML 2024 · similarity 0.90)
- PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulation (NeurIPS 2024 · similarity 0.89)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.89)
- RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation (NeurIPS 2026 · similarity 0.89)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.