Research map / Video / Video segmentation and tracking
SAM 2: Segment Anything in Images and Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SAM 2, a unified promptable visual segmentation model for images and videos with streaming memory, trained on the SA-V dataset collected via a three-phase model-in-the-loop data engine.
Problem
Extending promptable segmentation from static images to videos, where objects change appearance, get occluded, and require efficient real-time processing, and where existing video segmentation models and datasets lack coverage of arbitrary objects and parts.
Approach
A streaming transformer that conditions each frame's embedding on memory of past predictions and prompted frames (memory attention over a FIFO memory bank plus object pointers), allowing clicks/boxes/masks on any frame to interactively refine masklets; a model-in-the-loop data engine iteratively improves both model and data.
Key results
- SAM 2 achieves 58.9 1-click mIoU on SA-23 vs SAM's 58.1 while running 6x faster (130.1 vs 21.7 FPS).
- Data engine Phase 3 with SAM 2 in the loop is 8.4x faster than per-frame SAM annotation (4.5 s vs 37.8 s per frame).
- SAM 2 (Hiera-L) reaches 77.9 J&F on MOSE val and 78.4 J&F on SA-V test, outperforming prior VOS methods such as Cutie-base+ and DEVA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hiera-L instead of Hiera-B+ (ablation) | SAM 2 · MOSE val | J&F: 76.6 -> 77.9 | Helped |
| Train with the data engine phases and Auto data instead of only VOS + SA-1B (ablation) | SAM 2 · SA-V val | J&F: 50.0 -> 63.2 | Helped |
| Add internal, SA-V, and SA-1B data to VOS training data (ablation) | SAM 2 · 9 zero-shot video datasets | J&F: 59.7 -> 71.8 | Helped |
| Add SA-1B images to training (ablation) | SAM 2 trained with SA-V data · SA-23 | 1-click mIoU: 53.0 -> 58.6 | Helped |
| Select the 50k most-edited SA-V masklets rather than 50k random masklets (ablation) | SAM 2 · SA-V val | J&F: 63.7 -> 66.2 | Helped |
| Use all 190k SA-V masklets instead of 50k most-edited masklets (ablation) | SAM 2 · SA-V val | J&F: 66.2 -> 69.9 | Helped |
| Increase input resolution from 512 to 768 (ablation) | SAM 2 · SA-V val | J&F: 68.3 -> 71.1 | Helped |
| Reduce training sequence length from 8 frames to 4 frames (ablation) | SAM 2 · SA-V val | J&F: 68.3 -> 60.0 | Hurt |
| Increase image encoder from Hiera-B+ to Hiera-L (ablation) | SAM 2 · SA-23 | mIoU: 59.7 -> 61.1 | Helped |
| Add 2d-RoPE in memory attention (ablation) | SAM 2 · 9 zero-shot video datasets | J&F: 70.3 -> 70.7 | Helped |
| Add recurrent GRU processing to memory features (ablation) | SAM 2 · 9 zero-shot video datasets | J&F: 70.9 -> 70.5 | No clear effect |
| Cross-attend to object pointers from other frames (ablation) | SAM 2 · 9 zero-shot video datasets | J&F: 70.9 -> 70.7 | No clear effect |
Surprising findings
- Training with GRU-based recurrent memory did not improve results over direct memory bank storage, except slightly on LVOSv2.
- Training only on the VOS datasets (DAVIS, MOSE, YouTubeVOS) performs poorly on most zero-shot benchmarks, even though it works well in-domain on MOSE.
Limitations
- Can fail across shot changes, in crowded scenes, after long occlusions, and with thin or fast-moving fine structures.
- Processes each object separately without inter-object communication.
- Data engine still relies on human annotators for verification and frame selection.
Experimental setup
Datasets: SA-V SA-1B DAVIS MOSE YouTube-VOS Internal video data
Benchmarks: SA-V val SA-V test MOSE DAVIS 2017 LVOS YouTube-VOS 2019 9 zero-shot video datasets 17 video datasets SA-23 37 zero-shot image datasets
Metrics: J&F G mIoU FPS annotation time per frame
Baselines: SAM+XMem++ SAM+Cutie SAM STCN XMem DEVA Cutie-base+ SwinB-DeAOT
Builds on: Segment Anything (SAM) Hiera MAE XMem Cutie
Closest papers by meaning
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.93)
- SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation (CVPR 2025 · similarity 0.90)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.89)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.89)
- EdgeTAM: On-Device Track Anything Model (CVPR 2025 · similarity 0.89)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.