Research map / Video / Video segmentation and tracking
SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Upgrades a pretrained image SAM (TinySAM backbone) into a promptable video segmentation model via a temporal feature integrator, memory selective associator, and memory-as-prompt generator, reaching ~90% of SAM 2.1 performance at 0.2% of its training cost.
Problem
Promptable video segmentation (PVS) foundation models like SAM 2 require enormous training compute (256 A100 GPUs for 108 hours), limiting research and deployment; the paper asks whether a resource-efficient route exists.
Approach
Instead of training a video segmenter from scratch, upgrade a pretrained image SAM by adding a temporal feature integrator to its image encoder, a relevance-based memory selector over past frames, and learnable memory prompt tokens fed to the mask decoder.
Key results
- SAM-I2V (TinySAM backbone, 18.9M params) reaches overall online PVS score 65.3 vs SAM 2.1's 71.9 (about 91%), at 4.6k GPU-hours vs 2.2M.
- Offline PVS overall score 66.0 vs SAM 2.1's 73.3 (about 90%).
- Applied to EdgeSAM, the TFI-based upgrade raises online 3-click OA from 58.5 (TA) to 65.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Upgrade TinySAM with SAM-I2V instead of using SAM 2.1 | PVS models · PVS-online, average | J&F: 71.9 -> 65.3 | Hurt |
| Upgrade TinySAM with SAM-I2V instead of using SAM 2.1 | PVS models · PVS-offline, average | J&F: 73.3 -> 66.0 | Hurt |
| Add TFI to the TinySAM temporal-associator baseline (ablation) | TinySAM + Temporal Associator · PVS-online, 3-click average | J&F: 61.7 -> 66.1 | Helped |
| Add TFI instead of ST-Adapter (ablation) | TinySAM + temporal adaptation · PVS-online, 3-click average | J&F: 64.2 -> 66.1 | Helped |
| Add TFI instead of Parallel Adapter (ablation) | TinySAM + temporal adaptation · PVS-online, 3-click average | J&F: 63.9 -> 66.1 | Helped |
| Select 3 global and 3 local frames using MSA instead of 6 global frames (ablation) | SAM-I2V · PVS-online, 3-click average | J&F: 67.1 -> 69.3 | Helped |
| Select 3 global and 3 local frames instead of 1 global and 5 local frames (ablation) | SAM-I2V with MSA · PVS-online, 3-click average | J&F: 67.7 -> 69.3 | Helped |
| Select 3 global and 3 local frames instead of 2 global and 4 local frames (ablation) | SAM-I2V with MSA · PVS-online, 3-click average | J&F: 67.8 -> 69.3 | Helped |
| Replace relevance-based frame selection with uniform frame sampling (ablation) | SAM-I2V with MSA · PVS-online, 3-click average | J&F: 69.3 -> 67.4 | Hurt |
| Add MPG (ablation) | SAM-I2V without MPG · PVS-online, 3-click average | J&F: 66.3 -> 69.3 | Helped |
| Use 3 memory prompt tokens instead of 1 (ablation) | SAM-I2V with MPG · PVS-online, 3-click average | J&F: 67.8 -> 69.3 | Helped |
| Use 3 memory prompt tokens instead of 5 (ablation) | SAM-I2V with MPG · PVS-online, 3-click average | J&F: 68.5 -> 69.3 | Helped |
Surprising findings
- Selecting memory frames by similarity to the current frame (3 global + 3 local) outperforms uniform sampling of frames and the consecutive-frame baseline.
Limitations
- Accuracy remains below SAM 2.1 (about 90-91% of its overall score).
- SAM 2.1 was trained also on 62.9k proprietary videos that the method could not use.
- Evaluated only with point-prompt J&F settings on four benchmarks.
Experimental setup
Datasets: SA-1B SAV
Benchmarks: ESD PUMaVOS LV-VIS SAV-Test
Metrics: J&F
Baselines: SAM 2.1 SAM + XMem++ SAM + Cutie TinySAM + XMem++ TinySAM + Cutie TinySAM + TA TinySAM + SA + TA EdgeSAM + TA
Builds on: Segment Anything Model (SAM) SAM 2 TinySAM XMem++ Cutie
Closest papers by meaning
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.93)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.88)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.88)
- SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation (CVPR 2025 · similarity 0.88)
- RMP-SAM: Towards Real-Time Multi-Purpose Segment Anything (ICLR 2025 · similarity 0.88)
- Endow SAM with Keen Eyes: Temporal-spatial Prompt Learning for Video Camouflaged Object Detection (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.