Research map / Video / Video segmentation and tracking
EdgeTAM: On-Device Track Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Makes SAM 2 efficient enough for on-device video segmentation by replacing dense memory attention with a 2D Spatial Perceiver that compresses memories while preserving spatial structure, plus feature distillation from SAM 2, reaching 16 FPS on iPhone 15 Pro Max.
Problem
SAM 2 is too slow for on-device promptable video segmentation and tracking; prior efficiency work only compresses the image encoder, while the dense cross-attention over memory features is the latency bottleneck.
Approach
A plug-in 2D Spatial Perceiver compresses each frame-level memory map into a fixed set of learnable latents split into global (unstructured, attend globally) and 2D patch-local (window-partitioned) groups, keeping spatial layout while cutting memory-attention cost ~T times; a two-stage feature distillation from SAM 2 (image encoder and memory-attention output) adds accuracy at no inference cost.
Key results
- EdgeTAM reaches 87.7 DAVIS 2017, 70.0 MOSE, 72.3 SA-V val and 71.7 SA-V test J&F while running at 16 FPS on iPhone 15 Pro Max.
- 2D Spatial Perceiver with RepViT-M1 and two memory attention blocks speeds up the baseline from 2.5 FPS to 15.7 FPS on iPhone and improves SA-V val from 63.5 to 64.4 (ablation).
- Feature distillation from SAM 2 adds +1.3 J&F on SA-V val and +3.3 on SA-V test without inference overhead.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the 2D Spatial Perceiver memory compressor (ablation) | EdgeTAM with RepViT-M1 and 2 memory attention blocks · SA-V val | J&F: 63.5 -> 64.4 | Helped |
| Replace the memory compressor with 4×4 average pooling (ablation) | EdgeTAM with RepViT-M1 and 2 memory attention blocks · SA-V val | J&F: 63.5 -> 61.8 | Hurt |
| Add distillation to the 2D Perceiver model (ablation) | EdgeTAM with RepViT-M1 and 2 memory attention blocks · SA-V val | J&F: 64.4 -> 65.7 | Helped |
| Add distillation to the 2D Perceiver model (ablation) | EdgeTAM with RepViT-M1 and 2 memory attention blocks · SA-V test | J&F: 62.5 -> 65.8 | Helped |
| Use 256 global latents without 2D latents (ablation) | EdgeTAM memory compressor · SA-V val | J&F: 63.5 -> 62.0 | Hurt |
| Use 256 global latents without 2D latents (ablation) | EdgeTAM memory compressor · SA-V test | J&F: 62.1 -> 60.6 | Hurt |
| Use 256 2D latents without global latents (ablation) | EdgeTAM memory compressor · SA-V val | J&F: 63.5 -> 63.1 | No clear effect |
| Use 256 2D latents without global latents (ablation) | EdgeTAM memory compressor · SA-V test | J&F: 62.1 -> 62.4 | No clear effect |
| Use both 256 global latents and 256 2D latents (ablation) | EdgeTAM memory compressor · SA-V val | J&F: 63.5 -> 64.4 | Helped |
| Use both 256 global latents and 256 2D latents (ablation) | EdgeTAM memory compressor · SA-V test | J&F: 62.1 -> 62.5 | Helped |
| Use self-attention in the 2D Perceiver (ablation) | EdgeTAM memory compressor · SA-V val | J&F: 62.6 -> 64.4 | Helped |
| Use self-attention in the 2D Perceiver (ablation) | EdgeTAM memory compressor · SA-V test | J&F: 62.7 -> 62.5 | Hurt |
Surprising findings
- Naive average pooling of memories degrades accuracy substantially versus the uncompressed baseline (SA-V val 63.5 to 61.8), and a plain Perceiver also hurts, motivating the spatially structured design.
- Replacing the image encoder with compact backbones alone barely reduces latency because memory attention is the bottleneck.
Limitations
- Tracking granularity follows SAM 2; fails in cases like fast-moving objects where previously invisible parts (bird feet) are not recovered.
- On high-end GPUs, CUDA kernel launch overhead becomes the bottleneck, so the efficiency gains are mainly for edge devices.
- Slightly lower than SAM 2 on some settings (e.g., SA-23 mIoU 55.5 vs 58.9 SAM 2 at 1 click).
Experimental setup
Datasets: SA-1B SA-V DAVIS MOSE YTVOS SA-23 DAVIS 2017 YouTubeVOS 2019
Benchmarks: DAVIS 2017 MOSE SA-V val SA-V test YouTubeVOS 2019 SA-23 PVS 9-dataset zero-shot suite
Metrics: J&F G mIoU FPS
Baselines: SAM 2 SAM 2.1 SAM STCN XMem Cutie DEVA SAM 2-B+ RDE SimVOS-B
Builds on: SAM 2 Perceiver EdgeSAM RepViT Hiera
Closest papers by meaning
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.89)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.87)
- A Distractor-Aware Memory for Visual Object Tracking with SAM2 (CVPR 2025 · similarity 0.86)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.85)
- SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model (AAAI 2024 · similarity 0.85)
- BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.