Research map / Detection and segmentation / Open-set and few-shot detection
EntitySAM: Segment Everything in Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends SAM 2 with query-based entity discovery (prompt generator, entity decoder) and a DINOv2 semantic encoder to automatically segment and track all entities in video, trained class-agnostically on COCO.
Problem
SAM 2 requires explicit per-object prompts and tracks each object independently, so it cannot automatically discover and track all entities, including ones appearing in later frames. Existing unified video segmentation methods also struggle with this class-agnostic entity setting.
Approach
Keep SAM 2's frozen image encoder, memory encoder and memory attention, and add learnable object queries that generate SAM 2-compatible prompts automatically, an entity decoder with inter-entity self-attention and separate query groups, and a frozen-fused DINOv2 feature branch for better semantic awareness.
Key results
- On VIPEntitySeg with ViT-L, EntitySAM reaches 54.6 VEQ versus 50.8 for DEVA and 50.6 for SAM 2.1 with Mask2Former init.
- With ViT-S, EntitySAM reaches 44.9 AP on YouTube-VIS 2019 versus 34.5 for OV-DVIS++ (online) with ResNet-50.
- Ablation on VIPSeg shows DINOv2 dual encoder raises VEQ from 50.2 to 51.8 over SAM 2 features alone (ViT-S).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace prompted SAM 2 tracking with automatic entity discovery and tracking in EntitySAM | SAM 2 · VIPEntitySeg | VEQ: 30.5 -> 54.6 | Helped |
| Use EntitySAM instead of the strongest reported baseline | Video segmentation models · VIPSeg | VPQ: 30.4 -> 31.4 | Helped |
| Use EntitySAM instead of the strongest reported baseline | Video segmentation models · VSPW | mIoU: 34.3 -> 35.5 | Helped |
| Use EntitySAM instead of the strongest reported baseline | Video instance segmentation models · YouTube-VIS 2019 | AP: 34.5 -> 44.9 | Helped |
| Use EntitySAM instead of the strongest reported baseline | Video instance segmentation models · YouTube-VIS 2021 | AP: 31.0 -> 38.8 | Helped |
| Use EntitySAM instead of the strongest reported baseline | Video instance segmentation models · OVIS | AP: 14.8 -> 21.0 | Helped |
| Add query-based EntityDecoder tracking without memory-based tracking (ablation) | SAM 2 · VIPEntitySeg | VEQ: 26.9 -> 41.3 | Helped |
| Add memory-based tracking to query-based EntityDecoder (ablation) | EntitySAM · VIPEntitySeg | VEQ: 41.3 -> 50.2 | Helped |
| Add DINOv2 features through a dual visual encoder (ablation) | EntitySAM · VIPEntitySeg | VEQ: 50.2 -> 51.8 | Helped |
| Separate queries for mask and IoU prediction instead of shared queries (ablation) | EntitySAM · VIPEntitySeg | VEQ: 49.7 -> 51.8 | Helped |
| Set EntityDecoder depth to 2 instead of 4 (ablation) | EntitySAM · VIPEntitySeg | VEQ: 51.8 -> 51.3 | Hurt |
| Set EntityDecoder depth to 6 instead of 4 (ablation) | EntitySAM · VIPEntitySeg | VEQ: 51.8 -> 51.6 | Hurt |
Surprising findings
- Adding CLIP or SAM features alongside SAM 2 features gave no zero-shot gain, while DINOv2 did.
- Class-agnostic training on COCO without category labels outperformed open-vocabulary models trained with category annotations on VIPSeg panoptic segmentation when combined with GPT-4o post-classification.
Limitations
- Class-specific evaluation relies on external GPT-4o post-classification, adding cost and dependence on a closed model.
- Lower STQ_EN than DEVA, which uses external video object segmentation data for its association model.
- Image-only training stage alone yields limited video performance (VEQ 31.4), requiring the pseudo-video stage.
Experimental setup
Datasets: COCO panoptic (class labels removed) VIPSeg VSPW YouTube-VIS 2019 YouTube-VIS 2021 OVIS
Benchmarks: VIPEntitySeg VIPSeg VSPW YouTube-VIS 2019 YouTube-VIS 2021 OVIS
Metrics: VEQ VEQSQ VEQRQ STQ_EN VPQ STQ mIoU mVC AP AR
Baselines: SAM 2.1 (everything init) SAM 2.1 (Mask2Former init) DEVA FC-CLIP OV-DVIS++ (online) OV-DVIS++ (offline) Detic-OWTB MindVLT
Builds on: SAM 2 Mask2Former DINOv2 MinVIS
Closest papers by meaning
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.87)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.87)
- SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation (NeurIPS 2025 · similarity 0.87)
- SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation (CVPR 2025 · similarity 0.86)
- TinySAM: Pushing the Envelope for Efficient Segment Anything Model (AAAI 2025 · similarity 0.86)
- Matching Anything by Segmenting Anything (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.