Research map / Detection and segmentation / Tracking
Matching Anything by Segmenting Anything
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns a universal instance-association embedding (MASA) from unlabeled SA-1B images using SAM masks as dense instance proposals and strong augmentations, and plugs a frozen-backbone adapter onto SAM/Detic/Grounding-DINO to track any detected object zero-shot.
Problem
Multi-object tracking association requires in-domain labeled video data, which limits cross-domain generalization of instance embeddings; a universal tracker for any object is lacking.
Approach
Use SAM's automatic instance masks on unlabeled images, combined with two strongly augmented views (known geometric correspondences), to build dense instance-level contrastive supervision; a frozen-backbone MASA adapter with deformable/dynamic fusion and a distilled SAM-prior detection head enables tracking any detected object and speeds up SAM's everything mode.
Key results
- On TAO TETA, Ours-Detic reaches 46.3 TETA vs. 34.6 for the fully supervised TETer, and the same-detection Ours-SAM-H/Grounding-DINO variants reach ~34.5-34.9 TETA with zero-shot association.
- On BDD100K MOTS, Ours-SAM-H† achieves 49.7 mIDF1 and 54.5 AssocA, the best association among compared methods including UNINEXT-H (48.5 mIDF1, 53.2 AssocA).
- On UVO, Ours-SAM-H reaches 28.4 video Track AR100 zero-shot, above fully supervised QDTrack (20.9) and SeqFormer (21.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MASA-Grounding-DINO instead of TETer with the same detections | TAO TETA tracking · TAO TETA val | AssocA: 36.7 -> 37.6 | Helped |
| use MASA-Detic instead of OVTrack | Open-vocabulary MOT · Open-vocabulary MOT novel | AssocA: 33.2 -> 41.2 | Helped |
| use MASA-SAM-H instead of UNINEXT-H with the same detections | BDD100K MOTS · BDD100K MOTS val | AssocA: 53.2 -> 54.5 | Helped |
| use MASA-Grounding-DINO instead of ByteTrack with the same detections | BDD100K MOT · BDD100K MOT val | IDF1: 70.4 -> 71.7 | Helped |
| add MASA training to off-the-shelf SAM features (ablation) | MASA-SAM-B · BDD100K MOT | AssocA: 32.9 -> 48.5 | Helped |
| add MASA training to off-the-shelf SAM features (ablation) | MASA-SAM-B · BDD100K MOT | mIDF1: 37.3 -> 51.7 | Helped |
| add dynamic feature fusion (ablation) | MASA-SAM-B · BDD100K MOT | AssocA: 48.5 -> 50.1 | Helped |
| add object prior distillation (ablation) | MASA-SAM-B · BDD100K MOT | AssocA: 50.1 -> 51.9 | Helped |
| replace Mask2Former proposals with SAM proposals (ablation) | MASA proposal generation · BDD MOT | AssocA: 46.4 -> 50.9 | Helped |
| replace Mask2Former proposals with SAM proposals (ablation) | MASA proposal generation · TAO | AssocA: 29.8 -> 34.1 | Helped |
| increase the proposal limit from 64 to 128 (ablation) | MASA proposal generation · BDD MOT | AssocA: 47.1 -> 50.9 | Helped |
| increase the proposal limit from 128 to 256 (ablation) | MASA proposal generation · BDD MOT | AssocA: 50.9 -> 51.9 | Helped |
Surprising findings
- Off-the-shelf SAM features perform poorly for association (32.9 AssocA on BDD MOT) before MASA training, indicating segmentation features are not instance-discriminative by default.
- Removing strong augmentations drops mIDF1 on BDD MOT by 6.7%, showing strong augmentation is necessary when training only on static images.
Limitations
- Relies on detection observations from external detectors for standard MOT benchmarks; association results are with those fixed observations.
- Video association is evaluated with frame-by-frame online matching; the method's gains on mMOTA lag ByteTrack, which uses low-confidence boxes.
- Training relies on strong augmentations since only static images are used, and the approach has not been tested on video-based training signals.
Experimental setup
Datasets: SA-1B BDD100K COCO LVIS
Benchmarks: TAO TETA TAO Track mAP Open-vocabulary MOT BDD100K MOT BDD100K MOTS UVO
Metrics: TETA AssocA LocA ClsA Track mAP50 mIDF1 IDF1 mMOTA mHOTA AR100
Baselines: TETer QDTrack GTR UNINEXT-H Deva SAM-PT UniTrack ByteTrack OVTrack VFS MoCov2 DINO
Builds on: SAM QDTrack DyHead UniTrack
Closest papers by meaning
- Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation (CVPR 2021 · similarity 0.89)
- Learning to Track Instances without Video Annotations (CVPR 2021 · similarity 0.88)
- From Detection to Association: Learning Discriminative Object Embeddings for Multi-Object Tracking (CVPR 2026 · similarity 0.88)
- SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation (NeurIPS 2025 · similarity 0.86)
- EntitySAM: Segment Everything in Video (CVPR 2025 · similarity 0.86)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.