Research map / Detection and segmentation / Tracking
A Distractor-Aware Memory for Visual Object Tracking with SAM2
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DAM4SAM, a training-free distractor-aware memory for SAM2.1 that splits memory into recent-appearance and distractor-resolving parts with an introspection-based update, and releases the DiDi distractor-distilled tracking dataset.
Problem
Memory-based trackers built on video segmentation models (e.g., SAM2) still fail in the presence of distractors, since their memory stores entire frames without regard to tracking function and updates indiscriminately.
Approach
Split the SAM2 memory into a recent-appearance memory (FIFO, updated every 5 frames, skipped when target absent) and a distractor-resolving memory (initial frame plus FIFO anchors, not time-stamped), where anchor frames are chosen by divergence between SAM2's predicted and alternative masks during reliable tracking periods.
Key results
- DAM4SAM reaches 0.694 tracking quality on DiDi vs 0.649 for SAM2.1 and 0.680 for SAMURAI.
- On VOT2022, DAM4SAM achieves 0.753 EAO, outperforming the challenge winner MS AOT (0.673) and SAM2.1 (0.692).
- On LaSOText, DAM4SAM reaches 60.9 AUC vs 56.9 for SAM2.1 and 56.6 for LORAT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use distractor-aware memory and its update protocol | SAM2.1 · DiDi | Quality: 0.649 -> 0.694 | Helped |
| Use distractor-aware memory and its update protocol | SAM2.1 · VOT2020 | EAO: 0.681 -> 0.729 | Helped |
| Use distractor-aware memory and its update protocol | SAM2.1 · VOT2022 | EAO: 0.692 -> 0.753 | Helped |
| Use distractor-aware memory and its update protocol | SAM2.1 · VOTS2024 | Quality: 0.661 -> 0.711 | Helped |
| Use distractor-aware memory and its update protocol | SAM2.1 · LaSoT | AUC: 70.0 -> 75.1 | Helped |
| Use distractor-aware memory and its update protocol | SAM2.1 · LaSoText | AUC: 56.9 -> 60.9 | Helped |
| Use distractor-aware memory and its update protocol | SAM2.1 · GoT10k | AO: 80.7 -> 81.1 | Helped |
| Update memory only when the predicted mask is non-empty (ablation) | SAM2.1 · DiDi | Quality: 0.649 -> 0.665 | Helped |
| Update memory every 5th frame (ablation) | SAM2.1PRES · DiDi | Quality: 0.665 -> 0.667 | Helped |
| Add DRM and update it only during reliable tracking periods (ablation) | SAM2.1Δ=5 · DiDi | Robustness: 0.914 -> 0.932 | Helped |
| Update DRM only when a distractor is detected, without requiring reliable tracking (ablation) | SAM2.1Δ=5 · DiDi | Accuracy: 0.718 -> 0.691 | Hurt |
| Add distractor-aware memory with both proposed DRM update rules (ablation) | SAM2.1Δ=5 · DiDi | Quality: 0.667 -> 0.694 | Helped |
Surprising findings
- Updating memory with frames where the target is absent degrades SAM2 performance, and simply reducing update frequency to every 5th frame improves robustness.
- Updating the distractor memory on every detected distractor (without reliability gating) lowers accuracy, so gating on tracking reliability is necessary.
Limitations
- Gains depend on heuristic thresholds for anchor detection and reliability gating, although authors say insensitive to them.
- Moderate speed reduction (about 20%) relative to SAM2.1.
- DiDi is built from a subset of existing benchmarks using a DINOv2-based heuristic distractor criterion.
Experimental setup
Datasets: DiDi GOT-10k LaSOT LaSOText UAV123 VOT2020 VOT2022 VOTS2024
Benchmarks: DiDi VOT2020 VOT2022 VOTS2024 LaSOT LaSOText GOT-10k
Metrics: tracking quality Q accuracy robustness EAO AUC average overlap
Baselines: SAM2.1 SAM2.1Long SAMURAI ODTrack Cutie AOT KeepTrack SeqTrack LORAT MS AOT S3-Track
Builds on: SAM2 Cutie AOT SAMURAI
Closest papers by meaning
- When Trackers Fail: VLM-Guided Verification and Recovery for Robust Video Object Segmentation (NeurIPS 2026 · similarity 0.88)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.88)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.87)
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.87)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.86)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.