Research map / Video / Video segmentation and tracking
Segment Any Motion in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a trajectory-based moving object segmentation model that classifies long-range point tracks as dynamic using motion encoding plus DINOv2 semantics, then densifies sparse track prompts into masks with SAM2 via iterative prompting.
Problem
Moving object segmentation in videos, where optical-flow cues are short-range and fragile under occlusion, blur, and camera motion, and SAM2 cannot natively determine which objects are moving.
Approach
Combine long-range point tracks (BootsTAP) with a spatio-temporal trajectory attention encoder and a Motion-Semantic Decoupled Embedding that injects DINOv2 features only in the decoder, to label dynamic tracks; then use those tracks as point prompts for SAM2 in a two-stage iterative grouping and mask densification process.
Key results
- On DAVIS17-Moving fine-grained MOS, the method reaches J&F 80.5 versus 51.1 for the best baseline ABR.
- On DAVIS2016-Moving MOS, the method reaches J&F 89.5 versus 78.5 for OCLR-TTA and 79.6 for RCF-All.
- Ablation on DAVIS17-Moving shows removing tracks drops J&F to 19.6 and removing the decoupled embedding (w/o MSDE) drops it to 63.0 from 80.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use long-range trajectory tracks with DINO features and SAM2 prompting | Ours · DAVIS16-Moving | J &F: 79.6 -> 89.5 | Helped |
| Use long-range trajectory tracks with DINO features and SAM2 prompting | Ours · SegTrackv2 | F: 79.6 -> 82.8 | Helped |
| Use long-range trajectory tracks with DINO features and SAM2 prompting | Ours · FBMS-59 | F: 79.6 -> 82.8 | Helped |
| Use long-range trajectory tracks with DINO features and SAM2 prompting | Ours · DAVIS2016 | J &F: 80.7 -> 90.9 | Helped |
| Use the proposed method for fine-grained per-object segmentation | Ours · DAVIS17-Moving fine-grained MOS | J &F: 51.1 -> 80.5 | Helped |
| Add depth input (ablation) | Ours · DAVIS17-Moving | J &F: 69.2 -> 80.5 | Helped |
| Add depth input (ablation) | Ours · DAVIS16-Moving | J &F: 82.5 -> 89.1 | Helped |
| Add trajectory tracks (ablation) | Ours · DAVIS17-Moving | J &F: 19.6 -> 80.5 | Helped |
| Add trajectory tracks (ablation) | Ours · DAVIS16-Moving | J &F: 20.9 -> 89.1 | Helped |
| Add DINO features (ablation) | Ours · DAVIS17-Moving | J &F: 65.0 -> 80.5 | Helped |
| Add DINO features (ablation) | Ours · DAVIS16-Moving | J &F: 75.5 -> 89.1 | Helped |
| Use Motion-Semantic Decoupled Embedding instead of concatenating DINO before motion encoding (ablation) | Ours · DAVIS17-Moving | J &F: 72.0 -> 80.5 | Helped |
Surprising findings
- Simply feeding DINO features into the motion encoder makes the model over-rely on semantics, assuming same-category objects share motion state; decoupling them in the decoder fixes this.
Limitations
- Performance depends heavily on the accuracy of the off-the-shelf long-range tracker.
- Relies on off-the-shelf depth and tracking estimators rather than end-to-end training.
- Training data limited to synthetic and egocentric datasets (Kubric, Dynamic Replica, HOI4D).
Experimental setup
Datasets: Kubric Dynamic Replica HOI4D
Benchmarks: DAVIS16-Moving DAVIS17-Moving DAVIS2016 SegTrackv2 FBMS-59
Metrics: J (region similarity) F (contour similarity) J&F
Baselines: CIS EM RCF OCLR ABR
Builds on: SAM2 ParticleSfM BootsTAP DINOv2 Depth-Anything
Closest papers by meaning
- Learning Segmentation from Point Trajectories (NeurIPS 2024 · similarity 0.89)
- Video Segmentation via Object Flow (CVPR 2016 · similarity 0.89)
- GMOS: Grounding Moving Object Segmentation in 3D Space and Time (NeurIPS 2026 · similarity 0.88)
- Know Where You Are From: Event-Based Segmentation via Spatio-Temporal Propagation (AAAI 2025 · similarity 0.88)
- Learning Spatial-Semantic Features for Robust Video Object Segmentation (ICLR 2025 · similarity 0.87)
- Decoupled Motion Expression Video Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.