Research map / Detection and segmentation / Open-set and few-shot detection
Learning Segmentation from Point Trajectories
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an image segmentation network without labels using a low-rank trajectory loss on long-term point tracks, combined with an optical-flow parametric loss and temporal smoothing.
Problem
Unsupervised segmentation of independently moving objects in video using motion as the only supervision, where long-term motion from point trajectories is hard to model parametrically.
Approach
Inspired by subspace clustering, a loss penalizes singular values beyond rank r of masked trajectory matrices, so that trajectories of the same object are approximately linear combinations of each other; this is combined with a flow-based quadratic motion-fit loss and trained end-to-end over a video dataset.
Key results
- On DAVIS, LRTL reaches 82.2 J, versus 80.9 for RCF (multi-stage) and 72.1 for OCLR, at 192x352 resolution.
- On MOVi-F, the trajectory loss reaches 46.07 ARI versus 15.26 for K-Means and 11.12 for SSC in per-sequence optimisation.
- On FBMS, LRTL reaches 79.6 J and on SegTrackv2 81.2 J, the best reported among the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LRTL trajectory grouping instead of trajectory clustering baselines | Motion segmentation on MOVi-F · MOVi-F | ARI: 15.26 -> 46.07 | Helped |
| Use LRTL trajectory grouping instead of trajectory clustering baselines | Motion segmentation on MOVi-F · MOVi-F | FG-ARI: 42.53 -> 65.76 | Helped |
| Train with LRTL rather than the strongest listed prior method | Unsupervised video segmentation · DAVIS | J: 80.9 -> 82.2 | Helped |
| Train with LRTL rather than the strongest listed prior method | Unsupervised video segmentation · SegTrackv2 | J: 79.9 -> 81.2 | Helped |
| Train with LRTL rather than the strongest listed prior method | Unsupervised video segmentation · FBMS | J: 78.4 -> 79.6 | Helped |
| Add trajectory loss to flow loss (ablation) | LRTL · DAVIS | J: 78.5 -> 81.7 | Helped |
| Add temporal smoothing loss to flow and trajectory losses (ablation) | LRTL · DAVIS | J: 81.7 -> 82.2 | Helped |
| Use trajectory loss alone instead of flow loss alone (ablation) | LRTL · DAVIS | J: 78.5 -> 71.9 | Hurt |
| Replace singular-value trajectory loss with reconstruction, perspective, or tracks-as-flow losses (ablation) | LRTL · DAVIS | J: 71.9 -> 11.1; 18.2; 14.6; 65.3 | Hurt |
| Set rank r below 5 (ablation) | LRTL · DAVIS | J: 82.2 -> 76.0; 79.6 | Hurt |
| Set rank r above 5 (ablation) | LRTL · DAVIS | J: 82.2 -> 80.9 | Hurt |
| Vary predicted component count from k=4 (ablation) | LRTL · DAVIS | J: 82.2 -> 78.0; 82.0; 72.8 | Mixed |
Surprising findings
- Minimising singular values beyond rank r works better than the truncated-SVD reconstruction loss; full-reconstruction variants fail to train (e.g., 11.1 J for rec@3).
- Trajectory loss alone (71.9 J on DAVIS) underperforms flow alone (78.5 J), but combining them gives 82.2 J.
Limitations
- Occluded points have less accurate tracker predictions, and the method lacks explicit occlusion handling.
- Only nearby-frame trajectories are used, so the same point is tracked repeatedly; caching was not explored.
- Non-rigid motion is handled only via over-segmentation; multiple non-rigid objects are not addressed.
Experimental setup
Datasets: MOVi-F (Kubric) DAVIS 2016 SegTrackv2 FBMS
Benchmarks: MOVi-F DAVIS 2016 SegTrackv2 FBMS
Metrics: ARI FG-ARI Jaccard (J)
Baselines: K-Means SSC LRR OCLR DivA Meunier et al. GWM RCF LOCATE VideoCutLER VideoSAUR
Builds on: RAFT CoTracker MaskFormer RCF
Closest papers by meaning
- Learning Motion Patterns in Videos (CVPR 2017 · similarity 0.91)
- Unsupervised Space-Time Network for Temporally-Consistent Segmentation of Multiple Motions (CVPR 2023 · similarity 0.91)
- Learned Trajectory Embedding for Subspace Clustering (CVPR 2024 · similarity 0.90)
- Segment Any Motion in Videos (CVPR 2025 · similarity 0.89)
- GMOS: Grounding Moving Object Segmentation in 3D Space and Time (NeurIPS 2026 · similarity 0.87)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.