Research map / Video / Video segmentation and tracking
Point-VOS: Pointing Up Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Point-VOS, a video object segmentation task trained and tested with sparse spatio-temporal point annotations, plus a semi-automatic annotation pipeline that yields the PV-Oops and PV-Kinetics datasets (19M points, 133K objects, 32K videos).
Problem
Dense per-object mask annotations for training and testing video object segmentation are costly and time-consuming, limiting dataset scale.
Approach
Replace dense masks with spatio-temporally sparse point annotations for both training and test-time initialization, generate points semi-automatically from VidLN mouse traces via DynaMITe and STCN, and have annotators verify candidate points, then adapt STCN to point supervision or train it on pseudo-masks.
Key results
- Point-STCN trained on PV-Oops + PV-Kinetics reaches 71.9-72.9 mean J&F on PV-DAVIS with 10-point initialization after fine-tuning, versus 85.3 J&F for original STCN with masks.
- STCN trained on pseudo-masks from points reaches 76.9 J&F on DAVIS, more than 90% of the mask-supervised VOS quality (85.3).
- Adding PV-Kinetics VNG data to RF-VNG raises UVO-VNG from 46.4 to 52.8 J&F and OVIS-VNG from 32.7 to 35.0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PV-Oops and PV-Kinetics point data before fine-tuning | Point-STCN · PV-DAVIS | J &F: 61.3% to 63.1% | Helped |
| Add PV-Oops and PV-Kinetics point data before fine-tuning | Point-STCN · PV-YT | G: 51.9% to 57.2% | Helped |
| Train on PV-Oops point annotations instead of only PV-DAVIS and PV-YT | Point-STCN · PV-Oops | J &F: 48.6% to 61.2% | Helped |
| Add PV-Oops and PV-Kinetics pseudo-mask data to training | STCN with pseudo-masks · PV-DAVIS | J &F: 65.6% to 70.4% | Helped |
| Add PV-Oops and PV-Kinetics pseudo-mask data to training | STCN with pseudo-masks · PV-YT | G: 63.0% to 68.3% | Helped |
| Train with temporally sparse points and initialize with points instead of using dense masks (ablation) | STCN · DAVIS validation | J &F: 85.3 -> 67.7 | Hurt |
| Use pseudo-masks instead of points for training and test-time initialization (ablation) | STCN, Point-VOS · DAVIS validation | J &F: 67.7 -> 76.9 | Helped |
| Use point-based initialization instead of mask initialization while retaining mask training (ablation) | STCN, Hybrid · DAVIS validation | J &F: 85.3 -> 78.3 | Hurt |
| Use SAM pseudo-masks instead of points for hybrid test-time initialization (ablation) | STCN, Hybrid · DAVIS validation | J &F: 78.3 -> 80.4 | Helped |
| Pre-train RF-VNG on PV-Kinetics | RF-VNG · OVIS-VNG | J &F: 32.7% to 35.0% | Helped |
| Pre-train RF-VNG on PV-Kinetics | RF-VNG · UVO-VNG | J &F: 46.4% to 52.8% | Helped |
Surprising findings
- Training on PV-Kinetics alone improves results without fine-tuning only when more than 5 test-time points are available, which the authors attribute to domain mismatch.
- SAM pseudo-masks give worse results than DynaMITe for 1 or 2 initialization points, likely because SAM produces part-based segmentations for few points.
Limitations
- Test-time evaluation on PV-Oops uses dense masks for only 3 frames per object, a reduced evaluation protocol.
- Pseudo-mask baseline needs an additional segmentation model (DynaMITe or SAM) at test time, increasing run-time.
- Annotations are derived from VidLN mouse traces, so the pipeline depends on that existing corpus.
Experimental setup
Datasets: DAVIS YouTube-VOS PV-Oops PV-Kinetics COCO-PNG UVO-VNG OVIS-VNG
Benchmarks: DAVIS 2017 validation (PV-DAVIS) YouTube-VOS (PV-YT) PV-Oops OVIS-VNG UVO-VNG
Metrics: J&F G J F
Baselines: STCN Point-STCN RF-VNG PET SAM pseudo-masks
Builds on: STCN DynaMITe Video Localized Narratives RF-VNG
Closest papers by meaning
- Learning Fast and Robust Target Models for Video Object Segmentation (CVPR 2020 · similarity 0.91)
- Two-Shot Video Object Segmentation (CVPR 2023 · similarity 0.90)
- Breaking the "Object" in Video Object Segmentation (CVPR 2023 · similarity 0.90)
- RMem: Restricted Memory Banks Improve Video Object Segmentation (CVPR 2024 · similarity 0.88)
- Multi-Granularity Video Object Segmentation (AAAI 2025 · similarity 0.88)
- M^3-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.