Research map / Detection and segmentation / Tracking
Tracking and Understanding Object Transformations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Track Any State (tracking objects through transformations while detecting and describing state changes) with the VOST-TAS benchmark, and TubeletGraph, a zero-shot system that recovers missed tracks via spatiotemporal tubelet partitioning and builds a VLM-described state graph.
Problem
Object trackers such as SAM2 lose the target object after it undergoes state transformations (cutting, emerging, fragmenting), producing mainly false-negative misses; no method jointly tracks through transformations and describes the state changes.
Approach
Partition every video region into tubelets by entity segmentation and SAM2 propagation, then recover candidate tubelets that pass spatial proximity (overlap with SAM2 candidate masks of the prompt) and semantic consistency (CLIP mask-pooled similarity) priors, and describe each recovered transformation with a multimodal LLM to build a state graph.
Key results
- On VOST validation, TubeletGraph achieves J=50.9 and J_tr=36.7, versus SAM2.1 at 48.4 and 32.4, and above the finetuned SAM2.1 (ft) J_tr of 36.4.
- On VOST-TAS, the full system reaches temporal localization precision 43.1 and recall 20.4, with overall recall H_ST=12.0 and H=6.5.
- On VSCOS, TubeletGraph reaches J=75.9 and J_tr=72.2, the best among compared methods; on DAVIS17 it is comparable to baselines (J=85.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TubeletGraph rather than SAMURAI | object tracking · VOST val | J: 49.8 -> 50.9 | Helped |
| use TubeletGraph rather than SAMURAI | object tracking · VOST val | J_tr: 34.0 -> 36.7 | Helped |
| use TubeletGraph rather than SAM2Long | object tracking · VSCOS val | J: 73.0 -> 75.9 | Helped |
| use TubeletGraph rather than ReVOS | object tracking · M³-VOS val | J: 75.6 -> 74.1 | Hurt |
| use TubeletGraph rather than SAM2Long | object tracking · DAVIS17 val | J: 87.1 -> 85.6 | Hurt |
| add semantic consistency and spatial proximity constraints (ablation) | TubeletGraph with spatiotemporal partition only · VOST val | J: 25.7 -> 50.9 | Helped |
| add spatial proximity constraint (ablation) | TubeletGraph with spatiotemporal partition and semantic consistency · VOST val | J: 49.2 -> 50.9 | Helped |
| add semantic consistency constraint (ablation) | TubeletGraph with spatiotemporal partition and spatial proximity · VOST val | J: 50.7 -> 50.9 | Helped |
| use CropFormer rather than SAM automasks for entity detection (ablation) | TubeletGraph · VOST val | J: 49.2 -> 50.9 | Helped |
| use SAM2.1 rather than Cutie for tubelet propagation (ablation) | TubeletGraph · VOST val | J: 47.6 -> 50.9 | Helped |
| replace CLIP semantic similarity with DINOv2 (ablation) | TubeletGraph · VOST val | J: 50.9 -> 50.9 | No clear effect |
| use GPT-4.1 rather than Qwen as the VLM (ablation) | TubeletGraph · VOST-TAS | Action verb accuracy: 31.8 -> 81.8 | Helped |
Surprising findings
- Fine-tuned SAM2.1 gains are limited when combined with TubeletGraph (J 54.1 vs 50.9), since fine-tuning already targets the same false negatives.
- Replacing GPT-4.1 with Qwen-2.5VL sharply lowers semantic accuracy (action verb 31.8 vs 81.8) without changing tracking performance.
Limitations
- High computational cost (about 7 s per frame) from tracking every spatial region.
- Modular design makes systematic error attribution and diagnosis difficult.
- Low recall on temporal localization (20.4) and overall state-graph recall (H=6.5), since transformations are detected only when a missed object is recovered.
Experimental setup
Datasets: VOST VSCOS M3-VOS DAVIS 2017 Ego4D EPIC-Kitchens
Benchmarks: VOST VSCOS M3-VOS DAVIS 2017 VOST-TAS
Metrics: J J_tr precision recall temporal localization precision/recall action verb accuracy resulting object accuracy H_ST H
Baselines: SAM2.1 SAM2.1 (ft) SAM2Long SAMURAI DAM4SAM ReVOS Cutie XMem
Builds on: SAM2 CropFormer FC-CLIP VOST
Closest papers by meaning
- Learning Object State Changes in Videos: An Open-World Perspective (CVPR 2024 · similarity 0.85)
- Tracking objects that change in appearance with phase synchrony (ICLR 2025 · similarity 0.84)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.83)
- Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking (ICLR 2025 · similarity 0.83)
- (How) Do Language Models Track State? (ICML 2025 · similarity 0.83)
- Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.