Research map / 3D vision / 3D generation
TAPVid-360: Tracking Any Point in 360 from Narrow Field of View Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TAPVid-360, a task of tracking 3D directions to query points across narrow-FoV video even when they leave the view, with a dataset (TAPVid360-10k) built from 360 videos and a CoTracker3-based baseline (CoTracker360) that predicts per-point rotations.
Problem
Existing tracking-any-point methods operate in 2D image space or 2.5D and cannot persistently track points that leave the camera's field of view, and 3D ground-truth for dynamic scenes is hard to obtain.
Approach
Use 360 videos as supervision: track objects on the full equirectangular panorama with a 2D segmentation and TAP pipeline, convert tracks to unit direction vectors, then resample narrow-FoV perspective videos with random camera motions so that tracked points leave the view. The baseline fine-tunes CoTracker3 to predict a per-point rotation applied to the initial direction.
Key results
- CoTracker360 achieves out-of-frame delta_avg of 0.1160, about 1.3x the next-best baseline TAPIP3D (0.0850).
- CoTracker360 reduces out-of-frame angular distance to 10.98 versus 45.80 for TAPIP3D, a more than 4-fold reduction.
- CoTracker360 achieves all-points angular distance of 8.27 versus 36.44 for TAPIP3D, while in-frame accuracy of CoTracker3 (offline) is higher (0.5588 vs 0.4060).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt CoTracker3 to predict per-point direction rotations and fine-tune on TAPVid-360 data | CoTracker3 · TAPVid360-10k out-of-frame | <δ_avg oof: 0.0850 -> 0.1160 | Helped |
| Adapt CoTracker3 to predict per-point direction rotations and fine-tune on TAPVid-360 data | CoTracker3 · TAPVid360-10k out-of-frame | AD_avg oof: 45.7951 -> 10.9829 | Helped |
Surprising findings
- CoTracker3 and SpatialTracker have high in-frame precision but very large angular errors on wrong predictions, so precision alone misleads.
- Spin Y (roll-inducing) is the hardest camera motion for CoTracker360, while Spiral is easiest.
Limitations
- Dataset uses a fixed field of view, so sensitivity to zoom or varying FoV is not tested.
- The baseline uses fixed positional encoding per patch and does not model uncertainty over directions.
- Training is limited to 32 query points across 32 frames due to memory constraints.
Experimental setup
Datasets: 360-1M TAPVid360-10k
Benchmarks: TAPVid360-10k
Metrics: delta_avg angular threshold accuracy mean angular distance
Baselines: TAPNext TAPIR BootsTAPIR TAPIP3D SpatialTracker CoTracker3 (offline)
Builds on: CoTracker3 SAM2 Lang-SAM 360-1M
Closest papers by meaning
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.90)
- Seurat: From Moving Points to Depth (CVPR 2025 · similarity 0.89)
- TAPIP3D: Tracking Any Point in Persistent 3D Geometry (NeurIPS 2025 · similarity 0.89)
- Leader360V: A Large-scale, Real-world 360 Video Dataset for Multi-task Learning in Diverse Environment (NeurIPS 2025 · similarity 0.87)
- From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos (NeurIPS 2024 · similarity 0.87)
- PanoWorld: Geometry-Consistent Panoramic Video World Modeling (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.