Research map / 3D vision / Depth and stereo
Tracking Everything Everywhere across Multiple Cameras
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends pixel tracking to multi-camera settings using a canonical 3D space with per-view latent codes, spatial and distillation losses, and an incremental learning strategy.
Problem
Dense pixel tracking across multiple synchronized cameras and timesteps, handling occlusions and appearance changes between views, without camera poses, and adapting incrementally to new frames without catastrophic forgetting.
Approach
Builds on OmniMotion's canonical 3D volume with a bijective mapping conditioned on learnable per-view latent codes and time, adds a DINO-weighted spatial rigidity loss, and performs incremental tracking via sliding-window fine-tuning with partial layer freezing and a distillation regularizer on the canonical mapping.
Key results
- On the Plenoptic benchmark, the full method reaches AJ 76.5 and δx_avg 87.8 versus DINO's δx_avg 35.8.
- On the Immersive benchmark, the method reaches AJ 62.6 and δx_avg 78.5 versus DINO's δx_avg 37.1.
- Incremental tracking retains most of the offline performance (Plenoptic AJ 68.6 vs 76.5 offline).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed multiview tracker instead of DINO | Multiview pixel tracking · Plenoptic | < δx_avg: 35.8 -> 87.8 | Helped |
| Use the proposed multiview tracker instead of DINO | Multiview pixel tracking · Immersive | MTE: 50.8 -> 14.2 | Helped |
| Use incremental tracking rather than offline tracking | Ours · Plenoptic | AJ: 76.5 -> 68.6 | Hurt |
| Use incremental tracking rather than offline tracking | Ours · Immersive | MTE: 14.2 -> 26.8 | Hurt |
| Add distillation loss (ablation) | Incremental multiview tracking · Plenoptic | MTE: 20.6 -> 15.4 | Helped |
| Add distillation loss (ablation) | Incremental multiview tracking · Immersive | MTE: 37.1 -> 29.0 | Helped |
| Freeze the last two color-MLP layers (ablation) | Incremental multiview tracking · Plenoptic | AJ: 66.3 -> 66.4 | No clear effect |
| Freeze the last two color-MLP layers (ablation) | Incremental multiview tracking · Immersive | OA: 78.4 -> 85.3 | Helped |
| Add spatial loss (ablation) | Multiview tracking · Plenoptic | < δx_avg: 84.3 -> 87.0 | Helped |
| Add spatial loss (ablation) | Multiview tracking · Immersive | AJ: 64.3 -> 69.5 | Hurt |
| Use an optimizable per-view latent instead of a predetermined view index (ablation) | Multiview tracking · Plenoptic | AJ: 76.6 -> 78.7 | No clear effect |
| Use an optimizable per-view latent instead of a predetermined view index (ablation) | Multiview tracking · Immersive | AJ: 68.4 -> 69.5 | Helped |
Limitations
- Ground truth is manually labeled only for the first frame correspondences; subsequent labels come from a single-view tracker, so evaluation relies on a pretrained model.
- Requires per-scene test-time optimization over 200,000 iterations.
- Only DINO is compared as a baseline because existing multi-view tracking methods target object-level tracking.
Experimental setup
Datasets: Plenoptic Video Dataset Immersive Video Dataset
Benchmarks: Plenoptic Video Dataset Immersive Video Dataset
Metrics: AJ <δx_avg MTE OA
Baselines: DINO Ours w/o distillation loss Ours w/o parameter freezing Ours w/o spatial loss Ours w/o per-view latent
Builds on: OmniMotion NeRF PATS DINO
Closest papers by meaning
- SpatialTracker: Tracking Any 2D Pixels in 3D Space (CVPR 2024 · similarity 0.87)
- TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.87)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.86)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.86)
- DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO (ICLR 2025 · similarity 0.86)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.