Research map / Point clouds / Point cloud classification
Fast Encoder-Based 3D from Casual Videos via Point Track Processing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TracksTo4D, a feed-forward network that maps 2D point tracks from casual videos to per-frame 3D points and camera poses, trained unsupervised with reprojection losses on COP3D pet videos.
Problem
Reconstructing dynamic 3D structure and camera poses from casual monocular videos is ill-posed and existing methods either assume orthographic cameras or require slow per-video optimization.
Approach
A transformer that alternates attention over time and point dimensions, equivariant to time-translation and point permutation, predicts input-specific low-rank bases and per-frame coefficients plus per-point motion levels (Cauchy-weighted static loss), operating only on 2D point tracks in a single feed-forward pass.
Key results
- On the pet test set, the method reaches Abs Rel 0.12 (dynamic) with ATE 8.11 mm, runtime about 0.15 minutes versus 1.3E+02 minutes for CasualSAM.
- With bundle adjustment, ATE drops to 4.21 mm, beating all baselines on pet sequences including CasualSAM's 6.90 mm.
- Per-sequence fine-tuning (FT) reaches Abs Rel 0.09 (dynamic) and ATE 3.98 mm, matching CasualSAM's depth accuracy at about 4.86 minutes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add static reprojection loss (ablation) | TracksTo4D · Pet test set | Abs Rel (Dyn.): 0.77 -> 0.11 | Helped |
| Add movement-level gamma weighting (ablation) | TracksTo4D · Pet test set | Abs Rel (Dyn.): 0.22 -> 0.11 | Helped |
| Add sparsity loss (ablation) | TracksTo4D · Pet test set | Abs Rel (Dyn.): 0.17 -> 0.11 | Helped |
| Use the proposed attention architecture instead of Set of Sets (ablation) | TracksTo4D · Pet test set | Abs Rel (Dyn.): 0.27 -> 0.11 | Helped |
| Use the proposed attention architecture instead of DSS linear layers (ablation) | TracksTo4D · Pet test set | Abs Rel (Dyn.): 1.65 -> 0.11 | Helped |
| Use K=12 bases instead of K=30 (ablation) | TracksTo4D · Pet test set | Abs Rel (Dyn.): 0.14 -> 0.11 | Helped |
| Use K=12 bases instead of K=2 (ablation) | TracksTo4D · Pet test set | Rep. (pix.), Dyn.: 8.58 -> 3.98 | Hurt |
| Apply Bundle Adjustment after inference | TracksTo4D (C) · Pet test set | ATE (mm): 8.96 -> 4.22 | Helped |
| Apply per-sequence fine-tuning | TracksTo4D (C) · Pet test set | Abs Rel (Dyn.): 0.11 -> 0.09 | Helped |
| Apply Bundle Adjustment after inference | TracksTo4D (C) · Out-of-domain videos | ATE (mm): 32.06 -> 8.67 | Helped |
| Apply per-sequence fine-tuning | TracksTo4D (D) · Out-of-domain videos | Abs Rel (Dyn.): 0.08 -> 0.05 | Helped |
| Reduce inference tracking grid from 15 to 5 (ablation) | TracksTo4D · Pet test set | RPE Trans (mm): 3.79 -> 4.97 | Hurt |
Surprising findings
- Trained only on cats, the model generalizes comparably to dogs and to out-of-domain non-pet videos.
Limitations
- Fails on rapid motion and is bounded by the accuracy of the point tracker.
- Requires sufficient motion parallax to constrain depth; fails to produce accurate cameras without it.
- Accuracy on depth is below CasualSAM without fine-tuning.
Experimental setup
Datasets: COP3D Nvidia Dynamic Scenes Dataset
Benchmarks: COP3D pet test set (21 RGBD iPhone videos) Nvidia Dynamic Scenes Dataset
Metrics: Abs Rel delta<1.25 ATE RPE Trans RPE Rot runtime
Baselines: DROID-SLAM ParticleSfM RCVD CasualSAM MiDaS Set of Sets DSS
Builds on: Deep Sets for Symmetric Elements (DSS) Set of Sets non-rigid structure from motion low-rank basis CasualSAM BootsTAPIR
Closest papers by meaning
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.88)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.88)
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.87)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.86)
- Recovering Dynamic 3D Sketches from Videos (CVPR 2025 · similarity 0.86)
- L4D-Track: Language-to-4D Modeling Towards 6-DoF Tracking and Shape Reconstruction in 3D Point Cloud Stream (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.