Research map / 3D vision / 3D generation
Prior-free 3D Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BIT, a prior-free, training-free RGB 3D object tracker that alternates SoftRas-based mesh inverse rendering from SAM masks with SLOT-based pose tracking, iteratively building a reusable 3D model.
Problem
3D object tracking normally needs CAD/reconstructed models or training on pose-annotated data; the paper targets tracking with neither, from monocular RGB only.
Approach
A bidirectional loop: a geometry generation module inverse-renders a mesh from a sphere using SAM masks and tracked poses, while a pose optimization module tracks with that mesh and selects new key frames (by view-angle novelty and bounding-box validity) to feed back into geometry generation.
Key results
- On MOPED, Ours(0) without priors or reference frames averages 74.2 ADD-AUC, and Ours(3) with 3 reference frames reaches 86.2, above all compared methods (best prior-based baseline 73.7 for LatentFusion RGB-D).
- On RBOT with SLOT tracker, models generated with 12 reference frames/key frames reach 68.8 average accuracy versus 84.8 with ground-truth CAD models.
- Generated models converge quickly, with the model close to the final one by the second iteration; a model is obtained in about 30 seconds over 3 iterations on a single GTX 3080.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3 reference frames (ablation) | BIT · MOPED | ADD-AUC (0–10 cm): 74.2 -> 86.2 | Helped |
| Increase references and key frames from 3/3 to 6/12 (ablation) | BIT model generation with Song23 tracking · RBOT Avg. | Tracking accuracy: 49.2 -> 77.5 | Helped |
| Increase references from 6 to 12 while keeping 12 key frames (ablation) | BIT model generation with Song23 tracking · RBOT Avg. | Tracking accuracy: 77.5 -> 81.5 | Helped |
| Increase references and key frames from 3/3 to 6/12 (ablation) | BIT-generated model · RBOT | Normalized Hausdorff distance: 0.035 -> 0.011 | Helped |
| Increase references from 6 to 12 while keeping 12 key frames (ablation) | BIT-generated model · RBOT | Normalized Hausdorff distance: 0.011 -> 0.010 | Helped |
| Use BIT-generated models instead of GT models | Song23 · RBOT Avg. | Tracking accuracy: 92.4 -> 81.5 | Hurt |
| Use BIT-generated models instead of GT models | SLOT · RBOT Avg. | Tracking accuracy: 84.8 -> 68.8 | Hurt |
| Use BIT-generated models instead of GT models | RBOT tracker · RBOT Avg. | Tracking accuracy: 72.8 -> 60.0 | Hurt |
Surprising findings
- Without any reference frames, the method still outperforms all compared methods on MOPED average ADD-AUC (74.2), including training-based and RGB-D methods.
Limitations
- Cannot model recessed features or topologies such as ring-shaped or hollow structures.
- SAM mask errors under occlusion, motion blur, or similar colors degrade the generated model and tracking.
- Tracking failures occur with fast-moving objects and when reference frames are lacking.
Experimental setup
Datasets: MOPED RBOT
Benchmarks: MOPED RBOT
Metrics: ADD-AUC (0-10 cm) 5cm/5deg tracking accuracy normalized Hausdorff distance
Baselines: PoseRBPF LatentFusion PVNet Gen6D RBOT SRT3D RBGT SLOT Song23
Builds on: SLOT SoftRas SAM
Closest papers by meaning
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.86)
- Instance Tracking in 3D Scenes from Egocentric Videos (CVPR 2024 · similarity 0.85)
- TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.85)
- Robust 3D Tracking with Quality-Aware Shape Completion (AAAI 2024 · similarity 0.85)
- ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation (NeurIPS 2026 · similarity 0.85)
- TAPIP3D: Tracking Any Point in Persistent 3D Geometry (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.