Research map / Video / Video generation

TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking

NeurIPS 2026 · Video › Video generation · approach: Efficient video diffusion models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Repurposes a pretrained video diffusion transformer (Wan2.1-T2V) with LoRA into a single-pass feed-forward dense 3D tracker that predicts reference-anchored tracking pointmaps and visibility from a monocular video and its frame-anchored pointmaps.

Problem

Dense 3D tracking from monocular video needs reference-anchored predictions (following physical points of the first frame over time), while video diffusion transformers produce frame-anchored outputs; existing 3D trackers are iterative and trained from scratch on synthetic data or rely on static mul

Approach

A dual-latent representation (per-frame geometry latents plus first-frame-anchored track latents used as dense queries) combined with temporal RoPE alignment (assigning each track latent the target timestamp's RoPE index) lets full 3D attention in a video DiT regress tracking pointmaps in one forward pass.</put_idea>

Key results

• TrackCraft3R + DA3 reaches average AJ 0.7931 / APD3D 0.9250-range OA (average AJ 0.6785 on Kubric-inclusive table; reported average AJ 0.7931 and OA 0.9250 across sparse/dense benchmarks), outperforming DELTAv2 + DA3.
• TrackCraft3R runs 1.3x faster and uses 4.6x less peak memory than DELTAv2 for 12-frame clips (3.91s vs 5.00s; 7.63GB vs 35.46GB on A6000).
• Random initialization instead of pre-trained Wan weights drops average AJ from 0.5639 to 0.4698 (ViPE input).

Surprising findings

• Although trained on only 4 synthetic 3D/4D datasets, TrackCraft3R + V-DPM matches V-DPM's APD3D while exceeding it in AJ, despite V-DPM using 23 datasets.
• The model is trained on 12-frame clips yet generalizes to videos up to 120 frames with a sliding window and to large temporal strides.

Limitations

• Accuracy is bounded by the quality of external per-frame depth and camera pose inputs.
• Slightly lower APD3D than V-DPM when using V-DPM geometry.
• Relies on training with synthetic 3D/4D datasets only.

Experimental setup

Datasets: Kubric PointOdyssey DynamicReplica TartanAir ADT PStudio ITTO DAVIS

Benchmarks: TAPVid-3D ADT TAPVid-3D PStudio Dynamic Replica PointOdyssey Kubric

Metrics: AJ APD_3D OA

Baselines: DELTA DELTAv2 St4RTrack Any4D TraceAnything MotionCrafter AllTracker SpatialTrackerV2 TAPIP3D V-DPM

Builds on: Wan 2.1 DELTAv2 CoTracker 3D RoPE LoRA

Closest papers by meaning

• TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.90)
• Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.90)
• Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.89)
• DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO (ICLR 2025 · similarity 0.89)
• Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.89)
• Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.89)