Research map / 3D vision / Gaussian splatting
GS-DiT: Advancing Video Generation with Dynamic 3D Gaussian Fields through Efficient Dense 3D Point Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds dynamic 3D Gaussian fields from monocular videos via a new dense 3D point tracker (D3D-PT) and fine-tunes CogVideoX to generate videos conditioned on the rendered Gaussian-field video, enabling multi-camera shooting, dolly zoom, and object editing.
Problem
Video diffusion transformers lack precise 4D control (multi-camera shooting, dolly zoom, object motion editing), and existing multi-view training data is expensive while per-video 4D Gaussian optimization is slow and artifact-prone.
Approach
Construct dynamic 3D Gaussian fields directly from dense 3D point trajectories (no per-video optimization) using a loosely coupled depth-refinement tracker, render guidance videos under chosen camera parameters, and use them as conditioning for fine-tuning a pretrained video DiT to restore realistic video.
Key results
- D3D-PT achieves 3D-AJ of 9.0 average on TAPVid-3D minival vs 8.3 for SpatialTracker (Ours + ZoeDepth), and 11.4 with Depth Anything V2.
- Inference time per video drops from 89.80 s (SpatialTracker) to 0.98 s, roughly two orders of magnitude faster.
- On DAVIS, GS-DiT reaches PSNR 19.18 / LPIPS 0.23 vs MonST3R 14.12 / 0.31 and GCD 15.04 / 0.48 for multi-shooting generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D3D-PT with Depth Anything V2 instead of SpatialTracker | 3D point tracking · TAPVid-3D average | 3D-AJ: 8.3 -> 11.4 | Helped |
| Use D3D-PT with Depth Anything V2 instead of SpatialTracker | 3D point tracking · TAPVid-3D average | APD: 14.3 -> 18.3 | Helped |
| Use D3D-PT with Depth Anything V2 instead of SpatialTracker | 3D point tracking · TAPVid-3D average | OA: 83.4 -> 83.9 | Helped |
| Use D3D-PT instead of SpatialTracker | 3D point tracking inference · DAVIS | inference time per frame: 89.80 seconds -> 0.98 seconds | Helped |
| Use GS-DiT instead of MonST3R | multi-camera shooting video generation · DAVIS | PSNR: 14.12 -> 19.18 | Helped |
| Use GS-DiT instead of MonST3R | multi-camera shooting video generation · DAVIS | SSIM: 0.59 -> 0.60 | Helped |
| Use GS-DiT instead of MonST3R | multi-camera shooting video generation · DAVIS | LPIPS: 0.31 -> 0.23 | Helped |
| Use GS-DiT instead of MonST3R | multi-camera shooting video generation · Sora | PSNR: 15.32 -> 17.92 | Helped |
| Use GS-DiT instead of MonST3R | multi-camera shooting video generation · Sora | LPIPS: 0.30 -> 0.20 | Helped |
| Use GS-DiT instead of MonST3R | multi-camera shooting video generation · Pixabay | PSNR: 19.78 -> 22.66 | Helped |
Surprising findings
- GCD nearly collapses on real-world videos due to poor generalization from its simulator-trained data.
Limitations
- The dynamic 3D representation is not optimized, so rendered guidance videos contain artifacts.
- Gaussian scale and opacity are held constant and the field is built by point tracking rather than an explicit 4D Gaussian optimization.
Experimental setup
Datasets: Kubric WebVid-10M TAPVid TAPVid-3D DAVIS Sora Pixabay
Benchmarks: TAPVid TAPVid-3D DAVIS Sora Pixabay
Metrics: AJ delta_avg OA 3D-AJ APD PSNR SSIM LPIPS inference time
Baselines: GCD MonST3R SpatialTracker BootsTAPIR DOT CoTracker TAPIR PIPs TAP-Net
Builds on: CogVideoX DOT CoTracker RAFT 3D Gaussian Splatting
Closest papers by meaning
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.88)
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models (CVPR 2024 · similarity 0.88)
- Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle (CVPR 2024 · similarity 0.88)
- 4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos (NeurIPS 2025 · similarity 0.87)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.87)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.