Research map / Video / Video segmentation and tracking
4D-Fly: Fast 4D Reconstruction from a Single Monocular Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4D-Fly, a streaming framework that reconstructs dynamic Gaussian scenes from a monocular video in minutes by propagating Gaussians with anchor-based 2D track/depth priors, expanding the map via a canonical Gaussian map, and running a short per-frame optimization.
Problem
Reconstructing 4D scenes from a single monocular video is ill-posed, and existing prior-regularized Gaussian methods require hours of global optimization per video, limiting practical use.
Approach
Instead of aligning splatted feature maps through slow global optimization, it directly lifts 2D tracks and depth priors onto explicit 3D Gaussians via anchor-based propagation, adds new Gaussians from a canonical Gaussian map for unobserved regions, and performs a fast streaming per-frame optimization.
Key results
- On DyCheck iPhone, 4D-Fly reaches 17.03 mPSNR in 5.3 minutes per 100 frames versus 109.4 minutes for Shape of Motion (16.42 mPSNR), a ~20x speed-up.
- On NVIDIA Dynamic Scenes, 4D-Fly achieves 22.52 mPSNR and 0.14 mLPIPS, outperforming Dynamic Gaussian Marbles (22.36) and Shape of Motion (22.07).
- For point tracking on DyCheck iPhone, 4D-Fly matches Shape of Motion on 3D EPE (0.08) and gets the best OA (85.4) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add segmentation masks to distinguish foreground from background (ablation) | 4D-Fly · DyCheck iPhone | mSSIM: 0.42 -> 0.60 | Helped |
| add anchor-based Gaussian propagation (ablation) | 4D-Fly · DyCheck iPhone | mSSIM: 0.45 -> 0.60 | Helped |
| add the Canonical Gaussian Map (ablation) | 4D-Fly · DyCheck iPhone | mSSIM: 0.52 -> 0.60 | Helped |
| add fast 4D scene optimization (ablation) | 4D-Fly · DyCheck iPhone | mSSIM: 0.43 -> 0.60 | Helped |
| add rigidity regularization (ablation) | 4D-Fly · DyCheck iPhone | mSSIM: 0.54 -> 0.60 | Helped |
| include velocity distance in rigidity regularization (ablation) | 4D-Fly · DyCheck iPhone | mSSIM: 0.57 -> 0.60 | Helped |
| add color regularization (ablation) | 4D-Fly · DyCheck iPhone | mSSIM: 0.50 -> 0.60 | Helped |
| use 4D-Fly instead of Shape of Motion | novel view synthesis · NVIDIA Dynamic Scenes | mPSNR: 22.07 -> 22.52 | Helped |
Surprising findings
- Ablation shows removing the fast optimization or anchor-based propagation hurts most, while dropping the velocity distance from the rigidity loss still leaves a sizable gap in mPSNR.
Limitations
- Relies on priors from vision foundation models (depth, tracks, segmentation) that may be inaccurate.
- Struggles with complex illumination variation and non-rigid deformation modeling.
- Lower 2D tracking AJ and 3D delta accuracy than some baselines on DyCheck (e.g., 40.1 vs 43.2 delta 5cm for Shape of Motion).
Experimental setup
Datasets: DyCheck iPhone NVIDIA Dynamic Scenes DAVIS
Benchmarks: DyCheck iPhone NVIDIA Dynamic Scenes
Metrics: mPSNR mSSIM mLPIPS 3D EPE 3D delta accuracy AJ delta avg OA runtime FPS
Baselines: T-NeRF HyperNeRF 4D Gaussians Dynamic Gaussian Marbles Shape of Motion CoTracker TAPIR
Builds on: 3D Gaussian Splatting Dynamic Gaussian Marbles Shape of Motion DepthCrafter TAPIR
Closest papers by meaning
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.84)
- 4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos (NeurIPS 2025 · similarity 0.84)
- GFlow: Recovering 4D World from Monocular Video (AAAI 2025 · similarity 0.84)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.82)
- UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.82)
- 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.