Research map / Video / Video segmentation and tracking
4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4DGT, a feed-forward transformer that predicts pixel-aligned 4D Gaussians from posed monocular videos, trained only on real-world datasets with density control and multi-level attention for long space-time inputs.
Problem
Feed-forward dynamic scene reconstruction from posed monocular videos is hard because real-world monocular videos lack space-time supervision, and optimization-based methods are slow while prior feed-forward Gaussian models rely on synthetic object-level data.
Approach
A transformer predicts 4D Gaussians (with temporal center, lifespan, velocity, angular velocity) unifying static and dynamic content, trained on real monocular videos with depth/normal expert supervision and motion regularization; a two-stage histogram-based pruning and densification scheme plus multi-level spatiotemporal attention enable denser space-time sampling at fixed compute.
Key results
- On the averaged rendering benchmark, 4DGT reaches 23.20 PSNR versus 19.24 for MonST3R and 8.49 for L4GM, at 25 ms/frame reconstruction time.
- Motion segmentation mIoU of 81.2 on ADT versus 77.4 for MegaSaM, which is about 200x faster (1.5 s vs 300 s).
- After 10 s finetuning (Ours_tune10s), PSNR on ADT reaches 31.98, exceeding SoM's 30.30 which requires about 60 s per frame of optimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on EgoExo4D rather than only the small AEA training set (ablation) | 4DGT stage one · ADT | PSNR: 15.49 -> 22.72 | Helped |
| Add depth, normal, lifespan, angular velocity and velocity losses (ablation) | 4DGT stage one · ADT | PSNR: 22.72 -> 26.45 | Helped |
| Use the dynamic 4D Gaussian method instead of a static LRM (ablation) | 4DGT stage one · ADT | PSNR: 19.29 -> 26.45 | Helped |
| Use the dynamic 4D Gaussian method instead of per-frame prediction (ablation) | 4DGT stage one · ADT | PSNR: 10.07 -> 26.45 | Helped |
| Add velocity, angular velocity and lifespan regularization (ablation) | 4DGT · ADT motion segmentation | mIoU × 100: $9.4^{\pm4.1}$ -> $\mathbf{81.2^{\pm1.8}}$ | Helped |
| Use histogram-based pruning and densification rather than random Gaussian sampling (ablation) | 4DGT stage two · ADT | PSNR: 25.79 -> 28.79 | Helped |
| Add multi-level attention to density control (ablation) | 4DGT stage two · ADT | PSNR: 28.79 -> 28.71 | No clear effect |
| Mix training datasets instead of training only on EgoExo4D (ablation) | 4DGT stage two · DyCheck | PSNR: 15.34 -> 16.12 | Helped |
| Use a shared pruning pattern rather than an on-the-fly pattern (ablation) | 4DGT · ADT | PSNR: $28.36$ -> $\mathbf{28.79}$ | Helped |
| Fine-tune 4DGT for 10 seconds (ablation) | 4DGT · ADT | PSNR: 28.31$^{\pm1.51}$ -> 31.98$^{\pm1.01}$ | Helped |
| Use 4DGT rather than L4GM | dynamic reconstruction · Average across ADT, TUM, DyCheck and Aria | PSNR: 8.490 -> 23.20 | Helped |
| Use 4DGT rather than SoM | dynamic reconstruction · Average across ADT, TUM, DyCheck and Aria | reconstruction time: 60000 ms / f -> 25 ms / f | Helped |
Surprising findings
- Without the velocity/angular-velocity/lifespan regularizer the model collapses to a purely white motion mask, making all Gaussians transient and dynamic.
- Naively scaling space-time resolution runs out of memory, while the proposed pruning reduces Gaussians by 80% while raising the space-time sampling rate 16x.
Limitations
- Quality degrades on videos from unseen device types due to inaccurate metric-scale calibration; requires reliable camera calibration and poses.
- Visible artifacts when viewing Gaussians from extreme view angles far from the input trajectory.
- Blurriness and artifacts during fast motion; no flow or tracking expert guidance for temporal consistency due to cost and instability.
Experimental setup
Datasets: EgoExo4D Nymeria Hot3D Aria Everyday Activities Epic-Fields Cop3D ARKitTrack ADT DyCheck TUM-SLAM
Benchmarks: ADT DyCheck TUM-SLAM Aria test set
Metrics: PSNR LPIPS depth RMSE normal angle error reconstruction time mIoU
Baselines: L4GM Static-LRM Expert monocular models MonST3R Shape of Motion
Builds on: 4D Gaussian Splatting 2D Gaussian Splatting Large Reconstruction Model DINOv2 3D Gaussian Splatting
Closest papers by meaning
- 4D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Time (NeurIPS 2025 · similarity 0.90)
- Instant4D: 4D Gaussian Splatting in Minutes (NeurIPS 2025 · similarity 0.89)
- 4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos (NeurIPS 2025 · similarity 0.89)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.89)
- 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization (NeurIPS 2024 · similarity 0.89)
- Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.