Research map / 3D vision / Gaussian splatting
Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Represents a dynamic scene as a set of 4D Gaussians with 4D rotations and time-evolving 4D spherindrical harmonics, trained end-to-end for real-time, high-quality dynamic novel view synthesis.
Problem
Novel view synthesis of dynamic 3D scenes from monocular or multi-view video, requiring photorealistic rendering at any time with real-time speed while sharing information across space and time.
Approach
Approximates the spatio-temporal 4D volume with anisotropic 4D Gaussians whose 4D rotations (pair of quaternions) let them capture motion, factorizes each into a conditional 3D Gaussian and a marginal temporal 1D Gaussian for rasterization, and models time-varying view-dependent color with 4D spherindrical harmonics (Fourier in time times SH in view direction).
Key results
- On the Plenoptic Video benchmark, 4DGS reaches 32.01 PSNR, 0.014 DSSIM and 0.055 LPIPS at 114 FPS, versus 31.70 PSNR for HexPlane and 31.02 for the concurrent 4DGS at 36 FPS.
- On D-NeRF monocular synthetic scenes, 4DGS achieves 34.09 PSNR, 0.98 SSIM and 0.02 LPIPS, above V4D (33.72 PSNR).
- Ablation on flame salmon and cut roasted beef: the full model averages 31.62 PSNR versus 30.79 for the No-4DRot variant, 31.38 for No-4DSH, and 30.25 for No-Time split.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 4DGS rather than HexPlane | Dynamic novel view synthesis · Plenoptic Video | PSNR: 31.70 -> 32.01 | Helped |
| use 4DGS rather than HexPlane | Dynamic novel view synthesis · Plenoptic Video | DSSIM: 0.014 -> 0.014 | No clear effect |
| use 4DGS rather than HexPlane | Dynamic novel view synthesis · Plenoptic Video | LPIPS: 0.075 -> 0.055 | Helped |
| use 4DGS rather than MixVoxels-L | Dynamic novel view synthesis · Plenoptic Video | FPS: 16.7 -> 114 | Helped |
| use 4DGS rather than V4D | Dynamic novel view synthesis · D-NeRF | PSNR: 33.72 -> 34.09 | Helped |
| use 4DGS rather than V4D | Dynamic novel view synthesis · D-NeRF | SSIM: 0.98 -> 0.98 | No clear effect |
| use 4DGS rather than V4D | Dynamic novel view synthesis · D-NeRF | LPIPS: 0.02 -> 0.02 | No clear effect |
| add 4D rotation instead of restricting space and time to be independent (ablation) | 4DGS · Flame Salmon and Cut Roasted Beef (average) | PSNR: 30.79 -> 31.62 | Helped |
| add 4D rotation instead of restricting space and time to be independent (ablation) | 4DGS · Flame Salmon and Cut Roasted Beef (average) | SSIM: 0.96 -> 0.97 | Helped |
| add 4D spherindrical harmonics (ablation) | 4DGS · Flame Salmon and Cut Roasted Beef (average) | PSNR: 31.38 -> 31.62 | Helped |
| add 4D spherindrical harmonics (ablation) | 4DGS · Flame Salmon and Cut Roasted Beef (average) | SSIM: 0.97 -> 0.97 | No clear effect |
| add temporal splitting during densification (ablation) | 4DGS · Flame Salmon and Cut Roasted Beef (average) | PSNR: 30.25 -> 31.62 | Helped |
Surprising findings
- Optimizing the rendering loss alone, without motion supervision, yields coarse scene dynamics that match optical flow from VideoFlow.
- Simple batch sampling in time was found superior to prior regularization for reducing temporal flickering and jitter.
Limitations
- Distant static background is hard to capture without initial points; spherical initialization only yields a background sphere of Gaussians.
- Initialization relies on COLMAP point clouds from the first frame or random points, so performance depends on initialization.
- Urban experiments require LiDAR points and sky/cube-map modeling, and not all settings are fully specified.
Experimental setup
Datasets: Plenoptic Video dataset D-NeRF dataset Waymo Open Dataset
Benchmarks: Plenoptic Video dataset D-NeRF dataset
Metrics: PSNR DSSIM SSIM LPIPS FPS
Baselines: DyNeRF HexPlane K-Planes MixVoxels-L NeRFPlayer HyperReel 4DGS (Yang et al. concurrent) T-NeRF D-NeRF TiNeuVox V4D
Builds on: 3D Gaussian Splatting NeRF spherical harmonics
Closest papers by meaning
- 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (CVPR 2024 · similarity 0.92)
- Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (CVPR 2024 · similarity 0.92)
- FreeTimeGS: Free Gaussian Primitives at Anytime Anywhere for Dynamic Scene Reconstruction (CVPR 2025 · similarity 0.91)
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation (AAAI 2025 · similarity 0.91)
- Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene (ICLR 2025 · similarity 0.91)
- Fully Explicit Dynamic Gaussian Splatting (NeurIPS 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.