Research map / 3D vision / Gaussian splatting
SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Combines 3D Gaussian Splatting with physically-based rendering and deformation fields, adding residual normal correction, a deformable environment map, and coarse-to-fine training to reconstruct dynamic specular scenes from monocular video.
Problem
Reconstructing and rendering dynamic scenes containing specular (reflective) surfaces from monocular video, where existing 3DGS dynamic extensions represent specular surfaces and normals poorly.
Approach
Computes physically consistent normals for deformed 3D Gaussians by aligning canonical and deformed axes, scaling a normal residual by oblateness change, and uses a deformable reflection MLP over a canonical environment cubemap to model time-varying lighting, trained in static, dynamic, and specular stages.
Key results
- On NeRF-DS, achieves mean PSNR 24.17, SSIM 0.8522, LPIPS 0.1778, outperforming Deformable 3DGS (23.43 PSNR) and NeRF-DS (23.24 PSNR).
- On HyperNeRF, achieves PSNR 22.22, SSIM 0.6088, LPIPS 0.3295, on par with state-of-the-art rather than best.
- Real-time rendering above 30 FPS on an RTX 4090 when the number of Gaussians is below 178k.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SpectroMotion instead of Deformable 3DGS | Dynamic specular scene reconstruction · NeRF-DS test, mean | PSNR: 23.43 -> 24.17 | Helped |
| Use SpectroMotion instead of Deformable 3DGS | Dynamic specular scene reconstruction · NeRF-DS test, mean | LPIPS (VGG): 0.2201 -> 0.1778 | Helped |
| Use SpectroMotion instead of 4DGS | Novel view synthesis · HyperNeRF | PSNR: 24.89 -> 22.22 | Hurt |
| Add the dynamic stage after the static stage (ablation) | SpectroMotion · NeRF-DS test | PSNR: 20.26 -> 24.02 | Helped |
| Add the specular stage after static and dynamic stages (ablation) | SpectroMotion · NeRF-DS test | LPIPS: 0.1831 -> 0.1778 | Helped |
| Add normal loss (ablation) | SpectroMotion · NeRF-DS test | PSNR: 24.09 -> 24.17 | Helped |
| Add normal regularization (ablation) | SpectroMotion · NeRF-DS test | PSNR: 24.15 -> 24.17 | Helped |
| Add γk to normal regularization (ablation) | SpectroMotion · NeRF-DS test | PSNR: 23.16 -> 24.17 | Helped |
| Use a deformable environment map instead of SH (ablation) | SpectroMotion · NeRF-DS test | PSNR: 23.63 -> 24.17 | Helped |
| Use a deformable environment map instead of a static environment map (ablation) | SpectroMotion · NeRF-DS test | PSNR: 24.02 -> 24.17 | Helped |
| Use physical normal estimation instead of the shortest Gaussian axis as the normal (ablation) | SpectroMotion · NeRF-DS test | PSNR: 23.89 -> 24.17 | Helped |
| Use 3D Gaussians instead of 2D Gaussians (ablation) | SpectroMotion · NeRF-DS test | PSNR: 23.22 -> 24.17 | Helped |
Surprising findings
- On HyperNeRF (non-specular scenes), the method is not the best in PSNR (22.22 vs 24.89 for 4DGS), only on par with state of the art.
Limitations
- Some failure cases still occur despite coarse-to-fine stabilization (details deferred to supplementary).
- Rendering speed depends on the number of Gaussians; real-time only below ~178k Gaussians.
Experimental setup
Datasets: NeRF-DS HyperNeRF
Benchmarks: NeRF-DS HyperNeRF
Metrics: PSNR SSIM LPIPS
Baselines: Deformable 3DGS 4DGS GaussianShader GS-IR NeRF-DS HyperNeRF 2DGS
Builds on: 3D Gaussian Splatting Deformable 3DGS GaussianShader Cook-Torrance reflectance model split-sum approximation
Closest papers by meaning
- SpecTRe-GS: Modeling Highly Specular Surfaces with Reflected Nearby Objects by Tracing Rays in 3D Gaussian Splatting (CVPR 2025 · similarity 0.91)
- Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.90)
- IBGS: Image-Based Gaussian Splatting (NeurIPS 2025 · similarity 0.88)
- 3D Convex Splatting: Radiance Field Rendering with 3D Smooth Convexes (CVPR 2025 · similarity 0.87)
- Motion Decoupled 3D Gaussian Splatting for Dynamic Object Representation (AAAI 2025 · similarity 0.87)
- Fully Explicit Dynamic Gaussian Splatting (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.