Research map / 3D vision / 3D generation
Geometry-guided Online 3D Video Synthesis with Multi-View Temporal Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Geometry-guided online multi-view video synthesis that fuses temporally filtered depth into an image-space TSDF to guide a U-Net blending network over forward-rendered 3D Gaussian input views, improving view and temporal consistency at ~100 ms per frame.
Problem
Novel-view synthesis from sparse multi-camera video suffers from view-inconsistent and temporally flickering results, while dense-camera methods are too expensive for online use.
Approach
Aggregate temporally filtered input depths (using color-difference masks) into an image-space TSDF in the novel view, including the previous frame's rendered depth, and use the resulting view- and temporally consistent depth to guide a blending network that fuses forward-rendered 3D Gaussian-splatted input views.
Key results
- On DyNeRF, the method reaches 31.17 PSNR and 0.936 SSIM, versus 29.51 PSNR for ENeRF.
- On D3DMV, the method reaches 32.58 PSNR and 0.098 LPIPS, improving over ENeRF's 27.79 PSNR.
- Total runtime is about 100 ms per frame without TensorRT (blending 65 ms), faster than GPSG's 143.4 ms; on ENeRF-Outdoor it trails ENeRF on PSNR (24.01 vs 25.79).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use geometry-guided blending of forward-rendered views instead of rendering all input-view Gaussians in one pass (ablation) | Ours · DyNeRF | PSNR: 24.64 -> 31.17 | Helped |
| Use geometry-guided blending instead of distance-based blending (ablation) | Ours · DyNeRF | PSNR: 26.08 -> 31.17 | Helped |
| Add TSDF depth guidance to the blending network (ablation) | Ours · DyNeRF | TCC: 0.869 -> 0.872 | Helped |
| Add difference-mask temporal filtering of input-view depths (ablation) | Ours · DyNeRF | TCC: 0.869 -> 0.872 | Helped |
| Use Ours rather than ENeRF | novel-view synthesis · DyNeRF | PSNR: 29.51 -> 31.17 | Helped |
| Use Ours rather than ENeRF | novel-view synthesis · ENeRF-Outdoor | PSNR: 25.79 -> 24.01 | Hurt |
| Use Ours rather than ENeRF | novel-view synthesis · D3DMV | PSNR: 27.79 -> 32.58 | Helped |
Surprising findings
- On the ENeRF-Outdoor dataset the method has lower PSNR than ENeRF (24.01 vs 25.79), attributed partly to camera mis-synchronization and color calibration issues.
- Rendering all input-view Gaussians in a single pass with no blending yields much lower temporal consistency than blending.
Limitations
- Relies strongly on accurate geometry; fails in specular regions where depth is poor, causing view and temporal artifacts.
- Requires well-synchronized and calibrated cameras; calibration errors cause blurring.
- Runtime grows roughly linearly with the number of input views K.
Experimental setup
Datasets: ScanNet Google Spaces DyNeRF ENeRF-Outdoor D3DMV
Benchmarks: DyNeRF ENeRF-Outdoor D3DMV
Metrics: PSNR SSIM LPIPS L1 TCC STED SDT SDV
Baselines: FWD IBRNet GPSG ENeRF
Builds on: 3D Gaussian splatting image-based TSDF (Lawrence et al.) ENeRF IBRNet RAFT-Stereo
Closest papers by meaning
- SimVS: Simulating World Inconsistencies for Robust View Synthesis (CVPR 2025 · similarity 0.87)
- MultiDiff: Consistent Novel View Synthesis from a Single Image (CVPR 2024 · similarity 0.87)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.86)
- SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input (CVPR 2025 · similarity 0.86)
- GeoVideo: Introducing Geometric Regularization into Video Generation Model (NeurIPS 2025 · similarity 0.86)
- MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.