Research map / 3D vision / Gaussian splatting
3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Streams photo-realistic free-viewpoint videos by per-frame on-the-fly training of 3D Gaussians, using a hash-encoded neural transformation cache plus adaptively spawned frame-specific Gaussians for emerging objects.
Problem
Building photo-realistic free-viewpoint videos of dynamic scenes from multi-view video typically requires offline training on the full sequence and does not achieve real-time rendering, so such videos cannot be streamed.
Approach
Carry 3D Gaussians across timesteps and model their per-frame motion with a compact Neural Transformation Cache (multi-resolution hash encoding plus a shallow fully-fused MLP) trained on-the-fly, while spawning frame-specific Gaussians near high view-space gradient regions to capture emerging objects, storing only the NTC and these additions per frame.
Key results
- On N3DV, 3DGStream reaches 31.67 dB PSNR with 7.6 MB storage per frame, 0.20 min training per frame and 215 FPS rendering, versus StreamRF's 30.68 dB, 17.7 MB, 0.25 min, 8.3 FPS.
- On Meet Room, 3DGStream achieves 30.79 dB PSNR, 0.10 min training per frame and 288 FPS, outperforming StreamRF's 26.72 dB at 0.17 min and 10 FPS.
- Adaptive addition on flame salmon reaches 28.42 dB using 477.7 additional 3DGs on average, versus 28.39 dB baseline with no additions and 971.9 randomly spawned 3DGs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3DGStream instead of StreamRF | online FVV reconstruction · N3DV | PSNR (dB): 30.68 -> 31.67 | Helped |
| use 3DGStream instead of StreamRF | online FVV reconstruction · N3DV | Rendering speed (FPS): 8.3 -> 215 | Helped |
| use 3DGStream instead of StreamRF | online FVV reconstruction · Meet Room | PSNR (dB): 26.72 -> 30.79 | Helped |
| use 3DGStream instead of StreamRF | online FVV reconstruction · Meet Room | Rendering speed (FPS): 10 -> 288 | Helped |
| remove adaptive 3DG addition (Stage 2) (ablation) | 3DGStream · Flame salmon | PSNR (dB): 28.39 -> 28.42 | No clear effect |
| randomly spawn additional 3DGs instead of using adaptive spawning (ablation) | 3DGStream · Flame salmon | PSNR (dB): 28.39 -> 28.42 | No clear effect |
| remove adaptive 3DG quantity control (ablation) | 3DGStream · Flame salmon | Additional 3DGs: 477.7 -> 8710.8 | Hurt |
| remove multi-resolution hash encoding from the NTC (ablation) | 3DGStream · Flame salmon | PSNR | Hurt |
| remove NTC warm-up (ablation) | 3DGStream · Flame salmon | PSNR | Hurt |
| directly optimize the previous frame's 3DGs instead of using NTC (ablation) | 3DGStream · Flame salmon | Storage | Hurt |
Surprising findings
- Direct per-frame optimization of transformed 3DGs gives the best quality but at much larger storage, while NTC achieves comparable quality with far less storage.
- SH rotation is the most time-consuming rendering step yet has minimal impact on reconstruction quality, according to the authors.
Limitations
- Performance depends heavily on the quality of the initial 3DG-S reconstruction and its point cloud; distant landscapes show artifacts.
- Limited training iterations restrict modeling of drastic motion and complex emerging objects.
- Stored per-frame NTC and additional 3DGs still accumulate storage over long sequences.
Experimental setup
Datasets: N3DV Meet Room
Benchmarks: N3DV Meet Room
Metrics: PSNR training time storage rendering FPS
Baselines: Plenoxels I-NGP 3DG-S DyNeRF NeRFPlayer HexPlane K-Planes HyperReel MixVoxels StreamRF
Builds on: 3D Gaussian Splatting Neural Radiance Caching I-NGP tiny-cuda-nn
Closest papers by meaning
- Compressing Streamable Free-Viewpoint Videos to 0.1 MB per Frame (AAAI 2025 · similarity 0.92)
- Instant Gaussian Stream: Fast and Generalizable Streaming of Dynamic Scene Reconstruction via Gaussian Splatting (CVPR 2025 · similarity 0.91)
- Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction (NeurIPS 2025 · similarity 0.91)
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.90)
- 4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video (CVPR 2025 · similarity 0.90)
- QUEEN: QUantized Efficient ENcoding of Dynamic Gaussians for Streaming Free-viewpoint Videos (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.