Research map / 3D vision / Gaussian splatting
Compressing Streamable Free-Viewpoint Videos to 0.1 MB per Frame
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iFVC, a Gaussian-based streamable free-viewpoint video compression framework that uses anchor-based 3DGS with a binary transformation cache (saliency-assisted tri-plane) trained under joint rate-distortion loss, reaching ~0.1 MB per frame with on-the-fly training and real-time rendering.
Problem
Streamable free-viewpoint video reconstruction from multi-view video needs photo-realistic quality, fast per-frame online training, real-time rendering, and low storage, which existing methods do not jointly achieve.
Approach
Model inter-frame changes of anchor-based Gaussians with a compact binary transformation cache (high-resolution transformation tri-plane plus a saliency grid to reduce hash collisions), and optimize rendering loss plus an entropy estimate of the binary cache end-to-end for rate-distortion.
Key results
- On N3DV, iFVC reaches 32.32 dB PSNR with 0.09 MB storage per frame and 0.22 min training, versus 3DGStream's 31.67 dB at 7.8 MB.
- On MeetRoom, iFVC achieves 32.38 dB PSNR at 0.09 MB per frame and 157 FPS rendering, versus 3DGStream's 30.79 dB at 4.1 MB.
- Ablation on N3DV: the binary transformation cache with saliency grid gives 32.32 dB at 0.09 MB, versus NTC at 32.29 dB and 7.60 MB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use iFVC instead of 3DGStream | Online FVV reconstruction · N3DV | PSNR (dB): 31.67 -> 32.32 | Helped |
| Use iFVC instead of 3DGStream | Online FVV reconstruction · MeetRoom | PSNR (dB): 30.79 -> 32.38 | Helped |
| Use iFVC instead of 3DGStream | Online FVV reconstruction · VRU Basketball Game | PSNR (dB): 26.20 -> 29.55 | Helped |
| Replace the hybrid 2D-3D binary grid with a high-resolution transformation tri-plane (ablation) | iFVC BTC · N3DV | PSNR (dB): 31.89 -> 32.24 | Helped |
| Add a saliency grid to the binary transformation tri-plane (ablation) | iFVC BTC · N3DV | PSNR (dB): 32.24 -> 32.32 | Helped |
| Add a saliency grid to the binary transformation tri-plane (ablation) | iFVC BTC · N3DV | Storage (MB): 0.12 -> 0.09 | Helped |
| Keep anchor location fixed rather than modeling its change Δx (ablation) | iFVC dynamic anchors · N3DV | PSNR (dB): 32.03 -> 32.32 | Helped |
| Keep anchor location fixed rather than modeling its change Δx (ablation) | iFVC dynamic anchors · N3DV | Storage (MB): 0.20 -> 0.09 | Helped |
| Keep anchor scaling factor fixed rather than modeling its change Δl (ablation) | iFVC dynamic anchors · N3DV | PSNR (dB): 31.76 -> 32.32 | Helped |
| Increase tri-plane resolutions from 512, 1024, 2048, 4096 to 1024, 2048, 4096, 8192 (ablation) | iFVC BTC · N3DV | PSNR (dB): 32.32 -> 32.36 | Helped |
| Increase tri-plane resolutions from 512, 1024, 2048, 4096 to 1024, 2048, 4096, 8192 (ablation) | iFVC BTC · N3DV | Storage (MB): 0.092 -> 0.098 | Hurt |
Surprising findings
- The commonly used hybrid 2D-3D binary grid from BiRF drops quality from 32.29 dB to 31.89 dB when applied to dynamic transformations.
Limitations
- Quality of the initial timestep-0 representation is critical and errors accumulate across frames.
- Limited training iterations may fit large-scale motions poorly.
- Large-scale complex motion scenes (VRU) remain challenging.
Experimental setup
Datasets: N3DV MeetRoom VRU Basketball Game
Benchmarks: N3DV MeetRoom VRU Basketball Game
Metrics: PSNR storage (MB) training time (mins) rendering FPS
Baselines: 3DGStream StreamRF Spacetime-GS K-Planes HyperReel MixVoxels TeTriRF NeRFPlayer 4DGS 4DRotor-GS Gaussian-Flow Realtime-4DGS
Builds on: 3D Gaussian Splatting Scaffold-GS 3DGStream BiRF Instant-NGP
Closest papers by meaning
- 3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos (CVPR 2024 · similarity 0.92)
- 4DGC: Rate-Aware 4D Gaussian Compression for Efficient Streamable Free-Viewpoint Video (CVPR 2025 · similarity 0.92)
- Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction (NeurIPS 2025 · similarity 0.91)
- Instant Gaussian Stream: Fast and Generalizable Streaming of Dynamic Scene Reconstruction via Gaussian Splatting (CVPR 2025 · similarity 0.90)
- HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting (NeurIPS 2024 · similarity 0.90)
- Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.