Research map / 3D vision / NeRF and radiance fields
VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VideoRF, which serializes a dynamic human-centric radiance field into a 2D feature image stream trained with video-codec-friendly losses, enabling H.264 compression and real-time mobile rendering via deferred shading.
Problem
Real-time streaming and rendering of long-duration dynamic radiance fields on mobile devices is hard due to storage and computational constraints.
Approach
Represent each frame's 4D radiance field as a 2D feature image with a 3D-to-2D mapping table (Morton-sorted, grouped adaptively), train it with spatial total-variation and inter-frame temporal losses so that off-the-shelf video codecs and hardware decoders can compress and play it, and render with a deferred shading tiny MLP in a fragment shader.
Key results
- On the ReRF Kpop dataset VideoRF reaches 32.01 PSNR / 0.023 LPIPS at 0.658 MB, versus ReRF's 31.84 PSNR at 0.645 MB.
- On the Actors-HQ dataset VideoRF achieves 28.46 PSNR at 0.426 MB, below HumanRF's 28.98 PSNR (2.286 MB), with HumanRF reported as best quality.
- Runtime at 1920x1080: 116 FPS on desktop (RTX3090), 40 FPS on tablet (iPad Pro M2), 23 FPS on phone (iPhone 14 Pro), versus ReRF at 4 FPS on desktop.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spatial consistency loss to VideoRF (ablation) | VideoRF · ReRF | PSNR | Helped |
| Add temporal consistency loss to VideoRF (ablation) | VideoRF · ReRF | PSNR | Helped |
| Use 3D and 2D Morton sorting (ablation) | VideoRF · ReRF | PSNR | Helped |
| Use block-wise storage for the 2D feature space (ablation) | VideoRF · ReRF | PSNR | Helped |
| Use adaptive grouping instead of per-frame mapping tables (ablation) | VideoRF · ReRF | PSNR | Helped |
Surprising findings
- TiNeuVox has the smallest storage but suffers increasing blur as frame count grows.
- ReRF shows slightly better quality than VideoRF in the 700-800KB storage range.
Limitations
- Phone rendering reaches only 23 FPS and laptop 25 FPS at 1920x1080.
- On the HumanRF Actor3 sequence, quality is second to HumanRF.
Experimental setup
Datasets: ReRF dataset HumanRF dataset (Actor3 sequence 1) self-captured dynamic dataset
Benchmarks: ReRF dataset (Kpop) HumanRF dataset (Actor3) Actors-HQ
Metrics: PSNR SSIM LPIPS storage size (MB) FPS decoding/rendering time (ms)
Baselines: ReRF MeRF HumanRF TiNeuVox
Builds on: MeRF ReRF TensoRF-style grid NeRF deferred shading H.264
Closest papers by meaning
- TeTriRF: Temporal Tri-Plane Radiance Fields for Efficient Free-Viewpoint Video (CVPR 2024 · similarity 0.90)
- Ced-NeRF: A Compact and Efficient Method for Dynamic Neural Radiance Fields (AAAI 2024 · similarity 0.89)
- MuRF: Multi-Baseline Radiance Fields (CVPR 2024 · similarity 0.89)
- BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling (AAAI 2024 · similarity 0.89)
- NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild (CVPR 2024 · similarity 0.88)
- SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.