Research map / 3D vision / 3D generation
Pseudo-Generalized Dynamic View Synthesis from a Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an analysis framework for generalized dynamic novel view synthesis from monocular video by combining a pretrained GNT (adapted with masked attention) for static content with depth- and flow-based point-cloud rendering for dynamic content, showing that consistent depth removes the need for per-scene appearance optimization.
Problem
Rendering dynamic scenes observed in a monocular video from novel viewpoints and times has so far required scene-specific appearance optimization; it is unknown whether a generalized (no test-time fitting) approach is possible.
Approach
Separate static and dynamic rendering: adapt a pretrained generalizable NeRF transformer (GNT) for static content using masked attention that excludes source views with dynamic content, and render dynamic content by lifting masked pixels to 3D with consistent depth and optical flow, interpolating linearly in time and splatting into the target view. Consistent depth is identified as the sufficient
Key results
- On NVIDIA Dynamic Scenes the method (no appearance fitting) reaches LPIPS 64.29 on full images, better than TiNeuVox, Nerfies, HyperNeRF, DVS (81.93) and NSFF (62.11 is better than 64.29, so it is only ahead of DVS and others), with PSNR 26.15.
- On DyCheck iPhone it achieves mLPIPS 0.340, on par with Nerfies (0.339) and better than NSFF (0.396), T-NeRF (0.379), and TiNeuVox (0.532), while having mPSNR 15.88.
- Replacing consistent depth with ZoeDepth drops full-image PSNR from 26.15 to 21.15 and LPIPS rises from 64.29 to 142.3, showing consistent depth is necessary.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use pseudo-generalized rendering without scene-specific appearance fitting | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 31.20 -> 64.29 | Hurt |
| use pseudo-generalized rendering without scene-specific appearance fitting | GNT-based method · DyCheck iPhone | mLPIPS: 0.332 -> 0.340 | Hurt |
| replace aggregated depth point-cloud static rendering with GNT static rendering (ablation) | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 211.2 -> 69.70 | Helped |
| add masked attention to GNT (ablation) | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 69.70 -> 64.29 | Helped |
| replace consistent depth with ZoeDepth predictions (ablation) | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 64.29 -> 142.3 | Hurt |
| add TAPIR tracking to aggregate dynamic content from more frames (ablation) | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 64.29 -> 70.34 | Hurt |
| add CoTracker tracking to aggregate dynamic content from more frames (ablation) | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 64.29 -> 69.65 | Hurt |
| replace SoftSplat with point-based rendering (ablation) | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 64.29 -> 65.67 | Hurt |
| replace SoftSplat with mesh-based rendering (ablation) | GNT-based method · NVIDIA Dynamic Scenes | LPIPS: 64.29 -> 65.63 | Hurt |
| replace PointRenderer with SoftSplat (ablation) | GNT-based method · DyCheck iPhone | mLPIPS: 0.340 -> 0.342 | No clear effect |
Surprising findings
- Using point tracking (TAPIR, CoTracker) to aggregate more temporal information reduced performance rather than improving it.
- Consistent depth alone is not enough: a point-cloud static rendering with CD performed far worse than GNT-based static rendering.
Limitations
- Still requires scene-specific consistent depth optimization (about three GPU hours per video), so it is only pseudo-generalized.
- Trails the best scene-specific methods (e.g., DynIBaR) in PSNR/SSIM and LPIPS on NVIDIA Dynamic Scenes.
- Artifacts from missing foreground parts and background blur, given the lack of context-aware inpainting.
Experimental setup
Datasets: NVIDIA Dynamic Scenes DyCheck iPhone DAVIS ADE20K COCO
Benchmarks: NVIDIA Dynamic Scenes DyCheck iPhone
Metrics: PSNR SSIM LPIPS mPSNR mSSIM mLPIPS
Baselines: TiNeuVox Nerfies HyperNeRF NSFF DVS DynIBaR T-NeRF ZoeDepth
Builds on: GNT NSFF DVS SoftSplat COLMAP
Closest papers by meaning
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.91)
- Dynamic View Synthesis as an Inverse Problem (NeurIPS 2025 · similarity 0.89)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.89)
- MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors (ICLR 2025 · similarity 0.88)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.88)
- Novel View Synthesis with Pixel-Space Diffusion Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.