Research map / 3D vision / 3D generation
Dynamic View Synthesis as an Inverse Problem
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free dynamic view synthesis from monocular video by redesigning the noise initialization of CogVideoX-5B-I2V via K-order Recursive Noise Representation (K-RNR) and Stochastic Latent Modulation (SLM).
Problem
Dynamic view synthesis from monocular videos requires preserving scene identity and completing newly visible regions under camera motion, but existing methods need weight updates, auxiliary modules, or synthetic training data, and DDIM inversion fails under zero-terminal SNR video diffusion schedule
Approach
Identify that zero-terminal SNR makes the forward map non-injective so DDIM inversion cannot recover the source latent, then build a recursive noise initialization (K-RNR) with a closed-form expression that aligns the noise with the VAE-encoded pivot latent, plus adaptive AdaIN scale stabilization and latent-space visibility-aware permutation to fill occluded regions.
Key results
- On the 1100-video evaluation set, the method reports FID 53.15 and FVD 103.44 versus ReCamMaster's 58.12 and 118.82.
- Mean PSNR 14.50 and LPIPS 0.487 across OpenVid, DAVIS and Synthetic sets, versus ReCamMaster's mean PSNR 13.77 and LPIPS 0.508.
- Camera rotation error 1.31 vs 1.46 for ReCamMaster (best among compared methods).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed noise-space framework instead of ReCamMaster | Dynamic view synthesis on OpenVid-1M · OpenVid-1M (1000 samples) | FID: 58.12 -> 53.15 | Helped |
| Use the proposed noise-space framework instead of ReCamMaster | Dynamic view synthesis on OpenVid-1M · OpenVid-1M (1000 samples) | CLIP-F: 98.89 -> 98.54 | Hurt |
| Use the proposed noise-space framework instead of ReCamMaster | Dynamic view synthesis on OpenVid-1M · OpenVid-1M (1000 samples) | RotErr: 1.46 -> 1.31 | Helped |
| Use the proposed noise-space framework instead of ReCamMaster | Dynamic view synthesis on OpenVid-1M · OpenVid-1M (1000 samples) | CLIP-V: 89.91 -> 92.04 | Helped |
| Use the proposed noise-space framework instead of ReCamMaster | Dynamic view synthesis · Curated benchmark (Mean) | PSNR: 13.77 -> 14.50 | Helped |
| Use the proposed noise-space framework instead of ReCamMaster | Dynamic view synthesis · Curated benchmark (Mean) | SSIM: 0.524 -> 0.531 | Helped |
| Use the proposed noise-space framework instead of ReCamMaster | Dynamic view synthesis · Curated benchmark (Mean) | LPIPS: 0.508 -> 0.487 | Helped |
| Add K-RNR initialization (ablation) | CogVideoX-5B-I2V video reconstruction · Video reconstruction (no camera transformation) | PSNR | Helped |
| Increase K-RNR recursion depth beyond k = 3 (ablation) | K-RNR with adaptive scaling · Video reconstruction | PSNR | Helped |
| Increase K-RNR recursion depth to k = 8 (ablation) | K-RNR with adaptive scaling · Video reconstruction | PSNR | Hurt |
| Set K-RNR recursion depth to k = 3 (ablation) | K-RNR with adaptive scaling · Video reconstruction | LPIPS: 0.465 | Helped |
| Set adaptive reference index to δ = 3 (ablation) | Adaptive K-RNR · Video reconstruction | PSNR: 24.97 | Helped |
Surprising findings
- Standard DDIM inversion-initialized noise yields washed-out, oversaturated outputs, and K-RNR's scale explodes without the adaptive AdaIN correction.
- Simply repeating background patches at the input level produced artifacts, motivating latent-space modulation instead.
Limitations
- Inherits biases of the base diffusion model, which may fail on uncommon objects or heavy occlusion.
- Stochastic latent modulation can produce unstable or incoherent results when large regions become newly visible.
- Dependence on monocular depth estimation (DepthCrafter) for point cloud rendering and visibility masks.
Experimental setup
Datasets: OpenVid-1M DAVIS Sora-generated videos
Benchmarks: OpenVid-1M DAVIS curated 1100-video benchmark
Metrics: FID FVD CLIP-T CLIP-F RotErr TransErr Mat. Pix. FVD-V CLIP-V PSNR
Baselines: GCD TrajectoryAttention DaS TrajectoryCrafter ReCamMaster
Builds on: DDIM inversion CogVideoX DepthCrafter Diffusion-as-Shader
Closest papers by meaning
- Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos (NeurIPS 2025 · similarity 0.91)
- NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer (ICLR 2025 · similarity 0.90)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.89)
- Pseudo-Generalized Dynamic View Synthesis from a Video (ICLR 2024 · similarity 0.89)
- UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis (NeurIPS 2025 · similarity 0.89)
- NAMVIS: Next-Scale Autoregressive Multi-View Image Synthesis (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.