Research map / 3D vision / 3D generation
Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CogNVS, a video inpainting diffusion model fine-tuned from CogVideoX-5B on self-supervised reconstruction-masked 2D video pairs, applied to dynamic novel-view synthesis with test-time finetuning on the target video.
Problem
Novel-view synthesis of dynamic scenes from casually captured monocular videos, where test-time optimization is slow and feed-forward models lack geometric accuracy and generalize poorly.
Approach
Decompose dynamic view synthesis into reconstructing the scene (MegaSAM), rendering covisible pixels from novel views, and inpainting hidden regions with a video diffusion model trained self-supervisedly on 2D videos with reconstruction-derived covisibility masks, then test-time finetuning on the target video.
Key results
- On Kubric-4D, CogNVS reaches 22.63 PSNR / 0.760 SSIM / 0.232 LPIPS, beating TrajCrafter (20.93 PSNR) and Gen3C (19.41 PSNR) despite zero-shot evaluation.
- On DyCheck, CogNVS (MoSca) achieves the best FID (92.83) and KID (0.031 for MoSca, 0.030 for MegaSAM variant), though PSNR/SSIM are behind MoSca on SSIM.
- Ablation on Kubric-4D: removing pretraining drops PSNR from 23.29 to 18.62; removing test-time finetuning drops PSNR to 20.06.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply CogNVS instead of the strongest baseline | dynamic novel-view synthesis · Kubric-4D | PSNR: 20.93 -> 22.63 | Helped |
| apply CogNVS instead of the strongest baseline | dynamic novel-view synthesis · Kubric-4D | FID: 98.58 -> 102.47 | No clear effect |
| apply CogNVS instead of the strongest baseline | dynamic novel-view synthesis · ParallelDomain-4D | PSNR: 21.77 -> 24.34 | Helped |
| apply CogNVS instead of the strongest baseline | dynamic novel-view synthesis · DyCheck | FID: 140.35 -> 92.83 | Helped |
| add large-scale pretraining (ablation) | CogNVS with test-time finetuning · Kubric-4D (3 sequences) | PSNR: 18.62 -> 23.29 | Helped |
| add test-time finetuning (ablation) | pretrained CogNVS · Kubric-4D (3 sequences) | PSNR: 20.06 -> 23.29 | Helped |
| replace random masking with structured masking (ablation) | CogNVS · Kubric-4D (3 sequences) | PSNR: 20.62 -> 23.29 | Helped |
| replace tube masking with structured masking (ablation) | CogNVS · Kubric-4D (3 sequences) | PSNR: 21.75 -> 23.29 | Helped |
| use groundtruth depth instead of MegaSAM reconstruction (ablation) | CogNVS · Kubric-4D | PSNR: 19.62 -> 22.63 | Helped |
| add static background stacking (ablation) | CogNVS using MegaSAM · DyCheck | PSNR: 11.55 -> 14.27 | Helped |
| add noise addition during training (ablation) | CogNVS with background stacking · DyCheck | FID: 180.67 -> 156.83 | Helped |
| increase test-time finetuning epochs from 0 to 50 (ablation) | CogNVS · Kubric-4D | PSNR | Helped |
Surprising findings
- Test-time finetuning alone without large-scale pretraining cannot escape poor local minima, yielding substantially worse PSNR (18.62 vs 23.29).
- Optimization-based baselines such as MoSca achieve higher PSNR/SSIM on DyCheck than CogNVS (MegaSAM), with CogNVS winning mainly on FID/KID.
Limitations
- Performance depends on quality of off-the-shelf reconstruction (MegaSAM, MoSca).
- Training uses a relatively small set of about 10,000 2D videos and no 3D/4D data.
- Random camera trajectory sampling does not match real-world camera trajectory diversity; test-time finetuning costs about 140 min per video.
Experimental setup
Datasets: SA-V TAO Youtube-VOS DAVIS Kubric-4D ParallelDomain-4D DyCheck
Benchmarks: Kubric-4D ParallelDomain-4D DyCheck
Metrics: PSNR SSIM LPIPS FID KID mPSNR mSSIM mLPIPS
Baselines: GCD Gen3C TrajectoryCrafter Shape-of-Motion MoSca CAT4D MegaSAM
Builds on: CogVideoX MegaSAM MoSca GCD TrajectoryCrafter
Closest papers by meaning
- Dynamic View Synthesis as an Inverse Problem (NeurIPS 2025 · similarity 0.91)
- ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models (CVPR 2024 · similarity 0.91)
- Pseudo-Generalized Dynamic View Synthesis from a Video (ICLR 2024 · similarity 0.91)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.90)
- World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video (NeurIPS 2026 · similarity 0.90)
- ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.