Research map / Video / Video generation
SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stable Video 4D (SV4D) fine-tunes a unified latent video diffusion model with frame and view attention to jointly generate multi-view, multi-frame novel-view videos from a monocular video, then optimizes a dynamic NeRF from them without SDS.
Problem
Generating dynamic 3D (4D) objects from a single monocular video requires novel-view videos that are consistent across both time (frames) and camera views, but prior methods combine separately trained video and multi-view models or rely on slow SDS optimization.
Approach
Extend SVD with view attention and frame attention layers (initialized from SV3D and SVD respectively) so one diffusion model jointly reasons over the view and frame axes, conditioned on the input video and reference multi-views of the first frame; a mixed interleaved/dense sampling scheme handles long videos, and the output is fit with a dynamic NeRF using photometric and geometric losses only.
Key results
- On ObjaverseDy novel view video synthesis, SV4D reduces FVD-F to 585.09 from 729.67 for SV3D and FV4D to 470.46 from 546.56 for STAG4D.
- On Consistent4D 4D generation, SV4D achieves LPIPS 0.118 and FVD-F 732.40, versus 0.126 and 992.21 for STAG4D.
- User study: SV4D novel-view videos preferred 73.3% over SV3D (13.7%), Diffusion^2 (5.0%), and STAG4D (8.0%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the unified SV4D model instead of SV3D for novel-view video synthesis | novel view video synthesis · Consistent4D | FVD-F: 989.53 -> 677.68 | Helped |
| use the unified SV4D model instead of STAG4D for novel-view video synthesis | novel view video synthesis · Consistent4D | FVD-F: 861.88 -> 677.68 | Helped |
| use the unified SV4D model instead of STAG4D for novel-view video synthesis | novel view video synthesis · ObjaverseDy | FV4D: 546.56 -> 470.46 | Helped |
| use the unified SV4D model instead of DreamGaussian4D for 4D generation | 4D generation · ObjaverseDy | LPIPS: 0.152 -> 0.131 | Helped |
| use the unified SV4D model instead of DreamGaussian4D for 4D generation | 4D generation · ObjaverseDy | FV4D: 638.15 -> 614.35 | Helped |
| add frame attention (ablation) | SV4D · ObjaverseDy | FVD-F: 729.67 -> 585.09 | Helped |
| add frame attention (ablation) | SV4D · ObjaverseDy | FV4D: 690.49 -> 470.46 | Helped |
| add reference multi-views (ablation) | SV4D · ObjaverseDy | FVD-F: 876.07 -> 585.09 | Helped |
| use VAE latents of reference multi-views rather than CLIP embeddings and no reference multi-views (ablation) | SV4D · ObjaverseDy | FVD-F: 1174.47 -> 585.09 | Helped |
| use SV4D sampling rather than AMT interpolation (ablation) | SV4D novel-view video sampling · ObjaverseDy | FVD-F: 612.61 -> 585.09 | Helped |
| use SV4D sampling rather than independent sampling (ablation) | SV4D novel-view video sampling · ObjaverseDy | FV4D: 488.31 -> 470.46 | Helped |
| use SV4D sampling rather than interleaved sampling (ablation) | SV4D novel-view video sampling · ObjaverseDy | FV4D: 717.21 -> 470.46 | Helped |
Surprising findings
- The SV4D network's own LPIPS (0.136) is slightly worse than SV3D (0.131) on ObjaverseDy, so image quality is not improved while temporal and multi-view consistency are.
Limitations
- Memory limits require sequential submatrix sampling, which can introduce inconsistencies between submatrices without the mixed sampling scheme.
- Training data is limited to CC-licensed animated Objaverse objects, so generalization is evaluated mainly on synthetic data plus qualitative DAVIS results.
Experimental setup
Datasets: Objaverse ObjaverseDy
Benchmarks: ObjaverseDy Consistent4D DAVIS
Metrics: LPIPS CLIP-S FVD-F FVD-V FVD-Diag FV4D
Baselines: SV3D Diffusion^2 STAG4D 4Diffusion Consistent4D DreamGaussian4D GaussianFlow 4DGen Efficient4D
Builds on: Stable Video Diffusion SV3D D-NeRF Instant-NGP
Closest papers by meaning
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.93)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.93)
- Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video (ICLR 2024 · similarity 0.92)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.92)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.92)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.