Research map / Video / Video generation
Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Consistent4D generates 360-degree 4D dynamic objects from a static monocular video by optimizing a Cascade DyNeRF with Zero123-XL SDS supervision plus an interpolation-driven consistency loss from RIFE, followed by a GAN video enhancer.
Problem
Generating 360-degree dynamic 3D objects (4D) from uncalibrated monocular videos captured by a static camera, where multi-view supervision is unavailable and existing dynamic NeRFs fail on unobserved regions.
Approach
Use a 3D-aware image-to-image diffusion model (Zero123-XL) as SDS supervision for a coarse-to-fine residual K-planes DyNeRF (Cascade DyNeRF), and add an Interpolation-driven Consistency Loss that matches rendered frames to frames interpolated by a pretrained video interpolation model (RIFE), which also reduces the multi-face Janus problem in text-to-3D.
Key results
- On 7 synthetic Sketchfab objects, Consistent4D averages LPIPS 0.16 and CLIP 0.87, versus K-planes 0.38/0.72 and D-NeRF 0.51/0.68.
- In a user study on 20 video-to-4D objects, results with ICL were preferred in 75.5% of cases over w/o ICL.
- For text-to-3D with DreamFusion on 230 animal prompts, ICL raised the user-judged success rate from 19.3% to 28.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Consistent4D instead of D-NeRF | Video-to-4D generation · Video-to-4D synthetic, average over 7 objects | LPIPS: 0.51 -> 0.16 | Helped |
| Use Consistent4D instead of K-planes | Video-to-4D generation · Video-to-4D synthetic, average over 7 objects | CLIP: 0.72 -> 0.87 | Helped |
| Add Interpolation-driven Consistency Loss (ICL) (ablation) | Consistent4D · Video-to-4D user study | preference rate: 24.5 -> 75.5 | Helped |
| Add Interpolation-driven Consistency Loss (ICL) (ablation) | DreamFusion · Text-to-3D user study | success rate: 19.3 -> 28.6 | Helped |
| Replace Cascade DyNeRF with the original K-planes architecture (ablation) | Consistent4D · Video-to-4D | generation quality | Hurt |
| Add spatial ICL (ablation) | Consistent4D · Video-to-4D | spatial consistency | Helped |
| Add temporal ICL (ablation) | Consistent4D · Video-to-4D | temporal consistency | Helped |
| Use both spatial and temporal ICL (ablation) | Consistent4D · Video-to-4D | spatiotemporal consistency | Helped |
| Add the cross-frame video enhancer (ablation) | Consistent4D rendered video · Video-to-4D | visual quality | Helped |
| Use input videos without sampling them to 32 frames (ablation) | Consistent4D · Video-to-4D | generation quality | Helped |
Limitations
- Fails on complex or abrupt motion and when parts are occluded in the input view
- Multi-face issues persist in some text-to-3D cases even with ICL
- Relies on video interpolation between rendered frames, and evaluation uses only 7 synthetic objects
Experimental setup
Datasets: Sketchfab animated models (synthetic) in-the-wild internet videos
Benchmarks: Sketchfab animated models (7 objects, rendered by authors)
Metrics: LPIPS CLIP similarity user preference rate success rate
Baselines: D-NeRF K-planes DreamFusion
Builds on: DreamFusion Zero123 K-planes RIFE pix2pix
Closest papers by meaning
- SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency (ICLR 2025 · similarity 0.92)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videos (NeurIPS 2024 · similarity 0.90)
- CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models (CVPR 2025 · similarity 0.90)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.89)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.