Research map / 3D vision / 3D generation
4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 4D-fy, a text-to-4D generator that alternates score distillation gradients from a 3D-aware text-to-image model, a fine-tuned VSD text-to-image model, and a text-to-video model over a hash-grid 4D radiance field.
Problem
Text-to-4D generation faces a tradeoff between appearance quality, 3D structure (Janus problem), and motion when using SDS with text-to-image, 3D-aware, or text-to-video diffusion models.
Approach
Hybrid score distillation sampling: a three-stage alternating optimization that blends SDS gradients from MVDream (3D-aware T2I), VSD with Stable Diffusion, and video SDS with Zeroscope, using a static+dynamic multiresolution hash-grid representation.
Key results
- Full 4D-fy achieves CLIP score 35.0 versus 33.9 for MAV3D, and 67% overall user preference over MAV3D.
- Removing 3D-aware guidance drops 3D structure preference handling; w/o hybrid SDS yields 95% overall preference for 4D-fy.
- Replacing hash grid with HexPlane reduces CLIP score to 34.5 versus 35.0 for the full method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hybrid SDS instead of MAV3D | 4D-fy · Text-to-4D prompts | CLIP Score: 33.9 -> 34.2 | Helped |
| add 3D-aware SDS and image guidance (ablation) | 4D-fy · Text-to-4D ablation prompts | CLIP Score: 29.3 -> 35.0 | Helped |
| add 3D-aware SDS (ablation) | 4D-fy · Text-to-4D ablation prompts | 3D structure quality | Helped |
| add VSD image guidance (ablation) | 4D-fy · Text-to-4D ablation prompts | Appearance quality | Helped |
| retain image guidance in stage 3 with hybrid SDS (ablation) | 4D-fy · Text-to-4D ablation prompts | CLIP Score: 33.8 -> 35.0 | Helped |
| use the hash-grid representation instead of HexPlane (ablation) | 4D-fy · Text-to-4D ablation prompts | CLIP Score: 34.5 -> 35.0 | Helped |
| use text-to-video guidance only, removing 3D-aware SDS and image guidance (ablation) | 4D-fy · Text-to-4D ablation prompts | Appearance quality | Hurt |
Surprising findings
- Removing the 3D-aware guidance gives higher CLIP score (35.1) than the full method (35.0), while users still prefer the full method on 3D structure.
Limitations
- Motion complexity limited to simple movements, bounded by text-to-video model quality.
- Scene generation is slow, taking significant time per scene.
- Existing metrics (CLIP Score, user study) are insufficient for 3D/4D evaluation.
Experimental setup
Benchmarks: user study (28 prompts vs MAV3D, 5 prompts for ablations)
Metrics: CLIP Score human preference (appearance, 3D structure, motion, text ali
Baselines: MAV3D Make-A-Video3D
Builds on: score distillation sampling variational score distillation MVDream Instant-NGP hash encoding threestudio
Closest papers by meaning
- A Unified Approach for Text- and Image-guided 4D Scene Generation (CVPR 2024 · similarity 0.90)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models (CVPR 2024 · similarity 0.90)
- EG4D: Explicit Generation of 4D Object without Score Distillation (ICLR 2025 · similarity 0.90)
- Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior (CVPR 2024 · similarity 0.90)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.