Research map / 3D vision / Gaussian splatting
Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Align Your Gaussians (AYG) synthesizes text-to-4D dynamic 3D objects by optimizing a deformation field over dynamic 3D Gaussians via classifier score distillation that composes text-to-video, text-to-image, and multiview (MVDream) diffusion models.
Problem
Text-to-4D generation of dynamic 3D objects/scenes is underexplored; prior work (MAV3D) uses NeRF/HexPlane representations, produces short sequences, and cannot easily compose objects into scenes.
Approach
Represent 4D content as dynamic 3D Gaussians with a deformation field, initialize a static 3D object with MVDream plus SD, then optimize the deformation field using composed gradients from a frame-rate-conditioned text-to-video model and a text-to-image model, with JSD-based distribution regularization, motion amplification, and autoregressive extension.
Key results
- In a user study on 28 prompts, AYG was preferred over MAV3D for overall quality by 53.6% vs 38.8% (7.6% equal).
- Ablations show all components matter for overall quality; full AYG is strongly preferred over variants without initial 3D stage, JSD regularization, or image DM score in the 4D stage.
- AYG achieves R-Precision on par with MAV3D on a larger prompt set, though the authors note R-Precision is not meaningful for dynamic scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add rigidity regularization (ablation) | AYG · 30 text prompts | Overall Quality preference: 45.8/13.3 | Helped |
| add motion amplifier (ablation) | AYG · 30 text prompts | Overall Quality preference: 43.3/23.3 | Helped |
| add an initial 3D stage (ablation) | AYG · 30 text prompts | Overall Quality preference: 67.5/15.0 | Helped |
| add JSD-based regularization (ablation) | AYG · 30 text prompts | Overall Quality preference: 40.0/25.0 | Helped |
| add image DM score in the 4D stage (ablation) | AYG · 30 text prompts | Overall Quality preference: 42.5/22.5 | Helped |
| use CSD instead of SDS (ablation) | AYG · 30 text prompts | Overall Quality preference: 44.2/35.8 | Helped |
| use MVDream in the 3D stage (ablation) | AYG · 30 text prompts | Overall Quality preference: 66.7/21.7 | Helped |
| remove MVDream from the 4D stage (ablation) | AYG · 30 text prompts | Overall Quality preference: 50.8/27.5 | Helped |
| use multiple frame rates rather than only fps 4 (ablation) | AYG video model · 30 text prompts | Overall Quality preference: 46.7/15.8 | Helped |
| use multiple frame rates rather than only fps 12 (ablation) | AYG video model · 30 text prompts | Overall Quality preference: 48.3/29.2 | Helped |
| use dynamic cameras (ablation) | AYG · 30 text prompts | Overall Quality preference: 32.5/25.0 | Helped |
| use negative prompting (ablation) | AYG · 30 text prompts | Overall Quality preference: 44.2/28.3 | Helped |
Surprising findings
- The authors state R-Precision is a meaningless metric for evaluating dynamic scenes, so they rely mainly on user studies.
Limitations
- Cannot easily produce topological changes of dynamic objects
- Sequences are short by default; relies on autoregressive extension
- No background model; MAV3D uses an extra background model
Experimental setup
Datasets: HDVG-130M WebVid-10M Objaverse
Benchmarks: MAV3D project page prompts (28 prompts) ablation prompt set (30 prompts)
Metrics: user preference percentage R-Precision
Baselines: MAV3D
Builds on: DreamFusion 3D Gaussian Splatting Make-A-Video3D MVDream classifier score distillation
Closest papers by meaning
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- 4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling (CVPR 2024 · similarity 0.90)
- 4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.90)
- Fully Explicit Dynamic Gaussian Splatting (NeurIPS 2024 · similarity 0.89)
- Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle (CVPR 2024 · similarity 0.89)
- A Unified Approach for Text- and Image-guided 4D Scene Generation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.