Research map / 3D vision / 3D generation
Diffusion Time-step Curriculum for One Image to 3D Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTC123, a one-image-to-3D pipeline that applies a coarse-to-fine diffusion time-step curriculum to SDS, coordinating a time-annealed sampling schedule with progressive NeRF/DMTet representations and Zero-1-to-3 plus Stable Diffusion teacher guidance.
Problem
SDS-based single-image-to-3D generation suffers from geometric artifacts, collapsed geometry, Janus faces and texture saturation because diffusion time-steps are sampled uniformly, entangling coarse and fine modeling.
Approach
Anneal the diffusion time-step sampling interval from large to small over training, and coordinate it with student representation resolution (hash grid then tetrahedral grid) and teacher choice (Zero-1-to-3 for geometry, Stable Diffusion for texture) so that large time-steps teach coarse structure and small time-steps teach fine detail.
Key results
- On RealFusion15, DTC123 reaches CLIP-similarity 0.87, PSNR 21.42 and LPIPS 0.08, versus 0.82 / 19.50 / 0.10 for Magic123.
- On NeRF4, DTC123 reaches CLIP-similarity 0.84, PSNR 25.14 and LPIPS 0.02, versus 0.80 / 24.62 / 0.03 for Magic123.
- Failure analysis on Level50 shows DTC123 has consistently lower failure rates than RealFusion and Magic123 across random seeds and difficulty levels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DTC123 instead of Magic123 | Image-to-3D generation · NeRF4 | CLIP-Similarity: 0.80 -> 0.84 | Helped |
| Use DTC123 instead of Magic123 | Image-to-3D generation · NeRF4 | PSNR: 24.62 -> 25.14 | Helped |
| Use DTC123 instead of Magic123 | Image-to-3D generation · NeRF4 | LPIPS: 0.03 -> 0.02 | Helped |
| Use DTC123 instead of Magic123 | Image-to-3D generation · RealFusion15 | CLIP-Similarity: 0.82 -> 0.87 | Helped |
| Use DTC123 instead of Magic123 | Image-to-3D generation · RealFusion15 | PSNR: 19.50 -> 21.42 | Helped |
| Use DTC123 instead of Magic123 | Image-to-3D generation · RealFusion15 | LPIPS: 0.10 -> 0.08 | Helped |
| Add an annealed time-step sampling schedule (ablation) | DTC123 · Image-to-3D generation | generation quality | Helped |
| Add teacher-student collaboration with the annealed time-step (ablation) | DTC123 · Image-to-3D generation | generation quality | Helped |
Limitations
- Relies on pre-trained 2D teachers (Zero-1-to-3, Stable Diffusion) whose biases cause Janus face issues that need hand-designed azimuth-range corrections.
- Two-stage pipeline with NeRF at 64x64 before DMTet fine-tuning adds design complexity.
Experimental setup
Datasets: NeRF4 RealFusion15 GSO OmniObject3D Objaverse
Benchmarks: NeRF4 RealFusion15 GSO Level50
Metrics: CLIP-similarity PSNR LPIPS failure rate
Baselines: Zero123-SDS RealFusion NeuralLift Magic123 One-2-3-45 SyncDreamer
Builds on: score distillation sampling Zero-1-to-3 Magic123 DMTet Instant-NGP
Closest papers by meaning
- Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior (CVPR 2024 · similarity 0.90)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.89)
- DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation (ICLR 2024 · similarity 0.89)
- Score Distillation via Reparametrized DDIM (NeurIPS 2024 · similarity 0.89)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.88)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.