Research map / Image generation / Text-to-image diffusion
Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Phidias uses a retrieved or user-provided 3D reference to guide a multi-view diffusion model (via meta-ControlNet, dynamic reference routing, and self-reference augmentation) followed by sparse-view LGM reconstruction for reference-augmented 3D generation.
Problem
Single-image and text-to-3D generation suffers from ill-posed ambiguity, poor generalization to out-of-domain inputs, and limited controllability; existing ControlNet-style conditioning fails when 3D references are misaligned with the concept image.
Approach
Treat a retrieved or user-given 3D model as auxiliary reference guidance (as CCM maps) for a multi-view diffusion model, using a meta-ControlNet that modulates conditioning strength by image-reference similarity, a timestep-dependent reference resolution schedule, and self-reference augmentations with curriculum training to handle misalignment.
Key results
- On GSO, Ours (GT Ref.) reaches PSNR 20.37 and LPIPS 0.117 vs. best baseline CRM 16.35 PSNR / 0.182 LPIPS.
- With retrieved reference, Phidias achieves PSNR 17.02, CLIP-P 0.887 and F-Score 0.833, outperforming all five baselines on visual metrics.
- Top-1 retrieved references outperform top-5 and random references (PSNR 17.02 vs 15.96 vs 14.74), and the model still produces plausible results with random references.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a more similar top-1 rather than top-5 retrieved reference (ablation) | Phidias · GSO | PSNR: 15.96 -> 17.02 | Helped |
| use a more similar top-3 rather than top-5 retrieved reference (ablation) | Phidias · GSO | PSNR: 15.96 -> 16.75 | Helped |
| use a random reference instead of no reference (ablation) | Phidias · GSO | PSNR: 15.90 -> 14.74 | Hurt |
| add meta-ControlNet for adaptive reference conditioning (ablation) | Phidias · image-to-3D generation | reference utilization | Helped |
| add dynamic reference routing (ablation) | Phidias · image-to-3D generation | concept-image detail fidelity | Helped |
| add self-reference augmentation (ablation) | Phidias · image-to-3D generation | concept-image fidelity | Helped |
Surprising findings
- Using a random 3D reference yields lower quality than no reference at all on several metrics (PSNR 14.74 vs 15.90 without reference).
Limitations
- Performance depends on retrieval quality and the small 40K-object retrieval database.
- Multi-view diffusion operates at a limited 320x320 resolution, restricting high-resolution inputs.
- Retrieved references with misaligned pose or structure can produce undesired outputs.
Experimental setup
Datasets: Objaverse Google Scanned Objects
Benchmarks: Google Scanned Objects
Metrics: PSNR SSIM LPIPS CLIP-P CLIP-I Chamfer Distance F-Score user preference
Baselines: OpenLRM LGM CRM SV3D InstantMesh
Builds on: Zero123++ ControlNet LGM Uni3D
Closest papers by meaning
- PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion (CVPR 2024 · similarity 0.88)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.88)
- MVDream: Multi-view Diffusion for 3D Generation (ICLR 2024 · similarity 0.87)
- EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Priors (CVPR 2024 · similarity 0.87)
- Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation (CVPR 2025 · similarity 0.87)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.