Research map / Image generation / Fast diffusion sampling
DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DI-PCG, a lightweight 7.6M-parameter diffusion transformer that treats procedural generator parameters as denoising targets conditioned on DINOv2 image features, enabling fast inverse PCG for 3D asset creation.
Problem
Inverse procedural content generation: finding procedural generator parameters that reproduce a desired 3D shape from an input condition image, which sampling-based (MCMC) methods solve slowly and prior neural methods solve with limited controllability and generalization.
Approach
Treat the canonicalized procedural generator parameters as the denoising target of a small diffusion transformer, conditioned on frozen DINOv2 patch features via cross-attention, so the posterior over parameters given an image is sampled in seconds; trained self-contained on generator-synthesized image-parameter pairs.
Key results
- On the DI-PCG chair test split, DI-PCG achieves CD 0.033, EMD 0.028, F-Score 0.896, far above InstantMesh (CD 0.098, F-Score 0.416) and 3DShape2VecSet (CD 0.093).
- On ShapeNet chairs, DI-PCG reaches CD 0.093 and F-Score 0.452, comparable to or better than prior SOTA such as InstantMesh (CD 0.095, F-Score 0.473 is higher, so comparable overall).
- DI-PCG samples parameters in about 5 seconds, whereas the MCMC baseline needs roughly 167 minutes for 10k iterations and still gives worse results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with multi-view images and augmentations instead of single-view images without augmentations (ablation) | DI-PCG · ShapeNet chairs subset | F-Score: 0.321 -> 0.452 | Helped |
| Use DINOv2 instead of CLIP condition features (ablation) | DI-PCG · ShapeNet chairs subset | F-Score: 0.288 -> 0.452 | Helped |
| Use the full-size model instead of the 1.6M-parameter model (ablation) | DI-PCG · ShapeNet chairs subset | F-Score: 0.423 -> 0.452 | Helped |
| Increase model size to 39M parameters (ablation) | DI-PCG · ShapeNet chairs subset | F-Score: 0.452 -> 0.452 | No clear effect |
Surprising findings
- A larger 39M-parameter diffusion model does not improve results over the 7.6M model, indicating the small model suffices.
- On the ShapeNet chair subset, DI-PCG's F-Score is slightly below InstantMesh and 3DShape2VecSet's ShapeNet-trained variant in some columns, so it is not uniformly best.
Limitations
- Generation scope is bounded by the off-the-shelf procedural generators and cannot produce out-of-domain objects.
- Only image (and sketch) conditions are supported; text conditions are not.
- Applicability to scene-level procedural generation is not verified.
Experimental setup
Datasets: Infinigen procedural generators (synthetic image-parameter pairs) Infinigen Indoors generators ShapeNet chairs (test)
Benchmarks: DI-PCG test split (generated chairs) ShapeNet chairs subset (218 models)
Metrics: Chamfer Distance Earth Mover's Distance F-Score
Baselines: Shap-E SDFusion Michelangelo CraftsMan InstantMesh IM-NET LION 3DShape2VecSet MCMC CLIP-conditioned DI-PCG
Builds on: DiT DINOv2 Infinigen Metropolis-Hastings MCMC
Closest papers by meaning
- Cycle3D: High-quality and Consistent Image-to-3D Generation via Generation-Reconstruction Cycle (AAAI 2025 · similarity 0.87)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.85)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.85)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.85)
- Diffusion$^2$: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models (ICLR 2025 · similarity 0.85)
- Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.