Research map / 3D vision / Face and shape
Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sin3DM trains a triplane latent diffusion model with a small receptive field and triplane-aware convolutions on a single 3D textured shape to generate high-quality variations.
Problem
Generating novel 3D textured shapes resembling a single input example, where large 3D datasets are unavailable and training diffusion models directly in 3D is memory-expensive.
Approach
Compress the single textured shape into a triplane latent via an autoencoder, then train a limited-receptive-field 2D U-Net diffusion model with triplane-aware convolution blocks that predicts the clean latent instead of noise.
Key results
- Sin3DM average G-Qual. 0.156 vs SSG 0.238 and Sin3DGen 3.390 (lower is better).
- Sin3DM average T-Qual. 0.028 vs SSG 0.099 and Sin3DGen 0.036.
- Average G-Div. 0.224 for Sin3DM vs 0.209 for SSG and 0.261 for Sin3DGen; diversity comparable while quality is better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add triplane-aware convolution to the denoising network (ablation) | Sin3DM · 10 testing examples (Avg.) | G-Qual.: 0.347 -> 0.156 | Helped |
| add triplane-aware convolution to the denoising network (ablation) | Sin3DM · 10 testing examples (Avg.) | T-Qual.: 0.068 -> 0.028 | Helped |
| add a triplane encoder instead of fitting the latent with an auto-decoder (ablation) | Sin3DM · 10 testing examples (Avg.) | G-Qual.: 0.608 -> 0.156 | Helped |
| add a triplane encoder instead of fitting the latent with an auto-decoder (ablation) | Sin3DM · 10 testing examples (Avg.) | T-Qual.: 0.060 -> 0.028 | Helped |
| predict the clean latent instead of added noise ε (ablation) | Sin3DM · 10 testing examples (Avg.) | G-Qual.: 0.925 -> 0.156 | Helped |
| predict the clean latent instead of added noise ε (ablation) | Sin3DM · 10 testing examples (Avg.) | T-Qual.: 0.156 -> 0.028 | Helped |
| add triplane-aware convolution to the denoising network (ablation) | Sin3DM · 10 testing examples (Avg.) | G-Div.: 0.310 -> 0.224 | Mixed |
| add triplane-aware convolution to the denoising network (ablation) | Sin3DM · 10 testing examples (Avg.) | T-Div.: 0.171 -> 0.133 | Mixed |
| add a triplane encoder instead of fitting the latent with an auto-decoder (ablation) | Sin3DM · 10 testing examples (Avg.) | G-Div.: 0.355 -> 0.224 | Mixed |
| add a triplane encoder instead of fitting the latent with an auto-decoder (ablation) | Sin3DM · 10 testing examples (Avg.) | T-Div.: 0.179 -> 0.133 | Mixed |
| predict the clean latent instead of added noise ε (ablation) | Sin3DM · 10 testing examples (Avg.) | G-Div.: 0.402 -> 0.224 | Mixed |
| predict the clean latent instead of added noise ε (ablation) | Sin3DM · 10 testing examples (Avg.) | T-Div.: 0.187 -> 0.133 | Mixed |
Surprising findings
- Removing the triplane-aware convolution or encoder increases diversity but sharply reduces quality, showing a quality-diversity trade-off.
- Generated variations primarily occur along the three axis directions of the triplane representation.
Limitations
- Variations mainly occur along the three axis directions of the triplane.
- Quality-diversity trade-off controlled only by receptive field size.
- Inputs with strong semantic structure may not preserve semantics in generated variations.
Experimental setup
Datasets: single 3D textured shape per example (10 test examples across categories)
Benchmarks: 10 single-shape testing examples (acropolis, canyon, house, tree, etc.)
Metrics: G-Qual. (SSFID) G-Div. (1-IoU) T-Qual. (SIFID) T-Div. (LPIPS) LP-IoU LP-F-score
Baselines: SSG Sin3DGen w/o triplaneConv w/o encoder epsilon-prediction
Builds on: DDPM triplane representation SinGAN SSG DreamFusion-style triplane autoencoders
Closest papers by meaning
- Single Mesh Diffusion Models with Field Latents for Texture Generation (CVPR 2024 · similarity 0.91)
- GPLD3D: Latent Diffusion of 3D Shape Generative Models by Enforcing Geometric and Physical Priors (CVPR 2024 · similarity 0.89)
- Kiss3DGen: Repurposing Image Diffusion Models for 3D Asset Generation (CVPR 2025 · similarity 0.89)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.88)
- ShapeShifter: 3D Variations Using Multiscale and Sparse Point-Voxel Diffusion (CVPR 2025 · similarity 0.88)
- Generating Images with 3D Annotations Using Diffusion Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.