Research map / 3D vision / Face and shape
Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates precise 3D chair shapes from abstract freehand sketches via a part-aligned latent diffusion model over a pre-trained auto-decoder, trained on synthetic CLIPasso edgemaps without human sketch-3D pairs.
Problem
Sketch-to-3D generation degrades with poor drawing skill because prior methods map sketches to shapes deterministically, so inaccurate sketches yield deformed shapes; paired human sketch-3D data is also scarce.
Approach
Decompose shapes into aligned part latents (Wasserstein-matched part indices over GMM part-volumes) inside an implicit decoder, train a latent diffusion model on these aligned part latents, and map sketches to part-disentangled encodings supervised by projected 3D part segmentation maps on synthetic CLIPasso sketches, enabling generalisation to doodles and localized sketch edits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the part-aligned latent diffusion method rather than SENS | sketch-conditioned shape generation · AmateurSketch-3D | CD: 0.121 -> 0.109 | Helped |
| use the part-aligned latent diffusion method rather than SENS | sketch-conditioned shape generation · AmateurSketch-3D | EMD: 0.096 -> 0.091 | Helped |
| use the part-aligned latent diffusion method rather than LAS-D | sketch-conditioned shape generation · ProSketch-3D | CD: 0.195 -> 0.093 | Helped |
| use the part-aligned latent diffusion method rather than LAS-D | sketch-conditioned shape generation · ProSketch-3D | EMD: 0.147 -> 0.087 | Helped |
| aggregate sketch representations from multiple views | Ours · AmateurSketch-3D | CD: 0.0109 -> 0.0097 | Helped |
| aggregate sketch representations from multiple views | Ours · ProSketch-3D | CD: 0.0109 -> 0.0097 | Helped |
| remove latent alignment (ablation) | Ours · AmateurSketch-3D | CD: 0.109 -> 0.171 | Hurt |
| add latent alignment (ablation) | Ours without alignment · AmateurSketch-3D | CD: 0.171 -> 0.109 | Helped |
| replace part-aware sketch encodings with naive ImageNet features (ablation) | Ours · AmateurSketch-3D | CD: 0.109 -> 0.162 | Hurt |
| add part-disentangled sketch encodings (ablation) | Ours with naive ImageNet features · AmateurSketch-3D | CD: 0.162 -> 0.109 | Helped |
| use 1 multi-head attention layer instead of the full configuration (ablation) | Ours · AmateurSketch-3D | CD: 0.109 -> 0.127 | Hurt |
| use 2 multi-head attention layers instead of the full configuration (ablation) | Ours · AmateurSketch-3D | CD: 0.109 -> 0.114 | Hurt |
Surprising findings
- Part-aware sketch encodings improve generalisation to human doodles more than to synthetic sketches (+0.0053 CD vs +0.0031 without them).
Limitations
- Evaluated only on the chair category.
- Single-view sketches are ambiguous, causing shape variance that requires multi-view input to reduce.
- Unconditional generation performs only at par with prior diffusion methods rather than clearly better.
Experimental setup
Datasets: ShapeNet chairs (6755 shapes) CLIPasso synthetic sketches AmateurSketch-3D ProSketch-3D Quick-Draw!
Benchmarks: AmateurSketch-3D ProSketch-3D ShapeNet chairs unconditional generation Quick-Draw!
Metrics: Chamfer Distance Earth Mover's Distance FID COV MMD 1-NNA
Baselines: LAS-D SENS Sketch2Model SketchSampler Sketch2Mesh SDFusion SDF-StyleGAN Diffusion-SDF Mesh-Diffusion Wavelet-Diffusion
Builds on: LAS-D Diffusion-SDF CLIPasso DeepSDF-style auto-decoder (Chen et al. part-aware implicit decoder / ImAuto-dec DDPM
Closest papers by meaning
- Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering (ICLR 2025 · similarity 0.88)
- PartGen: Part-level 3D Generation and Reconstruction with Multi-view Diffusion Models (CVPR 2025 · similarity 0.86)
- Image Sculpting: Precise Object Editing with 3D Geometry Control (CVPR 2024 · similarity 0.85)
- Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.85)
- Rapid 3D Model Generation with Intuitive 3D Input (CVPR 2024 · similarity 0.85)
- MeshArt: Generating Articulated Meshes with Structure-Guided Transformers (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.