Research map / 3D vision / 3D generation
Rapid 3D Model Generation with Intuitive 3D Input
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Deep3DVRSketch generates high-quality 3D shapes from novice users' VR 3D sketches using a conditional SDF diffusion model pre-trained on rendered images, a sketch encoder mapped into its latent space, joint fine-tuning, and adaptive curriculum learning, and also releases the KO3D+ sketch-shape dataset.
Problem
Conventional CAD 3D modeling requires expertise and is hard for novices; existing 3D sketch-to-shape methods are deterministic regressions that reflect drawing imprecision and are limited to one category, with very little 3D sketch-shape training data.
Approach
Formulate 3D VR sketch-to-model as conditional generation: pre-train a diffusion generator on shapes conditioned on CLIP image embeddings, map 3D sketches (via a Uni3D point encoder) into the same CLIP latent space, jointly fine-tune, and use adaptive curriculum learning based on SDF difficulty to exploit scarce data.
Key results
- Single-category chair: FID 8.77 and CD 0.0220 versus 11.53 and 0.0305 for Luo et al.
- Multi-category KO3D+: FID 15.54 and CD 0.0304 versus 30.53 and 0.0455 for Luo et al.
- User study: designers modeled over 10 times faster (89.4 s vs 869.2 s with CAD) with about 7.3 s inference on A100
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Deep3DVRSketch instead of Luo et al. | Chair [39] · Chair [39] | FID: 11.5313 -> 8.7701 | Helped |
| Use Deep3DVRSketch instead of Luo et al. | Car · Car | CD: 0.0303 -> 0.0047 | Helped |
| Use Deep3DVRSketch instead of Luo et al. | KO3D+ multi-category · KO3D+ multi-category | FID: 30.5313 -> 15.5397 | Helped |
| Use Deep3DVRSketch instead of Luo et al. | User study · User study | Fidelity MOS: 2.02 ± 0.84 -> 4.06 ± 0.76 | Helped |
| Use Deep3DVRSketch instead of conventional CAD software | 3D modeling tasks · 3D modeling tasks | Time (s): 869.2 ± 410.2 -> 89.4 ± 29.6 | Helped |
| Add Stage 1 generative pre-training (ablation) | Deep3DVRSketch · Chair [39] | FID: 25.0112 -> 8.7701 | Helped |
| Add Stage 3 joint fine-tuning (ablation) | Deep3DVRSketch · Chair [39] | CD: 0.0256 -> 0.0220 | Helped |
| Use pretrained encoder at Stage 2 (ablation) | Deep3DVRSketch · Chair [39] | CD: 0.0405 -> 0.0220 | Helped |
| Add curriculum learning (ablation) | Deep3DVRSketch · Chair [39] | FID: 9.5470 -> 8.7701 | Helped |
Surprising findings
- Without joint fine-tuning, generated shapes keep high visual quality but stop matching the input sketch.
Limitations
- Trained on a single chair dataset or on limited sketch-shape pairs (about 600 per category), so data scarcity remains
- Evaluation on a small number of categories and sketches from novice users; user study with only 3 to 12 designers
Experimental setup
Datasets: ShapeNetCore-v2 Luo et al. chair sketch-shape dataset KO3D+
Benchmarks: Luo et al. chair sketch dataset KO3D+ car category KO3D+ multi-category
Metrics: Chamfer distance FID MOS user rating modeling time
Baselines: Luo et al. [40] LAS-Diffusion Deep3DSketch Deep3DSketch+ conventional CAD (ZBrush)
Builds on: Luo et al. normalizing-flow 3D sketch modeling LAS-Diffusion Uni3D Curriculum DeepSDF classifier-free guidance
Closest papers by meaning
- CraftsMan3D: High-fidelity Mesh Generation with 3D Native Diffusion and Interactive Geometry Refiner (CVPR 2025 · similarity 0.88)
- 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.88)
- CAD: Photorealistic 3D Generation via Adversarial Distillation (CVPR 2024 · similarity 0.87)
- Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior (CVPR 2024 · similarity 0.86)
- ViewCraft3D: High-fidelity and View-Consistent 3D Vector Graphics Synthesis (NeurIPS 2025 · similarity 0.86)
- En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.