Research map / 3D vision / 3D generation
Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diff3DS optimizes 3D rational Bézier curve strokes through perspective projection and a depth-aware DiffVG-based rasterizer, using SDS-style distillation from 2D/multi-view/image-conditioned diffusion models to generate view-consistent 3D sketches from text or a single image.
Problem
Generating view-consistent 3D sketches from flexible inputs such as text or a single image, since prior 3D sketch methods require interactive artistry or multi-view images, and DiffVG-based 3D curve rendering ignores depth ordering and uses orthographic approximation.
Approach
Represent 3D sketches as 3D rational Bézier curves, project them perspectively to 2D rational Béziers, and rasterize with a DiffVG-based rasterizer that sorts overlapping points by per-point depth; then optimize with Score Distillation Sampling from Stable Diffusion/MVDream or Zero-1-to-3/Stable-Zero123, with Dynamic Noise Deletion and time annealing.
Key results
- Text-to-3D sketch: Diff3DS reaches CLIP-Score T 0.3046 (ViT B/16) vs 0.2763 for MVDream and 0.2561 for DreamGaussian.
- Image-to-3D sketch: Diff3DS has the highest novel-view CLIP-Score I (0.6768 B/16, 0.6846 B/32) and LPIPS 0.2647, close to 3Doodle's 0.2567.
- Training takes about 1 hour (text-to-3D) and 2 hours (image-to-3D) on a single NVIDIA A10 GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Diff3DS instead of the strongest text-to-3D baseline | Text-to-3D sketch generation · Text-to-3D sketch | CLIP-Score^T (ViT B/16): 0.2763 -> 0.3046 | Helped |
| Use Diff3DS instead of the strongest text-to-3D baseline | Text-to-3D sketch generation · Text-to-3D sketch | CLIP-Score^T (ViT B/32): 0.2653 -> 0.3034 | Helped |
| Use Diff3DS instead of the strongest text-to-3D baseline | Text-to-3D sketch generation · Text-to-3D sketch | BLIP-Score: 0.4990 -> 0.5038 | Helped |
| Use Diff3DS instead of the strongest image-to-3D baseline | Image-to-3D sketch generation · Image-to-3D sketch, novel views | CLIP-Score^I (ViT B/16): 0.6734 -> 0.6768 | Helped |
| Use Diff3DS instead of the strongest image-to-3D baseline | Image-to-3D sketch generation · Image-to-3D sketch, novel views | CLIP-Score^I (ViT B/32): 0.6746 -> 0.6846 | Helped |
| Use Diff3DS instead of 3Doodle | Image-to-3D sketch generation · Image-to-3D sketch, reference view | LPIPS: 0.2567 -> 0.2647 | Hurt |
| Increase initial curve number from 56 to 112 (ablation) | Diff3DS · Text-to-3D sketch | CLIP-Score^T (ViT B/16): 0.3046 -> 0.3074 | Helped |
| Increase initial curve number from 56 to 112 (ablation) | Diff3DS · Text-to-3D sketch | CLIP-Score^T (ViT B/32): 0.3034 -> 0.3032 | No clear effect |
| Increase initial curve number from 56 to 112 (ablation) | Diff3DS · Text-to-3D sketch | Average Training Time: 60min -> 80min | Hurt |
| Add the Time Annealing Schedule (ablation) | Diff3DS · Text-to-3D sketch and image-to-3D sketch | semantic conformity and detail | Helped |
| Add Dynamic Noise Deletion (ablation) | Diff3DS · 3D sketch generation | noise in generated sketches | Helped |
| Use MVDream instead of Stable Diffusion (ablation) | Diff3DS text-to-3D sketch generation · Text-to-3D sketch | view consistency | Helped |
Surprising findings
- Optimizing curve width yields unstable outcomes, as the rendered image is highly sensitive to width changes.
- Increasing the initial curve count beyond 56 to 112 gives no significant CLIP-score gain, while training time rises to 80 minutes.
Limitations
- Inherits sparse gradient issue from DiffVG and can only optimize continuous parameters.
- Initial curve number is set manually to balance accuracy and complexity.
- Does not distinguish view-independent feature lines from view-dependent contour curves.
Experimental setup
Datasets: NeRF Synthetic (hotdog, ship, lego) 3Doodle dataset (toyhorse, toycar)
Benchmarks: 35 text prompts 25 Imagen-generated images
Metrics: CLIP-Score T (ViT B/16, B/32) BLIP-Score CLIP-Score I LPIPS user study Likert 1-5
Baselines: DreamGaussian MVDream DiffSketcher NEF 3Doodle CLIPasso
Builds on: DiffVG 3Doodle DreamFusion MVDream
Closest papers by meaning
- ViewCraft3D: High-fidelity and View-Consistent 3D Vector Graphics Synthesis (NeurIPS 2025 · similarity 0.86)
- Consistent Flow Distillation for Text-to-3D Generation (ICLR 2025 · similarity 0.85)
- Consistent3D: Towards Consistent High-Fidelity Text-to-3D Generation with Deterministic Sampling Prior (CVPR 2024 · similarity 0.85)
- CraftsMan3D: High-fidelity Mesh Generation with 3D Native Diffusion and Interactive Geometry Refiner (CVPR 2025 · similarity 0.85)
- Rapid 3D Model Generation with Intuitive 3D Input (CVPR 2024 · similarity 0.85)
- Interactive3D: Create What You Want by Interactive 3D Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.