Research map / 3D vision / Face and shape
Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Diff3F distills Stable Diffusion (ControlNet-guided) and DINOv2 features from multi-view depth/normal-conditioned renders onto untextured 3D shapes to get zero-shot semantic point descriptors for correspondence.
Problem
Extracting reliable, semantic per-point feature descriptors from untextured 3D shapes (meshes or point clouds) without 3D training data or per-shape optimization, for shape correspondence.
Approach
Render untextured shapes from many views, use depth and normal maps as ControlNet conditions to synthesize textured images, take intermediate Stable Diffusion UNet features over diffusion timesteps, fuse with DINOv2 features, and unproject and average them across views onto the surface; inconsistent textures are tolerated because the features are robust under aggregation.
Key results
- Achieves 26.41% accuracy at 1% tolerance on SHREC'19 versus 21.41% for SE-ORNet, with error 1.69.
- On SHREC'20 reaches 72.60% accuracy at 1% tolerance and 0.93 error, far above DPC (31.08%) and SE-ORNet (31.70%).
- On TOSCA gets 20.27% accuracy, below SE-ORNet's 33.25%, so it is comparable rather than best on isometric animals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ControlNet texturing (ablation) | DIFF3F · SHREC’19 | acc: 17.20 -> 26.41 | Helped |
| Add ControlNet texturing (ablation) | DIFF3F · SHREC’20 | acc: 65.48 -> 72.60 | Helped |
| Replace TEXTure+DINO with ControlNet texturing and diffusion features (ablation) | DIFF3F · SHREC’19 | acc: 17.20 -> 26.41 | Helped |
| Replace TEXTure+DINO with ControlNet texturing and diffusion features (ablation) | DIFF3F · SHREC’20 | acc: 65.48 -> 72.60 | Helped |
| Add DINO feature fusion (ablation) | DIFF3F · SHREC’19 | acc: 26.53 -> 26.41 | Hurt |
| Add DINO feature fusion (ablation) | DIFF3F · SHREC’20 | acc: 64.89 -> 72.60 | Helped |
| Add normal-map conditioning (ablation) | DIFF3F · SHREC’19 | acc: 25.68 -> 26.41 | Helped |
| Add normal-map conditioning (ablation) | DIFF3F · SHREC’20 | acc: 69.71 -> 72.60 | Helped |
| Add diffusion time-step feature aggregation (ablation) | DIFF3F · SHREC’19 | acc: 25.73 -> 26.41 | Helped |
| Add diffusion time-step feature aggregation (ablation) | DIFF3F · SHREC’20 | acc: 68.95 -> 72.60 | Helped |
| Add ball-query local feature sharing (ablation) | DIFF3F · SHREC’19 | acc: 25.72 -> 26.41 | Helped |
| Add ball-query local feature sharing (ablation) | DIFF3F · SHREC’20 | acc: 74.10 -> 72.60 | Hurt |
Surprising findings
- Multi-view image generations are inconsistent across views, yet aggregating their features still yields robust semantic descriptors.
- Functional maps with Diff3F descriptors work on non-isometric animal pairs where WKS-based functional maps struggle.
Limitations
- Fails to produce features on parts invisible from all sampled views (self-occlusion).
- Inherits dataset and view biases of image diffusion models, e.g., weaker features on less-seen regions like a horse's belly.
- Lower accuracy than SE-ORNet on TOSCA isometric animal pairs.
Experimental setup
Datasets: SHREC'19 SHREC'20 FAUST TOSCA
Benchmarks: SHREC'19 SHREC'20 FAUST TOSCA
Metrics: correspondence accuracy at 1% tolerance average correspondence error average geodesic error
Baselines: DPC SE-ORNet 3D-CODED FM+WKS TEXTure+DINO
Builds on: ControlNet Stable Diffusion DINOv2 Functional Maps TEXTure Wave Kernel Signature
Closest papers by meaning
- Large-Vocabulary 3D Diffusion Model with Transformer (ICLR 2024 · similarity 0.89)
- Geometry Image Diffusion: Fast and Data-Efficient Text-to-3D with Image-Based Surface Representation (ICLR 2025 · similarity 0.88)
- Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape (ICLR 2024 · similarity 0.88)
- DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis (CVPR 2024 · similarity 0.87)
- DiffCorr: Conditional Diffusion Model with Reliable Pseudo-Label Guidance for Unsupervised Point Cloud Shape Correspondence (AAAI 2025 · similarity 0.87)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.