Research map / 3D vision / Face and shape
TextureDreamer: Image-Guided Texture Synthesis Through Geometry-Aware Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Transfers relightable BRDF textures from 3-5 casually captured images onto arbitrary 3D meshes by personalizing a diffusion model with DreamBooth and optimizing a neural BRDF field with ControlNet-normal-conditioned variational score distillation.
Problem
Texture creation for 3D shapes usually needs dense views with aligned geometry or category-specific learned models, and text-only guidance cannot express complex image textures. The paper targets image-guided texture transfer from very sparse views to arbitrary-category target meshes.
Approach
Personalized geometry-aware score distillation (PGSD): a DreamBooth-finetuned diffusion model with a pre-trained normal-map ControlNet is used as the denoiser for variational score distillation (without LoRA and without classifier-free guidance) to optimize a neural BRDF field on the target mesh.
Key results
- CLIP similarity between reference images and renders: Ours 0.8296 vs TEXTure 0.7988 and Latent-Paint 0.7969.
- Amazon Turk user study: Ours preferred over TEXTure for texture photorealism 85.52% and shape-texture consistency 85.16% (TEXTure); over Latent-Paint 77.03% and 78.49% respectively.
- Ablations show normal-map ControlNet gives the best texture-geometry consistency, and removing LoRA preserves more input texture than keeping it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TextureDreamer instead of Latent-Paint | Image-guided texture transfer · Image-guided texture transfer | CLIP similarity: 0.7969 -> 0.8296 | Helped |
| Use TextureDreamer instead of TEXTure | Image-guided texture transfer · Image-guided texture transfer | CLIP similarity: 0.7988 -> 0.8296 | Helped |
| Use normal-conditioned ControlNet (ablation) | TextureDreamer · Image-guided texture transfer | shape-texture consistency | Helped |
| Remove ControlNet conditioning (ablation) | TextureDreamer · Image-guided texture transfer | shape-texture consistency | Hurt |
| Replace normal-conditioned ControlNet with depth-conditioned ControlNet (ablation) | TextureDreamer · Image-guided texture transfer | shape-texture consistency | Hurt |
| Replace VSD with SDS (ablation) | TextureDreamer · Image-guided texture transfer | texture fidelity | Hurt |
| Keep the LoRA module instead of removing it (ablation) | TextureDreamer · Image-guided texture transfer | texture fidelity | Hurt |
| Use one input image instead of multiple views (ablation) | TextureDreamer · Image-guided texture transfer | texture quality | Hurt |
| Use 11 input images instead of 5 views (ablation) | TextureDreamer · Image-guided texture transfer | texture quality | No clear effect |
| Jointly fine-tune the text encoder (ablation) | Dreambooth texture extraction in TextureDreamer · Texture information extraction | texture quality | No clear effect |
| Replace the diffusion denoising network with a pre-trained ControlNet during texture extraction (ablation) | Dreambooth texture extraction in TextureDreamer · Texture information extraction | texture quality | No clear effect |
Surprising findings
- The method does not benefit from classifier-free guidance, since the personalized model is fine-tuned on few images and the goal is faithful transfer rather than diversity.
- Initializing the score-distillation denoiser with the DreamBooth weights removes texture details, so the generic pre-trained weights are used instead.
Limitations
- Bakes lighting and strong specular highlights from input images into the texture.
- Janus (multi-face) artifacts when input views do not cover the whole object.
- Fails to transfer special, non-repeated patterns from the input.
Experimental setup
Datasets: 3D-FUTURE Online 3D repositories (CGTrader, Sketchfab) Casually captured image sets (32 sets across sofa, bed, mug/bowl, plush toy)
Benchmarks: Custom image-guided texture transfer evaluation set (32 image sets, 3 meshes per category)
Metrics: CLIP similarity user preference study
Baselines: Latent-Paint TEXTure
Builds on: DreamBooth ControlNet ProlificDreamer (variational score distillation) DreamFusion (score distillation sampling) Fantasia3D
Closest papers by meaning
- Wukong's 72 Transformations: High-fidelity Textured 3D Morphing via Flow Models (NeurIPS 2025 · similarity 0.87)
- TriTex: Learning Texture from a Single Mesh via Triplane Semantic Features (CVPR 2025 · similarity 0.87)
- TexTailor: Customized Text-aligned Texturing via Effective Resampling (ICLR 2025 · similarity 0.87)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.87)
- TUVF: Learning Generalizable Texture UV Radiance Fields (ICLR 2024 · similarity 0.86)
- Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.