Research map / 3D vision / Face and shape
Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Paint3D generates lighting-less 2K UV texture maps for untextured meshes from text or image prompts via a coarse multi-view depth-aware 2D diffusion stage followed by UV-space inpainting and UVHD refinement diffusion models.
Problem
Generating high-quality, diverse 3D mesh textures from text or image conditions that are free of baked-in illumination, so they can be relit in standard graphics pipelines, while keeping view consistency and completeness.
Approach
Two-stage pipeline: progressive multi-view depth-aware 2D diffusion sampling and back-projection yields a coarse UV texture, then a position-map-conditioned ControlNet-style UV diffusion model (trained on illumination-free Objaverse UV textures) inpaints holes and removes lighting, producing lighting-less textures.
Key results
- Text-to-texture FID 27.28 and KID 4.81 versus Text2Tex 38.93 and 7.94 (user study overall quality 4.45 vs 3.57).
- Image-to-texture FID 26.86 vs TEXTure 40.83; KID 4.94 vs 9.76.
- Ablation: removing the refinement stage or UV inpainting/UVHD raises FID to 33.42 to 48.81 from 27.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Paint3D instead of Text2Tex | Text-to-texture · Text-to-texture | FID: 38.93 -> 27.28 | Helped |
| Use Paint3D instead of TEXTure | Image-to-texture · Image-to-texture | FID: 40.83 -> 26.86 | Helped |
| Add the coarse stage before UV refinement (ablation) | Paint3D · Text-to-texture | FID: 48.81 -> 27.28 | Helped |
| Add the refinement stage after the coarse stage (ablation) | Paint3D · Text-to-texture | FID: 41.84 -> 27.28 | Helped |
| Add UV inpainting instead of using bilinear interpolation (ablation) | Paint3D · Text-to-texture | FID: 33.42 -> 27.28 | Helped |
| Add UVHD (ablation) | Paint3D · Text-to-texture | FID: 37.84 -> 27.28 | Helped |
| Increase total viewpoints from 2 to 6 (ablation) | Paint3D coarse stage · Text-to-texture | FID: 41.74 -> 27.28 | Helped |
| Increase total viewpoints from 6 to 8 (ablation) | Paint3D coarse stage · Text-to-texture | FID: 27.28 -> 27.71 | Hurt |
Surprising findings
- Increasing the number of coarse-stage viewpoints beyond six did not help (8 views gave FID 30.15 vs 29.02 at 6 for single-iteration), since the 2D model introduces illumination artifacts.
Limitations
- Multi-face inconsistency in the coarse stage because the 2D diffusion model is not trained on multi-view data.
- Does not generate material maps (PBR), so physically based material support is missing.
- Cannot generate or edit mesh geometry.
Experimental setup
Datasets: Objaverse
Benchmarks: Objaverse held-out textured meshes (301) plus 30 in-the-wild meshes
Metrics: FID KID user study overall quality user study text fidelity user study image fidelity
Baselines: Latent-Paint TEXTure Text2Tex
Builds on: Stable Diffusion v1.5 ControlNet IP-Adapter TEXTure
Closest papers by meaning
- Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering (CVPR 2024 · similarity 0.89)
- UV-free Texture Generation with Denoising and Geodesic Heat Diffusion (NeurIPS 2024 · similarity 0.88)
- One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion (CVPR 2024 · similarity 0.88)
- DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior (ICLR 2024 · similarity 0.87)
- Single Mesh Diffusion Models with Field Latents for Texture Generation (CVPR 2024 · similarity 0.87)
- Wonder3D: Single Image to 3D using Cross-Domain Diffusion (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.