Research map / 3D vision / Face and shape
Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Paint-it synthesizes PBR texture maps for 3D meshes from text by optimizing a randomly initialized convolutional U-Net (DC-PBR re-parameterization) with Score-Distillation Sampling, yielding frequency-scheduled, noise-robust texture optimization.
Problem
Text-driven texture map synthesis for 3D meshes via SDS suffers from noisy gradients and poor texture quality when optimizing pixel-based texture maps, and prior methods are limited to RGB diffuse textures.
Approach
Re-parameterize PBR texture maps (diffuse, roughness-metalness, normal) as the output of a randomly initialized convolutional U-Net with a fixed noise input, so that the convolutional prior imposes coarse-to-fine frequency scheduling that filters noisy SDS gradients.
Key results
- Paint-it achieves FID 34.46 versus 37.89 for Text2Tex and 38.40 for TEXTure on an Objaverse subset.
- User realism score of 4.37 for Paint-it versus 3.34 for the next best (Text2Tex), with 30 users.
- Paint-it produces PBR texture maps in about 15-30 minutes per mesh.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Paint-it with DC-PBR instead of the competing texture-synthesis methods | Objaverse subset · Objaverse subset | FID: 34.46 vs 37.89 | Helped |
| Use Paint-it with DC-PBR instead of the competing texture-synthesis methods | Objaverse subset · Objaverse subset | user score: 4.37 vs 3.34 | Helped |
| Add roughness, metalness, and normal maps to diffuse-only texture optimization (ablation) | Paint-it · Text-driven mesh texture synthesis | visual realism | Helped |
| Replace pixel-based PBR optimization with DC-PBR neural re-parameterization (ablation) | Paint-it · Text-driven mesh texture synthesis | texture quality | Helped |
Surprising findings
- Pixel-based SDS optimization with a total variation regularizer still produces severely jittered, degraded textures, while the neural re-parameterization filters high-frequency noise without explicit smoothing losses.
Limitations
- Optimization time of 15-30 minutes per mesh
- Per-mesh optimization without a feed-forward model
Experimental setup
Datasets: Objaverse RenderPeople SMAL
Benchmarks: Objaverse subset (FID and user study)
Metrics: FID user score (realism)
Baselines: Latent-Paint Fantasia3D Text2Tex TEXTure Pixel optimization with SDS
Builds on: Score-Distillation Sampling (DreamFusion) Cook-Torrance microfacet BRDF NVDiffRast
Closest papers by meaning
- PaintHuman: Towards High-Fidelity Text-to-3D Human Texturing via Denoised Score Distillation (AAAI 2024 · similarity 0.89)
- Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models (CVPR 2024 · similarity 0.89)
- TexTailor: Customized Text-aligned Texturing via Effective Resampling (ICLR 2025 · similarity 0.88)
- 3D Paintbrush: Local Stylization of 3D Shapes with Cascaded Score Distillation (CVPR 2024 · similarity 0.87)
- IntrinsiX: High-Quality PBR Generation using Image Priors (NeurIPS 2025 · similarity 0.87)
- HIFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.