Research map / 3D vision / 3D generation
Material Palette: Extraction of Materials from a Single Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extracts tileable PBR materials from regions of a single real-world image by finetuning Stable Diffusion with a concept token and decomposing the generated textures into SVBRDF maps via unsupervised domain adaptation on a new diffusion-generated dataset, TexSD.
Problem
Extracting spatially varying BRDF (PBR) materials from a single in-the-wild real-world image, without lighting, geometry, or capture-apparatus priors, is highly ill-posed and shifted from existing synthetic SVBRDF datasets.
Approach
Map an image region to a learned concept token via LoRA DreamBooth finetuning of Stable Diffusion to generate flat tileable texture images, then decompose those textures into albedo/normal/roughness using a network adapted from synthetic SVBRDF libraries with pseudo-label unsupervised domain adaptation on diffusion-generated textures.
Key results
- Domain adaptation (ACG to PH) improves over the source-only model with relative gain of +5.87% over {A,N,R} on in-distribution PolyHaven materials.
- Material extraction using OpenSurfaces masks gives LPIPS of 0.7959/0.5730/0.7142 (A/N/R), beating the class-random upper bound (0.8288/0.5915/0.7255).
- In end-to-end re-rendering, extracted materials achieve LPIPS of 0.6839/0.6254/0.5951 versus 0.8163/0.6895/0.7441 for random Stable Diffusion generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt the ACG-trained decomposition model using TexSD pseudo-labels | Material Palette SVBRDF decomposition · PolyHaven ID | relative performance over ACG-only (%): +5.87 | Helped |
| Adapt the ACG-trained decomposition model using TexSD pseudo-labels | Material Palette SVBRDF decomposition · CGBookcase ID | relative performance over ACG-only (%): +2.62 | Helped |
| Adapt the ACG-trained decomposition model using TexSD pseudo-labels | Material Palette SVBRDF decomposition · PolyHaven OOD | relative performance over ACG-only (%): +5.24 | Helped |
| Use “realistic Sc texture in top view” instead of generic generation prompts (ablation) | Stable Diffusion material texture generation · ACG material generation | FID: 1.89 -> 1.55 (albedo); 1.74 -> 1.27 (normal); 1.83 -> 1.59 (roughness) | Helped |
| Add “texture” to the generation prompt (ablation) | Stable Diffusion material texture generation · ACG material generation | FID: 1.89 -> 1.72 (albedo); 1.74 -> 1.54 (normal); 1.83 -> 1.63 (roughness) | Helped |
| Use “high resolution” in the generation prompt (ablation) | Stable Diffusion material texture generation · ACG material generation | FID: 1.55 -> 1.55 (albedo); 1.27 -> 1.30 (normal); 1.59 -> 1.55 (roughness) | No clear effect |
Surprising findings
- Plain text-to-image generation from class names yields materials much further from ground truth than the proposed extraction pipeline, while prior patch/region texture extraction methods leak shading and geometry.
- The method struggles more with simple uniform materials, where the learned concept collapses into color artifacts.
Limitations
- Struggles with simple uniform materials, where the concept collapses causing color artifacts.
- Illumination ambiguities cause inconsistent colors on shaded surfaces, and slanted surfaces or strong distortion are not rectified.
- Evaluation of end-to-end extraction lacks a real dataset with region-wise SVBRDF ground truth, relying on proxy metrics.
Experimental setup
Datasets: AmbientCG PolyHaven CGBookcase OpenSurfaces TexSD
Benchmarks: AmbientCG PolyHaven CGBookcase OpenSurfaces
Metrics: MSE SSIM LPIPS FID KID CLIP zero-shot classification accuracy
Baselines: Deep Materials SurfaceNet Source-only (ACG) Li et al. texture extraction DeepTex PSGAN Quilting Materialistic SAM
Builds on: DreamBooth Stable Diffusion ControlMat Deep Materials
Closest papers by meaning
- Material Anything: Generating Materials for Any 3D Object via Diffusion (CVPR 2025 · similarity 0.90)
- TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting (CVPR 2025 · similarity 0.89)
- Eclipse: Disambiguating Illumination and Materials using Unintended Shadows (CVPR 2024 · similarity 0.89)
- Intrinsic Image Diffusion for Indoor Single-view Material Estimation (CVPR 2024 · similarity 0.89)
- MatSynth: A Modern PBR Materials Dataset (CVPR 2024 · similarity 0.88)
- MaterialRefGS: Reflective Gaussian Splatting with Multi-view Consistent Material Inference (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.