Research map / Image generation / Text-to-image diffusion
IntrinsiX: High-Quality PBR Generation using Image Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates PBR maps (albedo, roughness, metallic, normal) directly from text by adapting FLUX.1-dev with per-property LoRAs, cross-intrinsic attention, and a differentiable rendering loss.
Problem
Text-to-image models output RGB images with baked-in lighting, so they cannot yield PBR maps usable in graphics pipelines; existing intrinsic decomposition methods need an input RGB image and generalize poorly out of distribution.
Approach
Turn a pretrained text-to-image diffusion transformer into a direct PBR map generator: train separate LoRAs per intrinsic property on small curated datasets, then jointly fine-tune them with cross-intrinsic attention (concatenated keys/values across modalities, with dropout to self-attention) and a rendering loss using importance-based light sampling on roughness.
Key results
- Albedo FID on out-of-distribution GObjaVerse improves to 71.39 vs 90.12 for RGBX and 98.77 for IID.
- In the user study, 67.36% of albedo preferences go to the proposed method vs 15.63% for RGBX and 14.24% for IID.
- Rendering quality R-PQ 3.93 for the proposed method vs 2.96 for RGBX and 2.95 for IID (1-5 scale).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| directly generate PBR maps from text instead of decomposing an input RGB image | IntrinsiX vs. IID w/ FLUX-LoRA · GObjaverse (A-OOD-FID) | FID: 79.29 -> 71.39 | Helped |
| add the rendering loss (ablation) | IntrinsiX · InteriorVerse test (A-ID-FID) | FID: 78.77 -> 72.09 | Hurt |
| add the rendering loss (ablation) | IntrinsiX · GObjaverse (A-OOD-FID) | FID: 72.23 -> 71.39 | Helped |
| add the rendering loss (ablation) | IntrinsiX · Rendered-image user study | R-PQ: 3.42 {{$\pm$}0.92} -> 3.93 {{$\pm$}0.88} | Helped |
| add cross-intrinsic attention dropout (ablation) | IntrinsiX · InteriorVerse test (A-ID-FID) | FID: 71.47 -> 72.09 | Hurt |
| add cross-intrinsic attention dropout (ablation) | IntrinsiX · GObjaverse (A-OOD-FID) | FID: 75.54 -> 71.39 | Helped |
| use BRDF-sampled rather than importance-based light directions (ablation) | IntrinsiX · InteriorVerse test (A-ID-FID) | FID: 72.09 -> 75.26 | Hurt |
| use BRDF-sampled rather than importance-based light directions (ablation) | IntrinsiX · GObjaverse (A-OOD-FID) | FID: 71.39 -> 68.87 | Helped |
| increase the albedo training set from 20 to 1k examples (ablation) | Albedo LoRA · InteriorVerse test | A-ID-FID: 187.83 -> 154.58 | Helped |
| increase the albedo training set from 20 to 1k examples (ablation) | Albedo LoRA · Albedo generation | A-Diversity: 398.36 -> 366.35 | Hurt |
| use an image-conditional prior instead of the text-only base model (ablation) | IntrinsiX · GObjaverse (A-OOD-FID) | FID: 71.39 -> 70.51 | Helped |
| use uniform instead of importance-based light sampling (ablation) | IntrinsiX · PBR map sharpness | roughness/metallic detail | Hurt |
Surprising findings
- Training LoRAs on larger datasets (100+ images) causes albedos to include baked-in lighting, so only 20 samples are used for albedo/normal.
- Dropping self-attention in some blocks during training (CIA dropout) improves sharpness, and an image-conditioned variant generalizes better than direct IID training.
Limitations
- Generated PBR maps contain less detail than FLUX RGB images and normals are sometimes incorrect.
- Screen-space renderer gives limited 2D results; a neural renderer may be needed.
- Fine-tuning on small datasets sacrifices some compositional diversity.
Experimental setup
Datasets: InteriorVerse GObjaVerse 3D-FRONT
Benchmarks: InteriorVerse test set GObjaVerse Daily-Used subset
Metrics: FID user preference (A-PQ) rendering quality (R-PQ) specularity quality (S-PQ) prompt coherence (PC) diversity
Baselines: IID RGBX ColorfulShading IID w/ FLUX-LoRA
Builds on: FLUX.1-dev LoRA flow matching SceneTex Disney BRDF
Closest papers by meaning
- MeshGen: Generating PBR Textured Mesh with Render-Enhanced Auto-Encoder and Generative Data Augmentation (CVPR 2025 · similarity 0.88)
- PBR-SR: Mesh PBR Texture Super Resolution from 2D Image Priors (NeurIPS 2025 · similarity 0.87)
- Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering (CVPR 2024 · similarity 0.87)
- TexGaussian: Generating High-quality PBR Material via Octree-based 3D Gaussian Splatting (CVPR 2025 · similarity 0.87)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.87)
- ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.