Research map / 3D vision / 3D generation
IDArb: Intrinsic Decomposition for Arbitrary Number of Input Views and Illuminations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IDArb, a diffusion-based feed-forward model that jointly decomposes albedo, normal, metallic and roughness from an arbitrary number of multi-illumination input views, trained on a new 5.7M-image ARB-Objaverse dataset.
Problem
Intrinsic decomposition (albedo, normal, metallic, roughness) from images is ambiguous under varying lighting, and existing learning-based single-image methods produce inconsistent results across multiple views, while optimization-based methods are slow.
Approach
Fine-tunes Stable Diffusion to denoise three intrinsic triplets (albedo, normal, metallic+roughness) with cross-view and cross-component attention (adapted from Wonder3D), plus illumination-augmented sampling and a view-adapted multi/single-image training schedule so it handles any number of views and lightings.
Key results
- On synthetic data, Ours(multi) reaches albedo PSNR 33.62 and SSIM 0.937, normal cosine similarity 0.941, metallic MSE 0.016 and roughness MSE 0.033, beating IID, RGB↔X and IntrinsicAnything.
- Photometric stereo on NeRFactor with 8 OLAT images gives albedo PSNR 25.15 and normal 0.85, the best among compared methods including SDM-UniPS.
- Using IDArb decompositions as pseudo-labels improves NVDiffRecMC on Synthetic4Relight: relighting 24.05 to 31.01 and raw albedo 17.03 to 26.42.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multiple input views instead of processing images independently | IDArb · Synthetic data | Metallic MSE: 0.037 -> 0.016 | Helped |
| use multiple input views instead of processing images independently | IDArb · Synthetic data | Albedo PSNR: 32.79 -> 33.62 | Helped |
| add cross-component attention (ablation) | IDArb · Intrinsic decomposition, particularly metallic and roughness | material ambiguity reduction | Helped |
| interleave single-image and multi-view training instead of training exclusively on multi-view inputs (ablation) | IDArb · Single-image inputs | decomposition performance | Helped |
| shift the noise scheduler toward higher noise levels (ablation) | IDArb · Metallic and roughness prediction | prediction performance | Helped |
| increase the number of viewpoints from 1 to 8 with 1 lighting condition (ablation) | IDArb · Synthetic data | Albedo performance: 29.16 -> 30.49 | Helped |
| increase lighting conditions from 1 to 3 with 1 viewpoint (ablation) | IDArb · Synthetic data | Metallic MSE: 0.105 -> 0.061 | Helped |
| add IDArb intrinsic decompositions as pseudo-material labels | NVDiffRecMC · NeRFactor | Relighting: 22.65 -> 27.20 | Helped |
| add IDArb intrinsic decompositions as pseudo-material labels | NVDiffRecMC · Synthetic4Relight | Roughness: 0.046 -> 0.014 | Helped |
Surprising findings
- Training only on multi-view inputs degrades single-image decomposition, motivating the view-adapted interleaved training.
- Performance gains from adding more viewpoints diminish beyond eight views.
Limitations
- Cross-view attention concatenates all views, giving O(N^2) cost that limits dense high-resolution inputs.
- Struggles with outdoor scenes, text, and fine-grained material details since training data is object-centric and synthetic with simple global material values.
- Complex real-world materials such as corroded bronze with spatially varying metallic and roughness are not handled well.
Experimental setup
Datasets: ARB-Objaverse G-Objaverse ABO Objaverse Poly Haven HDR environment maps Pixabay images NeRFactor Synthetic4Relight MIT-Intrinsic Stanford-ORB OpenIllumination NeRD
Benchmarks: ARB-Objaverse/G-Objaverse synthetic evaluation MIT-Intrinsic Stanford-ORB NeRFactor Synthetic4Relight Pixabay real-world images
Metrics: SSIM PSNR cosine similarity MSE LPIPS
Baselines: IID RGB↔X IntrinsicAnything GeoWizard SDM-UniPS NVDiffRecMC StableNormal IntrinsicNeRF
Builds on: Stable Diffusion Wonder3D Objaverse RGB↔X
Closest papers by meaning
- Intrinsic Image Diffusion for Indoor Single-view Material Estimation (CVPR 2024 · similarity 0.90)
- Material Palette: Extraction of Materials from a Single Image (CVPR 2024 · similarity 0.88)
- Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion (CVPR 2025 · similarity 0.88)
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior (CVPR 2025 · similarity 0.87)
- MAGE : Single Image to Material-Aware 3D via the Multi-View G-Buffer Estimation Model (CVPR 2025 · similarity 0.86)
- Latent Intrinsics Emerge from Training to Relight (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.