Research map / 3D vision / Depth and stereo
Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Factored-NeuS is a three-stage implicit inverse-rendering method that jointly uses volume and surface rendering with diffuse/specular decomposition to reconstruct SDF geometry, then materials and lighting, for glossy objects from posed multi-view images.
Problem
Recovering surface geometry, materials (BRDF), and illumination from posed multi-view images of glossy objects, where specular highlights corrupt geometry and diffuse/specular decomposition.
Approach
Jointly optimize volume rendering and surface rendering, splitting radiance into diffuse and normal-conditioned specular components in both, so highlights do not bend geometry; then distill continuous light visibility and indirect illumination from the SDF and add a learnable specular albedo to the Spherical Gaussian BRDF.
Key results
- On SK3D and DTU glossy scenes, Factored-NeuS achieves mean Chamfer distance 1.55 (SK3D) and 0.87 (DTU high-specularity scenes), best among compared methods except Geo-NeuS (PC) on DTU.
- On the IndiSG dataset, mean albedo PSNR of 26.28 and rendering PSNR of 35.41, versus IndiSG's 23.63 and 26.51.
- Ablation on the Hotdog scene shows diffuse albedo PSNR rising from 26.44 (IndiSG) to 30.76 with the full method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Factored-NeuS instead of NeuS for glossy surface reconstruction | Factored-NeuS · SK3D glossy scenes (mean) | Chamfer distance: 2.31 -> 1.55 | Helped |
| Use Factored-NeuS instead of NeuS for surface reconstruction | Factored-NeuS · DTU scenes (mean) | Chamfer distance: 0.83 -> 0.76 | Helped |
| Use Factored-NeuS instead of Geo-NeuS with point-cloud supervision | Factored-NeuS · DTU glossy scenes (mean) | Chamfer distance: 0.73 -> 0.87 | Hurt |
| Use Factored-NeuS instead of NVDiffrec | Factored-NeuS · IndiSG dataset (mean) | albedo PSNR: 20.41 -> 26.28 | Helped |
| Use Factored-NeuS instead of NVDiffrec | Factored-NeuS · IndiSG dataset (mean) | rendering PSNR: 29.51 -> 35.41 | Helped |
| Use Factored-NeuS instead of IndiSG | Factored-NeuS · IndiSG dataset (mean) | illumination PSNR: 22.47 -> 21.21 | Hurt |
| Add joint volume and decomposed surface rendering (ablation) | Factored-NeuS stage 1 · SK3D Pot | Chamfer distance: 2.01 -> 1.88 | Helped |
| Add diffuse/specular decomposition to volume rendering (ablation) | Factored-NeuS stage 1 · SK3D Pot | Chamfer distance: 1.91 -> 1.88 | Helped |
| Add indirect illumination (ablation) | Factored-NeuS · IndiSG dataset (mean) | rendering PSNR: 31.51 -> 35.41 | Helped |
| Add specular albedo improvement (ablation) | Factored-NeuS · IndiSG dataset (mean) | illumination PSNR: 23.00 -> 21.21 | Hurt |
| Add specular albedo improvement (ablation) | Factored-NeuS · Hotdog | roughness PSNR: 18.83 -> 23.10 | Helped |
| Add continuous SDF-based light-visibility supervision (ablation) | Factored-NeuS · Hotdog | roughness PSNR: 16.98 -> 17.86 | Helped |
Surprising findings
- Geo-NeuS with extra SfM point cloud supervision beats the method on DTU Chamfer distance, though the method wins on glossy SK3D scenes.
- Specular albedo improvement boosts roughness and rendering quality but lowers illumination reconstruction quality in the ablation.
Limitations
- Decomposing materials for fine structures is noted as an open challenge.
- Geo-NeuS with extra point clouds performs better on DTU, so the method does not dominate all regular scenes without extra data.
Experimental setup
Datasets: SK3D DTU Shiny Glossy IndiSG dataset
Benchmarks: SK3D DTU Shiny Glossy IndiSG dataset
Metrics: Chamfer distance PSNR
Baselines: NeuS Geo-NeuS NeRO PhySG IndiSG NVDiffrec Ref-NeRF Ref-NeuS UniSDF
Builds on: NeuS Geo-NeuS Ref-NeRF IndiSG PhySG NeRFactor
Closest papers by meaning
- Inverse Rendering of Glossy Objects via the Neural Plenoptic Function and Radiance Fields (CVPR 2024 · similarity 0.91)
- Eclipse: Disambiguating Illumination and Materials using Unintended Shadows (CVPR 2024 · similarity 0.90)
- RobIR: Robust Inverse Rendering for High-Illumination Scenes (NeurIPS 2024 · similarity 0.89)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.89)
- Latent Intrinsics Emerge from Training to Relight (NeurIPS 2024 · similarity 0.89)
- Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.