Research map / 3D vision / NeRF and radiance fields
LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LIRM, a feed-forward transformer that jointly reconstructs shape, materials and view-dependent radiance fields from sparse posed images in under a second, with progressive view updates, hexa-plane neural SDF, and neural directional encoding.
Problem
Existing large reconstruction models (LRMs) reconstruct radiance fields without view-dependent effects, struggle with unseen regions, cannot add more views without memory growth, and cannot decompose shape, materials and lighting into standard graphics-ready assets.
Approach
Builds on MeshLRM with (1) an update module that feeds predicted plane tokens back into the transformer so more input views can be added progressively at constant memory, (2) a hexa-plane neural SDF representation with six planes to reduce feature leakage between object sides, and (3) multiple neural directional encoding panoramas predicted by the transformer to model view-dependent effects in a f
Key results
- On GSO uniform-lighting view synthesis, LIRM-hexa 3rd stage reaches 30.65 PSNR vs 28.13 for MeshLRM and 30.52 for GS-LRM.
- On Stanford-ORB, LIRM-NDE with 3 sets of inputs achieves CD 0.31, the best shape error among compared methods including optimization-based Neural-PBIR (0.43).
- Inference takes about 0.3 seconds per step on an A100 GPU, compared with minutes to hours for optimization-based inverse rendering.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hexa-plane instead of tri-plane representation (ablation) | LIRM · GSO, uniform-lighting view synthesis | PSNR: 29.61 -> 30.65 | Helped |
| Use hexa-plane instead of tri-plane representation (ablation) | LIRM · ABO, uniform-lighting view synthesis | PSNR: 32.58 -> 32.83 | Helped |
| Add a second set of input views (ablation) | LIRM-hexa · GSO, uniform-lighting view synthesis | PSNR: 29.27 -> 30.48 | Helped |
| Add a third set of input views (ablation) | LIRM-hexa · ABO, uniform-lighting view synthesis | PSNR: 33.08 -> 32.98 | No clear effect |
| Add neural directional encoding for view-dependent radiance (ablation) | LIRM · ABO, environment-lighting view synthesis | PSNR: 29.50 -> 29.92 | Helped |
| Add neural directional encoding for view-dependent radiance (ablation) | LIRM · DTC, environment-lighting view synthesis | PSNR: 28.98 -> 29.35 | Helped |
| Increase the number of NDE panoramas through progressive updates (ablation) | LIRM-NDE · ABO, environment-lighting view synthesis | PSNR: 29.01 -> 29.92 | Helped |
| Increase the number of NDE panoramas through progressive updates (ablation) | LIRM-NDE · DTC, environment-lighting view synthesis | PSNR: 27.59 -> 29.35 | Helped |
| Add a third set of input views (ablation) | LIRM-diff · Stanford-ORB, shape reconstruction | CD: 0.48 -> 0.38 | Helped |
| Add a third set of input views (ablation) | LIRM-NDE · Stanford-ORB, shape reconstruction | CD: 0.33 -> 0.31 | Helped |
| Use NDE instead of diffuse-only radiance modeling (ablation) | LIRM · Stanford-ORB, relighting | PSNR-H: 25.09 -> 24.60 | Hurt |
| Use NDE instead of diffuse-only radiance modeling (ablation) | LIRM · Stanford-ORB, shape reconstruction | CD: 0.38 -> 0.31 | Helped |
Surprising findings
- LIRM-hexa outperforms the MeshLRM baseline even with only the first set of 4 input views, attributed to more plane tokens and the hexa-plane representation.
Limitations
- Gap between hexa-plane and tri-plane is small on ABO, possibly due to simpler textures.
- Numerical-normal loss is costly (3x feature queries), so it is used only in the third training stage.
- Full supplementary limitations were not included in the main text.
Experimental setup
Datasets: Shutterstock 3D objects (600k, with GT PBR materials) Laval HDR dataset Polyhaven HDR environments GSO ABO DTC Stanford-ORB
Benchmarks: GSO ABO DTC Stanford-ORB
Metrics: PSNR SSIM LPIPS Chamfer distance PSNR-H PSNR-L
Baselines: MeshLRM GS-LRM MetaLRM InvRender NVDiffrecMC Neural-PBIR
Builds on: MeshLRM LRM neural directional encoding NerfAcc K-Planes
Closest papers by meaning
- LRM: Large Reconstruction Model for Single Image to 3D (ICLR 2024 · similarity 0.91)
- GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement (ICLR 2025 · similarity 0.89)
- LRM-Zero: Training Large Reconstruction Models with Synthesized Data (NeurIPS 2024 · similarity 0.89)
- RelitLRM: Generative Relightable Radiance for Large Reconstruction Models (ICLR 2025 · similarity 0.88)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.87)
- ProbeSDF: Light Field Probes For Neural Surface Reconstruction (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.