Research map / 3D vision / Face and shape
SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents SHINOBI, an end-to-end neural-field framework that jointly recovers shape, Cook-Torrance BRDF, illumination and camera poses from unposed in-the-wild object image collections using hybrid hash/Fourier encoding and camera multiplex constraints.
Problem
Reconstructing 3D shape, material (BRDF) and per-image illumination from unconstrained in-the-wild object image collections with varying lighting, backgrounds and coarse, noisy camera poses.
Approach
Replace the NeRF positional encoding in SAMURAI-style joint pose-shape-BRDF optimization with a hybrid multiresolution hash grid plus annealed Fourier encoding, and stabilize camera optimization via a camera multiplex projection loss, per-view importance weighting, and patch-based silhouette/color losses.
Key results
- On NAVI with GT poses, SHINOBI reaches 27.69 PSNR (SC) and 0.0607 LPIPS vs SAMURAI's 25.34 PSNR and 0.0958 LPIPS, at about 4 hours per object versus 12 hours for SAMURAI.
- With pose estimation from coarse quadrant initialization, SHINOBI matches SAMURAI in PSNR (25.15 vs 25.34 on SC) while improving LPIPS (0.090 vs 0.096) and lowering rotation error (22.84 vs 26.16 degrees on SC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SHINOBI instead of SAMURAI with GT poses | NAVI view synthesis · NAVI SC (GT poses) | PSNR: 25.34 -> 27.69 | Helped |
| add per-view importance weighting (ablation) | SHINOBI · Keywest and School Bus (NAVI) | PSNR: 22.43 -> 25.87 | Helped |
| add coarse-to-fine annealing (ablation) | SHINOBI · Keywest and School Bus (NAVI) | PSNR: 21.47 -> 25.87 | Helped |
| add patch-based training (ablation) | SHINOBI · Keywest and School Bus (NAVI) | PSNR: 20.60 -> 25.87 | Helped |
| add camera multiplex consistency loss (ablation) | SHINOBI · Keywest and School Bus (NAVI) | PSNR: 25.80 -> 25.87 | No clear effect |
| add hybrid encoding (ablation) | SHINOBI · Keywest and School Bus (NAVI) | PSNR: 25.31 -> 25.87 | No clear effect |
Surprising findings
- Hash grid encoding alone reduces reconstruction quality and pose-optimization robustness; the Fourier/hash hybrid and patch losses are needed to recover benefits.
Limitations
- Symmetric objects and highly specular materials cause pose and reconstruction failures.
- BRDF/illumination decomposition cannot model shadows or inter-reflections.
- Struggles with thin/transparent structures and high-frequency detail under extreme illumination changes.
Experimental setup
Datasets: NAVI
Benchmarks: NAVI
Metrics: PSNR SSIM LPIPS translation error rotation error runtime
Baselines: SAMURAI NeROIC NeRS GNeRF
Builds on: SAMURAI Instant-NGP Neural-PIL BARF NeRF
Closest papers by meaning
- SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections (NeurIPS 2022 · similarity 0.90)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.87)
- Factored-NeuS: Reconstructing Surfaces, Illumination, and Materials of Possibly Glossy Objects (CVPR 2025 · similarity 0.85)
- LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields (CVPR 2025 · similarity 0.85)
- A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields (AAAI 2024 · similarity 0.85)
- ProbeSDF: Light Field Probes For Neural Surface Reconstruction (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.