Research map / 3D vision / NeRF and radiance fields
Robust Inverse Graphics via Probabilistic Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes robust inverse graphics (RIG), which performs full Bayesian posterior inference over a NeRF scene prior and a NeRF corruption model with an improper uniform prior, plus ReGAL diffusion conditioning with auxiliary latents, for single-image depth estimation under rain, cloud, and FOV corruptions.
Problem
Inferring 3D scenes from a single image robust to unknown corruptions (rain, clouds/fog, floaters, camera FOV errors) without knowing the corruption family ahead of time; MAP estimation yields degenerate 'billboard' corruption solutions.
Approach
Treat corruptions as a secondary NeRF with an uninformative uniform prior and perform full posterior inference (variational inference for ProbNeRF; ReGAL importance sampling/SMC for diffusion priors) over scene and corruption jointly, rather than MAP, which avoids billboard solutions; also introduces ReGAL, a diffusion conditioning method for models with auxiliary latents.
Key results
- On ShapeNet with rain corruption, SSDNeRF ReGAL (K=4) achieves VSD 0.25 vs 0.63 for DPT and 0.43 for ProbNeRF MAP (lower is better).
- ProbNeRF VI outperforms ProbNeRF MAP on corrupted ShapeNet (e.g., rain VSD 0.32 vs 0.43; cloud 0.45 vs 0.48; FOV 0.36 vs 0.46).
- On MultiShapeNet, DPT is competitive; ReGAL reaches VSD 0.87-0.88 on clean/rain/cloud versus DPT's 0.90-0.91, while FOV favors DPT (0.92 vs 0.95 for ReGAL, lower is better)— ReGAL is worse on FOV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full posterior variational inference instead of MAP point estimation | ProbNeRF · ShapeNet rain | VSD: 0.43 ± 0.0044 -> 0.32 ± 0.0039 | Helped |
| Use full posterior variational inference instead of MAP point estimation | ProbNeRF · ShapeNet cloud | VSD: 0.48 ± 0.0042 -> 0.45 ± 0.0043 | Helped |
| Use full posterior variational inference instead of MAP point estimation | ProbNeRF · ShapeNet FOV | VSD: 0.46 ± 0.0047 -> 0.36 ± 0.004 | Helped |
| Use full posterior variational inference instead of MAP point estimation | ProbNeRF · ShapeNet clean | PSNR: 27.08 ± 0.17 -> 27.11 ± 0.2 | No clear effect |
| Increase ReGAL importance samples from K=1 to K=8 (ablation) | SSDNeRF ReGAL · ShapeNet FOV | VSD: 0.49 ± 0.0052 -> 0.34 ± 0.0041 | Helped |
| Increase ReGAL importance samples from K=1 to K=8 (ablation) | SSDNeRF ReGAL · MultiShapeNet FOV | VSD: 0.95 ± 0.0013 -> 0.95 ± 0.0013 | No clear effect |
| Increase ReGAL importance samples from K=1 to K=8 (ablation) | SSDNeRF ReGAL · MultiShapeNet rain | PSNR: 19.01 ± 0.38 -> 15.65 ± 0.34 | Hurt |
| Use ReGAL with SSDNeRF rather than DPT | monocular depth estimation · ShapeNet rain | VSD: 0.63 ± 0.0065 -> 0.25 ± 0.0027 | Helped |
| Use ReGAL with SSDNeRF rather than DPT | monocular depth estimation · MultiShapeNet FOV | VSD: 0.92 ± 0.0031 -> 0.95 ± 0.0013 | Hurt |
| Use ReGAL with SSDNeRF rather than DPT | monocular depth estimation · MultiShapeNet clean | VSD: 0.91 ± 0.0037 -> 0.87 ± 0.0015 | Helped |
Surprising findings
- On MultiShapeNet FOV estimation, DPT (0.92) outperforms all ReGAL variants (0.95), attributed to the scale of FOV errors causing large depth changes and DPT's calibration advantage.
- MAP inference produces 'billboard' solutions where the corruption NeRF explains the scene, a degeneracy the authors prove is intrinsic to MAP under a uniform corruption prior.
Limitations
- Inference is very slow (about 7.5 to 9.5 minutes per image on an A100), prohibiting real-time use without amortized inference.
- Relies on a 3D scene prior that is hard to build for real-world scenes; the MultiShapeNet prior lacks fidelity for fine object details.
- Evaluated only on synthetic Kubric/ShapeNet renderings with a few corruption types.
Experimental setup
Datasets: ShapeNet cars MultiShapeNet
Benchmarks: ShapeNet cars (corrupted rain, cloud, FOV) MultiShapeNet (corrupted rain, cloud, FOV)
Metrics: VSD PSNR
Baselines: DPT ProbNeRF MAP
Builds on: ProbNeRF SSDNeRF reconstruction guidance RealNVP DPT
Closest papers by meaning
- ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field (NeurIPS 2024 · similarity 0.87)
- Exact Posterior Score Estimation for Solving Linear Inverse Problems (NeurIPS 2026 · similarity 0.87)
- PBR-NeRF: Inverse Rendering with Physics-Based Neural Fields (CVPR 2025 · similarity 0.87)
- Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views (AAAI 2024 · similarity 0.86)
- Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstruction (NeurIPS 2024 · similarity 0.86)
- ReconFusion: 3D Reconstruction with Diffusion Priors (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.