Research map / 3D vision / 3D generation
3DFIRES: Few Image 3D REconstruction for Scenes with Hidden Surfaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3DFIRES, a feed-forward system that fuses multi-view image features in feature space to predict a Directed Ray Distance Function, reconstructing full scene geometry including hidden surfaces from one or a few posed images.
Problem
Scene-level 3D reconstruction of both visible and occluded surfaces from sparse, wide-baseline posed images, with consistency across views.
Approach
Extends single-view DRDF to multiple views by attention-based fusion of per-camera pixel-aligned features with query-ray/camera geometric encodings, rather than fusing predicted point clouds, plus Gaussian sampling of points near ray intersections during training.
Key results
- On 3-view inputs, 3DFIRES improves consistency to 85.48 vs 76.44 for SV-DRDF on Scene consistency metric (>9% absolute gain on 3-view case cited in text).
- On 3-view inputs, all-point F-score is 66.56 for 3DFIRES vs 65.61 for SV-DRDF and 54.78 for MCC.
- With 1 view, 3DFIRES matches SV-DRDF (all-point F-score 65.71 vs 65.21), and it generalizes to 5 views without training on them (all F-score 71.41).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse image features across views instead of predicting each view independently | 3DFIRES · 3 views | Consistency: 76.44 -> 85.48 | Helped |
| Fuse image features across views instead of predicting each view independently | 3DFIRES · 5 views | Consistency: 78.13 -> 85.92 | Helped |
| Fuse image features across views instead of predicting each view independently | 3DFIRES · 3 views | All points F-score: 65.61 -> 66.56 | Helped |
| Use feature fusion rather than independent per-view prediction | 3DFIRES · 1 view | All points F-score: 65.21 -> 65.71 | No clear effect |
| Use LoFTR-estimated poses with ground-truth translation scale | 3DFIRES · 3 views, LoFTR poses | Consistency: 57.65 -> 65.71 | Helped |
| Remove Gaussian sampling near ray intersections (ablation) | 3DFIRES · Training strategy ablation | Hidden F-score: 50.20 -> 43.07 | Hurt |
| Remove Gaussian sampling near ray intersections (ablation) | 3DFIRES · Training strategy ablation | All points F-score: 66.46 -> 64.81 | Hurt |
| Add ray attention without Gaussian sampling (ablation) | 3DFIRES · Training strategy ablation | Hidden F-score: 50.20 -> 47.09 | Hurt |
| Add ray attention without Gaussian sampling (ablation) | 3DFIRES · Training strategy ablation | All points F-score: 66.46 -> 65.58 | Hurt |
| Add ray attention while retaining Gaussian sampling (ablation) | 3DFIRES · Training strategy ablation | Hidden F-score: 50.20 -> 14.85 | Hurt |
| Add ray attention while retaining Gaussian sampling (ablation) | 3DFIRES · Training strategy ablation | Consistency: 85.45 -> 33.56 | Hurt |
Surprising findings
- Combining ray attention with Gaussian sampling makes the network exploit a train-time shortcut, collapsing test performance (F-score 13.29).
- Single-view DRDF baseline inconsistency leads to doubled walls and ceilings when aggregating independent per-view predictions.
Limitations
- Requires accurate camera poses from off-the-shelf estimators; noisy LoFTR poses reduce feature alignment quality.
- Relies on ground-truth translation scale in the noisy-pose experiment since LoFTR gives no scale.
- Evaluated only on Omnidata/Gibson-derived indoor scenes.
Experimental setup
Datasets: Omnidata (Gibson / Taskonomy Medium scenes)
Benchmarks: Omnidata held-out test scenes (Gibson/Taskonomy)
Metrics: Scene F-score (rho=0.2) multiview consistency accuracy completeness
Baselines: Depth only (MiDaS) MCC SV-DRDF LoFTR-based pose variant
Builds on: Directed Ray Distance Function (DRDF) MiDaS PixelNeRF Multiview Compressive Coding
Closest papers by meaning
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.86)
- SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Images (CVPR 2025 · similarity 0.86)
- Coherent 3D Scene Diffusion From a Single RGB Image (NeurIPS 2024 · similarity 0.85)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.85)
- Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses (CVPR 2024 · similarity 0.85)
- LIRM: Large Inverse Rendering Model for Progressive Reconstruction of Shape, Materials and View-dependent Radiance Fields (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.