Research map / 3D vision / NeRF and radiance fields
Seeing the World through Your Eyes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs a NeRF-based radiance field of the scene beyond the camera's line of sight from eye (cornea) reflections in portrait video frames, jointly optimizing cornea poses, a 2D iris texture field, and a radial regularizer.
Problem
Recovering the 3D radiance field of the scene a person is observing from reflections in their eyes captured by a stationary camera, which is hard due to inaccurate cornea pose estimates in small eye images and entanglement of iris texture with scene reflections.
Approach
Treat the cornea as a mirror with a known ellipsoid geometry, trace reflected rays to train a NeRF, and jointly learn a shared 2D iris texture field plus 6DoF cornea pose refinement, regularized by a radial-symmetry loss on the iris texture.
Key results
- Pose optimization keeps novel view synthesis quality robust to up to about 1% eye-center error, 2.5% radius error, or 5 degrees rotation error in simulation.
- Iris texture decomposition improves LPIPS on synthetic scenes (e.g., 0.39 to 0.56 for Barbershop Front... reported as with vs. without in Figure 7).
- Pose optimization is identified as the most critical component for coherent reconstruction on real captures.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add iris texture decomposition (ablation) | radiance field reconstruction from eye reflections · Barbershop Front | LPIPS: 0.39 -> 0.32 | Helped |
| add iris texture decomposition (ablation) | radiance field reconstruction from eye reflections · Barbershop Back | LPIPS: 0.56 -> 0.51 | Helped |
| add iris texture decomposition (ablation) | radiance field reconstruction from eye reflections · Classroom | LPIPS: 0.7 -> 0.67 | Helped |
| add iris texture decomposition (ablation) | radiance field reconstruction from eye reflections · Kitchen | LPIPS: 0.49 -> 0.4 | Helped |
| add iris texture decomposition (ablation) | radiance field reconstruction from eye reflections · Living Room | LPIPS: 0.39 -> 0.34 | Helped |
| optimize cornea poses during training (ablation) | radiance field reconstruction from eye reflections · real-world captures | reconstruction quality | Helped |
| add radial regularization to the iris texture (ablation) | radiance field reconstruction from eye reflections · scenes with limited disparity across views | reconstruction quality | Helped |
Surprising findings
- Naively training a radiance field on eye reflections is insufficient; iris texture entanglement creates floaters and low-disparity scene regions get absorbed into the iris texture.
Limitations
- Real-world results come from controlled laboratory captures with deliberate head movement and zoomed-in faces.
- Unconstrained settings such as video conferencing are challenging due to lower resolution, compression, and motion blur.
- PSNR is not computed because of large intensity differences between reflection and scene.
Experimental setup
Datasets: synthetic Blender eye-reflection scenes real captured portrait frames (Sony RX IV)
Benchmarks: synthetic Blender eye-reflection scenes
Metrics: LPIPS SSIM
Baselines: Orca Nerfren
Builds on: NeRF nerfstudio Nishino and Nayar eye catadioptric model Orca Nerfren
Closest papers by meaning
- ORCa: Glossy Objects As Radiance-Field Cameras (CVPR 2023 · similarity 0.87)
- Humans As Light Bulbs: 3D Human Reconstruction From Thermal Reflection (CVPR 2023 · similarity 0.85)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.84)
- UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections (NeurIPS 2024 · similarity 0.83)
- IRIS: Inverse Rendering of Indoor Scenes from Low Dynamic Range Images (CVPR 2025 · similarity 0.83)
- GNeRP: Gaussian-guided Neural Reconstruction of Reflective Objects with Noisy Polarization Priors (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.