Research map / 3D vision / 3D generation
ReCap: Better Gaussian Relighting with Cross-Environment Captures
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReCap, a Gaussian relighting method that jointly optimizes multiple learnable environment maps sharing one set of material attributes, using cross-environment object captures to resolve albedo-lighting ambiguity.
Problem
Relighting 3D Gaussian Splatting objects in unseen environments is unfaithful because of albedo-lighting ambiguity: with single-environment captures, learned environment maps absorb residual terms and do not match real lighting.
Approach
Treat captures under multiple unknown environments as multi-task targets: each environment gets its own learnable cube-map lighting, all sharing one Gaussian material field, plus a split-sum shading function with a free specular tint vector (no metallic parameter) and clipping plus gamma post-processing so learned maps lie in a linear HDR space.
Key results
- Dual-environment ReCap reaches 25.82 average unseen-relight PSNR versus 24.49 for TensoIR with GT albedo scaling (22.33 for R3DG single-env baseline, 21.22 for 3DGS-DR).
- Ablation: the proposed shading function raises average relight PSNR from 24.62 to 25.29, and adding the saturation and energy losses reaches 25.58.
- Increasing the number of training environments from 1 to 5 raises relight PSNR from 23.98 to 27.36 on the three held-out scenes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dual-environment captures with shared material attributes and independent lighting maps | ReCap · Unseen relights (dual-env), Avg. | PSNR: 24.49 -> 25.82 | Helped |
| Use dual-environment captures with shared material attributes and independent lighting maps | ReCap · Unseen relights (dual-env), Avg. | SSIM: 0.906 -> 0.930 | Helped |
| Use dual-environment captures with shared material attributes and independent lighting maps | ReCap · Unseen relights (dual-env), Avg. | LPIPS: 0.090 -> 0.068 | Helped |
| Replace the original shading function with the proposed shading function (ablation) | ReCap · Unseen relights | PSNR: 24.62 -> 25.29 | Helped |
| Add the specular saturation loss (ablation) | ReCap with proposed shading · Unseen relights | PSNR: 25.29 -> 25.38 | Helped |
| Add the energy conservation loss (ablation) | ReCap with proposed shading and Lsat · Unseen relights | PSNR: 25.38 -> 25.58 | Helped |
| Add extra unique views within the same environment (ablation) | ReCap · Unseen relights (sunset, fireplace, night) | PSNR: 23.98 -> 24.07 | No clear effect |
| Add a second environment with identical camera poses (ablation) | ReCap · Unseen relights (sunset, fireplace, night) | PSNR: 24.07 -> 26.14 | Helped |
| Use extra unique views in the second environment instead of identical camera poses (ablation) | ReCap · Unseen relights (sunset, fireplace, night) | PSNR: 26.14 -> 26.25 | No clear effect |
| Increase the training environments from 2 to 3 (ablation) | ReCap · Unseen relights (sunset, fireplace, night) | PSNR: 26.25 -> 26.35 | Helped |
| Increase the training environments from 3 to 4 (ablation) | ReCap · Unseen relights (sunset, fireplace, night) | PSNR: 26.35 -> 26.68 | Helped |
| Increase the training environments from 4 to 5 (ablation) | ReCap · Unseen relights (sunset, fireplace, night) | PSNR: 26.68 -> 27.36 | Helped |
Surprising findings
- Replacing learned environment maps with ground-truth HDR maps degrades NVS quality, showing that learned maps act as sinks for residual terms.
- Directly optimizing existing Gaussian models over dual-environment inputs with a single lighting representation yields mixed relighting and significant NVS artifacts.
Limitations
- Indirect illumination and subsurface scattering are not modeled.
- Over-exposed highlights cause opacity estimation errors in Gaussian relighting methods.
- Requires pose calibration across environments, which is non-trivial for real captures.
Experimental setup
Datasets: NeRF Synthetic Shiny Blender StanfordORB
Benchmarks: RelightObj StanfordORB
Metrics: PSNR SSIM LPIPS
Baselines: 3DGS-DR GS-IR R3DG GShader TensoIR
Builds on: 3D Gaussian Splatting split-sum approximation GShader GS-IR TensoIR
Closest papers by meaning
- MetaGS: A Meta-Learned Gaussian-Phong Model for Out-of-Distribution 3D Scene Relighting (NeurIPS 2025 · similarity 0.90)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.90)
- Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation (CVPR 2025 · similarity 0.90)
- ROGR: Relightable 3D Objects using Generative Relighting (NeurIPS 2025 · similarity 0.89)
- IllumiNeRF: 3D Relighting Without Inverse Rendering (NeurIPS 2024 · similarity 0.88)
- Neural Gaffer: Relighting Any Object via Diffusion (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.