Research map / Video / Video generation
Real-time 3D-aware Portrait Video Relighting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a real-time, 3D-aware NeRF-based portrait video relighting method using dual tri-plane encoders (albedo and shading) plus a temporal consistency transformer network, running at 32.98 fps.
Problem
Relighting in-the-wild talking-face videos under novel lighting and viewpoints is either too slow or restricted to input viewpoints, and existing video methods flicker.
Approach
Feed-forward dual encoders predict albedo and shading tri-planes from each frame (shading conditioned on albedo and SH lighting), with a cross-attention temporal consistency network over tri-planes of neighboring frames, distilled from a pretrained EG3D-based relightable generator (LP3D-style inversion).
Key results
- Runs at 32.98 fps (30.32 ms per frame) on a single RTX 4090, the lowest time cost among compared methods.
- Lowest warping error (0.0003) and LPIPS between consecutive frames (0.0159) on INSTA, with highest identity preservation (0.5396) among compared methods.
- Achieves lowest FID (45.08) on cropped FFHQ input-view relighting versus SMFR (51.16) and Lumos (55.18).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the temporal consistency network (ablation) | Real-time 3D-aware Portrait Video Relighting · INSTA video dataset | warping error: 0.0006 -> 0.0003 | Helped |
| Use the temporal consistency network (ablation) | Real-time 3D-aware Portrait Video Relighting · INSTA video dataset | LPIPS between consecutive frames: 0.0304 -> 0.0159 | Helped |
| Use the temporal consistency network (ablation) | Real-time 3D-aware Portrait Video Relighting · INSTA video dataset | lighting error: 0.7707 -> 0.7710 | Hurt |
| Use the temporal consistency network (ablation) | Real-time 3D-aware Portrait Video Relighting · INSTA video dataset | lighting instability: 0.2526 -> 0.2533 | Hurt |
| Replace the latent-code encoder with tri-plane dual-encoders (ablation) | Real-time 3D-aware Portrait Video Relighting · portrait reconstruction | identity preservation | Helped |
Limitations
- Fails to model glares on eyeglasses.
- Does not separate motion from identity, so video-driven animation is limited.
- Lighting error is only second-best, not best, in some comparisons.
Experimental setup
Datasets: INSTA FFHQ synthetic data from pretrained generator
Benchmarks: INSTA FFHQ Lumos input-view protocol
Metrics: lighting error lighting instability identity preservation warping error LPIPS FID time cost DISTS pose error
Baselines: B-DPR B-SMFR B-E4E B-PTI SMFR DPR ReliTalk Lumos TR NVPR SIPR-W HeadNeRF
Builds on: LP3D EG3D DPR Lighting-aware NeRF portrait relighting (Jiang et al.) PTI
Closest papers by meaning
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.89)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.89)
- UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting (NeurIPS 2025 · similarity 0.87)
- High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model (CVPR 2025 · similarity 0.87)
- IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular VideosC (CVPR 2025 · similarity 0.87)
- Comprehensive Relighting: Generalizable and Consistent Monocular Human Relighting and Harmonization (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.