Research map / Video / Video generation
High-Fidelity Relightable Monocular Portrait Animation with Lighting-Controllable Video Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LCVD, a lighting-controllable video diffusion model that animates a reference portrait with driving-video pose and user-specified lighting by separating intrinsic and extrinsic features via shading and reference adapters on Stable Video Diffusion.
Problem
Relightable portrait animation: animating a static reference portrait to follow a driving video's head pose while controlling its lighting, which existing portrait animation methods cannot do because identity/appearance and pose/lighting features are entangled.
Approach
Render shading hints from the reference mesh, driving pose, and target spherical-harmonics lighting; map the hints (extrinsic) and reference image (intrinsic) through two adapters into separate feature subspaces of a pretrained I2V diffusion model, then merge them and use multi-condition classifier-free guidance to control lighting strength.
Key results
- On HDTF relighting, LCVD achieves FVD 273.3 vs 403.0 for the second-best (DPR), a 32% improvement, and FID 37.46 vs 44.57.
- On cross-identity animation (HDTF), LCVD reaches FID 49.11 vs 69.85 for AnimateAnyone and ID 0.876 vs 0.806.
- User study with 17 participants gives LCVD the top scores on LA-User (3.534), ID-User (4.000), and VQ-User (3.398) for relighting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LCVD instead of the strongest listed baseline | HDTF portrait relighting · HDTF | IQ: 2.646 -> 3.034 | Helped |
| use LCVD instead of the strongest listed baseline | HDTF portrait relighting · HDTF | ID: 0.730 -> 0.585 | Hurt |
| use LCVD instead of the strongest listed baseline | FFHQ portrait relighting · FFHQ | ID: 0.687 -> 0.765 | Helped |
| use LCVD instead of the strongest listed baseline | FFHQ portrait relighting · FFHQ | LE: 0.749 -> 0.938 | Hurt |
| use LCVD instead of the strongest listed baseline | HDTF cross-identity portrait animation · HDTF | FID: 69.85 -> 49.11 | Helped |
| add portrait mask during training (ablation) | LCVD · HDTF portrait relighting | LE: 1.090 -> 0.738 | Helped |
| add portrait mask during training (ablation) | LCVD · HDTF portrait relighting | IQ: 1.781 -> 3.034 | Helped |
| use both shading and reference adapters instead of either adapter alone (ablation) | LCVD portrait generation · portrait generation under relighting and animation conditions | identity, pose and lighting alignment | Helped |
| increase multi-condition guidance strength ω (ablation) | LCVD · portrait relighting | lighting alignment | Helped |
| increase multi-condition guidance strength ω (ablation) | LCVD · portrait relighting | identity retention and image quality | Hurt |
| use only the shading adapter (ablation) | LCVD · portrait generation | pose and lighting transfer | Helped |
| use only the reference adapter (ablation) | LCVD · portrait generation | identity and appearance preservation | Helped |
Surprising findings
- Relighting ID score is not the best (0.585 vs DPR 0.730), which the authors attribute to per-frame relighting in baselines versus animation-based relighting in LCVD.
Limitations
- Identity preservation drops at higher guidance strength, trading identity for lighting strength.
- Trained on video datasets, so slightly worse lighting error and FID than methods trained on FFHQ on FFHQ images.
Experimental setup
Datasets: CelebV-HQ VFHQ HDTF FFHQ
Benchmarks: HDTF FFHQ
Metrics: LE ID LPIPS IQ VQ FID FVD POSE user study scores (LA, ID, VQ)
Baselines: DPR SMFR NFL StyleFlow DiFaReli DaGAN StyleHEAT AnimateAnyone
Builds on: Stable Video Diffusion DECA Composer AnimateAnyone
Closest papers by meaning
- Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset (CVPR 2025 · similarity 0.89)
- RealPortrait: Realistic Portrait Animation with Diffusion Transformers (AAAI 2025 · similarity 0.88)
- SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces (CVPR 2025 · similarity 0.88)
- HunyuanPortrait: Implicit Condition Control for Enhanced Portrait Animation (CVPR 2025 · similarity 0.88)
- Real-time 3D-aware Portrait Video Relighting (CVPR 2024 · similarity 0.87)
- IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.