Research map / Video / Video generation
Occlusion-Insensitive Talking Head Video Generation via Facelet Compensation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a facelet semantic bank and facelet compensation module that repopulates occluded or profile-pose facial regions in warped features for occlusion-insensitive talking head video generation.
Problem
Warping-based talking head generation assumes facial keypoints keep consistent relative positions, which fails under self-occlusion and profile poses, causing identity inconsistency and blurring.
Approach
Replace spatial keypoint/flow correlation with semantic correlation: a spatially-free facelet semantic bank (per-region tokens from face parsing) is learned from source features and used via cross-attention to compensate occluded regions in warped features, with a localization loss aligning attention to target face masks and a spatially-invariant pose code from a 3D shape model.
Key results
- Achieves SSIM 0.896, PSNR 26.88, LPIPS 0.136 on same-identity VoxCeleb1, outperforming MRFA (0.889, 26.26, 0.143).
- On the 1,300-pair self-occlusion set, reaches SSIM 0.837 and AED 0.207 versus TPSM's 0.829 and 0.220.
- Cross-identity CelebV CSIM 0.711 and ARD 2.265 are best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full facelet-compensation method instead of MRFA | Talking head generation · VoxCeleb1 same-identity reenactment | SSIM: 0.889 -> 0.896 | Helped |
| Use the full facelet-compensation method instead of MRFA | Talking head generation · VoxCeleb1 same-identity reenactment | L1: 0.0329 -> 0.0306 | Helped |
| Use the full facelet-compensation method instead of MRFA | Talking head generation · VoxCeleb1 same-identity reenactment | LPIPS: 0.143 -> 0.136 | Helped |
| Use the full method instead of the strongest baseline | Cross-identity talking head generation · VoxCeleb1 | CSIM: 0.759 -> 0.779 | Helped |
| Use the full method instead of the strongest baseline | Cross-identity talking head generation · VoxCeleb1 | ARD: 2.206 -> 2.124 | Helped |
| Use the full method instead of the strongest baseline | Cross-identity talking head generation · CelebV | AUH: 0.215 -> 0.201 | Helped |
| Use the full method instead of MRFA | Talking head generation · VoxCeleb1 self-occlusion benchmark | SSIM: 0.823 -> 0.837 | Helped |
| Add Pose-Aware Motion (PAM) to the baseline (ablation) | Proposed method · VoxCeleb1 same-identity reenactment | L1: 0.0329 -> 0.0326 | Helped |
| Add Pose-Aware Decoder (PAD) after PAM (ablation) | Proposed method · VoxCeleb1 same-identity reenactment | AKD: 1.172 -> 1.168 | Helped |
| Add Identity Refinement (IR) after PAD (ablation) | Proposed method · VoxCeleb1 same-identity reenactment | AED: 0.108 -> 0.099 | Helped |
| Add localization loss (Lloc) after IR (ablation) | Proposed method · VoxCeleb1 same-identity reenactment | SSIM: 0.895 -> 0.896 | Helped |
| Integrate the proposed components into FOMM (ablation) | FOMM · VoxCeleb1 same-identity reenactment | SSIM: 0.851 -> 0.881 | Helped |
Limitations
- Facial shape may be slightly altered to match the driving frame, especially in cross-identity animation.
- Relies on keypoints that capture shape information, leading to deformations beyond expression.
Experimental setup
Datasets: VoxCeleb1 CelebV
Benchmarks: VoxCeleb1 CelebV
Metrics: SSIM L1 PSNR LPIPS AKD AED CSIM ARD AUH
Baselines: FOMM MRAA DaGAN TPSM MRFA FNeVR MCNet
Builds on: FOMM TPSM MRFA MCNet DaGAN
Closest papers by meaning
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.91)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- FG-EmoTalk: Talking Head Video Generation with Fine-Grained Controllable Facial Expressions (AAAI 2024 · similarity 0.88)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.87)
- PTUS: Photo-Realistic Talking Upper-Body Synthesis via 3D-Aware Motion Decomposition Warping (AAAI 2024 · similarity 0.87)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.