Research map / Video / Video generation
Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes jointly learned multi-scale motion and appearance codebooks with transformer-based compensation to refine motion flows and warped source features for talking head video generation.
Problem
Talking head video generation from a one-shot source image struggles to model complex facial motion precisely and lacks sufficient appearance information under large pose changes, producing artifacts and detail loss.
Approach
Learn multi-scale motion-flow and appearance codebooks jointly with the whole framework, with scale-allocated code groups, and use transformer cross-attention retrieval to refine motion flow residuals and warped appearance features at each scale.
Key results
- On VoxCeleb1 same-identity reconstruction, the method reaches FID 43.15 and LPIPS 0.1846, the best among compared methods (MRFA FID 48.49, LPIPS 0.1872).
- On CelebV-HQ, the method achieves the best AKD among compared methods (3.2562 vs MRFA 3.7166) but is not best on FID (71.78 vs MCNet 78.33 and MRFA 75.73 for those rows; AniPortrait 61.56).
- Ablation on VoxCeleb1: the full model with multi-scale motion and appearance codebooks reduces FID from 47.83 (baseline) to 43.15.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ours instead of FYE | talking head generation · VoxCeleb1 same-identity reconstruction | FID: 43.25 -> 43.15 | Helped |
| Use Ours instead of MRFA | talking head generation · VoxCeleb1 same-identity reconstruction | PSNR: 25.26 -> 25.30 | Helped |
| Use Ours instead of MRFA | talking head generation · VoxCeleb1 same-identity reconstruction | AKD: 1.1823 -> 1.2039 | Hurt |
| Use Ours instead of AniPortrait | talking head generation · CelebV-HQ same-identity reconstruction | FID: 61.56 -> 71.78 | Hurt |
| Use Ours instead of MRFA | talking head generation · CelebV-HQ same-identity reconstruction | PSNR: 22.41 -> 22.40 | Hurt |
| Use Ours instead of AniPortrait | talking head generation · CelebV-HQ same-identity reconstruction | AKD: 2.2397 -> 3.2562 | Hurt |
| Add single-scale motion codebook compensation (ablation) | Baseline · VoxCeleb1 same-identity reconstruction | FID: 47.83 -> 49.00 | Hurt |
| Add single-scale motion codebook compensation (ablation) | Baseline · VoxCeleb1 same-identity reconstruction | AKD: 1.2384 -> 1.2183 | Helped |
| Add multi-scale motion codebook compensation (ablation) | Baseline · VoxCeleb1 same-identity reconstruction | FID: 47.83 -> 44.86 | Helped |
| Add multi-scale motion codebook compensation (ablation) | Baseline · VoxCeleb1 same-identity reconstruction | PSNR: 24.93 -> 25.27 | Helped |
| Add single-scale appearance codebook compensation to multi-scale motion compensation (ablation) | Baseline+MMC · VoxCeleb1 same-identity reconstruction | FID: 44.86 -> 44.32 | Helped |
| Add single-scale appearance codebook compensation to multi-scale motion compensation (ablation) | Baseline+MMC · VoxCeleb1 same-identity reconstruction | PSNR: 25.27 -> 25.16 | Hurt |
Surprising findings
- Single-scale appearance codebook compensation slightly lowers PSNR, L1, and AED, which the authors attribute to conflict with warping artifacts at other scales.
Limitations
- Diffusion-based baselines were trained on larger-scale datasets, so comparisons are not fully matched in data scale.
- Generalization is shown on only 50 CelebV-HQ test videos and FID is not best on CelebV-HQ.
Experimental setup
Datasets: VoxCeleb1 CelebV-HQ
Benchmarks: VoxCeleb1 CelebV-HQ
Metrics: FID PSNR L1 LPIPS AKD AED
Baselines: FOMM LIA DaGAN MCNet MRFA AniPortrait Follow-Your-Emoji
Builds on: VQ-VAE First Order Motion Model MRFA VQGAN
Closest papers by meaning
- Occlusion-Insensitive Talking Head Video Generation via Facelet Compensation (AAAI 2025 · similarity 0.91)
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.90)
- MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes (NeurIPS 2024 · similarity 0.89)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.89)
- AnyTalk: Multi-modal Driven Multi-domain Talking Head Generation (AAAI 2025 · similarity 0.88)
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.