Research map / 3D vision / Face and shape
Let's Chorus: Partner-aware Hybrid Song-Driven 3D Head Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PaChorus, a partner-aware framework that generates 3D head and facial animation for each singer from mixed duet audio, and releases the ChorusHead dataset of chorus singing with FLAME motion.
Problem
Existing 3D facial animation methods are driven by single-speaker or mono-singer audio and ignore interaction between singers in a mixed choral/duet recording with background music.
Approach
A two-branch design where the host singer's vocal drives lip and facial coefficients, while a CVAE-based partner-aware motion prior conditioned on the partner's vocal and background music predicts residual head pose, letting partners' audio shape head movement.
Key results
- PaChorus reaches LVE 1.1922, FVE 1.3083, FDD 1.2314 on ChorusHead, beating the best fine-tuned baseline (FaceFormer LVE 1.5883, FDD 2.1 range for DiffSpeaker 2.2121).
- Ablations show removing the partner-aware prior raises FDD from 1.2314 to 1.3421 and removing BGM raises PPE from 0.0948 to 0.1295.
- In an AMT user study with 30 participants, PaChorus win rates were 78.6% to 93.3% against baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PaChorus instead of the strongest finetuned baseline, Diffspeaker | 3D singing animation generation · ChorusHead | LVE: 1.5833 -> 1.1922 | Helped |
| Use PaChorus instead of the strongest finetuned baseline, Imitator | 3D singing animation generation · ChorusHead | FVE: 1.4219 -> 1.3083 | Helped |
| Use PaChorus instead of the strongest finetuned baseline, SelfTalk | 3D singing animation generation · ChorusHead | FDD: 1.9458 -> 1.2314 | Helped |
| Add background music input (ablation) | PaChorus · ChorusHead | PPE: 0.1295 -> 0.0948 | Helped |
| Add KL-divergence loss (ablation) | PaChorus CVAE · ChorusHead | PPE: 0.1461 -> 0.0948 | Helped |
| Add partner-aware motion prior (ablation) | PaChorus · ChorusHead | PPE: 0.1246 -> 0.0948 | Helped |
| Add partner-aware motion prior (ablation) | PaChorus · ChorusHead | LVE: 1.2409 -> 1.1922 | Helped |
Surprising findings
- Pretrained talking-head models perform worse than fine-tuned ones, which the authors attribute to a domain gap between speech and singing.
Limitations
- Audio separation tool only reliably handles two singers, so the dataset and training are limited to duets.
- Multi-singer demonstration uses re-recorded covers rather than real separated tracks.
- Dataset release is promised after publication rather than given as a link.
Experimental setup
Datasets: ChorusHead
Benchmarks: ChorusHead
Metrics: LVE FVE FDD PPE user study win rate
Baselines: FaceFormer CodeTalker Imitator SelfTalk DiffSpeaker
Builds on: FaceFormer wav2vec 2.0 FLAME CVAE EMOCA v2
Closest papers by meaning
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
- Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style (AAAI 2024 · similarity 0.87)
- KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation (CVPR 2025 · similarity 0.87)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.86)
- Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation (CVPR 2025 · similarity 0.86)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.